Filed bug 73396 for the "logs don't make it to logstash" issue.

Node's docs claim that they already use SO_REUSEADDR:

http://nodejs.org/api/net.html#net_server_listen_port_host_backlog_callback

This makes our EADDRINUSE errors very mysterious.

However, the 'cluster' package does mysterious things: http://nodejs.org/api/cluster.html#cluster_how_it_works

It's possible that's what's going wrong here, somehow?

Given that we have had this code for a long time, and also given that we have also had a couple deploys post node 0.10 upgrade, and that this is the first deploy after we enabled timeouts, I would say this is related to timeouts (5 mins).

Possibly we're not shutting down cleanly when the service is stopped, and the zombie process continues holding on to the port until the timeout expires.

So, this is the code that we use to cleanly terminate the workers (see api/server.js)

cluster.disconnect(function() {
        logger.log( "info", "exiting" );
        process.exit(0);
});

This waits for worker to shut down. However, in the common case, the title should get processed fairly quicly and clear the timeout => all the workers should shut down in a timely fashion. I don't think (or at least, I would be surprised if it were) all those cases of hung processes that do indeed take 5 mins and eventually get killed by the timeout.

So, one question I now have is if we are holding onto timeouts after request processing and not clearing those out cleanly in all cases.

Well, it was 3-4 of the parsoid hosts which had this problem. So it's possible that these were hung processes which needed to wait for the timeout.

But in this case I would expect 'service parsoid stop' not to actually complete until the timeout had finished and the process had actually stopped. The problem was that the service was restarted while the old parsoid was still running (or so it would seem).

Ah, if 3-4 then maybe that is what it is then. Ping gwicke about the service restart part then. He might have a clue what is going on.

This might actually be the same problem I was dealing with quite a while ago.

There are some race conditions in upstart that are related to its hacky handling of forks & it's use of ptrace rather than cgroups. IIRC it sometimes thinks one of the workers is the leader process if a bunch of workers were re-forked since the last full restart. The same problem doesn't happen with the init script (start-stop-daemon really) & systemd. Fortunately, we are moving to Debian & are getting rid of upstart.

We have a fine init script that works both on Ubuntu & Debian in the deploy repo (debian/parsoid.init IIRC). We could plug that into puppet if this gets too annoying.

Can someone test this by using the init script on one instance to see if this still happens?

In T75395#807879, @yuvipanda wrote:

Can someone test this by using the init script on one instance to see if this still happens?

@yuvipanda: We don't have root on those boxes, so that would need to be somebody in ops.

In T75395#834249, @GWicke wrote:

@yuvipanda: We don't have root on those boxes, so that would need to be somebody in ops.

See also: https://gerrit.wikimedia.org/r/#/c/182585/

In T75395#956056, @GWicke wrote:

In T75395#834249, @GWicke wrote:

@yuvipanda: We don't have root on those boxes, so that would need to be somebody in ops.

See also: https://gerrit.wikimedia.org/r/#/c/182585/

@GWicke, now that the perms have landed can you poke at this?

@chasemp, yes.

cool, so since it looks like ops was roped in for perms stuff and that is resolved. I'm removing us, but only so we don't keep bugging you. If something comes up let us know :)

I think this problem may have been mitigated with this: https://gerrit.wikimedia.org/r/#/c/182644/ .. Last few deploys, I haven't see any stuck processes that needed explicit kills.

The eventual move to jessie & systemd will further improve this, as systemd systematically tracks process groups with cgroups.

I hope that we can create a generic systemd unit for node services, leveraging the default interface provided by https://github.com/wikimedia/service-runner. See T89901.