Fenari has a script for doing this: /home/wikipedia/bin/apache-graceful-all

No point reinventing the wheel

It's cool fenari has it, but as long as it's not open source which is publicly available it's not of much use...

it doesn't really need to be open source, but if that code was public it would be enough

http://noc.wikimedia.org/~reedy/apache-graceful-all

http://noc.wikimedia.org/~reedy/apache-graceful

http://noc.wikimedia.org/~reedy/apache-sanity-check

(In reply to comment #2)

It's cool fenari has it, but as long as it's not open source which is publicly
available it's not of much use...

It was directed at Antoine, as he does

Bleh, this is a crappy script dependency tree...

The problem is not that much about having the apache-graceful scripts but more about setting up a system that would let us ssh on the other hosts.

Currently dsh is of no use :-/

We might want to import/reuse the apache* production scripts in /home/wikipedia/bin

We should first have a list of all apaches. Maybe there's a ldap magic command we could use?

dsh does support netgroups which could be provided through LDAP. But that would require a LDAP schema change.

Meanwhile, we will have to edit and maintain some flat file in /etc/dsh/group or something.

damian wrote:

Pulling the data from LDAP into a static file via a script could work. In theory you could do it based off puppet classes, for not-yet-puppetized things you'd have to go off hostname formats or such.

Assuming you don't mind forwarding keys to the deployment 'bastion' host allowing ssh should be possible. The alternative would be to look at using Salt once there is an API in place to allow authentication via Nova (I believe production is moving towards Salt for deployment anyway).

I'm not sure what the current state of salt in labs is but this seems like a perfect case to use salt.

Asked salt status in labs on the labs mailing list: http://permalink.gmane.org/gmane.org.wikimedia.labs/1077

Ryan and I are working on adding a dedicated salt master for the beta cluster. When this is doing it should be fairly trivial to implement a script that will tell each apache host to perform a graceful restart.

The eqiad instances of deployment-prep now have a local salt master. I'll take a stab at making this a reality.

Salt has a built in module for communicating with apache [0]. Our puppet configuration now creates a salt grain for each system role [1]. Putting the two of these together allows us to send all apaches acting as application servers a restart signal from deployment-salt.eqiad.wmflabs:

sudo salt -G 'rolename:role::applicationserver::appserver::beta' apache.signal restart

All apache instances across the whole beta cluster could be restarted as well:

sudo salt -G '*' apache.signal restart

[0]: http://docs.saltstack.com/en/latest/ref/modules/all/salt.modules.apache.html
[1]: https://gerrit.wikimedia.org/r/#/c/107831/

Great! Would it be worth adding a few shell wrappers such as:

beta-apaches (restart|reload|graceful|stop|start) ?

Not sure if it is worth it, but sounds easier to remember than the salt command.

Change 125888 had a related patch set uploaded by BryanDavis:
beta: New script to restart apaches

https://gerrit.wikimedia.org/r/125888

Change 125888 merged by Alexandros Kosiaris:
beta: New script to restart apaches

https://gerrit.wikimedia.org/r/125888

My patch was merged, but puppet<->salt integration is disabled/broken due to race conditions that occur when new instances are added and need their salt client certs to be accepted on the salt master.

Seems it is fixed now :) Thank you Bryan.