Thanks for taking the time to report this!

Which webserver? A URL to reproduce is welcome in bug reports. :)

metatron wrote:

php-sites on tools-webserver retrun Error 500 i.e.

https://tools.wmflabs.org/catnap/

This error relates to tools-webserver-01

tools-webserver-01 seems to be fixed and is no longer constantly throwing a 500 error.

The other two webservers are still up. Only about 1/3 tools were affected.

Logs show there was a burst of activity with glamtools that lasted a couple of hours.

If this was about http://tools.wmflabs.org/ then it was also brought up on http://lists.wikimedia.org/pipermail/wikitech-l/2013-October/072336.html

tools-webserver-01 is experiencing the same problems as above, reopening.

The current scheme is, annoyingly, vulnerable to single tools consuming every resources. There is a new system in place that is more robust and more performing for all but the very simplest of tools:

https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/Help/NewWeb

While the system is currently opt-in, I would very much recommend that the heavier duty move to it as soon as possible; this will lighten the load on the apaches (alleviating the issue) and insulate those tools from the others.