No, it's a database issue: catscan2 has queries holding locks that have been running for several hours; I expect further queries simply pile up waiting to grab locks.

I don't think so. The test script

https://tools.wmflabs.org/catscan2/catscan2_test.php

consists of the following code:
<?PHP
header('Content-type: text/html');
print "TEST" ;
?>

It's not loading either, and I would think it's not waiting for a database query...

No, that script isn't, but all of your lighttpd workers are (which means that attempting to run that test fails because it's starved).

This implies that the issue might be solvable with a timeout for the cases where user input causes a very long query to stall.

But it also fails immediately after I restart lighttpd. That should have cleared the workers, right?

So, how do I get the tool working again?

Also, will adding a PHP set_time_limit prevent this from happening again? If not, how?

Finally, why did it stop today, when it apparently worked for months without incident? (there was one, a few month ago IIRC, but otherwise it seemed to be fine).

OK, seems to be back. I'd still like to hear some specific prevention method for this though.

set_time_limit will indeed try to abort the /script/, but depending on how exactly you connect to the DB it may not abort a query in progress.

A quick google finds that the problem is not new; there are possible avenues of investigation at

http://stackoverflow.com/questions/9562124/php-mysql-set-connection-timeout
http://www.phpbb2refugees.com/viewtopic.php?p=7647&sid=028921ede498fddbc1ebfbd71ee37dcc

The short of it is that there doesn't seem to be a way to set a query timeout, but you could set up an alarm() to abort one that takes too long.