For the last couple days the php entry points were giving an Error 500 because the Thanks extension was not in mediawiki/extensions.git (that is fixed now). Lucene search poll all the wiki via the OAI extension, that definitely served error 500 page that might have broke the search system.

The two search instances are using puppetmaster::self so their puppet configuration have to be done manually. I have updated them a few hours ago.

Doing a search does not work right now:

http://en.wikipedia.beta.wmflabs.org/w/api.php?format=json&action=opensearch&search=F&namespace=0&suggest=

Gives out:

["F",[]]

Need to investigate the PHP error logs and look at the search box logs.

deployment-search01:~$ curl -x localhost:8123 http://localhost/search/enwiki/Main
curl: (7) couldn't connect to host
deployment-search01:~$

I have restarted lucene-search2 there

Search is working again. What is troublesome is that lucene-search2 should be restart by puppet automatically whenever it dies. I am leaving this bug open to monitor it a bit more.

The lucene process is probably killed by the OOM catcher. We need to tweak the java -Xm parameter to limit the amount of memory being used.

Both en.m.wikipedia.beta.wmflabs.org and en.m.wikipedia.org have San Francisco article:

http://en.m.wikipedia.beta.wmflabs.org/wiki/San_Francisco
http://en.m.wikipedia.org/wiki/San_Francisco

At en.m.wikipedia.org when you enter San in search box, several search suggestions appear (wikipedia.png attachment). No search suggestions appear when the same is done at en.m.wikipedia.beta.wmflabs.org (wmflabs.png attachment).

Created attachment 12054
wikipedia screenshot

Attached:

Created attachment 12055
wmflabs screenshot

Attached:

Passing en.m.wikipedia.org Jenkins and Sauce Labs jobs:

https://wmf.ci.cloudbees.com/job/_debug-MobileFrontend-template/6/
https://saucelabs.com/tests/ba82274d4aaf4449919ff77c17292969

Failing en.m.wikipedia.beta.wmflabs.org Jenkins and Sauce Labs jobs:

https://wmf.ci.cloudbees.com/job/_debug-MobileFrontend-template/7/
http://saucelabs.com/jobs/a1b31beeaa2749d88335f2b94307bef8

Rewording the summary. The root cause is the java process asking for 20GB memory on an instance having 4GB.

I have hacked the script locally to limit memory to 2GB. Will see how well it goes then hack the puppet class and init.d script to let us easily tweak the memory settings for lucene.

Command running right now is:

/usr/bin/java -Xmx2000m -Dsun.rmi.transport.tcp.handshakeTimeout=10000 -Djava.rmi.server.codebase=file:///a/search/lucene-search/LuceneSearch.jar -Djava.rmi.server.hostname=deployment-search01 -classpath :/usr/share/java/udp2log-log4j.jar:/a/search/lucene-search/LuceneSearch.jar org.wikimedia.lsearch.config.StartupManager

:)

Taking bug, raising priority. I need to fix that this week.

The deployment-search01 Icinga report is http://icinga.wmflabs.org/cgi-bin/icinga/extinfo.cgi?type=2&host=deployment-search01.pmtpa.wmflabs&service=Lucene+frontend

I have restarted the lucene-search-2 service that was apparently no more listening although there has been no OOM message :-] So we have some progress!

Patches are:

https://gerrit.wikimedia.org/r/#/c/59995/ convert java opts to shell variables
https://gerrit.wikimedia.org/r/#/c/59996/ conf file for lucene.jobs.sh
https://gerrit.wikimedia.org/r/#/c/60000/ enable conf file loading

pending ops review, updating summary to reflect that.

Peter has merged the changes and deployed them in production. I have to make sure that works fine in labs and will most probably recreate the existing instances.

Most of the work has been completed, thus lowering priority.

Chad and Nik have migrated beta to CirrusSearch extension which uses an
ElasticSearch backend. Hence this Lucene search bug is no more valid :-)