Page MenuHomePhabricator
Create Task
Maniphest T53350

solr stress out the NFS server
Closed, DeclinedPublic
Actions

Description

This morning we found out both beta and tools projects were "slow". The root cause is some jobs running on the solr project which exhaust the NFS server (labstore3) I/O operations.

The workaround is to reboot the instance solr-mw2.pmtpa.wmflabs to disable the stressing job as instructed by Nikolas Everett http://lists.wikimedia.org/pipermail/labs-l/2013-July/001381.html

The Solr experiment should be run on a different NFS system than the shared one. I guess a dedicated one.

Version: unspecified
Severity: normal

Details

Reference
bz51350

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 1:56 AM
• bzimport added projects: Cloud-VPS, Performance Issue.
• bzimport set Reference to bz51350.
hashar created this task.Jul 15 2013, 10:45 AM
hashar added a comment.Jul 15 2013, 10:47 AM

Created attachment 12846
network usage of solr project between 7/13/2013 8:00 and 7/13/2013 20:00

Attached:

labs-solr.png (211×397 px, 14 KB)

hashar added a comment.Jul 15 2013, 10:47 AM

Created attachment 12847
Ganglia CPU report of labstore 3 between 7/13/2013 8:00 and 7/13/2013 20:00

Attached:

labstore3_cpu_report.png (259×397 px, 15 KB)

ArielGlenn added a comment.Jul 15 2013, 11:15 AM

So in accordance with the above I rebooted solr-mw2 and load on labstore3 looks much better.

• Manybubbles added a comment.Jul 15 2013, 1:33 PM

I should let everyone know what I'm doing:

I'm loading a copy of enwiki with all the current text (as of some backup) so I can index it. No historical revisions as we won't be indexing them. I've been told that I can't use a prod replica because it won't contain any text.

What is actually killing the NFS server is mysqld, not solr, elasticsearch, or any other new system. I make no claims that those systems wouldn't put a similar load on nfs at some point in the future though.

I'd run this on my local system then I wouldn't be able to properly interact with other systems in labs that I need for the experiment.

hashar added a comment.Nov 22 2013, 5:14 PM

Closing this since Elastic Search has been deployed in production so I guess there is less need nowadays to load huge amount of data in labs instance.

Aklapper added a project: Cloud-Services.Apr 29 2015, 12:20 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL