Page MenuHomePhabricator

CirrusSearch: Fail updates quickly if Elasticsearch is down or otherwise broken
Closed, ResolvedPublic

Description

Fail updates quickly if Elasticsearch is down or otherwise broken. This should include a fail fast pool counter in case we have a bunch of workers and a timeout. While we're at it we need to make sure that this only effects web users. Maintenance scripts should just wait.


Version: unspecified
Severity: normal
Whiteboard: cirrus_reenable

Details

Reference
bz57215

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 2:21 AM
bzimport added a project: CirrusSearch.
bzimport set Reference to bz57215.

Oh, and these failures should be invisible to users. They should be logged but we shouldn't bother users with this.

This will require a change to ElasticaConnection - that class seems to retry connection errors without regards to any overall timeout.

Change 96639 had a related patch set uploaded by Manybubbles:
Add timeouts to updates and better log failures

https://gerrit.wikimedia.org/r/96639

Change 96639 merged by jenkins-bot:
Add timeouts to updates and better log failures

https://gerrit.wikimedia.org/r/96639