Page MenuHomePhabricator

CirrusSearch: Figure out why shards needed reassignment around 19:35 UTC Nov 18th
Closed, ResolvedPublic

Description

Figure out why shards needed reassignment around 19:35 UTC Nov 18th and then file a new bug for that.


Version: unspecified
Severity: normal
Whiteboard: cirrus_reenable

Details

Reference
bz57222

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 2:21 AM
bzimport added a project: CirrusSearch.
bzimport set Reference to bz57222.

I've deployed an update to logging so we can see _exactly_ why shards are moved in gory detail. Also, in the process of doing Bug 57243 I'm doing the same careful shutdowns that we did yesterday to see if I can cause this again.

One problem I know we have: I missed some of the cirrus wikis (the wikivoyages) when I was initializing redundancy. That _might_ have caused the unassigned shards if we didn't follow the careful shutdown procedures correctly. Normally, though, with the careful shutdown procedure everything should still work without any replicas and replicas are just for handling unexpected outages.

Anyway, I'm still not sure what caused this but if I can't reproduce it by doing careful updates I'm going to sleep on it again and close this bug if nothing comes to me.

As promised in my last comment I'm closing this bug: I haven't been able to reproduce it with sane shutdowns. I'm going to close this under the assumption that the problem was the wikivoyages.