Page MenuHomePhabricator

HTTP 5xx Wikimedia Errors
Closed, ResolvedPublic

Description

on https://meta.wikimedia.org/wiki/Main_Page


Version: wmf-deployment
Severity: critical

Details

Reference
bz60970

Event Timeline

bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 3:02 AM
bzimport set Reference to bz60970.
bzimport added a subscriber: Unknown Object (MLST).

I don't see any error. In general, describing the error is a good first step for a bug report. :) See https://www.mediawiki.org/wiki/How_to_report_a_bug

It's being investigated in -operations right now.

Good spike in https://gdash.wikimedia.org/dashboards/reqerror/

Assuming this is about
"WIKIMEDIA FOUNDATION
Error
Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.",

Operations is looking at it in #wikimedia-operations on IRC.

Issue should be fixed now:

<Reedy> What caused it is fairly obvious (trying to run 3 versions of MW simultaneously). Why that was happening is a differnt matter
<Reedy> It should be fine now...

Appears to have been caused by some issue with the wmf12 deploy. Everything has been reverted to wmf11. The root cause of the issue was not identified. Can anyone else add anymore details?

(In reply to comment #8)

Appears to have been caused by some issue with the wmf12 deploy. Everything
has been reverted to wmf11. The root cause of the issue was not identified.
Can anyone else add anymore details?

Due to Math extension (texvc) apparently.

Created attachment 14515
Error 5xx rate today, for the records

Basically we had two partial outages lasting 10-20 minutes each on random pages served, if the 5xx error rate graph is to be trusted. I don't know if someone is able to calculate what portion of users/viewers was affected: 125k errors/s is three times the total pageviews/s and the same as total requests, as reported by https://gdash.wikimedia.org/dashboards/reqsum/

(In reply to comment #9)

Due to Math extension (texvc) apparently.

http://lists.wikimedia.org/pipermail/wikitech-l/2014-February/074321.html
When some conditions piled up.

Attached:

2014-02-06-Error503.png (768×1 px, 111 KB)