Page MenuHomePhabricator
Create Task
Maniphest T62105

Better metrics from the job queue
Closed, ResolvedPublic
Actions

Description

It'd be really helpful to get some information from the job queue per job type (and maybe per wiki?):

  1. time spent processing that job type
  2. number of jobs of that type started
  3. average execution time (basically 1 divided by 2 if both are rrdtool counters)
  4. number of jobs in the queue
  5. average time between job finished and job queued

Linked from: https://wikitech.wikimedia.org/wiki/Incident_documentation/20150825-Redis.

Details

Reference
bz60105

Related Objects

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:57 AM
• bzimport added a project: WMF-General-or-Unknown.
• bzimport set Reference to bz60105.
• bzimport added a subscriber: Unknown Object (MLST).
• Manybubbles created this task.Jan 15 2014, 8:37 PM
Aklapper added a comment.Feb 11 2014, 1:56 PM

Nik: Do you know if this would require changes in JobQueue ("MediaWiki" product) first, or server setup only (ganglia etc.), or both?

• Manybubbles added a comment.Feb 11 2014, 2:04 PM

I think it is likely.

• Manybubbles added a comment.Feb 11 2014, 2:05 PM

Sorry, to be more clear: I imagine to get them all you'd need to change Mediawiki in some way.

• GWicke merged a task: T93833: Job queue length, push, pop & latency metrics.May 3 2015, 5:12 PM
• GWicke added subscribers: aaron, Aklapper, • GWicke.
Krenair subscribed.May 3 2015, 5:14 PM
fgiunchedi subscribed.May 4 2015, 7:08 AM

note that per-status stats for each wiki is a non-starter, T95913 (too many metrics essentially)

fgiunchedi merged a task: Restricted Task.Jun 1 2015, 11:06 PM
fgiunchedi added a subscriber: Joe.
jcrespo raised the priority of this task from Medium to High.Aug 25 2015, 9:20 AM
jcrespo subscribed.

Elevating this to high, after polling other ops and devs on IRC, as it could have detected (maybe prevented most of its consequences?) a recent outage before it was too late.

bd808 added a project: MediaWiki-Core-JobQueue.Oct 20 2015, 10:14 PM
bd808 subscribed.
jcrespo added a comment.Nov 6 2015, 4:15 PM

I see a much improved graph here: https://grafana.wikimedia.org/dashboard/db/job-queue-rate

Maybe this has already been resolved?

Krinkle closed this task as Resolved.Jul 4 2017, 2:10 AM
Krinkle assigned this task to aaron.
Krinkle added a project: Performance-Team.
Krinkle subscribed.

Yep. Marking as resolved.

https://grafana.wikimedia.org/dashboard/db/job-queue-rate
https://grafana.wikimedia.org/dashboard/db/job-queue-health

Krinkle added a project: Wikimedia-Incident.Aug 4 2018, 4:18 PM
Krinkle moved this task from Active investigation to Follow-up prevention on the Wikimedia-Incident board.
Krinkle updated the task description. (Show Details)
Krinkle removed a subscriber: • wikibugs-l-list.
Krinkle edited projects, added Sustainability (Incident Followup); removed Wikimedia-Incident.Apr 28 2020, 9:50 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL