Page MenuHomePhabricator

Filter redundant jobs from the queue
Closed, ResolvedPublic

Description

Under some circumstances, like multiple successive edits to a much used template, the job queue may get quite full, and many of the jobs will be redundant: After a template is edited three times during a short period of time, the affected pages have all to be re-rendered, but only once, not three times.

For jobs that re-render page content, this can be done using the page_touched field. When the job is created and scheduled, it should remember the current value of page_touched. When the job is later executed, it can then first check whether page_touched is still the same as it was when the job was created. if not, the job can be canceled and the page is not rendered again.

Consider this scenario:

1: Template T gets edited, page P needs to be updated.
2: Job J1 is scheduled for updating the page
3: J1.page_touched is set to the page's current touch timestamp, P.page_touched.
4: Template T gets edited again before J1 was executed, page P (still) needs to be updated.
5: Job J2 is scheduled for updating the page
6: J2.page_touched is set to the page's current touch timestamp, P.page_touched. This is still the same as J1.page_touched, since we assumed the page wasn't re-rendered in the mean time.
7: J1 gets executed.
8: J1 checks J1.page_updated against P.page_updated and find it's still the same.
9: J1 re-renders P. This sets P.page_touched to the current wall clock time.
10: J2 gets executed.
11: J2 checks J2.page_updated against P.page_updated and find it's not the same.
12: J2 aborts.

This also covers the case that P gets re-rendered for some other reason:

4: P gets edited and re-rendered. This sets P.page_touched to the current wall clock time.
5,6: nothing
7: J1 gets executed.
8: J1 checks J1.page_updated against P.page_updated and find it's not the same.
9: J1 aborts.


Version: 1.21.x
Severity: enhancement

Details

Reference
bz42065

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 1:02 AM
bzimport set Reference to bz42065.
bzimport added a subscriber: Unknown Object (MLST).

Merged gerrit 32488 links here, bug maybe resolved

There was also change I58cfa98e and some other heavy work on the jobqueue today, cf. http://article.gmane.org/gmane.science.linguistics.wikipedia.technical/65703

Closed older resolved bugs as verified.