Page MenuHomePhabricator

$wgDeferredUpdateList and long-running scripts
Closed, ResolvedPublic

Description

Import scripts and probably other maintenance script call things like doEdit() or (onArticleEdit() directly) which builds up $wgDeferredUpdateList. These updates are not done until it finishes whatever it was doing, which means the $wgDeferredUpdateList array can get huge and slow.

Maybe:
(a) they should call wfDoUpdates() around wfWaitForSlaves(). This is simple but requires script writers to remember non-obvious crap
(b) stuff that uses wgDeferredUpdateList does it immediately if in cli mode
(c) stuff that uses wgDeferredUpdateList does do (nor defer) updates when in cli mode. Callers would have to do whatever themselves (which could lead to duplicated code).


Version: 1.17.x
Severity: minor

Event Timeline

bzimport raised the priority of this task from to Low.Nov 21 2014, 11:08 PM
bzimport set Reference to bz24375.

Ugh, (c) should be:
"(c) stuff that uses wgDeferredUpdateList does not do (nor defer) updates when in
cli mode. Callers would have to do whatever themselves (which could lead to
duplicated code)."

(a) seems like a good solution: https://gerrit.wikimedia.org/r/48412

If anybody thinks (b) or (c) would be preferred then I'll abandon that patch.

Well that makes sure they are done, but it doesn't deal with the array bloat problem (which really slows down as the array gets larger).

It might help to use a wrapper function instead of just $wgDeferredUpdateList[]=stuff. Then would could have an option (d) where it starts doing some updates if the list is too big (in cli mode).

Agreed, but at the very least that patch is a good first step. I mean, it's only natural that a maintenance script should commit deferred updates upon exiting the script.

Change 211881 had a related patch set uploaded (by Aaron Schulz):
Avoid OOMing on deferred updates in CLI mode

https://gerrit.wikimedia.org/r/211881

Change 211881 abandoned by Gilles:
Avoid reaching OOM via deferred updates in CLI mode

Reason:
Already merged as part of Iff6625ddc04a15751d2bb07dc6558145e7ceb14a

https://gerrit.wikimedia.org/r/211881