Page MenuHomePhabricator

User-controlled job throttle for GWToolset (tracking)
Closed, ResolvedPublic

Description

• allow the user the ability to control how many media file jobs per minute are added to the MediaWiki job queue.
• minimum of 1
• default 10
• maximum of 20

chris steipp is concerned about someone setting up files that are near the max upload size, approximately 1GB on Commons. At 60/min, that’s 8 Gbps .... which would likely dos any other server, and be a significant chunk of MediaWiki bandwidth. chris would like us to keep the max at 20. if anyone wants to increase that value, they should add that request to this ticket and get ops approval and have them monitor server activity while they carry out their bulk upload.


Version: unspecified
Severity: normal

Details

Reference
bz58417

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:18 AM
bzimport set Reference to bz58417.
bzimport added a subscriber: Unknown Object (MLST).

Change 101008 had a related patch set uploaded by Dan-nl:
user-job-throttle

https://gerrit.wikimedia.org/r/101008

Change 101008 merged by jenkins-bot:
user-job-throttle

https://gerrit.wikimedia.org/r/101008

Regarding the number of uploads: 20/min per batch upload, for, say, 10 batch uploads at the same time, should be fine for Swift. It's not really that big of a number, even with large files.

Regarding bandwidth: it's unlikely that we can do more than 60-70MB/s right now on the Swift cluster. I doubt that we can have enough third-party servers able to push us in aggregate files at that rate, though.

This has been deployed. Change status to verified?

dan: Is this ticket RESOLVED FIXED or is there more work to do?

this is supposed to be a "tracking" bug that keeps track of anyone who requests a higher throttle for GWToolset because it may be too slow in retrieving mediafiles from their server. maybe i haven't set-up the "tracking" bug properly. is there another way to do it?

Gilles triaged this task as Unbreak Now! priority.Dec 4 2014, 10:11 AM
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to Needs Triage.Dec 4 2014, 11:21 AM
Aklapper subscribed.

[adding the Tracking-Neverending project to tasks blocking (now deprecated) T4007 as part of T93366]