Page MenuHomePhabricator

wikimedia-job-runner should be able to run a specific job type
Closed, ResolvedPublic

Description

We need to be able to setup instances that will only run a specific kind of job such as video transcoding. The patch is at

https://gerrit.wikimedia.org/r/#/c/11610/

Need the Debian package to be updated and deployed.


Version: unspecified
Severity: normal

Details

Reference
bz38455

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 12:46 AM
bzimport set Reference to bz38455.
bzimport added a subscriber: Unknown Object (MLST).

jgerber wrote:

for jobs-loop.sh to work for transcoding, the hardcoded maxtime of 300 needs to be an option too.

jgerber wrote:

https://gerrit.wikimedia.org/r/#/c/15954/ makes jobs and maxtime
optional options that can be passed to jobs-loop.sh

video encoding nodes would have

JR_TYPE="-j webVideoTranscode -t 14400"

in /etc/default/mw-job-runner

(Possibly JR_TYPE should be renamed to JR_ARGS in https://gerrit.wikimedia.org/r/#/c/11610/)

Another JR_ can be added to operations/debs/wikimedia-job-runner.git

Example:
https://gerrit.wikimedia.org/r/#/c/11610/

Then would need ops to update the package on apt.wikimedia.org.

Would be great to have JR_TIMEOUT. We would then be able to generate or override /etc/default/mw-job-runner using puppet.

jgerber wrote:

without renaming JS_TYPES to JS_ARGS you can already just use puppet to set
JR_TYPE="-j webVideoTranscode -t 14400"

no need to have a new variable for each option, having one variable JR_ARGS to hold all arguments that are passed to the script seams more future proof. that way adding new arguments to the script does not require a new deb package.

a grep for _ARGS / _OPTS in /etc/defaults also suggest that this is a common pattern for ubuntu packages.

https://gerrit.wikimedia.org/r/#/c/11610/ has not been merged in nor the package has been rebuild yet. So you could amend that change and add both JR_TIMEOUT and JR_EXTRA_ARGS arguments to it. That would do it.

jgerber wrote:

sounds reasonable pushed a change to https://gerrit.wikimedia.org/r/#/c/11610/ please review.

Following a discussion with Faidon and Mark on 23rd, we will probably move that Debian package to puppet and bring the loop shell script from WikimediaMaintenance with it. That would add consistency and let us write some nice puppet class to set up job workers.

That is merged in and migrated to puppet.