Page MenuHomePhabricator

Jenkins: Investigate using concurrent builds of the same job
Closed, ResolvedPublic

Description

Jenkins can run a job several time by isolating the code in different workspace. I have tried it out by creating a job that sleep for 30 seconds and triggered it several time. The disk shows:

$ ls -1 -d workspace*
workspace/
workspace@2/
workspace@3/
workspace@4/
$

That does not work with custom workspace though, we will have to handle it ourselves.

A use case would be the MediaWiki parser tests, the risk is that we could end up with all the Jenkins executor being busy running parser tests which would delay the other faster tests.


Version: wmf-deployment
Severity: enhancement

Details

Reference
bz47063

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:25 AM
bzimport set Reference to bz47063.
bzimport added a subscriber: Unknown Object (MLST).

Concurrently running different jobs or concurrent builds of the same job?

As far as I know we already run concurrent builds of different jobs. Eg. on patchset-create mediawiki-core-phpunit-misc runs simultaneously with mediawiki-core-phpunit-api, that is already the case.

Running concurrent builds of the same job (if that is even possible, is it?) will likely get rather tricky.

For one, it will no longer make our gate an effective gate as it will no longer be testing the result as it will be when merged, other conflicting commits could make it into the repository separate from each other, thus breaking the end result with no warning *at all* anywhere to be found (until the next unrelated commit comes in and fails with nobody having a clue).

Secondly, we have various jobs (such as in Parsoid) that have certain hacks in place to run twice in a row to use last-run build artefacts. Not sure how that goes with concurrency.

Except for this last category (Parsoid double-runs) concurrency would be fine as long as the gate pipeline does not use it (for aforementioned reasons). On the check and test pipelines for patchset creations it should be fine.

I have made lot of jobs concurrent in December. The mediawiki-core-phpunit-parsertests had a race condition (bug 58094) which is apparently fixed.

Change 102152 had a related patch set uploaded by Hashar:
(WIP) mediawiki-core-phpunit-* made concurrent (WIP)

https://gerrit.wikimedia.org/r/102152

Change 102152 merged by jenkins-bot:
mediawiki-core-phpunit-* made concurrent

https://gerrit.wikimedia.org/r/102152

The Wikibase / Wikidata jobs need to be made concurrent as well.

Wikibase / Wikidata got moved to another CI system.

For mw/core the last non concurrent job was the qunit one which Timo modified last Friday with https://gerrit.wikimedia.org/r/#/c/117603/