Page MenuHomePhabricator

JJB: generates duplicates jobs with different parameters
Closed, ResolvedPublic

Description

Daniel Kinzler wrote:

There seems to be an issue with Jenkins. It appears to use an old version of
other extensions under some circumstances.

It's like this:

If you submit change 33 for extension A, which needs change 44 in extension B
(which isn't merged yet), jenkins will fail correctly fail.

BUT: When change 44 got merged into extension B, and you force Jenkins to re-run
(e.g. by rebasing change 33), it will *still* fail, apparently using an old
version of extension B.

It seems this is only the case for the "testextensions-master" job, not the
standalone "repo" and "client" jobs.

Here are some examples:

https://gerrit.wikimedia.org/r/#/c/72962/ fails for no good reason
https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-master/3099/console

https://gerrit.wikimedia.org/r/#/c/73772/ fails for no good reason
https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-master/3093/console

Please gather more evidence/insights if you come across this issue.

Thanks,
Daniel


Version: wmf-deployment
Severity: normal

Details

Reference
bz51461

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 2:03 AM
bzimport set Reference to bz51461.
bzimport added a subscriber: Unknown Object (MLST).

The issue must be related to the script fetching out the MediaWiki dependencies. It might not get the laster master version, possibly because it is just cloning the dependency and the workspace is never wiped.

will work on that tomorrow morning. I stay out of comp tonight sorry.

I found out the issue. The dependencies are no more fetched :(

Seems to be an issue in Jenkins Job builder:

$ jenkins-jobs --conf jenkins_jobs.ini update config/ mwext-Wikibase-testextensions-master
INFO:root:Updating jobs in config/ (['mwext-Wikibase-testextensions-master'])
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mwext-Wikibase-testextensions-master
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mwext-Wikibase-testextensions-master
$

It generates two version of the job, one of them must be lacking the dependencies :/

Jenkins jobs indeed build the job twice, at first with:

<command>/var/lib/jenkins/tools/fetch-mw-ext</command>

Then another time with the proper command:

<command>/var/lib/jenkins/tools/fetch-mw-ext Ask,Serialization,Diff,DataValues,WikibaseDataModel,Validator</command>

Jenkins Jobs always proceed them in that order so the job always ends up with the dependencies. BUT, whenever the job generation is aborted (and I did so on 2013-06-26_14-46-02), we end up with missing dependencies :/

The history link:
https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-master/jobConfigHistory/showDiffFiles?timestamp1=2013-07-16_19-35-57&timestamp2=2013-07-16_19-36-05

All the past configurations for the job:

https://integration.wikimedia.org/ci/job/mwext-Wikibase-testextensions-master/jobConfigHistory/?

You will notice that there are double entries a few seconds apart, that highlight the fact that Jenkins Job Builder regenerate the job twice.

Workaround: make sure both versions are pushed (ie do not abort the run)

Goal: fix up jenkins job builder to handle that and only push the second (last) version.

I have lowered the bug priority since it is fixed but I keep this bug open to fix up Jenkins Job Builder.

Sorry the issue apparently started on 2013-07-11_08-53-39

This is an issue in how we craft our jobs in Jenkins Job Builder. The workaround is to make sure the JJB command refresh the duplicate jobs.

Unassigning self for now, will revisit later on.

This is still occurring. But there are now less jobs being duplicated (once with defaults, and the second time with a variable set). Clean up is https://gerrit.wikimedia.org/r/#/c/155249/

Krinkle claimed this task.

This appears to be resolved. Feel free to re-open.