Maniphest T61935

Jenkins: Zuul is consistently losing job mwext-TemplateData-qunit to Gearman
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	Krinkle
	Jan 11 2014, 3:55 AM

Description

I've recently created (I78b948aa0e) and enabled (I1053812d8ac) the job "mwext-TemplateData-qunit".

However, it is consistently being lost by Zuul.

Example change set: https://gerrit.wikimedia.org/r/#/c/85400/

Zuul log of two events where it happened:

2014-01-11 03:41:25,410 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99110 85400,38> to <Pipeline check-only>
2014-01-11 03:41:25,410 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/TemplateData
2014-01-11 03:41:25,410 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99110 85400,38> to <Pipeline check-voter>
2014-01-11 03:41:25,410 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/TemplateData
2014-01-11 03:41:25,411 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99110 85400,38> to <Pipeline test>
2014-01-11 03:41:25,411 INFO zuul.IndependentPipelineManager: Change <Change 0x7f3008a99110 85400,38> depends on changes []
2014-01-11 03:41:25,933 INFO zuul.IndependentPipelineManager: Change <Change 0x7f3008a99110 85400,38> depends on changes []
2014-01-11 03:41:25,933 INFO zuul.Gearman: Launch job mwext-TemplateData-lint for change <Change 0x7f3008a99110 85400,38> with dependent changes []
2014-01-11 03:41:26,155 INFO zuul.Gearman: Launch job mwext-TemplateData-jslint for change <Change 0x7f3008a99110 85400,38> with dependent changes []
2014-01-11 03:41:26,159 INFO zuul.Gearman: Launch job mwext-TemplateData-qunit for change <Change 0x7f3008a99110 85400,38> with dependent changes []
2014-01-11 03:41:26,159 ERROR zuul.Gearman: Job <gear.Job 0x7f30086439d0 handle: None name: build:mwext-TemplateData-qunit unique: 6153b4d1d7e941bca65b7f72d2a1dede> is not registered with Gearman
2014-01-11 03:41:26,159 INFO zuul.Gearman: Build <gear.Job 0x7f30086439d0 handle: None name: build:mwext-TemplateData-qunit unique: 6153b4d1d7e941bca65b7f72d2a1dede> complete, result LOST
2014-01-11 03:41:26,160 INFO zuul.Gearman: Launch job mwext-TemplateData-testextensions-master for change <Change 0x7f3008a99110 85400,38> with dependent changes []
2014-01-11 03:41:26,162 INFO zuul.Gearman: Launch job mwext-TemplateData-phpcs-HEAD for change <Change 0x7f3008a99110 85400,38> with dependent changes []
2014-01-11 03:41:26,202 INFO zuul.Gearman: Build <gear.Job 0x7f3008643c10 handle: H:127.0.0.1:1 name: build:mwext-TemplateData-lint unique: 0431dedc6b0d49529ad283996b297557> started
2014-01-11 03:41:26,242 INFO zuul.Gearman: Build <gear.Job 0x2582bd0 handle: H:127.0.0.1:2 name: build:mwext-TemplateData-jslint unique: 8e1b020350de4d68a5dea386548de8d9> started
2014-01-11 03:41:26,246 INFO zuul.Gearman: Build <gear.Job 0x7f3008643bd0 handle: H:127.0.0.1:3 name: build:mwext-TemplateData-testextensions-master unique: 911e8877845f42c496e1cf833dd56aea> started
2014-01-11 03:41:26,250 INFO zuul.Gearman: Build <gear.Job 0x7f3008643ad0 handle: H:127.0.0.1:4 name: build:mwext-TemplateData-phpcs-HEAD unique: 16fb1eec2d0f459f88a267ea39d2064e> started
2014-01-11 03:41:26,900 INFO zuul.Gearman: Build <gear.Job 0x7f3008643c10 handle: H:127.0.0.1:1 name: build:mwext-TemplateData-lint unique: 0431dedc6b0d49529ad283996b297557> complete, result SUCCESS
2014-01-11 03:41:27,121 INFO zuul.Gearman: Build <gear.Job 0x7f3008643ad0 handle: H:127.0.0.1:4 name: build:mwext-TemplateData-phpcs-HEAD unique: 16fb1eec2d0f459f88a267ea39d2064e> complete, result SUCCESS
2014-01-11 03:41:27,396 INFO zuul.Gearman: Build <gear.Job 0x2582bd0 handle: H:127.0.0.1:2 name: build:mwext-TemplateData-jslint unique: 8e1b020350de4d68a5dea386548de8d9> complete, result SUCCESS
2014-01-11 03:41:29,748 INFO zuul.Gearman: Build <gear.Job 0x7f3008643bd0 handle: H:127.0.0.1:3 name: build:mwext-TemplateData-testextensions-master unique: 911e8877845f42c496e1cf833dd56aea> complete, result SUCCESS
2014-01-11 03:41:29,825 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f3008a99110 85400,38>, actions: [<ActionReporter <zuul.reporter.gerrit.Reporter object at 0x25821d0>, {'verified': -1}>]

2014-01-11 03:46:38,349 INFO zuul.Gerrit: Updating information for 85400,39
2014-01-11 03:46:39,010 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99f10 85400,39> to <Pipeline check-only>
2014-01-11 03:46:39,010 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/TemplateData
2014-01-11 03:46:39,011 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99f10 85400,39> to <Pipeline check-voter>
2014-01-11 03:46:39,011 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/TemplateData
2014-01-11 03:46:39,011 INFO zuul.Scheduler: Adding mediawiki/extensions/TemplateData, <Change 0x7f3008a99f10 85400,39> to <Pipeline test>
2014-01-11 03:46:39,012 INFO zuul.IndependentPipelineManager: Change <Change 0x7f3008a99f10 85400,39> depends on changes []
2014-01-11 03:46:39,501 INFO zuul.IndependentPipelineManager: Change <Change 0x7f3008a99f10 85400,39> depends on changes []
2014-01-11 03:46:39,501 INFO zuul.Gearman: Launch job mwext-TemplateData-lint for change <Change 0x7f3008a99f10 85400,39> with dependent changes []
2014-01-11 03:46:39,503 INFO zuul.Gearman: Launch job mwext-TemplateData-jslint for change <Change 0x7f3008a99f10 85400,39> with dependent changes []
2014-01-11 03:46:39,507 INFO zuul.Gearman: Launch job mwext-TemplateData-qunit for change <Change 0x7f3008a99f10 85400,39> with dependent changes []
2014-01-11 03:46:39,707 INFO zuul.Gearman: Build <gear.Job 0x7f3018219350 handle: H:127.0.0.1:38 name: build:mwext-TemplateData-lint unique: f340f7a6889b4b9191d7beecfb528598> started
2014-01-11 03:46:39,708 INFO zuul.Gearman: Build <gear.Job 0x7f3018219310 handle: H:127.0.0.1:39 name: build:mwext-TemplateData-jslint unique: 8268199135204f4cbf65f0bf301a94b8> started
2014-01-11 03:46:39,726 ERROR zuul.Gearman: Job <gear.Job 0x7f3018219590 handle: None name: build:mwext-TemplateData-qunit unique: e6da1e8d1d7c492a9778d8a4eba7098f> is not registered with Gearman
2014-01-11 03:46:39,727 INFO zuul.Gearman: Build <gear.Job 0x7f3018219590 handle: None name: build:mwext-TemplateData-qunit unique: e6da1e8d1d7c492a9778d8a4eba7098f> complete, result LOST
2014-01-11 03:46:39,727 INFO zuul.Gearman: Launch job mwext-TemplateData-testextensions-master for change <Change 0x7f3008a99f10 85400,39> with dependent changes []
2014-01-11 03:46:39,731 INFO zuul.Gearman: Launch job mwext-TemplateData-phpcs-HEAD for change <Change 0x7f3008a99f10 85400,39> with dependent changes []
2014-01-11 03:46:39,741 INFO zuul.Gearman: Build <gear.Job 0x7f3018219710 handle: H:127.0.0.1:40 name: build:mwext-TemplateData-testextensions-master unique: b3aa6d0a429b4a22972c024d68a1f729> started
2014-01-11 03:46:39,781 INFO zuul.Gearman: Build <gear.Job 0x7f3018219610 handle: H:127.0.0.1:41 name: build:mwext-TemplateData-phpcs-HEAD unique: ef222487a2454b669d44f11d536eb3e2> started
2014-01-11 03:46:40,277 INFO zuul.Gearman: Build <gear.Job 0x7f3018219350 handle: H:127.0.0.1:38 name: build:mwext-TemplateData-lint unique: f340f7a6889b4b9191d7beecfb528598> complete, result SUCCESS
2014-01-11 03:46:40,677 INFO zuul.Gearman: Build <gear.Job 0x7f3018219310 handle: H:127.0.0.1:39 name: build:mwext-TemplateData-jslint unique: 8268199135204f4cbf65f0bf301a94b8> complete, result SUCCESS
2014-01-11 03:46:40,730 INFO zuul.Gearman: Build <gear.Job 0x7f3018219610 handle: H:127.0.0.1:41 name: build:mwext-TemplateData-phpcs-HEAD unique: ef222487a2454b669d44f11d536eb3e2> complete, result SUCCESS
2014-01-11 03:46:43,023 INFO zuul.Gearman: Build <gear.Job 0x7f3018219710 handle: H:127.0.0.1:40 name: build:mwext-TemplateData-testextensions-master unique: b3aa6d0a429b4a22972c024d68a1f729> complete, result SUCCESS
2014-01-11 03:46:43,101 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f3008a99f10 85400,39>, actions: [<ActionReporter <zuul.reporter.gerrit.Reporter object at 0x25821d0>, {'verified': -1}>]
2014-01-11 03:46:43,157 INFO zuul.Gerrit: Updating information for 85400,39

Version: wmf-deployment
Severity: normal

Details

Reference: bz59935

Event Timeline

• bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 2:38 AM

• bzimport added a project: Continuous-Integration-Infrastructure.

• bzimport set Reference to bz59935.

Krinkle created this task.Jan 11 2014, 3:55 AM

https://gerrit.wikimedia.org/r/#/c/106860/

I tried doing a manual build from Jenkins (filling in Zuul paramaters), that one went fine: https://integration.wikimedia.org/ci/job/mwext-TemplateData-qunit/1/

In case it is a one-time bug, I verified that it is still broken even after I did the manual one.

https://gerrit.wikimedia.org/r/106860

And the relevant error is:

ERROR zuul.Gearman: Job <gear.Job 0x7f30086439d0

handle: None
name: build:mwext-TemplateData-qunit
unique: 6153b4d1d7e941bca65b7f72d2a1dede>

is not registered with Gearman

$ echo status|nc -q 3 localhost 4730|grep TemplateData
build:mwext-TemplateData-phpcs-HEAD:hasSlaveScripts 0 0 13
build:mwext-TemplateData-lint 0 0 13
build:mwext-TemplateData-lint:hasSlaveScripts 0 0 13
build:mwext-TemplateData-testextensions-master:hasSlaveScripts 0 0 13
build:mwext-TemplateData-testextensions-master 0 0 13
build:mwext-TemplateData-jslint 0 0 13
build:mwext-TemplateData-phpcs-HEAD 0 0 13
build:mwext-TemplateData-jslint:hasSlaveScripts 0 0 13
$

(need to document that last command somewhere)

So yeah the plugin did not register the job in Zuul Gearman server :/ Seems like a bug in Jenkins gearman plugin.

And now? All other jobs seem to work fine. I doubt this is related to the individual job configuration (TemplateData-qunit), since it just a copy of the mwext-name-qunit job template, and the others are working fine.

Perhaps something regressed in Jenkins/Zuul/Gearman recently that is only affecting new jobs created after that point.

Either way, TemplateData is now unmergable and we need those qunit tests.

Change 106860 had a related patch set uploaded by Hashar:
Sample commit for Jenkins

https://gerrit.wikimedia.org/r/106860

I have deleted the job and recreated it. That caused the Jenkins Gearman plugin to reregister the jobs with Zuul internal Gearman server.

The qunit job is registered now.

Retriggered change https://gerrit.wikimedia.org/r/#/c/106860/ by sending a new patchset.

Job build successfully https://integration.wikimedia.org/ci/job/mwext-TemplateData-qunit/1/console

I have added the command:

$ echo status|nc -q 3 localhost 4730|grep TemplateData

On Zuul documentation page https://www.mediawiki.org/w/index.php?title=Continuous_integration/Zuul&diff=882953&oldid=882952

Change 106860 abandoned by Krinkle:
Sample commit for Jenkins

Reason:
Thanks!

https://gerrit.wikimedia.org/r/106860

Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.Apr 7 2015, 2:04 PM

greg moved this task from Backlog to Done on the Continuous-Integration-Infrastructure board.Apr 14 2015, 3:04 PM

Jenkins: Zuul is consistently losing job mwext-TemplateData-qunit to GearmanClosed, ResolvedPublicActions

Description

Details

Event Timeline

Jenkins: Zuul is consistently losing job mwext-TemplateData-qunit to Gearman
Closed, ResolvedPublic
Actions