Page MenuHomePhabricator

Zuul: Certain events cause Zuul to trigger a pipeline with 0 jobs
Closed, ResolvedPublic

Description

Screenshot of Zuul Status at the time.

This happened a few weeks back as well. Entries are stuck in the pipeline indefinitely. No Jenkins jobs attached that I can see.

See screenshot.


Version: wmf-deployment
Severity: minor

Attached:

Screen_Shot_2013-07-28_at_1.36.12_AM.png (954×1 px, 82 KB)

Details

Reference
bz52161

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:56 AM
bzimport set Reference to bz52161.

All Jenkins executors are idle and stand-by.

$ tail -n100 /var/log/zuul/zuul.log
2013-07-27 22:47:54,239 INFO zuul.Gerrit: Getting information for 76288,1
2013-07-27 22:47:54,354 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94beef90 76288,1> to <Pipeline postmerge>
2013-07-27 22:47:54,354 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 22:52:10,847 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:52:10,847 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:52:10,972 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:54:14,905 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:54:14,906 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:54:15,036 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:54:15,151 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94bd4b10 76318,1> to <Pipeline gate-and-submit>
2013-07-27 22:54:15,152 INFO zuul.IndependentPipelineManager: Reporting start, action {'verified': 0} change <Change 0x7f2b94bd4b10 76318,1>
2013-07-27 22:54:15,234 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b9418c950 76318,1> to <Pipeline gate>
2013-07-27 22:54:15,235 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 22:54:15,236 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:54:15,236 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:54:15,356 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:56:22,230 INFO zuul.Scheduler: Fetching references for operations/mediawiki-config
2013-07-27 22:56:22,230 INFO zuul.Merger: Updating local repository operations/mediawiki-config
2013-07-27 22:56:22,696 INFO zuul.Gerrit: Getting information for 76237,2
2013-07-27 22:56:22,817 INFO zuul.Gerrit: Getting information for 76119,1
2013-07-27 22:57:49,863 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid
2013-07-27 22:57:49,863 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid
2013-07-27 22:57:50,244 INFO zuul.Gerrit: Getting information for 76315,2
2013-07-27 22:57:50,368 INFO zuul.Gerrit: Getting information for 76302,1
2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b9418cbd0 76315,2> to <Pipeline check-only>
2013-07-27 22:57:50,547 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid
2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94a36a10 76315,2> to <Pipeline test>
2013-07-27 22:57:52,171 INFO zuul.Jenkins: Launch job mwext-Parsoid-jslint for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:52,399 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 started, url: https://integration.wikimedia.org/ci/job/mwext-Parsoid-jslint/3099/
2013-07-27 22:57:52,598 INFO zuul.Jenkins: Launch job parsoid-parse-tool-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:52,808 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 started, url: https://integration.wikimedia.org/ci/job/parsoid-parse-tool-check/1015/
2013-07-27 22:57:53,022 INFO zuul.Jenkins: Launch job parsoid-roundtrip-test-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:53,349 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 started, url: https://integration.wikimedia.org/ci/job/parsoid-roundtrip-test-check/1012/
2013-07-27 22:57:53,446 INFO zuul.Jenkins: Launch job parsoid-parsertests-run-harder for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:53,764 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 started, url: https://integration.wikimedia.org/ci/job/parsoid-parsertests-run-harder/1078/
2013-07-27 22:57:53,878 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94334cd0 76315,2> to <Pipeline check-voter>
2013-07-27 22:57:53,878 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid
2013-07-27 22:57:58,904 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 complete, status SUCCESS
2013-07-27 22:58:00,614 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: None
parsoid-roundtrip-test-check: None
parsoid-parsertests-run-harder: None

2013-07-27 22:58:01,725 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 complete, status SUCCESS
2013-07-27 22:58:02,268 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: None
parsoid-parsertests-run-harder: None

2013-07-27 22:58:40,093 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 complete, status SUCCESS
2013-07-27 22:58:40,095 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: SUCCESS
parsoid-parsertests-run-harder: None

2013-07-27 22:59:02,017 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 complete, status SUCCESS
2013-07-27 22:59:02,019 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: SUCCESS
parsoid-parsertests-run-harder: SUCCESS

2013-07-27 22:59:02,020 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f2b94a36a10 76315,2>, action: {'verified': 2, 'code-review': 0}
2013-07-27 22:59:08,080 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid
2013-07-27 22:59:08,080 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid
2013-07-27 22:59:08,452 INFO zuul.Gerrit: Getting information for 76315,2
2013-07-27 22:59:08,577 INFO zuul.Gerrit: Getting information for 76302,1
2013-07-27 23:17:51,491 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,491 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:51,635 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 23:17:51,894 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,895 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:51,985 INFO zuul.Scheduler: Adding pywikibot/compat, <Ref 0x7f2b94391090 master updated 484f8cc860f192c72df8b9a54210df3c7cdd8ab8..91f565cf3d6c678df317db7d6f91563de95a1bcc> to <Pipeline post>
2013-07-27 23:17:51,985 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 23:17:51,986 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,987 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:52,103 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 23:17:52,226 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94391f50 76318,1> to <Pipeline postmerge>
2013-07-27 23:17:52,226 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 23:28:48,091 INFO zuul.Scheduler: Fetching references for pywikibot/core
2013-07-27 23:28:48,093 INFO zuul.Merger: Updating local repository pywikibot/core
2013-07-27 23:28:48,304 INFO zuul.Gerrit: Getting information for 76319,1
2013-07-27 23:28:48,415 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check-only>
2013-07-27 23:28:48,415 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core
2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check>
2013-07-27 23:28:48,417 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core
2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391410 76319,1> to <Pipeline check-voter>
2013-07-27 23:30:00,501 INFO zuul.Scheduler: Fetching references for pywikibot/core
2013-07-27 23:30:00,502 INFO zuul.Merger: Updating local repository pywikibot/core
2013-07-27 23:30:00,653 INFO zuul.Gerrit: Getting information for 76319,1
2013-07-27 23:31:32,970 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:31:32,971 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:31:33,091 INFO zuul.Gerrit: Getting information for 76318,1

$ tail -n100 /var/log/zuul/zuul.log
2013-07-27 22:47:54,239 INFO zuul.Gerrit: Getting information for 76288,1
2013-07-27 22:47:54,354 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94beef90 76288,1> to <Pipeline postmerge>
2013-07-27 22:47:54,354 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 22:52:10,847 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:52:10,847 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:52:10,972 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:54:14,905 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:54:14,906 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:54:15,036 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:54:15,151 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94bd4b10 76318,1> to <Pipeline gate-and-submit>
2013-07-27 22:54:15,152 INFO zuul.IndependentPipelineManager: Reporting start, action {'verified': 0} change <Change 0x7f2b94bd4b10 76318,1>
2013-07-27 22:54:15,234 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b9418c950 76318,1> to <Pipeline gate>
2013-07-27 22:54:15,235 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 22:54:15,236 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 22:54:15,236 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 22:54:15,356 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 22:56:22,230 INFO zuul.Scheduler: Fetching references for operations/mediawiki-config
2013-07-27 22:56:22,230 INFO zuul.Merger: Updating local repository operations/mediawiki-config
2013-07-27 22:56:22,696 INFO zuul.Gerrit: Getting information for 76237,2
2013-07-27 22:56:22,817 INFO zuul.Gerrit: Getting information for 76119,1
2013-07-27 22:57:49,863 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid
2013-07-27 22:57:49,863 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid
2013-07-27 22:57:50,244 INFO zuul.Gerrit: Getting information for 76315,2
2013-07-27 22:57:50,368 INFO zuul.Gerrit: Getting information for 76302,1
2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b9418cbd0 76315,2> to <Pipeline check-only>
2013-07-27 22:57:50,547 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid
2013-07-27 22:57:50,547 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94a36a10 76315,2> to <Pipeline test>
2013-07-27 22:57:52,171 INFO zuul.Jenkins: Launch job mwext-Parsoid-jslint for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:52,399 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 started, url: https://integration.wikimedia.org/ci/job/mwext-Parsoid-jslint/3099/
2013-07-27 22:57:52,598 INFO zuul.Jenkins: Launch job parsoid-parse-tool-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:52,808 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 started, url: https://integration.wikimedia.org/ci/job/parsoid-parse-tool-check/1015/
2013-07-27 22:57:53,022 INFO zuul.Jenkins: Launch job parsoid-roundtrip-test-check for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:53,349 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 started, url: https://integration.wikimedia.org/ci/job/parsoid-roundtrip-test-check/1012/
2013-07-27 22:57:53,446 INFO zuul.Jenkins: Launch job parsoid-parsertests-run-harder for change <Change 0x7f2b94a36a10 76315,2> with dependent changes []
2013-07-27 22:57:53,764 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 started, url: https://integration.wikimedia.org/ci/job/parsoid-parsertests-run-harder/1078/
2013-07-27 22:57:53,878 INFO zuul.Scheduler: Adding mediawiki/extensions/Parsoid, <Change 0x7f2b94334cd0 76315,2> to <Pipeline check-voter>
2013-07-27 22:57:53,878 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project mediawiki/extensions/Parsoid
2013-07-27 22:57:58,904 INFO zuul.Jenkins: Build 42eaabc1c06b490cb7f8f0cb494ff4f6 #3099 complete, status SUCCESS
2013-07-27 22:58:00,614 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: None
parsoid-roundtrip-test-check: None
parsoid-parsertests-run-harder: None

2013-07-27 22:58:01,725 INFO zuul.Jenkins: Build c25fbfb0324640f790a35554651ed5bf #1015 complete, status SUCCESS
2013-07-27 22:58:02,268 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: None
parsoid-parsertests-run-harder: None

2013-07-27 22:58:40,093 INFO zuul.Jenkins: Build 9b991d35362c46a0bbf79795b830b887 #1012 complete, status SUCCESS
2013-07-27 22:58:40,095 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: SUCCESS
parsoid-parsertests-run-harder: None

2013-07-27 22:59:02,017 INFO zuul.Jenkins: Build 1da6bca292444578af7582df9f1b6046 #1078 complete, status SUCCESS
2013-07-27 22:59:02,019 INFO zuul.IndependentPipelineManager: Change <Change 0x7f2b94a36a10 76315,2> status is now:
Project mediawiki/extensions/Parsoid change 76315,2

mwext-Parsoid-jslint: SUCCESS
parsoid-parse-tool-check: SUCCESS
parsoid-roundtrip-test-check: SUCCESS
parsoid-parsertests-run-harder: SUCCESS

2013-07-27 22:59:02,020 INFO zuul.IndependentPipelineManager: Reporting change <Change 0x7f2b94a36a10 76315,2>, action: {'verified': 2, 'code-review': 0}
2013-07-27 22:59:08,080 INFO zuul.Scheduler: Fetching references for mediawiki/extensions/Parsoid
2013-07-27 22:59:08,080 INFO zuul.Merger: Updating local repository mediawiki/extensions/Parsoid
2013-07-27 22:59:08,452 INFO zuul.Gerrit: Getting information for 76315,2
2013-07-27 22:59:08,577 INFO zuul.Gerrit: Getting information for 76302,1
2013-07-27 23:17:51,491 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,491 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:51,635 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 23:17:51,894 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,895 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:51,985 INFO zuul.Scheduler: Adding pywikibot/compat, <Ref 0x7f2b94391090 master updated 484f8cc860f192c72df8b9a54210df3c7cdd8ab8..91f565cf3d6c678df317db7d6f91563de95a1bcc> to <Pipeline post>
2013-07-27 23:17:51,985 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 23:17:51,986 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:17:51,987 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:17:52,103 INFO zuul.Gerrit: Getting information for 76318,1
2013-07-27 23:17:52,226 INFO zuul.Scheduler: Adding pywikibot/compat, <Change 0x7f2b94391f50 76318,1> to <Pipeline postmerge>
2013-07-27 23:17:52,226 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/compat
2013-07-27 23:28:48,091 INFO zuul.Scheduler: Fetching references for pywikibot/core
2013-07-27 23:28:48,093 INFO zuul.Merger: Updating local repository pywikibot/core
2013-07-27 23:28:48,304 INFO zuul.Gerrit: Getting information for 76319,1
2013-07-27 23:28:48,415 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check-only>
2013-07-27 23:28:48,415 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core
2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391b90 76319,1> to <Pipeline check>
2013-07-27 23:28:48,417 ERROR zuul.IndependentPipelineManager: Unable to find change queue for project pywikibot/core
2013-07-27 23:28:48,417 INFO zuul.Scheduler: Adding pywikibot/core, <Change 0x7f2b94391410 76319,1> to <Pipeline check-voter>
2013-07-27 23:30:00,501 INFO zuul.Scheduler: Fetching references for pywikibot/core
2013-07-27 23:30:00,502 INFO zuul.Merger: Updating local repository pywikibot/core
2013-07-27 23:30:00,653 INFO zuul.Gerrit: Getting information for 76319,1
2013-07-27 23:31:32,970 INFO zuul.Scheduler: Fetching references for pywikibot/compat
2013-07-27 23:31:32,971 INFO zuul.Merger: Updating local repository pywikibot/compat
2013-07-27 23:31:33,091 INFO zuul.Gerrit: Getting information for 76318,1

The events in question were supposed to trigger gate-and-submit on e.g. https://gerrit.wikimedia.org/r/#/c/76318/ which didn't happen (it left the comment that it started, but no build report was ever posted and no merge was performed).

I executed '/etc/init.d/zuul reload' (live reload, no restart, no lost events), and that caused the ghost entries to go away.

However that doesn't solve the problem, those changes didn't get a build report and no merge.

python/compat has jobs to run pep8 and pyflakes. They are only triggered when a python file is modified.

Change 76318 altered a .git file, thus Zuul does create an entry in the pipeline which is stuck waiting for nothing. That is a bug in Zuul :-]

This is most probably fixed upstream, we will see when we get Zuul upgraded (bug 50664).

I can confirm this has been fixed when we upgraded Zuul back in January. An example is a change against a python repository: https://gerrit.wikimedia.org/r/#/c/112036/ . The change did not trigger any job but the change has been reported as succeeding.