this is a test ticket for the gerrit->BZ notification bot/gerrit plugin
Version: wmf-deployment
Severity: major
Dzahn | |
May 15 2014, 8:38 PM |
F13682: hooks-bugzilla.src.zip | |
Nov 22 2014, 3:14 AM |
this is a test ticket for the gerrit->BZ notification bot/gerrit plugin
Version: wmf-deployment
Severity: major
Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.
[2014-05-15 20:49:56,495] ERROR com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor : Error while executing action status PATCH_TO_REVIEW
java.io.IOException: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410
at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:189) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.performAction(BugzillaItsFacade.java:97) at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:72) at com.googlesource.gerrit.plugins.hooks.workflow.ActionExecutor.execute(ActionExecutor.java:84) at com.googlesource.gerrit.plugins.hooks.workflow.ActionController.onChangeEvent(ActionController.java:55) at com.google.gerrit.common.ChangeHookRunner.fireEventForUnrestrictedListeners(ChangeHookRunner.java:619) at com.google.gerrit.common.ChangeHookRunner.fireEvent(ChangeHookRunner.java:630) at com.google.gerrit.common.ChangeHookRunner.doPatchsetCreatedHook(ChangeHookRunner.java:356) at com.google.gerrit.server.change.ChangeInserter.insert(ChangeInserter.java:183) at com.google.gerrit.server.change.CherryPickChange.createNewChange(CherryPickChange.java:244) at com.google.gerrit.server.change.CherryPickChange.cherryPick(CherryPickChange.java:182) at com.google.gerrit.server.change.CherryPick.apply(CherryPick.java:84) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:627) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:51) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:744)
Caused by: com.j2bugzilla.base.BugzillaException: An unknown error was encountered; fault code: 410
at com.j2bugzilla.base.XmlExceptionHandler.handleFault(XmlExceptionHandler.java:96) at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:169) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaClient.performAction(BugzillaClient.java:128) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.doPerformAction(BugzillaItsFacade.java:109) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.access$400(BugzillaItsFacade.java:31) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:102) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade$3.call(BugzillaItsFacade.java:97) at com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade.execute(BugzillaItsFacade.java:177) ... 56 more
Caused by: org.apache.xmlrpc.XmlRpcException: You must log in before using this part of Bugzilla.
at org.apache.xmlrpc.client.XmlRpcStreamTransport.readResponse(XmlRpcStreamTransport.java:197) at org.apache.xmlrpc.client.XmlRpcStreamTransport.sendRequest(XmlRpcStreamTransport.java:156) at org.apache.xmlrpc.client.XmlRpcHttpTransport.sendRequest(XmlRpcHttpTransport.java:143) at org.apache.xmlrpc.client.XmlRpcSunHttpTransport.sendRequest(XmlRpcSunHttpTransport.java:69) at org.apache.xmlrpc.client.XmlRpcClientWorker.execute(XmlRpcClientWorker.java:56) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:167) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:137) at org.apache.xmlrpc.client.XmlRpcClient.execute(XmlRpcClient.java:126) at com.j2bugzilla.base.BugzillaConnector.executeMethod(BugzillaConnector.java:164) ... 62 more
12:40 < JohnLewis> Hm. Gerrit not notifying Bugzilla of patches?
12:41 < mutante> JohnLewis: do you know since when?
12:42 < JohnLewis> mutante: https://gerrit.wikimedia.org/r/#/c/133364/ was uploaded yesterday and the Bugzilla bug was not notified. That's as far as I know.
12:43 < JohnLewis> I just patched a bug and that wasn't updated so it's still an on-going issue for reference
12:50 < mutante> http://www.mediawiki.org/wiki/Gerrit/Commit_message_guidelines#Auto-linking_and_cross-referencing
12:50 < mutante> "A bot will automatically notify Bugzilla ... "
12:52 < mutante> ^d: ideas why that could have changed recently?
12:52 < ^d> godog restarting gerrit?
12:53 < mutante> i hear it broke within the last 9 hours
12:53 < mutante> ah
12:53 < andre> so the plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla
12:54 < andre> and qchris set it up once upon a time
12:55 < andre__> very last activity in Bugzilla was Gerrit Notification Bot (gerritadmin@wikimedia.org) 2014-05-15 11:41:52 UTC in
https://bugzilla.wikimedia.org/show_bug.cgi?id=65225#c3
13:01 < mutante> the bugzilla plugin is listed as enabled in gerrit
13:02 < mutante> hooks-bugzilla2.8.1 Enabled .. hrmm
13:02 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/index.html
13:03 < mutante> https://gerrit.wikimedia.org/r/plugins/hooks-bugzilla/Documentation/config.html even
13:03 < mutante> "Bugzilla credentials and connectivity details are asked and verified during the Gerrit init."
13:03 < mutante> that?
13:12 < andre__> mutante, gerritadmin@wikimedia.org
13:13 < ^d> fwiw, gerrit is connecting.
13:13 < ^d> [2014-05-15 12:57:28,709] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaModule : Bugzilla is configured as ITS
13:13 < ^d> [2014-05-15 12:57:30,106] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to https://bugzilla.wikimedia.org as
username=gerritadmin@wikimedia.org, userid=16761
13:13 < ^d> [2014-05-15 12:57:30,646] INFO com.googlesource.gerrit.plugins.hooks.bz.BugzillaItsFacade : Connected to Bugzilla at
https://bugzilla.wikimedia.org/xmlrpc.cgi, reported version is 4.4.4
13:58 < andre__> Bugzilla DID change some cookie/login related stuff between 4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago
13:20 < ^d> Hmm, I reloaded the plugin and cleared the error log.
13:21 < ^d> Once again, Connected to https://bugzilla.wikimedia.org as username=gerritadmin@wikimedia.org, userid=16761
13:21 < ^d> Wonder if it'll last.
..but then it didn't
Gerrit log excerpt above from /var/lib/gerrit2/review_site/log/error_log
There were firewall changes but timing does not fit, as last activity in Bugzilla by Gerrit Notification Bot (gerritadmin@wikimedia.org) was 2014-05-15 11:41:52 UTC in bug 65225 comment 3.
<mutante> ^d: did you know about openjdk being upgraded btw?
<mutante> @ytterbium:/var/log# grep jdk /var/log/apt/history.log
<andre> plugin in Gerrit is https://gerrit-review.googlesource.com/#/admin/projects/plugins/its-bugzilla
<andre> for the records, the quick'n'dirty check if it's running again would be any patch created in Gerrit and correctly refering to a Bugzilla ID, which would create a result shown for https://bugzilla.wikimedia.org/buglist.cgi?chfield=bug_status&chfieldfrom=-1h&chfieldto=Now&chfieldvalue=PATCH_TO_REVIEW&email1=gerritadmin%40wikimedia.org&emaillongdesc1=1&emailtype1=exact&query_format=advanced (that query covers the last one hour)
yea, regarding the openjdk update, no idea if it's related but:
Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk
Install: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2)
Upgrade: icedtea-6-jre-cacao:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-lib:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre-headless:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), icedtea-6-jre-jamvm:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2), openjdk-6-jre:amd64 (6b30-1.13.1-1ubuntu2~0.12.04.3, 6b31-1.13.3-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:48:13
Start-Date: 2014-05-15 11:49:39
Commandline: apt-get purge openjdk-6-jdk
Purge: openjdk-6-jdk:amd64 (6b31-1.13.3-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:49:39
Start-Date: 2014-05-15 11:49:57
Commandline: apt-get install openjdk-7-jdk
Install: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-lib:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, automatic)
Downgrade: openjdk-7-jre:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2), openjdk-7-jre-headless:amd64 (7u51-2.4.4-0ubuntu0.12.04.2, 7u25-2.3.10-1ubuntu0.12.04.2)
Remove: icedtea-7-jre-jamvm:amd64 (7u51-2.4.4-0ubuntu0.12.04.2)
End-Date: 2014-05-15 11:50:02
Start-Date: 2014-05-15 11:51:04
Commandline: apt-get remove openjdk-7-jdk
Remove: openjdk-7-jdk:amd64 (7u25-2.3.10-1ubuntu0.12.04.2)
End-Date: 2014-05-15 11:51:04
Start-Date: 2014-05-15 11:53:14
Commandline: apt-get install openjdk-7-jre=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jdk=7u55-2.4.7-1ubuntu1~0.12.04.2 openjdk-7-jre-headless=7u55-2.4.7-1ubuntu1~0.12.04.2
Install: openjdk-7-jdk:amd64 (7u55-2.4.7-1ubuntu1~0.12.04.2)
Upgrade: openjdk-7-jre:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2), openjdk-7-jre-headless:amd64 (7u25-2.3.10-1ubuntu0.12.04.2, 7u55-2.4.7-1ubuntu1~0.12.04.2)
End-Date: 2014-05-15 11:53:18
(In reply to Daniel Zahn from comment #5)
yea, regarding the openjdk update, no idea if it's related but:
Start-Date: 2014-05-15 11:48:08
Commandline: apt-get install openjdk-6-jdk
Assuming that time is UTC, that fits reasonably well to the last activity in Bugzilla at 11:41:52 UTC.
(In reply to Daniel Zahn from comment #2)
13:58 < andre__> Bugzilla DID change some cookie/login related stuff between
4.4.1 and 4.4.4, but we upgraded to 4.4.4 more than one day ago
To clarify what I refered to, quoting the 4.4.3 release notes:
"The User.login WebService method now also returns a token argument containing a login token which you can use in subsequent calls to authenticate. For security reasons, this method no longer generates login cookies. ( https://bugzilla.mozilla.org/show_bug.cgi?id=893195 )"
The openjdk change time fits really, really nicely. But I it should
(yes. Vague :-/ ) not affect us here, as Gerrit uses the java implementation
that's referenced in it's config file (key: container.javaHome), which is
currently
/usr/lib/jvm/java-7-openjdk-amd64/jre
I checked, and this jvm seems basically functional still.
I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.
(Bugzilla 4.2.4 works nicely)
Prodding down that line, it seems comment #8 is relevant.
(Bugzilla's 410 error is "login_required")
j2bugzilla (the third part bugzilla xmlrpc library used in hooks-bugzilla) is
still current, and has not seen adaptions for bugzilla 4.4.3.
So I'll see if j2bugzilla needs fixing for the login change of bugzilla 4.4.3.
(In reply to christian from comment #9)
I tried a local bugzilla-4.4.4 instance, and can reproduce the issue there.
Weird, because we upgraded Bugzilla to 4.4.4 already on 2014-04-22 (bug 61287 comment 5) and the notifications still worked until yesterday.
(Oops, potentially confusing last comment: We skipped the broken 4.4.3 and upgraded from 4.4.1 or 4.4.2 directly to 4.4.4)
Had a chat with qchris who asked if downgrading Bugzilla to 4.4.2 is possible (otherwise fixing would take a bit longer).
We're still wondering why the problem only arose yesterday because our 4.4.4 upgrade was three weeks ago (cached login cookie? shrug).
My guess is that reverting http://bzr.mozilla.org/bugzilla/4.4/revision/8664 (for https://bugzilla.mozilla.org/show_bug.cgi?id=893195 ) should be sufficient.
As I got to leave in 15min (will be back later tonight) I'd love to get an opinion from mutante.
Change 133589 had a related patch set uploaded by QChris:
test gerrit->bz notification
Deployed a patched j2bugzilla/hooks-bugzilla plugin that allows token passing
for bugzilla-4.4.4 instead of using sessions.
Seems to be working again (see comment 13).
Change 133732 had a related patch set uploaded by QChris:
Update hoobs-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76
Change 133589 abandoned by QChris:
test gerrit->bz notification
Reason:
gerrit -> bugzilla notifications are working again.
Change 133732 merged by Chad:
Update hooks-bugzilla to 5edd392d926daaa58917b1c8bb174cdb022e4c76
aric wrote:
Hi guys, I would like to get this change pushed upstream to https://gerrit.googlesource.com/plugins/its-bugzilla/
I only see a binary diff in gerrit, can I get a link to the source code changed?
Thanks!
-Aric
Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaClient.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class and old/com/googlesource/gerrit/plugins/hooks/bz/BugzillaItsFacade.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/IssueExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class and old/com/googlesource/gerrit/plugins/hooks/util/PropertyExtractor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class and old/com/googlesource/gerrit/plugins/hooks/validation/ItsValidateComment.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class and old/com/googlesource/gerrit/plugins/hooks/workflow/action/LogEvent.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionController.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class and old/com/googlesource/gerrit/plugins/hooks/workflow/ActionExecutor.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class and old/com/googlesource/gerrit/plugins/hooks/workflow/GerritHookFilterChangeState$Transition.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class and old/com/googlesource/gerrit/plugins/hooks/workflow/RuleBase.class differ
Binary files new/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class and old/com/googlesource/gerrit/plugins/hooks/workflow/Rule.class differ
Binary files new/com/j2bugzilla/rpc/BugzillaVersion.class and old/com/j2bugzilla/rpc/BugzillaVersion.class differ
Binary files new/com/j2bugzilla/rpc/CommentBug.class and old/com/j2bugzilla/rpc/CommentBug.class differ
Binary files new/com/j2bugzilla/rpc/GetBug.class and old/com/j2bugzilla/rpc/GetBug.class differ
Binary files new/com/j2bugzilla/rpc/GetLegalValues.class and old/com/j2bugzilla/rpc/GetLegalValues.class differ
Binary files new/com/j2bugzilla/rpc/LogIn.class and old/com/j2bugzilla/rpc/LogIn.class differ
Binary files new/com/j2bugzilla/rpc/LogOut.class and old/com/j2bugzilla/rpc/LogOut.class differ
Binary files new/com/j2bugzilla/rpc/UpdateBug.class and old/com/j2bugzilla/rpc/UpdateBug.class differ
Only in new/: hooks-bugzilla_new-01f21a43649f50302f9d4ff2cff7d7bad6a4ffb9.jar
Only in old/: hooks-bugzilla_old-bd5dce77c691d823caafd79df7a58a7a992c0687.jar
diff -r new/META-INF/MANIFEST.MF old/META-INF/MANIFEST.MF
3c3
Implementation-Version: 2.8.1
8c8
Gerrit-ApiVersion: 2.8.1
11c11
Build-Jdk: 1.6.0_24
diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.properties
2,3c2,3
< #Fri May 16 18:01:44 CEST 2014
#Mon Jan 27 13:37:07 CET 2014
version=2.8.1
diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml old/META-INF/maven/com.googlesource.gerrit.plugins.bugzilla/hooks-bugzilla/pom.xml
24c24
<version>2.8.1</version>
99c99
<version>${project.version}</version>
104c104
<version>2.2</version>
diff -r new/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties old/META-INF/maven/com.googlesource.gerrit.plugins.its/hooks-its/pom.properties
2c2
#Mon Jan 27 13:35:30 CET 2014
diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.properties
1,5c1,5
< #Generated by Maven
< #Fri May 16 18:01:15 CEST 2014
< version=2.3-SNAPSHOT
< groupId=com.j2bugzilla
#Generated by Maven
#Thu Jun 13 00:23:50 EDT 2013
version=2.2
groupId=com.j2bugzilla
artifactId=j2bugzilla
diff -r new/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml old/META-INF/maven/com.j2bugzilla/j2bugzilla/pom.xml
12c12
<version>2.2</version>
114d113
< <!--
129d127
< -->
160c158
</project>
\ No newline at end of file
ehm... ??
aric wrote:
@dan, Yeah, from the contents of the .jar I can't actually see inside the .class files to see what was changed to make the gerrit -> Buzilla posting start working. I am considering emailing qchris, but I thought it more polite to ask here first. I'd like to help make sure that the its-bugzilla plugin continues being viable going forward. Currently it's support is stuck at gerrit 2.9. Though I'm no java programmer, I'm imagining that If I can lay the task out clearly that I'll be able to enlist someone.
(In reply to Aric Gardner from comment #19)
Hi guys, I would like to get this change pushed upstream to
https://gerrit.googlesource.com/plugins/its-bugzilla/
I only see a binary diff in gerrit, can I get a link to the source code
changed?
A patch to hooks-bugzilla is not sufficient, I had to patch j2bugzilla too.
I sent you the source files by email.
(The fix is an ugly hack. I do not like it.)
(Meanwhile (in June) j2bugzilla saw an update too. Maybe that allows for
an easier fix?)