| Krinkle | |
| May 3 2013, 11:35 PM |
| F10882: Screen_Shot_2013-05-22_at_8.50.46_PM.png | |
| Nov 22 2014, 1:37 AM |
Screenshot of error page
Location: https://gerrit.wikimedia.org/r/#/c/62207/
HTTP ERROR: 500
Problem accessing /r/. Reason:
Guice provision errors:
1 error
Version: wmf-deployment
Severity: major
Attached:
Looks like this has escalated. Most users don't even get an error, it just times out. Gerrit has become completely unresponsive.
Gerrit was restared, https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log&diff=68980&oldid=68979. Seems to be working now.
In case this becomes relevant at some point of investigating the root cause:
~2 hours before this happened MatmaRex told me that gerrit was really slow
to respond. Not having access to logs, I checked ganglia, and the graphs
showed the usual increase around that time of the day.
The now reported error message surfaces if gerrit could connect to the
database at startup but now cannot open new connections to the
database.
Do the logs show a root cause?
Here's the full stacktraces from the failure: http://p.defau.lt/?B4I7N8sdqihW2Dj7tQOUNw
(In reply to comment #3)
Gerrit was restared, Seems to be working now.
Lowering priority.
Seen again with exactly the same error message, on https://gerrit.wikimedia.org/r/#/c/63824/ .
Created attachment 12371
Proxy error, usually accompanies the previously captured Guice error
The problem is back.
Attached:
The following error appeared today (a few minutes ago):
Guice provision errors:
1 error
And its back coming to eat more innocent souls:
Guice provision errors:
1 error
There's no one bug here so I'm not sure what this is supposed to accomplish.
"Don't break the database connection with bad config, networking hiccups or otherwise crash the ability of Gerrit to talk to its DB?"
Happened again just now:
Guice provision errors: 1) Cannot open ReviewDb at com.google.gerrit.server.util.ThreadLocalRequestContext$1.provideReviewDb(ThreadLocalRequestContext.java:70) while locating com.google.gerrit.reviewdb.server.ReviewDb 1 error
The problem seems to not happen any more in over 1 and half years.
We are upgrading to gerrit 2.12 soon so maybe that will finally fix anything.
@Paladox: Noone is aware of any fix hence this is not resolved... Declining for the time being.
Please reopen if still happening with 2.12 (to be deployed soon).
Yeah, but we are talking about upgrading gerrit.wikimedia.org, not some other subdomain. Plus not really relevant for this task.
This was not fixed on the existing install and should not be declined either because it's a real bug. Can resolve after upgrade.
So I'm closing this as invalid. Couple of things:
I'm not sure what else is actionable here.