Page MenuHomePhabricator

Database problems. Unable to write and intermittent problems on reading
Closed, ResolvedPublic

Description

I'm unable to write to the database. Programs are just hanging. Having troubles reading in that it can take awhile to receive data. Things were ok 12 hours ago. Looks like problem has been happening atleast for 3 hours.


Version: unspecified
Severity: normal

Details

Reference
bz60382

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:59 AM
bzimport added a project: Toolforge.
bzimport set Reference to bz60382.

Now unable to connect to database via the website.

I get "Could not connect to database: User 'p50380g50450' has exceeded the 'max_user_connections' resource (current value: 80)"

I only have on hung program on the queue and nothing else that should be doing so many connctions.

I can access the database again. Coincidentally(?), about 50 mysql processes are no longer running on tools-login. Though there are still a few running by users since last year.

Which database server were you connecting to, i. e. tools-db or the replicas?

(We had a full disk on tools-db, so that could have saturated that server (accepting connections, but then not being able to answer queries).)

Sorry, that would have been helpful: tools-db
Having a full disk could have caused. Thanks for telling me.

This was, indeed, caused by a disk full on the (now gone) old database instance.