Too many connection errors in mysql
Closed, ResolvedPublic
Actions

Assigned To

None

Authored By

	Hydriz
	May 29 2012, 4:19 AM

Description

Recently, the deployment-prep cluster is experiencing many technical failures with regards to the MySQL server (I believe?). A sample output error:

Can't contact the database server: Host 'i-0000026f.pmtpa.wmflabs' is blocked because of many connection errors; unblock with 'mysqladmin flush-hosts' (deployment-sql)

This blocks *everyone* (including bots) from accessing the beta cluster, and defeats the purpose of being an alternative beta cluster.

Please look into this issue. Its a blocker.

Version: unspecified
Severity: blocker

Details

Reference: bz37173

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 12:26 AM

• bzimport added a project: Beta-Cluster-Infrastructure.

• bzimport set Reference to bz37173.

• bzimport added a subscriber: Unknown Object (MLST).

Hydriz created this task.May 29 2012, 4:19 AM

Someone spammed the beta cluster which caused the apaches to generate a ton of connections which got them blacklisted.

I ran FLUSH HOSTS; and raised max_connect_errors;

For the record, the incident occurred between 2012/05/28 21:45 UTC and roughly 2012/05/29 02:00 UTC. Every apaches show a nice CPU spike.

I have banned all public IP ranges from Amazon Elastic Cloud. That will prevents their crawlers to hammer our servers.

How did you perform the ban?

squid config, apache config, iptables, rules from labsconsole... ?

From squid, see /etc/squid/squid.conf on deployment-squid.

I have just fixed the config right now.

Too many connection errors in mysqlClosed, ResolvedPublicActions

Description

Details

Event Timeline

Too many connection errors in mysql
Closed, ResolvedPublic
Actions