Page MenuHomePhabricator

deployment-graphite.eqiad.wmflabs went away?
Closed, ResolvedPublic

Description

The OCG service on beta is crashing because:

Host deployment-graphite.eqiad.wmflabs not found: 3(NXDOMAIN)

on deployment-pdf01.

Where did it go?


Version: unspecified
Severity: normal

Details

Reference
bz71031

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:55 AM
bzimport set Reference to bz71031.
bzimport added a subscriber: Unknown Object (MLST).

It has been deployed on Sep. 11 16:31 UTC by YuviPanda: Delete deployment-graphite instance.

Having statsd/graphite on labs instance did not fit the needs of beta cluster monitoring based on graphite. Instead, a real hardware box has been setup on the labs infrastructure and is maintained by ops. That is much more stronger.

One should thus use:

labmon1001.eqiad.wmnet 10.64.37.13

If it got broken, the host configuration should be in puppet and/or operations/mediawiki-config.git so it can be properly updated whenever it changes again.

It is, presumably, I just need to hunt down the puppet configuration for it. Mwalker set it up.

Ok, I've changed the statsd configuration from deployment-graphite.eqiad.wmflabs to labmon1001.eqiad.wmnet on both deployment-pdf01 and deployment-pdf02. Fingers crossed.