Page MenuHomePhabricator

Logstash stopped recording apache2 events (fatalmonitor) at 2014-02-01
Closed, ResolvedPublic

Description

It appears that no events of the "apache2" type have been stored in logstash since 2014-01-31T23:59:59.000Z. These events are used to drive the fatalmonitor dashboard.

I tried restarting the logstash instance on logstash1001 but this did not seem to restore the event stream. Events of other types are being recorded which are delivered via the udp2log transport stream. fluorine.eqiad.wmnet is still adding new lines to /a/mw-log/apache2.log.


Version: wmf-deployment
Severity: normal

Details

Reference
bz60772

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:06 AM
bzimport set Reference to bz60772.

The root cause seems to be a bad grok parse pattern. See https://gerrit.wikimedia.org/r/#/c/110971 for a fix that matches config that I have manually deployed on logstash1001.

I forced a puppet run after the merge and was able to verify the fix.