This ticket needs Ops power. I filed RT 8554 for it.

https://ganglia.wikimedia.org/latest/?r=day&cs=&ce=&c=Miscellaneous+eqiad&h=stat1001.wikimedia.org&tab=m&vn=&hide-hf=false&mc=2&z=medium&metric_group=ALLGROUPS showa that at some point 1 GB memory was freed and then traffic dropped.

Hoo concluded that apache2 died and puppet doesn't configure the machine to restart it.

Change 164914 had a related patch set uploaded by QChris:
End stats.wikimedia.org certificate in newline

https://gerrit.wikimedia.org/r/164914

Change 164914 merged by Filippo Giunchedi:
End stats.wikimedia.org certificate in newline

https://gerrit.wikimedia.org/r/164914

godog restarted apache on stat1001.

https://stats.wikimedia.org/
https://datasets.wikimedia.org/

are working again.

It seems certificate chaining choked on stats.wikimedia.org's
certificate not ending in a newline. Stop-gap fix is in commet #3.
But godog and _joe_ said this setting should be caught by the
certificate chaining itself, which makes sense.
The RT ticket has been updated accordingly.

Thanks godog and _joe_!

Page request that match "undefined" from the october sample logs so far.

Page request that match "undefined" from the october sample logs so far.

attachment undefined-october.txt.gz ignored as private

Please ignore prior assignment.