Root cause of traffic drop is that we had to retstart udp2log to make effective our removal of one of the filters. Alarms were raised for what seemed a longer event but the traffic drop just lasted a few minutes.

See request drop around 22:45 hours on the 16th for mobile traffic:

2014-08-16T22:39 4482
2014-08-16T22:40 4391
2014-08-16T22:41 4408
2014-08-16T22:42 4354
2014-08-16T22:43 1628
2014-08-16T22:44 4
2014-08-16T22:46 1
2014-08-16T22:47 3
2014-08-16T22:48 631
2014-08-16T22:49 4419
2014-08-16T22:50 4460
2014-08-16T22:51 4312

A graph that ilustrates the same drop: http://i.imgur.com/7p96Wmh.png

Resolving bug and updating log of events that affect feeds on wikitech.

Reopening as it looks there were other intervals affected.

We had alarms about traffic drop on Aug 16th 23:03 (packetloss), Aug 17th 00:11, Aug 17th 05:51, Aug 17th 07:51 (oxygen).

Recovery was sent on Aug 17th 04:51 (packetloss) and Aug 17th 13:17 (oxygen).

Looked at the files for 17th and 18th again and the only event I can find with significant loss is printed below.

2014-08-17T06:24 3474
2014-08-17T06:25 3516
2014-08-17T06:26 1385 (*)
2014-08-17T06:29 3059
2014-08-17T06:30 3605

Assigning to Christian per his request to take all prod issues in the upcoming weeks.

Oxygen's alarms (see comment #3) around packet loss and udp2log from
2014-08-16 23:03:09 until 2014-08-17 07:51:08 were just artifacts of
bug 69661.

Loss in TSVs (from comment #1 and comment #3) is real though.

The loss on 2014-08-16 ~22:46 was due to the root mount effectively
getting full, hence services panicing, CPU usage jumping up.

The loss starting on 2014-08-16 ~06:25 was due to logrotation kicking
in, and reshuffling some files on the root mount a bit. Thereby, a bit
of disk space was freed up for <20mins, and services recovered a bit
until the root mount got full again. CPU usage going up further.

The losses affected all the multicast udp2log filters on oxygen:

zero tsvs
edits tsvs
mobile-sampled-100 tsvs
5xx tsvs
webstatscollector