(In reply to christian from comment #0)

Looking at the Ganglia graphs, it seems we'll see the same issue also
for today (2014-09-29).

Yes, we did.
The affected period is 2014-09-29T18:41:48--2014-09-29T19:55:21.
Again, only all esams bits caches.
Again, both duplicate and missing lines.

Ottomata restarted varnishkafka on cp3019 on 19:41, and cp3019
immediately recovered. Its queues being back to normal, and no longer
getting critical again. No more losses on cp3019.

This nicely matches yesterday's theory of esams bits traffic spikes are
above what 3 brokers can take.

It happened again during for the 5 bits partitions from
2014-10-14T16:xx:xx up to and including 2014-10-14T20:xx:xx.
Again only esams bits.

Since I've been around when it happened, and historic ganglia graphs
don't expose this: The kafka drerr's were not constant, but grew,
died off again, and stayed off for the rest of the interval in
intervals that were ~25-minutes long.
(See attachment kafka.varnishkafka.kafka_drerr.per_second-2014-10-15.png)

All affected caches showed this ~25-minutes long pattern.
But the pattern was not synchronous across machines.

While the drerrs showed this pattern, the outbuf_cnt did not show such
a pattern. It was high the whole time.

Created attachment 16773
kafka.varnishkafka.kafka_drerr.per_second-2014-10-15.png

Attached:

It happened again for 2014-10-16T17:xx:xx up to and including 2014-10-16T19:xx:xx