Page MenuHomePhabricator

Raw webrequest partitions for 2014-08-29T20:xx:xx not marked successful
Closed, ResolvedPublic

Description

For the hour 2014-08-29T20:xx:xx, none [1] of the the four sources'
bucket was marked successful.

What happened?

[1]


qchris@stat1002 jobs: 0 time: 11:19:13 // exit code: 0
cwd: ~/cluster-scripts
./dump_webrequest_status.sh

+---------------------+--------+--------+--------+--------+
| Date                |  bits  |  text  | mobile | upload |
+---------------------+--------+--------+--------+--------+

[...]

| 2014-09-29T18:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-09-29T19:xx:xx |    X   |    .   |    .   |    .   |    
| 2014-09-29T20:xx:xx |    X   |    X   |    X   |    X   |    
| 2014-09-29T21:xx:xx |    .   |    .   |    .   |    .   |    
| 2014-09-29T22:xx:xx |    .   |    .   |    .   |    .   |    
+---------------------+--------+--------+--------+--------+

Statuses:

. --> Partition is ok
X --> Partition is not ok (duplicates, missing, or nulls)

For the bits failures for 2014-09-29T18:xx:xx, and 2014-09-29T19:xx:xx
see bug 71435.


Version: unspecified
Severity: normal

Details

Reference
bz71463

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:50 AM
bzimport set Reference to bz71463.
bzimport added a subscriber: Unknown Object (MLST).

The issue covered each and every cache.
For each cache, at some point between 20:30:00 and 20:57:00, partition
numbers reset, and the minute before the reset, we see missing lines
(no duplicates).

This nicely matches merge of resetting queue.buffering.max.ms

b62d61a0eda950f202484a4d27972405cb6f124d

and the subsequent scattered puppet runs.