Page MenuHomePhabricator

Raw webrequest partitions for 2014-10-21T11/1H not marked successful
Closed, DeclinedPublic

Description

None of the webrequest partitions [1] for 2014-10-21T11/1H have been
been marked successful.

What happened?

[1]


qchris@stat1002 jobs: 0 time: 10:12:12 // exit code: 0
cwd: ~
~/cluster-scripts/dump_webrequest_status.sh

+------------------+--------+--------+--------+--------+
| Date             |  bits  | mobile |  text  | upload |
+------------------+--------+--------+--------+--------+

[...]

| 2014-10-21T09/1H |    .   |    .   |    .   |    .   |    
| 2014-10-21T10/1H |    .   |    .   |    .   |    .   |    
| 2014-10-21T11/1H |    X   |    X   |    X   |    X   |    
| 2014-10-21T12/1H |    .   |    .   |    .   |    .   |    
| 2014-10-21T13/1H |    X   |    X   |    .   |    .   |

[...]

+------------------+--------+--------+--------+--------+

Statuses:

. --> Partition is ok
M --> Partition manually marked ok
X --> Partition is not ok (duplicates, missing, or nulls)

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


Version: unspecified
Severity: normal

Details

Reference
bz72352

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:56 AM
bzimport set Reference to bz72352.
bzimport added a subscriber: Unknown Object (MLST).

(2014-10-21T13/1H is handled in bug 72353)

The affected period is 2014-10-21T11:41:19/2014-10-21T11:59:09.
It affected only ulsfo caches, but all ulsfo caches.

The affected period shows ~5M duplicates, which are worth

  • 4 minutes of ulsfo data, or
  • 36 seconds of total data.

The affected period shows ~6M missing lines, which are worth

  • 5 minutes of ulsfo data, or
  • 41 seconds of total data.

Ops reported [1] that the ulsfo->eqiad connection again caused issues.
According to the IRC logs [2], the connection issues started around
10:30, which matches the affected period.

[1] https://lists.wikimedia.org/mailman/private/ops/2014-October/042427.html
[2] http://bots.wmflabs.org/~wm-bot/logs/%23wikimedia-operations/20141021.txt