Page MenuHomePhabricator

No new Pagecounts-all-sites files since 2014-10-10 17:00
Closed, ResolvedPublic

Description

http://dumps.wikimedia.org/other/pagecounts-all-sites/2014/2014-10/
shows the files for 2014-10-10 16:00 as most current, available hour.

Why are no new files appearing there?

Is this related to the refinery deploy from 2014-10-10?
(See also bug 71948)


Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=71948

Details

Reference
bz71994

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:50 AM
bzimport set Reference to bz71994.
bzimport added a subscriber: Unknown Object (MLST).

(In reply to christian from comment #0)

Is this related to the refinery deploy from 2014-10-10?
(See also bug 71948)

Seems so.
Oozie does not show the insert_hourly_pagecounts coordinator :-(

I submitted the insert_hourly_pagecounts coordinator, and the files are
getting generated again.

(Will keep this bug open, until all missing files have been generated)

Back-filling caught up.
We're back to normal.

otto wrote:

Oh, hm, we only re-submitted the bundles. Was there a coordinator we had to manually submit too?

Change 167185 had a related patch set uploaded by QChris:
Add overview diagram for Oozie jobs

https://gerrit.wikimedia.org/r/167185

(In reply to Andrew Otto from comment #4)

Was there a coordinator we had to
manually submit too?

Yes, the Oozie job that fills webstats table is only a coordinator.
See If29198245ac99b58b7bd6184cdc48920490d14b9.

The commit from comment 5 adds an overview over Oozie jobs, where
scopes are the job types.

I uploaded the rendered file to commons:

https://commons.wikimedia.org/wiki/File:Refinery-oozie-overview.png

(However, after linking it on

https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Oozie

it was visible for some seconds (minutes?) ... now it's only a link.
It least it's linked there.)

Change 167185 merged by Ottomata:
Add overview diagram for Oozie jobs

https://gerrit.wikimedia.org/r/167185