Generate data on the cluster, move it on labsdbs for wikimetrics or make it directly available to Vital Signs
There are 2 metrics: Monthly Pageviews and Daily Pageviews.
They are broken down by project (wiki) and by target-site (same definition of target site as in bug https://bugzilla.wikimedia.org/show_bug.cgi?id=72737
Version: unspecified
Severity: enhancement
Whiteboard: u=VSUser c=Dashiki p=34 s=2014-11-13
Change 169974 had a related patch set uploaded by Milimetric:
Transform projectcounts hourly files
Change 172195 had a related patch set uploaded by QChris:
Add basic implementation of projectcount aggregation
Change 172196 had a related patch set uploaded by QChris:
Add basic monitoring script for projectcount aggregates
Change 172197 had a related patch set uploaded by QChris:
Allow additional logging to disk for projectcounts aggregation
Change 172198 had a related patch set uploaded by QChris:
Add switch for automatic pushing of data repo for projectcounts aggregation
Change 172181 had a related patch set uploaded by QChris:
Create empty CSVs for relevant wikis
Change 172182 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2008
Change 172183 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2009
Change 172185 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2010
Change 172186 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2011
Change 172187 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2012
Change 172188 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2013
Change 172189 had a related patch set uploaded by QChris:
Backfill daily projectcounts for 2014 up to 2014-09-22
Change 172190 had a related patch set uploaded by QChris:
Backfill daily projectcounts up to 2014-11-08
Change 172201 had a related patch set uploaded by QChris:
Add jobs for aggregating hourly projectcount files to daily per wiki csvs
Change 172285 had a related patch set uploaded by QChris:
Link aggregator dataset into wikimetrics public webspace
Per discussions last week, we changed the scope to daily page views only.
Monthly page views is another story.
Change 172195 merged by jenkins-bot:
Add basic implementation of projectcount aggregation
Change 172196 merged by jenkins-bot:
Add basic monitoring script for projectcount aggregates
Change 172197 merged by jenkins-bot:
Allow additional logging to disk for projectcounts aggregation
Change 172198 merged by jenkins-bot:
Add switch for automatic pushing of data repo for projectcounts aggregation
Change 172189 merged by Mforns:
Backfill daily projectcounts for 2014 up to 2014-09-22
Interesting! Will the CSVs end up on http://datasets.wikimedia.org/aggregate-datasets/ or elsewhere? Wherever they end up, please add a link from the raw files (under http://dumps.wikimedia.org/other/ ).
The cvs will be in a repo that we were thinking of checking out under wikimetrics directory for the time coming but with a git clone so anyone can check it anywhere. At this time we are vetting out the data, though.
Change 172285 merged by Ottomata:
Link aggregator dataset into wikimetrics public webspace
UI is showing '0' values for lines that didn't exist (Zero) before a certain date
These values need to be changed to 'undefined'
This may break the metric.
Change 169974 abandoned by Milimetric:
Transform projectcounts hourly files
Reason:
turned into the analytics/aggregator project