Page MenuHomePhabricator
Create Task
Maniphest T27205

Automate generation of fundraiser tracking data, long term storage, and data mining
Closed, DeclinedPublic
Actions

Description

Currently we generate stats on locke to one landingpage or banner impressions file. Then someone has to rsync it for long term archive storage on dataset. After, we have to run the appropriate mining script and ship the fundraiser data.

Lets automate this.

We'll need to setup a regular job that pulls a specific set of log files from locke to dataset1. Then a cron job can kick off at a specific point in time (say 30min post fundraiser) that will email the fundraisers with a csv version of the data.

Version: unspecified
Severity: enhancement

Details

Reference
bz25205

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 21 2014, 11:21 PM
• bzimport added a project: WMF-General-or-Unknown.
• bzimport set Reference to bz25205.
• Tfinc created this task.Sep 18 2010, 12:17 AM
• Tfinc added a comment.Sep 28 2010, 9:28 PM

The 2step proces is now well automated thanks to Nimish. Documentation to come next.

• bzimport added a comment.Oct 8 2010, 5:01 PM

ngautam wrote:

There is now a cron running on locke which has user file_mover automatically compress the data, send it to dataset1, run analysis on it, and email the CSV to tfinc@wikimedia at 55 min past the hour.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL