https://mingle.corp.wikimedia.org/projects/analytics/cards/342

nilesh wrote:

Could you give some more info about the problem? I can see that this has something to do with parallelizing some task, but can't be sure as to what it exactly entails. Some pointers please?

Sorry if it's too obvious.

See https://bugzilla.wikimedia.org/show_bug.cgi?id=60826, where a somewhat less elegant and less powerful but simpler solution is proposed, and more explanation is given.

Main difference: here the idea is to allow processing of any number of wikis in parallel, by removing any chance that files will be updated simultaneously by several threads. In 60826 the idea is to keep the limitation of processing one wiki per project, but stub and full archive dumps will be processed round robin on different servers. Periodically a rather small script takes care of harvesting
the extra metrics from key metrics file StatisticsMontly.csv on the full archive server and update empty columns in same file on stub dump server). 60826 is less elegant as it requires syncing between two servers, and less powerful as it still doesn't allow ad hoc processing of dumps without suspending round robin process, but it is much simpler to implement. This bug would require maintenance in tens of places in several source files as Wikistats counts job generates so many files.