Change 124500 had a related patch set uploaded by BryanDavis:
Maintain file order when iterating wikiversions.json

https://gerrit.wikimedia.org/r/124500

Change 124500 merged by jenkins-bot:
Maintain file order when iterating wikiversions.json

https://gerrit.wikimedia.org/r/124500

The JSON standard specifically states that "an object is an unordered set of name/value pairs". If the order matters, it's a bug. Either the order shouldn't matter, or a JSON object is the wrong way to represent the data. Change I3c5ca78ae is a kludge, in my opinion.

Cpython will randomize the value used to seed its hashing algorithm when invoked with -R or when the PYTHONHASHSEED environment var is set to "random". On Python 3.3 and greater, hash randomization is turned on by default.

The patch I just submitted is at most a bandaid, but it's useful for testing the hypothesis.

Change 124508 had a related patch set uploaded by BryanDavis:
Revert "Maintain file order when iterating wikiversions.json"

https://gerrit.wikimedia.org/r/124508

Apparently testwiki is the only wiki version that creates a proper ExtensionMessages file. Or better stated, running mwscript mergeMessageFileList.php --wiki=testwiki generates a different ExtensionMessages file that using test2wiki or mediawikiwiki does and the testwiki variant seems to not cause l10nupdate-1 to break core translations.

Change 124508 merged by jenkins-bot:
Revert "Maintain file order when iterating wikiversions.json"

https://gerrit.wikimedia.org/r/124508

• mw-update-l10n gets version=wikidb data via mwversionsinuse
• mwversionsinuse reads from /a/common/wikiversions.json

• mwscript uses MWMultiVersion::initializeForMaintenance() to process the --wiki=<WIKIDB> argument and choose a php-1.XwmfY version to run scripts from.
• MWMultiVersion::getVersion() reads the compiled /usr/local/apache/common-local/wikiversions.cdb to find the version for a given database

• Scap does not compile /usr/local/apache/common-local/wikiversions.json into /usr/local/apache/common-local/wikiversions.cdb until the final step in the scap process.

This combination means that when mw-update-l10n runs mwscript mergeMessageFileList.php the version seen by mergeMessageFileList will always be the version that database was at on after the *prior* scap (or sync-wikiversions) execution. This is why it looks like ExtensionMessages-1.23wmf21 was built for the prior version; it was.

If you run scap twice without changing wikiversions.json, the second time mergeMessageFileList.php will see the expected version of MediaWiki.

The fix for this is to add a step early in scap (just after sync-common) to compile /usr/local/apache/common-local/wikiversions.json to wikiversions.cdb. This shouldn't be done before or during sync-common because that would push the updated version out to the apaches during their runs of sync-common and thus before they have compiled the l10n cache locally.

Change 124627 had a related patch set uploaded by BryanDavis:
Compile wikiversions.json to cdb for local sync

https://gerrit.wikimedia.org/r/124627

Change 124627 merged by jenkins-bot:
Compile wikiversions.json to cdb for local sync

https://gerrit.wikimedia.org/r/124627

Change 124762 had a related patch set uploaded by BryanDavis:
Compile wikiversions.json to cdb on deploy server

https://gerrit.wikimedia.org/r/124762

Change 124763 had a related patch set uploaded by BryanDavis:
Create symlink for compile-wikiversions in /usr/local/bin

https://gerrit.wikimedia.org/r/124763

Change 124763 merged by Alexandros Kosiaris:
Create symlink for compile-wikiversions in /usr/local/bin

https://gerrit.wikimedia.org/r/124763

Change 124762 merged by jenkins-bot:
Compile wikiversions.json to cdb on deploy server

https://gerrit.wikimedia.org/r/124762

Looks like it did the right thing:

$ ll /a/common/wmf-config/ExtensionMessages-1.23wmf2*.php
-rw-rw-r-- 1 bd808 wikidev 24179 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf20.php
-rw-rw-r-- 1 reedy wikidev 30070 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf21.php
-rw-rw-r-- 1 reedy wikidev 30288 Apr 10 16:16 /a/common/wmf-config/ExtensionMessages-1.23wmf22.php