Seems ok this week.. :/

(In reply to comment #1)

Seems ok this week.. :/

What was different? Were any new extensions deployed?

(In reply to comment #2)

(In reply to comment #1)

Seems ok this week.. :/

What was different? Were any new extensions deployed?

Nothing I don't think. It should create a ExtensionMessages-1.22wmfXX.php each time

Would be great to update this after today's deployment, either as WORKSFORME or as still valid.

(In reply to comment #4)

Would be great to update this after today's deployment, either as WORKSFORME
or
as still valid.

We're not deploying a new version, scap won't be used today

So what's the state of this, a few weeks later?

This seems to have come back again:

reedy@tin:/a/common$ scap testwiki to 1.23wmf4 and build l10n cache
Checking syntax... done
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.23wmf3.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf3... done
Updating ExtensionMessages-1.23wmf2.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf2... done
Updating ExtensionMessages-1.23wmf4.php...PHP Warning: require(/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636

Warning: require(/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php' (include_path='/a/common/php-1.23wmf4/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf4:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf4/../wmf-config/ExtensionMessages-1.23wmf4.php' (include_path='/a/common/php-1.23wmf4/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf4:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2636
Update of MediaWiki localisation messages failed
sync failed
reedy@tin:/a/common$ ./multiversion/activeMWVersions
1.23wmf3 1.23wmf2 1.23wmf4
reedy@tin:/a/common$

Reopened, still highest/major?

Not exactly, it's easily "worked around" (make a copy of the previous versions file) I just don't get why it works fine for a few weeks, and then it doesn't. Which is annoying

Not seen this for a few weeks... Seems to have come back again today

reedy@tin:/a/common$ scap testwiki to 1.23wmf13 and build l10n cache
[2014-02-06 16:04:31,041] started scap: testwiki to 1.23wmf13 and build l10n cache
[2014-02-06 16:04:31,056] Checking syntax
Copying to tin from tin.eqiad.wmnet...ok
Done
Updating ExtensionMessages-1.23wmf12.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf12... done
Updating ExtensionMessages-1.23wmf11.php...
done
Copying to local copy...
done
Updating LocalisationCache for 1.23wmf11... done
Updating ExtensionMessages-1.23wmf13.php...PHP Warning: require(/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660

Warning: require(/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php' (include_path='/a/common/php-1.23wmf13/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf13:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf13/../wmf-config/ExtensionMessages-1.23wmf13.php' (include_path='/a/common/php-1.23wmf13/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf13:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2660
Update of MediaWiki localisation messages failed
Traceback (most recent call last):

File "/usr/local/bin/scap", line 220, in <module>
  scap()
File "/usr/local/bin/scap", line 177, in scap
  subprocess.check_call('/usr/local/bin/mw-update-l10n')
File "/usr/lib/python2.7/subprocess.py", line 511, in check_call
  raise CalledProcessError(retcode, cmd)

subprocess.CalledProcessError: Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1

Updating ExtensionMessages-1.23wmf14.php...PHP Warning: require(/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673

Warning: require(/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php): failed to open stream: No such file or directory in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673
PHP Fatal error: require(): Failed opening required '/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php' (include_path='/a/common/php-1.23wmf14/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf14:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673

Fatal error: require(): Failed opening required '/a/common/php-1.23wmf14/../wmf-config/ExtensionMessages-1.23wmf14.php' (include_path='/a/common/php-1.23wmf14/extensions/TimedMediaHandler/handlers/OggHandler/PEAR/File_Ogg:/a/common/php-1.23wmf14:/usr/local/lib/php:/usr/share/php') in /usr/local/apache/common-local/wmf-config/CommonSettings.php on line 2673
Update of MediaWiki localisation messages failed
18:09:25 ERROR - scap failed: CalledProcessError Command '/usr/local/bin/mw-update-l10n' returned non-zero exit status 1 (duration: 03m 04s)

Change 113260 had a related patch set uploaded by BryanDavis:
Touch ExtensionMessages-VERSION if not present

https://gerrit.wikimedia.org/r/113260

Change 113260 merged by Ori.livneh:
Add bootstrap initialization for l10n update in new branches

https://gerrit.wikimedia.org/r/113260

Tested fix on tin by renaming ExtensionMessages-1.23wmf14.php to ExtensionMessages-1.23wmf14.php-bak and running mw-update-l10n. ExtensionMessages-1.23wmf14.php was recreated with no diff from backup (and no crash).

The patch seemed to work for the wmf15 rollout, but due to other issues we should probably wait to see what happens with wmf16 before we mark this as resolved.

During wmf16 deploy we noticed missing en translations on www.mediawiki.org.

Running a full scap for just wmf16 branch (scap --versions php-1.23wmf16 'rebuild php-1.23wmf16 l10n cache') failed with:

Warning: opendir(/upstream): failed to open dir: No such file or directory in /srv/scap/bin/mergeCdbFileUpdates on line 76
Could not open directory '/upstream'.

This looks to be a bug in the scap-rebuild-cdbs call when versions are specified manually.

Rerunning as a full scap (scap 'full scap; rebuild php-1.23wmf16 l10n cache') performed the scap-rebuild-cdbs step correctly. It also seemed to fix the l10n cache on mw.o.

Change 117154 had a related patch set uploaded by BryanDavis:
Delete stub l10n file after generating ExtensionMessages

https://gerrit.wikimedia.org/r/117154

Change 117154 merged by jenkins-bot:
Delete stub l10n file after generating ExtensionMessages

https://gerrit.wikimedia.org/r/117154

Change 117236 had a related patch set uploaded by BryanDavis:
Remove l10n file as user l10nupdate

https://gerrit.wikimedia.org/r/117236

Change 117236 merged by jenkins-bot:
Remove l10n file as user l10nupdate

https://gerrit.wikimedia.org/r/117236

It worked!

Or not:

14:03] < bd808> greg-g: Nemo_bis found an l10n bug in wmf17 -- https://www.mediawiki.org/wiki/Special:Preferences#mw-prefsection-betafeatures
[14:03] < huh> [MatmaRex]
[14:04] < MatmaRex> [21:58] <huh> Did I add i18n for sp-contributions-newonly correctly? :/
[14:04] < MatmaRex> [21:59] <huh> It looks ok here http://deployment.wikimedia.beta.wmflabs.org/wiki/Special:Contributions/108.32.45.153
[14:04] < MatmaRex> [21:59] <huh> but broken here https://www.mediawiki.org/wiki/Special:Contributions/PiRSquared17

It just dawned on me that rebuildLocalisationCache.php has a --force option. This might be the answer rather than the attempts that I've made to clean up after the stub l10n generation.

Change 117821 had a related patch set uploaded by BryanDavis:
Force l10n rebuild after bootstrapping

https://gerrit.wikimedia.org/r/117821

Change 117821 merged by jenkins-bot:
Force l10n rebuild after bootstrapping

https://gerrit.wikimedia.org/r/117821

I'm going to declare victory on this now. I added the --force flag and the only l10n issues that have been reported against the 1.23wmf18 branch were due to an unrelated partial scap failure.

(In reply to Bryan Davis from comment #26)

I'm going to declare victory on this now.

Or not: https://wikitech.wikimedia.org/wiki/Incident_documentation/20140403-Deploy#Bryan