Page MenuHomePhabricator

Local versions of system messages are ignored
Closed, ResolvedPublic

Description

Yesterday a user reported on the Italian Wikipedia that several Mediawiki messages which have a local version there are being ignored (I guess the system is showing the original message available at Translatewiki, but can't check since that site is currently down). This doesn't seem to be targeting one product only.

Examples:
The button to launch the new TemplateData editor says "Gestisci documentazione template" (you can check at the top left of https://it.wikipedia.org/w/index.php?title=Template:Cassetto/man&action=edit ). It should read "Gestisci TemplateData" instead, as per https://it.wikipedia.org/wiki/MediaWiki:Templatedata-editbutton .

Or see Table 1 at https://it.wikipedia.org/wiki/Discussioni_utente:Rotpunkt#Messaggi_che_non_funzionano . The link says "Comprimi"/"Espandi", but the correct messages, linked a few lines above the table, should actually be "Nascondi"/"Mostra".


Version: unspecified
Severity: major
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=68781

Details

Reference
bz67805

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:34 AM
bzimport set Reference to bz67805.
bzimport added a subscriber: Unknown Object (MLST).

The original can be checked in the i18n directory of the source repo, e.g. https://git.wikimedia.org/tree/mediawiki%2Fcore.git/HEAD/languages%2Fi18n
(Or in Special:AllMessages *if* it works.)

Important note, Rotpunkt said this started happening between the 8th and the 9th, dates when no new MediaWiki version was deployed on it.wiki AFAICS.

When I open this link in a private browsing window:

https://it.wikipedia.org/w/index.php?title=Template:Cassetto/man&action=edit&uselang=it

I see the default, non-customized button label for TemplateData, but I see the local, custom message for https://it.wikipedia.org/wiki/MediaWiki:Anoneditwarning

Escalating as this is now active on Ukranian Wikipedia too – https://uk.wikipedia.org/wiki/MediaWiki:Visualeditor-ca-editsource is correct but instead the Russian version, "Править викитекст" is showing in production.

Yes, this is pretty bad.

Amir/Niklas: Can you take a look into this ASAP?

  • Bug 67911 has been marked as a duplicate of this bug. ***

Doesn't fit the definition of "critical", AFAICS.

Did someone try rebuilding the l10n cache? Were there changed to the LocalisationUpdate or deploy scripts in the last few days? Did someone inspect their logs?

I have made multiple observations:

  1. Typing unicode to mwscript eval.php does not work in WMF
  2. This particular message has existed in uk.json since April
  3. Other messages in the same file seem to work on random testing
  4. MessageCache thinks the message does not exist:

mwscript maintenance/eval.php --wiki=ukwiki

var_dump( MessageCache::singleton()->getMsgFromNamespace( 'Visualeditor-ca-editsource', 'uk' ) );

bool(false)

Because of 1) I was not able to test whether calling MessageCache::replace would have fixed that.

  1. MessageCache can find the message if instructed manually to load messages directly from the database:

mwscript maintenance/eval.php --wiki=ukwiki
var_dump( MessageCache::singleton()->loadFromDB( 'uk' ) );

..
["Visualeditor-ca-editsource"]=>
string(28) " Редагувати код"
..
  1. Localisation cache returns the ru version:

var_dump( $wgLang->getLocalisationCache()->getSubItem( 'uk', 'messages', 'visualeditor-ca-editsource' ) );

string(33) "Править викитекст"

  1. I cannot reproduce this issue locally
  1. LC issue is possibly caused by incorrect fallback merging, because L10nupdate contains update for ru, but not for uk (because the translation is already up to date for uk):

nikerabbit@tin:/var/lib/l10nupdate/cache-1.24wmf12$ grep visualeditor-ca-editsource *-ru.* *-uk.*
Binary file l10n_cache-ru.cdb matches
l10nupdate-ru.json: "visualeditor-ca-editsource": "\u041f\u0440\u0430\u0432\u0438\u0442\u044c \u0432\u0438\u043a\u0438\u0442\u0435\u043a\u0441\u0442",
l10nupdate-ru.json: "visualeditor-ca-editsource-section": "\u043f\u0440\u0430\u0432\u0438\u0442\u044c \u0432\u0438\u043a\u0438\u0442\u0435\u043a\u0441\u0442",
Binary file l10n_cache-uk.cdb matches

  1. Something fishy with message cache:

mwscript maintenance/eval.php --wiki=ukwiki

var_dump( MessageCache::singleton() );

object(MessageCache)#108 (8) {

...
["mDisable":protected]=>
bool(false)
..

}

mwscript maintenance/eval.php --wiki=ukwiki

var_dump( MessageCache::singleton()->getAllMessageKeys( 'uk' ) );

NULL

var_dump( MessageCache::singleton()->load( 'uk' ) );

bool(true)

var_dump( MessageCache::singleton() );

object(MessageCache)#108 (8) {

["mCache":protected]=>
bool(false)
["mDisable":protected]=>
bool(true)
..
["mLoadedLanguages":protected]=>
array(0) {
}
..

}

My conclusions so far are that there seems to be at least two separate issues here:
I) MessageCache is failing to load its cache
II) Localisation cache returns ru when uk exists
III) (bonus) Unicode does not work on eval.php at WMF

For I) I would like other people to inspect why that is happening.
For II) My observation in 8 is a good starting point for my investigations for someone else while I continue my vacation. This might be specific to the way WMF runs l10n update.
For III) It would be nice for someone to file a separate bug.

(In reply to Niklas Laxström from comment #9)

III) (bonus) Unicode does not work on eval.php at WMF
For III) It would be nice for someone to file a separate bug.

Done, see bug 67924

string(28) " Редагувати код"

or "Редагувати код"?
Maybe first space in this message in database causes error?

(In reply to Володимир Ф from comment #11)

Maybe first space in this message in database causes error?

That is just how MessageCache::loadFromDB presents them internally. There is no space in front of that message in the database.

In reply to Niklas Laxström from comment #12)
Understood. Thanks.

In case this wasn't clear enough, this is affecting several wikis, including large ones like the French or the German WP.

Please not that this bug is critically important for Ukrainian: the message affected (VE Edit tab) is a prominent one, and the fallback displayed (Russian despite both local and global translations being correct) is very badly received by the community (to the point that some people suggest disabling VE).

Re-adjusting importance fields. I'd say this constitutes a "major loss of functionality", and definitely should not take "one to six months" to fix.

Are there any examples where the incorrect message is being displayed in the served HTML, rather than being served via ResourceLoader for use in JavaScript? I'm suspecting not, since I can open https://it.wikipedia.org/wiki/Discussioni_utente:Rotpunkt#Messaggi_che_non_funzionano and add "{{int:Collapsible-expand}} {{int:Collapsible-collapse}}" and see the correct messages despite the JS-added link in the table being wrong.

That makes me suspect either RL's message cache isn't being cleared properly, or else something in it is broken such that it's not picking up the customized messages.

As a test, I removed the database row for ResourceLoader's message cache for jquery.makeCollapsible for Italian on itwiki. And upon reload, it started serving the correct messages for that expand/collapse example.

Which makes me suspect even more that the RL cache clearing is broken.

Yeah, when we ran into this with a VE message a few days ago, I checked this. wfMessage( $msg )->plain() returned the correct version, but mw.msg( msg ) did not.

Certainly RL cache clearing: after the fix above, I reset the timestamp on the new row and re-ran the RL cache clearer (refreshMessageBlobs.php in WikimediaMaintenance) on itwiki. And it's re-broken.

More information: Something is screwy on tin, which is where the cache flush is run.

This command:

wfMessage( 'collapsible-collapse' )->inLanguage( 'it' )->plain()

returns "nascondi" on terbium (and presumably other servers) but "Comprimi" on tin.

Ok, I think this is it. Turning on logging gives this from tin:

> var_dump( MessageCache::singleton()->load( 'it' ) );
[memcached] get(itwiki:messages:it:hash)
[memcached-serious] Memcached error for key "itwiki:messages:it:hash" on server "127.0.0.1:11212": CONNECTION FAILURE
[memcached] get(itwiki:messages:it)
[memcached-serious] Memcached error for key "itwiki:messages:it" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] add(itwiki:messages:it:status)
[memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] get(itwiki:messages:it:status)
[memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] add(itwiki:messages:it:lock)
[memcached-serious] Memcached error for key "itwiki:messages:it:lock" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] set(itwiki:test)
[memcached-serious] Memcached error for key "itwiki:test" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] delete(itwiki:messages:it:lock)
[memcached-serious] Memcached error for key "itwiki:messages:it:lock" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] get(itwiki:messages:it)
[memcached-serious] Memcached error for key "itwiki:messages:it" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
[memcached] add(itwiki:messages:it:status)
[memcached-serious] Memcached error for key "itwiki:messages:it:status" on server "127.0.0.1:11212": SERVER HAS FAILED AND IS DISABLED UNTIL TIMED RETRY
MessageCache::load: Loading it... global cache is empty, waited for other thread to complete, global cache is empty, could not acquire status key., loading FAILED - cache is disabled
bool(false)

I think at this point we need to get an ops person involved.

[01:09:35] <Reedy> I suspect it just hasn't had nutcracker updated on it
[01:09:40] <Reedy> tin
[01:09:41] <Reedy> nobody 23483 0.0 0.0 18732 1704 ? Sl Apr25 8:59 /usr/local/bin/nutcracker -m 65536 -a 127.0.0.1 -c /usr/local/apache/common-local/wmf-config/twemproxy-eqiad.yaml
[01:09:46] <Reedy> mw1017
[01:09:47] <Reedy> 112 14857 0.1 0.0 21652 2144 ? Ssl Jul07 13:11 /usr/sbin/nutcracker --mbuf-size=65536 --stats-port=22223
[01:10:04] -*- jgage takes a look
[01:10:10] <Reedy> Yeah
[01:10:15] <Reedy> tin still is on "twemproxy"
[01:10:20] <Reedy> apaches are on "nutcracker"

<jgage> andre__: nutcracker is now updated on tin

reedy@tin:~$ mwscript eval.php itwiki

var_dump( MessageCache::singleton()->load( 'it' ) );

bool(true)

Not sure if I should've actually closed it...

Do the ResourceLoader caches need to be flushed to take care of any lingering issues? The entries in msg_resource with mr_timestamp less than 20140715002850 (based on comment 24) and mr_blob != '{}' are potentially problematic.

At a quick look, this might apply to the following languages on wikis still on 1.24wmf12: ak af aln am ang anp arc ary as ast avk bbc bcl bg bi bo br brh ch ceb chr chy cr cps crh crh-latn cu dv dz ee en en-ca en-gb eu ext fj fo fy ga gom-latn got grc gu gv ha haw hi hif hif-latn hil hy ie ig ik ike-cans ike-latn ilo io iu jam jbo ka kab kbd kbd-cyrl kg khw ki km kri krj ks ks-arab ks-deva ku ku-latn kw ky la lfn lg li lo loz lrc lt lus lzh mai mdf mi mn mr my na ne new niu nl nl-informal nov nrm nso nv ny oc om or pag pam pap pi pih pnb pnt prg ps rif rm roa-tara rup rw sa sat sc sco sd sdc se sei sgs sh shi si sm sma sn so sq srn ss st sw te tet tg tg-cyrl tg-latn ti tk tl tly tn to tokipona tpi tru ts tt-latn tw ug ug-arab ug-latn ur uz ve vls vo war xh xmf yo yue zea zu. Other languages (except maybe uselang hacks) should have been updated already by the l10nupdate run at 02:00 UTC this morning.

It looks like all languages on 1.24wmf13 wikis should have been updated by the l10nupdate run at 02:00 UTC this morning. This afternoon's update of non-Wikipedias to wmf13 should fix them too, BTW.

Does the nutcracker issue completely explain findings I-II in comment 9? If there are further underlying causes or possible improvements, separate bug reports would be very useful once this one about the specific incident is closed.

(In reply to Nemo from comment #29)

Does the nutcracker issue completely explain findings I-II in comment 9? If
there are further underlying causes or possible improvements, separate bug
reports would be very useful once this one about the specific incident is
closed.

Finding I, yes. Probably not finding II.

I tracked that one as far as observing that the json file for uk that the cdb file gets built from has "Править вики-текст" as the value for messages:visualeditor-ca-editsource rather than "Редагувати код". That seems to be a separate issue from this one, and so should have its own bug report if it doesn't already.

Hello. On it.wiki we had changed the shortcut for the button "Show preview" from alt+shift+v to alt+shift+- (see it:MediaWiki:Accesskey-diff). That shortcut doesn't work for a few days (or weeks, dunno). Since I thought that problem had been solved (comment #27), it may be related to this bug or not?

(In reply to Lorenzo Marconi from comment #31)

Hello. On it.wiki we had changed the shortcut for the button "Show preview"

This is separate issue and the cause is that "-" means that the message is disabled and thus "-" cannot be used as an accesskey.

(In reply to Niklas Laxström from comment #32)

(In reply to Lorenzo Marconi from comment #31)

Hello. On it.wiki we had changed the shortcut for the button "Show preview"

This is separate issue and the cause is that "-" means that the message is
disabled and thus "-" cannot be used as an accesskey.

Excuse me for the OT, but why it doesn't work only since a few days?

Lorenzo: Please create a separate ticket. This is offtopic.

NickK: Is this still a problem on Ukrainian Wikimedia sites?

(In reply to Andre Klapper from comment #35)

NickK: Is this still a problem on Ukrainian Wikimedia sites?

And is "II) Localisation cache returns ru when uk exists" still a problem?

(In reply to Andre Klapper from comment #36)

(In reply to Andre Klapper from comment #35)

NickK: Is this still a problem on Ukrainian Wikimedia sites?

And is "II) Localisation cache returns ru when uk exists" still a problem?

No one has identified a fix, so I would assume yes (even if the symptoms are not exposed right now).

Filed as bug 68781.

(In reply to Matthew Flaschen from comment #37)

And is "II) Localisation cache returns ru when uk exists" still a problem?

No one has identified a fix, so I would assume yes (even if the symptoms are
not exposed right now).

Filed as bug 68781.

Thanks. CLosing this ticket about I) as FIXED as II) is handled there now.