| siebrand | |
| Apr 2 2014, 1:46 PM |
| F41990: message-usage-counts.log | |
| Feb 17 2015, 2:32 AM |
The file wikimedia-mostused-2011.txt should be updated, so that the translation group core-0-mostused at translatewiki.net can be updated.
Blog post: http://laxstrom.name/blag/2015/02/19/prioritizing-mediawikis-translation-strings/
Version: unspecified
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=63415
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Resolved | Nikerabbit | T65415 Support core-0-mostused with JSON file format | |||
| Resolved | Nikerabbit | T65416 Update list of most often used messages for MediaWiki core at Wikimedia |
I'm moving this out of Translate component. Perhaps we could use EventLogging together with the new hook in message cache and suitable sampling to as a way to collect data this time.
(In reply to Niklas Laxström from comment #1)
I'm moving this out of Translate component. Perhaps we could use
EventLogging together with the new hook in message cache and suitable
sampling to as a way to collect data this time.
For this high volume this sounds like complete overkill. For previous versions, this took about 30 minutes of Tim Starling's time to gather the data, IIRC.
Tim is difficult to reproduce on demand.
It's been three weeks since this bug was filed, and years since data was collected last time.
(In reply to Niklas Laxström from comment #3)
Tim is difficult to reproduce on demand.
It's been three weeks since this bug was filed, and years since data was
collected last time.
That's not because Tim is hard to get a hold of, it's because I'm bad at scheduling an update.
(In reply to Siebrand Mazeland from comment #4)
That's not because Tim is hard to get a hold of, it's because I'm bad at
scheduling an update.
Typically it happens at the hackathon, so we have hopes. :)
Copying here a comment I made some time ago on https://meta.wikimedia.org/wiki/Talk:Language_proposal_policy: «the WMF could give assistance in updating the "500 most used" core messages, and adding a second "1500 most used" (or whatever) core+extensions messages (measured in a way that ensures the most used messages on e.g. Wikisource only are all included too)». Whatever is measured in the end (even just core), it would be nice to ensure it represents all Wikimedia projects (maybe Tim's method already does but we don't know).
(In reply to Siebrand Mazeland from comment #2)
For previous
versions, this took about 30 minutes of Tim Starling's time to gather the
data, IIRC.
I was wondering, was that a live hack or is there a commit on some wmf branch in SVN? The code may also be available to shell users on Tim's home directory on fenari or whatever; it takes a couple minutes to look it up.
(In reply to Nemo from comment #6)
(In reply to Siebrand Mazeland from comment #2)
For previous
versions, this took about 30 minutes of Tim Starling's time to gather the
data, IIRC.I was wondering, was that a live hack or is there a commit on some wmf
branch in SVN? The code may also be available to shell users on Tim's home
directory on fenari or whatever; it takes a couple minutes to look it up.
There's a patch set somewhere in Subversion, I think (or maybe in Gerrit already). I say that, because I think I remember he reverted that to stop profiling.
r88126, but nothing in https://www.mediawiki.org/w/index.php?title=Special:Code/MediaWiki&path=%2Fbranches%2Fwmf&author=tstarling or anything else around the time.
Over the year multiple people has been begging me for to get a refreshed list enabled at translatewiki.net. Let's get this bug fixed for real. That means devising a reproducible solution and not hunting old hacks or waiting for someone to ping someone. I will try to take advantage of the developer summit to get this fixed.
Talked with Ori briefly about this. He said using tcpdump in one of the memcached hosts could be a solution (though unless scripted that doesn't seem to be easy to update the list). Hopefully will get more details later.
Tim told me that he just hacked some logging for an hour. Ori said he would teach me how to gather the data when I have time.
Change 190777 had a related patch set uploaded (by Nikerabbit):
Temporarily log message key lookups on four app servers
My recollection of the conversation here was that any list of the most used messages would invariably include messages used in the standard user interface (sidebar messages, footer messages, etc.), which really brought into question the value of such a report. The concern is that a high level of noise would drown out any signal. I don't really care if we do a sampled collection of hits, I just personally doubt there will be much utility in the results. That said, I'm pretty detached from the interface translation process and the people closer to it seem to really think that a refreshed report will help.
The list will be post processed to filter out noise as well as we can. I don't understand your point about "would invariably include messages used in the standard user interface". That's exactly what we want: the most important messages (using requests/visibility as a proxy). The core itself has 3000 messages, many of them obscure things like exif info and rare error messages or disabled features. Not to speak about crucial extensions which also contain important messages. Finding those in midst of 24k messages is what we are here doing, to help the translators to prioritize their time and work.
Thank you for clarifying. As someone who's not as closely involved in the translation process and who was largely relying on the task description here, this wasn't obvious to me. :-)
Right... I think everyone is of the view that ideally all of the messages should be translated in every available language.
Ordering by most used feels a bit artificial and strange to me, but @Nemo_bis pointed out that https://meta.wikimedia.org/wiki/Language_proposal_policy does (in my opinion, bizarrely) reference this list:
As a baseline, it is recommended that you begin by translating the "most used MediaWiki messages".
And this is apparently a policy created/followed by the Language committee (LangCom). It's even marked as a global policy... wild.
In any case, if we're going continue to update the most used messages list regularly, I agree with you that we should find a way to make the process more automated.
Change 190777 merged by jenkins-bot:
Temporarily log message key lookups on four app servers
I was tracking this in real-time and after a few minutes the frontrunners did not change, so I ran this for half an hour total.
During this time, there were 15,324,011 message key lookups in total. I have attached the full list. The top 100 are:
| Message key | Hits |
|---|---|
| word-separator | 643974 |
| interlanguage-link-title | 552468 |
| parentheses | 512980 |
| brackets | 468465 |
| mainpage | 365891 |
| scribunto-doc-page-name | 352744 |
| editsectionhint | 314825 |
| editsection | 250348 |
| pipe-separator | 170227 |
| cite_reference_link_suffix | 161050 |
| cite_reference_link_prefix | 161049 |
| cite_references_link_suffix | 143605 |
| cite_references_link_prefix | 143605 |
| red-link-title | 139745 |
| comma-separator | 125058 |
| talkpagelinktext | 111075 |
| aboutsite | 108211 |
| rc-change-size-new | 103157 |
| pagetitle | 101121 |
| february | 94908 |
| disclaimers | 93210 |
| conversion-ns-1 | 88336 |
| privacy | 86732 |
| conversion-ns6 | 85100 |
| conversion-ns14 | 81630 |
| conversion-ns3 | 81542 |
| conversion-ns10 | 81302 |
| conversion-ns2 | 81288 |
| conversion-ns1 | 81276 |
| conversion-ns15 | 81248 |
| conversion-ns7 | 81242 |
| conversion-ns8 | 81238 |
| conversion-ns11 | 81236 |
| conversion-ns829 | 81234 |
| conversion-ns828 | 81234 |
| conversion-ns13 | 81234 |
| conversion-ns12 | 81234 |
| conversion-ns9 | 81233 |
| cite_reference_link | 80971 |
| conversion-ns100 | 80934 |
| conversion-ns101 | 80922 |
| conversion-ns-2 | 79108 |
| contribslink | 72930 |
| cite_reference_link_key_with_num | 70317 |
| mobile-frontend-editor-edit | 66118 |
| printableversion | 58057 |
| feb | 57425 |
| cite_references_link_one | 55100 |
| january | 53215 |
| aboutpage | 50192 |
| search | 49969 |
| privacypage | 49852 |
| disclaimerpage | 49844 |
| cirrussearch-boost-templates | 49372 |
| december | 49309 |
| wikilove-err-not-logged-in | 48929 |
| july | 47898 |
| november | 47888 |
| october | 47664 |
| september | 47353 |
| march | 47290 |
| wikibase-repo-name | 47256 |
| august | 47226 |
| june | 46604 |
| april | 46554 |
| may_long | 46498 |
| anonnotice | 46478 |
| jan | 46396 |
| dec | 45668 |
| nov | 45255 |
| oct | 45165 |
| mar | 44989 |
| aug | 44813 |
| sep | 44712 |
| jun | 44657 |
| apr | 44531 |
| jul | 44449 |
| may | 44443 |
| searchbutton | 43511 |
| opensearch-desc | 43383 |
| wikimedia-developers-url | 43366 |
| wikimedia-developers | 43366 |
| retrievedfrom | 43366 |
| ffeed-enable-sidebar-links | 43366 |
| mobile-frontend-view | 43364 |
| site-atom-feed | 42878 |
| conversion-ns4 | 41778 |
| conversion-ns5 | 41757 |
| interlanguage-link-title-langonly | 41479 |
| tooltip-search-fulltext | 40841 |
| tooltip-search | 40841 |
| searchsuggest-search | 40841 |
| accesskey-search-fulltext | 40837 |
| accesskey-search | 40837 |
| pt-login | 40064 |
| pt-createaccount | 40064 |
| sitenotice | 37918 |
| colon-separator | 37430 |
| contact-url | 37234 |
| size-kilobytes | 36492 |
Change 191072 had a related patch set uploaded (by Nemo bis):
Add 2015 list of most used MediaWiki messages
When I look at https://translatewiki.net/w/i.php?title=Translating:MediaWiki/Most_used_messages_table,_2015&oldid=6022702 , it's not without significance that the first message standing out is mobile-frontend-editor-edit, 12th in the rank but 1 order of magnitude fewer translations than its neighbours. Thanks to this list, we're going to fix it!
Thank to everyone involved, especially Ori and Nemo_bis for their time so that I myself was able to focus actually setting this message group up – it's now deployed to production.
Not sure how index got into the list; the only index message translatewiki.net knows about is a message from an obscure/unmantained IndexFunction extension.
I guess this is thanks to ru.wikipedia (and a few other wikis) using MediaWiki:Index to include “A–Z index” link in the sidebar?
That's plausible, let's remove "index". It's fine to keep improving the list with further patches, we made it a bit longer on purpose.
cleanup-most-used.php should perhaps be amended to check this stuff as well.
Jhs, see https://gerrit.wikimedia.org/r/192288 for metadata-fields which you reported (I can't find you in gerrit).
How? List the primary group it belongs to?
I've been thinking more about it but no better idea so far. What could be used to compare contents as LocalisationUpdate would do? Perhaps get a cdb l10n file from production?
Change 200822 had a related patch set uploaded (by Mormegil):
Remove 'index' from most-used list