duplicate/invalid language codes
Open, MediumPublic
Actions

Assigned To

None

Authored By

	Merl
	Nov 23 2012, 7:55 PM

Description

Archived discussion at http://www.wikidata.org/wiki/Wikidata:Contact_the_development_team/Archive/2012/11#language_code_problems

als, be-x-old and zh-yue are not a valid language code that should be used for labels, aliases and descriptions. Instead gsw, be-tarask and yue are the valid lang codes.

You you also check the lang setting of these wikis: http://als.wikipedia.org/w/api.php?action=query&meta=siteinfo&siprop=general

Version: wmf-deployment
URL: https://meta.wikimedia.org/wiki/Special_language_codes
See Also:
T53071: Utils::getLanguageCodes() should filter [[mw:Manual:$wgDummyLanguageCodes]] out
T39459: [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag
T39754: Handling for missing language codes
T21986: [DO NOT USE] Wikis waiting to be renamed (tracking) [superseded by #Wiki-Setup (Rename)]
T101086: Standardize invalid language codes for Babel extension

Details

Reference: bz42396

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T124286 [Epic] Wikidata language support
Open		None	T145535 language code disparity (Wikidata tracking)
Open	BUG REPORT	None	T279874 SVG language tag als reported as Swiss German; gsw is Swiss German.
Open		None	T44396 duplicate/invalid language codes
Open	Feature	None	T284808 Add a configuration variable that allows disabling language codes for labels, descriptions, and aliases
Open	BUG REPORT	None	T257415 Language code zh-classical is invalid
Open		None	T51024 [Task] Removal of de-formal from allowed language for labels

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 12:55 AM

• bzimport added projects: Wikidata.org, I18n.

• bzimport set Reference to bz42396.

• bzimport added a subscriber: Unknown Object (MLST).

Merl created this task.Nov 23 2012, 7:55 PM

This bug is probably too general to be useful (perhaps transform into a tracking bug?), but as we have another equally general report let me copy it here:

Small update: I went through the language list at

https://github.com/mkroetzsch/wda/blob/master/includes/epTurtleFileWriter.py#L472

and added a number of TODOs to the most obvious problematic cases. Typical problems are:

Malformed language codes ('tokipona')
Correctly formed language codes without any official meaning (e.g., 'cbk-zam')
Correctly formed codes with the wrong meaning (e.g., 'sr-ec': Serbian from Ecuador?!)
Language codes with redundant information (e.g., 'kk-cyrl' should be the same as 'kk' according to IANA, but we have both)
Use of macrolanguages instead of languages (e.g., "zh" is not "Mandarin" but just "Chinese"; I guess we mean Mandarin; less sure about Kurdish ...)
Language codes with incomplete information (e.g., "sr" should be "sr-Cyrl" or "sr-Latn", both of which already exist; same for "zh" and "zh-Hans"/"zh-Hant", but also for "zh-HK" [is this simplified or traditional?]).

Small update: I went through the language list at

https://github.com/mkroetzsch/wda/blob/master/includes/epTurtleFileWriter.py#L472

and added a number of TODOs to the most obvious problematic cases. Typical problems are:

Malformed language codes ('tokipona')
Correctly formed language codes without any official meaning (e.g., 'cbk-zam')
Correctly formed codes with the wrong meaning (e.g., 'sr-ec': Serbian from Ecuador?!)
Language codes with redundant information (e.g., 'kk-cyrl' should be the same as 'kk' according to IANA, but we have both)
Use of macrolanguages instead of languages (e.g., "zh" is not "Mandarin" but just "Chinese"; I guess we mean Mandarin; less sure about Kurdish ...)
Language codes with incomplete information (e.g., "sr" should be "sr-Cyrl" or "sr-Latn", both of which already exist; same for "zh" and "zh-Hans"/"zh-Hant", but also for "zh-HK" [is this simplified or traditional?]).

Copy and paste fail... URL http://thread.gmane.org/gmane.org.wikimedia.wikidata/2524

It seems fixed to me. I just made this edit with uselang=be-x-old: https://www.wikidata.org/w/index.php?title=Q1&diff=112330190&oldid=112313552

Tacsipacsi mentioned this in T21986: [DO NOT USE] Wikis waiting to be renamed (tracking) [superseded by #Wiki-Setup (Rename)].Nov 30 2014, 3:57 PM

matej_suchanek mentioned this in T39459: [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag.Jun 15 2015, 12:40 PM

Smalyshev mentioned this in T105430: [Task] Ensure that language tags generated in RDF output are standard language names .Jul 17 2015, 8:19 AM

matej_suchanek updated the task description. (Show Details)Jul 20 2015, 2:36 PM

matej_suchanek set Security to None.

Restricted Application added a project: Wikidata. · View Herald TranscriptJul 20 2015, 2:36 PM

Reopened. It is not fixed. It is still possible to add unwanted values via API:

Description with language code be-x-old: https://www.wikidata.org/w/index.php?title=Q4115189&diff=235922241
Description with language code be-tarask: https://www.wikidata.org/w/index.php?title=Q4115189&diff=235922524

In T44396#444530, @Lydia_Pintscher wrote:

It seems fixed to me. I just made this edit with uselang=be-x-old: https://www.wikidata.org/w/index.php?title=Q1&diff=112330190&oldid=112313552

Here you got the correct language code be-tarask because MediaWiki core converts the URL parameter uselang=be-x-old to the user interface language be-tarask.

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptJul 21 2015, 1:11 PM

Isn't that T39459: [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag?

Liuxinyu970226 subscribed.Jul 22 2015, 11:44 AM

Liuxinyu970226 updated the task description. (Show Details)Aug 3 2015, 11:43 PM

Koavf subscribed.Aug 25 2015, 6:36 AM

In T44396#1467435, @Nemo_bis wrote:

Isn't that T39459: [Task] Don't try to add labels in non-existing languages: restrict to Language::isKnownLanguageTag?

No. T39459 request to restrict the user interface to known languages. The database is already restricted to known languages. Except for duplicate language codes like als/gsw, be-x-old/be-tarask, ...

This task requests to restrict the database even with API requests to disallow unwanted language codes that are defined in wgDummyLanguageCodes.

T39459 request to restrict the user interface to known languages

No, it doesn't.

It is not possible to add a label/description/alias with language code xyzzy. Neither via GUI nor via API. When you try to do it, you get an error message.

When you open the GUI with uselang=xyzzy you get a UI which gives you input elements for label/description/alias in language xyzzy. You showed me exactly this in T39459#1468881.

daniel added a parent task: T78006: [Story] Determine list of available languages in a uniform way.Oct 8 2015, 11:00 AM

daniel added a subtask: T102533: [Bug] Disallow (or resolve) dummy language codes..Oct 8 2015, 11:08 AM

adrianheine mentioned this in T53071: Utils::getLanguageCodes() should filter [[mw:Manual:$wgDummyLanguageCodes]] out.Jan 28 2016, 2:36 PM

Amire80 added a subtask: T132899: Remove tokipona support.Apr 18 2016, 9:10 AM

adrianheine mentioned this in T137065: Language "norsk" ("no") added in the signature section, but only "nynorsk" ("nn") and "bokmål" ("nb") should be used.Jun 7 2016, 10:09 AM

Liuxinyu970226 mentioned this in T145535: language code disparity (Wikidata tracking).Sep 23 2016, 10:06 AM

Can we close this and just make new tasks for anything that is still outstanding?

In T44396#2777658, @Esc3300 wrote:

Can we close this and just make new tasks for anything that is still outstanding?

You can not close this task as resolved, because it is not solved. But you can merge this task with a similar task for example with T102533: [Bug] Disallow (or resolve) dummy language codes..

For solving this task several subtasks are necessary. The first task should be disallowing adding new entries with deprecated language codes.

! In T44396#2787104, @Fomafix wrote:

You can not close this task as resolved, because it is not solved. But you can merge this task with a similar task for example with T102533: [Bug] Disallow (or resolve) dummy language codes..

For solving this task several subtasks are necessary. The first task should be disallowing adding new entries with deprecated language codes.

Good point! But

You can not create a relationship to object "PHID-TASK-hubexo6f7fq5spgvmjqd" because objects can not be related to themselves.

...

Liuxinyu970226 closed this task as a duplicate of T145535: language code disparity (Wikidata tracking).Nov 11 2016, 12:51 AM

Restricted Application removed a subscriber: Liuxinyu970226. · View Herald TranscriptNov 11 2016, 12:51 AM

Liuxinyu970226 removed a subtask: T102533: [Bug] Disallow (or resolve) dummy language codes..Nov 11 2016, 12:57 AM

Liuxinyu970226 removed a subtask: T132899: Remove tokipona support.

Nemo_bis reopened this task as Open.Nov 11 2016, 1:01 PM

Nemo_bis merged a task: T145535: language code disparity (Wikidata tracking).

Nemo_bis added subscribers: Krenair, zhuyifei1999, Liuxinyu970226, Amire80.

NOTE: This is tracked at: T122677

From bzimport added a subscriber: Unknown Object (MLST).? Probably wikibugs-l, IIRC they removed this functionality from Phabricator

Liuxinyu970226 added a parent task: T145535: language code disparity (Wikidata tracking).Dec 23 2016, 4:31 AM

Quest_for_Truth subscribed.Jan 4 2017, 3:25 AM

Aklapper removed a project: Verified.Jan 14 2017, 2:22 AM

Restricted Application added a subscriber: PokestarFan. · View Herald TranscriptJul 25 2017, 3:29 AM

So there are currently more than 30,000 invalid terms in Wikidata, mostly in als, es-formal, no and simple. Doing cleanup again and again is pointless.

Language codes in question:

als
bat-smg
bh
de-formal
es-formal
fiu-vro
hu-formal
nl-informal
no
roa-rup
simple
zh-classical
zh-min-nan
zh-yue

They are all supported by MediaWiki but should be blacklisted in Wikibase.

Lucas_Werkmeister_WMDE mentioned this in T250114: STOP adding zh-yue captions on Commons. Migrate existing ones to yue..Apr 14 2020, 11:41 AM

Nikki mentioned this in T257415: Language code zh-classical is invalid.Jul 8 2020, 10:15 AM

jhsoby added a project: Language codes.Sep 16 2020, 10:02 AM

I have recently migrated all uses of "bat-smg", "bh", "fiu-vro", "roa-rup", "zh-classical, "zh-min-nan", and "zh-yue" on labels, descriptions, and aliases to "sgs", "bho", "vro", "rup", "lzh", "nan", and "yue" respectively, so now would be a great time to at least disallow those language codes.

I have been semi-regularly migrating occasional additions of labels/descriptions/aliases in the languages codes noted in my original comment since that comment, in addition to an "no" to "nb" migration with @jhsoby's approval. The sooner these codes can be disallowed, the less work this will be for everyone.

Is there a blacklist feature or would that need to be created?

@Mbch331 what do you think?

In T44396#7150835, @Esc3300 wrote:

Is there a blacklist feature or would that need to be created?

! In T44396#7150849, @Esc3300 wrote:

@Mbch331 what do you think?

That's more for WMDE. This goes deeper into the codebase.

The request is from 2012, but maybe it was designed since. If you haven't come across it, I suppose it doesn't exists and needs to be requested: see T284808 for the termbox.

Esc3300 added a subtask: T284808: Add a configuration variable that allows disabling language codes for labels, descriptions, and aliases.Jun 11 2021, 8:32 AM

In T44396#7150835, @Esc3300 wrote:

Is there a blacklist feature or would that need to be created?

The DifferenceContentLanguages thing. It is already in place for DefaultMonolingualTextLanguages, but not for DefaultTermsLanguages.
I think this all has been waiting for T66649 fixed.