Canonical URL should include language variant
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• RobLa-WMF
	Aug 1 2013, 11:14 PM

Description

Acceptance criteria (AC):

Example page: https://zh.wikipedia.org/wiki/汉语 ( shortened as /wiki/汉语 format below )

Language variant in URL	Action	`link rel="canonical"`	`link rel="alternate" hreflang="zh-Hant-TW"`	Notes
non-specified	view	`/wiki/汉语`	`/zh-tw/汉语`
zh-tw	view	`/zh-tw/汉语`	`/zh-tw/汉语`
non-specified	edit	`/wiki/汉语`	`/zh-tw/汉语`	See T67402
zh-tw	edit	`/zh-tw/汉语`	`/zh-tw/汉语`	for consistency
non-specified	history	`/w/index.php?title=汉语&action=history`	(without HTML meta alternate tag)	See Func's cmt.
zh-tw	history	`/w/index.php?title=汉语&action=history`	(without HTML meta alternate tag)	See Func's cmt.
non-specified	info	`/w/index.php?title=汉语&action=info`	(without HTML meta alternate tag)	See Func's cmt.
zh-tw	info	`/w/index.php?title=汉语&action=info`	(without HTML meta alternate tag)	See Func's cmt.

Func commented on https://gerrit.wikimedia.org/r/879583 :

I think we don't need to append the variant parameter to the history or info page,
variants only make sense when editing or viewing the page.

NOTE: Maybe we should prefer to use the BCP 47 code for language variant path?

NOTE: Maybe we would like to introduce new configuration option(s) to exclude some of the language variants?

Language variants currently point to the same canonical URL. For example, on this page:

http://zh.wikipedia.org/zh-tw/%E6%B1%89%E8%AF%AD

...there is a rel=”canonical” pointing to
http://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD

This rel=”canonical” link asks search engines to index the Simplified Chinese page to represent the content on both pages, instead of separately indexing the Simplified Chinese and Traditional Chinese pages. Similar rel=”canonical” links are found on all zh-TW pages. Google is reporting that we see a similar problem on other Chinese (e.g. zh-SG) and Serbian content pages.

(this may be caused by the fix to bug 48402 ( T50402: rel=canonical of https pages should point to http )

Version: 1.23.0
Severity: normal
See Also:

Details

Reference: bz52429

	Subject	Repo	Branch	Lines +/-
	OutputPage: Fix the behavior for canonical URL and alternate URLs	mediawiki/core	master	+125 -6

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Open		None	T99587 Make the Mobile site more discoverable by search engines
Open		None	T93550 Fix canonical URL issues (tracking)
Open	Feature	None	T16720 robots.txt (tracking)
Resolved		Jdlrobson	T305540 Google search traffic sending users to the wrong Chinese language versions of Wikipedia
Resolved		Jdlrobson	T108443 Google doesn't honor canonical URLs of zh.wiki
Resolved		Jdlrobson	T54429 Canonical URL should include language variant

Event Timeline

• bzimport raised the priority of this task from to High.Nov 22 2014, 1:45 AM

• bzimport added projects: MediaWiki-Language-converter, I18n.

• bzimport set Reference to bz52429.

• RobLa-WMF created this task.Aug 1 2013, 11:14 PM

If I understand the semantic meaning of rel="canonical" correctly, what it does now is the expected behavior.

http://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD is not "the Simplified Chinese
page", but an automatically converted page based on requests (prefs for users and Accept-Language for anons). We want all these links to show up in Google search results instead of links specifying a particular variant.

However Google seems not respecting it and indexing links to pages in every variant, and we have to workaround it: https://zh.wikipedia.org/w/index.php?title=MediaWiki:Gadget-variant-link-fix.js

Currently Google seems to mainly index /zh/ links instead of /wiki/'s (which is unexpected). /zh-tw/ or /zh-cn/'s are not indexed as expected though.

RobLa: So should this still be high priority wrt Liangent's comment 1 here?

If still high priority:
Tim: Do you plan to work on this at some point?

(In reply to Andre Klapper from comment #3)

RobLa: So should this still be high priority wrt Liangent's comment 1 here?

If still high priority:
Tim: Do you plan to work on this at some point?

I guess Tim is just the default CC, but actually this issue seems not Wikimedia-specific.

Change 154240 had a related patch set uploaded by Tim Starling:
Don't send rel=canonical to variant-neutral page

https://gerrit.wikimedia.org/r/154240

Change 154240 merged by jenkins-bot:
Don't send rel=canonical to variant-neutral page

https://gerrit.wikimedia.org/r/154240

All patches mentioned in this report were merged or abandoned - is there more work left to do here (if yes: please reset the bug report status to NEW or ASSIGNED), or can you close this ticket as RESOLVED FIXED?

(In reply to Rob Lanphier from comment #0)

Language variants currently point to the same canonical URL. For example, on
this page:

http://zh.wikipedia.org/zh-tw/%E6%B1%89%E8%AF%AD

...there is a rel=”canonical” pointing to
http://zh.wikipedia.org/wiki/%E6%B1%89%E8%AF%AD

Now has:

But I'm not sure this is properly fixed in general, because this is still an issue:

(In reply to fireattack from comment #2)

Currently Google seems to mainly index /zh/ links instead of /wiki/'s (which
is unexpected). /zh-tw/ or /zh-cn/'s are not indexed as expected though.

The two URLs for "zh" version don't agree on which is canonical:

/zh/ says

/wiki/ says

Created attachment 16795
Google search in Italian for [[zh:汉语]]

If I search a Latin alphabet string of that article I manage to get 4 variants from Google after asking to show me duplicate pages as well. None of them is /wiki/

Searching '"漢語，又称中文、华语" site:wikipedia.org' yielded two results including zh.wap.wikipedia.org/zh-tw/汉语 but that's another bug.

Attached:

Nemo_bis mentioned this in T3433: Add metainformation for interlanguage links (link rel="alternate" hreflang).Feb 9 2015, 3:35 PM

Aklapper mentioned this in T93550: Fix canonical URL issues (tracking).Mar 26 2015, 2:23 PM

Shizhao closed this task as a duplicate of T33838: zh-tw still gives simplified Chinese in link titles on Facebook and Google.May 12 2015, 1:19 PM

Liuxinyu970226 unsubscribed.May 12 2015, 1:36 PM

Not a duplicate.

Krinkle added a parent task: T93550: Fix canonical URL issues (tracking).Jun 18 2015, 12:43 AM

Nemo_bis added a project: SEO.Jun 23 2015, 8:49 AM

Krinkle renamed this task from Language variants currently point to the same canonical URL to Canonical URL should include language variant.Jul 7 2015, 10:34 AM

Krinkle set Security to None.

I don't know why the title of this page reads "Canonical URL should include language variant" (because it shouldn't). But anyway, I'm here to report the exact weired behavior mentioned above:

In T54429#541103, @Nemo_bis wrote:

Created attachment 16795
Google search in Italian for [[zh:汉语]]

If I search a Latin alphabet string of that article I manage to get 4 variants from Google after asking to show me duplicate pages as well. None of them is /wiki/

Searching '"漢語，又称中文、华语" site:wikipedia.org' yielded two results including zh.wap.wikipedia.org/zh-tw/汉语 but that's another bug.

Attached:

If you use google to search "汉语维基百科", the first result would be https://zh.wikipedia.org/zh/汉语 and all the other results below are using /zh/.

This is NOT optimal because it will show page in original variant instead of user's preference (in my case, zh-cn).

However if you search with "汉语维基百科 site:wikipedia.org", the first result will become https://zh.wikipedia.org/wiki/汉语 as well as other links. This is optimal because /wiki/ links would automatically jump to language variant that user wants.

I have no idea what causes this strange situation (it's even maybe Google's fault), but it needs to be fixed. It's quite annoying that users need to manually change language variant from Google result.

So I think this bug should be "Canonical URL should be /wiki/ links, but somehow Google doesn't honor it".

Liuxinyu970226 subscribed.Jul 22 2015, 12:30 AM

Aklapper mentioned this in T108443: Google doesn't honor canonical URLs of zh.wiki.Aug 18 2015, 3:42 PM

Nemo_bis mentioned this in T93213: Improve access to local language wikis by fixing bug in generation of hreflang tags in <head> of article pages.Jan 25 2016, 8:27 AM

Shizhao added a subtask: T125033: [DO NOT USE] Chinese Wikimedia projects (tracking) [superseded by #Chinese-Sites].Jan 28 2016, 2:29 AM

Liuxinyu970226 removed a subtask: T125033: [DO NOT USE] Chinese Wikimedia projects (tracking) [superseded by #Chinese-Sites].Jan 28 2016, 3:58 AM

Liuxinyu970226 added a parent task: T125033: [DO NOT USE] Chinese Wikimedia projects (tracking) [superseded by #Chinese-Sites].

Restricted Application added a subscriber: Stang. · View Herald TranscriptSep 28 2016, 3:33 AM

Aklapper added a project: Chinese-Sites.Dec 21 2016, 9:58 AM

Aklapper removed a parent task: T125033: [DO NOT USE] Chinese Wikimedia projects (tracking) [superseded by #Chinese-Sites].Dec 21 2016, 10:03 AM

zhuyifei1999 moved this task from Backlog to MediaWiki core on the Chinese-Sites board.Dec 21 2016, 8:07 PM

Liuxinyu970226 updated the task description. (Show Details)Dec 30 2016, 1:23 PM

Liuxinyu970226 mentioned this in T71026: Canonical links may be in fallback encoding.

Liuxinyu970226 updated the task description. (Show Details)Jan 5 2017, 1:02 PM

Amire80 moved this task from Untriaged to Script & term conversion on the I18n board.Feb 4 2018, 10:47 AM

Nemo_bis updated the task description. (Show Details)Sep 11 2018, 10:22 PM

Restricted Application added a subscriber: • Petar.petkovic. · View Herald TranscriptSep 11 2018, 10:22 PM

Change 609513 had a related patch set uploaded (by VulpesVulpes825; owner: VulpesVulpes825):
[mediawiki/core@master] Write language varaint link as child element rather than individual entry in sitemap

https://gerrit.wikimedia.org/r/609513

gerritbot added a project: Patch-For-Review.Jul 4 2020, 12:21 PM

VulpesVulpes825 claimed this task.Jul 4 2020, 2:11 PM

VulpesVulpes825 added a subscriber: tstarling.

As T198965#4438037 suggests, fixing Sitemap will not solve this issue unless T87140 gets implemented. Hence removing myself as the assignee of this task.

VulpesVulpes825 mentioned this in T257211: Determine frequency of usage of language variants.Sep 8 2020, 3:43 PM

I think we shouldn't be setting canonical url to /wiki on chinese variants pages. The localized pages have their own title. Setting the canonical url to /wiki makes them all show the same title in google search results.

As a Taiwanese user, it always bothers me to see simplified chinese title in my search results.

I would suggest we remove the canonical url.

I'm happy to help making the code changes if we agree on doing this.

Kindly refer to this discussion on Search Console Help

The combination of such subtle differences with correct implementation of hreflang and country targeted folders should exclude any necessity for canonicals anyway.

https://support.google.com/webmasters/thread/130615008?hl=en&msgid=130651877

Restricted Application added a subscriber: Ericliu1912. · View Herald TranscriptJun 5 2022, 2:48 PM

YLJ mentioned this in T305367: Google search traffic sending users to the wrong Chinese language versions of Wikipedia.Jul 3 2022, 5:08 PM

Winston_Sung subscribed.Dec 6 2022, 3:33 PM

Change 879579 had a related patch set uploaded (by Winston Sung; author: Winston Sung):

[mediawiki/core@master] Make meta canonical URL variant-language-aware

https://gerrit.wikimedia.org/r/879579

In T54429#7981483, @YLJ wrote:

I would suggest we remove the canonical url.

Kindly refer to this discussion on Search Console Help

The combination of such subtle differences with correct implementation of hreflang and country targeted folders should exclude any necessity for canonicals anyway.

https://support.google.com/webmasters/thread/130615008?hl=en&msgid=130651877

I think we do need a canonical set since we have multiple ways to access variants, /zh-*/ subpage and ?variant=zh-* query string.

@Winston_Sung We may need to use subpage as the canonical since links in the variants tab use subpage syntax.

In T54429#8522568, @Func wrote:

In T54429#7981483, @YLJ wrote:

I would suggest we remove the canonical url.

Kindly refer to this discussion on Search Console Help

The combination of such subtle differences with correct implementation of hreflang and country targeted folders should exclude any necessity for canonicals anyway.

https://support.google.com/webmasters/thread/130615008?hl=en&msgid=130651877

I think we do need a canonical set since we have multiple ways to access variants, /zh-*/ subpage and ?variant=zh-* query string.

@Winston_Sung We may need to use subpage as the canonical since links in the variants tab use subpage syntax.

@Func :

Due to technical limitations (as you mentioned, there should be only one canonical URL for each variant), it is controlled by the $wgVariantArticlePath configuration option.

Winston_Sung added a parent task: T108443: Google doesn't honor canonical URLs of zh.wiki.Jan 13 2023, 7:23 AM

In T54429#8522573, @Winston_Sung wrote:

Due to technical limitations (there should be only one canonical for one variant), it is controlled by the $wgVariantArticlePath configuration option.

Of course only one canonical, I mean the subpage should be used for WMF sites. Yes, so you should make use of it and keep things consistent.

Oh, I now understand what you meant.

As long as the configuration option $wgVariantArticlePath being set properly (as the language code directory (the "subpage")) on WMF sites, it will be used (for WMF sites).

Func mentioned this in T305540: Google search traffic sending users to the wrong Chinese language versions of Wikipedia.Jan 13 2023, 7:32 AM

@Func :

I think you misunderstand about the code

The part you mentioned is the parameter for $this->getTitle()->getCanonicalURL() , which is independent from the URL format configuration.

See Title::getCanonicalURL( $params )

Xiplus subscribed.Jan 13 2023, 8:37 AM

Jdlrobson added a project: Web-Team-Backlog.Jan 13 2023, 5:12 PM

Winston_Sung updated the task description. (Show Details)Jan 25 2023, 10:44 AM

Winston_Sung updated the task description. (Show Details)Jan 25 2023, 11:14 AM

Jdlrobson moved this task from Incoming to Needs Prioritization (Tech) on the Web-Team-Backlog board.Jan 26 2023, 8:12 PM

Jdlrobson edited projects, added Web-Team-Backlog (Needs Prioritization (Tech)); removed Web-Team-Backlog.

Jdlrobson moved this task from Backlog to Future Unplanned on the Web-Team-Backlog (Needs Prioritization (Tech)) board.Jan 31 2023, 12:47 AM

Jdlrobson mentioned this in T294716: hreflang Tags' alternate URLs are not fully-qualified.Feb 6 2023, 10:28 PM

Jdlrobson updated the task description. (Show Details)Feb 22 2023, 8:05 PM

The canonical behaviour described in the table in the description is working as requested.

In terms of ?action=(info|history|edit) the link[hreflang] will always point to the article page.

As far as I can see the remaining potential ask (from the original task description) here is to drop all link[hreflang] tags on ?action=info ?action=history and ?action=edit pages. After chatting to a Google representative, there doesn't seem to be any benefits from dropping those link tags.

If you feel there is any follow up work to be done here, please create a new ticket with detailed description of what is required and I will forward that question to Google.

Stang moved this task from MediaWiki core to Closed on the Chinese-Sites board.Feb 22 2023, 8:31 PM

Stang unsubscribed.Feb 25 2023, 10:33 PM

Change 879579 merged by jenkins-bot:

[mediawiki/core@master] OutputPage: Fix the behavior for canonical URL and alternate URLs

https://gerrit.wikimedia.org/r/879579

Jdlrobson mentioned this in T334365: +2 in mediawiki/ for Winston Sung.Apr 12 2023, 3:15 PM

Winston_Sung moved this task from Backlog to Core on the MediaWiki-Language-converter board.Dec 27 2023, 4:35 PM

Winston_Sung removed a project: Patch-For-Review.Dec 27 2023, 4:57 PM