Crawlers may still index pages that should be disallowed due to encoded characters, see the following example:
Disallow: /wiki/Wikipedia:Arbitration/
Disallow: /wiki/Wikipedia3AArbitration/
Disallow: /wiki/Wikipedia3AArbitration%2F
Disallow: /wiki/Wikipedia:Arbitration%2F
MediaWiki should generate these extra rules automatically for users.
Version: unspecified
Severity: enhancement
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | Feature | None | T16720 robots.txt (tracking) | ||
| Declined | Krinkle | T31162 Automatically add encoded URL lines for entries in MediaWiki:robots.txt |
Better still, use <link rel="canonical" href="http://en.wikipedia.org/wiki/X"> for this purpose.
More info: http://googlewebmastercentral.blogspot.com/2009/02/specify-your-canonical.html
FT2.wiki wrote:
Very useful, however we need to check what happens if some of the alternate URLs to the canonical page are excluded by robots.txt and some aren't.
Does it apply the robots.txt rule that it has for the "canonical" page, to all alternatives? Or does it get confused?
Example:
/Wikipedia%3AArbitration%2FExample is stated to have /Wikipedia:Arbitration/Example as its canonical link. However one of these is NOINDEXed via robots.txt or in its header and one isn't. Knowing the canonicity helps to identify these as "duplicates" and "the same page". But does it guarantee both of these will be treated as NOINDEXED if one of them is and the other isn't? Or do we still have to cover all variants of the URL in robots.txt?
FT2.wiki wrote:
To clarify, URL variants where robots.txt or header tags prohibit spidering will probably be excluded from spidering in the first place. So Google will be left to collate those URL variants it came across where robots,txt or header tags _didn't_ prevent spidering -- and a "canonical" setting which states these are all the same page.
Ie this setting could help avoid duplicates but my guess is it probably _won't_ prevent URLs not stopped by robots.txt or header tags from being listed in results.
Changing product/component to Wikimedia/Site requests, MediaWiki:Robots.txt is a WMF hack, there's no such feature in MediaWiki core.
(In reply to comment #0)
MediaWiki should generate these extra rules automatically for users.
(In reply to comment #4)
MediaWiki:Robots.txt is a WMF hack, there's no such feature in MediaWiki core.
Now, how to prioritize...