The Robots Exclusion Standard (http://www.robotstxt.org/orig.html), which defines the syntax of robots.txt files, says, among other things, that:
- there may not be more than one record with "User-Agent: *", and
- sections may not contain blank lines.
Thus, the only standard-conforming (and thus reasonably reliable) way for users editing the local part of robots.txt, as specific in [[MediaWiki:Robots.txt]], to include rules pertaining to all robots is to include them at the very top with no blank lines preceding them, so that they get appended directly to the "User-Agent: *" section of the global robots.txt.
Unfortunately, even this doesn't currently work right, since the global part of Wikimedia's robots.txt contains some blank lines at the end. Please remove said lines or replace them with comments.
(Actually, the current implementation is somewhat annoyingly fragile in general. It might be better for MediaWiki itself to parse the content of both the local and global parts of robots.txt (it's not hard), preferably with fairly relaxed parsing rules, and merge them properly into a single file guaranteed to have correct syntax. While it it, the software could try to provide notification of any unrecognized lines and other potential errors detected during the parsing.)
Version: unspecified
Severity: normal
URL: http://en.wikipedia.org/wiki/MediaWiki:Robots.txt