Are there any specific examples?

Are whitespace mismatches due to problems parsing the way whitespace is encoded in the XML, or due to the XML dumps actually containing incorrect whitespace?

(The dumps may well contain incorrect whitespace, most likely due to inconsistencies in parsing the previous whitespace when doing multiple passes combining text from previous dumps with new stub dumps, etc.)

(In reply to comment #1)

Are there any specific examples?

Are whitespace mismatches due to problems parsing the way whitespace is encoded
in the XML, or due to the XML dumps actually containing incorrect whitespace?

Do the XML dumps use the xml:space="preserve" attribute?

I would like a specific page ID, revision ID and dump file to look at, if someone can point me to one.

Anarchism(12)
RevisionId: 233194

From the 2010-01-30 XML dump at the end of the 233194 revision (notice the line breaks before the closing </text> tag)

[...]
/Talk &lt;br&gt;

/Todo &lt;br&gt;

[[Anarchy/Talk]] [http://www.wikipedia.com/wiki.cgi?action=history&amp;id=Anarchy Anarchy History] (The content of Anarchy and Anarchism have since been merged into this version)

</text>

From the API (http://en.wikipedia.org/w/api.php?action=query&prop=revisions&revids=233194&rvprop=content&format=jsonfm) (notice that the string ends right after the last non-whitespace character)

{
"query": {

		"pages": {
			"12": {
				"pageid": 12,
				"ns": 0,
				"title": "Anarchism",
				"revisions": [
					{
						"*": "''Anarchism'' is <removed most of the text here -Aaron Halfaker> (The content of Anarchy and Anarchism have since been merged into this version)"
					}
				]
			}
		}

}

}

(Yes, the XML files have <text xml:space="preserve"> in them.)

I had a look at the output we get from ExternalStore::fetchFromURL()

The text we get back has a newline after the final parenthesis.

That text is 8884 bytes long, which matches the rev_len recorded in the revision table and in the XML dump file. When I apply the various conversions for & < > " and strip the ^Ms I get the byte count of the text entry in the xml file: 8930.

When I do the same conversions for the json format (for " \r \n and /) I come up one byte longer, 9160, than the actual json output text, 9159. My conclusion is that the json formatter or perhaps generally the API loses that newline at the end.

In T29773#1193262, @Nemo_bis wrote:

My conclusion is that the json formatter or perhaps generally the API loses that newline at the end.

Then, if still current/interesting, it should be reported as an API bug. Cc @Anomie.

The one example given, https://en.wikipedia.org/w/api.php?action=query&prop=revisions&revids=233194&rvprop=content&format=jsonfm, does not match the reported output with respect to trailing newlines. The page content from the json object is currently 8884 bytes long.