Bumping to critical, since it may result in data loss for clients that cannot process the old style format. We really do not want them to implement that, we changed for a reason...

Btw: In order to check for old style serializations, grep for "entity". To detect new style serialization, check for "descriptions" (plural).

See also https://lists.wikimedia.org/pipermail/wikidata-l/2014-October/004843.html

Just confirming, this only applies to XML dumps, and not the new JSON dumps?

The reason seems to be backupTextPass.inc, see bug 72361.

• Bug 72613 has been marked as a duplicate of this bug. ***

Can I please have a status update on this? Do we know why it is happening?

In T74348#787697, @Lydia_Pintscher wrote:

Can I please have a status update on this? Do we know why it is happening?

As far as I know the problem is that during dump creation content from the last dump is being scraped in case nothing changed. That's probably fine for wikitext, but of course that bypasses our on-the-fly serialization change.

Old revisions are indeed read from the old dump, as long as the length of the revision text is correct. And indeed this is a necessity; the db servers cannot handle requests for all revisions anew, and even if they could the dumps would take many times loger to generate as well. The only thing that can be done is a manual run of the specfic pass without prefetch, which will take... as long as it takes. I need to check with Sean (DBA) about it before doing so.

@Ariel: My patch for T74361 hooks into TextPassDumper::getText() and applies on-the-fly transformation regardless of whether the text comes from a previous dump (prefetch), or the database. Can you confirm that this will indeed fix the issu? The relevant diff is here: https://gerrit.wikimedia.org/r/#/c/168126/15/maintenance/backupTextPass.inc

Thanks for the patch! I will check it out in the next couple of days. I'm really sorry for the long delay; I've been out for medical reasons and am now trying to get caught up on everything.

I ran a series of tests locally and also checked production output. I can verify that the transform is actually applied, the output looks good to me for prefetch or from the database, but a consumer of the data should probably look at it for 5 seconds to verify that the output format is they way you want it.

Hello? Any wikidata dumps consumers on this ticket? Otherwise I'll ask in xmlatadumps-l.

@hoo: could you have a look?

In T74348#1062351, @Lydia_Pintscher wrote:

@hoo: could you have a look?

Just kicked of the download of a dump, I'll verify some old revisions once that's done (later today).

In T74348#1059331, @ArielGlenn wrote:

Hello? Any wikidata dumps consumers on this ticket? Otherwise I'll ask in xmlatadumps-l.

In T74348#768660, @daniel wrote:

Bumping to critical, since it may result in data loss for clients that cannot process the old style format. We really do not want them to implement that, we changed for a reason...

Btw: In order to check for old style serializations, grep for "entity". To detect new style serialization, check for "descriptions" (plural).

hoo@tools-dev:~$ grep -c '"entity"' wikidatawiki-20150207-pages-articles.xml 
129630

:(

right. this is what you want; the old style 'entity' is gone, the new style 'descriptions' is present. or am I missing something?

In T74348#1069658, @ArielGlenn wrote:

right. this is what you want; the old style 'entity' is gone, the new style 'descriptions' is present. or am I missing something?

To me it seems like the old style entity is still present.

ugh, I stare it for an hour and I'm still blind. Let me look at it for another hour... sorry.

OK, I no longer feel as stupid. The number of items with the 'entity' format is small in comparison to the total number of qualities, we would expect the opposite if old revisions were being kept as is. And as I said I had checked with local testing that the export transform is indeed being called and changing the content. So I had a look at the problematic entries. It turns out that all but 27 are of the form

<text xml:space="preserve">{&quot;entity&quot;:&quot;Q547932&quot;,&quot;redirect&quot;:&quot;Q6150957&quot;}</text>

so I guess serializing of redirects needs work. I checked that newly added redirects are dumped with this format. The few remaining matches are likely discussions that happen to include the string; I spot checked some and found that to be the case.

Um, "with this format" means new redirects are dumped with {&quot;entity&quot; ... etc.

Is anyone looking at the redirects serialization?

@daniel: Could you have a quick look at this? Looks fixed to me, but I think you're the only one who can tell for sure.

Fore redirects, the encoding {&quot;entity&quot;} is correct. There is no "old" encoding for redirects, entity redirects didn't exist when we used the old serialization format.

So, searching for &quot;entity&quot; is not a good indicator for detecting old-style serialization.

@JanZerebecki: Redirects are serialized like this:

{"entity":"Q23","redirect":"Q42"}

Old style serialization ends with this:

,"entity":"q207"}

So, if you egrep for ,&quot;entity&quot;:&quot;[qQpP][0-9]*&quot;\}, you should find only old style serializations.

Also, old style serialization will contain &quot;label&quot;:{, while new style should contain &quot;labels&quot;:{ (using lables, plural).

Btw, if someone can tell me where to find a full history dump of wikidata, I'd be happy to check this myself. The annoying part here is to download and store the behemoth...

other examples of old serializations can be found here:
https://github.com/dbpedia/extraction-framework/blob/master/core/src/main/scala/org/dbpedia/extraction/wikiparser/impl/json/JsonWikiParser.scala#L62-L67

@Jimkont: broken serialization of empty lists is a separate issue, unrelated to unconverted old-style serializations.

I'm now running the following on tool labs to find "old" serializations:

daniel@tools-bastion-01:/public/dumps/public/wikidatawiki/20150330$ bzgrep ',"entity":"[qQpP][0-9]*"\}' wikidatawiki-20150330-pages-meta-history.xml.bz2 | tee ~/wikidatawiki-20150330-pages-meta-history.bad-serialization.txt

The double-check didn't turn anything up either. The dump seems to be clean.

Great news!