herd wrote:

from http://toolserver.org/~amidaniel/chanlogs/%23mediawiki/20080726.txt ->

[09:35:22] <Sadik_Khalid> Hi, when I tried to edit this page (http://ml.wikipedia.org/wiki/%E0%B4%B2%E0%B5%82%E0%B4%AF%E0%B4%BF_%E0%B4%AA%E0%B4%BE%E0%B4%B8%E0%B5%8D%E0%B4%9A%E0%B4%B0%E0%B5%8D%E2%80%8D) I am getting Egypt page (http://ml.wikipedia.org/wiki/Egypt)
[09:37:45] <Sadik_Khalid> History page don't mach with the content of the article

Changing title since this occurs outside enwiki.

Possibly related to bug 14930

Also may be related to the recent ext. storage problems on one cluster (https://wikitech.leuksman.com/view/Server_admin_log)

OK, I can't find any relevant software changes. I'm almost sure this is due to the above issue. As things are now, as of now, no *new* edits should be recorded wrongly anymore.

jeluf wrote:

This happened due to a master switch on the external storage cluster.

Apparently, the new master didn't have an up-to-date replica of the master, a few records were missing. Due to this, the same text IDs were used twice. The edits saved on the old master that were not replicated to the new master are lost, no way to get them back.

I have to close this bug as "FIXED" because there's no "CANTFIX"

It wasn't fixed. srv104 still had an old copy of the configuration (because it's not reachable by ssh), and so it was still writing blobs to srv101. I've taken srv104 out of LVS rotation now. Maybe we'll be able to recover the edits from srv101 at some point, but it looks like it might be hanging on I/O now.

stigmj wrote:

Occured here as well: http://no.wikipedia.org/w/index.php?title=Vinterkrigen&diff=next&oldid=4096373

jeroenvrp wrote:

I can confirm this on nl.wikipedia to.

See e.g. http://nl.wikipedia.org/w/index.php?title=Yang_Yaozu&diff=13286529&oldid=13139324

In the recent changes this revision have added 15 bytes, but the page is empty:
http://nl.wikipedia.org/w/index.php?title=Yang_Yaozu&action=edit&oldid=13286529

See also http://nl.wikipedia.org/w/index.php?title=Yang_Yaozu&action=history (2.159 bytes vs. 2.144 bytes).

jeroenvrp wrote:

Ok I didn't saw it was fixed.

Still happening:
http://es.wikipedia.org/w/index.php?title=Wikipedia:Caf%C3%A9/Portal/Archivo/Propuestas/Actual&diff=19106298&oldid=19104167

Next two edits (both trying to revert it) also failed:
http://es.wikipedia.org/w/index.php?title=Wikipedia:Caf%C3%A9/Portal/Archivo/Propuestas/Actual&diff=next&oldid=19106298
http://es.wikipedia.org/w/index.php?title=Wikipedia:Caf%C3%A9/Portal/Archivo/Propuestas/Actual&diff=next&oldid=19106364

herd wrote:

Unsure if related, but these do not show the revision #798283:

• http://meta.wikimedia.org/w/index.php?title=Help:Magic_words&oldid=798283
• http://meta.wikimedia.org/w/index.php?title=-&oldid=798283
• http://meta.wikimedia.org/w/index.php?title=Help:Magic_words&diff=prev&oldid=798283

And yet, these do (sort of):

• http://meta.wikimedia.org/w/index.php?title=Help:Magic_words&diff=798284&oldid=798283
• http://meta.wikimedia.org/w/api.php?action=query&prop=revisions&revids=798283&rvprop=size|content

Although, Per VP/T Tim said:

It looks like the anomalous blank revisions are just cache pollution, and will
fix themselves when the cache expires in a week. The revisions that show the
wrong article are due to database corruption, and will need to be fixed manually.

daniel wrote:

This edit is attributed to my bot
http://commons.wikimedia.org/w/index.php?title=Image%3AHyena_pup.jpg&diff=13062289&oldid=12189366

But it is pretty much impossible that the bot performed it (nothing remotely similar to CopyVio tagging is in the source code).

Might be due to the same server issue, although the nature of the glitch seems different from the ones reported.

(In reply to comment #13)

But it is pretty much impossible that the bot performed it (nothing remotely
similar to CopyVio tagging is in the source code).

Might be due to the same server issue, although the nature of the glitch seems
different from the ones reported.

Also note that the length reported in the history is larger than the edit.
I understand this happens becaouse the write goes to the false master and
then the real one reuses the same revision id.

Probably we could find between the deleted revisions at a similar time,
another with that same content.

Another magic blanking:
http://es.wikipedia.org/w/index.php?title=Wikipedia:Vandalismo_en_curso&diff=19107113&oldid=19107017

Should be fixed as of July 30, 03:00 UTC. Initially, ordinary edits processed by srv101/srv104 polluted the revision cache, which has an expiry of one week. This was identified and fixed (without me ever seeing this bug report) on July 27, by removing those servers from HTTP LVS. However, they continued to run the job queue, and refreshLinks jobs would have continued to pollute the revision cache. This was fixed on July 30, by firewalling srv101/104 from all core DB servers.

I'm running a script to fix the revision cache. This will make the old revision view and old revision edit work properly. Any broken diffs will have to be fixed manually by appending &action=purge to the diff URL.

Note that the script only affects page blankings (which are due to cache pollution), not replacement with unrelated text, which is due to corruption of the core DB with incorrect text rows referencing blob_ids on the old cluster17 master, srv101.