Aaron is going to take an initial look at this one, but we might ask someone else to wrap this one up.

*** Bug 35011 has been marked as a duplicate of this bug. ***

I just ran a cleanup script on all wikis.

I also removed a non-suppress block from one user in order to fix the
suppression block (avoiding a duplicate key error). Inspection of UserList
looks fine.

I've found a number of local blocks + suppression on random users instead of the intended target from the global lock + oversight.

Several of these users don't even have an account on the wiki. I've unblocked a number, but do I have to go thru all of the global oversights? If so, could I have an estimate of from where to where so I can limit the work necessary?

It just struck me, did you by any chance removed the logs of the block while leaving the underlaying block still there? 'cause I couldn't find anywhere the logs for these blocks on the local wiki, while they should be in Special:Log/suppress, yet the blocks were there and fully working.

reopening as the issue is clearly not fixed

No log entries were removed.

If a user was blocked locally and later suppressed, I don't think the script fixed those (to avoid duplicate key errors). The local blocks need to be dealt with, probably manually.

Which Special:Log/suppress? The one on meta. crosswiki blocks are only logged there.

These users couldn't have been already blocked locally, as some of them were actually non existent locally...

What I found so far is on hiwiki (the public unblocked entries in https://hi.wikipedia.org/w/index.php?title=%E0%A4%B5%E0%A4%BF%E0%A4%B6%E0%A5%87%E0%A4%B7%3ALog&type=block&user=Snowolf&page=&year=&month=-1&tagfilter= ) and bowiki (the three unblock entries in https://bo.wikipedia.org/w/index.php?title=Special%3ALog&type=block&user=Snowolf&page=&year=&month=-1 ).

They were not logged anywhere at all, not central because some of them don't even have SUL and in any case they global account was unaffected, what they got was a local block+hideuser. I can start checking the affected wikis, but I kinda need a timeframe where the bug was in place to limit the scope of the search.

I would say it was broken since 1.18. From the logs:

• Monday, September 19 (-20), 23:00-01:00 UTC (4pm-6pm PDT): MediaWiki 1.18 deployment to test2

...so the timeframe would start there and end when bug 34539 was closed.

(In reply to comment #9)

I would say it was broken since 1.18. From the logs:

• Monday, September 19 (-20), 23:00-01:00 UTC (4pm-6pm PDT): MediaWiki 1.18

deployment to test2
...so the timeframe would start there and end when bug 34539 was closed.

Bloody hell this is a really critical issue to solve!

Two more from nlwiki: https://nl.wikipedia.org/wiki/Speciaal:Deblokkeren/PoloLine and https://nl.wikipedia.org/wiki/Speciaal:Deblokkeren/Owenwho - thought I'm unable to unblock them as I get an error (a lock of the database: "1205: Lock wait timeout exceeded; try restarting transaction (10.0.6.23)").

Do we have a timeline on this?

To be more specific this is related to bug 34995 which was merged with this one but the symptoms there are pretty critical. (sorry for rapid fire emails)

apparently I fail in many ways at this time in the morning. bug 35011 is the correct one regarding the incorrect users being blocked.

Avoiding assignment battles -- gave it to Sam since he confirmed the remaining issues are shell, but adding Aaron to CC.

(In reply to comment #8)

These users couldn't have been already blocked locally, as some of them were
actually non existent locally...

I ran a query to find oversight blocks for accounts that did not exist. There
are almost none that ipb_user = 0 and good few scattered about that have
ipb_user equal to some other value. In the latter case, the ipb_user is
actually correct and the name wrong. The name corresponds to the metawiki user
name for the user ID with the value of ipb_user. It should be the local user
name for that user ID.

Judging from the block timestamps, this is additional code bug that is ongoing.

I just ran a new script to fix the affected rows for the above problem.

Aaron is working out this issue. Here's the situation as I understand it thus far:

• There is a problem where MediaWiki will block the incorrect user on a local wiki when a global user block is performed. This problem was likely introduced when MediaWiki 1.17 was deployed (last year: 2011-02-16), and went unnoticed for quite some time.
• A manifestation of one of the root causes (bug 34539) was reported 2012-02-20. Aaron fixed that root cause on 2012-02-28. That didn't retroactively fix any problems caused prior to 2012-02-28, but Aaron believed that it should have stopped new instances from occurring.
• Vituzzu reported Bug 35011on 2012-03-06, which I believe was the first time someone noticed innocent users getting blocked.
• Aaron closed that issue as a duplicate of this issue (bug 34995), believing that any problems remaining were problems lingering from prior to 2012-02-28.
• Aaron closed this issue on 2012-03-08, believing that the cleanup script he ran should have caught all of the lingering issues.
• Snowolf reopened this 2012-03-15, having found more instances. At the time, it was unclear if those were from before 2012-02-28 or after, but we now know that's irrelevant.
• There was some back-and-forth about how to clean up what was thought to be the remaining problems. What's clear now is that the list was growing.
• Earlier today, Aaron ran a cleanup script, referenced in comment 16.
• Later today, Aaron discovered another problem, which he fixed (r114672) and just deployed (2012-04-03 00:37 UTC). That means *should* see an end to new cases of this.

So, that's the chronology up until this point. Here's what should happen moving forward:

• Aaron will also run the cleanup script again. At this point, the issue should be resolved.
• If it isn't resolved, please be sure to take note of the timestamp of any erroneous blocks, and reference that timestamp in new issues. Problems that occur due to blocks after 2012-04-03 00:37 UTC should especially be noted as such.

Thanks for your patience, and sorry for the confusion. There are a lot of moving parts here, and it's possible that there are more root causes that we haven't discovered yet.

Ok, confirmed with Aaron that he's run this script. So, this issue should be resolved now. Please let us know if you're seeing other issues here. Thanks!

There's a block from right before the reference time that however was a global suppression suppressing the IP of the blocking steward, I would like to get in contact with a dev to make sure this was fixed since but would not like to disclose the involved IP publicily here.

I am re-opening this bug per new information which cannot be shared here. Please contact me, mafk or Vito on IRC for details. It is rather urgent.

(In reply to comment #21)

I am re-opening this bug per new information which cannot be shared here.
Please contact me, mafk or Vito on IRC for details. It is rather urgent.

Can you email rob and I?

(In reply to comment #22)

(In reply to comment #21)

I am re-opening this bug per new information which cannot be shared here.
Please contact me, mafk or Vito on IRC for details. It is rather urgent.

Can you email rob and I?

I added you to the security bug that was opened. For some reason you were missed on it.

Aaron tells me this is fixed. Please reopen if there are further problems.

To note that I was today pointed to a suppressed account (@cswikiquote) that was visible to an admin in local special:listblocks; special:contributions and special:listusers

In mid April 2012 (towards the end of all the above cited work)

• account was blocked by admin
• subsequently account was locked and suppressed by steward

Today's work

1. Removing the block, removed one issue (though was still visible in Special:Listusers at csWQ)
2. subsequent removing the suppression and reapplying resolved the view in Sp:LU
3. having to unset oversight exposed it to view, and publically logged it which needed further suppression (so not sustainable to resolve)

So this specific issue is resolved, though one would think that this issue would still exist for other accounts. [Noting that I haven't reopened this bugzilla at this point.]

Thanks.