Page MenuHomePhabricator

skipped item IDs
Open, HighPublic

Description

Nov 2020 update

So I looked at a small gap here:

So the skips don't even appear to be during a time of mass creations?
I think the best way to investigate this would probably to be to add some fairly verbose logging around the entity creation flows in it's own log channel?

Perhaps:

  • When Special:NewItem starts the process of creating a new item?
  • When wbeditentity start the process of creating a new item?
  • When an ID is claimed in the IdGenerator

This should then allow us to progress toward seeing exactly which requests end up generating an ID that end up not completing / saving an entity.

Original report:

After http://www.wikidata.org/wiki/Q50450 there are many skipped IDs. They're not deleted. Next page after Q50450 is Q50464.

What is going on?

Details

Reference
bz42362

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 12:52 AM
bzimport set Reference to bz42362.
bzimport added a subscriber: Unknown Object (MLST).

Q50514 - Q50567 is missing, too.

It could be a wikidata bug. "If an entry is posted via API and error occurs due to the interwiki conflict, new entry is not created. But new entry number is consumed and NEVER used again". These gap can be found other bot editings.

It's true that IDs may be "swallowed" by unsuccessful attempts to create an item. But I don't see this as a problem. The IDs have no significance, they could just as well be chosen randomly instead of sequentially. The ID system may well be changed to something else entirely at some point, e.g. GUIDs.

I want to thank you for reporting this anyway, because it *might* have been an indication for Something Bad happening, like items just vanishing without a trace. But I don't see any evidence of that. So, closing WONTFIX.

From what I've heard, it's enough to visit Special:NewItem to "consume" an ID.
True, the only damage is that it hurts our milestones addiction. :)

(In reply to comment #4)

From what I've heard, it's enough to visit Special:NewItem to "consume" an
ID.
True, the only damage is that it hurts our milestones addiction. :)

That's not correct. But any attempt to create an entity will consume an ID, including failed attempts.

(In reply to comment #5)

That's not correct.

Ok, let me rephrase that: if that was true, it would really be a bug, and one that should be fixed. But before filing a bug report, please make very sure that IDs are actually being consumed without any attempt to create anything.

It is possible to create through a "visit" to a link, but then the "link" must use a post request and contain a valid edit token for the user, so it must be created through a js-call.

Restricted Application added a subscriber: StudiesWorld. · View Herald Transcript
Bugreporter subscribed.

Boldly reopen, ideally no Qid should be "lost" - and no database transaction should be made for failed entity creation.

See another task for actual reason.

Reopen as a more general tracking task.

Reminder that this is still a thing:

I maintain a weekly updated page that is tracking the number of skipped item IDs (among other things) at https://www.wikidata.org/wiki/User:MisterSynergy/itemstats. During the past week (diff), the QID counter was increased by 586.750 from Q101579998 to Q102166748. However, 450.082 (76.7%) were skipped QIDs, and only around 130.000 QIDs are actual new items. This means that more than three out of four QIDs were skipped over the period of the past week; it also means that we have increased the number of skipped QIDs by almost 7% within only one week to more than 7 million meanwhile.

I know that we do not run out of QIDs and this seems not actually a very serious issue, but in some sense a densly packed QID space still seems desirable. I thus suggest to start another investigation how this can happen and eventually be mitigated.

Urgh. Yeah, this is not good. I thought we make major headway on this issue with T232620 but apparently, that's not enough. Skipping over three quarters is _bad_. Adding this to camp so we can look at some more of the subtickets @Bugreporter created.
Thanks for creating the stats!

Another idea is to allow user to reclaim Qids that are never used. Don't know how bad it is.

So I looked at a small gap here:

So the skips don't even appear to be during a time of mass creations?
I think the best way to investigate this would probably to be to add some fairly verbose logging around the entity creation flows in it's own log channel?
Perhaps:

  • When Special:NewItem starts the process of creating a new item?
  • When wbeditentity start the process of creating a new item?
  • When an ID is claimed in the IdGenerator

This should then allow us to progress toward seeing exactly which requests end up generating an ID that end up not completing / saving an entity.

(I probably should have written this on T268625)

Addshore updated the task description. (Show Details)

New numbers from the past week, in order to keep the momentum here:

Between 28 Nov 1:42 AM and 5 Dec 1:42 AM, around 1.250.000 QIDs have been skipped and around 175.000 new items have been created. The skip ratio on average over the entire week was ~87.5%.

Related activity started on Nov 29, around noon, and ended on Dec 3 in the evening. The skip rate during that time was constantly somewhere between 3 and 4 QIDs per second.

If anyone is interested, I would upload plots similar as previously here.

Week ending ...Total skipped itemsweekly increase
5 December 20208303905+1256459
12 December 20208351248+47343
19 December 20208380252+29004
26 December 20208420623+40371
2 January 20218431286+10663
9 January 20218459454+28168
16 January 20218473979+14525
23 January 20218487360+13381
30 January 20218505173+17813
6 February 20218514746+9573
13 February 20218524740+9994
20 February 20218535448+10708
27 February 20218542979+7531

This does not look overly concerning to me in the past weeks, and I do not see excessive streaks of skipped QIDs either. I have no further insight as to why these Q-IDs are being skipped, but I could provide lists with skipped QIDs if this helps.

Any further changes to avoid skipped Qids would likely need some larger changes to some core Wikibase code or how ids are generated.

Skipping probably started happening more once we fixed T194299: Lock wait timeout exceeded in SqlIdGenerator::generateNewId

We did have another idea for solving that issue back then in T194299#4712763

One idea that @daniel and I had for this was simply to spread the ItemId counter out over 2 or more rows.. having one row deal with an odd counter and another deal with an even counter.
This would mean that ids would no longer be guaranteed to be created in order, and the counters might drift (but we could put something in place to fix that?), but the issue of too much stuff trying to work on a single row would be solved.

This would mean we could move back to fetching Ids in the same transaction as the main writes (so the id claim would also be rolled back on failure).
But as long as we are okay having some skipped IDs I would propose not touching it more now, they are after all, only numbers!

@MisterSynergy Could you have another look at the current situation after the recent changes we made?

Continuation of the table above (numbers taken from the revision history of https://www.wikidata.org/wiki/User:MisterSynergy/itemstats):

Week ending ...Total skipped itemsweekly increase
27 February 20218542979
6 March 20218550632+7653
13 March 20218596041+45409
20 March 20218630017+33976
27 March 20218652766+22749
3 April 20218666101+13335
10 April 20218679152+13051
17 April 20218688972+9820
24 April 20218704612+15640
1 May 20218714560+9948
7 May 20218724841+10281
15 May 20218735465+10624

Looks fine to me. Maybe something happened during two weeks in March, but not excessively so that we would need to worry.

This would mean we could move back to fetching Ids in the same transaction as the main writes (so the id claim would also be rolled back on failure).
But as long as we are okay having some skipped IDs I would propose not touching it more now, they are after all, only numbers!

I am strongly against doing so in one transaction. Previously this results in database lock timeout, and this may be exploited by anyone without being detected (i.e. a strong DoS potential).

Continuation of the table above (numbers taken from the revision history of https://www.wikidata.org/wiki/User:MisterSynergy/itemstats):

Week ending ...Total skipped itemsweekly increase
27 February 20218542979
6 March 20218550632+7653
13 March 20218596041+45409
20 March 20218630017+33976
27 March 20218652766+22749
3 April 20218666101+13335
10 April 20218679152+13051
17 April 20218688972+9820
24 April 20218704612+15640
1 May 20218714560+9948
7 May 20218724841+10281
15 May 20218735465+10624

Looks fine to me. Maybe something happened during two weeks in March, but not excessively so that we would need to worry.

Thank you! It seems then we're still skipping somewhere around 1% of IDs if I didn't calculate wrong. That still seems too high to me. Will dig into it more then.

Thank you! It seems then we're still skipping somewhere around 1% of IDs if I didn't calculate wrong. That still seems too high to me. Will dig into it more then.

One option we can still consider is “punishing” users for bad API requests, as sketched out in T272032#6834213. Right now we let bad API requests burn through 90 IDs per minute, which is better than not rate limiting ID generation at all, but still allows for a lot of skipped IDs, I think.