Page MenuHomePhabricator

Respect wikisite and EditThis throttling
Closed, ResolvedPublic

Description

See http://wikistats.wmflabs.org/display.php?t=et&s=http_desc
The utmost of them work in the browser.


Version: unspecified
Severity: major

Details

Reference
bz59742

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:23 AM
bzimport set Reference to bz59742.

That's because they have a very aggressive throttling, like wiki-site.
Only 1 request every 60 seconds is allowed on EditThis, more or less. On wiki-site I usually make 1 every 240 s even.
It's easy to sleep more, but if the cronjobs are daily I'm not sure how long each of them can last. Probably we'd need to stop after a certain amount of hours, or something.

yep, there are currently no cron jobs for wikisite and editthis until this is resolved somehow

Robih, feel like calculating that? <current number of wikis> x <seconds we have to wait> .. is that under or over 24h?

WikiSite needs to allow us an interval of 72 seconds to get around in 48 hours.
EditThis needs to allow us an interval of 257 seconds to get around in 96 hours.

Change 166162 had a related patch set uploaded by Dzahn:
add a delay between updates for wikisite/editthis

https://gerrit.wikimedia.org/r/166162

Change 166162 merged by Dzahn:
add a delay between updates for wikisite/editthis

https://gerrit.wikimedia.org/r/166162

Change 166164 had a related patch set uploaded by Dzahn:
add update cronjobs for wikisite/editthis

https://gerrit.wikimedia.org/r/166164

Change 166164 merged by Dzahn:
add update cronjobs for wikisite/editthis

https://gerrit.wikimedia.org/r/166164

thanks! .. so i:

added delay:

https://gerrit.wikimedia.org/r/#/c/166162/

added cron jobs:

https://gerrit.wikimedia.org/r/#/c/166164/

please keep an eye on them over the weekend. let's just see how far it gets

(In reply to Daniel Zahn from comment #9)

please keep an eye on them over the weekend. let's just see how far it gets

Since this morning (9 UTC) it seems to be going well, Before that, only an endless spiral of 503 errors in quick succession, last ones being 2014-10-10 15:23:19 / 2014-10-10 15:23:20

Seems to have worked well! Only a few dozens wikis errored out now.
http://wikistats.wmflabs.org/display.php?t=et&s=http_desc
http://wikistats.wmflabs.org/display.php?t=si&s=http_desc
wiki-site shows some 1200 wikis with 302, seemingly some now-deleted wikis; separate issue.