itwiki looks like it would make a good wiki to set the debugging code up on. Simple is small enough, too, I think.

Now, to get the debugging code.

• Bug 29691 has been marked as a duplicate of this bug. ***

mr.heat wrote:

More broken files:
http://commons.wikimedia.org/wiki/File:Flies_1.JPG
http://commons.wikimedia.org/wiki/File:Flies3.JPG
http://commons.wikimedia.org/wiki/File:Flies4.JPG
I tried purging and even an other browser but nothing works. All thumbnails show the old image.

Bryan.TongMinh wrote:

Checked the first image, it gives an cache hit. If I add a query string it gives a cache miss, but still shows the wrong image. So probably not a squid issue in this case?

Another one: http://commons.wikimedia.org/wiki/File:Bucharest-Metro-Map-2011.png

(In reply to comment #29)

Checked the first image, it gives an cache hit. If I add a query string it
gives a cache miss, but still shows the wrong image. So probably not a squid
issue in this case?

What were the exact URLs you got hits and misses for? Are you sure you weren't hitting the description page instead or something like that?

Bryan.TongMinh wrote:

http://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/Flies_1.JPG/120px-Flies_1.JPG

GET /wikipedia/commons/thumb/d/d3/Flies_1.JPG/120px-Flies_1.JPG HTTP/1.1
Host: upload.wikimedia.org
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:5.0) Gecko/20100101 Firefox/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Referer: http://commons.wikimedia.org/wiki/File:Flies_1.JPG
Cookie: __qca=P0-1940361223-1289932920877

HTTP/1.0 200 OK
Server: nginx/0.7.65
Date: Mon, 04 Jul 2011 08:43:13 GMT
Content-Type: image/jpeg
Content-Length: 1033
Last-Modified: Thu, 03 Feb 2011 00:45:16 GMT
Accept-Ranges: bytes
Age: 411
X-Cache: HIT from sq80.wikimedia.org, MISS from sq84.wikimedia.org
X-Cache-Lookup: HIT from sq80.wikimedia.org:3128, MISS from sq84.wikimedia.org:80

http://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/Flies_1.JPG/120px-Flies_1.JPG?test

GET /wikipedia/commons/thumb/d/d3/Flies_1.JPG/120px-Flies_1.JPG?test HTTP/1.1
Host: upload.wikimedia.org
User-Agent: Mozilla/5.0 (Windows NT 6.0; rv:5.0) Gecko/20100101 Firefox/5.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip, deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Connection: keep-alive
Cookie: __qca=P0-1940361223-1289932920877

HTTP/1.0 200 OK
Server: nginx/0.7.65
Date: Mon, 04 Jul 2011 09:01:45 GMT
Content-Type: image/jpeg
Content-Length: 1033
Last-Modified: Thu, 03 Feb 2011 00:45:16 GMT
Accept-Ranges: bytes
X-Cache: MISS from sq58.wikimedia.org, MISS from sq82.wikimedia.org
X-Cache-Lookup: MISS from sq58.wikimedia.org:3128, MISS from sq82.wikimedia.org:80
Connection: keep-alive

Bryan.TongMinh wrote:

(In reply to comment #26)

Now, to get the debugging code.

Remove the error suppression operator on @unlink() in LocalFile::purgeThumbnails(), catch the E_NOTICES that it generates and log them somewhere?

darwin wrote:

I removed the border from this image in June 23:

http://commons.wikimedia.org/wiki/File:Southern_Railway_USA_class_on_the_Swanage_Railway.jpg

After 11 days, it is still showing the wrong version, though the thumbnail seems to have been updated meanwhile.

Bryan.TongMinh wrote:

Patch that logs failed unlink()s

(In reply to comment #26)

Now, to get the debugging code.

Here you go. Please find somebody to deploy it; it appears that this bug is manifesting itself much more lately.

Attached:

Cant(In reply to comment #35)

Created attachment 8740 [details]
Patch that logs failed unlink()s

Shouldn't we use wfSuppressWarnings() and wfRestoreWarnings() around the @unlink() call?

Attached:

Ariel says this issue may be related to multicast issues having to do
with our routers which are up for replacment. From IRC:

<apergos> the image/thumb purge issues may be multicast issues which have to

do with our broken routers

<apergos> as you might recall last time we had a flare up mark upgraded and

rebooted those

<apergos> and we were ok for awhile
<apergos> anyways new ones are on order, they are simply old
<hexmode> so, what is the ETA on the new ones?
<apergos> mark's gonna try to find out about it
<apergos> they have to be custom built I guess, they are junipers
<apergos> we talked about it in the ops meeting
<apergos> he didn't know there was another round of issues
<apergos> and I had forgotten about the routers
<apergos> being bad.

saibotrash wrote:

Another case:
http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Bfds_logo.svg/113px-Bfds_logo.svg.png
http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Bfds_logo.svg/563px-Bfds_logo.svg.png
Are broken. Should show a red-ish logo. Not a blue-ish. See e.g. http://upload.wikimedia.org/wikipedia/commons/thumb/e/e6/Bfds_logo.svg/363px-Bfds_logo.svg.png

bidgee-wiki wrote:

Original photograph was corrupted and reuploaded the photograph (which is uncorrupted) however the corrupted photograph thumbnails still show even after purging and deleting the cache on the computer.
http://commons.wikimedia.org/wiki/File:Old_Wagga_Wagga_Police_Station_%281%29.jpg

This "should" be fixed now, as there has been some router replacements etc

So any new purge requests/reuploads should behave as expected

alexander_berlin wrote:

I have just tested it. It is NOT fixed. The bug is easy reproducable with the given examples. So please first test it and then set it to fixed.

They look like they're working for me

On a wiki not related to Wikipedia, we seems to have the same issue, so the
routers might not be involved. I should add that we have only one cache server.

alexander_berlin wrote:

http://commons.wikimedia.org/wiki/File:Kaulitz_Berlin_1708.jpg

.. I have tried new upload, purge, browser cache cleared - no success.

X-Cache HIT from amssq58.esams.wikimedia.org, HIT from amssq61.esams.wikimedia.org
X-Cache-Lookup HIT from amssq58.esams.wikimedia.org:3128, HIT from amssq61.esams.wikimedia.org:80

Bryan.TongMinh wrote:

See my comment #29. I don't believe this is a squid purge issue, so I wouldn't have expected that problems with multicast were the cause. I suggest adding logging in LocalFile::purgeThumbnails() per my patch in comment #35.

Bryan.TongMinh wrote:

I did some debugging with Roan and Sam on http://commons.wikimedia.org/w/index.php?title=File:Kaulitz_Berlin_1708.jpg

• MediaWiki does not fail on the unlink() calls in purgeThumbnails()
• var_dump( wfFindFile( 'Kaulitz Berlin 1708.jpg' )->getThumbnails() ); returns an empty array on fenari
• ls /mnt/thumbs/wikipedia/commons/thumb/0/03/Kaulitz_Berlin_1708.jpg/ returns an empty directory
• http://upload.wikimedia.org/wikipedia/commons/thumb/0/03/Kaulitz_Berlin_1708.jpg/800px-Kaulitz_Berlin_1708.jpg?blsdsgfsd=dsgdry gives two cache MISSes and a Last-Modified header on 20 November 2010

So, according to MediaWiki there are no thumbnails left. However according to the webserver(?) they are still there. I'm not sure what the next step in debugging this would be.

saibotrash wrote:

examples:
http://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/%C3%96RF_2011.png/280px-%C3%96RF_2011.png should show a font with white outline like this does: http://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/%C3%96RF_2011.png/281px-%C3%96RF_2011.png

http://upload.wikimedia.org/wikipedia/commons/thumb/5/56/L%C3%B6rrach_-_Heilige_Familie6.jpg/800px-L%C3%B6rrach_-_Heilige_Familie6.jpg should be cropped like this: http://upload.wikimedia.org/wikipedia/commons/thumb/5/56/L%C3%B6rrach_-_Heilige_Familie6.jpg/801px-L%C3%B6rrach_-_Heilige_Familie6.jpg
using a wrong directory hash "5w" leads, as always, to a correct thumb: http://upload.wikimedia.org/wikipedia/commons/thumb/5/5w/L%C3%B6rrach_-_Heilige_Familie6.jpg/800px-L%C3%B6rrach_-_Heilige_Familie6.jpg

darwin wrote:

This issue is not resolved at all and is causing significant disruption at Commons.

See for instance:

http://commons.wikimedia.org/wiki/Commons:Administrators%27_noticeboard/Blocks_and_protections#Protection_of_File:Political_Regions_of_Sudan.2C_July_2006.svg

http://commons.wikimedia.org/wiki/Commons:Village_pump#Help:_I_cant.27t_restore_original_File:LocationSouthernSudan.svg

These situations are of particular significance, since it took only one user uploading once the wrong version of one file to start a "edit war" with a lot of users trying in good faith to revert it back to the original image. This is bond to happen again and again and again, and is wasting a lot of server space, since each time a file is reverted it is uploaded again to the servers.

It's also forcing the files to be moved to new names in order to bypass that bug, creating unnecessary redirects and name replacements in the projects using those files, and making the file log quite hard to follow, since deleted revisions are left behind each time the file is moved.

Please solve this bug as soon as you can, it is really damaging the project.

Paulo

darwin wrote:

By the way, this problem is not only about thumbnails. In the cases presented above, as well as in many others, the actual file failed to update, while the thumbnails themselves updated. The situation with this bug is now worse than it has ever been.

alexander_berlin wrote:

Is there at least a workaround available? Restart a server every Sunday?

In the meantime a hint should be placed on Commons, that there is a known problem with replacing images! Otherwise people are getting frustrated more and more. There are endless discussions on Wikipedia and Commons every day, because most users don't know about that bug.

darwin wrote:

(In reply to comment #50)

Is there at least a workaround available? Restart a server every Sunday?

In the meantime a hint should be placed on Commons, that there is a known
problem with replacing images! Otherwise people are getting frustrated more and
more. There are endless discussions on Wikipedia and Commons every day, because
most users don't know about that bug.

I agree, it should be in the site notice, indeed. I don't know how to do it or what to write there, though, but hopefully someone following this thread does.

Main image thumb:

GET /wikipedia/commons/thumb/3/3a/Gluehlampe_01_KMJ.jpg/462px-Gluehlampe_01_KMJ.jpg HTTP/1.1
Host: upload.wikimedia.org
Connection: close
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7
Cache-Control: no-cache
Accept-Language: de,en;q=0.7,en-us;q=0.3

Status: HTTP/1.0 200 OK
Server: nginx/0.7.65
Date: Fri, 08 Jul 2011 20:49:26 GMT
Content-Type: image/jpeg
Content-Length: 48173
Last-Modified: Sat, 15 Aug 2009 06:42:09 GMT
Accept-Ranges: bytes
Age: 135562
X-Cache: HIT from amssq59.esams.wikimedia.org
X-Cache-Lookup: HIT from amssq59.esams.wikimedia.org:3128
X-Cache: MISS from knsq18.knams.wikimedia.org
X-Cache-Lookup: MISS from knsq18.knams.wikimedia.org:80
Connection: close

Main imagethumb with purge command:

GET /wikipedia/commons/thumb/3/3a/Gluehlampe_01_KMJ.jpg/462px-Gluehlampe_01_KMJ.jpg?action=purge HTTP/1.1
Host: upload.wikimedia.org
Connection: close
User-Agent: Mozilla/5.0 (Windows NT 5.1) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30
Accept-Encoding: gzip
Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7
Cache-Control: no-cache
Accept-Language: de,en;q=0.7,en-us;q=0.3

Status: HTTP/1.0 200 OK
Server: nginx/0.7.65
Date: Sun, 10 Jul 2011 10:29:44 GMT
Content-Type: image/jpeg
Content-Length: 48173
Last-Modified: Sat, 15 Aug 2009 06:42:09 GMT
Accept-Ranges: bytes
X-Cache: MISS from amssq52.esams.wikimedia.org
X-Cache-Lookup: MISS from amssq52.esams.wikimedia.org:3128
X-Cache: MISS from amssq61.esams.wikimedia.org
X-Cache-Lookup: MISS from amssq61.esams.wikimedia.org:80
Connection: close

Despite the fact that the squid reports a MISS on both lookups, I still got served the wrong, old file. Either the squid fails to retrieve the updated thumb, or the server fails to generate and send the new thumb to the squid.

(The following might be really stupid and wrong) We use some sort of networked file system for where all the thumbs live right? Is it possible something is wrong with the networked file system such that when mediawiki deletes the files, it deletes it on the local server, but the changes don't get propagated? That's the only thing I could think of that would make sense in light of comment 46.

(In reply to comment #43)

On a wiki not related to Wikipedia, we seems to have the same issue, so the
routers might not be involved. I should add that we have only one cache server.

Is this your own wiki? If so, could you give us some more information so that we can track this down?

The commenter on comment #43 has since told me that what xe was seeing was due to a browser cache issue.

/me sighs

I've seen this bug multiple times in the past week. It's definitely not a browser caching issue. Adding ?action=purge to the thumbnail URLs fixes it, but it's rather tedious having to do this for all the thumbnails each time I upload a new version of an image. I hadn't noticed this bug much in the past year, but it seems to have become common in the past week.

Actually it looks like all the thumbs that I fixed with ?action=purge are now showing the bad thumbs again and ?action=purge won't fix them. I'm also seeing lots of people needlessly reverting images in an effort to work around this bug, which is causing some disruption on Commons.

Here's an example of an image which is currently showing the wrong thumbnail for me:
http://en.wikipedia.org/wiki/File:Team_Barnstar_Hires.png
It's showing the thumbnail for the original version of the image (with the bottom cropped off) rather than the fixed version. I imagine some people may be seeing the correct thumbnail, however, if the cache servers are not all in sync. My best guess as to what is happening is that some of the squid servers are refusing to flush their caches. Maybe Ryan Lane or someone could compare...
http://upload.wikimedia.org/wikipedia/commons/thumb/2/27/Team_Barnstar_Hires.png/120px-Team_Barnstar_Hires.png
...across different squid servers and see which ones have the bad version.

• Bug 29766 has been marked as a duplicate of this bug. ***

superjesse10 wrote:

I am sitting on an SVG animation that I haven't been able to use in a Wikipedia page because of this very issue.

I have posted a small outline of something I noticed here: http://en.wikipedia.org/wiki/User_talk:AquaGeneral

The image thumbnail is working at 140px, though at default sizes and 180px it doesn't.

saibotrash wrote:

more examples needed?
correct: http://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/BS11_BTS.JPG/400px-BS11_BTS.JPG
wrong: http://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/BS11_BTS.JPG/401px-BS11_BTS.JPG

matthiasbecker1967 wrote:

I observed that changing the image size using the upright parameter does produce a new thumb but only for that size. For example in
http://de.wikipedia.org/wiki/Diskussion:Nuklearkatastrophe_von_Fukushima#Tropisches_Wettergeschehen
the map on the right does not actualize. But it did, when I changed it from upright=1.5 to upright=1.4. A later reverting to the former size brought back the older image version.

With the specific file
http://commons.wikimedia.org/wiki/File:JTWC_wp0811.gif
I made yet another observation which is valid at least for the last four or five uploads.

When I uploaded the second version (edit comment "Adv. 12") the preview updated as it should. From the third version ("Adv. 13") onwards the preview and the thumb for the latest version didn't refresh anymore, so is both showed the map to "Adv. 12" all the time. But: Uploading a new version did make the second last thumb in the file versions history updating, i.e. after I uploaded version "Adv. 20" the version "Adv. 19" got it's correct thumb. Before uploading "Adv. 20" the "Adv. 19" thumb showed the same as the second version in row, "Adv. 12".

RT ticket filed to get Ops to put multicast listeners in place since we think there is some packet happening still. http://rt.wikimedia.org/Ticket/Display.html?id=1174

saibotrash wrote:

(In reply to comment #61)

I observed that changing the image size using the upright parameter does
produce a new thumb but only for that size. For example in
http://de.wikipedia.org/wiki/Diskussion:Nuklearkatastrophe_von_Fukushima#Tropisches_Wettergeschehen
the map on the right does not actualize. But it did, when I changed it from
upright=1.5 to upright=1.4. A later reverting to the former size brought back
the older image version.

This happens always (to my knowledge): only the thumbnail size/sizes which was/were in use somewhere before the new version was uploaded stay(s) old. All thumbnail sizes which were not in use before the upload of a new version will work as expected.

bidgee-wiki wrote:

Not sure what the issue is (could be a chmod issue which is preventing the new thumb to override the old thumb) but I've tried deleting, reuploading with some more deleting, deleting then uploading it, however still get the same old thumbnail which will not go away.
http://commons.wikimedia.org/wiki/File:Old_Wagga_Wagga_Police_Station_%281%29.jpg
http://commons.wikimedia.org/w/index.php?title=Special%3ALog&type=&user=&page=File%3AOld+Wagga+Wagga+Police+Station+%281%29.jpg&year=&month=-1&tagfilter=&hide_patrol_log=1

Has there been any change recently in the thumb's directory/filenaming naming scheme? It puzzles me that the squids keep reporting a MISS on purge, while the server continues to serve the old file. Is it possible the scaler puts the new file in the wrong location, and for that reason fails to delete the old thumbs?

http://rt.wikimedia.org/Ticket/Display.html?id=1174 to request a multicast listener.

saibotrash wrote:

Update:

(In reply to comment #47)

examples:
http://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/%C3%96RF_2011.png/280px-%C3%96RF_2011.png
should show a font with white outline like this does:
http://upload.wikimedia.org/wikipedia/commons/thumb/c/ca/%C3%96RF_2011.png/281px-%C3%96RF_2011.png

Did not work. [http://commons.wikimedia.org/w/index.php?title=File:%C3%96RF_2011.png&action=purge Purged the file page] - did not help. ?action=purge appended to the thumbnail URL did help. Works now.

http://upload.wikimedia.org/wikipedia/commons/thumb/5/56/L%C3%B6rrach_-_Heilige_Familie6.jpg/800px-L%C3%B6rrach_-_Heilige_Familie6.jpg
should be cropped like this:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/56/L%C3%B6rrach_-_Heilige_Familie6.jpg/801px-L%C3%B6rrach_-_Heilige_Familie6.jpg
using a wrong directory hash "5w" leads, as always, to a correct thumb:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/5w/L%C3%B6rrach_-_Heilige_Familie6.jpg/800px-L%C3%B6rrach_-_Heilige_Familie6.jpg

Works now (I have not purged it)

(In reply to comment #60)

more examples needed?
correct:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/BS11_BTS.JPG/400px-BS11_BTS.JPG
wrong:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/5a/BS11_BTS.JPG/401px-BS11_BTS.JPG

Works now (I have not purged it)

However, purging the thumbnails shouldn't be needed if the systems would work correctly.

saibotrash wrote:

(where is the edit function in bugzilla?)... But I did not have problems with recently uploaded files (like on 2011-07-26 [http://commons.wikimedia.org/wiki/File:2011_Oslo_attacks_map_deutsch.png#filehistory this file]).

russnelson wrote:

Might this be related to https://bugzilla.wikimedia.org/show_bug.cgi?id=30192 perhaps?

Reports from Saibo seem to show this is resolved now. Closing.

I and a few others at etwiki currently get an error message at this URL: http://upload.wikimedia.org/wikipedia/commons/thumb/d/d3/Bear_cub.jpg/220px-Bear_cub.jpg (other sizes seem to work).

In Nov-2010 image [[commons:File:Gulen.2 066.jpg]] wasn't available at 220px. (So a duplicate was uploaded to etwiki, which was avilable at 220px). Now the Commons image also scales at 220px. Checking other round numbers now, I get an error message here: http://upload.wikimedia.org/wikipedia/commons/thumb/f/f6/Gulen.2_066.jpg/300px-Gulen.2_066.jpg

Though the bear images hasn't been updated since 2008, is it related?

mr.heat wrote:

http://commons.wikimedia.org/wiki/File:Flies3.JPG is still broken.

alexander_berlin wrote:

also still broken
http://commons.wikimedia.org/w/index.php?title=File:Satellite_Image_of_Ruegen.jpg

This ticket requires a crises investigation team. It's been open WAY too long already and is way too obscure.

Resources need to be allocated here by WMF, since it seems it is very much infrastructure related.

(In reply to comment #71)

In Nov-2010 image [[commons:File:Gulen.2 066.jpg]] wasn't available at 220px.
(So a duplicate was uploaded to etwiki, which was avilable at 220px). Now the
Commons image also scales at 220px. Checking other round numbers now, I get an
error message here:
http://upload.wikimedia.org/wikipedia/commons/thumb/f/f6/Gulen.2_066.jpg/300px-Gulen.2_066.jpg

It doesn't sound like this problem is related to this bug.

(In reply to comment #72)

http://commons.wikimedia.org/wiki/File:Flies3.JPG is still broken.

Apparently works for me. Could you please explain what you expect to see that you are not seeing?

(In reply to comment #73)

also still broken
http://commons.wikimedia.org/w/index.php?title=File:Satellite_Image_of_Ruegen.jpg

This looks like it is working for me. Could you please explain what you expect to see that you are not seeing?

(In reply to comment #74)

Resources need to be allocated here by WMF, since it seems it is very much
infrastructure related.

I'm not sure what else we can do. The reports that cropped up in the past few hours are apparently not related to this problem.

This ticket has gotten a *lot* of attention. For example, we had the VP of the router maker expedite delivery of a new router because of this ticket.

Last night, after talking with Saibo on IRC (who left the comments above), I decided this problem was fixed, partly because of Saibo's tests and experience at Commons, but also because new reports had apparently stopped showing up here.

(Note that some of the issues talked about on this bug are really Bug #30192 which was just filed last night and is a high priority concern.)

alexander_berlin wrote:

(In reply to comment #77)

(In reply to comment #73)

also still broken
http://commons.wikimedia.org/w/index.php?title=File:Satellite_Image_of_Ruegen.jpg

This looks like it is working for me. Could you please explain what you expect
to see that you are not seeing?

compare the following two:

http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg

http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/552px-Satellite_Image_of_Ruegen.jpg

(In reply to comment #79)

(In reply to comment #77)

This looks like it is working for me. Could you please explain what you expect
to see that you are not seeing?

compare the following two:

Thanks, that helped. Appending ?action=purge to the darker thumb URL cleared it.

I'll update the RT ticket with this new information.

foxyshadis wrote:

I still see a few images with thumbnails that don't match, although new uploads are fixed. Perhaps instead of waiting for new uploads or for someone in the know to manually purge them as they come across them, there should be a mass thumbnail purge and rebuild. (Since people and pages can use any random size they want, it probably isn't even possible to rely on that.) Not all at once, because I could see that bringing the servers to their knees and frustrating everyone, but staged by hash group or as a low priority constant process over a week or two.

mr.heat wrote:

(In reply to comment #76)

(In reply to comment #72)

http://commons.wikimedia.org/wiki/File:Flies3.JPG is still broken.

Apparently works for me. Could you please explain what you expect to see that
you are not seeing?

Sorry, I forgot. In my 640px setting it still shows the wrong version:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Flies3.JPG/640px-Flies3.JPG

(In reply to comment #82)

(In reply to comment #76)

(In reply to comment #72)

http://commons.wikimedia.org/wiki/File:Flies3.JPG is still broken.

Apparently works for me. Could you please explain what you expect to see that
you are not seeing?

Sorry, I forgot. In my 640px setting it still shows the wrong version:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Flies3.JPG/640px-Flies3.JPG

Fixed

mr.heat wrote:

This is ridiculous. In the moment I previewed an edit in the sandbox, the broken 640px thumbnail was recreated. Now it shows the right version.

http://commons.wikimedia.org/w/index.php?title=Commons:Sandbox&diff=57617170&oldid=57579026

I tried to use purge before (multiple times) but it did not fixed the problem.

I just appended ?action=purge to the full url and it fixed it

mr.heat wrote:

(In reply to comment #85)

I just appended ?action=purge to the full url and it fixed it

No, it did not. I tried ?action=purge multiple times since I first reported the problem here and it never did anything. It seems my sandbox edit recreated the thumbnail. But even if this is true, it does not solve the problem that caused all the broken thumbnails.

(In reply to comment #86)

(In reply to comment #85)

I just appended ?action=purge to the full url and it fixed it

No, it did not. I tried ?action=purge multiple times since I first reported the
problem here and it never did anything. It seems my sandbox edit recreated the
thumbnail. But even if this is true, it does not solve the problem that caused
all the broken thumbnails.

Loading the image once showed the old one, second request with the purge and it loaded correctly? Please don't tell me what I did or didn't see.

Maybe so. It is reported as is working for new uploads, and I've tested it myself and it worked today.

Fixing stuff like this in retrospect isn't so easy, as there is no point purging everything

(In reply to comment #81)

I still see a few images with thumbnails that don't match, although new uploads
are fixed. Perhaps instead of waiting for new uploads or for someone in the
know to manually purge them as they come across them, there should be a mass
thumbnail purge and rebuild. (Since people and pages can use any random size
they want, it probably isn't even possible to rely on that.) Not all at once,
because I could see that bringing the servers to their knees and frustrating
everyone, but staged by hash group or as a low priority constant process over a
week or two.

mr.heat wrote:

(In reply to comment #87)

Loading the image once showed the old one, second request with the purge and it
loaded correctly?

I did exactly the same a few minutes ago and it did not fixed the problem. Why should it work when you do it but not when I do it? I think this was coincidence because I did my sandbox edit the same second. However, I don't see a reason why such an edit should make a difference. Maybe the developers know?

mr.heat wrote:

Here is another funny thing:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg?action=purge
and
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg
shows two different images. How is this possible?

(In reply to comment #88)

(In reply to comment #87)

Loading the image once showed the old one, second request with the purge and it
loaded correctly?

I did exactly the same a few minutes ago and it did not fixed the problem. Why
should it work when you do it but not when I do it? I think this was
coincidence because I did my sandbox edit the same second. However, I don't see
a reason why such an edit should make a difference. Maybe the developers know?

I am a developer. Maybe it is co-incidence, but it wouldn't be the first time I've done a purge for someone else and it worked, when it didn't for them...

(In reply to comment #89)

Here is another funny thing:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg?action=purge
and
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg
shows two different images. How is this possible?

Well you cannot just stick action=purge behind any url. You have to append it to the File description page,, and then clear your browser cache, to get rid of your local copies.

mr.heat wrote:

(In reply to comment #91)

Well you cannot just stick action=purge behind any url. You have to append it
to the File description page,, and then clear your browser cache, to get rid of
your local copies.

I did everything you said multiple times. Appending to the description page never did anything for me, as I wrote above. From my point of view it seems that purging is broken or disabled for my account or for regular users or something like this. Maybe only administrators or developers can use it? I have no idea. It feels like (and you obviously think) that I'm to stupid to use a web browser. This is not the case. I know how my Opera works, Shift+reloaded, cleared everything, even tried different machines and different browsers. I guarantee there is no client cache involved.

Appending to the image itself returned two different images as described above. Currently, this is fixed. I assume (since nobody seems to be able to tell us) that caching is done using the full URL and what I have seen was a new and an old cached image.

What I write about the sandbox was coincidence, but most probably the other way around: Reedy was able to purge the image in the second I edited the sandbox. What I don't understand (this is the Bug) is why purging does not work sometimes but does work if it's done by a developer? Should I open an other bug for this?

gavtroy wrote:

(In reply to comment #92)

From my point of view it seems
that purging is broken or disabled for my account or for regular users or
something like this. Maybe only administrators or developers can use it?

While there may be something along those lines happening, there have been times where purging worked for neither developer nor user. However I suppose that if the main issue is in fact now fixed, this could be a separate issue. Probably worth a new report if it happens again, but hopefully manual purging should no longer be necessary for images at all.

It feels like (and you obviously think) that I'm to stupid to use a
web browser. This is not the case.

Nobody was trying to insult you. Hartman seems to be saying that you can't purge thumbnails directly (if so - something I did not know), and in any case it is unreasonable to assume that because one can use the Internet, they know about lower level issues like caching.

I know how my Opera works, Shift+reloaded

In Opera you cannot bypass the cache in this way, as you can in Firefox. You can use Dragonfly to temporarily disable caching for a page, but I usually (like you) just clear my entire cache to be sure it's really gone.

saibotrash wrote:

(In reply to comment #82)

(In reply to comment #76)

(In reply to comment #72)

http://commons.wikimedia.org/wiki/File:Flies3.JPG is still broken.

Apparently works for me. Could you please explain what you expect to see that
you are not seeing?

Sorry, I forgot. In my 640px setting it still shows the wrong version:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/91/Flies3.JPG/640px-Flies3.JPG

(In reply to comment #89)

Here is another funny thing:
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg?action=purge
and
http://upload.wikimedia.org/wikipedia/commons/thumb/9/98/Satellite_Image_of_Ruegen.jpg/551px-Satellite_Image_of_Ruegen.jpg
shows two different images. How is this possible?

Work for me. Logged in and logged off. I did not notice any re-occurrences of this problem the last days. Would be nice of all damaged thumbnails could be purged automatically as the vast majority probably does not know how to purge... However - can be closed now.

I'm seeing this problem on images uploaded to meta. action=purge and clearing cache on the client side won't fix it.

Here's an example file:
http://meta.wikimedia.org/wiki/File:Cerrar.png
The first thumbnail should be black and the second should be red. For me they are both black, even if I purge the individual thumbs. This seems to be the case for any new versions of images on meta at the moment.

(In reply to comment #96)

Here's an example file:
http://meta.wikimedia.org/wiki/File:Cerrar.png
The first thumbnail should be black and the second should be red. For me they
are both black, even if I purge the individual thumbs. This seems to be the
case for any new versions of images on meta at the moment.

Purge the individual thumbs how? (We don't provide a user facing way of purging individual thumbnails afaik. Do you mean you manually did something on the server-side?).

For example:
http://upload.wikimedia.org/wikipedia/meta/thumb/6/69/Cerrar.png/85px-Cerrar.png?action=purge
That URL shows me the correct red thumbnail, but
http://upload.wikimedia.org/wikipedia/meta/thumb/6/69/Cerrar.png/85px-Cerrar.png
never does, it's always black. And yes, I've purged my cache on the client side.

Actually, if I put anything on the query string of the thumb, it renders correctly (red), but the normal URL is stuck on the black version. Can anyone else confirm (either with this image or a new one)? We need this fixed for our fundraising banner graphics.

[mid-air collision]
(In reply to comment #98)

For example:
http://upload.wikimedia.org/wikipedia/meta/thumb/6/69/Cerrar.png/85px-Cerrar.png?action=purge
That URL shows me the correct red thumbnail, but
http://upload.wikimedia.org/wikipedia/meta/thumb/6/69/Cerrar.png/85px-Cerrar.png
never does, it's always black. And yes, I've purged my cache on the client
side.

?action=purge doesn't actually purge a thumbnail (It does change the url to one
the squids probably haven't cached, but that is no different from say going to
http://upload.wikimedia.org/wikipedia/meta/thumb/6/69/Cerrar.png/85px-Cerrar.png?catInTheHat
) So it does confirm the issue is with squids not getting proper purges (or
getting purges and not actually purging, etc).

btw, for images that are displayed directly on the image page. Perhaps its some sort of race condition. If the order of events happen to be: User uploads new version of image, squid purge goes out, user gets redirected to the image page, (old version of) images get back into squid cache (from user loading image page), MediaWiki replaces the original image with the new version. the result would be consistent with some of the symptoms shown here (at least in terms of the version in the image history being wrong). Is it possible for the events to take place in that order?

(I was uploading a new version of an image on my testwiki, and noticed the loaded page had old version on it, which made me think of that)

(in case its useful to anyone else, I've been testing mediawiki's htcp support on my local install using the command: while true; do echo | nc -l -p 4827 -u -q 0|hd; done. (which sort of works, but needs to have a delay added to mw) So far the result seems to be images are purged a lot more often then I would expect them to be, but I need to double check to see if that's unexpected )

Something odd that I'm not sure is related or not: If $wgInternalServer does not contain a protocol (For example $wgInternalServer = 'foo';) Image purges end up having http: prefixed to that, while normal page purges don't. Not directly related (Since wikimedia should have a protocol in wgInternalServer i believe), but kind of odd, and suggests the code differs between normal purges and image purges in ways it perhaps shouldn't.

Well the squid cache finally cleared for my test case, but I uploaded a new version and the same bug happened:
http://meta.wikimedia.org/wiki/File:Cerrar.png
The current version should be blue, but both thumbnails show as red for me.

While this issue is important, it's not going to stop us from deploying 1.18 to the rest of the cluster. Removing as 1.18 blocker.

It is important that this get fixed before the fundraiser begins in November. All of the banner graphics are managed on meta.

russnelson wrote:

Earlier in this bug, somebody said that this also happened with the original image, not just the thumbnails. Is that still true?

I touched this code some weeks ago to solve a different problem. Unfortunately I didn't accidentally fix the bug. I don't know much about the Squid control, so I was careful not to touch that code (or the way it gets called). Maybe it's not getting called properly?

aaron.adrignola wrote:

Examine http://en.wikipedia.org/wiki/File:Tilted_Kilt_logo.jpg

Newest upload's resolution is recognized but the first upload has been stretched to match the larger size of the latest upload. The image is not large enough to be scaled down on the description page. Only in the article where it's used do I get the latest version if I change (in this case, add) the displayed size.

Also, OTRS email on the issue in ticket 2011100510006016.

That email addressed http://commons.wikimedia.org/wiki/File:Station_Hietzing_Otto_Wagner_Teppich.JPG

For that file, a new version was uploaded September 6 and the first upload is *still* showing (dark background).

saibotrash wrote:

(In reply to comment #107)
As noted in https://commons.wikimedia.org/wiki/Commons:Village_pump#Full_size_.28orig._version.29_of_files_does_not_update_on_upload_of_a_new_version I think this is a differnt bug. Opened: https://bugzilla.wikimedia.org/show_bug.cgi?id=31382

r99041 should help.

I'm assuming r99041 fixed this problem. Please reopen if this is still an issue.

(In reply to comment #110)

I'm assuming r99041 fixed this problem. Please reopen if this is still an
issue.

?

Unless I'm missing something, r99041 doesn't touch any code paths that are used on Wikimedia. (SquidPurgeClient does HTTP purges, we use HTCP purges as far as i know. At least the config files at noc.wikimedia.org suggest we do not use SquidPurgeClient for anything)

(In reply to comment #111)

(In reply to comment #110)

I'm assuming r99041 fixed this problem. Please reopen if this is still an
issue.

?

Unless I'm missing something, r99041 doesn't touch any code paths that are used
on Wikimedia. (SquidPurgeClient does HTTP purges, we use HTCP purges as far as
i know. At least the config files at noc.wikimedia.org suggest we do not use
SquidPurgeClient for anything)

See r99043.

The main thumb image at [[:Commons:File:Al Gore - Ohhh, you aren't going to get me on this one.jpg]] (at my default setting of 1024x768: http://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/Al_Gore_-_Ohhh%2C_you_aren%27t_going_to_get_me_on_this_one.jpg/1024px-Al_Gore_-_Ohhh%2C_you_aren%27t_going_to_get_me_on_this_one.jpg) still refuses to purge after uploading a new version. Other sizes do work though.

(In reply to comment #113)

The main thumb image at [[:Commons:File:Al Gore - Ohhh, you aren't going to get
me on this one.jpg]] (at my default setting of 1024x768:
http://upload.wikimedia.org/wikipedia/commons/thumb/5/5d/Al_Gore_-_Ohhh%2C_you_aren%27t_going_to_get_me_on_this_one.jpg/1024px-Al_Gore_-_Ohhh%2C_you_aren%27t_going_to_get_me_on_this_one.jpg)
still refuses to purge after uploading a new version. Other sizes do work
though.

Purging seemed to work for me to fix it (Note, that's purging the image description page, not the thumbnail).

I tried that to... in three different browsers! But the page kept showing the old (1024px) thumbnail. Only just now does the new image appear.

saibotrash wrote:

Non-refreshing thumb again... (Germany)
even trying some obscure direct thumb purge doesn't help:

wget https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=purge
--2011-12-04 14:53:35-- https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=purge
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 68441 (67K) [image/jpeg]
In »250px-PopulusNigra2.jpg?action=purge« speichern.

100%[======================================>] 68.441 212K/s in 0,3s

2011-12-04 14:53:36 (212 KB/s) - »250px-PopulusNigra2.jpg?action=purge« gespeichert [68441/68441]

Gets me the old version with blue sky. Same with wget https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg
Tried to purge the file desc page before....

https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/220px-PopulusNigra2.jpg instead is the new version with white sky.

Sidenote: the 220px version also was the old one until I purged the file desc page https://commons.wikimedia.org/w/index.php?title=File:PopulusNigra2.jpg&action=purge So purging after the upload of the new version did completely fail.

https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=purge

Note, purging the actual thumbnail doesn't do much (well I suppose it changes the url, which would cause the squid cache to possibly re-fetch it from the image servers). But in essence https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=purge isn't that different then going to https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=puppy as far as i know

For me (with doing the /etc/hosts trick so the request goes through the europe servers) https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg returns the newest version of the file (as expected). https://upload.wikimedia.org/wikipedia/commons/thumb/3/34/PopulusNigra2.jpg/250px-PopulusNigra2.jpg?action=purge returns the old version of the file (which isn't that unexpected since non-canonical thumb urls don't get purged, so earlier it returned the outdated image, and since then it just kept the outdated image cached for the url with ?action=purge appended).

So in conclusion, this looks like everything is working properly now (?)

saibotrash wrote:

Hmm, okay. Thanks for the comments.

One more needed? 221px works, 220px not (misrotated, old version).

wget https://upload.wikimedia.org/wikipedia/commons/thumb/e/e2/Denkmal_Robert_Oettel.JPG/220px-Denkmal_Robert_Oettel.JPG
--2011-12-08 03:20:19-- https://upload.wikimedia.org/wikipedia/commons/thumb/e/e2/Denkmal_Robert_Oettel.JPG/220px-Denkmal_Robert_Oettel.JPG
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 13098 (13K) [image/jpeg]
In »220px-Denkmal_Robert_Oettel.JPG« speichern.

100%[========================================================================>] 13.098 --.-K/s in 0,04s

2011-12-08 03:20:19 (296 KB/s) - »220px-Denkmal_Robert_Oettel.JPG« gespeichert [13098/13098]

+++++++++++++++++++++++++++++++++++++

wget https://upload.wikimedia.org/wikipedia/commons/thumb/e/e2/Denkmal_Robert_Oettel.JPG/221px-Denkmal_Robert_Oettel.JPG
--2011-12-08 03:20:26-- https://upload.wikimedia.org/wikipedia/commons/thumb/e/e2/Denkmal_Robert_Oettel.JPG/221px-Denkmal_Robert_Oettel.JPG
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 21888 (21K) [image/jpeg]
In »221px-Denkmal_Robert_Oettel.JPG« speichern.

100%[========================================================================>] 21.888 --.-K/s in 0,08s

2011-12-08 03:20:26 (278 KB/s) - »221px-Denkmal_Robert_Oettel.JPG« gespeichert [21888/21888]

saibotrash wrote:

Another old version (450px). This was directly after a purge - you notice that the 180px takes ten seconds to be scaled ([[:bugzilla:32432]]) but the 450px thumb (which is -with my settings- shown on the file page) is served instantaneously. → 450px isn't purged.

wget https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Pole_Vault_Sequence_2.jpg/450px-Pole_Vault_Sequence_2.jpg
--2011-12-08 18:08:23-- https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Pole_Vault_Sequence_2.jpg/450px-Pole_Vault_Sequence_2.jpg
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: 35925 (35K) [image/jpeg]
In »450px-Pole_Vault_Sequence_2.jpg.1« speichern.

100%[===============================================================================>] 35.925 --.-K/s in 0,1s

2011-12-08 18:08:23 (272 KB/s) - »450px-Pole_Vault_Sequence_2.jpg.1« gespeichert [35925/35925]

+++++++++++++++

wget https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Pole_Vault_Sequence_2.jpg/180px-Pole_Vault_Sequence_2.jpg
--2011-12-08 18:08:26-- https://upload.wikimedia.org/wikipedia/commons/thumb/8/8f/Pole_Vault_Sequence_2.jpg/180px-Pole_Vault_Sequence_2.jpg
Auflösen des Hostnamen »upload.wikimedia.org«.... 91.198.174.234
Verbindungsaufbau zu upload.wikimedia.org|91.198.174.234|:443... verbunden.
HTTP Anforderung gesendet, warte auf Antwort... 200 OK
Länge: nicht spezifiziert [image/jpeg]
In »180px-Pole_Vault_Sequence_2.jpg.1« speichern.

[ <=>                                                                            ] 12.149      --.-K/s   in 0,1s

2011-12-08 18:08:36 (83,5 KB/s) - »180px-Pole_Vault_Sequence_2.jpg.1« gespeichert [12149]

Reclosing. This bug was originally more specific.

Intermittent purge fails (usually wmf packet loss) should go on bug 31680.

alexander_berlin wrote:

Seems that thi problem again becomes critical. Several error reports on Commons:Village_pump.

Example:

http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/529px-Wappen_Landkreis_Aurich.svg.png

Connection:keep-alive
Content-Type:image/png
Date:Sun, 27 Jan 2013 14:17:28 GMT
Last-Modified:Sun, 27 Jan 2013 14:17:28 GMT
X-Cache:HIT from amssq53.esams.wikimedia.org
X-Cache:MISS from amssq52.esams.wikimedia.org
X-Cache-Lookup:HIT from amssq53.esams.wikimedia.org:3128
X-Cache-Lookup:MISS from amssq52.esams.wikimedia.org:80

http://upload.wikimedia.org/wikipedia/commons/thumb/c/c2/Wappen_Landkreis_Aurich.svg/529px-Wappen_Landkreis_Aurich.svg.png?1

Age:900
Connection:keep-alive
Content-Type:image/png
Date:Sun, 27 Jan 2013 14:17:28 GMT
Last-Modified:Sun, 27 Jan 2013 14:17:28 GMT
X-Cache:HIT from amssq53.esams.wikimedia.org
X-Cache:MISS from amssq61.esams.wikimedia.org
X-Cache-Lookup:HIT from amssq53.esams.wikimedia.org:3128
X-Cache-Lookup:MISS from amssq61.esams.wikimedia.org:80

Reclosing. This bug is due to the protocal relative issue. Please use something like bug 41130 instead (arguably that was also about a much more specific issue but it has become more broad)

Change 780890 had a related patch set uploaded (by Xqt; author: Xqt):

[pywikibot/core@master] Revert "[IMPR] use pg.XMLDumpPageGenerator in replace.py"

https://gerrit.wikimedia.org/r/780890