Page MenuHomePhabricator

Some squid caches serving stale image and thumbnail versions
Closed, ResolvedPublic

Description

Some but not all of the squid caches seem to be serving stale versions of both
thumbnails and original images. Purging or reuploading does not seem to help.
I have personally only encountered this problem with SVG images so far, but I
expect it probably affects all images.

See for example [[Wikipedia talk:WikiProject Mathematics/Graphics#Blank thumb in
svg?]]: the image
http://upload.wikimedia.org/wikipedia/en/thumb/0/05/Paley13.svg/200px-Paley13.svg.png
shows up for me as an old, blank version, while the 201-pixel thumb shows up
just fine. The HTTP response headers I get for the affected thumbnail are:

Connection: close
Date: Sat, 04 Nov 2006 20:12:44 GMT
Via: 1.0 sq6.pmtpa.wmnet:80 (squid/2.6.STABLE3), 1.0 sq12.wikimedia.org:80
(squid/2.6.STABLE4), 1.0 ragweed.knams.wikimedia.org:80 (squid/2.6.STABLE5), 1.0
vandale.knams.wikimedia.org:80 (squid/2.6.STABLE3)
Age: 2618
Server: lighttpd/1.4.13
Content-Length: 271
Content-Type: image/png
Client-Date: Sat, 04 Nov 2006 20:55:41 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq6.pmtpa.wmnet
X-Cache: MISS from sq12.wikimedia.org
X-Cache: HIT from ragweed.knams.wikimedia.org
X-Cache: MISS from vandale.knams.wikimedia.org
X-Cache-Lookup: HIT from sq6.pmtpa.wmnet:80
X-Cache-Lookup: MISS from sq12.wikimedia.org:80
X-Cache-Lookup: HIT from ragweed.knams.wikimedia.org:80
X-Cache-Lookup: MISS from vandale.knams.wikimedia.org:80
X-Powered-By: PHP/5.1.4

whereas the headers for the seemingly unaffected 201-pixel thumbnail are:

Connection: close
Date: Sat, 04 Nov 2006 19:59:07 GMT
Via: 1.0 sq7.pmtpa.wmnet:80 (squid/2.6.STABLE3), 1.0 sq12.wikimedia.org:80
(squid/2.6.STABLE4), 1.0 ragweed.knams.wikimedia.org:80 (squid/2.6.STABLE5)
Age: 3452
Server: lighttpd/1.4.13
Content-Type: image/png
Client-Date: Sat, 04 Nov 2006 20:55:57 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq7.pmtpa.wmnet
X-Cache: MISS from sq12.wikimedia.org
X-Cache: HIT from ragweed.knams.wikimedia.org
X-Cache-Lookup: HIT from sq7.pmtpa.wmnet:80
X-Cache-Lookup: MISS from sq12.wikimedia.org:80
X-Cache-Lookup: HIT from ragweed.knams.wikimedia.org:80
X-Powered-By: PHP/5.1.4

Of course, it's possible the both thumbs are affected, but the latter one just
happens to have been generated after the original image was fixed, and thus does
not appear blank.


Version: unspecified
Severity: normal
URL: http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Mathematics/Graphics#Blank_thumb_in_svg.3F

Details

Reference
bz7807

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 9:28 PM
bzimport set Reference to bz7807.
bzimport added a subscriber: Unknown Object (MLST).

http://upload.wikimedia.org/wikipedia/en/thumb/8/8a/RobertJohson.jpg/250px-RobertJohson.jpg
is also affected, serving a thumbnail of a version that's been deleted since 15
October! See original complaint at
http://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Distorted_image

I've tried purging, reverting, deleting, undeleting and various combinations
thereof without results. Manually issuing HTTP "Cache-Control: no-cache"
request headers doesn't help either. For what it's worth, the response headers
I'm getting include:

GET
http://upload.wikimedia.org/wikipedia/en/thumb/8/8a/RobertJohson.jpg/250px-RobertJohson.jpg
--> 200 OK
Connection: close
Date: Sun, 12 Nov 2006 20:53:06 GMT
Via: 1.0 sq14.wikimedia.org:80 (squid/2.6.STABLE5), 1.0 sq13.wikimedia.org:80
(squid/2.6.STABLE5), 1.0 mayflower.knams.wikimedia.org:80 (squid/2.6.STABLE3),
1.0 ragweed.knams.wikimedia.org:80 (squid/2.6.STABLE5)
Accept-Ranges: bytes
Age: 909
ETag: "-299518909"
Server: lighttpd/1.4.13
Content-Length: 31440
Content-Type: image/jpeg
Last-Modified: Wed, 28 Jun 2006 23:25:07 GMT
Client-Date: Sun, 12 Nov 2006 21:08:16 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq14.wikimedia.org
X-Cache: MISS from sq13.wikimedia.org
X-Cache: HIT from mayflower.knams.wikimedia.org
X-Cache: MISS from ragweed.knams.wikimedia.org
X-Cache-Lookup: HIT from sq14.wikimedia.org:80
X-Cache-Lookup: MISS from sq13.wikimedia.org:80
X-Cache-Lookup: HIT from mayflower.knams.wikimedia.org:80
X-Cache-Lookup: MISS from ragweed.knams.wikimedia.org:80

with some variants including

GET
http://upload.wikimedia.org/wikipedia/en/thumb/8/8a/RobertJohson.jpg/250px-RobertJohson.jpg
--> 200 OK
Connection: close
Date: Sun, 12 Nov 2006 20:53:06 GMT
Via: 1.0 sq14.wikimedia.org:80 (squid/2.6.STABLE5), 1.0 sq13.wikimedia.org:80
(squid/2.6.STABLE5), 1.0 mayflower.knams.wikimedia.org:80 (squid/2.6.STABLE3),
1.0 vandale.knams.wikimedia.org:80 (squid/2.6.STABLE3)
Accept-Ranges: bytes
Age: 906
ETag: "-299518909"
Server: lighttpd/1.4.13
Content-Length: 31440
Content-Type: image/jpeg
Last-Modified: Wed, 28 Jun 2006 23:25:07 GMT
Client-Date: Sun, 12 Nov 2006 21:08:12 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq14.wikimedia.org
X-Cache: MISS from sq13.wikimedia.org
X-Cache: HIT from mayflower.knams.wikimedia.org
X-Cache: MISS from vandale.knams.wikimedia.org
X-Cache-Lookup: HIT from sq14.wikimedia.org:80
X-Cache-Lookup: MISS from sq13.wikimedia.org:80
X-Cache-Lookup: HIT from mayflower.knams.wikimedia.org:80
X-Cache-Lookup: MISS from vandale.knams.wikimedia.org:80

and

GET
http://upload.wikimedia.org/wikipedia/en/thumb/8/8a/RobertJohson.jpg/250px-RobertJohson.jpg
--> 200 OK
Connection: close
Date: Sun, 12 Nov 2006 20:53:06 GMT
Via: 1.0 sq14.wikimedia.org:80 (squid/2.6.STABLE5), 1.0 sq13.wikimedia.org:80
(squid/2.6.STABLE5), 1.0 mayflower.knams.wikimedia.org:80 (squid/2.6.STABLE3),
1.0 sage.knams.wikimedia.org:80 (squid/2.6.STABLE3)
Accept-Ranges: bytes
Age: 895
ETag: "-299518909"
Server: lighttpd/1.4.13
Content-Length: 31440
Content-Type: image/jpeg
Last-Modified: Wed, 28 Jun 2006 23:25:07 GMT
Client-Date: Sun, 12 Nov 2006 21:08:02 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq14.wikimedia.org
X-Cache: MISS from sq13.wikimedia.org
X-Cache: HIT from mayflower.knams.wikimedia.org
X-Cache: MISS from sage.knams.wikimedia.org
X-Cache-Lookup: HIT from sq14.wikimedia.org:80
X-Cache-Lookup: MISS from sq13.wikimedia.org:80
X-Cache-Lookup: HIT from mayflower.knams.wikimedia.org:80
X-Cache-Lookup: MISS from sage.knams.wikimedia.org:80

and

GET
http://upload.wikimedia.org/wikipedia/en/thumb/8/8a/RobertJohson.jpg/250px-RobertJohson.jpg
--> 200 OK
Connection: close
Date: Sun, 12 Nov 2006 20:53:06 GMT
Via: 1.0 sq14.wikimedia.org:80 (squid/2.6.STABLE5), 1.0 sq13.wikimedia.org:80
(squid/2.6.STABLE5), 1.0 mayflower.knams.wikimedia.org:80 (squid/2.6.STABLE3)
Accept-Ranges: bytes
Age: 518
ETag: "-299518909"
Server: lighttpd/1.4.13
Content-Length: 31440
Content-Type: image/jpeg
Last-Modified: Wed, 28 Jun 2006 23:25:07 GMT
Client-Date: Sun, 12 Nov 2006 21:01:44 GMT
Client-Peer: 145.97.39.156:80
Client-Response-Num: 1
X-Cache: HIT from sq14.wikimedia.org
X-Cache: MISS from sq13.wikimedia.org
X-Cache: HIT from mayflower.knams.wikimedia.org
X-Cache-Lookup: HIT from sq14.wikimedia.org:80
X-Cache-Lookup: MISS from sq13.wikimedia.org:80
X-Cache-Lookup: HIT from mayflower.knams.wikimedia.org:80

mail wrote:

Hey there,

i'm not sure if I should've opened a new report, but I think your observations only prove mine:

I was having a (deeper) look at your includes/SquidUpdate.php to get a clue of how to deal with my own squids.

Here's the broken code: http://phpfi.com/267445 (relevant comments prefixed with "globe:")
Here's a simple test proving the bug: http://phpfi.com/267447

The programmer's supplying an index, where he definately shouldn't have.
$sockets[$so+1] should be $sockets[] and everything is fine :)

I've never used mediaWiki, so i've never actually run the code. But from reading (and the little test) i'd say it's not a feature ;)

It seems that the bug is in both, the branch and trunk:
http://svn.wikimedia.org/viewvc/mediawiki/branches/REL1_11/phase3/includes/SquidUpdate.php?view=markup
http://svn.wikimedia.org/viewvc/mediawiki/trunk/phase3/includes/SquidUpdate.php?view=markup

keep it up, cheers,
Rod

Patch to includes/SquidUpdate.php (not committing this myself, since I can't test it)

Wow, that indeed looks broken. And it's been that way since r13845! Here's a simple patch that should fix it; the only reason I'm not committing it myself is that I can't test it, not having any squids to purge.

Attached:

Change sounds reasonable -- applied on r36351.

This code path hasn't been exercised on our servers in a long long time (we'd switched to HTCP and multicast stuff long ago).