Page MenuHomePhabricator

Purging for all sites but eqiad is broken
Closed, ResolvedPublic

Description

+++ This bug was initially created as a clone of Bug #44508 comment 13 +++

  • Comment 13 by Marco 2013-03-19 23:54:38 UTC ---

New instance:
https://upload.wikimedia.org/wikipedia/commons/thumb/a/a5/PlacaenmonumentoaMarconi.jpg/120px-PlacaenmonumentoaMarconi.jpg

"Broken" response from esams:

HTTP request sent, awaiting response...

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Tue, 19 Mar 2013 23:35:35 GMT
Content-Type: image/jpeg
Content-Length: 3637
Connection: keep-alive
X-Object-Meta-Sha1base36: q311dfj910y0loxw51q2fm79rw8j32o
Content-Disposition: inline;filename*=UTF-8''PlacaenmonumentoaMarconi.jpg
X-Timestamp: 1363609261.89105
Accept-Ranges: bytes
Access-Control-Allow-Origin: *
Last-Modified: Mon, 18 Mar 2013 12:21:01 GMT
ETag: b737ec5a44b9f7975b69facfde655da0
X-Cache: HIT from sq82.wikimedia.org
X-Cache-Lookup: HIT from sq82.wikimedia.org:3128
Age: 41471
X-Cache: HIT from amssq59.esams.wikimedia.org
X-Cache-Lookup: HIT from amssq59.esams.wikimedia.org:3128
X-Cache: MISS from knsq19.esams.wikimedia.org
X-Cache-Lookup: MISS from knsq19.esams.wikimedia.org:80
Via: 1.0 sq82.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0

amssq59.esams.wikimedia.org:3128 (squid/2.7.STABLE9), 1.0
knsq19.esams.wikimedia.org:80 (squid/2.7.STABLE9)
Length: 3637 (3.6K) [image/jpeg]

"Valid" response via Varnish:

HTTP request sent, awaiting response...

HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Tue, 19 Mar 2013 23:35:24 GMT
Content-Type: image/jpeg
Content-Length: 3994
Connection: keep-alive
X-Object-Meta-Sha1base36: qtiacup2waiwupgx7sma7puadwpqeig
Content-Disposition: inline;filename*=UTF-8''PlacaenmonumentoaMarconi.jpg
Last-Modified: Tue, 19 Mar 2013 23:24:09 GMT
Etag: 92f154cb562f20cf648199f1bd88c464
X-Timestamp: 1363735449.88825
X-Varnish: 2316312654, 2904564637, 670145209
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 0
X-Cache: cp1028 miss (0), cp3009 miss (0), cp3009 frontend miss (0)
Access-Control-Allow-Origin: *

Length: 3994 (3.9K) [image/jpeg]

Please confirm. (I already purged the file page)


Version: wmf-deployment
Severity: critical

Details

Reference
bz46350

Related Objects

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 1:32 AM
bzimport set Reference to bz46350.

$:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg'
--2013-03-20 10:09:14-- http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg
Resolving upload-lb.eqiad.wikimedia.org... 208.80.154.235, 2620:0:861:ed1a::b
Connecting to upload-lb.eqiad.wikimedia.org|208.80.154.235|:80... connected.
HTTP request sent, awaiting response...

HTTP/1.1 200 OK
X-Object-Meta-Sha1base36: quwkq14ur1ogdvm9amhhkky19and4wy
Last-Modified: Tue, 19 Mar 2013 14:55:49 GMT
Etag: 5627db7e9874a4cf2ff47acddd548c8a
X-Timestamp: 1363704949.43621
Content-Type: image/jpeg
X-Varnish: 684751357, 3232235220
Via: 1.1 varnish, 1.1 varnish
Content-Length: 2112147
Accept-Ranges: bytes
Date: Wed, 20 Mar 2013 09:09:16 GMT
Age: 1
Connection: keep-alive
X-Cache: cp1027 miss (0), cp1035 frontend miss (0)
Access-Control-Allow-Origin: *

Length: 2112147 (2.0M) [image/jpeg]

2013-03-20 10:09:21 (449 KB/s) - “PlacaenmonumentoaMarconi.jpg” saved [2112147/2112147]

$:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg'
--2013-03-20 10:09:27-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/a/a5/PlacaenmonumentoaMarconi.jpg
Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b
Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected.
HTTP request sent, awaiting response...

HTTP/1.0 200 OK
X-Object-Meta-Sha1base36: 8xysnw9wchzvrlb71wizuqw9c86snkp
Last-Modified: Mon, 18 Mar 2013 12:20:53 GMT
ETag: 56d70ec74402692887b1ca918fff4bd9
X-Timestamp: 1363609253.81912
Accept-Ranges: bytes
Content-Length: 1333860
Content-Type: image/jpeg
Access-Control-Allow-Origin: *
Date: Tue, 19 Mar 2013 11:22:20 GMT
Age: 78427
X-Cache: HIT from sq53.wikimedia.org
X-Cache-Lookup: HIT from sq53.wikimedia.org:3128
X-Cache: MISS from knsq21.esams.wikimedia.org
X-Cache-Lookup: MISS from knsq21.esams.wikimedia.org:3128
X-Cache: MISS from amssq49.esams.wikimedia.org
X-Cache-Lookup: MISS from amssq49.esams.wikimedia.org:80
Connection: keep-alive

Length: rOPUP1333860d7e29 (1.3M) [image/jpeg]

2013-03-20 10:09:30 (594 KB/s) - “PlacaenmonumentoaMarconi.jpg.1” saved [rOPUP1333860d7e29/rOPUP1333860d7e29]

$:andre\>

Same for https://upload.wikimedia.org/wikipedia/commons/0/04/Wikipedia-logo-v2-uz.png (just reuploaded):

Last-Modified: Thu, 12 Jul 2012 20:11:39 GMT
[...]
X-Cache-Lookup: HIT from sq56.wikimedia.org:3128
Age: 9332151

But varnish is equally wrong:

Last-Modified: Thu, 12 Jul 2012 20:11:39 GMT
Etag: dc298c819efca185c0200435e68428a6
X-Timestamp: 1342123899.97366
X-Varnish: 2769044483 1786526384, 2956910772 864530478, 824447606 824386074
Via: 1.1 varnish, 1.1 varnish, 1.1 varnish
Accept-Ranges: bytes
Age: 2665367
X-Cache: cp1024 hit (8978), cp3009 hit (3621), cp3009 frontend hit (2)

Eqiad is correct:

$ wget -S --header 'host: upload.wikimedia.org' 'http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/0/04/Wikipedia-logo-v2-uz.png'
[...]

Last-Modified: Wed, 20 Mar 2013 01:12:41 GMT
Etag: 66bb874273a5120bb99c83c7827151d8
X-Timestamp: 1363741961.71944
Content-Type: image/png
X-Varnish: 1993846029, 2883426698
Via: 1.1 varnish, 1.1 varnish
Content-Length: 18240
Accept-Ranges: bytes
Date: Wed, 20 Mar 2013 14:59:35 GMT
Age: 0
Connection: keep-alive
X-Cache: cp1024 miss (0), cp1021 frontend miss (0)

15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)

Nemo was quicker (thanks!) and I had a mid-air collision when coming back. :)

https://wikitech.wikimedia.org/wiki/Server_admin_log says for March 20:

  • 15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)