sumanah wrote:

Thanks for the bug report!

(Whoops, sorry, accidentally marked this as a deployment blocker because I missed what wiki it was on.)

Confirming. I think I've hit the same issue:

• https://commons.wikimedia.org/w/thumb.php?f=Broom%20icon.svg&w=22 is a 404
• https://commons.wikimedia.org/w/thumb.php?f=Broom%20icon.svg&w=23 works

According to eval.php:

$be = FileBackendGroup::singleton()->get( 'local-swift' );

$root = $be->getRootStoragePath() . '/local-thumb';

$obj = '3/34/Flag_of_El_Salvador.svg/25px-Flag_of_El_Salvador.svg.png';

var_dump( $be->getFileStat( array( 'src' => "$root/$obj" ) ) );

bool(false)

The bad files are not in swift. So I guess that file is just in squid. The error message is apparently a swift 404 error message. So it some point swift have a 404 and either:
a) squid cached it wrongly, or
b) the response had the wrong headers, so squid didn't know it was a 404

• Bug 40980 has been marked as a duplicate of this bug. ***

• Bug 41130 has been marked as a duplicate of this bug. ***

• Bug 41088 has been marked as a duplicate of this bug. ***

• Bug 41174 has been marked as a duplicate of this bug. ***

Today, another thumbnail got broken:

https://pt.wiktionary.org/wiki/Predefini%C3%A7%C3%A3o:-da-

Is 41174 really a duplicate? It is not just the thumbnails that are broken. The file cannot be seen at all.

(In reply to comment #9)

Is 41174 really a duplicate? It is not just the thumbnails that are broken.
The file cannot be seen at all.

Probably not, I'd reopen that one.

(In reply to comment #3)

According to eval.php:

$be = FileBackendGroup::singleton()->get( 'local-swift' );

$root = $be->getRootStoragePath() . '/local-thumb';

$obj = '3/34/Flag_of_El_Salvador.svg/25px-Flag_of_El_Salvador.svg.png';

var_dump( $be->getFileStat( array( 'src' => "$root/$obj" ) ) );

bool(false)

According to Asher, some of the cached stat entries are bogus. This causes thumb.php to send a 200 status, length header, and so on and then try to stream the body of a file that does not exist (which streams the literal swift error html). This gets cached by the squids (apparently they ignore the content-length mismatch).

There are few stat cache update failures showing up in the logs. I've made some code cleanups in https://gerrit.wikimedia.org/r/#/c/28559/ to improve cache key normalization, but I have some doubts as to whether that is at play here.

Also see https://gerrit.wikimedia.org/r/#/c/28614/.

That template looks fine now. Are there some other examples?

(In reply to comment #13)

That template looks fine now. Are there some other examples?

I clicked through the URLs above and all seemed to work properly. Unless someone can come up with a broken test case in the next day or so, I think this bug can safely be marked resolved as fixed.

sumanah wrote:

Dropping this to "high" after conversation with Rob Lanphier -- bug 41130 is the highest-priority thumbnail-related bug right now.

sumanah wrote:

Tried to reproduce problem and could not; marking as fixed because I presume the changesets fixed the problem.