Page MenuHomePhabricator

Out of memory for some GIF thumbnail versions of animated GIF images on Commons (exceeding the 50 MP limit?)
Closed, DeclinedPublic

Description

Details

Reference
bz61711

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 2:58 AM
bzimport set Reference to bz61711.
bzimport added a subscriber: Unknown Object (MLST).

137 = OOM

In theory we are supposed to render only the first frame if image big enough to cause this. Maybe settings need to be tweaked.

The joke is that a error message ("Note: Due to technical limitations, thumbnails of high resolution GIF images such as this one will not be animated." - maybe from Bug 23063) and a static thumb only appears if the GIF is over 50MP! But the limit is really 12,5MP! (It was
a short time 50MP at 2013 https://commons.wikimedia.org/w/index.php?title=Special:Contributions&offset=201305261339&limit=500&tagfilter=&contribs=user&target=Slick)

Sorry my correction, the limitation is not somehow 12.5MP it is above ~23 MP (and seems for sure as of 26 MP, dependence by the frame size I've not tested). I've tested some in [[Category:Animated GIF files affected by MediaWiki restrictions]].

(In reply to PRO from comment #3)

Sorry my correction, the limitation is not somehow 12.5MP it is above ~23 MP
(and seems for sure as of 26 MP, dependence by the frame size I've not
tested). I've tested some in [[Category:Animated GIF files affected by
MediaWiki restrictions]].

Note, the megapixel limit and the memory limit are separate limits.

The amount of memory a file takes does not precisely correspond to a specific number of megapixels.

mcdevitd wrote:

This may be a separate issue, but that error message is particularly unhelpful. When you hit the MP limit, at least you get a message for that, in theory.

(In reply to Dominic from comment #5)

This may be a separate issue, but that error message is particularly
unhelpful. When you hit the MP limit, at least you get a message for that,
in theory.

Would actually writing out "out of memory" actually be anymore helpful? I cant imagine that that means much to average user.

(For reference 137 comes from 9 + 128, which is bash's way of saying process recieved SIGKILL (signal number 9), which gets caused by exceeding the linux cgroup memory limit)

mcdevitd wrote:

It may not help the average user experience (though if they are hitting this error, I am nit sure they can be helped), but I was just thinking how frustrated I get at vague error messages, which I can't look up to see what the cause is or if it is a known issue.

(In reply to Dominic from comment #7)

Would actually writing out "out of memory" actually be anymore helpful? I
cant imagine that that means much to average user.

Maybe not the average user but the uploader may think his GIF file is broken in some way other than being too large.

The error message for TIF seems to be more helpful. C.f: https://upload.wikimedia.org/wikipedia/commons/thumb/2/2f/19-N-9283-1720-42-K-Launching_BB-62_waterborne.tiff/lossy-page1-599px-19-N-9283-1720-42-K-Launching_BB-62_waterborne.tiff.jpg

Different error: oom can be triggered by different but related conditions. Tiff files can also be oom

Hmm, I was recently testing image magick, and it seems to have better memory performance then older versions, with me being able to scale a bunch of images locally that don't work on cluster.

Created attachment 15694
mem limit values I tested image magick with

So in my tests, newer versions of image magick seem to sometimes be able to recover from ulimit -v, and use less memory if they run out (at least that's what I think is happening). That doesn't seem to happen with cgroups.

There are various environmental variables you can set to tell image magick its memory limits. When I tried setting them on newer image magick (6.8.9-3), it caused the animated gif I was testing to be rendered without exceeding memory limit (where previously cgroups killed it for OOM) [See attached patch for values I was testing with].

However, testing on 6.6.0-4, this didn't seem to have a lot of affect. WMF is currently using "ImageMagick 6.6.9-7 2014-03-06 Q16", so I'm unsure what affect it would have there (Although rumour has it they are updating some applications in the next 2 months).

The values I was testing with were mostly pulled out of my hat/trial and error, so also unclear if they are good values. Anyways, more research needs to be done to see if this is practical/a good idea [which is why I added my patch here, not in gerrit].

The gif file I was testing with is https://upload.wikimedia.org/wikipedia/commons/archive/6/64/20140614071837%21BULGARI_-_a_gentleman%27s_Carbongold_Hong_Kong_chronograph_wrist_watch._Fellows-1438-2-A.gif . I was using cgroups with $wgMaxShellMemory set to 400*1024*1024

see also: http://www.imagemagick.org/script/command-line-options.php#limit

Attached:

  • Bug 70296 has been marked as a duplicate of this bug. ***
  • Bug 47409 has been marked as a duplicate of this bug. ***
Aklapper changed the task status from Open to Stalled.Nov 16 2017, 1:02 AM

All links provided in this task do not expose any problems nowadays. Is this issue resolved?
(Or if there are still problems, could you provide full links and explain what you see and what you'd expect to see where?)
Thanks.

All links provided in this task do not expose any problems nowadays. Is this issue resolved?

No reply, hence assuming so.

BFG added a subscriber: BFG.

This is still the bug which is associated with not being able to generate thumnails for larger Gifs. If this is resolved there exists another bug. See https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif for a recent example.

In T63711#4148021, @BFG wrote:

This is still the bug which is associated with not being able to generate thumnails for larger Gifs. If this is resolved there exists another bug. See https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif for a recent example.

@BFG: Where exactly to see a problem? That 20MB GIF is displayed for me in Firefox 59 when being logged in.

I'm not totally sure how this is supposed to work, but it's my understanding that preview versions should automatically be generated, which they are not. Please have a look at the usage at https://en.wikipedia.org/wiki/Unit_circle the included file there is not animated at all. If this is working as intended the commons category probably needs to be removed.

Aklapper renamed this task from out of memory for some animated GIF images on Commons to Out of memory for some GIF thumbnail versions of animated GIF images on Commons.Apr 22 2018, 9:41 PM

Ah, so it's the thumbnails themselves embedded. Thanks for clarifying!

(All the three links in the task description and the link in https://phabricator.wikimedia.org/T63711#661563 work for me without a problem.)

What's the difference of this task nowadays to T25063? Isn't it both about the 50MP limit?

@Aklapper from my point of view this and T25063 seems to refer to the same problem. This bug is however the stated reason in https://commons.wikimedia.org/wiki/Commons:Maximum_file_size for why you cannot upload large GIFs. If one is closed, the reference in commons should be updated to reflect this.

Aklapper renamed this task from Out of memory for some GIF thumbnail versions of animated GIF images on Commons to Out of memory for some GIF thumbnail versions of animated GIF images on Commons (exceeding the 50 MP limit?).Mar 2 2019, 4:59 PM
Aklapper raised the priority of this task from High to Needs Triage.
Aklapper edited projects, added Thumbor; removed WMF-General-or-Unknown.

Checking the links posted here after T63711#3765377:

However, some animated GIF thumbnails on https://commons.wikimedia.org/wiki/Category:Animated_GIF_files_exceeding_the_50_MP_limit indeed are not animated. (And I don't see how this is high priority.)

@Aklapper Why are you mentioning https://en.wikipedia.org/wiki/Unit_circle#/media/File:2pi-unrolled.gif ? - At 9.68 MiPixels that is totally irrelevant to this bug.

As for https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif - neither the thumbnail on https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif nor the embedded version on https://en.wikipedia.org/wiki/Unit_circle is working correctly for me. To make sure I've verified this on the following combinations - all with the same result:
Windows 7:

  • Firefox
  • Chrome
  • Internet Explorer
  • Opera

Windows 8:

  • Firefox
  • Chrome

OS X:

  • Safari
  • Firefox

Linux:

  • Chrome

I've also downloaded the thumbnail and verified that there's only 1 frame available.

Could you please recheck this on clean equipment? There has to be something special about your setup since all these combinations fail, and still you get correctly animated thumbnails.

In T63711#4995757, @BFG wrote:

@Aklapper Why are you mentioning https://en.wikipedia.org/wiki/Unit_circle#/media/File:2pi-unrolled.gif ? - At 9.68 MiPixels that is totally irrelevant to this bug.

As no GIF was explicitly mentioned in T63711#4148477 I obviously checked the wrong GIF in that article...meh.

As for https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif - neither the thumbnail on https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif nor the embedded version on https://en.wikipedia.org/wiki/Unit_circle is working correctly for me.

You seem to refer to the thumbnail under "File History" instead of what can be seen at the top of the page.
Indeed, I can confirm that neither that thumbnail at https://upload.wikimedia.org/wikipedia/commons/thumb/9/9c/Unitycircle-complex.gif/120px-Unitycircle-complex.gif nor https://upload.wikimedia.org/wikipedia/commons/thumb/9/9c/Unitycircle-complex.gif/220px-Unitycircle-complex.gif in https://en.wikipedia.org/wiki/Unit_circle are animated. Sorry for the misunderstanding!

Could you please recheck this on clean equipment?

I don't think I have dirty equipment. :P

In T63711#4995757, @BFG wrote:

You seem to refer to the thumbnail under "File History" instead of what can be seen at the top of the page.
Indeed, I can confirm that neither that thumbnail at https://upload.wikimedia.org/wikipedia/commons/thumb/9/9c/Unitycircle-complex.gif/120px-Unitycircle-complex.gif nor https://upload.wikimedia.org/wikipedia/commons/thumb/9/9c/Unitycircle-complex.gif/220px-Unitycircle-complex.gif in https://en.wikipedia.org/wiki/Unit_circle are animated. Sorry for the misunderstanding!

I might have been unclear here as well. The thumbnail in the history is the only one which appears, but it's a proxy for whether or not the thumbnail generation works.

Could you please recheck this on clean equipment?

I don't think I have dirty equipment. :P

I thought you might have some kind of dev-tools or something installed. For instance there is a setting on English Wikipedia which limits the thumbnail size, and I've got MathML enabled, whereI used to have a javascript to enable before it became a setting.

I've put in my todo list to make a bot for commons which identifies all the files with this problem. I'll try to get around to doing it before the summer.

For clarity: The thumbnail system on Wikimedia sites has been replaced since this task was first filed, and GIF thumbnailing is now handled by Thumbor. In our Thumbor configuration, MAX_ANIMATED_GIF_AREA = 100000000 (100 MP) and is calculated with length * width * frame_count. Files with a total area greater than their maximum are not animated in thumbnails to prevent huge thumbnails. This is not a bug, it is intended behavior.

Is this task about GIFs < 100 MP that fail to animate, Raising the 100 MP limit, or something else?

Aklapper changed the task status from Open to Stalled.May 25 2020, 3:34 PM

Is this task about GIFs < 100 MP that fail to animate, Raising the 100 MP limit, or something else?

Setting task status to stalled per last question. Once answered, please set the status of this task back to "Open" via the Add Action...Change Status dropdown. Thanks!

@Krenair: Could you please answer the last comment? Thanks in advance!

Unfortunately closing this Phabricator task as no further information has been provided.

@Krenair: After you have provided the information asked for and if this still happens, please set the status of this task back to "Open" via the Add Action...Change Status dropdown. Thanks!

I didn't see this until now anyway the bug still exists, see the file history of https://commons.wikimedia.org/wiki/File:Unitycircle-complex.gif

Although, I expect the reasons may be that the 100MP limit is not strictly a 100MP limit, but something else. Anyway, this seems to already be addressed in T64463 so I'm happy to let this be closed as duplicate.

Not a bug.

$ identify 20200317143113!Unitycircle-complex.gif | wc -l                 
1001

500 * 495 * 1001 = 247,747,500 px

$ units
Currency exchange rates from FloatRates (USD base) on 2020-10-03 
3677 units, 109 prefixes, 114 nonlinear units

You have: 247747500 P
You want: MP
	* 247.7475
	/ 0.0040363677
You have: 

248 MP > 100 MP. System is working as intended.