Page MenuHomePhabricator

OOM when rendering SVGs
Closed, InvalidPublic

Description

Author: david

Description:
On our wiki, as well as on many others ([0],[1],[2],..) I've come across this error report:

SvgHandler::rasterize: convert -background white -thumbnail 64x64\! '/var/www/hackerspace.lu/www/w/images/4/4f/LightOff.svg' PNG:'/var/www/hackerspace.lu/www/w/images/thumb/4/4f/LightOff.svg/64px-LightOff.svg.png' 2>&1
wfShellExec: '/var/www/hackerspace.lu/www/w/bin/ulimit4.sh' 180 102400 102400 'convert -background white -thumbnail 64x64\! '\''/var/www/hackerspace.lu/www/w/images/4/4f/LightOff.svg'\'' PNG:'\''/var/www/hackerspace.lu/www/w/images/thumb/4/4f/LightOff.svg/64px-LightOff.svg.png'\'' 2>&1'
thumbnail failed on syndi: error 1 "sh: wmf2eps: command not found
convert: Delegate failed `"wmf2eps" -o "%o" "%i"'.
convert: unable to open image `/tmp/magick-XXjbtgDb': No such file or directory.
convert: unable to load module `/usr/lib/ImageMagick-6.3.7/modules-Q16/coders/svg.la': file not found.
convert: UnableToOpenBlob `/tmp/magick-XXjbtgDb': No such file or directory.
convert: missing an image filename `PNG:/var/www/hackerspace.lu/www/w/images/thumb/4/4f/LightOff.svg/64px-LightOff.svg.png'." from "convert -background white -thumbnail 64x64\! '/var/www/hackerspace.lu/www/w/images/4/4f/LightOff.svg' PNG:'/var/www/hackerspace.lu/www/w/images/thumb/4/4f/LightOff.svg/64px-LightOff.svg.png' 2>&1"

Note that this is independent of the wiki version. The real problem is the last message coming from convert. The "unable to load module" message is incorrect, because the svg.la file is present and the path is correct.

I've tried executing the same command on the shell and it works flawlessly when removing all of the single quotes. I think that convert is somehow interpreting both arguments, source file and output file as a single argument due to some erroneous escaping. (Removing one pair of "\'" from every argument works as well)

I've not dug into the mw source for his though.


Version: unspecified
Severity: normal
OS: Linux
URL: https://www.hackerspace.lu/w/thumb.php?f=LightOff.svg&width=64

Details

Reference
bz25403

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 11:23 PM
bzimport set Reference to bz25403.
bzimport added a subscriber: Unknown Object (MLST).

Bryan.TongMinh wrote:

Can you try to see if the (unreleased) 1.16.1 from SVN solves your problem? 1.16.1 has quite a lot fixes regarding IM escaping.

david wrote:

I'm trying out 1.16.2 from svn now. I'm having issues getting MW to use my $wgCustomConvertCommand right now. I'll have to look into this more thoroughly and will get back to you asap.

david wrote:

Well I can confirm that the $wgCustomConvertCommand = "gm convert %s -resize %wx%h %d"; is being ignored on r80886 when uploading svg files. I.e. svghandler::reasterize() does not respect that variable:

"SvgHandler::rasterize: convert -background white -thumbnail 800x427\! '/var/www/hackerspace.lu/www/w/images/8/84/Crowd.svg' PNG:'/var/www/hackerspace.lu/www/w/images/thumb/8/84/Crowd.svg/800px-Crowd.svg.png' 2>&1"

this fails (although graphicsmagick-imagemagick-compat is installed) with the following error message:

"thumbnail failed on syndi: error 1 "convert: Unrecognized option (-thumbnail)." from "convert -background white -thumbnail 800x427\! '/var/www/hackerspace.lu/www/w/images/8/84/Crowd.svg' PNG:'/var/www/hackerspace.lu/www/w/images/thumb/8/84/Crowd.svg/800px-Crowd.svg.png' 2>&1""

It is NOT being ignored by the BitmapHandler though:

"BitmapHandler::doTransform: Running custom convert command gm convert '/var/www/hackerspace.lu/www/w/images/d/df/Defcon-8.jpg' -resize 800x600 '/var/www/hackerspace.lu/www/w/images/thumb/d/df/Defcon-8.jpg/800px-Defcon-8.jpg'"

This works just fine.

Bryan.TongMinh wrote:

That is more or less intended, $wgCustomConvertCommand is only meant to work with bitmaps. For SVGs you need to set $wgSVGConverter and add your custom convertor to $wgSVGConverters. See includes/DefaultSettings.php for examples.

However, I think the real problem is that SVGHandler::rasterize does not use the IM specific escaping that is defined in BitmapHandler. (We should really split our MediaHandler and MediaTransformer code one day)

david wrote:

Oh, sorry, I wasn't aware of the $wgSVGConverter variable. I'll add it right away.

david wrote:

(In reply to comment #6)

Oh, sorry, I wasn't aware of the $wgSVGConverter variable. I'll add it right
away.

Just for completeness' sake I'll add the debugging output below. It seems odd though that it 496 bytes would not be allocatable.

SvgHandler::rasterize: rsvg -w120 -h78 '/var/www/hackerspace.lu/www/w/images/c/c7/Speaker_badge_rear.svg' '/var/www/hackerspace.lu/www/w/images/thumb/c/c7/Speaker_badge_rear.svg/120px-Speaker_badge_rear.svg.png' 2>&1
wfShellExec: '/var/www/hackerspace.lu/www/w/bin/ulimit4.sh' 180 102400 102400 'rsvg -w120 -h78 '\''/var/www/hackerspace.lu/www/w/images/c/c7/Speaker_badge_rear.svg'\'' '\''/var/www/hackerspace.lu/www/w/images/thumb/c/c7/Speaker_badge_rear.svg/120px-Speaker_badge_rear.svg.png'\'' 2>&1'
Removing bad 0-byte thumbnail "/var/www/hackerspace.lu/www/w/images/thumb/c/c7/Speaker_badge_rear.svg/120px-Speaker_badge_rear.svg.png"
thumbnail failed on syndi: error 0 "*MEMORY-ERROR*: rsvg-convert[16683]: GSlice: failed to allocate 496 bytes (alignment: 512): Cannot allocate memory" from "rsvg -w120 -h78 '/var/www/hackerspace.lu/www/w/images/c/c7/Speaker_badge_rear.svg' '/var/www/hackerspace.lu/www/w/images/thumb/c/c7/Speaker_badge_rear.svg/120px-Speaker_badge_rear.svg.png' 2>&1"

We're about to release 1.17RC1 ... could you update this after trying it? (Sorry I just found your bug... it is on my radar now.)

Bryan.TongMinh wrote:

Tweaking summary.

Works for me on both trunk (1.18 - r85088) as well as on 1.15.5.

The command used (according to debug messages) is:
/bin/bash '/var/www/w/phase3/bin/ulimit4.sh' 180 102400 102400 'convert -background white -thumbnail 128x128\! '\''/var/www/w/phase3/images/0/0e/Inkscape_logo_2.svg'\'' PNG:'\''/var/www/w/phase3/images/thumb/0/0e/Inkscape_logo_2.svg/128px-Inkscape_logo_2.svg.png'\'' 2>&1'

Version of inkscape is

bawolff@Bawolff-L:/var/www/w/phase3$ convert --version
Version: ImageMagick 6.3.7 08/07/09 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2008 ImageMagick Studio LLC

Version of wmf2eps is:

bawolff@Bawolff-L:/var/www/w/phase3$ wmf2eps --version
libwmf: version 0.2.8


I've tried executing the same command on the shell and it works flawlessly when
removing all of the single quotes. I think that convert is somehow interpreting
both arguments, source file and output file as a single argument due to some
erroneous escaping. (Removing one pair of "\'" from every argument works as
well)

When trying it on shell, did you include the ulimit.sh part to? Did you try at as the same user as apache, did increasing $wgMaxShellMemory do anything for making it work on mediawiki, etc.

(In reply to David from comment #7)

(In reply to comment #6)

Oh, sorry, I wasn't aware of the $wgSVGConverter variable. I'll add it right
away.

Just for completeness' sake I'll add the debugging output below. It seems
odd though that it 496 bytes would not be allocatable.

Its probably the last 496 bytes that give a problem (e.g. The limit was already almost over the limit, and the last allocation of 496 bytes is what brought it over).

Please increase $wgMaxShellMemory in your LocalSettings.php file


Closing invalid. This appears to be a configuration error. The default for $wgMaxShellMemory has been increased since this bug was filed.

Gilles raised the priority of this task from Medium to Unbreak Now!.Dec 4 2014, 10:20 AM
Gilles added a project: Multimedia.
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to Medium.Dec 4 2014, 11:22 AM