Page MenuHomePhabricator

wfShellExec() cuts off stdout at multiples of 8192 bytes
Closed, ResolvedPublic

Description

It doesn't seem to be related to any memory / time limits. The output of the script is just truncated. It works when run under php. (php below is aliased to hhvm)

php extensions/Translate/scripts/createMessageIndex.php --memory-limit=max
[e18f17bf] [no req] Exception from line 113 of /www/dev.translatewiki.net/w/extensions/Translate/ffs/PythonSingleFFS.php: Failed to decode python file /resources/projects/pywikibot/add_text.py
Backtrace:
#0 /www/dev.translatewiki.net/w/extensions/Translate/ffs/PythonSingleFFS.php(86): PythonSingleFFS->parseFile()
#1 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/FileBasedMessageGroup.php(52): PythonSingleFFS->read(string)
#2 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/MessageGroupBase.php(81): FileBasedMessageGroup->load(string)
#3 /www/dev.translatewiki.net/w/extensions/Translate/messagegroups/MessageGroupBase.php(183): MessageGroupBase->getDefinitions()
#4 /www/dev.translatewiki.net/w/extensions/Translate/utils/MessageIndex.php(186): MessageGroupBase->getKeys()
#5 /www/dev.translatewiki.net/w/extensions/Translate/utils/MessageIndex.php(119): MessageIndex->checkAndAdd(array, FileBasedMessageGroup)
#6 /www/dev.translatewiki.net/w/extensions/Translate/scripts/createMessageIndex.php(31): MessageIndex->rebuild()
#7 /www/dev.translatewiki.net/w/maintenance/doMaintenance.php(109): CreateMessageIndex->execute()
#8 /www/dev.translatewiki.net/w/extensions/Translate/scripts/createMessageIndex.php(36): include(string)
#9 {main}

The relevant code:

		$command = wfEscapeShellArg( "import json; execfile(\"$filename\"); " .
			"print json.dumps(msg)" );
		$json = wfShellExec( "python -c $command" );

		$parsed = FormatJson::decode( $json, true );
		if ( !is_array( $parsed ) ) {
			throw new MWException( "Failed to decode python file $filename" );
		}

Version: 1.24rc
Severity: normal

Details

Reference
bz67870

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:39 AM
bzimport set Reference to bz67870.
bzimport added a subscriber: Unknown Object (MLST).

Thanks for the report. Could you var_dump() the value of $json after the wfShellExec() invocation and the value of $parsed after the FormatJson::decode() invocation?

Created attachment 15922
Output returned by wfShellExec

I added the following line after the decode call:
var_dump( 'JSON', $json, 'PARSED', $parsed ); die();

The result is in the attachment. Interesting thing is that the length of the truncated output is 8192 bytes.

Attached:

It's a general wfShellExec() problem.

Change 147635 had a related patch set uploaded by MaxSem:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/147635

Change 147635 merged by jenkins-bot:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/147635

Change 148419 had a related patch set uploaded by MaxSem:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/148419

Change 148419 merged by jenkins-bot:
Fix race condition in wfShellExec()

https://gerrit.wikimedia.org/r/148419

Hi, I noticed that WfShellExecTest::testT69870() blocked my test job (make the job infinite waiting). I think this place should be a good place to ask this question. I using Windows 10 and PHP 7.2.9 as my development environment, the problem happened on here. But in my Linux system, everything looks good.

For my test (using die()), I think my job blocked by wfShellExec().