Page MenuHomePhabricator

Allow CSV result format to accommodate large result sets
Closed, ResolvedPublic

Description

Author: mediawiki

Description:
patch to use php://temp instead of php://memory

Hi Markus,
There was a discussion on semediawiki-user about CSV result printer limits ( | limit=40000) so I poked around and made a small change. (see http://us.php.net/wrappers.php).

Attached is the proposed patch.

Thanks,
Joel


Version: unspecified
Severity: enhancement

Attached:

Details

Reference
bz17760

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 21 2014, 10:31 PM
bzimport set Reference to bz17760.

mediawiki wrote:

Markus,
I went ahead and applied the patch after thorough testing with $smwgQMaxInlineLimit and "| limit" set to large values.
(http://svn.wikimedia.org/viewvc/mediawiki?view=rev&revision=47994)

Feel free to reverse the commit if need be.

Best,
Joel

Thanks a lot -- I am somewhat busy these weeks, hence the delay in incorporating contributions.

The change looks okay to me, although I vaguely recall that I had some reason for using memory at first. But we can just see how things work out in practice (if nobody complains, I assume that all is well).

mediawiki wrote:

No problem - I suspected as much especially with CeBit underway. I just wish I could have taken the time to go there for the 2nd SMW UGM but work didn't permit. Hopefully, arrangements will be made to get SMW users to participate virtually like when you Skyped in Boston.

FWIW, I did some stress testing with big result sets spanning more than 2M (the point, at which PHP starts swapping to the filesystem by default), and even set the swap point (php://temp/maxmemory:NNNbytes) to a ridiculously low number (100 bytes) and it worked without problems.

Ok, great (the tests). I still fear that query result formatting is not particularly memory efficient in the API right now (too much copying, whole result kept for serialization instead of keeping only current slice). I will try to improve this in some future.

Denny and Frank (SRF Graph extension) are at CeBit for demoing SMW, I escaped. Other things are due here, anyway (e.g. the 6th ontolog telco on Semantic Wikis tomorrow; see http://ontolog.cim3.net/cgi-bin/wiki.pl?ConferenceCall_2009_03_05).