Page MenuHomePhabricator

Accessing content from command line on private wikis
Closed, ResolvedPublic

Description

So, I was trying to index a private wiki in CirrusSearch. TMH doesn't like this, as it enforces all permission checks. Usually this makes sense, but doesn't from the command line (even though we're running as an anon, we should assume read access to everything). Here's the output:

$ mwscript extensions/CirrusSearch/forceSearchIndex.php --wiki officewiki --fromId 0 --toId 5990 --batch-size=50
Indexed 50 pages ending at 64 at 20/second
Indexed 50 pages ending at 127 at 24/second
Indexed 50 pages ending at 197 at 24/second
Indexed 50 pages ending at 265 at 23/second
Indexed 50 pages ending at 316 at 24/second
Indexed 50 pages ending at 375 at 25/second
Indexed 50 pages ending at 430 at 24/second
Indexed 50 pages ending at 488 at 22/second
Indexed 50 pages ending at 546 at 22/second
Indexed 50 pages ending at 606 at 22/second
Indexed 50 pages ending at 664 at 22/second
Indexed 50 pages ending at 730 at 21/second
Indexed 50 pages ending at 785 at 21/second
Indexed 50 pages ending at 839 at 21/second
Indexed 50 pages ending at 896 at 21/second
Indexed 50 pages ending at 952 at 22/second
Indexed 50 pages ending at 1002 at 22/second
Indexed 50 pages ending at 1055 at 22/second
Indexed 50 pages ending at 1107 at 22/second
Indexed 50 pages ending at 1163 at 22/second
Indexed 50 pages ending at 1217 at 23/second
Indexed 50 pages ending at 1275 at 23/second
Indexed 50 pages ending at 1330 at 23/second
Indexed 50 pages ending at 1380 at 23/second
Indexed 50 pages ending at 1435 at 22/second
Indexed 50 pages ending at 1491 at 23/second
Indexed 50 pages ending at 1543 at 22/second
Indexed 50 pages ending at 1596 at 22/second
Indexed 50 pages ending at 1657 at 22/second
Indexed 50 pages ending at 1710 at 22/second
Indexed 50 pages ending at 1764 at 22/second
Indexed 50 pages ending at 1815 at 23/second
Indexed 50 pages ending at 1869 at 23/second
Indexed 50 pages ending at 1923 at 23/second
Indexed 50 pages ending at 1977 at 23/second
Indexed 50 pages ending at 2034 at 23/second
Indexed 50 pages ending at 2089 at 23/second
Indexed 50 pages ending at 2151 at 23/second
Indexed 50 pages ending at 2208 at 23/second
Indexed 50 pages ending at 2268 at 22/second
Indexed 50 pages ending at 2332 at 22/second
Indexed 50 pages ending at 2394 at 22/second
Indexed 50 pages ending at 2446 at 22/second
Indexed 50 pages ending at 2497 at 22/second
Indexed 50 pages ending at 2629 at 22/second
Indexed 50 pages ending at 2711 at 22/second
Indexed 50 pages ending at 2770 at 21/second
Indexed 50 pages ending at 2852 at 21/second
Indexed 50 pages ending at 2910 at 21/second
Indexed 50 pages ending at 2974 at 21/second
Indexed 50 pages ending at 3034 at 21/second
Indexed 50 pages ending at 3094 at 21/second
Indexed 50 pages ending at 3149 at 21/second
Indexed 50 pages ending at 3206 at 21/second
Indexed 50 pages ending at 3263 at 21/second
Indexed 50 pages ending at 3320 at 21/second
Indexed 50 pages ending at 3387 at 21/second
Indexed 50 pages ending at 3448 at 21/second
Indexed 50 pages ending at 3499 at 20/second
Indexed 50 pages ending at 3554 at 20/second
Indexed 50 pages ending at 3604 at 20/second
Indexed 50 pages ending at 3661 at 20/second
Indexed 50 pages ending at 3756 at 20/second
You need read permission to use this module
Backtrace:
#0 /usr/local/apache/common-local/php-1.22wmf17/includes/api/ApiBase.php(1437): ApiBase->dieUsage('You need read p...', 'readapidenied')
#1 /usr/local/apache/common-local/php-1.22wmf17/includes/api/ApiMain.php(769): ApiBase->dieUsageMsg('readrequired')
#2 /usr/local/apache/common-local/php-1.22wmf17/includes/api/ApiMain.php(822): ApiMain->checkExecutePermissions(Object(ApiQuery))
#3 /usr/local/apache/common-local/php-1.22wmf17/includes/api/ApiMain.php(349): ApiMain->executeAction()
#4 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(282): ApiMain->execute()
#5 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(58): TimedMediaTransformOutput->getPosterFromApi(854)
#6 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(309): TimedMediaTransformOutput->getUrl(Array)
#7 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(249): TimedMediaTransformOutput->getMediaAttr(Array, true)
#8 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(173): TimedMediaTransformOutput->getHtmlMediaTagOutput(Array, true)
#9 /usr/local/apache/common-local/php-1.22wmf17/extensions/TimedMediaHandler/TimedMediaTransformOutput.php(136): TimedMediaTransformOutput->getImagePopUp()
#10 /usr/local/apache/common-local/php-1.22wmf17/includes/gallery/TraditionalImageGallery.php(137): TimedMediaTransformOutput->toHtml(Array)
#11 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Parser.php(5210): TraditionalImageGallery->toHTML()
#12 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/CoreTagHooks.php(120): Parser->renderImageGallery('?File:Multimedi...', Array)
#13 [internal function]: CoreTagHooks::gallery('?File:Multimedi...', Array, Object(Parser), Object(PPFrame_DOM))
#14 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Parser.php(4000): call_user_func_array(Array, Array)
#15 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Preprocessor_DOM.php(1181): Parser->extensionSubstitution(Array, Object(PPFrame_DOM))
#16 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Parser.php(3129): PPFrame_DOM->expand(Object(PPNode_DOM), 0)
#17 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Parser.php(1201): Parser->replaceVariables('{{Report-nav? |...')
#18 /usr/local/apache/common-local/php-1.22wmf17/includes/parser/Parser.php(388): Parser->internalParse('{{Report-nav? |...')
#19 /usr/local/apache/common-local/php-1.22wmf17/includes/content/WikitextContent.php(300): Parser->parse('{{Report-nav? |...', Object(Title), Object(ParserOptions), true, true, 27529)
#20 /usr/local/apache/common-local/php-1.22wmf17/includes/WikiPage.php(3514): WikitextContent->getParserOutput(Object(Title), 27529, Object(ParserOptions))
#21 /usr/local/apache/common-local/php-1.22wmf17/includes/PoolCounter.php(222): PoolWorkArticleView->doWork()
#22 /usr/local/apache/common-local/php-1.22wmf17/includes/WikiPage.php(1126): PoolCounterWork->execute()
#23 /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/CirrusSearchUpdater.php(182): WikiPage->getParserOutput(Object(ParserOptions), 27529)
#24 /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/CirrusSearchUpdater.php(124): CirrusSearchUpdater::buildDocumentforRevision(Array)
#25 /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/forceSearchIndex.php(113): CirrusSearchUpdater::updateRevisions(Array)
#26 /usr/local/apache/common-local/php-1.22wmf17/maintenance/doMaintenance.php(115): ForceSearchIndex->execute()
#27 /usr/local/apache/common-local/php-1.22wmf17/extensions/CirrusSearch/forceSearchIndex.php(298): require_once('/usr/local/apac...')
#28 /usr/local/apache/common-local/multiversion/MWScript.php(97): require_once('/usr/local/apac...')
#29 {main}


Version: master
Severity: normal

Details

Reference
bz54394

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:06 AM
bzimport set Reference to bz54394.

Is this actually high priority for WMF, if it only affects private wikis? It would be nice to have on private wikis, but is it really that important? Workarounds include not deploying CS to private wikis for now or taking TMH off of them. Suggest bumping priority down to Normal or Low, but the Multimedia team will definitely take a look at this.

(In reply to comment #1)

Is this actually high priority for WMF, if it only affects private wikis? It
would be nice to have on private wikis, but is it really that important?
Workarounds include not deploying CS to private wikis for now or taking TMH
off
of them.

I'm fine with holding off on CS for awhile on those wikis, but they can't hold off forever. I'd rather not take TMH away if people are using it though, I guess we can cross that bridge when the time comes.

Suggest bumping priority down to Normal or Low, but the Multimedia
team will definitely take a look at this.

Set to Normal/Normal, I mainly wanted to be sure you guys knew about it and were indeed going to take a look :)

Change 96932 had a related patch set uploaded by Brian Wolff:
Don't use API to fetch alternate poster sizes

https://gerrit.wikimedia.org/r/96932

Change 96932 merged by jenkins-bot:
Don't use API to fetch alternate poster sizes

https://gerrit.wikimedia.org/r/96932

Patch was merged a while ago - is there more work left to do here (if yes: please reset the bug report status to NEW or ASSIGNED), or can you close this ticket as RESOLVED FIXED?

Gilles raised the priority of this task from Medium to Unbreak Now!.Dec 4 2014, 10:24 AM
Gilles moved this task from Untriaged to Done on the Multimedia board.
Gilles lowered the priority of this task from Unbreak Now! to Medium.Dec 4 2014, 11:22 AM