Parsoid stacktrace on beta cluster
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• Cmcmahon
	Nov 8 2013, 3:30 PM

Description

http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA seen for both anon and logged in user:

[42c45aa0] /wiki/Talk:Flow_QA Exception from line 62 of /data/project/apache/common-local/php-master/extensions/Flow/includes/ParsoidUtils.php: Failed contacting parsoid

Backtrace:

#0 /data/project/apache/common-local/php-master/extensions/Flow/includes/ParsoidUtils.php(21): Flow\ParsoidUtils::parsoid(string, string, string)
#1 /data/project/apache/common-local/php-master/extensions/Flow/includes/Model/AbstractRevision.php(315): Flow\ParsoidUtils::convert(string, string, string)
#2 /data/project/apache/common-local/php-master/extensions/Flow/includes/Model/AbstractRevision.php(290): Flow\Model\AbstractRevision->getConvertedContent(string)
#3 /data/project/apache/common-local/php-master/extensions/Flow/templates/header.html.php(19): Flow\Model\AbstractRevision->getContent(User, string)
#4 /data/project/apache/common-local/php-master/extensions/Flow/includes/Templating.php(76): include(string)
#5 /data/project/apache/common-local/php-master/extensions/Flow/includes/Templating.php(50): Flow\Templating->_render(string, array)
#6 /data/project/apache/common-local/php-master/extensions/Flow/includes/Block/Header.php(116): Flow\Templating->render(string, array)
#7 /data/project/apache/common-local/php-master/extensions/Flow/includes/View.php(65): Flow\Block\HeaderBlock->render(Flow\Templating, array)
#8 /data/project/apache/common-local/php-master/extensions/Flow/Hooks.php(155): Flow\View->show(Flow\WorkflowLoader, string)
#9 [internal function]: FlowHooks::onPerformAction(OutputPage, Article, Title, User, WebRequest, MediaWiki)
#10 /data/project/apache/common-local/php-master/includes/Hooks.php(199): call_user_func_array(string, array)
#11 /data/project/apache/common-local/php-master/includes/GlobalFunctions.php(4032): Hooks::run(string, array)
#12 /data/project/apache/common-local/php-master/includes/Wiki.php(423): wfRunHooks(string, array)
#13 /data/project/apache/common-local/php-master/includes/Wiki.php(305): MediaWiki->performAction(Article, Title)
#14 /data/project/apache/common-local/php-master/includes/Wiki.php(596): MediaWiki->performRequest()
#15 /data/project/apache/common-local/php-master/includes/Wiki.php(460): MediaWiki->main()
#16 /data/project/apache/common-local/php-master/index.php(49): MediaWiki->run()
#17 /data/project/apache/common-local/w/index.php(3): require(string)
#18 {main}

Version: unspecified
Severity: blocker

Details

Reference: bz56781

Event Timeline

• bzimport raised the priority of this task from to Unbreak Now!.Nov 22 2014, 2:20 AM

• bzimport added a project: Parsoid-Web-API.

• bzimport set Reference to bz56781.

• Cmcmahon created this task.Nov 8 2013, 3:30 PM

For obvious reasons, moving to blocker/immediate - we can't really test any written code (or consider sandbox releases) without a working sandbox.

The WMF core features team tracks this bug on Mingle card https://mingle.corp.wikimedia.org/projects/flow/cards/429, but people from the community are welcome to contribute here and in Gerrit.

The WMF core features team tracks this bug on Mingle card https://mingle.corp.wikimedia.org/projects/flow/cards/434, but people from the community are welcome to contribute here and in Gerrit.

Note the error:

Failed contacting parsoid

So it seems the parsoid daemon on beta cluster was/is dead.

Well, plausibly, but given that we got it working again on ee-flow...

Erik/S, can you chip in and explain how you un-screwed the situation?

I have upgraded varnish on all caches including deployment-parsoidcache3.

When looking at that cache backend varnish (varnishncsa) I see requests passed to to http://parsoid//enwiki/Main_Page , the 'parsoid' host is not resolvable so that is unlikely to work.

Disregard my comment about 'parsoid' host not being found. The queries do hit the deployment-parsoid2 parsoid backend, using tcpdump I get:

POST //enwiki/Main_Page HTTP/1.0
Accept: */*
Accept-Encoding: deflate, gzip
Referer: http://en.wikipedia.beta.wmflabs.org/wiki/Talk:Flow_QA
User-Agent: MediaWiki/1.23alpha
Content-Length: 144
Content-Type: multipart/form-data; boundary=----------------------------3cae3ac95a02
X-Varnish: 553410961
host: parsoid
X-Forwarded-For: 10.4.0.187, 10.4.0.187, 127.0.0.1, 127.0.0.1
X-Varnish: 1458843141

It replies back with a stacktrace:

HTTP/1.1 500 Internal Server Error
X-Powered-By: Express
Content-Type: text/x-mediawiki; charset=UTF-8
Content-Length: 747
Date: Mon, 11 Nov 2013 14:05:06 GMT
Connection: close

TypeError: Cannot call method 'match' of undefined

at Object.DOMUtils.parseHTML (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.DOMUtils.js:1714:12)
at app.post.cb (/srv/deployment/parsoid/Parsoid/js/api/ParserService.js:677:13)
at getParserServiceEnv (/srv/deployment/parsoid/Parsoid/js/api/ParserService.js:385:4)
at MWParserEnvironment.getParserEnv (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.parser.environment.js:281:3)
at setupWikiConfig (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.parser.environment.js:318:3)
at ApiRequest._processListeners.processSome (/srv/deployment/parsoid/Parsoid/js/lib/mediawiki.ApiRequest.js:110:17)
at process.startup.processNextTick.process._tickCallback (node.js:244:9)

So that seems to be an issue in Parsoid backend.

I have changed Parsoid version from 4 months old cdbfdbb to 986c1e7 (the one in 1.23wmf3).

Deleted the node modules in /var/lib/parsoid/Parsoid/js/node_modules and reran npm install .

Stopped parsoid entirely using the init.d script and restarted it.

It seems to be working again :-)

Antoine, can Parsoid be updated from master along with all the other core+extensions in place already?

(In reply to comment #9)

Antoine, can Parsoid be updated from master along with all the other
core+extensions in place already?

There is some discussion on bug 56622 to have Parsoid support heterogeneous deployment on beta labs. Currently the Parsoid instance is updated manually (git pull and restart of node service).

Could https://gerrit.wikimedia.org/r/#/c/93939/ be deployed easily?

Parsoid stacktrace on beta clusterClosed, ResolvedPublicActions

Description

Details

Event Timeline

Parsoid stacktrace on beta cluster
Closed, ResolvedPublic
Actions