wikidata beta (item pages, etc.) inaccessible with 503 errors
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	aude
	Aug 18 2014, 4:53 PM

Description

wikidata beta is (and has been inaccessible) with 503 errors. hhvm is running there.

finally located logs in /var/log/upstart and we have fatal errors:

Fatal error: Argument 1 passed to Wikibase\\EntityContentFactory::getTitleForId() must be an instance of Wikibase\\EntityId, Wikibase\\DataModel\\Entity\\ItemId given in /srv/common-local/php-master/extensions/Wikidata/extensions/Wikibase/repo/includes/content/EntityContentFactory.php on line 83

this is a class alias issue, which i think is resolved in latest hhvm (things work in vagrant and travis no longer has this issue)

We cen remove the evil class aliases anyway but would be good to know for sure if the bug is gone in hhvm or if we somehow don't have the bug fix in the hhvm version we are using.

Version: unspecified
Severity: normal

Details

Reference: bz69708

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:40 AM

• bzimport added a project: Beta-Cluster-Infrastructure.

• bzimport set Reference to bz69708.

• bzimport added a subscriber: Unknown Object (MLST).

aude created this task.Aug 18 2014, 4:53 PM

mysteriously suddenly it works again. it has off and on, been working occasionally, so I am not yet convinced it is fixed!

the fix for https://github.com/facebook/hhvm/issues/2527 appears to be in the hhvm package deployed on labs

Here are the hhvm versions deployed on the two beta cluster instances:

deployment-mediawiki01:~$ hhvm --version
HipHop VM 3.3.0-dev (rel)
Compiler: heads/master-0-g6366fcf6ac4f437c9f16d6d4a468e21c8ba76d40
Repo schema: bf5f5d8158a21a0ea40abdfb3e2422f14d932365
Extension API: 20140727

deployment-mediawiki02:~$ hhvm --version
HipHop VM 3.3.0-dev (dbg)
Compiler: heads/master-0-g3bf9672f95e86742fc67454a389fd7be68653325
Repo schema: 2fcacb410e0d852615f9e06a36b56f26f5dc440a
Extension API: 20140702

Only mediawiki02 (the last above) is pooled currently. The hhvm package is:

$ apt-cache policy hhvm
hhvm:

Installed: 3.3-dev+20140728+wmf5
Candidate: 3.3-dev+20140728+wmf4
Version table:

3.3-dev+20140728+wmf5 0 100 /var/lib/dpkg/status 3.3-dev+20140728+wmf4 0 1001 http://apt.wikimedia.org/wikimedia/ trusty-wikimedia/main amd64 Packages

broke again...

Fatal error: Argument 3 passed to Wikibase\\EntityContentFactory::getPermissionStatusForEntityId() must be an instance of Wikibase\\EntityId, Wikibase\\DataModel\\Entity\\ItemId given in /srv/common-local/php-master/extensions/Wikidata/extensions/Wikibase/repo/includes/content/EntityContentFactory.php on line 236

now we are getting

Fatal error: Argument 1 passed to Wikibase\\EntityHandler::getTitleForId() must be an instance of Wikibase\\EntityId, Wikibase\\DataModel\\Entity\\ItemId given in /srv/common-local/php-master/extensions/Wikidata/extensions/Wikibase/repo/includes/content/EntityHandler.php on line 426

reproduced these issues in vagrant, which uses hhvm:

Fatal error: Argument 1 passed to Wikibase\\ItemContent::__construct() must be an instance of Wikibase\\Item, Wikibase\\DataModel\\Entity\\Item given in /vagrant/mediawiki/extensions/WikidataBuildResources/extensions/Wikibase/repo/includes/content/ItemContent.php on line 95

i then restarted hhvm and the error is gone! before restarting, i did composer install of all of wikibase etc., which maybe exceeded some limit or such?

I wonder if this is related to the hhvm cache issues that Ori has been looking into? I've only been following that via irc eavesdropping, but I think the just is that HHVM has pool sizes for certain types of bytecode and in some circumstances we have more code than can fit in the currently configured pools.

@bryan I think quite certain it is related.

this happens less often on test.wikidata, but I think because hhvm has higher limits in production. (JitAColdSize etc?)

probably resolving the type hints when it involves class aliases + inheritance is an intensive thing, maybe also with memory leak or some bug in hhvm, causing limits to be hit and triggering crash.

btw, we are working to eliminate use of class alias in Wikibase, as not a good idea generally to have them but same time not easy to eliminate completely.

restart hhvm
maybe do a thing or two (e.g. add a label)
click history
view a diff and hhvm crashes

viewing a diff in vagrant:

[Tue Sep 16 13:19:28 2014] [hphp] [9179:7f024bbff700:9:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138
[Tue Sep 16 13:19:29 2014] [hphp] [9179:7f0248fff700:3:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138
[Tue Sep 16 13:19:34 2014] [hphp] [9179:7f02497ff700:4:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138
[Tue Sep 16 13:19:39 2014] [hphp] [9302:7f85037ff700:1:000001] [] \nNotice: Cannot find site enwiki in sites table [Called from Wikibase\\Client\\WikibaseClient::newSiteGroup in /vagrant/mediawiki/extensions/WikidataBuildResources/extensions/Wikibase/client/includes/WikibaseClient.php at line 446] in /vagrant/mediawiki/includes/debug/MWDebug.php on line 302
[Fri Sep 19 16:05:36 2014] [hphp] [4697:7fcec1bff700:2:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138
[Fri Sep 19 16:05:37 2014] [hphp] [4697:7fcecefff700:7:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138
[Fri Sep 19 16:05:42 2014] [hphp] [4697:7fced7bff700:9:000001] [] \nFatal error: Argument 2 passed to Wikibase\\DataModel\\Deserializers\\ClaimDeserializer::setRankFromSerialization() must be an instance of Wikibase\\DataModel\\Claim\\Statement, Wikibase\\DataModel\\Statement\\Statement given in /vagrant/mediawiki/extensions/WikidataBuildResources/vendor/wikibase/data-model-serialization/src/Deserializers/ClaimDeserializer.php on line 138

Is this still an issue? I don't have wikidata going in a vagrant instance, but on beta I can go here without fail:
http://wikidata.beta.wmflabs.org/w/index.php?title=Q30926&curid=33206&diff=56321&oldid=56319

(ie: doing steps in comment 10 (ish), but on beta not vagrant)

greg moved this task from To Triage to Done on the Beta-Cluster-Infrastructure board.Jan 8 2015, 5:34 PM

wikidata beta (item pages, etc.) inaccessible with 503 errorsClosed, DeclinedPublicActions

Description

Details

Event Timeline

wikidata beta (item pages, etc.) inaccessible with 503 errors
Closed, DeclinedPublic
Actions