Page MenuHomePhabricator
Create Task
Maniphest T58439

Import of Commons partial export fails with error about plurals-mediawiki.xml
Closed, ResolvedPublic
Actions

Description

I exported the latest version of all of the pages starting with MediaWiki:Lang/ (MediaWiki:Lang/de, etc.), to test Template:LangSwitch. However, on import I get repeating warnings like the below, I believe for every revision:

Warning: DOMDocument::load(): I/O warning : failed to load external entity "/vagrant/mediawiki/languages/data/plurals-mediawiki.xml" in /vagrant/mediawiki/includes/cache/LocalisationCache.php on line 588

Call Stack:

 0.0045     778856   1. {main}() /vagrant/mediawiki/maintenance/importDump.php:0
 0.0121    1396960   2. require_once('/vagrant/mediawiki/maintenance/doMaintenance.php') /vagrant/mediawiki/maintenance/importDump.php:291
 0.3200   17065176   3. BackupReader->execute() /vagrant/mediawiki/maintenance/doMaintenance.php:113
 0.3202   17065624   4. BackupReader->importFromStdin() /vagrant/mediawiki/maintenance/importDump.php:97
 0.3206   17066456   5. BackupReader->importFromHandle() /vagrant/mediawiki/maintenance/importDump.php:253
 0.3279   17824496   6. WikiImporter->doImport() /vagrant/mediawiki/maintenance/importDump.php:286
98.5509  207738984   7. WikiImporter->handlePage() /vagrant/mediawiki/includes/Import.php:473
98.5575  207748280   8. WikiImporter->handleRevision() /vagrant/mediawiki/includes/Import.php:609
98.5603  207754536   9. WikiImporter->processRevision() /vagrant/mediawiki/includes/Import.php:658
98.5607  207758064  10. WikiImporter->revisionCallback() /vagrant/mediawiki/includes/Import.php:706
98.5607  207758696  11. call_user_func_array() /vagrant/mediawiki/includes/Import.php:351
98.5608  207758840  12. BackupReader->handleRevision() /vagrant/mediawiki/includes/Import.php:0
98.5608  207758920  13. call_user_func() /vagrant/mediawiki/maintenance/importDump.php:165
98.5608  207758976  14. WikiImporter->importRevision() /vagrant/mediawiki/maintenance/importDump.php:0
98.5610  207759760  15. DatabaseBase->deadlockLoop() /vagrant/mediawiki/includes/Import.php:257
98.5623  207760720  16. call_user_func_array() /vagrant/mediawiki/includes/db/Database.php:3004
98.5623  207761088  17. WikiRevision->importOldRevision() /vagrant/mediawiki/includes/db/Database.php:0
98.5918  207781560  18. WikiPage->doEditUpdates() /vagrant/mediawiki/includes/Import.php:1434
98.5927  207782424  19. WikiPage->prepareContentForEdit() /vagrant/mediawiki/includes/WikiPage.php:2069
98.6048  207799064  20. WikitextContent->getParserOutput() /vagrant/mediawiki/includes/WikiPage.php:2026
98.6048  207799384  21. Parser->parse() /vagrant/mediawiki/includes/content/WikitextContent.php:300
98.6058  207812752  22. Parser->internalParse() /vagrant/mediawiki/includes/parser/Parser.php:395
98.6079  207813128  23. Parser->replaceInternalLinks() /vagrant/mediawiki/includes/parser/Parser.php:1230
98.6079  207813264  24. Parser->replaceInternalLinks2() /vagrant/mediawiki/includes/parser/Parser.php:1825
98.6080  207815560  25. Parser->getTargetLanguage() /vagrant/mediawiki/includes/parser/Parser.php:1861
98.6081  207815560  26. Title->getPageLanguage() /vagrant/mediawiki/includes/parser/Parser.php:837
98.6086  207815624  27. ContentHandler->getPageLanguage() /vagrant/mediawiki/includes/Title.php:4824
98.6097  207815920  28. wfGetLangObj() /vagrant/mediawiki/includes/content/ContentHandler.php:620
98.6107  207922328  29. Language::factory() /vagrant/mediawiki/includes/GlobalFunctions.php:1304
98.6107  207922328  30. Language::newFromCode() /vagrant/mediawiki/languages/Language.php:196
98.6122  207922688  31. Language::getFallbacksFor() /vagrant/mediawiki/languages/Language.php:237
98.6123  207922928  32. LocalisationCache->getItem() /vagrant/mediawiki/languages/Language.php:4120
98.6123  207922928  33. LocalisationCache->loadItem() /vagrant/mediawiki/includes/cache/LocalisationCache.php:259
98.6124  207922928  34. LocalisationCache->initLanguage() /vagrant/mediawiki/includes/cache/LocalisationCache.php:324
98.6203  207923208  35. LocalisationCache->recache() /vagrant/mediawiki/includes/cache/LocalisationCache.php:442
98.6667  208613368  36. LocalisationCache->readSourceFilesAndRegisterDeps() /vagrant/mediawiki/includes/cache/LocalisationCache.php:772
98.7115  210162200  37. LocalisationCache->getPluralRuleTypes() /vagrant/mediawiki/includes/cache/LocalisationCache.php:630
98.7115  210162200  38. LocalisationCache->loadPluralFiles() /vagrant/mediawiki/includes/cache/LocalisationCache.php:558
98.7188  210162576  39. LocalisationCache->loadPluralFile() /vagrant/mediawiki/includes/cache/LocalisationCache.php:578
98.7188  210163240  40. DOMDocument->load() /vagrant/mediawiki/includes/cache/LocalisationCache.php:588

Version: 1.23.0
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=62368

Details

Reference
bz56439
SubjectRepoBranchLines +/-
LocalisationCache: Use file_get_contents instead of DOMDocument::loadmediawiki/coremaster+6 -1
Customize query in gerrit

Related Objects

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 2:27 AM
• bzimport added projects: MW-1.24-release, MediaWiki-Core-Snapshots, I18n.
• bzimport set Reference to bz56439.
• Mattflaschen-WMF created this task.Oct 31 2013, 11:27 PM
• Mattflaschen-WMF added a comment.Oct 31 2013, 11:28 PM

Created attachment 13641
Dump that shows problem

Attached:

Commons_MediaWiki_lang.xml190 KBDownload

• Mattflaschen-WMF added a comment.Oct 31 2013, 11:37 PM

Created attachment 13643
Comparison trivial import that works

The import with the warning also does not seem to work (though I have ctrl-Ced it every time) even partially.

A trivial one-page import (attached) works fine.

Attached:

Import_test_2.xml2 KBDownload

yuvipanda added a comment.Nov 8 2013, 12:29 AM

Also affects testwikidatawiki

yuvipanda added a comment.Nov 8 2013, 12:30 AM

Only the full history version of it, works fine for the current versions only dump

Nemo_bis added a comment.Mar 7 2014, 12:20 PM
  • Bug 60435 has been marked as a duplicate of this bug. ***
Nemo_bis added a comment.Mar 7 2014, 2:10 PM

I wonder if this has to do with insufficient caching: if we're hammering the localisation cache in MySQL for hundreds revisions per second, I wouldn't be surprised that something goes wrong.
In an instance set up by role::mediawiki-install::labs , I now uncommented

$wgCacheDirectory = "$IP/cache";

and

php maintenance/rebuildLocalisationCache.php --force

took only few seconds instead of minutes because it uses CDB files (I first checked write permissions in cache/ directory, owned by www-data rather than root).
From a cursory view of top, it seems that now the php process running importDump is as busy as before, but mysqld is one order of magnitude less loaded.

(In reply to Yuvi Panda from comment #3)

Also affects testwikidatawiki

What caching system did it use?

ori added a comment.May 10 2014, 1:06 PM

Adding libxml_disable_entity_loader(false); to the top of LocalisationCache::loadPluralFile makes the warning go away. We disable the entity loader for a reason (it's insecure), so we can't use that as a solution. But it does finger entity references as most likely being the underlying cause.

Andrew added a comment.May 10 2014, 4:48 PM

I have a fairly trivial import that fails in the same way.

https://github.com/wikimedia/operations-puppet/blob/production/modules/mediawiki_singlenode/files/privacy-policy.xml

TTO added a comment.Jul 26 2014, 7:40 AM

Apparently caused by the fact that libxml_disable_entity_loader is not thread-safe. Great one, PHP!

See https://bugs.php.net/bug.php?id=64938 and http://pyd.io/f/topic/failed-to-load-external-entity-boot-confmanifest-xml

gerritbot added a comment.Jul 26 2014, 7:41 AM

Change 149507 had a related patch set uploaded by TTO:
Use file_get_contents to load plurals XML data

https://gerrit.wikimedia.org/r/149507

TTO added a comment.Jul 26 2014, 7:43 AM
  • Bug 65116 has been marked as a duplicate of this bug. ***
matmarex added a comment.Sep 19 2014, 3:34 PM

According to comments on the patch, it is not clear if it actually fixes the issue or not.

TTO added a comment.Sep 19 2014, 10:40 PM

Not sure if any of those commenters actually tested it...

gerritbot added a comment.Nov 26 2014, 9:02 AM

Change 149507 had a related patch set uploaded (by TTO):
LocalisationCache: Use file_get_contents instead of DOMDocument::load

https://gerrit.wikimedia.org/r/149507

Patch-For-Review

gerritbot added a comment.Dec 14 2014, 6:38 AM

Change 149507 merged by jenkins-bot:
LocalisationCache: Use file_get_contents instead of DOMDocument::load

https://gerrit.wikimedia.org/r/149507

• Mattflaschen-WMF mentioned this in rMWc13912684145: LocalisationCache: Use file_get_contents instead of DOMDocument::load.Dec 14 2014, 6:39 AM
• Mattflaschen-WMF closed this task as Resolved.Jan 12 2015, 4:20 AM
TTO mentioned this in T86036: Import: XMLReader::open(): Unable to open source data.Jan 12 2015, 5:17 AM
• Phabricator_maintenance removed a subscriber: yuvipanda.Jun 7 2017, 7:02 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL