Page MenuHomePhabricator

Speed up MediaWiki PHPUnit build by running integration tests in parallel
Open, HighPublic

Description

Background

Long ago, our Jenkins configuration used to include multiple jobs that run PHPUnit on MediaWiki core commits. Each job ran a portion of the tests (by passing a filter or subdirectory to the phpunit command), thus allowing them to run concurrently. Over the years, as things have been optimised and improved, these were eventually merged into one large job that basically "just" runs phpunit. This is how it has been for about 7 years now.

As per T225730: Reduce runtime of MW shared gate Jenkins jobs to 5 min, in the last 5 years the job has gotten significantly slower due to growth in how many tests we run, and so now the time developers spend waiting for a CI response to a MediaWiki commit, is spent in the PHPUnit job, and so we want to make it faster again. We believe the job is not exhausting its allocated resources and could run much faster, if it was parallelised somehow. By approaching it as a single job with concurrency (instead of splitting the jobs) we have two other benefits: 1) Keeps CI configuration simple, 2) Means it will also become fast by default for developers running tests locally.

Work

Look into software that would help running these in parallel within the job (separate threads/processes)

See also:
T60771: Jenkins: MediaWiki extensions phpunit test should also run mediawiki core tests

Update in 2022 by @aaron:

  • The most promising so far is paratest.
  • A notable restriction is that it does not support a wrapped phpunit command. MediaWiki currently wraps phpunit in tests/phpunit.php, being fixed as part of T90875 is a blocker unless we use paratest 1.x (no longer maintained), which did support custom wrappers.
  • Distribution of tests between subproceses should be invisible to developers in practice. There are multiple ways to do it, e.g.
    • by "suite" (per top-level subdir of tests/phpunit/includes),
    • by class (one Test.php file),
    • by function (individual test cases, including such that e.g. the "setUpBeforeClass" hooks will run multiple times, once in each process that end sup running one or more of the test functions, and even individual data provided cases could end up split and e.g. clash or do more work than is needed). The "by function" is known to have many failures with our current setup and would take much effort to make pass and keep passing.

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
Resolvedkostajh
Resolvedkostajh
ResolvedKrinkle
Resolvedkostajh
Resolvedkostajh
Resolvedkostajh
ResolvedDaimona
Resolvedkostajh
ResolvedDaimona
Resolveddaniel
ResolvedBUG REPORTkostajh
ResolvedDaimona
ResolvedDaimona
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

I'd like to revisit this as an experiment using [fastest](https://github.com/liuggio/fastest). In my local environment using MediaWiki-Docker-Dev, running the full test suite takes 49 minutes. With fastest, it takes 39 minutes:

root@68e6b761fef5:/var/www/mediawiki# find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"
find: ‘extensions/**/tests/phpunit’: No such file or directory
find: ‘skins/**/tests/phpunit’: No such file or directory
- 662 shuffled test classes into the queue.
- Will be consumed by 4 parallel Processes.




662/662 [============================] 100% 39 mins 6.0 MiB

     8 failures.
[4] tests/phpunit/tests/MediaWikiTestCaseSchema2Test.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

E

Time: 2.62 seconds, Memory: 30.00MB

There was 1 error:

1) MediaWikiTestCaseSchema2Test::testMediaWikiTestCaseSchemaTestOrder
Error: Class 'MediaWikiTestCaseSchema1Test' not found

/var/www/mediawiki/tests/phpunit/tests/MediaWikiTestCaseSchema2Test.php:25
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

ERRORS!
Tests: 1, Assertions: 0, Errors: 1.

[2] tests/phpunit/includes/specials/SpecialShortpagesTest.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

F

Time: 2.52 seconds, Memory: 30.00MB

There was 1 failure:

1) SpecialShortpagesTest::testGetQueryInfoRespectsContentNS with data set #0 (array(0, 6), array(), array(0, 6))
=== Logs generated by test case
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
[wfDebug] [debug] SpecialPage::getContext called and $mContext is null. Return RequestContext::getMain(); for sanity {"private":false}
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
===
Failed asserting that two arrays are equal.
--- Expected
+++ Actual
@@ @@
 Array (
     0 => 0
-    1 => 6

/var/www/mediawiki/tests/phpunit/includes/specials/SpecialShortpagesTest.php:30
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

FAILURES!
Tests: 1, Assertions: 3, Failures: 1.

[1] tests/phpunit/includes/search/SearchResultTest.php
Fatal error: Class 'MediawikiTestCase' not found in /var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php on line 3

Call Stack:
    0.0087     406392   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
    0.0299     698800   2. require('/var/www/mediawiki/maintenance/doMaintenance.php') /var/www/mediawiki/tests/phpunit/phpunit.php:129
    0.3754   11037136   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
    0.4046   11902112   4. MediaWikiPHPUnitCommand->run(???, ???) /var/www/mediawiki/tests/phpunit/phpunit.php:90
    0.4508   12630272   5. MediaWikiTestRunner->getTest(???, ???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
    0.4511   12630272   6. MediaWikiTestRunner->loadSuiteClass(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
    0.4527   12647448   7. PHPUnit\Runner\StandardTestSuiteLoader->load(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
    0.4529   12667984   8. PHPUnit\Util\Fileloader::checkAndLoad(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
    0.4534   12668240   9. PHPUnit\Util\Fileloader::load(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
    0.4541   12679072  10. include_once('/var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php') /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

Using PHP 7.2.14
PHP Fatal error:  Class 'MediawikiTestCase' not found in /var/www/mediawiki/tests/phpunit/includes/search/SearchResultTest.php on line 3
PHP Stack trace:
PHP   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
PHP   2. require() /var/www/mediawiki/tests/phpunit/phpunit.php:129
PHP   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
PHP   4. MediaWikiPHPUnitCommand->run($argv = *uninitialized*, $exit = *uninitialized*) /var/www/mediawiki/tests/phpunit/phpunit.php:90
PHP   5. MediaWikiTestRunner->getTest($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*, $suffixes = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
PHP   6. MediaWikiTestRunner->loadSuiteClass($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
PHP   7. PHPUnit\Runner\StandardTestSuiteLoader->load($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
PHP   8. PHPUnit\Util\Fileloader::checkAndLoad($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
PHP   9. PHPUnit\Util\Fileloader::load($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
PHP  10. include_once() /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

[1] tests/phpunit/includes/libs/GenericArrayObjectTest.phpPHPUnit\Runner\Exception from line 102 of /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php: Class 'tests/phpunit/includes/libs/GenericArrayObjectTest' could not be found in '/var/www/mediawiki/tests/phpunit/includes/libs/GenericArrayObjectTest.php'.
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(130): PHPUnit\Runner\StandardTestSuiteLoader->load('tests/phpunit/i...', '/var/www/mediaw...')
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(73): PHPUnit\Runner\BaseTestRunner->loadSuiteClass('tests/phpunit/i...', '/var/www/mediaw...')
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest('tests/phpunit/i...', '/var/www/mediaw...', Array)
#3 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(Array, true)
#4 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require('/var/www/mediaw...')
#6 {main}
Using PHP 7.2.14

[3] tests/phpunit/suites/ParserIntegrationTest.phpUsing PHP 7.2.14
[4ca2f19e580cc04e08f5ae71] [no req]   ArgumentCountError from line 44 of /var/www/mediawiki/tests/phpunit/suites/ParserIntegrationTest.php: Too few arguments to function ParserIntegrationTest::__construct(), 0 passed in /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php on line 591 and exactly 3 expected
Backtrace:
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(591): ParserIntegrationTest->__construct()
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(882): PHPUnit\Framework\TestSuite::createTest(ReflectionClass, string)
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(187): PHPUnit\Framework\TestSuite->addTestMethod(ReflectionClass, ReflectionMethod)
#3 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(106): PHPUnit\Framework\TestSuite->__construct(ReflectionClass)
#4 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest(string, string, array)
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(array, boolean)
#6 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#7 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require(string)
#8 {main}

[3] tests/phpunit/LessFileCompilationTest.phpUsing PHP 7.2.14
[b865182a94074440ab541361] [no req]   ArgumentCountError from line 28 of /var/www/mediawiki/tests/phpunit/LessFileCompilationTest.php: Too few arguments to function LessFileCompilationTest::__construct(), 0 passed in /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php on line 591 and exactly 2 expected
Backtrace:
#0 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(591): LessFileCompilationTest->__construct()
#1 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(882): PHPUnit\Framework\TestSuite::createTest(ReflectionClass, string)
#2 /var/www/mediawiki/vendor/phpunit/phpunit/src/Framework/TestSuite.php(187): PHPUnit\Framework\TestSuite->addTestMethod(ReflectionClass, ReflectionMethod)
#3 /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php(106): PHPUnit\Framework\TestSuite->__construct(ReflectionClass)
#4 /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php(169): PHPUnit\Runner\BaseTestRunner->getTest(string, string, array)
#5 /var/www/mediawiki/tests/phpunit/phpunit.php(90): PHPUnit\TextUI\Command->run(array, boolean)
#6 /var/www/mediawiki/maintenance/doMaintenance.php(99): PHPUnitMaintClass->execute()
#7 /var/www/mediawiki/tests/phpunit/phpunit.php(129): require(string)
#8 {main}

[4] tests/phpunit/docs/ExportDemoTest.php
Fatal error: Class 'DumpTestCase' not found in /var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php on line 11

Call Stack:
    0.0118     406360   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
    0.0490     698768   2. require('/var/www/mediawiki/maintenance/doMaintenance.php') /var/www/mediawiki/tests/phpunit/phpunit.php:129
    1.0348   11037104   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
    1.0749   11902080   4. MediaWikiPHPUnitCommand->run(???, ???) /var/www/mediawiki/tests/phpunit/phpunit.php:90
    1.2131   12630224   5. MediaWikiTestRunner->getTest(???, ???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
    1.2143   12630224   6. MediaWikiTestRunner->loadSuiteClass(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
    1.2188   12647400   7. PHPUnit\Runner\StandardTestSuiteLoader->load(???, ???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
    1.2191   12667936   8. PHPUnit\Util\Fileloader::checkAndLoad(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
    1.2214   12668192   9. PHPUnit\Util\Fileloader::load(???) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
    1.2236   12672704  10. include_once('/var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php') /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

Using PHP 7.2.14
PHP Fatal error:  Class 'DumpTestCase' not found in /var/www/mediawiki/tests/phpunit/docs/ExportDemoTest.php on line 11
PHP Stack trace:
PHP   1. {main}() /var/www/mediawiki/tests/phpunit/phpunit.php:0
PHP   2. require() /var/www/mediawiki/tests/phpunit/phpunit.php:129
PHP   3. PHPUnitMaintClass->execute() /var/www/mediawiki/maintenance/doMaintenance.php:99
PHP   4. MediaWikiPHPUnitCommand->run($argv = *uninitialized*, $exit = *uninitialized*) /var/www/mediawiki/tests/phpunit/phpunit.php:90
PHP   5. MediaWikiTestRunner->getTest($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*, $suffixes = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/TextUI/Command.php:169
PHP   6. MediaWikiTestRunner->loadSuiteClass($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:73
PHP   7. PHPUnit\Runner\StandardTestSuiteLoader->load($suiteClassName = *uninitialized*, $suiteClassFile = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/BaseTestRunner.php:130
PHP   8. PHPUnit\Util\Fileloader::checkAndLoad($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Runner/StandardTestSuiteLoader.php:43
PHP   9. PHPUnit\Util\Fileloader::load($filename = *uninitialized*) /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:48
PHP  10. include_once() /var/www/mediawiki/vendor/phpunit/phpunit/src/Util/Fileloader.php:64

[4] tests/phpunit/includes/search/SearchEngineTest.phpUsing PHP 7.2.14
PHPUnit 6.5.14 by Sebastian Bergmann and contributors.

.........F

Time: 38.75 seconds, Memory: 34.00MB

There was 1 failure:

1) SearchEngineTest::testCompletionSearchMustRespectCapitalLinkOverrides with data set "Searching for "search is" will finds "search is not Search" on NS_CATEGORY" ('search is', 'Category:search is not Search', array(14))
=== Logs generated by test case
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
[wfDebug] [debug] LocalisationCache::isExpired(en): cache missing, need to make one {"private":false}
[wfDebug] [debug] LocalisationCache::recache: got localisation for en from source {"private":false}
[caches] [info] LocalisationCache: using store LCStoreNull {"private":false}
===
Failed asserting that 0 matches expected 1.

/var/www/mediawiki/tests/phpunit/includes/search/SearchEngineTest.php:265
/var/www/mediawiki/tests/phpunit/MediaWikiTestCase.php:427
/var/www/mediawiki/maintenance/doMaintenance.php:99

FAILURES!
Tests: 10, Assertions: 34, Failures: 1.


    ✘ ehm broken tests...
    Time: 39 minutes 21 seconds 130 milliseconds, Memory: 6.00 MiB

In our CI environment where we have 8 vCPUs, this might go a lot faster.

There are some failures above due to expectation of running tests serially (e.g. testMediaWikiTestCaseSchemaTestOrder) but overall the test failures here don't seem to be insurmountable.

This might make more sense to pursue further after T90875: Use vendor/bin/phpunit instead of tests/phpunit/phpunit.php is done.

My rough plan would be:

  1. Patch quibble so that it accepts a --phpunit-command argument. (That will also be needed for T90875, so that you could run quibble --phpunt-command=vendor/bin/phpunit.)
  2. Submit a patch for integration/config with quibble --phpunit-command='find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"'
  3. Create a patch for core which adds https://github.com/liuggio/fastest to require-dev
  4. Pair with someone to do a test run in CI to check speed
kostajh renamed this task from Speed up MediaWiki PHPUnit build by running tests in parallel to Speed up MediaWiki PHPUnit build by running integration tests in parallel.Jun 17 2019, 5:55 PM
kostajh added subscribers: Ladsgroup, awight.

There are a few gotchas though. Some testsuite define tests which would not be found by globbing files. Typically the parser tests. There are also alternative ways to register test files via the UnitTest hook.

running the full test suite takes 49 minutes. With fastest, it takes 39 minutes:

That suggest the suite is not that much CPU bound or surely running four in parallels would be way faster?

root@68e6b761fef5:/var/www/mediawiki# find tests/phpunit extensions/**/tests/phpunit skins/**/tests/phpunit -name "*Test.php" | ./vendor/liuggio/fastest/fastest "tests/phpunit/phpunit.php {};"
find: ‘extensions/**/tests/phpunit’: No such file or directory
find: ‘skins/**/tests/phpunit’: No such file or directory

From the command line, that seems to suggest to run test classes from mediawiki/core as well as all extensions and skins. Then it does not find any file for extensions or skins. So is that solely for mediawiki/core tests? And if so why does it take 49 minutes when being run serially? It should be ten time faster (3 mins 15 s on my computer with hhvm).

In our CI environment where we have 8 vCPUs, this might go a lot faster.

The devil there is that we do not have the capacity to add 8 times more CPU on the WMCS infra. Even if instances have 8 vCPUS, they currently can run up to 4 jobs in parallel and we still need lot of CPU cycles for eg MySQL :]

Change 581131 had a related patch set uploaded (by Krinkle; owner: Aaron Schulz):
[mediawiki/core@master] tests: Split includes/ tests into more suites for tools like paratest

https://gerrit.wikimedia.org/r/581131

Change 637574 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Move ApiQuery* tests under the /query subdirectory

https://gerrit.wikimedia.org/r/637574

Change 637574 merged by jenkins-bot:
[mediawiki/core@master] Move ApiQuery* tests under the /query subdirectory

https://gerrit.wikimedia.org/r/637574

Krinkle added subscribers: aaron, thcipriani.
@aaron wrote at T269894

Something like the phpunit/paratest wrapper at https://github.com/AaronSchulz/mediawiki-developer-tools would be nice to have in MediaWiki core.

  • Merge mediawiki-developer-tools suite.xml optimizations (or an improved version) into core (these breaks up the tests into more suites that can be run in parallel)
  • Add support for cloning to new sqlite DB files in CloneDatabase; this will replace wfCloneSqliteSchemaForParatest() from mediawiki-developer-tools
  • Add paratest to composer.json
  • Merge ParatestWrapperSettings.php into core from mediawiki-developer-tools
  • Include a simplified version of mediawiki-phpunit from mediawiki-developer-tools in core (e.g. just for "dev_phpunit" and "dev_paratest" modes)

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

To explain a bit about what I'm trying here:

  • In the short term, I'm skeptical about adding paratest or other wrappers like https://github.com/liuggio/fastest until T90875: Use vendor/bin/phpunit instead of tests/phpunit/phpunit.php is done; it seems like adding more code to wrap around tests/phpunit/phpunit.php will make removing it in favor of vendor/bin/phpunit that much harder. Longer term, having paratest would be nice to faciliate local test execution.
  • we have some parallelism utility code in Quibble that seemed simple to adapt for PHPUnit. In fact, it is quite easy to adapt, but it turns out that a bunch of our tests don't pass when run out of order, on their own, or even when they are executed with --testsuite={testSuiteName} ==testsuite={someOtherTestSuiteName}. So as @Krinkle linked to above, I'm trying to fix tests and making subtasks as needed under T297078: PHPUnit tests should pass when run on their own.

Change 742199 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Support splitting extension suite

https://gerrit.wikimedia.org/r/742199

Change 742200 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

https://gerrit.wikimedia.org/r/742200

To explain a bit about what I'm trying here:

  • In the short term, I'm skeptical about adding paratest or other wrappers like https://github.com/liuggio/fastest until T90875: Use vendor/bin/phpunit instead of tests/phpunit/phpunit.php is done; it seems like adding more code to wrap around tests/phpunit/phpunit.php will make removing it in favor of vendor/bin/phpunit that much harder. Longer term, having paratest would be nice to faciliate local test execution.
  • we have some parallelism utility code in Quibble that seemed simple to adapt for PHPUnit. In fact, it is quite easy to adapt, but it turns out that a bunch of our tests don't pass when run out of order, on their own, or even when they are executed with --testsuite={testSuiteName} ==testsuite={someOtherTestSuiteName}. So as @Krinkle linked to above, I'm trying to fix tests and making subtasks as needed under T297078: PHPUnit tests should pass when run on their own.

The last ones to fix (which are also harder because I can't reproduce them breaking when running locally) from https://integration.wikimedia.org/ci/job/integration-quibble-fullrun/504/console:

21:31:19 Time: 1.64 minutes, Memory: 904.50 MB
21:31:19 
21:31:19 There was 1 error:
21:31:19 
21:31:19 1) MediaWikiServicesTest::testDefaultServiceInstantiation
21:31:19 Use of GrowthExperimentsTaskSuggester service was deprecated in GrowthExperiments 1.35. [Called from Wikimedia\Services\ServiceContainer::createService in /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php at line 447]
21:31:19 
21:31:19 /workspace/src/includes/debug/MWDebug.php:377
21:31:19 /workspace/src/includes/debug/MWDebug.php:353
21:31:19 /workspace/src/includes/debug/MWDebug.php:232
21:31:19 /workspace/src/includes/GlobalFunctions.php:1005
21:31:19 /workspace/src/extensions/GrowthExperiments/ServiceWiring.php:589
21:31:19 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php:447
21:31:19 /workspace/src/vendor/wikimedia/services/src/ServiceContainer.php:416
21:31:19 /workspace/src/includes/MediaWikiServices.php:291
21:31:19 /workspace/src/tests/phpunit/includes/MediaWikiServicesTest.php:386
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [GlobalTitleFail] [info] RequestContext::getTitle called with no title set. {"exception":{}}
21:31:19 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
21:31:19 [ParserCache] [debug] Creating ParserCache instance for pcache []
21:31:19 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
21:31:19 [objectcache] [debug] fetchOrRegenerate(wikidbA-unittest_:page:8:0e6409a4846e9aee0a73fd5dfb53dd5e9839c4c6): miss, new value computed []
21:31:19 ===
21:31:19 
21:31:19 --
21:31:19 
21:31:19 There were 12 failures:
21:31:19 
21:31:19 1) MediaWiki\Tests\Revision\RevisionRendererTest::testGetRenderedRevision_multi
21:31:19 slot header
21:31:19 Failed asserting that '<div class="mw-parser-output"><p><a href="/index.php?title=Kittens&amp;action=edit&amp;redlink=1" class="new" title="Kittens (page does not exist)">Kittens</a>\n
21:31:19 </p><h1 class="mw-slot-header"><mediainfoslotheader /></h1><p><a href="/index.php?title=Goats&amp;action=edit&amp;redlink=1" class="new" title="Goats (page does not exist)">Goats</a>\n
21:31:19 </p></div>' contains ">aux<".
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/Revision/RevisionRendererTest.php:420
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
21:31:19 [localisation] [debug] LocalisationCache::isExpired(en): cache missing, need to make one []
21:31:19 [wfDebug] [debug] ParserFactory: using default preprocessor {"private":false}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"thisWikiIsTheRepo","logValue":"true"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'entityNamespaces' => \n    array (\n      'item' => 120,\n      'property' => 122,\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'repoDatabase' => false,\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"entitySources","logValue":"array (\n  'local' => \n  array (\n    'repoDatabase' => false,\n    'entityNamespaces' => \n    array (\n      'item' => '120\/main',\n      'property' => '122\/main',\n      'mediainfo' => '6\/mediainfo',\n    ),\n    'baseUri' => 'http:\/\/127.0.0.1:9412\/entity\/',\n    'rdfNodeNamespacePrefix' => 'wd',\n    'rdfPredicateNamespacePrefix' => '',\n    'interwikiPrefix' => '',\n  ),\n)"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"itemAndPropertySourceName","logValue":"'local'"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGlobalID","logValue":"'wikidbA'"}
21:31:19 [MessageCache] [debug] MessageCache using store {class} {"class":"HashBagOStuff"}
21:31:19 [Wikibase] [debug] {method}: setting {settingName} was given as a closure, resolve it to {logValue} {"method":"Wikibase\\Lib\\SettingsArray::getSetting","settingName":"siteGroup","logValue":"NULL"}
21:31:19 [DuplicateParse] [debug] MediaWiki\Parser\ParserObserver::notifyParse: Possibly redundant parse! {"page":"ns0:MediaWiki\\Tests\\Revision\\RevisionRendererTest","rev":null,"options-hash":"canonical","trace":"#0 \/workspace\/src\/includes\/content\/ContentHandler.php(1728): MediaWiki\\Parser\\ParserObserver->notifyParse(Object(Title), NULL, Object(ParserOptions), Object(ParserOutput))\n#1 \/workspace\/src\/includes\/content\/Renderer\/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Content\\Renderer\\ContentParseParams))\n#2 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(271): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Page\\PageIdentityValue), NULL, Object(ParserOptions), true)\n#3 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(238): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(Object(WikitextContent), true)\n#4 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(236): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput('aux', Array)\n#5 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(158): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#6 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#7 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(200): call_user_func(Object(Closure), Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#8 \/workspace\/src\/tests\/phpunit\/includes\/Revision\/RevisionRendererTest.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#9 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1472): MediaWiki\\Tests\\Revision\\RevisionRendererTest->testGetRenderedRevision_multi()\n#10 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1092): PHPUnit\\Framework\\TestCase->runTest()\n#11 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestResult.php(703): PHPUnit\\Framework\\TestCase->runBare()\n#12 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(820): PHPUnit\\Framework\\TestResult->run(Object(MediaWiki\\Tests\\Revision\\RevisionRendererTest))\n#13 \/workspace\/src\/tests\/phpunit\/MediaWikiIntegrationTestCase.php(456): PHPUnit\\Framework\\TestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#14 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): MediaWikiIntegrationTestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#15 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#16 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#17 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/TestRunner.php(656): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#18 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/Command.php(235): PHPUnit\\TextUI\\TestRunner->doRun(Object(PHPUnit\\Framework\\TestSuite), Array, Array, true)\n#19 \/workspace\/src\/tests\/phpunit\/phpunit.php(141): PHPUnit\\TextUI\\Command->run(Array, true)\n#20 \/workspace\/src\/tests\/phpunit\/phpunit.php(207): PHPUnitMaintClass->execute()\n#21 {main}","previous-trace":"#0 \/workspace\/src\/includes\/content\/ContentHandler.php(1728): MediaWiki\\Parser\\ParserObserver->notifyParse(Object(Title), NULL, Object(ParserOptions), Object(ParserOutput))\n#1 \/workspace\/src\/includes\/content\/Renderer\/ContentRenderer.php(47): ContentHandler->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Content\\Renderer\\ContentParseParams))\n#2 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(271): MediaWiki\\Content\\Renderer\\ContentRenderer->getParserOutput(Object(WikitextContent), Object(MediaWiki\\Page\\PageIdentityValue), NULL, Object(ParserOptions), true)\n#3 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(238): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutputUncached(Object(WikitextContent), true)\n#4 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(236): MediaWiki\\Revision\\RenderedRevision->getSlotParserOutput('main', Array)\n#5 \/workspace\/src\/includes\/Revision\/RevisionRenderer.php(158): MediaWiki\\Revision\\RevisionRenderer->combineSlotOutput(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#6 [internal function]: MediaWiki\\Revision\\RevisionRenderer->MediaWiki\\Revision\\{closure}(Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#7 \/workspace\/src\/includes\/Revision\/RenderedRevision.php(200): call_user_func(Object(Closure), Object(MediaWiki\\Revision\\RenderedRevision), Array)\n#8 \/workspace\/src\/tests\/phpunit\/includes\/Revision\/RevisionRendererTest.php(406): MediaWiki\\Revision\\RenderedRevision->getRevisionParserOutput()\n#9 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1472): MediaWiki\\Tests\\Revision\\RevisionRendererTest->testGetRenderedRevision_multi()\n#10 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(1092): PHPUnit\\Framework\\TestCase->runTest()\n#11 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestResult.php(703): PHPUnit\\Framework\\TestCase->runBare()\n#12 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestCase.php(820): PHPUnit\\Framework\\TestResult->run(Object(MediaWiki\\Tests\\Revision\\RevisionRendererTest))\n#13 \/workspace\/src\/tests\/phpunit\/MediaWikiIntegrationTestCase.php(456): PHPUnit\\Framework\\TestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#14 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): MediaWikiIntegrationTestCase->run(Object(PHPUnit\\Framework\\TestResult))\n#15 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#16 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/Framework\/TestSuite.php(627): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#17 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/TestRunner.php(656): PHPUnit\\Framework\\TestSuite->run(Object(PHPUnit\\Framework\\TestResult))\n#18 \/workspace\/src\/vendor\/phpunit\/phpunit\/src\/TextUI\/Command.php(235): PHPUnit\\TextUI\\TestRunner->doRun(Object(PHPUnit\\Framework\\TestSuite), Array, Array, true)\n#19 \/workspace\/src\/tests\/phpunit\/phpunit.php(141): PHPUnit\\TextUI\\Command->run(Array, true)\n#20 \/workspace\/src\/tests\/phpunit\/phpunit.php(207): PHPUnitMaintClass->execute()\n#21 {main}"}
21:31:19 ===
21:31:19 
21:31:19 2) MediaWiki\Tests\Parser\ParserCacheTest::testJsonEncodeUnicode
21:31:19 Failed asserting that '' contains "Э".
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/parser/ParserCacheTest.php:762
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 [localisation] [debug] LocalisationCache using store LCStoreNull []
21:31:19 [objectcache] [debug] MainObjectStash using store {class} {"class":"HashBagOStuff"}
21:31:19 [ContentHandler] [info] Registered handler for wikitext: WikitextContentHandler []
21:31:19 ===
21:31:19 
21:31:19 3) NamespaceInfoTest::testGetCanonicalNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      3 => 'User_talk'
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:842
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 4) NamespaceInfoTest::testGetValidNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:896
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 5) NamespaceInfoTest::testGetCanonicalNamespaces_NoCanonicalNamespaceNames
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19  Array &0 (
21:31:19      0 => ''
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:911
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 6) NamespaceInfoTest::testGetValidNamespaces_NoCanonicalNamespaceNames
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19  Array &0 (
21:31:19      0 => 0
21:31:19 +    1 => 250
21:31:19 +    2 => 251
21:31:19 +    3 => 252
21:31:19 +    4 => 253
21:31:19 +    5 => 710
21:31:19 +    6 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:940
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 7) NamespaceInfoTest::testGetCanonicalNamespaces_ExtensionNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19      12345 => 'Extended'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:963
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 8) NamespaceInfoTest::testGetValidNamespaces_ExtensionNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 -    4 => 12345
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19 +    10 => 12345
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:999
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 9) NamespaceInfoTest::testGetCanonicalNamespaces_ExtraNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19      1234567 => 'Extra'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1089
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 10) NamespaceInfoTest::testGetValidNamespaces_ExtraNamespaces
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 -    4 => 1234567
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19 +    10 => 1234567
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1121
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 11) NamespaceInfoTest::testGetCanonicalNamespaces_caching
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      3 => 'User_talk'
21:31:19      -1 => 'Special'
21:31:19      -2 => 'Media'
21:31:19 +    250 => 'Page'
21:31:19 +    251 => 'Page_talk'
21:31:19 +    252 => 'Index'
21:31:19 +    253 => 'Index_talk'
21:31:19 +    710 => 'TimedText'
21:31:19 +    711 => 'TimedText_talk'
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1144
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===
21:31:19 
21:31:19 12) NamespaceInfoTest::testGetValidNamespaces_caching
21:31:19 Failed asserting that two arrays are identical.
21:31:19 --- Expected
21:31:19 +++ Actual
21:31:19 @@ @@
21:31:19      1 => 1
21:31:19      2 => 2
21:31:19      3 => 3
21:31:19 +    4 => 250
21:31:19 +    5 => 251
21:31:19 +    6 => 252
21:31:19 +    7 => 253
21:31:19 +    8 => 710
21:31:19 +    9 => 711
21:31:19  )
21:31:19 
21:31:19 /workspace/src/tests/phpunit/includes/title/NamespaceInfoTest.php:1203
21:31:19 /workspace/src/tests/phpunit/MediaWikiIntegrationTestCase.php:456
21:31:19 /workspace/src/tests/phpunit/phpunit.php:141
21:31:19 /workspace/src/tests/phpunit/phpunit.php:207
21:31:19 === Logs generated by test case
21:31:19 [objectcache] [debug] MainWANObjectCache using store {class} {"class":"EmptyBagOStuff"}
21:31:19 ===

Change 758784 had a related patch set uploaded (by Kosta Harlan; author: Kosta Harlan):

[mediawiki/core@master] [WIP] phpunit: Use process ID in test DB prefix

https://gerrit.wikimedia.org/r/758784

Change 581131 abandoned by Aaron Schulz:

[mediawiki/core@master] tests: split includes/ tests into suites for tools like paratest

Reason:

Going with automatic generation from paratest wrapper

https://gerrit.wikimedia.org/r/581131

Change 763650 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] phpunit: Remove redundancy from \"skins\" test and \"extensions\" suites

https://gerrit.wikimedia.org/r/763650

Change 763650 abandoned by Aaron Schulz:

[mediawiki/core@master] phpunit: Remove redundancy from \"skins\" test and \"extensions\" suites

Reason:

Worked around this in paratest scripts

https://gerrit.wikimedia.org/r/763650

Change 763651 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Avoid unsafe use of setUpBeforeClass() in ApiFormatXmlTest

https://gerrit.wikimedia.org/r/763651

Change 767232 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Split out DatabaseSqliteUpgradeTest class

https://gerrit.wikimedia.org/r/767232

Change 763651 merged by jenkins-bot:

[mediawiki/core@master] tests: Avoid unsafe use of setUpBeforeClass() in ApiFormatXmlTest

https://gerrit.wikimedia.org/r/763651

Change 767232 merged by jenkins-bot:

[mediawiki/core@master] tests: Split out DatabaseSqliteUpgradeTest class

https://gerrit.wikimedia.org/r/767232

Change 776058 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] phpunit: parse any --boostrap parameter in getopt() call

https://gerrit.wikimedia.org/r/776058

Change 776058 merged by jenkins-bot:

[mediawiki/core@master] phpunit: parse any --boostrap parameter in getopt() call

https://gerrit.wikimedia.org/r/776058

Change 793140 had a related patch set uploaded (by Krinkle; author: Aaron Schulz):

[mediawiki/core@master] tests: Fix memcached test failure with multiple BagOStuff test classes

https://gerrit.wikimedia.org/r/793140

Change 793140 merged by jenkins-bot:

[mediawiki/core@master] tests: Fix memcached test failure with multiple BagOStuff test classes

https://gerrit.wikimedia.org/r/793140

Now that T90875 is almost done, I think this should be unblocked. We still have some cleanup to do for T90875, but that's mostly documentation; and then there's T227900 for having a single config and bootstrap, but I don't know if that would affect paratest.

I've fiddled a bit with paratest locally. It's easier (albeit almost useless) with unit tests because our unit tests are pretty self-contained as they should be and do not need a database, so paratest works out of the box for them. composer phpunit:unit took 8.6 seconds, whereas vendor/bin/paratest --colors=always --testsuite=core:unit,extensions:unit,skins:unit --runner WrapperRunner took 3.5 seconds (with 16 processes).

I then wanted to try integration (non-database) tests. At the moment this doesn't work out of the box due to T155147 causing some DB errors, but I checked out r937980 and applied r938387 to skip the DB setup. This results in a lot of errors, but at least all the tests can run. vendor/bin/phpunit --exclude-group Broken,ParserFuzz,Stub,Database,Standalone took 3m36s, whereas vendor/bin/paratest --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner took 36 seconds. For some reason the paratest run executed 26k tests instead of 30k and I don't know why, but I'm going to assume (and hope) that the 4k tests that didn't run for some reason aren't just the most expensive ones. Also, when I restricted the test run to the "includes" suite, it went from 1m5s to 19s.

Note that the "slow tests" report and our extension which prints logs after failures don't work with paratest (https://github.com/paratestphp/paratest/issues/771), but we that can be addressed later, when our tests will really be ready for parallelization.

I think paratest is one of the most promising things we can try for T225730. By splitting the suites we may be able to improve the run time even further. I don't know if the end result will really be a 6x speedup, but we should really, really try and get this done.

Adding to the above, the "missing" tests are from the extensions and skins suites (particularly unit tests, but also integration tests, I think). I suspect it might be because of our hacky subclassing of TestSuite, which we should really get rid of some day. If I remove those 4 suites from phpunit.xml.dist (as well as the core:unit suite, which is executed with or without paratest but shouldn't really add much), both runs have 14231 tests, and the time for a full run goes from 2m59s to 35 seconds. Another thing I noticed is that paratest seems to hang for roughly 10 seconds at some point towards the end; however, I think it's not due to a slow test, but rather due to it putting together the output from all processes (just an ineducated guess!). And since there are ~1000 failures due to tests using the database when they shouldn't, there's definitely a lot of output to put together and print. I'm quite convinced that this is the case because as soon as it unfreezes, it prints all the results. But I don't have a quick way to test that, and the random hacks I made to the paratest ResultPrinter didn't fix this. But we'll see better once T155147 is done.

Daimona raised the priority of this task from Low to High.Jul 19 2023, 3:01 PM

Another thing I noticed is that paratest seems to hang for roughly 10 seconds at some point towards the end; however, I think it's not due to a slow test, but rather due to it putting together the output from all processes (just an ineducated guess!). And since there are ~1000 failures due to tests using the database when they shouldn't, there's definitely a lot of output to put together and print. I'm quite convinced that this is the case because as soon as it unfreezes, it prints all the results. But I don't have a quick way to test that, and the random hacks I made to the paratest ResultPrinter didn't fix this. But we'll see better once T155147 is done.

I was wrong. It is actually LanguageConverterFactoryTest which takes something like 11 seconds on its own with paratest. I think it might be simply because it's a large test, but I haven't looked closely. At any rate, I've pulled r938387, applied r939777 to cut down on DB usage, deleted LanguageConverterFactoryTest.php and re-ran the includes suite (7054 tests).

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

38 seconds.

$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

8.5 seconds.

New benchmark! This time with r938387 checked out, and all non-database tests passing when DB access is disabled. Once again, I've also disabled LanguageConverterFactoryTest (by marking it as skipped).

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

Time: 00:39.145, Memory: 521.00 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 14014, Skipped: 607.
$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

Time: 00:07.781, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 14014, Skipped: 607.

This seems to confirm the 5x speedup observed previously.

When adding LanguageConverterFactoryTest back, the results are respectively

Time: 00:53.573, Memory: 523.00 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 15916, Skipped: 156.

and

Time: 00:16.575, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 15916, Skipped: 156.

which kinda sucks, but we can think about it later on.

And it's time again for a new benchmark :) Thanks to T342418, language creation is now much faster and so is LanguageConverterFactoryTest. Therefore, for the full 'includes' suite as-is:

$ vendor/bin/phpunit -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone

Time: 00:39.939, Memory: 503.50 MB

OK, but incomplete, skipped, or risky tests!
Tests: 7579, Assertions: 15918, Skipped: 156.
$ vendor/bin/paratest -c tests/phpunit/suite.xml --testsuite includes --exclude-group Broken,ParserFuzz,Stub,Database,Standalone --runner WrapperRunner

Time: 00:07.844, Memory: 115.00 MB

OK, but incomplete, skipped, or risky tests! 
Tests: 7579, Assertions: 15918, Skipped: 156.

This looks really promising. I'm still planning to do some general PHPUnit cleanup before tackling this task (e.g., T342428 and T342301), and then there'll be some other issues to figure out, for instance the way of specifying extension tests, whose list is dynamically generated and doesn't work with paratest. But I'm quite confident overall.

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

Change 758784 abandoned by Kosta Harlan:

[mediawiki/core@master] phpunit: Use process ID in test DB prefix

Reason:

https://gerrit.wikimedia.org/r/758784

Change 742199 abandoned by Kosta Harlan:

[mediawiki/core@master] [WIP] phpunit: Support splitting core and extension suites

Reason:

https://gerrit.wikimedia.org/r/742199

Change 742200 abandoned by Kosta Harlan:

[integration/quibble@master] [WIP] Run PHPUnit tests in parallel

Reason:

https://gerrit.wikimedia.org/r/742200

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

I think this may be from tests which extend the base PHPUnit classes, instead of MediaWikiUnitTest or MediaWikiIntegrationTest.

This time not a benchmark but a FYI: do NOT try to run paratest with DB tests in your local. Not even as an experiment. I did it a few times, and it truncated a few tables in my real DB. I have no idea why, nor have I looked into Aaron's config for paratest on sqlite, but whatever the cause, it definitely connects to the real DB at some point.

I think this may be from tests which extend the base PHPUnit classes, instead of MediaWikiUnitTest or MediaWikiIntegrationTest.

I believe I was able to reproduce the bug with WikiPageDbTest, which does extend MediaWikiIntegrationTestCase. Further, I remember writing something about this bug somewhere, but I can't find it right now, so maybe I didn't. Anyway, I read somewhere that SQLite can sometimes "forget" about the DB prefix when the connection is reloaded, or something like that, esp. in case of concurrent connections. I think this might be what Aaron's code in ParatestSettings.php is for; but again, I'm not an SQLite expert.

TBH, the results for r1005792 look really promising. That patch is just a POC, and has numerous issues that need to be fixed (both in the config/bootstrap and in the tests themselves), but the speed seems consistent with my local experiments: the core-only database suite finished in 50 seconds, vs the 4m54s it took for the parent patch. Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite). Still, these issues can be resolved as usual, and so far I don't think we've found any showstoppers.

Change 1006168 had a related patch set uploaded (by Daimona Eaytoy; author: Daimona Eaytoy):

[mediawiki/core@master] phpunit: Do not crash when paratest options are passed in

https://gerrit.wikimedia.org/r/1006168

Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite).

That seems kind of worrying, though. If we switch the coverage job to use paratest as well, then we'd have some additional assurances that we're still covering the same code.

Note that, amongst the issues to be fixed, paratest seems to be unable to report the actual number of tests it's running (notice how it starts from 36551 and ends at 10075); and it's not even clear if it's running all the tests or not (the parent patch has 12349 tests in the DB suite).

That seems kind of worrying, though. If we switch the coverage job to use paratest as well, then we'd have some additional assurances that we're still covering the same code.

It is, and that's why I consider this a blocker. While I'm not 100% sure about it, I think this bug is a consequence of "dynamic" suites (T345481), as I'm unable to reproduce it otherwise. My proposed fix for extension tests is already applied in the test run above, but we still need to fix parser tests and others, most notably scribunto (T358394).

Change 1006168 merged by jenkins-bot:

[mediawiki/core@master] phpunit: Do not crash when paratest options are passed in

https://gerrit.wikimedia.org/r/1006168