After some discussion on IRC...

We should probably split

https://github.com/wikimedia/mediawiki-core/blob/master/tests/phpunit/includes/parser/MediaWikiParserTest.php

into a 'core' and an 'extensions' part. Because in the Jenkins situation we only have a single extension loaded, this gives us the needed separation. Something along the lines of

$wgExtensionParserTestFiles['LabeledSectionTranscludion'][] = $path . 'lstParserTests.txt';

is probably the clearest method, and it also keeps the option open to do more specific filtering later.

Note that even if it's a change for an extension, we still want to check that it doesn't break core parser tests.

LabeledSectionTransclusion has a patch that introduce a test suite to let PHPUnit detects the parserTests: https://gerrit.wikimedia.org/r/#/c/38114

We ideally want to introduce such a feature in mw/core to avoid having extensions writer to duplicate that code :D

Not really have time to look at this bug right now. Moving back to pool.

Workaround by Merlijn van Deen is to create a dummy PHPUnit test class in the extension:e

https://gerrit.wikimedia.org/r/#/c/38114/2/tests/lstParserTest.php,unified

This way 'php phpunit.php extensions/Foobar' will actually find something to run :-)

So Wikibase has most probably been hit by that bug. Below is my mail summarizing it:

A few months ago Merlijn noticed an issue with the LabeledSectionTransclusion extension for which parser tests were not triggered. The workaround is to add a dummy parser tests in the extension as shown at:

https://gerrit.wikimedia.org/r/#/c/38114/

I have never been happy with that because the parser tests are already registered in a global variable and the MediaWiki phpunit wrapper is supposed to find them via the 'extensions' test suite.

The Wikibase issue we have been facing (where Jenkins does not find the Wikibase test) is most probably related in the sense Jenkins does not find the tests for Wikibase.

My suspicion is:

Back in the old days I have made phpunit for extensions to be called using the form:

php tests/phpunit.php -- extensions/<name of extension>

The point was to only run the tests for the extension, not the full core test suite since it is sooooo slow. Ideally we should have used PHPUnit v3.7.x --testsuite argument like:

php tests/phpunit.php --testsuite extensions

That would parse the tests/phpunit/suite.xml and lookup for a testsuite named 'extensions' and in turns load our wrapper:

/tests/phpunit/ExtensionsTestSuite.php

When I took over Jenkins, we used PHPUnit v3.6 which did not support --testsuite so I came with the workaround of passing the paths we want to test. The drawback is that it fallback to PHPUnit discovering mechanisme which is to look for file ending up with Test.php and in turns bypass any test registered via the global variable.

When running our phpunit wrapper from MediaWiki root directory, it does find the suite.xml file but since the paths to the suites are relative, it never loads them. Hence when invoking:

php tests/phpunit/phpunit.php --group Wikibase

You get no tests executed.

But running from the phpunit directory it does:

cd tests/phpunit; php phpunit.php --group Wikibase
<bunch of tests...>

So theorically Jenkins should be made to use:

cd tests/phpunit ; php phpunit.php --testsuite extensions

I will play with LabeledSectionTransclusion to fix that.

PHPUnit can not find the extension wrapper indeed:

$ strace -estat php tests/phpunit/phpunit.php --group Wikibase 2>&1|grep ExtensionsTestSuite
stat("suites/ExtensionsTestSuite.php", 0x7fffdaa4f790) = -1 ENOENT (No such file or directory)
stat("suites/ExtensionsTestSuite.php", 0x7fffdaa4eb00) = -1 ENOENT (No such file or directory)
stat("suites/ExtensionsTestSuite.php", 0x7fffdaa4ed00) = -1 ENOENT (No such file or directory)
$

Phpunit can not find the files in suite.xml because they are relative and phpunit.php is run from the root of MediaWiki.

The way to fix it is to cd to the tests/phpunit directory beforehand ie:

cd tests/phpunit && php phpunit.php ...

I have manually updated mwext-Wikibase-server-tests to use that and it did found some tests ( https://integration.wikimedia.org/ci/job/mwext-Wikibase-server-tests/ ).

The issue is very similar to bug 32022 'phpunit.php need to be run from tests/phpunit' though it is not fixable on our end apparently.

Related URL: https://gerrit.wikimedia.org/r/62622 (Gerrit Change Ib0fdffb97cdf237a49b43d7abaa81b81afe8c499)

Related URL: https://gerrit.wikimedia.org/r/63164 (Gerrit Change Icc3e9d30706b32149aa9dd18552e4241ec4af67e)

Worked a bit on this patch again. Patchset 4 of https://gerrit.wikimedia.org/r/#/c/63164/ has something more or less working by using the --testsuite parameter of PHPUnit.

There is also a bug in PHPUnit (3.7.21) that makes it lookup suite files relatively to where phpunit. Directories are not impacted though.

The PHPUnit bug is fixed with https://github.com/sebastianbergmann/phpunit/pull/938

https://gerrit.wikimedia.org/r/#/c/63164/ has been merged in. To run the extension tests one will have to use:

cd tests/phpunit && php phpunit.php --testsuite extensions