That would be awesome. And it could be made voting once it works properly.

Could you describe what would be the workflow to end up testing Semantic extensions? Seems to me it would need:

fetch MediaWiki @REL1_19
fetch out all extensions
run installer
run phpunit

Report. This could be done daily.

Seems to be similar to bug 52278 "Integration tests for Wikibase dependencies"

I made the SMW master/SMW 1.9 unit tests [1] compatible so that the same tests can actually run against a MW REL1_19 instance.

It would be marvellous to have existing SMW master unit tests run against a MW REL1_19 instance as supplement to the current MW master [1]. Last week I already tripped over an unknown method call due to MW 1.21/1.22, and I only caught them because I run the tests manually. In future I'd like to avoid this by having Jenkins to bounce back as long as MM 1.19 is an official LTS release, at least for what is now SMW master (and soon to be SMW 1.9).

If it is done daily, where can you see the report because right now only individual commits will show success/failure of a Jenkins run?

If it is done on a per commit basis, first execute [1] and if the job fails stop otherwise continue with the commit patchset and load REL1_19 instead.

PS: Not sure if this is relevant and only increase the necessary effort to create a new Jenkins template but at some point SMW master (which by then is either 1.10 or 1.11) doesn't have to care about MW 1.19 any longer but should be tested against MW 1.21 instead but since this is not an immediate issue, we should keep it simple and focus solely on having unit tests running against MW 1.19.

[1] https://integration.wikimedia.org/ci/job/mwext-SemanticMediaWiki-testextensions-master

See also Bug 54255

Since nothing is happing here and I don't have time to waste, I'm closing this issue.

PS: It somehow shows that non-WMF extensions do not necessarily can make sufficient use of the "given" WMF infrastructure.