Page MenuHomePhabricator

Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium)
Closed, ResolvedPublic40 Estimated Story Points

Description

Current pipeline uses the global wmfgrunt slave-script:

Fetch integration-jenkins.git to test it. As dependencies are pre-compiled and committed to git for Ubuntu Precise, on other platforms, first cd ./tools; rm -rf node_modules and npm install to provide for your system's architecture.

It's invoked from this macro: https://github.com/wikimedia/integration-config/blob/0b85d48e60/jjb/macro.yaml#L344

For many reasons, we should migrate from grunt-contrib-qunit to Karma, and use Chromium and/or Firefox as browser instead of PhantomJS:

  • Performance. Chromium is faster.
  • Accuracy. PhantomJS is not a browser we support. Its layout and JavaScript engine and version thereof are irrelevant. It's based on a port from an old Safari build. Aside from being old it's also customised and under various abstraction layers. Ideally we'd run in multiple browsers, but even one would be a great start. PhantomJS does support nice features most browsers can't or wont implement, but those are not features we need to run unit tests. We want a real browser.
  • Ability. While testing fallback behaviour is important, many new browser features are unavailable in PhantomJS. This is actively making it impossible to run tests against modern code such as from OOjs UI and VisualEditor that don't support Safari 4 or 5.

https://github.com/karma-runner/karma
https://github.com/karma-runner/karma-chrome-launcher
https://github.com/karma-runner/karma-qunit
https://github.com/karma-runner/grunt-karma

This is already being used by OOjs, OOjs UI and VisualEditor standalone. This task is about migrating the pipeline for projects that rely on running in a MediaWiki context. This brings many complexities due to registering and loading of scripts. As well as migrating the bootstrapping code to support Ubuntu Trusty and labs instances (as opposed to prod slavs running Precise).

Related Objects

Event Timeline

bzimport raised the priority of this task from to High.Nov 22 2014, 3:54 AM
bzimport set Reference to bz72063.

Change 168631 had a related patch set uploaded by Krinkle:
[WIP] contint: Apply contint::qunit_localhost to labs slaves

https://gerrit.wikimedia.org/r/168631

Step 1: Move qunit from prod/precise to labs/trusty.
Step 1a: Refactor puppet for localhost/qunit and apply to labs.
Step 1b: Refactor puppet for mediawiki testing and apply to labs (sqlite, php5-cli, imagemagic, etc.)
Step 2: Change integration/jenkins:bin/wmfgrunt to use Karma instead of Grunt-contrib-qunit/grunt-lib-phantomjs

localhost/qunit:

Looks like the qunit test doesn't pass in labs. Some seeminly-random API calls are failing.

https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit2/6/console

I've copied one of the qunit2 jobs to a static directory on integration-slave1008 and proxied it through ssh to my local Chrome to use Dev Tools to investigate the QUnit failures.

They are indeed timing out, on load.php requests. Some of which are taking over 12 seconds (the QUnit timeout for an individual test is 10 seconds, defined in our testrunner.js in mediawiki-core:/tests/qunit/data).

The main slowdown is the database layer. Increasing the timeout to 30 seconds made it work, but we don't want our tests to be that slow. I'll first try to match production speed more, and we could resort to increasing the timeout otherwise.

Will add a tmpfs to the integration slaves and make Jenkins use that when appropriate.

Change 173512 had a related patch set uploaded by Krinkle:
[WIP] contint: Add tmpfs mount in jenkins-deploy homedir for labs slaves

https://gerrit.wikimedia.org/r/173512

The sqlite installer auto-detects presence of a tmpfs mount in $HOME via mw-set-env:

https://github.com/wikimedia/integration-jenkins/blob/8abc7b6/bin/mw-set-env.sh#L19

I've mounted tmpfs drive in the $HOME of jenkins-deploy on the integration slaves

https://gerrit.wikimedia.org/r/173511
https://gerrit.wikimedia.org/r/173512

Tested a few in labs, and are now running fine. The compared runs in labs are an N+1 run since the first one involve zuul-cloner cloning mediawiki-core.

on gallium in 1m18s:
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/12309/

on integration-slave1003 in 1m34s:
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/12365/

on gallium in 1m23s:
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/12362/

on integration-slave1003 in 1m31s:
https://integration.wikimedia.org/ci/job/mwext-VisualEditor-qunit/12366/

on gallium in 29s:
https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/32092/

on integration-slave1003 in 32s:
https://integration.wikimedia.org/ci/job/mediawiki-core-qunit/32094/

Change 173529 had a related patch set uploaded by Krinkle:
Migrate mediawiki/qunit jobs from production slaves to labs

https://gerrit.wikimedia.org/r/173529

Krinkle renamed this task from Jenkins: Convert mwext qunit from grunt-contrib-qunit (PhanttomJS) to grunt-karma (Chromium) to Jenkins: Convert mwext qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium).Nov 24 2014, 4:28 PM

Blocked on https://gerrit.wikimedia.org/r/#/c/175949/ which removes the test that was leaking jQuery animations.

Blocked on https://gerrit.wikimedia.org/r/#/c/175949/ which removes the test that was leaking jQuery animations.

This has been merged.

In T74063#765974, gerritadmin wrote:

Change 173529 in integration/config by Krinkle:

Migrate mediawiki/qunit jobs from production slaves to labs

Change-Id: I69c4d6db114e2960c76ac01c4b255b7fb7bf8962

This change has been deployed to Jenkins. The QUnit jobs now install MediaWiki on a labs slave and also run PhantomJS there.

Change 173529 merged by jenkins-bot:
Migrate mediawiki/qunit jobs from production slaves to labs

https://gerrit.wikimedia.org/r/173529

Change 178551 had a related patch set uploaded (by Krinkle):
SpecialJavaScriptTest: Add export feature

https://gerrit.wikimedia.org/r/178551

Patch-For-Review

Change 178551 merged by jenkins-bot:
SpecialJavaScriptTest: Add export feature

https://gerrit.wikimedia.org/r/178551

Change 173512 merged by coren:
contint: Add tmpfs mount in jenkins-deploy homedir for labs slaves

https://gerrit.wikimedia.org/r/173512

Change 187638 had a related patch set uploaded (by Krinkle):
mediawiki-core-qunit: Migrate to simplified mw-set-env-qunit.sh

https://gerrit.wikimedia.org/r/187638

Patch-For-Review

Change 186934 had a related patch set uploaded (by Krinkle):
[WIP] Add qunit-karma macro and mediawiki-core-qunit-karma job

https://gerrit.wikimedia.org/r/186934

Patch-For-Review

Change 187638 merged by jenkins-bot:
mediawiki-core-qunit: Migrate to simplified mw-set-env-qunit.sh

https://gerrit.wikimedia.org/r/187638

Change 186934 merged by jenkins-bot:
Add qunit-karma macro and various *-qunit-karma jobs

https://gerrit.wikimedia.org/r/186934

Krinkle renamed this task from Jenkins: Convert mwext qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium) to Jenkins: Convert mediawiki qunit from grunt-contrib-qunit (PhantomJS) to grunt-karma (Chromium).Mar 20 2015, 11:05 AM

Change 198210 had a related patch set uploaded (by Krinkle):
Update mwcore-qunit to trusty/hhvm/mysql/chrome from precise/zend/sqlite/phantomjs

https://gerrit.wikimedia.org/r/198210

Change 198210 merged by jenkins-bot:
mediawiki-core-qunit: To trusty/hhvm/mysql/chrome from precise/zend/sqlite/phantomjs

https://gerrit.wikimedia.org/r/198210

Change 198257 had a related patch set uploaded (by Krinkle):
mwext-VisualEditor-qunit: To trusty/karma/chromium from precise/phantomjs

https://gerrit.wikimedia.org/r/198257

Change 198257 merged by jenkins-bot:
mwext-VisualEditor-qunit: To trusty/karma/chromium from precise/phantomjs

https://gerrit.wikimedia.org/r/198257

Change 198293 had a related patch set uploaded (by Krinkle):
mwext-{name}-qunit: To hhvm/karma/chromium from zend/php53/phantomjs

https://gerrit.wikimedia.org/r/198293

Change 198293 merged by jenkins-bot:
mwext-{name}-qunit: To hhvm/karma/chromium from zend/php53/phantomjs

https://gerrit.wikimedia.org/r/198293

The reported number of tests ran differs widely between running QUnit manually in the browser and running it via Karma. For example, some recent mediawiki-core-qunit run reports Executed 259 of 259, while this run from last Thursday said 1576 assertions passed. I suppose karma-qunit counts tests, while Special:JavaScriptTest/qunit counts assertions? Even if that is the explanation, it should be mentioned somewhere.

More issues I had when running grunt karma:main locally:

  • I had to pass MW_SERVER and MW_SCRIPT_PATH (which makes sense), but it's a) not documented anywhere and b) the error message is not helpful at all.
  • In karma-qunit (lib/adapter.js), QUnit.load and QUnit.start are executed (which is fine), and they try to set QUnit.config.autostart to false, but that doesn't work somehow in my setup. Instead, QUnit.config.autostart is undefined right before QUnit.load is called, and thus QUnit fails on trying to start() twice. I had to patch it locally to set autostart to false directly before calling load.

Sorry for the noise, but after all the nagging I also wanted to thank for this great improvement. I can now easily run the QUnit tests headlessly without having to clone integration/jenkins, it supports two browsers we actually support, and it's so fast: Down from 33 seconds to 18 seconds in Chromium and 23 seconds in Firefox. Awesome!