Add more VisualEditor tests via CloudBees jobs for OSes beyond Linux
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	• bzimport
	Dec 6 2013, 7:10 PM

Description

Author: jhall

Description:
Per a request from James F, we should start to expand the selection of environments (OS and browser combinations) that Visual Editor automated tests run via CloudBees.

For example, add a "Internet Explorer 10 on Windows 8" test, and related variations that CloudBees/SauceLabs can accommodate.

Per Chris M, a good way to get started is to copy the existing "Selenium" template in CloudBees and figure out what we need to do to add Windows into the mix.

Version: unspecified
Severity: normal

Details

Reference: bz58110

Event Timeline

• bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 2:26 AM

• bzimport added a project: Quality-Assurance.

• bzimport set Reference to bz58110.

• bzimport created this task.Dec 6 2013, 7:10 PM

jhall wrote:

Created a new test template to mess around with:

https://wmf.ci.cloudbees.com/template/81889c2c-ef15-4fad-b39b-8256eb9e841a/

jhall wrote:

In addition to the new template in CloudBees, I think we'll need to expand the config.yml file in the VisualEditor repo:

https://github.com/wikimedia/mediawiki-extensions-VisualEditor/blob/master/modules/ve-mw/test/browser/config/config.yml

For example, our existing "firefox" profile is hard-coded to Firefox version 23 on Linux, so we'll need to add more entries for Firefox on Windows 8, etc.

#1 I would hesitate to add more Jenkins jobs until existing ones[1] are rock solid (meaning there were no failures in a week or so).

#2 The entire config folder will soon(tm) be moved to mediawiki-selenium gem[2]. We should probably rename config.yml to browsers.yml (since it no longer has any other information, only browsers) and expand the number of combinations, like Wikibase has already done[3]

1: https://wmf.ci.cloudbees.com/view/r-ve/
2: https://gerrit.wikimedia.org/r/#/c/90560
3: https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/tests/browser/config/browsers.yml

jhall wrote:

(In reply to comment #3)

#1 I would hesitate to add more Jenkins jobs until existing ones[1] are rock
solid (meaning there were no failures in a week or so).

#2 The entire config folder will soon(tm) be moved to mediawiki-selenium
gem[2]. We should probably rename config.yml to browsers.yml (since it no
longer has any other information, only browsers) and expand the number of
combinations, like Wikibase has already done[3]

1: https://wmf.ci.cloudbees.com/view/r-ve/
2: https://gerrit.wikimedia.org/r/#/c/90560
3:
https://github.com/wikimedia/mediawiki-extensions-Wikibase/blob/master/tests/
browser/config/browsers.yml

Thanks Željko - I agree with both of your points, especially #1, which is an ongoing problem area (which I am working on!)

(In reply to comment #3)

#2 The entire config folder will soon(tm) be moved to mediawiki-selenium
gem[2]. We should probably rename config.yml to browsers.yml (since it no
longer has any other information, only browsers) and expand the number of
combinations, like Wikibase has already done[3]

config.yml is now moved to mediawiki-selenium gem. Feel free to expand the number of browser/OS combinations in the gem.

jhall wrote:

Per a chat with Rummana, the priority test environments to add (via CloudBees and Sauce Labs) are:

Chrome on Windows XP
Firefox on Windows XP

Change 110980 had a related patch set uploaded by Jhall:
Adding new priority Windows browser tests for VisualEditor.

https://gerrit.wikimedia.org/r/110980

jhall wrote:

Setting bug status back to resolved since the 110980 Gerrit patch is a first step only; if that change works out well, we can add additional (browser + OS) combinations to work towards eventually resolving this bug.

Change 110980 abandoned by Jhall:
Adding new priority Windows browser tests for VisualEditor.

Reason:
Abandoning this change per a discussion between Chris, Željko and myself. We have decided to implement new (browser + OS) combinations for use with SauceLabs via environment variable values rather than via hard-coded amendments to the mediawiki-selenium gem itself.

https://gerrit.wikimedia.org/r/110980

jhall wrote:

Tests have been setup in CloudBees at the following locations: