browser tests should assert there are no pink error boxes on the page
Closed, DeclinedPublic
Actions

Assigned To

None

Authored By

	• Spage
	Feb 13 2014, 8:09 AM

Description

When I kick back with some popcorn and watch Sauce Labs browser test movies, I sometimes notice error boxes appearing on the screen. Yet the test continues, only to fail later.

The classic example is

MEDIAWIKI_PASSWORD=wrong MEDIAWIKI_URL=http://en.wikipedia.beta.wmflabs.org/wiki/ MEDIAWIKI_USER=Selenium_user bundle exec cucumber features/flow_logged_in.feature

the login fails with "Login error Incorrect password..." yet the tests continue.

There are other examples where the page reports an API error and the tests keep rolling.

99% of the time, if anything appears on the page with class=errorbox or class=error, it's serious. Is there a way tests can implicitly assert these aren't on the page on every step, and if not the case report the contents of the .error/.errorbox ?

Version: unspecified
Severity: normal

Details

Reference: bz61304

Related Objects

Mentioned Here: T139740: Port Selenium tests from Ruby to Node.js

Event Timeline

• bzimport raised the priority of this task from to Low.Nov 22 2014, 3:07 AM

• bzimport added a project: Quality-Assurance.

• bzimport set Reference to bz61304.

• bzimport added a subscriber: Unknown Object (MLST).

• Spage created this task.Feb 13 2014, 8:09 AM

S, do you need help implementing this?

(In reply to Željko Filipin from comment #1)

S, do you need help implementing this?

Always :)

A simple test case for the utility of this is any browser test that logs in via Special:UserLogin, e.g.

cd extensions/Flow/tests/browser
export MEDIAWIKI_PASSWORD=WRONGwrong
bundle exec cucumber features/action_menu_permalink.feature

Current result is test fails with

Given I am logged in  # mediawiki_selenium-0.3.2/lib/mediawiki_selenium/step_definitions/login_steps.rb:12
  Connection refused - connect(2) for "127.0.0.1" port 9515 (Errno::ECONNREFUSED)

A more useful result is something like

Given I am logged in  # mediawiki_selenium-0.3.2/lib/mediawiki_selenium/step_definitions/login_steps.rb:12
  visible errorbox on page with contents "<strong>Login error</strong><br>Incorrect password entered. Please try again."

Chris, Dan and me should be able to help you. Feel free to ping me via e-mail if you would like to pair with me in your morning (my afternoon).

zeljkofilipin unsubscribed.Dec 3 2014, 1:01 PM

zeljkofilipin edited projects, added Browser-Tests-Infrastructure; removed Quality-Assurance.Mar 25 2015, 2:05 PM

zeljkofilipin set Security to None.

Anybody still cares about this, or can this be resolved?

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptNov 2 2015, 4:20 PM

zeljkofilipin moved this task from Inbox to Waiting on the Browser-Tests-Infrastructure board.Nov 2 2015, 4:20 PM

zeljkofilipin moved this task from Waiting to Gems on the Browser-Tests-Infrastructure board.Nov 5 2015, 11:55 AM

greg added a project: Release-Engineering-Team (Backlog).May 20 2017, 4:39 PM

Unlikely to ever be resolved because of T139740: Port Selenium tests from Ruby to Node.js.

browser tests should assert there are no pink error boxes on the pageClosed, DeclinedPublicActions

Description

Details

Related Objects

Event Timeline

browser tests should assert there are no pink error boxes on the page
Closed, DeclinedPublic
Actions