I ran bundle exec cucumber features/login.feature per [1] in my MediaWiki-Vagrant instance and got two failures:

 expected "Login error\nYou have not specified a valid username." to match "You\\ have\\ not\\ specified\\ a\\ valid\\ user\\ name\\." 

expected "Login error\nPassword entered was blank. Please try again." to match "The\\ password\\ field\\ was\\ empty\\.\\ Please\\ try\\ again\\." (RSpec::Expectations::ExpectationNotMetError)

The first one expects enwiki's modified noname message which adds a space in "user name" (https://test2.wikipedia.org/wiki/MediaWiki:Noname). The second one expects enwiki's rewritten wrongpasswordempty message (https://test2.wikipedia.org/wiki/MediaWiki:Wrongpasswordempty).

Obviously any developer who just installed MediaWiki won't have these customizations, so the tests fail. And admins on enwiki regularly change its messages, so the tests are brittle.

Some of enwiki's modified messages helpfully include a

<span id="mw-wrongpasswordempty">

which won't change, but the default form doesn't have this.

The tests could do more generic text matches such as '.*valid user.*', but almost nothing is in common between the default and enwiki for wrongpasswordempty. I think a better approach is to get tests to execute with the ?uselang=qqx language code in the query string, which renders message text as the message keys themselves and thus is independent of language and message tweaks.

[1] https://git.wikimedia.org/blob/mediawiki%2Fvagrant.git/HEAD/puppet%2Fmodules%2Fbrowsertests%2FREADME

Version: wmf-deployment
Severity: normal
See Also:
https://bugzilla.wikimedia.org/show_bug.cgi?id=60347