In gerrit change 40343, apparently the failures of the test suite were overruled by the successful lint checks. Even though in this particular case the test suite seems to be faulty, Jenkins should not do that.
(NB: I'm just looking at the Gerrit UI here where it says "jenkins-bot: Verified: +1"; I don't know how it looks to the Gerrit ruleset when it comes to deciding whether the patchset can be merged.)
Version: unspecified
Severity: major
This is likely a race condition.
I think we can solve this by ensuring we only execute one pipeline on an event, not two.
From Zuul's system it might make sense to allow multiple pipelines (e.g. if 'check' is not a subset of 'test'), however this currently doesn't make any sense as there is only 1 score field for Verified for jenkins-bot and the one arriving last will just override it.
So either Zuul needs to merge the pipelines into 1 on-the-fly, or we need to quit doing both 'check' and 'test'.
Related URL: https://gerrit.wikimedia.org/r/58273 (Gerrit Change Ic9a3411406fafdb0de19ed20d7e717b20280fd7d)
Not ideal, but good enough for now.
We could leave it open for a cleaner solution, however I think this is good enough given that having two pipelines (check and test) for the same event is misuse of Zuul to begin with.
This whole obnoxious set up is only because we don't have an isolated environment yet (bug 45499). Once we do, this whole thing can be gotten rid of.