Page MenuHomePhabricator

CI browser test dashboard takes 100 seconds to appear on first load
Closed, ResolvedPublic

Description

In the last few days, when I visit https://integration.wikimedia.org/ci/view/BrowserTests/ it redirects to https://integration.wikimedia.org/ci/view/BrowserTests/view/-Dashboard/ which takes over 1.5 minutes to appear. Thereafter reloads are much faster.

It definitely was not this slow in the past. The redirect page now includes two graphs as well as the build queue and list of browser tests, maybe their libraries slow it down. If that's the problem then perhaps https://integration.wikimedia.org/ci/view/BrowserTests/ should not go to a page with these graphs.

The slowness is in both Firefox 32 and chromium 37 on Ubuntu


Version: unspecified
Severity: minor

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:49 AM
bzimport set Reference to bz70671.
bzimport added a subscriber: Unknown Object (MLST).

The Jenkins dashboard loads the history of all builds on the first request. Jenkins save each build result as a standalone XML file and the server running Jenkins has slow disk I/O.

So, on first access the dashboard ends up loading thousands of files from disk. Once it is loaded, it is kept in memory for a while.

I am tempted to wontfix this bug. A better solution would be to generate a dashboard by ourself and asynchronously but I don't think it is worth the effort.

I don't expect miracles :) but it's getting worse. The last few times I've loaded the dashboard it times out altogether and I have to reload to get it to appear.

  • Could I have a dumb cron job that requests the dashboard every "a while"? Would that hurt other clients of the server?
  • Can we build dashboards for individual sets of tests like the [fl] and [ve] tabs that were in the cloudbees dashboard?

What we could do is discard old build results after a given time. I am not sure whether we need to keep them after say 15 days.

I have proposed the idea of reducing the browser tests history to the QA mailing list: https://lists.wikimedia.org/pipermail/qa/2014-October/002043.html

(In reply to Antoine "hashar" Musso (WMF) from comment #3)

What we could do is discard old build results after a given time. I am not
sure whether we need to keep them after say 15 days.

+1

"Jenkinks UI slow due to constant build record loading"
https://issues.jenkins-ci.org/browse/JENKINS-15858

Last comment:
"fixed in 2.8"

We're on Jenkins ver. 1.580.1

From Zeljko:
Dashboard View, jenkins extension, we have version 2.9.4, performance bug should be fixed in 2.8

I'm wrong :)

Change 176649 had a related patch set uploaded (by Hashar):
Limit browser tests history to 31 days

https://gerrit.wikimedia.org/r/176649

Patch-For-Review

Change 176649 merged by jenkins-bot:
Limit browser tests history to 31 days

https://gerrit.wikimedia.org/r/176649

Krinkle assigned this task to hashar.
Krinkle removed a project: Patch-For-Review.
Krinkle set Security to None.
Krinkle removed a subscriber: Unknown Object (MLST).