Page MenuHomePhabricator
Create Task
Maniphest T49120

[upstream] Jenkins is slow to start
Closed, ResolvedPublic
Actions

Description

jstack trace taken while Jenkins is busy reading files

Whenever restart Jenkins it takes roughly half an hour to start up. Running strace on the process it is reading all the build histories.

I have attached a jstack taken during startup. An interesting thread is 25437.

So I guess whenever Jenkins start a project, it ends up reading all the build history for all projects which is definitively a slow operation given the number of builds logs we keep.

Version: wmf-deployment
Severity: normal
Whiteboard: incident-report
See Also:
https://issues.jenkins-ci.org/browse/JENKINS-17837

Attached:

file_47120.txt67 KBDownload

Details

Reference
bz47120

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedNoneT49120 [upstream] Jenkins is slow to start
 ResolvedhasharT49744 update Jenkins to LTS 1.509.1

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 1:29 AM
• bzimport added projects: Continuous-Integration-Infrastructure, Upstream.
• bzimport set Reference to bz47120.
• bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Apr 11 2013, 10:09 AM
hashar added a comment.Apr 17 2013, 1:32 PM

Seems like upstream has been working on it recently.

The issue https://issues.jenkins-ci.org/browse/JENKINS-8754 "Improve Start-up Time" has some information about it.

1.485 seems to implements lazy loading. So I guess we will get it with the next LTS release.

hashar added a comment.May 2 2013, 2:28 PM

Jenkins 1.509.1 (LTS) still has the issue.

hashar added a comment.May 2 2013, 2:47 PM

Upstream bug is https://issues.jenkins-ci.org/browse/JENKINS-17837

hashar added a comment.May 2 2013, 2:53 PM

Maybe it is related to the Downstream buildview plugin which is reference jstack traces. https://wiki.jenkins-ci.org/display/JENKINS/Downstream+buildview+plugin

hashar added a comment.May 2 2013, 3:13 PM

I have removed the downstream-buildview plugin. That was triggered a full parse of all the build history!

The Jenkins upgraded did add the Lazy Loading system.

We are safe now, Jenkins restarts very fast now.

hashar added a comment.Dec 17 2013, 8:05 PM

Upstream bug ( https://issues.jenkins-ci.org/browse/JENKINS-17837 ) received some attention today. The issue was in the downstream-buildview plugin that would update the whole history of jobs apparently for no worthwhile reason.

Upstream fixed the issue by removing the faulty class with https://github.com/jenkinsci/downstream-buildview-plugin/commit/47801c278f000b34eae5bd9d674029cf470cd8b2

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL