Created attachment 12060
Jenkins log related to plugin upgrade

Attached:

I didn't upgrade any plugins, but we did have issues with Jenkins being super slow so we restarted.

Looking at the plugin files date under /var/lib/jenkins/plugin :

-rw-rw-r-- 1 jenkins jenkins 2924737 Apr 8 23:27 copyartifact.jpi
-rw-rw-r-- 1 jenkins jenkins 5134796 Apr 8 23:27 depgraph-view.jpi
-rw-rw-r-- 1 jenkins jenkins 2764886 Apr 8 23:27 git-client.jpi
-rw-rw-r-- 1 jenkins jenkins 3408310 Apr 8 23:27 git.jpi
-rw-rw-r-- 1 jenkins jenkins 129619 Apr 8 23:27 jobConfigHistory.jpi
-rw-rw-r-- 1 jenkins jenkins 2936677 Apr 8 23:27 xunit.jpi

copyartifact: 1.25 -> 1.26 | not a problem

depgraph-view 1.10 -> 1.11 | not a problem

xunit 1.52 -> 1.53 | not a problem

git plugin: 1.1.26 -> 1.3.0

addition of git-client plugin 1.0.5 which LUCKILY has the revert of the Jgit migration and still use the safe git-cli implementation.

job configuration history: 1.13 -> 2.3

The config history plugin has lost all of its history. This is due to version 2.0 that changed the location where files are saved.

Migrated all job history from the per job directory to the shared directory /var/lib/jenkins/config-history/jobs using the following script:

#!/bin/bash -e

src=/var/lib/jenkins/jobs
target=/var/lib/jenkins/config-history/jobs

cd "$src"
dirs=( * )
for f in "${dirs[@]}"
do

cp -r -v "$f/config-history" "$target/$f"

done

grmblbl that copied config-history as a subdirectory of the job. Need to be fixed later on.

Job configs have been copied properly restoring the access.

git plugin seems to be still working.

I guess that is solved for now :-]

For history purpose, this bug was intended has a log of what I did after the plugins got upgraded.

(In reply to comment #0)

The Jenkins plugins have been upgraded on Apr 8, 2013 11:27:03 and Jenkins
got restarted. That can cause various regressions in the installation.

They were upgraded by me. I logged into the admin panel and updates a few plugins (not all).

Then when Jenkins opted to restart later or now, I told it do it now.

However when it got back up it hung on "Please wait while Jenkins is getting ready to work..." for ever.

(In reply to comment #2)

I didn't upgrade any plugins, but we did have issues with Jenkins being super
slow so we restarted.

It wasn't slow. It was stuck restarting.

Roan then helped out (as root) by digging through the jtack and jmap[1], but that didn't help us at the time.

Then right before we were considering to kill it and start again, it finished all fo a sudden. By this time the "Please wait while Jenkins is getting ready to work..." screen was on the web gui for almost half an hour. Apparently that's just how long it takes to restart after updating plugins.

(In reply to comment #3)

job configuration history: 1.13 -> 2.3

The config history plugin has lost all of its history. This is due to version
2.0 that changed the location where files are saved.

I did look at the changelog of JobConfigHistory but missed the notice about the location changing. The migration script is available on the plugin page:
https://wiki.jenkins-ci.org/display/JENKINS/JobConfigHistory+Plugin

So that's the only thing that happened, right? The old versions of job configurations got disconnected and they were re-imported with that script.

[1] https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+is+hanging

It wasn't slow. It was stuck restarting.

<somestuff>

Then right before we were considering to kill it and start again, it finished

all fo a sudden.

Which highlight the fact that Jenkins is slow to reload, isn't it? :-]  On startup there is a background process that parse the build histories, while it is busy handling them, the web interface is not available :/

So yeah, luckily, I just had to migrate the config histories :-] I even patched upstream migration shell script to take care of jobs having spaces in their names \O/

The Job config history is a side effect, the main cause was careless upgrade of all the plugins which could have been much more impact.

I am putting back the summary to easy find out the bug later on.