root@openid-wiki:/srv/mediawiki# git pull
remote: Counting objects: 894, done
remote: Finding sources: 100% (477/477)
remote: Getting sizes: 100% (489/489)
remote: Compressing objects: 100% (72/72)
remote: Total 477 (delta 345), reused 405 (delta 331)
Receiving objects: 100% (477/477), 279.19 KiB, done.
Resolving deltas: 96% (379/393), completed with 222 local objects.
fatal: pack has 14 unresolved deltas
fatal: index-pack failed
root@openid-wiki:/srv/mediawiki#

root@openid-wiki:/srv/mediawiki# git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (43356/43356), done.

WFM.

This might be related to working on a shallow repo )side note: my local full clone (with some branches fetched apart from master) has ~400k objects instead of ~40k).

In my view, it is a blocker

even when having a shallow repo, git pull must not fail

This WFM as well on a shallow clone. Can you try on a fresh clone?

I can try. It means, that the puppet class which installed that wiki onm labsconsole is corrupted, apparently.

What does git fsck --full give you?

pls. wear your glasses : https://bugzilla.wikimedia.org/show_bug.cgi?id=44129#c2

I saw that, and you only seemed to have done git fsck. I asked for the results of git fsck --full

Uh, *sorry*,

root@openid-wiki:/srv/mediawiki# git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (43356/43356), done.
root@openid-wiki:/srv/mediawiki#

Could you clone a fresh copy of the repository?

(In reply to comment #12)

Could you clone a fresh copy of the repository?

Yes. "Worksforme"

But looks, as if the "puppet class" has a corrupted repo.

But anyway, if you wish, you can close this bug.

[ Reset to UNCO and moving to Wikimedia Labs. ]

Wondering if further info would help to track down the offending commits/deltas, e.g. by using the env variable

GIT_TRACE=1

for git pull?

(In reply to comment #15)

Wondering if further info would help to track down the offending
commits/deltas, e.g. by using the env variable

GIT_TRACE=1

for git pull?

I made a tgz of the whole installation (with that problem) before I made a fresh clone, perhaps I can find some time to track down the problem.

Decreasing priority as nobody can reproduce this, and severity as there is a workaround (clone a fresh copy).

closing this. nobody cares, and since I cloned a fresh copy, the problem does not exist anymore.

Reopening this bug... I can reproduce it. I used puppet to install mediawiki/Wikidata on two Labs instances a week ago. Today, I ran git pull for the first time since, and it didn't work. On a third instance, I first had puppet install it two days ago. Same error, but less "deltas".

root@wikidata-testrepo /srv/mediawiki (master)$ git pull
remote: Counting objects: 3626, done
remote: Finding sources: 100% (1746/1746)
remote: Getting sizes: 100% (2360/2360)
remote: Compressing objects: 100% (480/480)
remote: Total 1746 (delta 1247), reused 1310 (delta 1083)
Receiving objects: 100% (1746/1746), 3.84 MiB | 4.24 MiB/s, done.
Resolving deltas: 88% (1309/1479), completed with 652 local objects.
fatal: pack has 170 unresolved deltas
fatal: index-pack failed

root@wikidata-testrepo /srv/mediawiki (master)$ git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (43644/43644), done.

Note that I *only* encountered this problem when pulling core, everything worked fine for the extensions.

I'm still not convinced it's Gerrit's fault at all.

From my (brief) time on Google, it looks like it could be a generic problem with shallow clones.

Looking at operations/puppet/manifests/generic-definitions.pp where git::clone is defined, it says

$depth="full",

which doesn't sound shallow.

On a newly create mediawiki cloned from git, git log only shows me the latest three commits, though.

(In reply to comment #21)

Looking at operations/puppet/manifests/generic-definitions.pp where
git::clone
is defined, it says

$depth="full",

which doesn't sound shallow.

Looking at operations/puppet/manifests/mediawiki.pp where mediawiki/core is actually cloned, it says

depth => 1,

which makes a shallow clone.

Cannot reproduce it using the test repo over http:

$ git clone --depth 1 http://l.saper.info:8888/test-project.git
$ cd test-project
$ git log
commit 425953c4bb34df0e3304179261097766929828f4
Author: Marcin Cieślak <marcin.cieslak@gmail.com>
Date: Fri Apr 13 09:23:43 2012 +0200

Force merge

Change-Id: I419a1d16bb06600d8c9db2907bb8aef1bbf0593a

commit 15e2af9cde0578f21056980d7f608f459ce75af9
Author: Marcin Cieślak <marcin.cieslak@gmail.com>
Date: Fri Apr 13 09:09:50 2012 +0200

More

Change-Id: Iefe0f48624a059474ff9f69da02bdfe59a0394db

$

(...)

after push of some change to the repository:

$ git pull --depth 1
remote: Counting objects: 4, done
remote: Finding sources: 100% (3/3)
Unpacking objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 3 (delta 0)
From http://l.saper.info:8888/test-project

425953c..ea80f4a  master     -> origin/master

Updating 425953c..ea80f4a
Fast-forward
DOIT | 1 +
1 files changed, 1 insertions(+), 0 deletions(-)
create mode 100644 DOIT

$ git log
commit ea80f4a2e524a3db676cb4401c620c8fb5f9d291
Author: saper <saper@saper.info>
Date: Mon Feb 11 11:07:25 2013 +0100

Another change

Change-Id: I0f92085acb6281b8539f164d36a35f12cfae983f

commit 425953c4bb34df0e3304179261097766929828f4
Author: Marcin Cieślak <marcin.cieslak@gmail.com>
Date: Fri Apr 13 09:23:43 2012 +0200

Force merge

Change-Id: I419a1d16bb06600d8c9db2907bb8aef1bbf0593a

But this was fast-forward update, using gerrit close to master (2.5.1-1020-gef6b83b) and git version 1.7.3.4

Ok got it reproduced with git talking to my test gerrit instance (above), over HTTP using mediawiki core repository.

$ git remote -v
origin http://l.saper.info:8888/mediawiki-core (fetch)
origin http://l.saper.info:8888/mediawiki-core (push)
$ git branch -vv

• master cd382de [origin/master] Merge "Revert "Updated RELEASE NOTES for 1.20""

(this some local change for testing)
$ git pull
remote: Counting objects: 24778, done
remote: Finding sources: 100% (22391/22391)
remote: Total 22391 (delta 18522), reused 21888 (delta 18522)
Receiving objects: 100% (22391/22391), 23.16 MiB | 1.04 MiB/s, done.
Resolving deltas: 84% (15736/18522), completed with 1237 local objects.
fatal: pack has 2786 unresolved deltas
fatal: index-pack failed

As discussed on IRC with Marcin C.:

• this is no puppet issue, he was not using puppet when he reproduced the error.
• it is *probably* not related to http(s) vs. ssh either
• the mediawiki core pack seems to lack information; normally git pull is possible on shallow clones.

Using direct access to the GIT repository (avoiding Gerrit and JGit)
seems to avoid the problem:

$ git remote add directssh ssh://l.saper.info/home/saper/sw/test-site-pg/git/mediawiki-core.git

$ git fetch -k --depth 1 directssh refs/heads/master:refs/remotes/directssh/master

seems to load objects and the tree can be updated with

$ git merge directssh/master

For now I suspect that JGit does not fully support sending object packs for shallow clients.

(In reply to comment #9)

pls. wear your glasses :
https://bugzilla.wikimedia.org/show_bug.cgi?id=44129#c2

can we have more respect, even in bugzilla? There are other ways to express the same and they are more friendly. Thanks...

I flagged this as normal priority, since it affects only 1 user and is likely a problem of local copy... If puppet class is broken, fix it! But IMHO I don't see why a puppet class would clone a repository in a way that it's broken later.

(In reply to comment #28)

I flagged this as normal priority, since it affects only 1 user and is
likely a
problem of local copy... If puppet class is broken, fix it! But IMHO I don't
see why a puppet class would clone a repository in a way that it's broken
later.

It's been replicated, and it's not puppet. Please see recent comments from Marcin.

ok that's true but is it really "critical"?

This issue may not be related to git at all. Were you using /data/project or /home to store your repository? If so, it's possible that glusterfs got corrupted (git files were damaged or aren't accessible), which happens quite a lot these days and may need a recovery. Usually a best way is to reboot and mount glusterfs again using latest version of gluster daemon.

If it's not a problem of gluster I don't see how it is related to wikimedia labs.

(In reply to comment #30)

ok that's true but is it really "critical"?

No, I don't think it's critical (I wasn't disagreeing with lowering the priority). I was just making sure we were all clear that it's a jgit bug most likely.

I highly doubt it's a gluster issue, since replication of the bug was possible locally. In fact, moving it to Git/Gerrit where it belongs.

I cloned mediawiki to /srv/mediawiki, so no glusterfs partition.

(In reply to comment #27)

For now I suspect that JGit does not fully support sending object packs for
shallow clients.

Is this still the case?

This may now be moot... the actual use case (in Labs) now does a full clone, thanks to performance improvements in gerrit.

I reported this bug. I think, I can now close it.

Related URL: https://gerrit.wikimedia.org/r/65873 (Gerrit Change If5cf784f37aea49ca6c96c9fb1f50104d073b5b2)

The snapshots tool for MediaWiki nightly branches started throwing this warning recently

• Thu, 27 Feb 2014 17:00:07 +0000
• Starting update process for snapshots of mediawiki-core... --

Forced clean up and reset...
$ rm -f .git/index.lock
$ git clean -d -x --force -q
$ git reset --hard HEAD -q
$ git checkout 'master' -q

Fetch updates from remotes...
fatal: pack has 21 unresolved deltas
fatal: index-pack failed
error: Could not fetch origin

As a result 1.23wmf15 and 1.23wmf16 never made it there, for example.

Is there something I can do (short of rm -rf the clone, and re-init/re-clone every time) to defend against this so that I can have this cronjob run without any worries?

[17:24 UTC] local-snapshots at tools-login in ~/src/mwSnapshots/remotes/mediawiki-core (master)
$ git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (63675/63675), done.
dangling commit 6146a7d310181b23618b74fcba580f8919a8ba6d
dangling commit rMW10ba3d7caa1e

Didn't solve it apparently.

[17:24 UTC] local-snapshots at tools-login in ~/src/mwSnapshots/remotes/mediawiki-core (master)

$ git remote update
Fetching origin
remote: Counting objects: 10144, done
remote: Finding sources: 100% (1201/1201)
remote: Getting sizes: 100% (286/286)
remote: Compressing objects: 100% (2020458/2020458)
remote: Total 1201 (delta 788), reused 1023 (delta 768)
Receiving objects: 100% (1201/1201), 2.84 MiB | 2.58 MiB/s, done.
Resolving deltas: 97% (823/844), completed with 331 local objects.
fatal: pack has 21 unresolved deltas
fatal: index-pack failed
error: Could not fetch origin

1 $ git fsck --full
Checking object directories: 100% (256/256), done.
Checking objects: 100% (63675/63675), done.
dangling commit 6146a7d310181b23618b74fcba580f8919a8ba6d
dangling commit rMW10ba3d7caa1e

$ git remote update
Fetching origin
remote: Counting objects: 10144, done
remote: Finding sources: 100% (1201/1201)
remote: Getting sizes: 100% (286/286)
remote: Compressing objects: 100% (2020458/2020458)
remote: Total 1201 (delta 788), reused 1023 (delta 768)
Receiving objects: 100% (1201/1201), 2.84 MiB | 2.61 MiB/s, done.
Resolving deltas: 97% (823/844), completed with 331 local objects.
fatal: pack has 21 unresolved deltas
fatal: index-pack failed
error: Could not fetch origin

I tried the trick saper mentioned about using another git remote (e.g the git.wikimedia.org mirror) to fix it up.

$ git fetch origin
..
fatal: pack has 26 unresolved deltas
fatal: index-pack failed


$ git remote add mirror https://git.wikimedia.org/git/mediawiki/core.git

$ git remote -v
mirror	https://git.wikimedia.org/git/mediawiki/core.git (fetch)
mirror	https://git.wikimedia.org/git/mediawiki/core.git (push)
origin	https://gerrit.wikimedia.org/r/p/mediawiki/core.git (fetch)
origin	https://gerrit.wikimedia.org/r/p/mediawiki/core.git (push)

$ git fetch mirror
remote: Counting objects: 3886, done
remote: Finding sources: 100% (1927/1927)
remote: Getting sizes: 100% (3130/3130)
remote: Compressing objects:  99% (59991/60007)
remote: Total 1927 (delta 716), reused 1044 (delta 495)
Receiving objects: 100% (1927/1927), 14.18 MiB | 2.99 MiB/s, done.
Resolving deltas:  75% (903/1197), completed with 170 local objects.
fatal: pack has 294 unresolved deltas
fatal: index-pack failed

# Still fails.. Trying again with limited depth

$ git fetch -k --depth 1 mirror
remote: Counting objects: 9253, done
remote: Finding sources: 100% (9253/9253)
remote: Getting sizes: 100% (8258/8258)
remote: Compressing objects:  99% (165096/165109)
remote: Total 9253 (delta 2901), reused 4979 (delta 978)
Receiving objects: 100% (9253/9253), 55.62 MiB | 3.59 MiB/s, done.
Resolving deltas: 100% (3895/3895), done.
From https://git.wikimedia.org/git/mediawiki/core
 * [new branch]      wmf/1.23wmf16 -> mirror/wmf/1.23wmf16
 * [new tag]         1.22.3     -> 1.22.3

# That's better, now trying origin again

$ git remote rm mirror
$ git fetch origin
remote: Counting objects: 14492, done
remote: Finding sources: 100% (8/8)
remote: Getting sizes: 100% (2/2)
remote: Compressing objects: 100% (7196/7196)
remote: Total 8 (delta 4), reused 5 (delta 4)
Unpacking objects: 100% (8/8), done.
fatal: unresolved deltas left after unpacking
fatal: unpack-objects failed

Still broken..

fatal: unresolved deltas left after unpacking
fatal: unpack-objects failed
$ git fetch --prune
Command failed:
$ git fetch --prune
Return: 128

Created a new git clone from git.wikimedia.org in a temp directory and removed the old one and moved this one in its place.

Whatever happens, if you get this bug, don't waste time trying to fix it. Just re-clone and hope you don't have any local commits or branches you care about.

Meh.