Not being able to force a git pull on the mediawiki-installation nodes via dsh means that new scap features which are merged can't be immediately guaranteed to be available across the cluster.

The immediate problem could be solved by a root running:

dsh -g mediawiki-installation -M -F 40 -- 'chmod g+w /srv/scap/docs'

That won't solve the underlying problem however.

https://gerrit.wikimedia.org/r/#/c/115851/

Change 115851 had a related patch set uploaded by Hashar:
fix wrong scap/doc permissions.live hack->puppet

https://gerrit.wikimedia.org/r/115851

The /srv/scap repo on tin (and likely all the other nodes) is now blocked from update by non-root due to directories in .git/objects with the same 2755 permissions problem.

Change 115851 abandoned by Dzahn:
fix wrong scap/doc permissions.live hack->puppet

Reason:
per "This fixes a symptom, but the real problem is the behavior/configuration of git:clone. "

https://gerrit.wikimedia.org/r/115851

There is a bunch of comments on https://gerrit.wikimedia.org/r/115851 which might lead to a proper solution (aka get git to ignore user umask in favor of hardcoded settings).

Patch was abandoned. Root cause fix looks like it will require relatively deep puppet modifications (changing git::clone to set git configuration on new repos) or a rethinking of the deployment mechanism used.

(In reply to Bryan Davis from comment #8)

Root cause fix looks like it will require relatively
deep puppet modifications (changing git::clone to set git configuration on
new repos) or a rethinking of the deployment mechanism used.

Bryan:
So is either fixing this properly or finding an acceptable workaround still highest priority, relastically speaking? Or should priority be set lower?

(In reply to Andre Klapper from comment #9)

So is either fixing this properly or finding an acceptable workaround still
highest priority, relastically speaking? Or should priority be set lower?

Ori was working on this on Friday. I think it's pretty close to being fixed. We seem to have permissions fixed now for members of the wikidev group. In the process however the permissions on the /srv/scap top level directory were broken.

$ ls -ld /srv/scap
drwxrws--- 6 root wikidev 4096 Mar 10 03:34 /srv/scap

The permissions on the top level directory should be 2775 (or even 0775) instead of the 2770 that is now set. Permissions seem to be set correctly on files and directories within /srv/scap.

The incorrect permissions on the base directory is causing the nightly l10nupdate cron job to fail.

Ori has fixed the directory permissions for /srv/scap in puppet and then manually corrected all of cluster hosts. I'll leave this open for a day until I'm sure that further puppet runs don't cause some sort of regression.

As of today:
gjg@tin:~$ ls -ld /srv/scap
drwxrwsr-x 6 root wikidev 4096 Mar 19 10:19 /srv/scap

gjg@mw1163:~$ ls -ld /srv/scap
drwxrwsr-x 6 root wikidev 4096 Mar 19 15:02 /srv/scap