For testing, I've deleted /home/scfc-test, logged in as that and the directory was created just fine. Next up creating a new test account to see if there's something wrong with newly created users in LDAP.

Created [[wikitech:User:Tim Landscheidt (Test 2)]], added to Tools, uploaded ssh key, logged into tools.wmflabs.org, no problem at all. I'm out of ideas.

I'm wondering if that's a case of "logged in before the credentials were actually propagated to the NFS server" timing.

With scfc-test2, I tried logging in directly after adding to Tools and before /public/keys/scfc-test2 was created, and it just didn't accept the key, but it didn't half-create /home/scfc-test2.

Also, shouldn't the NFS server a) use LDAP, so any changes are instantaneous and site-wide, and b) not look up user names? I assumed that NFS servers accept any uids/gids and (if so) only handle 0 differently.

It does, but it's not instantaneous.

But also, we are using NFSv4; that doesn't just trust user IDs. :-)

So yeah, I'm not sure what happened in those cases either. Odd.

Apparently mga73 was another instance of this (only .bashrc, everything owned by root).

I think jorm was having this issue on unicorn.wmflabs.org.

It's possibly the same issue. Here's what happens when I try to log in:

zombieland:~ bharris$ ssh unicorn.wmflabs.org

If you are having access problems, please see:https://labsconsole.wikimedia.org/wiki/Access#Accessing_public_and_private_instances
Creating directory '/home/bharris'.
Unable to create and initialize directory '/home/bharris'.
Welcome to Ubuntu 12.04.2 LTS (GNU/Linux 3.2.0-57-virtual x86_64)

75 packages can be updated.
3 updates are security updates.

The last Puppet run was at Mon Mar 3 19:53:03 UTC 2014 (7 minutes ago).

• System restart required ***

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Last login: Mon Mar 3 19:31:09 2014 from 216.38.130.164
Connection to unicorn.wmflabs.org closed.

chown reports "Invalid argument" on new labs instances.

(In reply to christopher.johnson from comment #9)

chown reports "Invalid argument" on new labs instances.

Also, if the "Share home directories across instances" is disabled, the home directory cannot be initialized on login. This problem is specific to /home.

Yes, if there is no shared home, then it won't be able to mount a /home share. The puppet noise is annoying, but we haven't found a clean way to prevent that yet (whether or not the project requested shared homes is not visible from puppet). The warning is annoying but harmless.

Christopher, can you give me a bit more detail? chown with what values report invalid argument, and on which share?

Marc,

for example, adduser foo will fail if it is run as root, because the script chown's /home/foo. It is specific to /home only. NFSv.4 (idmapd) related?

Ah, that is normal behavior for NFS4, which deals in usernames and not user IDs. There are two ways around this: either make sure the username that needs to own shared files is known to the file server (in practice, "is in LDAP") or have local users' homes not be on NFS (/usr/lib is a reasonable place for 'system' users, for instance).

NFS4 does change the semantics of file ownership; rather than use user IDs and hope that they happen to match the same users on different clients, it relies on a central list of users for that.

What I have done, in cases where it's reasonable to do so, is add global system users to LDAP; this makes everything work as expected and is considerably more reliable and rely on user creation from puppet.

Is there still something I can do for you?