Page MenuHomePhabricator

Puppet failures on deployment-bastion
Closed, ResolvedPublic

Description

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Must pass trusted_group to Class[Keyholder] on node i-0000010b.eqiad.wmflabs


Version: unspecified
Severity: normal

Event Timeline

bzimport raised the priority of this task from to Needs Triage.Nov 22 2014, 3:48 AM
bzimport set Reference to bz73520.

Seems to be caused by https://gerrit.wikimedia.org/r/#/c/173353/ keyholder: add /etc/keyholder.d and keyholder arm subcommand

role::deployment::deployment_servers::common has:

 class { '::keyholder':
    trusted_group => 'wikidev',
}

But ::key holder::private_key has:

include ::key holder

Which indeed as no default value for the $trusted_group parameter.

Assigning to Ori.

greg triaged this task as Medium priority.Nov 24 2014, 11:26 PM

hey operations, can some puppet guru help sort out a puppet error we have please?

Must pass trusted_group to Class[Keyholder]

On deployment-bastion.eqiad.wmflabs , the equivalent of tin on the beta cluster. Thanks!

The deployment-bastion node directly includes the 'keyholder' class via wikitech. That class now takes an argument ('trusted_group') but no arg is specified, hence the error.

You'll need to either specify keyholder::trusted_group in heira (https://wikitech.wikimedia.org/wiki/Heira:deployment-prep) or wrap the keyholder include in a role. Or just remove it from that class from the instance config; I'm not clear on if it's actually needed given that that class is included elsewhere.

removed keyholder from the instance config because it's specified in the deployment role.

Change 179875 had a related patch set uploaded (by Hashar):
mwdeploy private key is only for production

https://gerrit.wikimedia.org/r/179875

Patch-For-Review

Thanks @Andrew @mmodell , I know start understanding the power of hiera for the beta cluster :-]

deployment-bastion now fails with:

Error: /Stage[main]/Role::Deployment::Deployment_servers::Common/Keyholder::Private_key[mwdeploy_rsa]/File[/etc/keyholder.d/mwdeploy_rsa]:
Could not evaluate: Could not retrieve information from environment production source(s) puppet:///private/ssh/tin/mwdeploy_rsa

That is because the prod/labs common class role::deployment::deployment_servers::common has:

keyholder::private_key { 'mwdeploy_rsa':
    source  => 'puppet:///private/ssh/tin/mwdeploy_rsa',
}

https://gerrit.wikimedia.org/r/179875 moves it from the common to the production role class. That fixed the puppet run on deployment-bastion.

Joe mentioned this in Unknown Object (Diffusion Commit).Dec 16 2014, 8:14 AM

I added the correct patch here https://gerrit.wikimedia.org/r/#/c/180126/

Now on beta you just need to configure role::deployment::deployment_servers::common::key_source to some file in the beta private repo via hiera.

@bd808: (copied from gerrit)
Keys can be placed in beta via local commits in deployment-salt:/var/lib/git/labs/private. That is how the ssh keypair for beta's scap wrapper were provisioned.

so we should probably move the key that's currently manually created and create a local commit on deployment-salt, so that it can be distributed with puppet and linked to the keyholder role via hiera (as in rOPUPa3e9446daf80)

Change 179875 abandoned by Hashar:
mwdeploy private key is only for production

Reason:
Seems fixed, this patch is no more on the beta puppet master. Thank you!

https://gerrit.wikimedia.org/r/179875