Page MenuHomePhabricator

Local private files on deployment host should be backed up somewhere
Closed, ResolvedPublic

Details

Reference
bz67818

Event Timeline

bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:36 AM
bzimport set Reference to bz67818.

14:53 < Reedy> Needs backup/remote repo (and backup of that)

Maybe put it the private ops repo we have for puppet configs?

(In reply to Krinkle from comment #2)

Maybe put it the private ops repo we have for puppet configs?

Then no one who can deploy, who may need to change PrivateSettings (most likely, adding one), will be able to do so. Presumably that then also depends on puppet ensuring it's in place on tin, to then be synced out. Unless it goes somewhere else on the apache (et al) file systems, and we include/symlink it from there...

I've created a private repo in /a/common/private, but it needs a remote to push somewhere (so it can be backed up), but obviously not gerrit. That's an aside. Not synced to production

AdminSettings.php merged into PrivateSettings.php in the 2nd commit to the repo.

private added to .gitignore in https://gerrit.wikimedia.org/r/145403

PrivateSettings.php and AdminSettings.php removed from .gitignore, and replaced with symlinks to ../private/PrivateSettings.php in https://gerrit.wikimedia.org/r/145404

Need to do something similar on beta for testing (and so it doesn't break it)

Andrew subscribed.

Many of these settings come from puppet, so the raw file can't be in mw repo; it's already puppetized as intended.

Wait, I'm totally wrong, confusing this with something else

Reedy removed Reedy as the assignee of this task.Sep 27 2015, 2:02 PM
Reedy subscribed.
Krenair renamed this task from PrivateSettings.php should be in a repo to Private repository should be pushed to somewhere remote.Sep 27 2015, 2:05 PM
Krenair updated the task description. (Show Details)
Krenair set Security to None.
Krenair subscribed.
Krinkle renamed this task from Private repository should be pushed to somewhere remote to Local private files on deployment host should be in a repo somewhere.Aug 7 2018, 8:56 PM
Krinkle renamed this task from Local private files on deployment host should be in a repo somewhere to Local private files on deployment host should be backed up somewhere.Aug 7 2018, 8:59 PM

The issue of having both AdminSettings and PrivateSettings was fixed (T69820, removed AdminSettings).

The issue of not having a backup of recent versions of PrivateSettings still remains, narrowing this task to be about that.

Is PrivateSettings.php a single file that exists on the deployment host and that is shared between everyone with deploy access? Or does everyone have their own?

Is PrivateSettings.php a single file that exists on the deployment host and that is shared between everyone with deploy access? Or does everyone have their own?

It's got a list of passwords and stuff used in wmf-config. So everyone "shares" it

The folder/repo is also used for some spam prevention things we don't want in public

Krinkle added a subscriber: jcrespo.

I believe this has since been done, per T125527: Backup all of /srv on mira and/or tin (deployment servers).

@jcrespo Can you confirm that /srv/mediawiki-staging/private/* is covered by baculate (eg. not somehow excluded from /srv/).

This is the contents on backups for deploy1002 as of yesterday (under NDA, because I don't know if the file names are sensitive):

(Go to line 279)
{P17346}

And this is the contents for deploy2002 (also NDA):

(Go to line 283)
{P17347}

So I consider it covered.

@Krinkle before closing this, could you ask me to restore a file at some point in time (within the last 2 months, and we don't have to replace existing files, we can recover to a separate path) to make sure the backups are working as expected- we should regularly test recoveries to make sure they are working, but "testing" tasks will vary from service to service.

Waiting for an answer, even if it is "no". :-)

@jcrespo OK, could you restore /srv/mediawiki-staging/private/PrivateSettings.php from ~4 weeks ago to my home directory?

Done, the exact file is at deploy1002:/home/krinkle/restore/srv/mediawiki-staging/private/PrivateSettings.php and its mod time is 2021-05-26 17:14:22 You can diff it against current one and see if it corresponds to the status at 2021-09-16 04:13:11, the moment the backup took place (4 weeks earlier- we have daily backups).

I didn't modify the original file permissions.

Please check the file is as expected, otherwise please communicate any issue so I can solve it.

If everything looks right, please delete it from your home (you should have the permissions, otherwise ping any root) and we can resolve this ticket :-). Thanks for testing the backups, they are forgotten sometimes but they help making sure they are ready at an unexpected time!

Log of commands for the record:

ssh backup1001.eqiad.wmnet
$ sudo bconsole
*restore
*6
*2021-09-16 08:41:00
*75
*2
*cd /srv/mediawiki-staging/private/
*ls
*mark PrivateSettings.php
*done
*mod
*9
*/home/krinkle/restore
*yes

I didn't modify the original file permissions.

The original on /srv/mediawiki-staging/private, as a deployer, I'm naturally able to read, write, and navigate/create files in:

$ l /srv/mediawiki-staging/private
drwxrwsr-x  3 wikidev 4.0K Dec  9 19:35 ./

The restored copy, I'm not able to open, and thus unable to verify the contents of.

$ l ~/restore/srv/mediawiki-staging/
drwxr-xr-x 3 root    4.0K Oct 14 08:44 ./
drwxr-xr-x 3 root    4.0K Oct 14 08:44 ../
drwxrw-r-- 2 wikidev 4.0K Oct 14 08:44 private/

$ cd ~/restore/srv/mediawiki-staging/private/
bash: cd: /home/krinkle/restore/srv/mediawiki-staging/private/: Permission denied

Maybe this was a genuine permission error at the time of the backup. If it was, maybe we can try again with a slightly more recent version. (If so, feel free to first recursively delete the root-owned /home/krinkle/restore/.)

Thanks, this was useful to detect something applicable for future automated recoveries. Bacula backups full paths, and keeps the permissions of those paths, so restore, restore/srv, and restore/srv/mediawiki-staging were kept as root.

I thought that those would be still browsable, but it turns that no.

I've alterted the permissions on those 3 dirs, while keeping everyhing else intact, and will wait for your feedback.

I see the group of the parent dirs is now wikidev, but note that the issue was not with the parent dirs. I was able to enter those fine, as they were root/root but world-readable. The issue is with the private/ directory itself, which was already wikidev-grouped, but lacking the needed executable bit on group or world for me to open it:

drwxrw-r-- 2 urbanecm wikidev 4.0K Oct 14 08:44 private/
bash: cd: private/: Permission denied

Actually, you may have found something that I am unable to answer you on: that directory (the original one) seems to have the setgid bit on, which of course bacula won't maintain. If that is intended, or how to handle, I cannot tell you. I would ask you to inquiry what is the best way to solve this for the original files, or if we have to workaround it.

ls -l /srv/mediawiki-staging
total 376
...
drwxrwsr-x  3 pybal-check wikidev   4096 Dec  9 19:35 private

Check now I've made another restore into /home/krinkle/restore2/ and I think you should be able to see it. I think I see the issue- the setgid forces the inheritance from the parent, which will do weird things when restoring to not-root. This is an important thing to notice when doing non-destructive restores, but I think it shouldn't hit us for a regular restore.

Still, I wonder if the Setgid is intended.