Page MenuHomePhabricator

sync-dir needs to use rsync proxies like scap
Closed, ResolvedPublic

Description

ie tin deploys to X hosts, and then Y hosts can use the "nearest"/"quickest" host to grab from

Like scap does..


Version: wmf-deployment
Severity: normal

Details

Reference
bz59225

Event Timeline

bzimport raised the priority of this task from to Low.Nov 22 2014, 2:16 AM
bzimport added a project: Deployments.
bzimport set Reference to bz59225.
bzimport added a subscriber: Unknown Object (MLST).

What if we just continue down the path of making scap (and trebuchet) fast enough to not worry about a sync-file/sync-dir at all? iow: always use scap.

It reduces the complexity and code paths needed to be tested with deployment changes and encourages us to make scap as fast as possible (already around 3 minutes for small/no-op changes and no i10n update).

A no-op scap is taking 3 minutes vs <10 seconds for sync-file today.

I think it's going to be a while until we have speed parity between scap and sync-*. If we could efficiently determine if l10n updates were needed or not it would probably mostly eliminate the difference. In beta (much smaller host set), a no-op scap is taking just under 1 minute. If we could get that down to 20-30 seconds we could probably convince people that killing sync-* is reasonable.

Converting sync-* to python is the next big thing on my TODO list, so after we do the straight port we can think about either doing this or waiting until we get a pre-computed diff transfer mechanism (aka Trebuchet) figured out.

Change 135513 had a related patch set uploaded by BryanDavis:
Convert sync-dir and sync-file to python

https://gerrit.wikimedia.org/r/135513

Change 135513 merged by jenkins-bot:
Convert sync-dir and sync-file to python

https://gerrit.wikimedia.org/r/135513

Use of rsync proxies for sync-dir and sync-file implemented in I5ab7b2fba2f68d3d9219947ef8ab8ac6d0160d8e.