Page MenuHomePhabricator
Create Task
Maniphest T70480

Scale merge operations (tracking)
Closed, ResolvedPublic
Actions

Description

Zuul merge operations are slowing down Zuul processing. git fetch is too slow when there are thousands and thousands of references. There is only one Zuul merger handling the work.

This is a tracking bug.

Details

Reference
bz68480

Related Objects
Search...

StatusSubtypeAssignedTask
 ResolvedhasharT70480 Scale merge operations (tracking)
 DeclinedhasharT70482 Zuul: setup a second merger on lanthanum
 ResolvedhasharT70481 Zuul repositories have too many refs causing slow updates
 DeclinedNoneT103528 Run zuul-clear-refs.py daily on all our repositories to reclaim Zuul references
 ResolvedhasharT103529 Package / puppetize zuul-clear-refs.py
 ResolvedelukeyT145057 Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger)
 ResolvedhasharT147073 Upgrade Zuul on scandium to 2.5.0-8-gcbc7f62-wmf3jessie1
 ResolvedhasharT95046 install/deploy scandium as zuul merger (ci) server
 ResolvedhasharT93076 eqiad: 2 hardware access request for CI isolation on labsnet
 ResolvedhasharT86661 Disable Gerrit replication to production slaves
 ResolvedhasharT86659 Migrate all jobs to labs slaves
 ResolvedKrinkleT86175 Migrate jsduck jobs to run in labs (and jsduck-publish via integration-publisher)
 ResolvedKrinkleT86174 Migrate JSDuck to Ubuntu Trusty
 ResolvedKrinkleT93143 Migrate .*testextension-zend jobs to labs slaves
 ResolvedhasharT93559 Migrate mediawiki-core-code-coverage job to labs
 ResolvedKrinkleT93558 Add publishing macro for integration.wikimedia.org/cover
 ResolvedhasharT96463 Move integration-zuul-layout* jobs to labs slaves
 ResolvedhasharT95675 Migrate Jenkins job "Global-Dev Dashboard Data" to JJB and Zuul trigger
 ResolvedhasharT98737 Migrate operations-dns-lint Jenkins job to labs instances
 ResolvedhasharT101966 Remove use of label "hasSlaveScripts" in jobs
 ResolvedhasharT116921 Remove CI root access from scandium
 ResolvedAndrewT119526 Give contint-admins sudo rights to start/stop zuul-merger
 Resolved• chasempT116975 Allow network flow between labs instance and scandium
 ResolvedhasharT117223 Package Zuul for Debian Jessie
 ResolvedhasharT118340 Upload new Zuul packages on apt.wikimedia.org for Precise / Trusty / Jessie
ResolvedhasharT118856 git-daemon for zuul-merger is not monitored
 ResolvedhasharT118861 On Jessie, puppet does not start zuul-merger via init scripts

Event Timeline

• bzimport raised the priority of this task from to Medium.Nov 22 2014, 3:28 AM
• bzimport added projects: Continuous-Integration-Infrastructure, Tracking-Neverending.
• bzimport set Reference to bz68480.
• bzimport added a subscriber: Unknown Object (MLST).
hashar created this task.Jul 23 2014, 9:30 PM
• demon unsubscribed.Dec 16 2014, 6:05 PM
hashar closed subtask T70482: Zuul: setup a second merger on lanthanum as Declined.Mar 17 2015, 9:35 AM
Krinkle moved this task from Untriaged to Backlog on the Continuous-Integration-Infrastructure board.Apr 13 2015, 4:39 PM
hashar added a subtask: T95046: install/deploy scandium as zuul merger (ci) server.Oct 28 2015, 2:10 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptOct 28 2015, 2:10 PM
hashar closed this task as Resolved.Oct 28 2015, 2:11 PM
hashar claimed this task.

I could net get a zuul-merger process on lanthanum because of network restrictions.

We are now going to setup scandium on the labs network ( T95046 ). From there it will be easy to add more zuul-merger instances.

This bug was from Bugzilla era to facilitate tracking. With Phabricator fulfill that role. Hence abandoning this tracking bug.

Dzahn closed subtask T95046: install/deploy scandium as zuul merger (ci) server as Resolved.Nov 30 2015, 6:49 PM
greg added a project: Essential-Work.Jan 13 2016, 8:38 PM
Danny_B renamed this task from Zuul: scale merge operations (tracking) to Scale merge operations (tracking).Jul 10 2016, 1:51 AM
Danny_B added a project: Zuul.
Danny_B updated the task description. (Show Details)
Danny_B removed a subscriber: • wikibugs-l-list.
• Phabricator_maintenance removed a parent task: T4007: [DO NOT USE] Tracking bug [superseded by #Tracking].Jul 28 2016, 2:33 AM
Liuxinyu970226 moved this task from Tag to Transition completed / Archived on the Tracking-Neverending board.Mar 21 2019, 10:31 AM
hashar closed subtask T70481: Zuul repositories have too many refs causing slow updates as Resolved.Apr 10 2019, 1:16 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL