A related post I sent to the labs mailing list
http://lists.wikimedia.org/pipermail/labs-l/2013-November/001796.html

-----------------8<---------------------8<--------------------8<--------

I could use a new labs project on the wmflabs project to boot up new instances that would be consumed by the Continuous integration system. The aim is to be able to properly isolate some tests we are running.

The idea is to maintain a pool of instances that would be dynamically registered as Jenkins slaves of the CI Jenkins master. The later would then be able to send test to run on those slaves and we would destroy the instance once the job is completed.

The requisites would be:

• a way to access the OpenStack API so I could script the creation of instances using https://pypi.python.org/pypi/python-novaclient/

• isolate the instances from the rest of the prod/labs networks using a yet to be written security matrix. The instances would at least need:
  • to be reachable from the Jenkins master by ssh
  • the availability to fetch from npm

• A virtual image using Ubuntu Precise and prepopulated withrole::ci::slave::labs::common and contint::slave-scripts. The idea is to have the new instances booting up as fast as possible.

• A host profile with 2GB of RAM, single CPU and 5GB of /dev/vdb disk.

Optionally:

• having the project to run on dedicated hardware, but that can happen later on.

Who shall I sync with to make it happen? :-]

Should I fill in a bunch of bugs in Bugzilla for ease of tracking?

-----------------8<---------------------8<--------------------8<--------

I guess I will fill above tasks as bugs depending on this one.

As this setup sounds *very* similar to the concept of Travis CI (https://travis-ci.org/), have you looked into that? It might be easier than to hammer Labs and Jenkins into shape, and WMF could just sponsor some hardware (or money :-)).

That sounds like a great idea!
As far as I am aware the only functionality travis doesn't currently have that we need would be gerrit integration.

We don't use hosted solutions when we can sanely host them ourselves and we don't use proprietary things unless there's no available alternative.

Setting up a labs project for this is doable, so we should be doing it. We just need to take the time to do it. I think priority wise, this should be after the Labs move to eqiad.

What Ryan said, we can't depend on a third party at all. That doesn't prevent us from reusing Travis open source code and integrate it in our build system, that would nicely scale out CI to more people.

Re Ryan and Hashar, https://github.com/travis-ci, Sorry that is what I meant!
Set up our own slightly modified travis!
I can see our CI taking a big step forward when this bug is resolved :D

We have labs instance in the integration projects: integration-slave1001.eqiad.wmflabs and integration-slave1002.eqiad.wmflabs.

They are fully puppetized and let us fetch packages from nodejs/rubygems/pip...

Still does not provide proper isolation nor the ability to spawn new instance on demand.

I am merging this up in its parent task T47499: [EPIC] Run CI jobs in disposable VMs. Will fill some new tasks for each of the item listed there.

I have filled new tasks for the list of item that were there:

T86168: Isolate contintcloud nova project from the rest of the wmflabs cloud
T86170: OpenStack API account to control `contintcloud` labs project
T84989: Support dedicating a specific virt node to a specific nova project