CI systems are well known for mitigating risks, improving a quality of the code and system consistency, speeding up cycle time, and eliminating time-consuming and error-prone manual tasks.

In this note, I'm going to describe the requirements to CI for OpenStack, and give an overview of the solution my team has worked on.

The problem

OpenStack cloud being used in production may mean the constant flow of changes such as follows:

  • Changes to OpenStack packages (e.g. Nova, Horizon, and so on)
  • Changes to plugins or 3d-party components with which the cloud is to be deployed
  • Specific OpenStack feature changes
  • Development of new OpenStack components

In order to handle all those changes in a right way, you definitely have to have a CI pipeline.

Notice, though, if there is no constant development, you won't get enough benefits out of the CI system. That's why you don't need to have CI just in case.

CI requirements

Let's have a look at the user stories that may be specified to design the CI system.

As a cloud owner, I'd like to have code management system, so that all changes from DevOps team are stored and versioned.

As a cloud owner, I'd like to have review system, so that all changes from DevOps team are reviewed before merging into the master branch.

As a cloud owner, I'd like to have CI orchestration system, so that CI jobs can be set up to build, test and deliver packages.

As a cloud owner, I'd like to have package repository set up, so that all packages the CI system built can be stored in that repository for an upcoming deployment.

As a cloud owner, I'd like to have all changes to the codebase tested automatically, so that the bad code doesn't get into the master branch.

Solution overview

Essential rules we've been trying to follow when designing the solution:

  • Automate everything - testing, building, deploying
  • Test and review each commit
  • Prohibit developers from merging anything into the master branch until a commit passes tests and code-review
  • Keep master branch green - whenever you decide to deploy code, it is supposed to work

The core components of the solution meeting the described above requirements.

  • Gerrit as a code management and code review system
  • Jenkins as a CI server that orchestrates build, test and delivery tasks
  • Jenkins job builder as a job managing tool
  • Deployment tool is Fuel
  • Testing frameworks and tools (see more below)
  • Virtual multi-node environment as a testing environment
  • LDAP as authentication service

Architecture of the solution:

One of possible CI installations is to have 2 bare-metal node. Here is the picture worth thousand words.

CI deployment

Development workflow

  1. Whenever a DevOps engineer submits a commit to Gerrit, it triggers corresponding Jenkins job to start building a package
  2. A publishing Jenkins job signs the package and sends it to the local package repository and the Mirror VM
  3. Jenkins runs jobs to deploy virtual environment for testing the package
  4. Testing Jenkins jobs execute test suites to assess the package from different points of quality
  5. If tests pass, Jenkins will send +1 to Gerrit, otherwise -1
  6. Now, the commit is to be reviewed by other engineers
  7. If the change gets +2 from a core reviewer, it is ready to merge into the master
  8. Once the commit merged, corresponding Jenkins jobs build the package from the master branch and publish it on the Mirror repository

At this point, the package is ready to deploy to staging or production.

Continuous testing

Test automation is an essential part of the CI system. Not only it helps us prevent bad code from merging into the master, it also generates metrics that lead the team towards developing quality code.

In our solution, we rely on the following testing tools.

  • pep8, puppet-lint for syntax checks for Python and Puppet
  • py27 for unit testing
  • OSTF for smoke testing
  • Tempest for integration testing
  • Rally for performance testing
  • Selenium for UI testing (Horizon part)


The described above solution lets us leverage many benefits of using CI to handle continuous flow of changes to OpenStack cloud.