What is a good CI build-process

Question

What constitutes a good CI build-process?

We use CI, but is deployment to production even a realistic CI goal when you have dependencies on several services that should be deployed too and other apps may depend on these too.

Is a good good CI build process good enough when its automated to QA and manual from there?

Answer 1

Be sure you understand the idea behind a CI build. CI stands for Continuous Integration and CI builds are really intended to be throw-away builds that are performed when a developer checks code in to the source control system (or at some specified interval) to ensure that the newest changes do not break the code base (hence the idea of continuously integrating the changes to the code base).

To that end, the technology used for the actual build server process is largely irrelevant compared to what actually happens during the build. As @pdavis mentioned, the CI build should compile the code base, execute some code analysis (FxCop, StyleCop, Lint, etc.), execute unit tests and code coverage, and execute any other custom analysis you want performed that should impact the concept of a "successful" or "failed" build.

Having a CI build automatically deploy to an environment really doesn't fall under the control of a build server. That being said, you can always create a separate project that runs on the build server that handles the deployment when it detects certain conditions (such as a build completes successfuly), but that should always be done as a completely independent thing.

Answer 2

A good CI process will have full or nearly-full unit test coverage. Unit tests test classes and methods, vs. integration tests, which test multiple parts of the system. When you set up your CI builds, have them automate the unit tests. That way, the CI builds can run multiple times per day. We have ours set to run every 2 hours.

You can have longer running builds that run once per day. These can use other services and run integration tests.

Answer 3

I am starting on a new project at work that I am really looking forward to. We are still in the initial design stage and have just recently completed the Logical System Architecture. We have ordered new servers for the testing and staging environments and are setting up a Continuous Integration (CI) build system based on Cruise Control (http://cruisecontrol.sourceforge.net/) and MSBuild (http://msdn2.microsoft.com/en-us/library/wea2sca5.aspx) which is basically an improved port of ANT. It appears that Visual Studio 2005 project and solution files are all now in MSBuild format. Cruise Control will be automatically pulling the source from Visual Source Safe (ok, it isn't Subversion but we can deal), compiling it, and then running it through fxCop (http://www.gotdotnet.com/Team/FxCop/), nUnit (http://www.nunit.org/), nCover (http://ncover.org/site/), and last but not lease Simian (http://www.redhillconsulting.com.au/products/simian/). Cruise Control has a pretty good website interface for displaying all of the compiled results from the various tools and can even display code changes from one build to the next. It also keeps track of all builds in a build history. I'm looking forward to the test driven development and think that this type of approach combined with nUnit/nCover should give us a pretty good idea before we roll out changes that we haven't broken anything. There are also plans to incorporate some type of automated user interface testing once we are far enough along in the project. Depending on the tool, this should be just a matter of installing the tool on the build server and calling it from Cruise Control. Sweet.

Answer 4

I was watching a ThoughtWorks presentation (creators of Cruise Control) and they actually addressed this issue. Their answer is that NO deployment is too complex to test. Why? Because otherwise, your customers become your testers, which is exactly where you don't want to be.

If you have a complex deployment structure, set up a visualization server. Have it pretend to be all the systems you need to talk to. They can always start in a known good state, because you can reset to a clean image.

To answer your initial question, a good process is one which enables communication between the repository and the developers. If the repository is in a bad state (non-compiling code, failed tests, etc.), the developers know about it as soon as possible, so that they can correct it.

Answer 5

Is a good good CI build process good enough when its automated to QA and manual from there?

I think, that "manual" deployment is used when person with deployment-engineer role have a fear that something after deployment can go wrong. He haven't confidence in deployment tools reliability and stability.

This feat can go away with automated deployment process testing on prod-like environment.It also has to test a rollback mechanism after deployment.

When this functions are tested enough you will gain a confidence in this process and tool you use.

Answer 6

Well "it depends" :)

We use our CI system to:

1. build & unit test
2. deploy to single box, run intergration tests and code analisys
3. deploy to lab environment
4. run acceptance tests in prod-like system
5. drop builds that pass to code drop for prod deployment

This is for a greenfield project of about a dozen services and databases deployed to 20+ servers, that also had dependencies on half a dozen other 'external' services.

Using a CI tool to deploy your product to a production environment as a realistic goal? again... "it depends"

Why would you want to do this?

• if you have the process you can roll changes (and roll back) faster and more often
• less chance for human error
• you can test the same deployment strategy in a test environment before going to production and catch issues earlier

Some technical things you have to address before you can answer this:

• what is the uptime requirements for your system -- Are you allowed to have downtime or does it need to be up 24/7?
• do you have change control processes in place that require human intervention/approval?
• is your deployment robust enough for any component to roll back to a known-good state if a deployment fails?
• is your system designed to handle different versions of services or clients in case one or several component deployments fails (and you have the above rollback to last known good)?
• does the process have the smarts to handle a partial deployment where a component cannot handle mixed versions of its dependencies/clients?
• how are you handing database deployment/upgrades?
• do you have monitoring in place so you know when something goes wrong?

Here are a couple of recent related links about automation and building the tools you need.

When it comes down to it the more complex your system the more difficult it is do automate everything, but that does not mean it is not a worthy goal, it just takes a lot more effort and willpower to get it done -- everything from knowing the difficulties you're going to face, the problems you have to account for (failure will happen), the political challenges of building infrastructure (vs. more product features).

Now heres the big secret... the technical challenges are challenging but not impossible... the political challenges may be insurmountable. Everything about this costs money whether its dev time or buying 3rd party solutions. So really, can you build the $1K, $10K, $100K, or $1M solution?

Whatever solution you go for make sure the automation is robust first, complete second... i.e. make sure you have as robust a solution as you can for getting deployment to a test environment rather than a fragile solution that deploys to production.