
Blogs Banner

Architecting for Continuous Delivery


While part of it is selecting the right tools for each aspect like version control, CD server, infrastructure configuration, monitoring and so on, an effective CD implementation doesn’t stop at that. For instance, you might use a particular version control tool and have a particular CI server set up, but if you are not checking-in code in small batches frequently, or don’t have automated tests, it's not really Continuous Integration.


In this article, I am going to talk about three recurring themes in CD enablement that I have seen in my experience as a developer:

  • The trouble with monolithic codebases and approaches to break it down
  • Designing the test suite for optimal feedback
  • Setting up a deployment pipeline as the backbone of CD


  • Sluggish build and application start-up time
  • Slow feedback loop with large test suites

Up to some point, you can attack this problem by parallelizing build tasks and test runs. Build tools like Gradle, Rake, and others, support parallel task capabilities that we can take advantage of. Additionally, CI tools including Snap CI support test parallelism with multiple build workers.

But for teams working on a monolithic codebase, this is usually not enough. Apart from the aforementioned problems, a large codebase can be difficult to work with and intimidating to new members on the team. When multiple functional teams work on the same codebase, there tends to be less ownership of the code, multiple implementation patterns begin to emerge, and it gets more difficult to get consensus. Technical debt accumulates and can easily spiral out of control.



There are examples of teams that have optimized their deployments for the monolith—Etsy, for instance, has written about its experiences with Continuous Development. However, it is worth noting that Etsy has also invested significant time building the custom tooling and infrastructure to support the approach.


Extract components


Componentization via Libraries

For a start, shared components (like a pagination UI component or a library to make database or API calls) can be extracted and added as binary dependencies on the application.

But when there are multiple functional teams working on the same repo, the biggest payoff comes when you can separate out each team’s work into components to be managed in their own repos.

One large-scale Open Source project doing this is the OpenMRS application, where independent modules are wired up together in the application runtime. A component in this case is a binary dependency that includes end-to-end functionality of a module including the UI. Deployment of a component can be managed on the deployment pipeline and means upgrading the component version on the parent application.



Componentization via Services

Depending on the situation, breaking down the monolith app into a composition of smaller runtime services can provide enormous leverage in terms of the autonomy it affords to the team, given that apart from service boundary or API, everything else is completely de-coupled from each other. A standalone service can be deployed independently and scaled appropriately based on its requirements. The Building Microservices book by Sam Newman is an excellent read on the subject.




When you start on a greenfield project, automated GUI tests or acceptance tests seem small and manageable and it is tempting to add as much coverage on that layer as possible. But unless tended to, these tests grow become unmanageable over time. [The ice cream cone anti-pattern for test suites is well described in this article.]

Indeed, teams that are saddled with a large number of acceptance tests end up getting slowed down by the very automation that was supposed to make the application delivery process easier and faster – tests taking many hours to run and producing random failures in the end. Acceptance tests also tend to be brittle and because they are harder to reproduce and fix, these tend to accumulate over time. In many instances, the quality starts being gauged in terms of the percentage of passed tests. “85% pass. Not bad compared to previous run!” This doesn’t say anything about the quality of the application.


Icecream Cone to Test Pyramid
(Adapted from watirmelon blog)





CI is essential but not sufficient for an effective application delivery workflow

Automating the deployment procedure is the first step and most CI tools like Jenkins or TeamCity do a good job of providing that capability. For example, you can set up build configurations for each phase – say a build phase or a unit test phase, acceptance tests – and also automate deployments to each environment.

While this is certainly useful, the trouble is that it is very difficult to answer the question: “Do we have confidence to release this version of software to Production?”


This is where the deployment pipeline helps.

CD Pipeline
(Source: Continuous Delivery book)



The deployment pipeline in essence represents a flow chart of your production process and enables you to automate and visualize your deployment process from source repo all the way to production. Each step of your build and deployment process can be modeled into the pipeline to provide high resolution visibility into your deployment workflow.

Below is the view of a simple pipeline on Snap CI:

Snap Simple Pipeline

Each commit goes through a series of stages that lead all the way to production. With each passing stage, you get higher confidence with that revision of the code. If something fails, the pipeline stops and you have to fix the build OR revert the commit that caused the failure. If the deploy to production fails, you can rollback by triggering the last successful “Deploy to Production” stage.



Snap Simple Pipeline

Above is the value stream map of all the dependencies involved on a deployment workflow down to production from the Go.CD website. It displays the state of each commit, and the dependencies of the application that need to be packaged. It provides the end-to-end visualization to production which indicates the confidence whether the application is releasable at any point.

You can see that the deployment pipeline can also easily support requirements like integration testing microservice dependencies and has the flexibility to support best practices like trunk-based development.

Thus, the deployment pipeline is the backbone that provides the right primitives for an effective CD implementation.



This article was originally published on Snap CI's The Pipeline blog.

Snap CI and Go.CD are both products from Thoughtworks. Snap CI is the hosted CI/CD service for repo based pipelines that have the essential capabilities for small teams working on cloud. Go.CD is an on-premise tool that supports complex pipeline dependencies that is useful for large teams and enterprises.


Keep up to date with our latest insights