InfoQ Homepage News How Continuous Delivery Impacts Testing

How Continuous Delivery Impacts Testing

This item in japanese


With continuous delivery we need to focus on quality as we write the code. Not every team will have testers, but if there are testers then they will work closely with developers, writing code to automate the small number of tests that cannot be covered by unit tests while helping developers creating unit tests.

Jeff Morgan, chief executive officer of Industrial Logic Canada, talked about how continuous delivery impacts the tester’s job at eXperience Agile 2018. InfoQ is covering eXperience Agile with Q&As, summaries, and articles.

The power of continuous delivery is to quickly do experiments in production, said Morgan. It allows us to rapidly test product ideas with our users and collect data on the results.

Unit test is where most of our application is tested. There are no manual tests; automation is what drives testing, argued Morgan. There is no hardening phase, we have to build in quality from the start.

Morgan suggested that if you need specialized testing knowledge, then bring in a specialist but have them work with the teams instead of writing tests themselves. The teams own the tests, not the specialist.

As testers have become more technical and developers have become more test aware, over time developers and testers have become indistinguishable, said Morgan.

InfoQ interviewed Morgan about how continuous delivery impacts testing.

InfoQ: What are the major changes to the way that software is built and delivered when organizations adopt continuous delivery?

Jeff Morgan: The primary change is that there is no longer time for the traditional approach to software development, where we build it and then enter a testing phase. With continuous delivery the code is deployed to production as soon as it is checked into source code management and traverses a deployment pipeline.

This significant change means that testing needs to happen simultaneously with development. In fact, we typically do not view development and testing as two separate activities. Instead, testing is just a part of all development. This change also means that there is a much higher reliance on automation - test automation, deployment automation, and environment automation.

InfoQ: How does continuous delivery impact the way that we should think about quality?

Morgan: Quality is often thought of as something we verify/add at the end of the development cycle just prior to release. This is often called a hardening phase. With continuous delivery there is no need to have a hardening phase as the code goes to production as soon as you check it in. Instead, we need to focus on quality at all times as we write the code.

For the teams I work with this means a fundamental change in the way we write code. We adopt practices like pair programming/mobbing, test driven development, elimination of branches, and feature toggles. Developers are far more test and quality aware. If there are testers on the teams, they are typically focused on the small amount of end-to-end automation and pairing with the developers on exploratory testing.

The testing of non-functional requirements belongs to "everybody" on the team instead of the traditional approach of it belonging exclusively to testers. On the very best teams there is very little distinction between many of the developers and testers.

We also use what many call DevOps to help us drive risk out of the system. Build pipelines run all of our tests in different ways to mitigate risk, as well as run the tests for concerns like security, capacity, and compliance issues. Containers or Infrastructure as Code are used to mitigate environmental risk. Controlled releases of small changes or experiments are used to mitigate risk to the users as well as product.

InfoQ: How does it impact the tester’s job?

Morgan: First of all I should state that not every team in this new world has testers. If there are testers then they are not executing manual test scripts. Instead, they are writing code to automate the small number of tests that cannot be covered by unit tests while helping developers to move more of the testing down the pyramid to unit tests.

They work in very tight collaboration loops with the developers and are assisting with the construction of the pipeline. They are also performing exploratory testing by pairing with developers throughout delivery. If defects are found they are pairing with the developers to rapidly fix and deploy the correction. All things are done in collaboration with developers.

InfoQ: What should testers do when there is no time for a "testing phase"?

Morgan: My experience is that in continuous delivery far more testing takes place than in what I would call traditional "agile". Also, testing is happening at different times in the development lifecycle - throughout instead of at the end. We heavily rely on automation (primarily unit tests) and our pipeline to give us a sense of the quality of the system.

Testers should work closely with the developers to ensure that quality is maintained and that defects almost never occur. Testers typically focus on the "User Tests" from the pyramid above. They can also pair with developers to help them acquire and improve their testing skills. Testers also have a role to play in the definition of the build pipeline.

Sometimes there is a need for a test specialist like a security or load/performance testing expert. In these cases, they would consult with the team to create and maintain the test scripts to run in their stage of the pipeline.

In many ways, the pipeline is the closest thing we have to a "testing phase". This is where all of our tests run each time we check in code and ultimately deploy our code to the production environment. Although all pipelines are different, there are some core patterns that I find common from team to team. Here is an example of the stages that a team might have:

  1. Build and run the unit tests
  2. Run static code analysis and fail if quality is not achieved
  3. Deploy with all work in progress feature toggles on and run only the tests that focus on un-released behaviour. Any backend system outside of our control should be mocked.
  4. Deploy with all work in progress feature toggles off and run all remaining tests to provide regression. The tests need to run fast so they should run in parallel. Also, any backend system outside of our control should be mocked.
  5. Deploy with toggles off and run a small subset of tests across multiple browsers and devices. Backends should be mocked.
  6. Deploy with toggles off and run a very small subset of tests without mocks. This ensures full integration works properly.
  7. Run static and dynamic security tests.
  8. Run load and performance tests.
  9. Run compliance tests and address any additional pre-production compliance tasks.
  10. Deploy to production.

If any stage fails, the pipeline stops.

InfoQ: What’s your advice to organizations that want to adopt continuous delivery?

Morgan: Many companies I talk to state that they are in the process of adopting continuous delivery. When I ask them why, they typically talk about faster delivery or improving quality. While these are good goals, I think there are more important benefits.

In my view, the primary reason to adopt continuous delivery is to change the way we plan and deliver products. Continuous delivery allows us to eliminate laborious and speculative long-term product roadmaps and instead adopt a more "Lean Startup" approach to product design. It allows us to quickly course correct based on real users' experience in order to deliver the correct product. This is where I think the focus should be.

For companies that are going down this path, they should understand that DevOps is not the solution to all of their problems. While it is an important component, it must be paired with high-quality development. In fact, I would suggest focusing on dramatically improving quality first and expect it to take some time. Once this is underway you can look at adding continuous integration so the developers can get rapid feedback on their build process. Over time you can extend it into a full fledged deployment pipeline.

Once you have the culture of quality installed in your organization, then you can add the DevOps automation necessary to delivery many times a day.

Rate this Article


Hello stranger!


Get the most out of the InfoQ experience.


Community comments






Company name:
Company role:
Company size: