Developing Good Automated Test Suites

An automated test suite is more than just a collection of scripts. It requires careful planning, development, and maintenance to build a test suite that can provide helpful feedback on software quality. Small boutique shops pushing hot new apps and gigantic enterprises configuring warehouse workflows alike face similar challenges when automating tests: What should be tested? How should tests be written? And, how can we make testing a repeatable process?

The process for developing good automated test suites has three phases to answer those three questions: Discovery, Formulation, and Automation. Let’s learn about this process and how to apply it to our testing work.

What’s the process?

When I think about development and testing, I don’t see them as two separate activities. Instead, I see them as both connected to the behaviors of a software product. Ultimately, the behaviors are what provide value to the end user, and so that is where we should put our focus. “Testing” then becomes reframed as an activity for verifying that the behaviors we develop meet our users’ expectations, and “automation” expedites its completion.

The Discovery → Formulation → Automation process comes from Behavior-Driven Development (or “BDD” for short) and, specifically, the BDD Books. Even though these three phases typically describe development, they can also apply more narrowly to test automation. Remember, test automation is software development and requires all the same skills.

Discovery

Discovery asks the question, “What are the behaviors to test?” Discovery is an activity of learning. Software systems are often very complex – sometimes more complex than any one person can individually comprehend! Teams need time to figure out all the behaviors in the system and determine how they should work.

While behaviors are still being designed, Discovery can be done with the Three Amigos (business, development, and testing roles) getting together to do activities like Story Mapping and Example Mapping. If behaviors are already implemented, and tests are being automated to increase regression coverage, Discovery could be time to explore the system, review documentation, and ask questions.

Formulation

Formulation asks the question, “How should the behaviors work?” Formulation is an activity of definition. It puts everything learned during Discovery into clearly written words. In simplest terms, this is when test cases are written. It is important to be clear and thorough when writing test cases because they become living documentation for the system under test. They also become the receipts for what was developed and tested.

One very common format for formulating tests is Gherkin, a simple language that denotes tests with Given-When-Then phrases. Teams may also test case management systems to store their test specifications. No matter how tests are recorded, they should be easy to find and intuitive to understand.

Automation

Automation answers the question, “What happens when we try the behaviors?” Automation is an activity of verification. A team should be able to know the state of the system’s quality at any moment. Automation enables continuous testing of the system by checking it against the expectations defined in the test specifications.

Again, automation is more than just a bunch of scripts. It includes the test execution platform with all the required machines and environments. It includes tooling for editing and debugging tests. It also includes reporting capabilities for test results.

Why do this process?

Well, if you are automating your tests, then you are already following the Discovery → Formulation → Automation process, even if you don’t call it by that name. The three phases describe work that is necessarily part of test automation.

The magic in the maxim is its intentionality. Many times when folks start automating tests, they jump right into code (the “Automation” phase) without taking the time to truly understand the system (the “Discovery” phase) or carefully set expectations (the “Formulation” phase). Poor coverage, invalid tests, and wasted time backtracking are the results. Being intentional about the process ensures that the right things are tested in the right ways. It also builds the body of living documentation for the system as an artifact of the process, so that folks in the future can more easily learn the system for further development and testing. That’s what makes the process “good.”

When should the process start?

Ideally, Discovery → Formulation → Automation should be a development-centric process that includes testing and automation. This would be an “Agile” or “shift left” approach, where testing activities start early in the development timeline. Discovery would be an activity for the full team, and Formulation would provide specifications for both the developer on what to build and for the tester on what to verify and automate. This is usually the intention as presented with BDD in mind.

However, not everyone works in that ideal world. Sometimes, testing needs to happen after the system is already built (or mostly built) for various reasons. It could be that an enterprise company is following the Waterfall Model or has legacy systems. It could be that a test team is hired to automate an existing manual regression suite. It could be that a team is testing third-party or outsourced software. In that case, test automation becomes its own dedicated development effort, but the Discovery, Formulation, and Automation phases still apply. And that’s okay – automated tests still deliver value.

How do we get started?

Discovery → Formulation → Automation is a very helpful process for test automation. As stated before, its value comes from being intentional about each phase. 

This process has been transformative at Cycle Labs as well. In fact, we are actively building features into the Cycle Testing Platform to improve tester experience in the areas of Discovery, Formulation, and Automation. We want to empower testers to automate not just good but excellent test suites from inspiration to implementation in the Cycle platform. In fact, when we talk about the “Cycle Testing Process,” we really just mean Discovery → Formulation → Automation.

Hopefully, like us, you can find value in this process as well!

This post was written by:
Andrew Knight
Principal Architect