Confidence in a Comprehensive Strategy: Why Test Automation Accelerates Transformation

Executive Summary

In the era of on-demand everything, consumers expect instant gratification and curated user experiences. Industries of every kind, as well as the global supply chain, are undergoing a massive transformation as they try to keep up.

Consistent, comprehensive testing of business-critical systems is a key part of this transformation. Automation can make testing more effective, efficient, and affordable than ever before.

For testing to be truly effective, it should begin with developing a strategy and integrating tests into a project plan. The strategy should provide a clear framework for deciding what to test and how to translate the results to business leaders.

By putting testing front and center, leaders can move forward with confidence, knowing their organizations’ processes, platforms, and people are the best they can be.

Download The White Paper

Cycle with Confidence

Change is inevitable. Bottlenecks, surprises, and stressful go-lives are not.

The speed of innovation today requires rapid feedback, faster validation, and a reduced time to value. Software testing, and test automation in particular, are indispensable for anyone who wants to focus on business growth rather than worry about how new initiatives might introduce risk.

There are a number of reasons an organization should leverage automation. Recently, a few significant common threads across industries underscore the importance of an automated quality solution.

The Power of Wanting It Now: Consumer Expectations

In the age of Uber and Amazon, instant gratification is industry standard. Lofty consumer expectations for things like on-demand services and same-day delivery put pressure on a supply chain that’s already struggling to keep pace.

As a result, the development of supply chain applications is increasing rapidly. Customers don’t just seek instant gratification. They want a curated experience. Companies of all kinds must be more responsive, flexible, and proactive along the entire supply chain to deliver that experience, because today’s consumer cares about it more than ever. To compete with giants like Amazon, businesses simply can’t afford any instances of customer dissatisfaction.

Fast Forward: The Pace of Change

The on-demand economy means enterprise applications are constantly in a state of rapid change. Deployment rates are rising in step with the adoption of agile philosophies. To meet consumer expectations, the systems that support businesses must evolve more swiftly than ever before.

But the implementation and deployment practices that support packaged software solutions haven’t matured quickly enough to match the increased speed of deployment cycles.

Many of Cycle Labs’ customers upgrade their enterprise solutions every two to three years. Some upgrade every five to seven years. Meanwhile, the OEMs of these solutions produce incremental value through hotfixes and upgrades as often as every two weeks. The result is a bottleneck in delivering value to customers.

With an industry-wide focus on moving to the cloud and true SaaS readiness, the agile methodologies that power these initiatives are critical. DevOps, continuous integration, continuous delivery, and continuous testing ensure packaged software deployment teams deliver high-value, on-time solutions.

Pandemonium: COVID-19 and the Supply Chain

The COVID-19 pandemic exposed numerous challenges facing the global supply chain. Across a variety of industries, organizations are re-evaluating core practices like “lean,” just-in-time inventory strategies and other cost-cutting measures. Many organizations don’t have processes in place that enable them to react quickly to a supply chain upheaval.

But there isn’t a one-size-fits-all solution now, and there likely won’t be one in the immediate years to come. Companies can, however, reintroduce a bit of slack into their supply chains and eliminate some rigidity while maintaining relatively lean operations.

Test automation is at the foundation of agile initiatives like these. Organizations built on agile practices, which prioritize flexibility and the ability to accelerate change, will thrive in today’s marketplace and lead the way toward whatever is next.

As they look to the future, business leaders need to make the right choices today to prepare for the next pandemic, the next Suez Canal-style crisis, or simply the next time a quick adjustment of processes is necessary.

Meet the Experts

This white paper is based on an on-demand webinar produced by Delaplex and Cycle Labs.

Josh Owen, Co-founder & CEO, Cycle Labs

Josh is a co-founder and the CEO of Cycle Labs. Before Cycle Labs, Josh founded and led a global supply chain consulting firm through 12 years of organic growth, where he also led the development of the Cycle product from its idea phase to the enterprise continuous testing platform it is today. With over 16 years of experience deploying enterprise solutions across industry verticals, Josh is dedicated to modernizing enterprise solution deployment and helping companies focus on business growth rather than business risk.

Bruce How, Regional VP, Delaplex

Bruce is an established leader and mentor in developing and delivering enterprise-level digital supply chain programs across a broad range of industries. Based in London, he leads the efforts of Delaplex, a global custom software development and business solutions provider, to grow in the UK and European markets. Before Delaplex, he served in several roles at Blue Yonder and NTT DATA Business Solutions APAC.

What Is Automated Testing?

According to software engineer, tech blogger, and test automation expert Andy Knight, test automation is “leveraging modern software solutions to automate the validating of goodness and the identifying of badness in software solutions, product code, and features for the purposes of enforcing high-quality standards.”

Supply chain organization leaders might wonder why software testing is in their purview, thinking: We’re not a software company. Isn’t testing the responsibility of those who developed the product?

That approach might have worked decades ago. But today, almost every company is a software company, and it’s on each one of them to ensure the strength of their link in the supply chain.

What does testing solve?

“Testing activities, done traditionally, will become the constraining factor … the alternative is that testing will not be done adequately, therefore putting your organization’s reputation at risk.”

World Quality Report 2020

Rather than developing code for specific features or functions the way actual software companies do, most businesses’ products and services are built with, or rely on, packaged enterprise software solutions integrated together to achieve critical business needs. These enterprise solutions themselves rely on a network of existing software applications. Every part of this network needs testing.

Those in the supply chain space experience many of the same challenges as companies that build software from the ground up. Issues like miscommunication, software bugs, and misaligned expectations can spell disaster for SaaS startups and warehouses alike. The difference lies in how these problems are solved.

The Three Categories of Software Testing

1. Functional: Does this solution work correctly?
2. Performance: Does the solution work within the desired system metrics?
3. Experimental: Does the solution work with improved features and metrics after some system change?

Testing in Action

To determine whether a solution works correctly, a functional test should have a clear pass-or-fail outcome. A “passed” test validates the achievement of a goal or a set of mutually agreed-upon results, which could include business requirements, software, or solution specifications. Ultimately, and arguably most importantly, the test should validate that the solution meets user expectations and provides a good user experience.

The Testing Pyramid for Functional Tests

The testing pyramid is an industry-standard guideline for functional test case strategy, intended to illustrate the ideal focus, testing type, and size of a comprehensive testing strategy. Each layer provides a guideline for developing these strategies. Here, the pyramid focuses on the packaged solutions used by most of Cycle Labs’ customers, with the goal of risk mitigation across core business functions.

1. Bottom layer, unit tests: These are generally smaller in size and less costly to develop, and they tend to focus on incremental validations. Enterprise organizations should use them with care and focus instead on end-to-end tests (a top-layer system test, described below) that provide higher value. The solutions most organizations choose work as designed. However, each enterprise must validate whether those solutions satisfy its critical business needs.
2. Middle layer, integration tests: These focus on the intersection of two or more different systems and measure how well the systems work together. For example, an integration test might validate the efficacy of a pair of applications, such as a parcel system and a transportation management system running in tandem.
3. Top layer, system tests (end-to-end tests): These are run the way an end user experiences the solution being evaluated. Supply chain organizations should spend the majority of their time and effort on this layer, which tends to provide the highest level of risk mitigation.

Today, most businesses rely on a complex network of interfacing packaged software solutions. It’s vital to ensure that changes to individual solutions don’t set off a chain reaction that interferes with overall operations.

When should such tests be automated? When they provide a positive return on investment.

Performance Tests

Once an organization has a functional testing strategy in place, and has automated critical business processes to ensure its needs are consistently met, the company can turn its focus to performance testing.

Performance testing determines whether a given solution performs well and within an organization’s desired system metrics. “Good performance” translates to minimal impact on primary software performance metrics like processor usage, memory usage, response time, and throughput.

In the supply chain world, performance is always a primary concern. It’s important to consider performance when tackling big challenges, like managing peak-season volumes or concurrency issues caused by shifting work between two functional areas. But when business-critical operations rely on a complex network of interrelated systems, seemingly small changes can also have an enormous impact.

Supply chain leaders ask questions like these every day:

1. What would happen if our fulfillment center had 60 people picking instead of 20? Can our system handle that change?
2. Will this warehouse management system function in an ideal manner if we serialize our inventorying?
3. What happens if we need to double up staff today on replenishments because we’re expecting higher picking volumes tomorrow?

A good performance testing strategy allows organizations to simulate what-if scenarios and move forward with confidence.

Control + Shift Strategy

When functional and performance testing strategies are ingrained in an organization’s processes, leaders can shift how they think about testing.

Traditionally, companies viewed testing as a defensive strategy, intended primarily to protect against system changes that result in regressions. In other words, testing prevents bugs from slipping into production. But with a solid functional and performance testing strategy, testing can be an offensive measure as well.

Automation can make the days, weeks, and hundreds of hours spent fixing issues in production a thing of the past. It gives teams the freedom to innovate and the ability to focus on business growth rather than business risk.

Test, Meet Talent: The Power of People in a Testing Framework

While automated testing can be a key driver of change, it can’t truly transform a business without human ingenuity.

Testing starts with people. Before a project starts, the people who run a business need to clarify their goals. They engage with fellow executives and internal development teams, and they might talk to vendors and consultants as well. The questions asked are usually something like:

1. What will the business get out of this?
2. What value can it add for our customers?
3. Why create this program or take this initiative now?

Typically, companies focus on creating features and functionality. But the big-picture visions and smaller, project-oriented goals often don’t address what they need to do from a testing perspective. Sometimes organizations pass the buck on testing back to their partners. OEMs in particular perform extensive, high-quality unit tests on their products, and warranties and support or maintenance agreements are in place if something goes awry.

This line of thinking is all too common. From a project planning perspective, it means testing silently moves into the project phase itself, yet the timelines and effort necessary for testing are not part of the equation.

People First: A Typical Testing Team

Vendors and IT specialists may do their own tests or weigh in on the process. But testing a new software update before going live, for example, generally falls to business users. Of course, those users still have day-to-day responsibilities and commitments.

If the organization works with agile methodologies, testing will be part of sprints, proving the software works. However, that testing may not be consistent or part of an overall strategy.

Business experts tend to focus on the path of least resistance. But large-scale transformation programs are almost always about solving a number of complex problems. There’s rarely a yes or no answer. The solution depends on expiration dates, the time when stock becomes available, or whether that stock is allocated to other customers. The path forward is far from straightforward, because there are so many scenarios to test.

In this kind of environment, even the most dedicated teams get frustrated and exhausted. If tasks are assigned piecemeal with no upfront understanding of the scope of the project, morale can plummet. But it doesn’t have to be this way.

Roles shown in the paper’s “typical testing team” graphic: Project Manager, Project Subject Matter Expert, Business User, Business Expert, Project Associate, Business IT Representative.

How to Support Teams Through Testing

The most transformative testing strategies are transparent and supportive to employees at every level. Often, that means augmenting (not replacing) certain teams.

A typical fulfillment solution, for example, will involve a project manager and a solution architect at the outset. To do their jobs well, they should know what satisfactory results look like. But many things can lead to variations in test use cases and test results. That’s why test specialists can be invaluable in supporting subject matter experts. People with experience writing test use cases know there is never just one path. The reality is usually that a number of nested paths can lead to correct outcomes. A test-use-case specialist will identify these paths, map them back to a traceability matrix of core business requirements, and communicate these possibilities to business users.

By augmenting the efforts of an organization’s core teams with specialists in test use cases, business experts get the support they need while the project stays on task and on time. It’s a win-win for everyone.

The Test Automation Puzzle

The test automation technology landscape can be thought of as a puzzle with six major components. Test automation is accomplished by combining a suite of modern technologies, integrated together, to program and orchestrate software testing at scale, without the need for human intervention.

Countless tools are available to achieve a variety of automation goals. For the supply chain, these are the technologies best suited to increase development velocity, maximize productivity, and achieve testing sustainability. Supply chain organizations, like all enterprise businesses, need tools that can span teams and technologies.

6 key puzzle pieces

1. Automation framework: The first decisions in any testing scenario involve the automation framework. What tools will be used to articulate the automated test cases? Especially in the enterprise, a framework that works across technologies is essential. Ideally, the same framework will cover everything from legacy systems to leading-edge systems and can evolve alongside new ones. Testing issues are often rooted in early miscommunications, so a focus on executable specifications earlier in the pipeline will better align testing with business expectations. Ideally, organizations should create a framework that provides inherent traceability to the underlying business requirement under evaluation. When tests provide context for business operations, everyone wins. The best frameworks fit into (and streamline) existing workflows and provide insight with no extra work required.
2. Revision control: Revision control tracks iterations of the target application or system being tested. Configuration management, source rollouts, and hotfixes all fall under this umbrella. Revision control allows branching that supports multiple variations on a core solution. These branches might represent several deployment permutations, such as different facility types. In the supply chain ecosystem, that might mean manufacturing and fulfillment facilities that each require different core configurations. A revision control solution can manage everything.
3. Automation server: The automation server is the primary engine of a test automation pipeline, serving as a console, a mission control center, and the central point of integration. A strong automation server integrates with task management solutions and can do things like send alerts via email or chat. It can pull data from revision control and orchestrate tests by itself, so people can view and interact with jobs, results, and reports all in one place.
4. Containers: Containerization is a form of operating system virtualization that runs applications in an isolated space called a container (under the same operating system). A container is essentially a fully packaged, portable computing environment. It dramatically simplifies testing infrastructure by minimizing OS licensing, componentizing an application stack, and spinning up disposable test agents in instances that match the system being tested. Containers allow repeatable deployment and testing, managed by a cluster management solution.
5. Cluster management: Cluster management solutions manage the startup, execution, and teardown of containers, creating scalable clusters and nearly endless possibilities. This puzzle piece makes it possible to simulate a full day’s warehousing staff at peak volume (or more) with minimal dedicated infrastructure, including machines, licensing, and even maintenance.
6. Compute: All of these solutions need to “live” somewhere, because tests need power to run. Compute options may be physical or virtual, and they may be in a private or public cloud. A number of companies offer solutions like hypervisors and infrastructure-as-a-service platforms such as Google Cloud, Microsoft Azure, and Amazon Web Services (AWS). The right solution for any given organization depends on its needs as well as its human expertise, budget, and governing policies. Compute resources should empower automated testing to run regularly, if not daily.

The Role of Process in an Automation Journey

Testing isn’t a singular event. It’s a process that includes strategy, planning, and ongoing execution through automation. The first element, strategy, should begin with a fundamental question: What exactly is being tested?

Sometimes, in early meetings about testing strategy, people will say things like: “We’re going to test everything. This solution is going to be 100%, with zero defects, no bugs.” Soon enough, they’ll start categorizing bugs. But then, when they learn of the potential to automate testing, they’ll say: “We’re going to automate everything.”

Risk-based testing is a strategy that helps focus an organization’s effort on what’s most important.

“While it’s natural to want to minimize defects before going live, speed is perhaps even more crucial to a business. Agile delivery at the pace of consumer demand can’t happen if teams attempt to make everything perfect.”

Josh Owen, Co-founder & CEO, Cycle Labs

Unless an organization has an unlimited budget, endless resources, and infinite time, these goals aren’t feasible.

To explain risk-based testing in action, imagine a warehouse that is overwhelmed with orders. The warehouse may have thousands of things to test, but it must prioritize testing the most mission-critical operations. For instance, if management wants to extend employees’ weekend shifts from seven to 10 hours to keep up with demand, it’s prudent to test this decision first. There are a number of outcomes for the business, and it’s not as simple as just paying everyone for an extra three hours.

In the paper’s example, the warehouse team is made up of people with various employment contracts and pay grades. A veteran of the company with 20 years of seniority has a union contract that stipulates a certain rate for overtime and an additional 15-minute break. A salaried employee is paid another rate for overtime. A contract worker gets no overtime, breaks, or special rates.

Failure to account for these factors could produce multiple correct test results, but that’s not useful to the company. Simple pass-or-fail criteria won’t work in any of these scenarios. What does? A strategy that scores each factor according to its relative risk. Scoring can be on a scale of one to 10, one to five, or any other scheme. The idea is to visualize the complexity of outcomes.

The use case here is quite simple (just four possible outcomes). More complex systems may involve deeply embedded heuristics with dozens of different outcomes. No matter the complexity, scoring factors such as overtime can prevent regression testing of every release, upgrade, or hotfix (and untold headaches for the teams behind them).

Scoring each factor involved in the hypothetical shift extension highlights the ones likely to affect business-critical operations. This is one of the best ways organizations can prioritize tests and make the implications accessible to everyone involved.

At its core, this testing strategy creates artifacts that provide long-term, ongoing value to a business. It’s a prescriptive, “shift-left” approach that articulates business requirements at the outset and bakes testing into the overall project plan and timeline. These artifacts provide value in the context of testing and also offer inherent advantages for communication with teams and alignment with big-picture goals.

When everyone understands what solutions are designed to achieve, the journey toward attaining them will likely be much smoother. The ROI is clear, and so are the expectations.

Conclusion

Automated testing isn’t just a new set of bells and whistles. It empowers organizations to leverage other technologies of all kinds without the fear that change may yield unpredictable or negative results. The ability to adapt quickly is a requirement for every business today. Resilience is no longer a luxury companies can afford to skip when disruption is commonplace. The future belongs to those who innovate every step of the way.

Ready to deploy with confidence?

Cycle® is the solution for simple, streamlined testing of all your systems, all in one place. Watch a demo of Cycle®: https://cyclelabs.io/demo/

Contact: cyclelabs.io | +1 (866) 305-3151 | info@cyclelabs.io