Why API Testing May Not Be Enough for Enterprise Execution Systems
You Can’t Improve Solution Quality By Adding Horsepower
API testing tools like Loadrunner and jMeter are widely used for execution system stress tests, but only targeting the back-end may miss performance bottlenecks in complex applications that can be caught with day-in-the-life (DITL) performance testing.
In order to uncover and proactively remediate production risks, we must simulate end-to-end user experiences and behaviors at scale. For transaction-based enterprise systems with non-linear processes, solution quality and resiliency cannot be fully verified simply by flooding specific API endpoints with transaction volumes that don’t recreate production workflows and timings.
In today’s mobile world, many execution systems optimize for inconsistent connectivity by minimizing back-end communication, particularly in non-enterprise environments with simpler business requirements and tasks that depend primarily on forms and straight-forward media usage. When processes are relatively linear with only a handful of API endpoints, it can make a lot of sense to focus primarily on API testing even though those tools do not re-create the entire user experience. However, enterprise execution systems introduce complex workflow interactions, multi-step synchronous transactions, and dependencies that may justify more thorough DITL test automation.
In order to uncover logical bottlenecks, we must simulate end-to-end user experiences and behaviors at scale. For complex enterprise systems with non-linear processes, solution quality cannot be verified simply by flooding specific API endpoints with transaction volumes that don’t re-create production data flows and timings.
The Disconnect Between Technical Support and End Users
If you have ever supported production environments for an enterprise execution system like warehouse management (WMS), you are probably familiar with end user tickets like “[insert essential system action] not working,” “unable to complete [key object identifier],” and (the most feared) “system slow.” Most support tickets start with a frustration about not being able to achieve a desired result – picking, shipping, closing a purchase order, etc. Users are generally unaware of the highly dynamic and “chatty” nature of their business applications, which often generate user workflows through a combination of:
- Dynamic UI elements
- Back-end validation calls firing during navigation
- Table-stored configuration
- Caching
- Larger data payloads
For example, a support analyst working a WMS support ticket for “unable to load shipment 12345” might work backwards to a root cause through queries (and technical assumptions) like the following:
- Is the user utilizing a web, native, or mobile interface?
- Is there a clear error message or is the workflow just hanging up?
- If workflow hangs up, is the solution configuration complete?
- Is the user following the SOP?
- If yes, might indicate unexpected/unhandled input values
- If no, might indicate solution defect
- Is the issue occurring after a button press?
- Might indicate a long-running synchronous processing API
- Is the issue occurring after a field entry/exit?
- Might indicate issue with dynamic validation/fetch API or UI behavior
- Is the issue happening on a screen transition?
- Might indicate caching, validation, or transaction commit steps
- Are there any locks or unexpected field statuses on relevant SQL database tables?
- Might indicate conflicting user interactions (perhaps from a shipping manager making customer-requested changes)
Many support and performance issues in enterprise systems result from logical contradictions and timing collisions that occur when hundreds of users across multiple functions are all working independently to accomplish a goal like shipping a load of goods to a customer accurately and efficiently. Data is constantly updated (and locked) by human users, host system integration, automation systems, and background tasks yet it must be displayed and acted upon moment by moment. How can we expect to uncover potential bottlenecks only by flooding API endpoints with transactions that don’t recreate the actual data flows and timings?
In an ideal environment, stress testing and DITL load testing would each be employed based on their respective strengths, not as an either/or decision.
You Can’t Improve Solution Quality By Adding Horsepower
In an ideal environment, stress testing and DITL load testing would each be employed based on their respective strengths, not as an either/or decision. Tools like Loadrunner excel at proving that infrastructure and application architectures can manage peak volumes, but DITL testing through Cycle proves that the business processes and solutions are able to stand up to real-world usage and exposes potential process bottlenecks that can be targeted with back-end stress testing.
Depending on your production volumes and risk profile, you may decide to implement more of one type or the other. However, in our experience, it has become much less common since the dawn of cloud computing to find sizing issues after initial infrastructure implementation and server/database maintenance is configured. (Pro tip: Don’t try to get tricky with sharing virtual resources. Dedicate resources to your mission critical systems and allocate sufficient cores to your database.) The most painful and costly production issues often come from obscure root causes like database locks, inefficient SQL, and incorrect business logic, which many of us have experienced first-hand in former careers. API stress testing certainly plays a role in reducing deployment risk, but continuous DITL testing protects you from the issues that can’t be solved by adding more horsepower.
Learn more about Cycle or hear from one of our clients about their experience with Cycle.
Test automation is the backbone of any successful enterprise testing strategy – especially as enterprise systems continue to rapidly evolve. However, implementing test automation requires commitment and investment. How do you justify it?
This calculator estimates how much money you spend on manual testing currently as a starting point for building a business case for switching to automated regression testing using the Cycle® platform.
