Moving Your Automated Testing Infrastructure to the Cloud

Let’s get specific on leveraging the cloud for your automated testing infrastructure. One of our responsibilities is to help bridge gaps between technical and non-technical folks. Over the last few months, we’ve been educating ourselves on test infrastructure options and trying to bridge our own gap when it comes to fully comprehending the flexibility and power of leveraging cloud computing for test automation. It wasn’t that long ago we were all giddy about the convenience of virtualized environments. As cloud platforms like Azure and Amazon Web Services (AWS) become easier to use and add feature sets, test automation engineers and managers – and really anyone in the realm of software testing – must now evaluate the advantages of running their tests on cloud infrastructure provisioned on-demand and managed over the Internet versus using in-house infrastructure/resources.

If you have very small-scale, relatively modest testing needs and are lucky enough to not need to scale up AND have hassle-free dedicated physical resources for testing, then using cloud resources may not be worth it for your situation. However, nearly everyone can benefit because we all want to scale up testing and who has hassle-free test resources? Testing teams with medium-to-large scale testing needs and/or frequently-run tests involving concurrent users (as often demanded when performance testing) in the system under test should especially consider moving to cloud infrastructure.

Regardless of where you fall on the “testing needs” spectrum, testing on cloud resources is undoubtedly worth it if you find that your team is continually wasting time and money setting up a varied assortment of unreliable “test PCs” and often installing and updating applications required by the system under test.

You want to hit the ground running clean when your test cases are ready to be executed, and not delay the valuable feedback gained from testing and/or experience time-wasting issues introduced by unreliable infrastructure.

You: We need some PCs to test out the latest version.
IT Manager: You can’t use the new “good ones” as they are for production…Here, take these from the IT closet:

This is obviously an exaggeration, but in the last five years you probably have either tested on a dusty old PC with a floppy drive or CRT monitor, or even worse, you’ve had to waste time updating a drastically outdated operating system for testing purposes.

Pros and Cons of Switching Test Infrastructure to the Cloud

Testing with Cloud Resources	Testing with In-house Resources
Dependably clean and repeatable environment	There is an initial learning phase, at least without something to help jumpstart your testing team
Cost can be precisely managed	May not be as beneficial for very small scale testing
On-demand start and stop of scalable resources
Personnel no longer busy configuring QA computers
Faster time to value, especially on test feedback
Dependable disaster recovery that is accessible off-site

Leveraging the Cloud + Infrastructure as Code

When we started talking about cloud automation, we started to envision building a cost-effective testing infrastructure on the Azure platform to enable our customers to move their Cycle licenses to cloud-based resources, or to enable new customers to start their warehouse management systems (WMS) testing in the cloud with Cycle. We built an infrastructure as code (IaC) template that would allow our delivery engineers – or the customer – to easily deploy the cloud resources to Azure to build this testing environment in an automated fashion. We think our automation template probably saves our customers with moderate knowledge of Azure about 80 hours worth of work and manual resource deployment.

One of the main reasons we started building this environment using Azure as our cloud provider, and Jenkins as our testing platform, is because of the tool sets that are available with the two. Jenkins and Microsoft collaboratively built a Jenkins plugin called Azure VM Agent plugin, which allows Jenkins to build and destroy temporary virtual machines on Azure, to run builds (aka Cycle tests) on. These build-on-demand agents help save our customers a lot of money by only building the agents when a build/test is executed, and then deleting them shortly after the build/test is finished. This way you don’t have idle agents, costing you money every hour in your cloud platform.

Another benefit of this, besides cost, is that you’re following a true automated testing best practice – which is to always test on a fresh agent. Since the agents are deployed and configured whenever a build/test is executed, you can be sure that your agents will be fresh and up to your specifications/configurations. Here is our best attempt to put simple math around this concept of cost savings:

• A standard D2s v3 (2 vCPU / 8GB of RAM) costs $0.19/hr or $137.24 a month if it’s online 24x7x30.
• Using the Azure VM agent, we spin up nodes only when we need them. For example, if you are running a Cycle test 10 times a day, and that test takes 14 minutes per run – it would cost you $13.30 (+ disk/networking cost, minimal) to run that Cycle test in Azure (per month) on a D2s v3 compute instance.
  
  

Reducing Your Time-to-Test with the Cycle Appliance

Not only does our automation deploy the cloud resources, but we also install and configure Jenkins, install about 20 different plugins, as well as automatically configure the Jenkins manager to be connected to your Azure environment securely using Azure Managed Identities – so that it can create and destroy those temporary agents on its own. We tried to take the manual leg work out of the process and make it easier for our customers to deploy the appliance so they can get straight to setting up their Cycle tests.

Pipeline Driven Testing

When using Jenkins to run Cycle tests, we are taking advantage of pipeline driven-testing. This makes the test repeatable, and versionable in source control. The Jenkins pipeline is driven by a Jenkinsfile to tell the pipeline what kind of agent to build in Azure, how many of them, how they should be configured, what software should be installed on them, and then last but not least – which Cycle commands to run on them. We provide example Jenkinsfile code to get you going in the right direction and to make it easier to approach a more DevOp-sy mindset of using Cycle for testing, by leveraging Cycle + Jenkins + Source Code Management + the Cloud all in one-fell-swoop.

Understanding the Cycle Appliance

The Cycle Appliance is deployed using an Azure resource manager (ARM) template that will build everything into your existing Azure tenant. The Azure tenant is the account with Microsoft that represents a single organization. You can have multiple subscriptions underneath the tenant, all set up to bill on their own, but they live under the tenant. The Cycle Appliance will deploy to, and live inside of its own resource group, inside your Azure tenant, and Azure subscription. So, an existing Azure tenant is a requirement for this – although if you don’t have an existing Azure tenant, we can help you provision and set this up so that you can use the appliance.

“A dedicated and trusted instance of Azure AD that’s automatically created when your organization signs up for a Microsoft cloud service subscription, such as Microsoft Azure, Microsoft Intune, or Microsoft 365. An Azure tenant represents a single organization.” -Microsoft

If you are interested in creating your own Azure account, you can start here and get $200 in credits! Once you create your Azure account, a tenant will automatically be created for you. You will then be able to set up a subscription, and then deploy the Cycle Appliance into that subscription. This brief training page helps you better understand the hierarchy of Azure tenant versus subscription, how billing works, and more. 

Getting Started with the Cycle Appliance

These are the high level steps of getting the Cycle Appliance deployed and functional in your Azure environment.

Let’s take a look at what needs to be done at at high level (our engineers would do this for you if you use the Cycle Appliance):

1. Create your own Azure account and tenant (the tenant automatically gets created when you make the account, mycompany.onmicrosoft.com, etc.)
2. Set up your subscription within the Azure tenant (pay-as-you-go)
3. Deploy the Cycle Appliance ARM template into that subscription, while logged in as the global admin
4. Connect Azure to your on-premise network using an Azure VPN Gateway

For those of you that use in-house testing infrastructure, we’ve armed you with enough information to at least have a conversation with your testing team. Get everyone together, run the numbers using one of the many Cloud cost calculators, ask the IT manager how much of a hassle it is to wrangle existing test infrastructure and mull it over. In highlighting our own Cycle Appliance, we also hopefully shed some light on IaC and our implementation of it that helps automate the building and deployment of test environments in Azure. 

Are you interested in learning more about implementing test automation in your warehouse management system? Check out our success stories, more of our blogs, or learn more about the Cycle platform.

This post was written by:
James Prior
Technical Pre-Sales Consultant

Ryan Berger
Cloud Automation Engineer II