Let’s get specific on leveraging the cloud for your automated testing infrastructure. I’m James Prior, and one of my responsibilities at Tryon Solutions is to help bridge gaps between technical and non-technical folks. Over the last few months, I’ve been educating myself on test infrastructure options, and trying to bridge my own gap when it comes to fully comprehending the flexibility and power of leveraging cloud computing for test automation. It wasn’t that long ago we were all giddy about the convenience of virtualized environments. As cloud platforms like Azure and Amazon Web Services (AWS) become easier to use and add feature sets, test automation engineers and managers – and really anyone in the realm of software testing – must now evaluate the advantages of running their tests on cloud infrastructure provisioned on-demand and managed over the Internet versus using in-house infrastructure/resources.
If you have very small-scale, relatively meager testing needs and are lucky enough to not need to scale up AND have hassle-free dedicated physical resources for testing, then using cloud resources may not be worth it for your situation. That’s another way of saying that nearly everyone can benefit, because we all want to scale up testing, and really, who has hassle free test resources? Testing teams with medium-to-huuuge scale testing needs and/or frequently-run tests involving concurrent users (as often demanded when performance testing) in the system under test should especially consider moving to Cloud infrastructure.
Regardless of where you fall on the “testing needs” spectrum, testing on cloud resources is undoubtedly worth it if you find that your team is continually wasting time and money setting up a varied, ramshackle assortment of unreliable “test PCs” and often installing and updating applications required by the system under test.
You want to hit the ground running clean when your test cases are ready to be executed, and not delay the valuable feedback gained from testing and/or experience time-wasting issues introduced by unreliable infrastructure.
You: We need some PCs to test out the latest version.
IT Manager: You can’t use the new “good ones” as they are for production…Here, take these from the IT closet:
This is obviously an exaggeration, but admit it, in the last five years you have either tested on a dusty old PC with a floppy drive or CRT monitor, or even worse, you’ve had to waste time updating a drastically outdated operating system for testing purposes.
Pros and Cons of Switching Test Infrastructure to the Cloud
Testing w/ Cloud Resources
Testing w/ In-house Resources
|Dependably clean and repeatable environment||There is an initial learning phase, at least without something to help jumpstart your testing team|
|Cost can be precisely managed||May not be as beneficial for very small scale testing|
|On-demand start and stop of scalable resources|
|Personnel no longer busy configuring QA computers|
|Faster time to value, especially on test feedback|
|Dependable disaster recovery that is accessible off-site|
Leveraging the Cloud + Infrastructure as Code
My name is Ryan Berger, and I’ve been with Tryon Solutions for about eight months now. My specialty is in cloud automation, and when I started talking about Cycle with the team – I started to envision building a cost-effective testing infrastructure on the Azure platform, to enable our customers to move their Cycle licenses to the cloud based resources, or to enable new customers to start their WMS testing in the cloud with Cycle. We built an IaC (Infrastructure-as-Code) template that would allow our delivery engineers – or the customer – to easily deploy the cloud resources to Azure to build this testing environment in an automated fashion. We think our automation template probably saves our customers with moderate knowledge of Azure about 80 hours worth of work and manual resource deployment.
One of the main reasons we started building this environment using Azure as our cloud provider, and Jenkins as our testing platform, is because of the tool sets that are available with the two. Jenkins and Microsoft collaboratively built a Jenkins plugin called Azure VM Agent plugin, which allows Jenkins to build and destroy temporary virtual machines on Azure, to run builds (aka Cycle tests) on. These build-on-demand agents help save our customers a lot of money by only building the agents when a build/test is executed, and then deleting them shortly after the build/test is finished. This way, you don’t have idle agents costing you money every hour in your cloud platform!
Another benefit of this besides cost, is that you’re always following a true automated testing best practice – which is to always test on a fresh agent. Since the agents are deployed and configured whenever a build/test is executed, you can be sure that your agents will be fresh and up to your specifications/configurations. Here is my best attempt to put simple math around this concept of cost savings:
- A standard D2s v3 (2 vCPU / 8GB of RAM) costs $0.19/hr or $137.24 a month if it’s online 24x7x30.
- Using the Azure VM agent, we spin up nodes only when we need them. For example, if you are running a Cycle test 10 times a day, and that test takes 14 minutes per run – it would cost you $13.30 (+ disk/networking cost, minimal) to run that Cycle test in Azure (per month) on a D2s v3 compute instance.
Reducing your time-to-test with the Cycle Appliance
Not only does out automation deploy the cloud resources, but we also install and configure Jenkins, install about 20 different plugins, as well as automatically configure the Jenkins manager to be connected to your Azure environment securely using Azure Managed Identities – so that it can create and destroy those temporary agents on its own. We tried to take the manual leg work out of the process, and make it easier for our customers to deploy the appliance and get straight to setting up their Cycle tests.
Pipeline Driven Testing
When using Jenkins to run Cycle tests, we are taking advantage of pipeline driven-testing. This makes the test repeatable, and versionable in source control. The Jenkins pipeline is driven by a Jenkinsfile to tell the pipeline what kind of agent to build in Azure, how many of them, how they should be configured, what software should be installed on them, and then last but not least – what Cycle commands to run on them. We provide example Jenkinsfile code to get you going in the right direction and to make it easier to approach a more DevOp-sy mindset of using Cycle for testing, by leveraging Cycle + Jenkins + Source Code Management + the Cloud all in one-fell-swoop.
Understanding the Cycle Appliance
The Cycle Appliance is deployed using an ARM template (Azure Resource Manager) that will build everything into your existing Azure tenant. The Azure tenant is the account with Microsoft that represents a single organization. You can have multiple subscriptions underneath the tenant, all set up to bill on their own, but they live under the tenant. The Cycle Appliance will deploy to, and live inside of its own resource group, inside your Azure tenant, and Azure subscription. So, an existing Azure tenant is a requirement for this – although if you don’t have an existing Azure tenant, we can help you provision and set this up so that you can use the appliance.
“A dedicated and trusted instance of Azure AD that’s automatically created when your organization signs up for a Microsoft cloud service subscription, such as Microsoft Azure, Microsoft Intune, or Microsoft 365. An Azure tenant represents a single organization.” -Microsoft
If you are interested in creating your own Azure account, you can start here and get $200 in credits! Once you create your Azure account, a tenant will automatically be created for you. You will then be able to set up a subscription, and then deploy the Cycle Appliance into that subscription. This brief training page helps you better understand the hierarchy of Azure tenant vs subscription, how billing works, and more. If these are new concepts for you, it’s worth the time to have a look!
Getting started with the Cycle Appliance
These are the high level steps of getting the Cycle Appliance deployed and functional in your Azure environment. Let’s take a look what needs to be done at at high level, and of course if you decided to use our Appliance our engineers would do this for you:
- Create your own Azure account and tenant (the tenant automatically gets created when you make the account, mycompany.onmicrosoft.com, etc)
- Set up your subscription within the Azure tenant (pay-as-you-go)
- Deploy the Cycle Appliance ARM template into that subscription, while logged in as the global admin
- Connect Azure to your on-premise network using an Azure VPN Gateway
For those of you that use in-house testing infrastructure, we’ve armed you with enough information to at least have a conversation with your testing team. Get everyone together, run the numbers using one of the many Cloud cost calculators, ask the IT manager how much of a hassle it is to wrangle existing test infrastructure and mull it over. In highlighting our own Cycle Appliance, we also hopefully shed some light on Infrastructure-as-Code and our implementation of it that helps automate the building and deployment of test environments in Azure.
Drop us a line and let us know what you test on and how it’s working for your testing team!
This post was written by:
Technical Pre-Sales Consultant
James has been working in software pre-sales and implementation since 2000, and has more recently settled into focusing on technical pre-sales. He takes care of our hands-on demonstrations, and eagerly awaits your request to see our Cycle test automation software in action. Drop him a line at: james.prior[at]tryonsolutions[dot]com.
Cloud Automation Engineer II
Ryan Berger is a Cloud Automation Engineer II at Tryon Solutions Group. Ryan’s main focus is on the Microsoft Azure platform, where he has spent the last four years learning how to automate the creation and configuration of cloud resources using various toolsets (ARM, PowerShell, Desired State Config, Cloud-init, etc). Automating the deployment and configuration of not only Windows and Linux based virtual machines, but also Azure services such as Azure SQL, Azure Kubernetes Services, Azure App Service, etc.