Like herding cats: managing test automation tools in an open-source world

In the world of testing and devops, it’s easy to take on a Dickensian view: “It was the best of times, it was the worst of times, it was the age of wisdom, it was the age of foolishness.” Technology has changed the practices of developing and testing tools irrevocably. There’s a wealth of tools, both commercial and open source, which companies can use to automate rote processes and speed up release cycles. The number and sophistication of these tools and platforms continually expands as the technology landscape shifts to incorporate developments like artificial intelligence and the internet of things.

In response to this environment of hyperautomation, companies tend to have two different approaches.

The first is one of restriction. A company will dictate the use of a few tools that their QA teams are allowed to use. Often, this company is in a highly regulated industry or has purchased a suite of vendor testing tools. However, team leaders may fail to provide adequate education, training, and guidance to optimize the sanctioned tool set. This creates little incentive to use the company’s tools. As a result, testers search open source forums and engage in online communities that lead to experiments with open source tools. This experimentation may not be the best choice for the project’s overall goals

The other scenario that plays out is more of a freewheeling environment. Do what you want and figure it out on your own, as long as you deliver great work on time. Technical people love that sort of freedom and independence. It’s fun to experiment and try out the latest, greatest tools. Yet in the bigger picture, things can get out of control quickly: too many applications with redundant functionality, a lack of tools integration leading to issues with reporting and visibility, and challenges in standardizing processes. Over time, this freewheeling environment may lead to lower productivity, higher costs, and/or quality issues.

Neither option is wrong, but there is a third way. The test infrastructure freeway, defined by Bryan Osterkamp of financial services company USAA at the recent QASymphony user conference, marries control with flexibility. The freeway includes a company-recommended set of tools for test automation, such as unit testing, performance testing, security testing, and test management. People who stay on the freeway don’t have to make a lot of decisions or research to get started. They get access to instructor training and guides such as wikis and help desk support, along with templates and blueprints to ramp up quickly.

This is all well and good but what if the team decides that these tools just won’t fit its project? That’s fine. It can choose to experiment with something else. However, it doesn’t get company support, training, and so on. That reality will make teams think twice before “going rogue.” On the other hand, if the team discovers a new tool that is the perfect fit for the project and future projects, it can be integrated onto the freeway later—perhaps replacing an older tool.

Here’s another advantage of the freeway: built-in reports and dashboards. The rogue tester or developer who wants to get off the freeway will have to comply with whatever metrics and reports are standard across the organization to ensure product quality and business needs. Off the freeway, they must create reports themselves from scratch. That’s time-consuming. Not everyone’s up for that task.

The freeway can include an integration service linking all the tools so that metrics can be easily shared across the department. That’s nice for for management but also individual team members who can better understand the bigger picture and collaborate on best practices.  

By creating a test infrastructure freeway, you can balance the organization’s business needs and requirements with technical innovation. Most testers can accept the following process:

• Start on the freeway and give the supported tools a shot.
• When needed, go off the freeway, understanding the ramifications.
• Crowdsource for alternatives that could replace an outdated freeway tool.

As software needs evolve, becoming ever more complex, QA teams will have to balance flexibility and structure. It helps to begin with an assessment of the current automation environment. Understand what is being used and why, how many tools overlap and how many tools operate in isolation with data that’s not accessible by managers. If your organization seems too strict, or too loose, consider whether a change in the form of a testing-tools freeway could help bring the balance your team needs to thrive.