How Automation Testing Can Significantly Benefit Your Testing Process

By exploring automation testing, you can increase your efficiency. For more information on optimising test case coverage, see test design techniques: equivalence partitioning & boundary values. To learn more about api test automation, see Postman and TestRail.

Why automate tests?

Saves your time and effort

Doing repetitive tests requires a large effort and is time consuming. When deploying new features to production, you certainly have to run a regression test to ensure that the old features are still working well. By automating the test run for regression, you can save time and effort, therefore, you can focus more on testing the under developed features.

Increases test coverage

You may have a large collection of test cases, and when you have limited time to perform manual testing, automated testing will come in handy for running the tests that you cannot cover manually in the given time. With the added capability of running the cases in parallel, you can cut down the execution time, while increasing the coverage.

Improving testing accuracy

Sometimes, testing is boring, especially running the repetitive tests. When it is combined with other sources of stress, this might lead to inaccurate tests. You might: make incorrect inputs, incorrectly document results, or even forget about the specifications of a certain test. However, automated testing will run exactly as you write it, precisely inputting given variables in the given places, documenting each result of every case run.

Saving cost

Despite high initial costs (e.g cost for the tools and also hiring the automation testers), once the automated framework is set up, there is an overall reduction in costs as you can reuse them 24/7 without any additional cost. The more automated test cases that you are going to automate, the higher the return on the investment. However, keep in mind that there are some test cases when manual testing is essential. 

Obtaining immediate feedback

Test execution can be run at a faster speed and bugs are identified sooner, getting attention early in the development cycle. 

What kinds of tests should you automate?

The tests that consume your time and effort because you have to repeat them over and over again, such as regression testing, smoke testing, or sanity testing. By automating those repetitive tests, you will have more time to do exploratory testing on features that are under developed.

If you already have a large base of test cases and scenarios, you can start with the most essential features of your product. You can also start with happypath cases or scenarios, ensuring that you can quickly detect if there are any bugs occurring on those critical user flows.

Step-by-Step to Automation Testing

You might be wondering what the real process of automation testing is. Are the results obtained able to be compared with the process? Before diving deeper, let’s take a quick look at the following steps of the automation testing process:

Test Automation Process (source: https://www.guru99.com/images/testautomationprocess.png)

Test Tool Selection

The tool selection solely depends on the technology on which the application tests are built. Having a deep understanding of multiple tool types is also a big factor in choosing the best tool. Some factors that can be used to determine the best tool are the test cases that will be automated, the programming expertise of the QA team in your company, and the project budget (whether to go for the free or paid tool). Following are the process of selecting the automation tools:

Tools Selection Process (source: https://www.guru99.com/images/TestManagement/testmanagement_article_6_7.png)

• Identify requirement for tools
  This is basically what you expect from the tool depending on your needs. For example, whether the tool can generate test cases automatically, or the tool can generate a report with desired format, or the tool can execute selected test cases with a given set of test data, or all of them.

• Evaluate the tool and vendors

This step is to dive into the tool options. We can list the available tools on the market, shortlist them based on our needs/requirements, check the vendor (reputation, after sale support, tool update frequency), evaluate the quality of the tool by taking the trial version of the tool. More details on how to evaluate the tool will be explained in this post.

• Estimate the cost and benefit

In this step we will compare between the cost of the tool and the value/benefit that we will receive from the tool. The balance between cost and value/benefit of the tool may affect the final decision.

• Make the final decision

After spending time doing the three steps above, we still need to try the tool in our actual working environment before deciding to buy the license. We might also need to discuss with the team and consult with experts before making a decision.

Test Automation Tool Evaluation Criteria

Team Capability on the Test Automation Skill

Before you select any test automation tool, you should know you have the necessary skilled resources that will be allocated for the automation testing. If currently there are no QA resources who will do the automation testing, and you want the automation tools which are the most powerful, it would be wasted.

One of the ways to fully optimize the automation tool is by designing the test case effectively to eloquent the test scripts that ensure maximum test coverage. This is only possible if the testers have the knowledge and skills of the programming languages, techniques and also other skills that will be required to use the automation tools and framework of choice. 

Hence, before selecting and purchasing the tools, the first thing that must be discussed with stakeholders and management is the team’s capabilities. If they don’t have any, there should be a plan to train the team for the skills required to obtain optimal benefits from the tool.

Team Budget

Besides the team’s capability, the other concerns that need to be considered are the financial concerns. Before deciding on which automation tools that you will choose to get started with test automation, you need to know how much budget that you could spend. You should also be aware that the cost is not only from the licensing but it’s also needed to count on how much the operational costs, maintenance costs, cost of support (if it’s required) and also the additional cost to train existing testers or resources so they could utilize the automation well. 

Once you’ve obtained how much budget you could spend, then you will start to find the automation tools accordingly.

The Desired Features

Each team or company will have different requirements. But there are some key factors that you should always take into consideration when choosing a suitable automation tool. These are the following key factors to consider:

1. Supported platforms
2. Programming languages
3. Paralel tests execution
4. CI/CD integration capabilities
5. Reporting functionality

Platform Supported and Tested

Before purchasing any automation tools for testing, it is important to know what the tech stack that the team or company is currently using. You may talk to the developers or your Engineering team and make a comprehensive list of the software and hardware then once you’ve got the list, check if your selection of automation tools are compatible with them. Or when you don’t have the automation tools to select, then you can do the other way round which is start to research on the automation testing tools based on the comprehensive list of tech stack that you have. 

Once you’ve found the automation tools based on your requirements of supported platforms, you could download the trial versions of the tools and verify if it works properly. You wouldn’t want to realize later on that the real apps don’t align with the requirements that you are looking for.

Ease of Use

One of the significant factors on test automation is script maintenance. This will have a significant impact especially on the cost. The more complex automation tools, the higher maintenance is required. So when you select the automation tools, you should choose the automation tools that could have the capabilities to reduce the effort on the test script maintenance (e.g eliminating object locator flakiness). Besides, you also could have the reusability script for the similar test cases as it will save you and the team a great deal of time.

Define scope of Automation

Not all test cases have to be automated. There are test cases that could be automated and others done manually. For example, test cases for usability testing and exploratory testing are better not be done automatically because the results will be faster and more accurate if done by humans directly. For more details, please check following table:

Test Cases that could not be Automated	Test Cases that could be Automated
UI test cases that are still dynamically updated	Test cases for major functionalities which are frequently used in the application
Test cases that have a complex flow	Test cases that have scenarios with a large amount of data
Test cases for usability testing	Repeatable test cases
Test cases for exploratory testing	Can be used for cross-browser and cross-device

Planning, Design and Development

Test planning stage in test automation process is really important. This stage represents the process to review long-lead-time test planning activities. These are the activities on this phase:

Test Planning

• Identifies test procedure creation standards and guidelines
• Define the software, hardware and network required to support the test environment
• Define the test data requirements
• Plan for the test schedule
• Define the measurement for the performance requirements
• Define the procedure to control the test configuration and environment
• Define the defect tracking procedures and the tracking tools that integrated with the automation tool.

One of the important things from this phase is defining the test data requirements. Having good test data will be extremely useful for the data driven testing. The data should be entered into input field during the automated test is usually stored in the external file. Using external data makes your automated tests reusable and easier to maintain. To add different test scenarios, these data files could be easily extended with new data without needing to edit the actual automated test.

Test Design

The test design stage is able to begin when the test requirements have been derived using the described techniques. In this phase, it consists of the definition of logical group of test procedures and also the naming convention for the suite of test procedures. It is important that the test procedure is defined as each procedure will identify as either an automated or a manual test.

Once the test procedure is defined, the next step is to identify those test procedures that standout, then these test procedures are flagged and the tester will prepare the detailed design document. During the test planning phase, the test data requirements have been defined. Now in this test design phase, since we have already detailed the test design document, we can map the test data requirements against the defined test procedures.

Before designing test cases, the test engineer needs to see the dependency between the test cases. If there are test cases that are dependent on each other, then they should order them in a respective manner, so that when it’s time for execution, the execution should be stopped on the first failure and the bus should be reported.

Test Development

Once the test planning and design stages are done, the test team is now ready to perform development. Test development involves the development of the test procedures that are reusable, simple, maintainable and also robust. By developing test procedures based on the development guidelines, the test team could create the initial building blocks for an automation infrastructure. The automation infrastructure will eventually contain a library of common and reusable scripts.

The test engineer could make use of the test automation infrastructure to support reuse of archived test procedures and minimize duplication This is will enhance the entire automation effort. Besides that, the test engineer needs to stick to the test procedure development and also execution schedule, test design information, test procedure development guidelines and also test tool user manuals. Armed with the proper guidelines and documentation, the test engineers will have the foundation that will allow them to develop more cohesive and structured set of test procedures.

When developing the test scripts, the testers should be able to write the scripts that could accommodate some degree of change. Automated tests created with scripts or keyword tests are dependent on the application under test. To make your automated tests resistant to the UI changes and ensures that the automation test will still work without having to make changes to the test itself, you could provide unique names for your controls. By having the unique names, it will also eliminate the automation testing tools from relying on location coordinates to find the control, which is less stable and breaks easily.

Hence to minimize the test automation maintenance, the test engineer should be able to foresee and could accommodate those changes when developing the scripts in the early stage.

Test Execution

After you are done with the test scripts, input test data into the tool and start to execute the test scripts. Run the test scripts (you will not do anything in this step just let the tool execute each test automatically) and finally get the report. The report usually provides how many tests are passed or failed.

The most important thing about the test execution plan is that it has to be planned before the creation scripts.

Once the test execution is finished, the test report should be generated. The information that you should consider when generating the test automation report:

• Report for batch of scripts

If multiple test cases are included in a batch, once that batch is executed, the following information needs to be included in the report:

1. Total number of scripts
2. List of all test cases
3. Test results (Passed or failed)
4. Duration
5. Environment details.

• Report for individual test case

A detailed report for an individual test case needs to be generated. The following information needs to be included in the report for the individual test case:

1. Test Case Name
2. Test Case ID
3. Duration
4. Status (Passed or failed)
5. Screenshots and error messages (only for the failed test cases)
6. The exact line number where the test case is failed
7. Any other helpful logs that the test engineer wrote in the script to be displayed in the report.

Maintenance

If in the future there are changes that are implemented to your application, you need to update your test scripts according to these changes, otherwise your test run will fail when executed. If you are using Agile methodology in your application development process, where one sprint will be ended after two weeks means you will need to maintain your test scripts once every two weeks.

Key Points

1. There are 5 reasons why you should automate your tests, which are: it will save your time and effort, increases testing coverage, improves testing accuracy, saves cost and it will have immediate feedback once the test is run.
2. The kinds of test cases that you should automate is test case that consumes your time and effort because you have to repeat it over and over again such as regression testing
3. To do the test automation, the first step is to select the correct tools. There are some factors that you need to consider when evaluating which tool is the most suitable for the team.
4. Once the tool is selected, the next step is to define the scope of automation, planning, design and develop the script, test execution and maintenance.

References

1. Test Automation trends
2. 5 Automation trends in software development
3. Evaluation Criteria for Automation Testing Tools
4. Choose the right tools for test automation