Test Automation in Times of Continuous Delivery

Test automation in times of continuous delivery

Continuous delivery has changed the challenges of automated software testing in unit, integration, and UI testing. But even specialized testers are faced with great difficulties here. Fortunately, there are up-to-date solutions for a smooth process.

Software has to be tested on several levels. There is the smallest unit unit tests. Mostly created by developers themselves, they check a completed functionality (for example, a class or a method) in full isolation. Dependencies on other units and outside influences are replaced by mocks. This will help developers ensure that a piece of code behaves as expected. Unit tests form the basis of the test coverage and are considered standard in software development.

However, the isolation in the unit test has consequences. The correct cooperation of several components is not part of the test. Responsible for this are integration tests as the next level in the test hierarchy. They focus on the interaction of several building blocks. Message-based interfaces to the outside or even the deployment configuration for an application server are about the subject. Examples include connection pools and factories, JNDI resources or third-party dependencies that are configured via Dependency Injection at the deployment time.

There are always several components involved in an integration test. Automation of the integration is therefore much more difficult than with unit tests. In order to enable meaningful tests, dependencies to other systems have to be simulated. The required components with their infrastructure and configuration must be set up. Both have a much higher complexity and expense.

After testing the interaction of individual components and their integration, UI testing is missing for the user interface check (for example, a web frontend or a rich client application). They simulate user interactions and thus serve the user interface. During the simulation, the appearance and functions of the UI are validated. The test should ensure that the data provided by the server are displayed correctly, which plays an important role in error handling.

Why test automation across all areas?

The three types of tests are among the most important in software development. Automation in all areas is playing an increasingly important role in today's software development. This is due to the fact that application development has to bring bug fixes and new features into production more frequently and faster. Teams are entrusted at ever shorter intervals with the task of checking a software version for release capability. Therefore, all tests must be repeatable and efficiently executable.

An extensive test phase before each release with perhaps a lot of manual tests can no longer afford a team. In order to succeed in the market, development-accompanying, fully automated tests continuously determine the release capability of a software version. This allows the stability of a software to be evaluated more often and earlier in the development process. An update of the software is thus faster and more often possible.

However, the construct of fully automated tests requires a rethinking of management. Many project managers and clients traditionally see the need for test automation in the area of??unit testing. In the areas of integration and UI tests, the willingness to provide appropriate funds is usually absent. One of the reasons for this is that the initial effort for a complete test automation is high compared to manual tests. All participants must be aware that test automation is not a minor matter.

If you compare the effort for manual testing, it quickly becomes clear that they are only cheaper in the short term. In the medium and long term, automated tests pay off in all areas - both financially and qualitatively. Ideally, manual testing is reserved for domain experts who specifically focus on the new features of an application prior to release before the new piece of software finds its way into production.

When test automation?

It is advisable to strive for development-accompanying automation right from the start. As with unit testing, integration and UI testing should be an integral part of the build process so that every test automatically performs all tests. When integrating into the build environment, the principle of the fail-fast approach applies. Unit tests are handled first, and if a test fails, all subsequent ones in the hierarchy will no longer be executed.

As shown in picture, the number of tests and the variety of tested variants are increasingly decreasing from bottom to top, resulting in the image of a pyramid. The decreasing number of tests up in the pyramid makes sense. On the one hand, integration and UI tests are much more complex in terms of care and set-up. On the other hand, the execution time of the integration or UI tests is higher than with the unit tests.

Thus, a more efficient runtime of the entire test run is achieved if the tests are always limited to the essentials. Only if the feedback of a test run is timely, the likelihood is high that developers can assign the result of the test without much effort to a dedicated code change. The risk of tedious and time-consuming troubleshooting can be significantly reduced.

Successful test automation often depends on the organization of the project. Either a fixed test team is appointed to take care of the test cases, or an overarching team is equally responsible for development and testing.

Cross functional teams have many advantages. Better communication and collaboration as well as a more even sharing of knowledge in the team are just two of the key points. Such a team avoids time-consuming communication loops between bug opening, test verification, and bug fixes. For this reason, it is recommended during project set-up to find a joint procedure model (Scrum, Kanban) for both development and testing.

issuesNot easy with test automation

The most common problems with test automation in software development are:

• infrastructural problems

• organizational hurdles

• Problems creating test data

• missing standards and guidelines

The infrastructure of a complete test environment for the integration or UI test can be expensive. The software to be tested (the so-called "System Under Test", SUT), for example, has to be deployed in advance in an application server. If necessary, a database is required or external interface partners are to be simulated. It's similar in UI testing, because testing usually requires a backend to show meaningful UI data. In addition, the UI must be displayed error-free on different devices and in different browsers.

A test also needs prepared test data available in the SUT. The biggest hurdle is that automated testing needs to be repeatable without having to rebuild the entire infrastructure every time. This requires either read-only data or a test data generator that prepares technical data in different states and technical sub-steps. This topic is often an underestimated task in test automation and causes problems for many companies. The tests must not influence each other. It is therefore best practice for each test to prepare its required test data in the SUT and clean it up again.

Care of the tests

Care and maintenance of the tests must be organizationally integrated as an integral part of the code base in the development process. A failed test, like a production bug, finds its way back into the development process for error analysis and resolution.

However, a critical situation in the development process does not arise until several errors accumulate over and over hours or days in the Continuous Build. This inevitably leads to a confusing situation, because no one can understand which error belongs to which code change. This makes troubleshooting unnecessarily difficult and must be prevented at all costs. If the team does not pay attention to the stability of the build, the team quickly gets used to a bad build. This increasingly reduces the willingness to eliminate the errors. After a short time the build can not give any information about the quality of the software. The state either prevents productive deployment or in the worst case leads to unrecognized errors in production.

The solution is to address and prioritize errors in the continuous build in the daily team meeting. This provides the team with continuous visibility of the tests and defines responsibilities for a failed build. In order to keep the quality permanently high, the build build feedback loop should promptly provide all data for error analysis. The test code should be the subject of Code Review to ensure lasting compliance with guidelines. It is also helpful to encourage developers to run code unit and selected integration tests locally before checking in to spot obvious bugs.

Start of a Continuous Delivery Pipeline

If the project team makes the decision to rely on test automation, the topic of continuous integration (CI) is on the agenda first. Test automation is the lowest level of full automation. Continuous integration is when artifacts, such as in the Java environment, are automatically created, tested, and placed in a repository by an Application Server WAR file.

The term "automated" means that after the developer has committed, no manual actions are necessary to reach the target - in this case the finished WAR file. CI tools like Jenkins or Travis provide good support to set up such a pipeline. In technical jargon, "continuous delivery" is used when the CI pipeline is extended by an automated deployment that updates any target system with a click.

Different levels of manual approval can be built in at the level. It is conceivable that a test person deploys the systems for acceptance tests with one click. For productive deployment, on the other hand, only a specific operations team is authorized. If a high degree of maturity of automated tests is achieved, continuous deployment can be started as the next step. The trigger for the productive deployment here is either the developer's commit or a time interval (for example, at 3 o'clock at night).

Which degree of automation makes sense for the respective project often results from organizational framework conditions. In the traditional large company, the company usually bears the responsibility for the availability of an application. As a result, he is also responsible for deployment, which speaks for manual approval steps in the continuous delivery pipeline. However, with growing, emerging software from a start-up, it can be helpful to deploy a stand that has been removed by automated testing directly. This allows users to quickly get excited about new features for the product.

No matter in which form, automated deployment should always be the goal. This allows the project team to focus on their core task of driving the software forward. Manual deployments create distraction, are in hindsight incomprehensible, difficult to repeat, usually poorly documented and therefore not worthwhile.

To implement the test pyramid into executable test code, various tools and frameworks provide support. Using the example of a classic Java EE system with backend and web frontend, a continuous delivery pipeline is outlined below. As often used for build tools that both build automated artifacts and perform various tests, Maven applies. Depending on the type of test, the frameworks mentioned in the following chapters are suitable.

Conclusion

Full test automation with unit, integration, and UI tests is the prerequisite for continuous delivery in software development. In the areas of integration and UI testing, automation is more complex and requires preparations for infrastructure and test data. By using specialized tools, the effort can be mastered. Automation becomes a powerful weapon in the fight against bugs. The automated tests enable fast and frequent deliveries of software and are valuable in future refactoring.