Actually, the use of an effective test automation system minimizes test case execution time (for example, up to one night), without which it is impossible to confirm that the product’s quality has been improved (namely the number of remaining errors has been reduced).
System testing is carried out as part of testing cycles (the phases during which a developed test set is run on the build of the software product being created). Before each cycle starts, the developed or corrected software build is registered and all its errors found during the test run are recorded. Then the errors are corrected, and a new build is prepared for the next test cycle. The testing is considered to be completed when there is experimental confirmation of the achieved quality level with respect to the selected test criterion or about the reduction in the density of undetected errors to a certain previously agreed value. Completion of the testing cycle in one day or several hours can be achieved solely through test automation.
Test automation companies are involved in performing tests in an automated manner to help to reduce test execution time and accelerate the pace of production releases.
A general test automation system stores the following information:
- Results of the test set execution, recorded in the log file. The log file contains traces (“protocols”) that represent the sequences of some events (the values of individual variables or their combinations) implemented in the test run and each node in the program’s graph representing each of these events. The traces may include sequences of explicitly and implicitly defined labels specifying the execution paths through the program’s control flow graph, the set of values of the variables for these labels, the values of the intermediate results achieved for some labels, and so on.
- Statistics of the test cycle, containing: 1) the results of skipping each test from the test set and comparing them with the reference values; 2) the facts that served as the basis for deciding whether to continue or terminate the testing; 3) the coverage criterion and degree of its satisfaction, achieved in the testing cycle.
- The result of each test run analysis is a list of problems such as errors and defects, which is entered in the project development database. Then the errors are analyzed, and each critical problem is identified, associated with the corresponding module and the developer, prioritized and monitored, which ensures its solution (correction or attribution to the list of known problems, which for one reason or another are postponed) in subsequent builds.