First, let’s look at the meaning of the term regression testing
According to Wikipedia, “regression testing is any type of software test that aims to reveal software regressions.” Such regressions occur when software functionality that previously worked correctly stops working as intended. Regressions usually occur as an unintended consequence of program changes. Common regression testing methods include rerunning previously run tests and checking for any errors that were previously resolved.
Final definition of “regression testing”:
It is very normal to make changes to the code during the development and maintenance phase of software. So a software program that was previously executed correctly cannot function in the same way after the changes. Therefore, repeating the test cases of a program, which functioned correctly before the changes, and for the purpose of debugging, can be concluded as “Regression tests”.
Regression testing is a quality control measure aimed at ensuring the following two conditions:
a) Newly modified code meets the specified requirements.
b) Unchanged code is not affected by the above change.
By definition, regression testing is repetitive. Therefore, most tests would be best suited for automation, where after running some test iterations, test automation would prove to be significantly cost effective compared to the heavy manual process.
Why are defects introduced during changes?
Correctly working applications fail due to incorrect or incomplete changes made to them. In the software industry, the percentage of defects in the applications is quite high. In general, one in six attempts made through the applications to correct them is themselves faulty.
The main reason for the high number of defects being introduced is
1) Bad system documentation from the developers.
2) Developers’ tendency to address the symptoms rather than identify the underlying causes.
3) Lack of developer experience.
Objectives of regression testing:
1) The purpose of regression testing is to identify unexpected defects. These defects or errors are the ones that have remained in the system, as while modifying the code, the developer probably couldn’t fully understand the internal correlations of the code.
Therefore, regression testing remains the only reliable way to provide sufficient confidence that changes or additions to the code are safe and do not tend to “break” the existing functionality of the application.
So regression testing becomes a must every time code is changed or used in a new environment. It is essential to verify the integrity of the code in order to identify and correct the defects as soon as possible.
2) The purpose of regression testing is not only limited to testing the correctness of an application, but also extends to monitoring the quality of the output. For example, when designing a compiler, regression testing can emphasize the size of the code, the time to simulate, and the time to run the test suites.
What are the best practices / strategies for regression testing?
1) Formulate a regression testing policy on a regular basis if we are to be successful in developing reliable and predictable software applications.
2) Perform standard actions defined in the test procedure and verify that the desired responses are correct. Any failure of the system to meet the set of desired responses becomes a clear indicator of system regression.
3) Make sure the regression test is correct and not outdated.
4) Carefully analyze any defect that escapes detection during the regression testing process. This would require careful updating of the regression test cases so that we do not slip from such defects in the future.
5) Generally, unit test cases and integration test cases are used to build regression test cases; therefore, instead of one large regression test, it is better to create a logical batch of such test cases in the form of an extensive test suite. This would provide flexibility in conducting tests on small units in case of urgency or time constraints.
6) Based on the famous 80/20 management principle, we can assume that there is a good chance that 20% of the functions will be used for 80% of the time. Therefore, regression test suites can be designed accordingly.
The severity of a problem in relatively less common functions would be less used by a small number of users. Thus, the main focus of our regression tests should be on overused trades, menus and screens etc.
7) For smaller projects, run a regression test after every successful compilation or at least once a week. Such repetitive activities can be automated using tools such as HP-QuickTest Professional etc.
8) Depending on the risk factors within the company, we can design the regression test suite. It has been seen that there are certain types of failures, which do not occur often, but when they do occur, they have a serious impact on the business process.
Therefore, whenever a change is made to the system or environment, we must perform regression tests on such sensitive areas that are adversely affecting the business.
9) We need to identify the application areas that are known to have a high failure rate and include more regression tests.
10) Closely tie the regression tests to the functional testing and create the regression tests based on successful test cases created and used during the functional tests.
11) Regularly rerunning successful functional test cases (which happen to verify the desired functionality of the application) as regression tests.
12) Regression tests should be like a chain, starting at the unit level, adjusting the unit tests and running again after any change to the unit is detected. The chain of regression testing continues through integration testing, system testing, user acceptance testing, and all operational phases under the SDLC.
13) As a best practice, strict regression testing should be performed before a build is released over the mass or in a live environment in the organization. Such an approach would help uncover any major defects that could otherwise seriously affect costs, productivity, schedule, and most importantly, the negative impact on the company’s reputation.
14) As a best practice, regular regression testing should be a continuous exercise in the case of web-based and all multi-user systems. Regular regression tests help continuously monitor the performance of important transactions within specified limits.
Any factor responsible for slowing down the response time for a large transaction can be easily identified by frequent regression tests.
15) Apart from testing the functional aspects, we have to perform regression tests that relate to non-functional features of the application, such as the performance, security, usability, etc. whether the design has a significant effect on system performance.
16) Before running the time-consuming regression test scenarios on newly supplied code, doing smoke tests or health tests can be useful to save time, as smoke tests or health tests usually reveal the obvious errors. Detection of such errors early can lead to early fixation.
17) We have to keep a close eye on the side effects of the bug fixes. Chances are the bug will be fixed but at the same time our fix could have introduced another bug into the system.
18) Regression testing should be treated as an integral part of the extreme programming method of software development. This includes extensive automated testing of the entire application at every stage of the SDLC.
Pitfalls of regression test results
1) We should not get happy and satisfied just because the system has responded as desired, as there may be all probability of the presence of introduced defects that may have escaped detection.
2) In most companies, the critical functionality of the applications is verified once and it is believed that it would continue to behave properly until deliberately changed. However, the fact remains that even minor routine changes to the code can have serious unexpected side effects, which can likely break the previously verified functionality.
What are the benefits of a good regression testing policy?
1) Great improvement in the effectiveness of the software development and testing personnel.
2) Great success of the development projects resulting in reliable and stable applications.
3) Development teams change the code without fear of breaking previously verified functionality.
4) Problems arising from code changes are identified early in the life cycle.
5) Great savings of man hours spent searching for and fixing software errors caused by code changes.
What are the characteristics of a good regression testing policy?
1) It defines solid guidelines for the use of regression systems.
2) It defines concrete plans to consistently implement and integrate the defined guidelines into the SDLC. This results in detecting existing errors as well as the newly introduced errors due to uncontrolled changes made to the code over time.
3) It provides a mechanism to measure and monitor the application of the policy and system to report the data being tracked.
4) Well-documented system for controlling and maintaining all types of regression test suites that are critical assets. The policy outlines the system for backup, configuration management, maintaining the latest and defining clear ownership and responsibility of specific personnel to manage the regression test suites.
What are the reasons that regression testing is not so popular?
Despite the many benefits of regression testing, it is not used by many organizations, probably for the following reasons.
1) Software development companies believe that regression testing is a bit complicated and difficult to maintain.
2) In most cases, companies do not have a well-defined and implemented regression testing policy.
3) Even if there is some policy in place, there is generally a lack of commitment from the management side to the implementation of such policy.