research-article

Practical selective regression testing with effective redundancy in interleaved tests

Authors:

Dusica Marijan,

Marius LiaaenAuthors Info & Claims

ICSE-SEIP '18: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice

Pages 153 - 162

https://doi.org/10.1145/3183519.3183532

Published: 27 May 2018 Publication History

Abstract

As software systems evolve and change over time, test suites used for checking the correctness of software typically grow larger. Together with size, test suites tend to grow in redundancy. This is especially problematic for complex highly-configurable software domains, as growing the size of test suites significantly impacts the cost of regression testing.

In this paper we present a practical approach for reducing ineffective redundancy of regression suites in continuous integration testing (strict constraints on time-efficiency) for highly-configurable software. The main idea of our approach consists in combining coverage based redundancy metrics (test overlap) with historical fault-detection effectiveness of integration tests, to identify ineffective redundancy that is eliminated from a regression test suite. We first apply and evaluate the approach in testing of industrial video conferencing software. We further evaluate the approach using a large set of artificial subjects, in terms of fault-detection effectiveness and timeliness of regression test feedback. We compare the results with an advanced retest-all approach and random test selection. The results show that regression test selection based on coverage and history analysis can: 1) reduce regression test feedback compared to industry practice (up to 39%), 2) reduce test feedback compared to the advanced retest-all approach (up to 45%) without significantly compromising fault-detection effectiveness (less than 0.5% on average), and 3) improve fault detection effectiveness compared to random selection (72% on average).

References

[1]

A. Andrews, S. Elakeili, and A. Alhaddad. 2015. Selective Regression Testing of Safety-Critical Systems: A Black Box Approach. In Software Quality, Reliability and Security - Companion (QRS-C), 2015 IEEE International Conference on. 22--31.

Digital Library

[2]

H. Baller, S. Lity, M. Lochau, and I. Schaefer. 2014. Multi-objective Test Suite Optimization for Incremental Product Family Testing. In 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation. 303--312.

Digital Library

[3]

K. Z. Bell and M. A. Vouk. 2005. On effectiveness of pairwise methodology for testing network-centric software. In 2005 International Conference on Information and Communication Technology. 221--235.

[4]

Yanping Chen, Robert L. Probert, and D. Paul Sims. 2002. Specification-based Regression Test Selection with Risk Analysis. In Proc. of the 2002 Conf. of the Centre for Advanced Studies on Collaborative Research. IBM Press. http://dl.acm.org/citation.cfm?id=782115.782116

Digital Library

[5]

Sebastian Elbaum, Gregg Rothermel, and John Penix. 2014. Techniques for Improving Regression Testing in Continuous Integration Development Environments. In ACM SIGSOFT International Symposium on Foundations of Software Engineering (FSE 2014). ACM, New York, NY, USA, 235--245.

Digital Library

[6]

Emelie Engstrom and Per Runeson. 2013. Test Overlay in an Emerging Software Product Line - An Industrial Case Study. Information and Software Technology 55, 3 (March 2013), 581--594.

Digital Library

[7]

Gordon Fraser and Franz Wotawa. 2007. Redundancy Based Test-suite Reduction. In Proceedings of the 10th International Conference on Fundamental Approaches to Software Engineering (FASE'07). Springer-Verlag, Berlin, Heidelberg, 291--305. http://dl.acm.org/citation.cfm?id=1759394.1759425

Digital Library

[8]

M. Gligoric, L. Eloussi, and D. Marinov. 2015. Ekstazi: Lightweight Test Selection. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. 713--716.

Digital Library

[9]

Arnaud Gotlieb and Dusica Marijan. 2014. FLOWER: Optimal Test Suite Reduction As a Network Maximum Flow. In Proceedings of the 2014 International Symposium on Software Testing and Analysis (ISSTA 2014). ACM, New York, NY, USA, 171--180.

Digital Library

[10]

Rajiv Gupta, M. Jean Harrold, and Mary Lou Soffa. 1996. Program Slicing-Based Regression Testing Techniques. Software Testing, Verification and Reliability 6, 2 (1996), 83--111.

[11]

M. Jean Harrold, Rajiv Gupta, and Mary Lou Soffa. 1993. A Methodology for Controlling the Size of a Test Suite. ACM Transactions on Software Engineering and Methodology 2, 3 (July 1993), 270--285.

Digital Library

[12]

M. P. E. Heimdahl and D. George. 2004. Test-suite reduction for model based tests: effects on test quality and implications for testing. In International Conference on Automated Software Engineering 2004. 176--185.

Digital Library

[13]

Aymeric Hervieu, Dusica Marijan, Arnaud Gotlieb, and Benoit Baudry. 2016. Practical Minimization of Pairwise-covering Test Configurations Using Constraint Programming. Information Software Technology 71, C (March 2016), 129--146.

[14]

H. Y. Hsu and A. Orso. 2009. MINTS: A general framework and tool for supporting test-suite minimization. In International Conference on Software Engineering.

Digital Library

[15]

D. Jeffrey and Neelam Gupta. 2005. Test suite reduction with selective redundancy. In 21st IEEE International Conference on Software Maintenance (ICSM'05). 549--558.

Digital Library

[16]

Dennis Jeffrey and Neelam Gupta. 2007. Improving Fault Detection Capability by Selectively Retaining Test Cases During Test Suite Reduction. IEEE Transactions on Software Engineering 33, 2 (Feb. 2007), 108--123.

Digital Library

[17]

Jung-Min Kim and Adam Porter. 2002. A History-based Test Prioritization Technique for Regression Testing in Resource Constrained Environments. In International Conference on Software Engineering. ACM, New York, NY, USA, 119--129.

Digital Library

[18]

Negar Koochakzadeh and Vahid Garousi. 2010. A Tester-assisted Methodology for Test Redundancy Detection. Advanc. Software Engineering 2010, Article 6 (Jan. 2010), 13 pages.

Digital Library

[19]

N. Koochakzadeh, V. Garousi, and F. Maurer. 2009. Test Redundancy Measurement Based on Coverage Information: Evaluations and Lessons Learned. In International Conference on Software Testing Verification and Validation. 220--229.

Digital Library

[20]

B. Korel, L. H. Tahat, and B. Vaysburg. 2002. Model based regression test reduction using dependence analysis. In Software Maintenance, 2002. Proceedings. International Conference on. 214--223.

Digital Library

[21]

Chengying Mao and Yansheng Lu. 2005. Regression testing for component-based software systems by enhancing change information. In 12th Asia-Pacific Software Engineering Conference (APSEC'05). 8 pp.-.

Digital Library

[22]

D. Marijan, A. Gotlieb, and S. Sen. 2013. Test Case Prioritization for Continuous Regression Testing: An Industrial Case Study. In International Conference on Software Maintenanse 2013. 540--543.

Digital Library

[23]

D. Marijan and M. Liaaen. 2016. Effect of Time Window on the Performance of Continuous Regression Testing. In International Conference on Software Maintenance and Evolution. 568--571.

[24]

W. Masri, A. Podgurski, and D. Leon. 2007. An Empirical Study of Test Case Filtering Techniques Based on Exercising Information Flows. IEEE Transactions on Software Engineering 33, 7 (July 2007), 454--477.

Digital Library

[25]

A. Jefferson Offutt, Jie Pan, and Jeffrey M. Voas. 1995. Procedures for reducing the size of coverage-based test sets. In International Conference Testing Computer Software. 111--123.

[26]

Alessandro Orso, Taweesup Apiwattanapong, and Mary Jean Harrold. 2003. Leveraging Field Data for Impact Analysis and Regression Testing. In Proceedings of the 9th European Software Engineering Conference (ESEC/FSE-11). ACM, New York, NY, USA, 128--137.

Digital Library

[27]

Xiaoxia Ren, Fenil Shah, Frank Tip, Barbara G. Ryder, and Ophelia Chesley. 2004. Chianti: A Tool for Change Impact Analysis of Java Programs. In Proceedings of the 19th Annual ACM SIGPLAN Conference on Object-oriented Programming, Systems, Languages, and Applications (OOPSLA '04). ACM, New York, NY, USA, 432--448.

Digital Library

[28]

Gregg Rothermel and Mary Jean Harrold. 1997. A Safe, Efficient Regression Test Selection Technique. ACM Transactiona on Software Engineering and Methodology 6, 2 (April 1997), 173--210.

Digital Library

[29]

G. Rothermel, M. J. Harrold, J. Ostrin, and C. Hong. 1998. An empirical study of the effects of minimization on the fault detection capabilities of test suites. In International Conference on Software Maintenance. 34--43.

Digital Library

[30]

Gregg Rothermel, Mary Jean Harrold, Jeffery von Ronne, and Christie Hong. 2002. Empirical Studies of Test-Suite Reduction. Journal of Software Testing, Verification, and Reliability 12 (2002), 219--249.

[31]

A. S. M. Sajeev and B. Wibowo. 2003. Regression test selection based on version changes of components. In Software Engineering Conference, 2003. Tenth Asia-Pacific. 78--85.

Digital Library

[32]

Amanda Schwartz and Hyunsook Do. 2016. Cost-effective regression testing through Adaptive Test Prioritization strategies. Journal of Systems and Software 115 (2016), 61 -- 81.

Digital Library

[33]

Lingming Zhang, Darko Marinov, Lu Zhang, and Sarfraz Khurshid. 2011. An Empirical Study of JUnit Test-Suite Reduction. In Proceedings of the 2011 IEEE 22Nd International Symposium on Software Reliability Engineering (ISSRE '11). IEEE Computer Society, Washington, DC, USA, 170--179.

Digital Library

Cited By

Douglas Ferrari Mendonça WAssunção WVergilio S(2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672592
Mendonça WAssunção WVergilio S(2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
https://doi.org/10.1016/j.jss.2024.112157
Singh MChauhan NPopli R(2024)Test case reduction and SWOA optimization for distributed agile software development using regression testingMultimedia Tools and Applications10.1007/s11042-024-19148-1Online publication date: 23-Apr-2024
https://doi.org/10.1007/s11042-024-19148-1
Show More Cited By

Index Terms

Practical selective regression testing with effective redundancy in interleaved tests
1. Software and its engineering
  1. Software creation and management
    1. Software verification and validation
      1. Empirical software validation

Recommendations

Fine-grained test minimization
ICSE '18: Proceedings of the 40th International Conference on Software Engineering

As a software system evolves, its test suite can accumulate redundancies over time. Test minimization aims at removing redundant test cases. However, current techniques remove whole test cases from the test suite using test adequacy criteria, such as ...
Test Redundancy Measurement Based on Coverage Information: Evaluations and Lessons Learned
ICST '09: Proceedings of the 2009 International Conference on Software Testing Verification and Validation

Measurement and detection of redundancy in test suites attempt to achieve test minimization which in turn can help reduce test maintenance costs, and to also ensure the integrity of test cases. Test suite reduction based on coverage information has been ...
Reverse Engineering of Test Cases for Selective Regression Testing
CSMR '04: Proceedings of the Eighth Euromicro Working Conference on Software Maintenance and Reengineering (CSMR'04)

A problem arises in the maintenance oflarge systems when the links between the specificationbased test cases and the code components they test arelost. It is no longer possible to perform selectiveregression testing because it is not known which ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE-SEIP '18: Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice

May 2018

336 pages

ISBN:9781450356596

DOI:10.1145/3183519

Conference Chairs:
Frances Paulisch
Siemens Healthineers, Germany
,
Jan Bosch
Chalmers University of Technology

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 May 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Norwegian Research Council

Conference

ICSE '18

Sponsor:

SIGSOFT
IEEE-CS

ICSE '18: 40th International Conference on Software Engineering

May 27 - June 3, 2018

Gothenburg, Sweden

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
183
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)3

Reflects downloads up to 19 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Douglas Ferrari Mendonça WAssunção WVergilio S(2024)Feature-oriented Test Case Prioritization Strategies: An Evaluation for Highly Configurable SystemsProceedings of the 28th ACM International Systems and Software Product Line Conference10.1145/3646548.3672592(72-83)Online publication date: 2-Sep-2024
https://dl.acm.org/doi/10.1145/3646548.3672592
Mendonça WAssunção WVergilio S(2024)Feature-oriented test case selection and prioritization during the evolution of highly-configurable systemsJournal of Systems and Software10.1016/j.jss.2024.112157217(112157)Online publication date: Nov-2024
https://doi.org/10.1016/j.jss.2024.112157
Singh MChauhan NPopli R(2024)Test case reduction and SWOA optimization for distributed agile software development using regression testingMultimedia Tools and Applications10.1007/s11042-024-19148-1Online publication date: 23-Apr-2024
https://doi.org/10.1007/s11042-024-19148-1
Mendonça WAssunção WVergilio S(2023)Feature-oriented Test Case Selection during Evolution of Highly-Configurable SystemsProceedings of the 27th ACM International Systems and Software Product Line Conference - Volume A10.1145/3579027.3608979(76-86)Online publication date: 28-Aug-2023
https://dl.acm.org/doi/10.1145/3579027.3608979
Bagies TLe WSheaffer J Jannesari A(2023)Reducing branch divergence to speed up parallel execution of unit testing on GPUsThe Journal of Supercomputing10.1007/s11227-023-05375-079:16(18340-18374)Online publication date: 13-May-2023
https://dl.acm.org/doi/10.1007/s11227-023-05375-0
Marijan D(2023)Comparative study of machine learning test case prioritization for continuous integration testingSoftware Quality Journal10.1007/s11219-023-09646-031:4(1415-1438)Online publication date: 28-Jul-2023
https://dl.acm.org/doi/10.1007/s11219-023-09646-0
Chang ZLi MWang JWang QLi SRoychoudhury ACadar CKim M(2022)Putting them under microscope: a fine-grained approach for detecting redundant test cases in natural languageProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3540250.3549089(1161-1172)Online publication date: 7-Nov-2022
https://dl.acm.org/doi/10.1145/3540250.3549089
Marijan DSen S(2022)Industry–Academia Research Collaboration and Knowledge Co-creation: Patterns and Anti-patternsACM Transactions on Software Engineering and Methodology10.1145/349451931:3(1-52)Online publication date: 7-Mar-2022
https://dl.acm.org/doi/10.1145/3494519
Rosero RGómez OVilla EAguilar RPardo C(2022)Software Regression Testing in Industrial Settings: Preliminary Findings from a Literature ReviewTrends in Artificial Intelligence and Computer Engineering10.1007/978-3-030-96147-3_18(227-237)Online publication date: 10-Feb-2022
https://doi.org/10.1007/978-3-030-96147-3_18
Elsner DHauer FPretschner AReimer SCadar CZhang X(2021)Empirically evaluating readily available information for regression test optimization in continuous integrationProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis10.1145/3460319.3464834(491-504)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3460319.3464834
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents