research-article

Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms

Authors:

Goiuria Sagardui,

Leire EtxeberriaAuthors Info & Claims

GECCO '16: Proceedings of the Genetic and Evolutionary Computation Conference 2016

Pages 1053 - 1060

https://doi.org/10.1145/2908812.2908871

Published: 20 July 2016 Publication History

Abstract

Cyber-Physical Systems (CPSs) can be found in many sectors (e.g., automotive and aerospace). These systems are usually configurable to give solutions based on different needs. The variability of these systems is large, which implies they can be set into millions of configurations. As a result, different testing processes are needed to efficiently test these systems: the appropriate configurations must be selected and relevant test cases for each configuration must be chosen as well as prioritized. Prioritizing the order in which the test cases are executed reduces the time for detecting faults in these kinds of systems. However, the test suite size is often large and exploring all the possible test case orders is infeasible. Search algorithms can help find optimal solutions from a large solution space. This paper presents an approach based on weight-based search algorithms for prioritizing the test cases for configurable CPSs. We empirically evaluate the performance of the following algorithms with two case studies: Weight-Based Genetic Algorithms, Random Weighted Genetic Algorithms, Greedy, Alternating Variable Method and Random Search (RS). Our results suggest that all the search algorithms outperform RS, which is taken as a baseline. Local search algorithms have shown better performance than global search algorithms.

References

[1]

A. Arcuri and L. Briand. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Software Engineering (ICSE), 2011 33rd International Conference on, pages 1--10. IEEE, 2011.

Digital Library

[2]

A. Arcuri, M. Z. Iqbal, and L. Briand. Black-box system testing of real-time embedded systems using random and search-based testing. In Proceedings of the 22Nd IFIP WG 6.1 International Conference on Testing Software and Systems, ICTSS'10, pages 95--110, Berlin, Heidelberg, 2010. Springer-Verlag.

Digital Library

[3]

A. Arrieta, G. Sagardui, and L. Etxeberria. Test control algorithms for the validation of cyber-physical systems product lines. In Proceedings of the 19th International Conference on Software Product Line, SPLC '15, pages 273--282, New York, NY, USA, 2015. ACM.

Digital Library

[4]

C. Catal and D. Mishra. Test case prioritization: A systematic mapping study. Software Quality Journal, 21(3):445--478, Sept. 2013.

Digital Library

[5]

P. Derler, E. A. Lee, and A. Sangiovanni-Vincentelli. Modeling cyber-physical systems. Proceedings of the IEEE (special issue on CPS), 100(1):13 -- 28, January 2011.

[6]

D. Greer and G. Ruhe. Software release planning: an evolutionary and iterative approach. Information and Software Technology, 46(4):243 -- 253, 2004.

[7]

M. Harman, S. A. Mansouri, and Y. Zhang. Search-based software engineering: Trends, techniques and applications. ACM Comput. Surv., 45(1):11:1--11:61, Dec. 2012.

Digital Library

[8]

R. Kuhn, R. Kacker, Y. Lei, and J. Hunter. Combinatorial software testing. Computer, 42:94--96, 2009.

Digital Library

[9]

R. E. Lopez-Herrejon, L. Linsbauer, and A. Egyed. A systematic mapping study of search-based software engineering for software product lines. Information and Software Technology, 61:33 -- 51, 2015.

Digital Library

[10]

A. G. Malishevsky, G. Rothermel, and S. Elbaum. Modeling the cost-benefits tradeoffs for regression testing techniques. In In Proceedings of the International Conference on Software Maintenance, pages 204--213, 2002.

Digital Library

[11]

R. Matinnejad. Simulink fault patterns. Technical report, 2015.

[12]

A. B. Sánchez, S. Segura, and A. Ruiz-Cortés. A comparison of test case prioritization criteria for software product lines. In IEEE International Conference on Software Testing, Verification, and Validation, pages 41--50, 2014.

Digital Library

[13]

K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. Timeaware test suite prioritization. In Proceedings of the 2006 International Symposium on Software Testing and Analysis, ISSTA '06, pages 1--12, New York, NY, USA, 2006. ACM.

Digital Library

[14]

S. Wang, S. Ali, and A. Gotlieb. Minimizing test suites in software product lines using weight-based genetic algorithms. In Proceedings of the 2013 Genetic and Evolutionary Computation Conference, pages 1493 -- 1500, Amsterdam, Netherlands, 2013.

Digital Library

[15]

S. Wang, S. Ali, and A. Gotlieb. Cost-effective test suite minimization in product lines using search techniques. Journal of Systems and Software, 103(0):370 -- 391, 2015.

Digital Library

[16]

S. Wang, D. Buchmann, S. Ali, A. Gotlieb, D. Pradhan, and M. Liaaen. Multi-objective test prioritization in software product line testing: An industrial case study. In Proceedings of the 18th International Software Product Line Conference - Volume 1, SPLC '14, pages 32--41, New York, NY, USA, 2014. ACM.

Digital Library

[17]

S. Yoo and M. Harman. Regression testing minimization, selection and prioritization: a survey. Software Testing, Verification and Reliability, 22(2):67--120, 2012.

[18]

J. Zander-Nowicka. Reactive testing and test control of hybrid embedded software. In Proceedings of the 5th Workshop on System Testing and Validation, pages 45--62, 2007.

[19]

J. Zander-Nowicka. Model-based Testing of Real-Time Embedded Systems in the Automotive Domain. PhD thesis, Technical University Berlin, 2008.

[20]

K. Zhai, B. Jiang, and W. K. Chan. Prioritizing test cases for regression testing of location-based services: Metrics, techniques, and case study. IEEE Trans. Serv. Comput., 7(1):54--67, Jan. 2014.

Digital Library

[21]

L. Zhang, S.-S. Hou, C. Guo, T. Xie, and H. Mei. Time-aware test-case prioritization using integer linear programming. In Proceedings of the Eighteenth International Symposium on Software Testing and Analysis, ISSTA '09, pages 213--224, New York, NY, USA, 2009. ACM.

Digital Library

Cited By

Formica FPetrunti NBruck LPantelic VLawford MMenghi CChandra SBlincoe KTonella P(2023)Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial SimulatorProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613894(1949-1960)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613894
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
May R(2022)Security and Configurable Storage Systems in Industry 4.0 Environments: A Systematic Literature StudyOpen Conference Proceedings10.52825/ocp.v2i.1492(151-156)Online publication date: 15-Dec-2022
https://doi.org/10.52825/ocp.v2i.149
Show More Cited By

Index Terms

Test Case Prioritization of Configurable Cyber-Physical Systems with Weight-Based Search Algorithms
1. Software and its engineering
  1. Software creation and management
    1. Search-based software engineering
    2. Software verification and validation

Recommendations

Test case prioritization for acceptance testing of cyber physical systems: a multi-objective search-based approach
ISSTA 2018: Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis

Acceptance testing validates that a system meets its requirements and determines whether it can be sufficiently trusted and put into operation. For cyber physical systems (CPS), acceptance testing is a hardware-in-the-loop process conducted in a (near-)...
A Simulation Study on Some Search Algorithms for Regression Test Case Prioritization
QSIC '10: Proceedings of the 2010 10th International Conference on Quality Software

Test case prioritization is an approach aiming at increasing the rate of faults detection during the testing phase, by reordering test case execution. Many techniques for regression test case prioritization have been proposed. In this paper, we perform ...
Automatic generation of test system instances for configurable cyber-physical systems

Cyber-physical systems (CPSs) are ubiquitous systems that integrate digital technologies with physical processes. These systems are becoming configurable to respond to the different needs that users demand. As a consequence, their variability is ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

GECCO '16: Proceedings of the Genetic and Evolutionary Computation Conference 2016

July 2016

1196 pages

ISBN:9781450342063

DOI:10.1145/2908812

Editor:
Tobias Friedrich
Hasso Plattner Institute
,
General Chair:
Frank Neumann
University of Adelaide
,
Program Chair:
Andrew M. Sutton
Hasso Plattner Institute

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGEVO: ACM Special Interest Group on Genetic and Evolutionary Computation

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Eusko Jaurlaritza

Conference

GECCO '16

Sponsor:

SIGEVO

GECCO '16: Genetic and Evolutionary Computation Conference

July 20 - 24, 2016

Colorado, Denver, USA

Acceptance Rates

GECCO '16 Paper Acceptance Rate 137 of 381 submissions, 36%;

Overall Acceptance Rate 1,669 of 4,410 submissions, 38%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
290
Total Downloads

Downloads (Last 12 months)16
Downloads (Last 6 weeks)2

Reflects downloads up to 24 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Formica FPetrunti NBruck LPantelic VLawford MMenghi CChandra SBlincoe KTonella P(2023)Test Case Generation for Drivability Requirements of an Automotive Cruise Controller: An Experience with an Industrial SimulatorProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering10.1145/3611643.3613894(1949-1960)Online publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1145/3611643.3613894
Arrieta AValle PAgirre JSagardui G(2023)Some Seeds Are Strong: Seeding Strategies for Search-based Test Case SelectionACM Transactions on Software Engineering and Methodology10.1145/353218232:1(1-47)Online publication date: 13-Feb-2023
https://dl.acm.org/doi/10.1145/3532182
May R(2022)Security and Configurable Storage Systems in Industry 4.0 Environments: A Systematic Literature StudyOpen Conference Proceedings10.52825/ocp.v2i.1492(151-156)Online publication date: 15-Dec-2022
https://doi.org/10.52825/ocp.v2i.149
Vaagensmith BSingh VIvans RMarino DWickramasinghe CLehmer JPhillips TRieger CManic M(2021)Review of Design Elements within Power Infrastructure Cyber–Physical Test Beds as Threat Analysis EnvironmentsEnergies10.3390/en1405140914:5(1409)Online publication date: 4-Mar-2021
https://doi.org/10.3390/en14051409
Singhal SJatana NSuri BMisra SFernandez-Sanz L(2021)Systematic Literature Review on Test Case Selection and Prioritization: A Tertiary StudyApplied Sciences10.3390/app11241212111:24(12121)Online publication date: 20-Dec-2021
https://doi.org/10.3390/app112412121
Ali SHafeez YHussain SYang SJamal M(2021) Requirement prioritization framework using case‐based reasoning: A mining‐based approach Expert Systems10.1111/exsy.1277038:8Online publication date: 19-Jul-2021
https://doi.org/10.1111/exsy.12770
Markiegi UArrieta AEtxeberria LSagardui G(2021)Dynamic test prioritization of product lines: An application on configurable simulation modelsSoftware Quality Journal10.1007/s11219-021-09571-0Online publication date: 20-Oct-2021
https://doi.org/10.1007/s11219-021-09571-0
Sadri‐Moshkenani ZBradley JRothermel G(2021)Survey on test case generation, selection and prioritization for cyber‐physical systemsSoftware Testing, Verification and Reliability10.1002/stvr.179432:1Online publication date: 15-Sep-2021
https://doi.org/10.1002/stvr.1794
Shin KLim D(2020)Model-Based Test Case Prioritization Using an Alternating Variable Method for Regression Testing of a UML-Based ModelApplied Sciences10.3390/app1021753710:21(7537)Online publication date: 26-Oct-2020
https://doi.org/10.3390/app10217537
Arrieta AAgirre JSagardui GCoello Coello C(2020)Seeding strategies for multi-objective test case selectionProceedings of the 2020 Genetic and Evolutionary Computation Conference10.1145/3377930.3389810(1222-1231)Online publication date: 25-Jun-2020
https://dl.acm.org/doi/10.1145/3377930.3389810
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents