research-article

Empirical evaluation of reliability improvement in an evolving software product line

Authors:

Sandeep Krishnan,

Katerina Goševa-PopstojanovaAuthors Info & Claims

MSR '11: Proceedings of the 8th Working Conference on Mining Software Repositories

Pages 103 - 112

https://doi.org/10.1145/1985441.1985458

Published: 21 May 2011 Publication History

Abstract

Reliability is important to software product-line developers since many product lines require reliable operation. It is typically assumed that as a software product line matures, its reliability improves. Since post-deployment failures impact reliability, we study this claim on an open-source software product line, Eclipse. We investigate the failure trend of common components (reused across all products), highreuse variation components (reused in five or six products) and low-reuse variation components (reused in one or two products) as Eclipse evolves. We also study how much the common and variation components change over time both in terms of addition of new files and modification of existing files. Quantitative results from mining and analysis of the Eclipse bug and release repositories show that as the product line evolves, fewer serious failures occur in components implementing commonality, and that these components also exhibit less change over time. These results were roughly as expected. However, contrary to expectation, components implementing variations, even when reused in five or more products, continue to evolve fairly rapidly. Perhaps as a result, the number of severe failures in variation components shows no uniform pattern of decrease over time. The paper describes and discusses this and related results.

References

[1]

Cvschangelogbuilder, tool for generating cvs log reports. http://cvschangelogb.sourceforge.net/.

[2]

Eclipse bugzilla wiki homepage. https://bugs.eclipse.org/bugs/.

[3]

Eclipse bugzilla wiki homepage. http://wiki.eclipse.org/Eclipse/Bug_Tracking.

[4]

Software engineering institute, software product lines. http://www.sei.cmu.edu/productlines/.

[5]

Software product line hall of fame. http://www.splc.net/fame.html.

[6]

A. Avizienis, J. claude Laprie, and B. Randell. Fundamental concepts of dependability, 2001.

[7]

J. A. Børretzen and R. Conradi. Results and experiences from an empirical study of fault reports in industrial projects. In PROFES 2006. LNCS, pages 389--394. Springer, 2006.

Digital Library

[8]

C. Catal and B. Diri. A systematic review of software fault prediction studies. Expert Systems with Applications, 36(4):7346--7354, 2009.

Digital Library

[9]

G. Chastek, J. McGregor, and L. Northrop. Observations from viewing eclipse as a product line. In Proceedings on the Third International Workshop on Open Source Software and Product Lines, pages 1--6, 2007.

[10]

N. E. Fenton and N. Ohlsson. Quantitative analysis of faults and failures in a complex software system. IEEE Trans. on Software Engineering, 26:797--814, 2000.

Digital Library

[11]

H. Gomaa. Designing Software Product Lines with UML: From Use Cases to Pattern-Based Software Architectures. Addison Wesley Longman Publishing Co., Inc., Redwood City, CA, USA, 2004.

Digital Library

[12]

P. J. Guo, T. Zimmermann, N. Nagappan, and B. Murphy. Characterizing and predicting which bugs get fixed: an empirical study of Microsoft Windows. In Proc. of the 32nd ACM/IEEE International Conference on Software Engineering - Volume 1, ICSE'10, pages 495--504, New York, NY, USA, 2010. ACM.

Digital Library

[13]

M. Hamill and K. Goševa-Popstojanova. Common trends in software fault and failure data. IEEE Trans. Softw. Eng., 35:484--496, July 2009.

Digital Library

[14]

Y. Jiang, B. Cukic, and T. Menzies. Can data transformation help in the detection of fault-prone modules? In Proc. of the 2008 workshop on Defects in large software systems, DEFECTS '08, pages 16--20, New York, NY, USA, 2008. ACM.

Digital Library

[15]

R. R. Lutz and I. C. Mikulski. Empirical analysis of safety-critical anomalies during operations. IEEE Transactions on Software Engineering, 30:172--180, 2004.

Digital Library

[16]

R. R. Lutz, D. M. Weiss, S. Krishnan, and J. Yang. Software product line engineering for long-lived, sustainable systems. In J. Bosch and J. Lee, editors, SPLC, volume 6287 of Lecture Notes in Computer Science, pages 430--434. Springer, 2010.

Digital Library

[17]

T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener. Defect prediction from static code features: current results, limitations, new approaches. Automated Software Engg., 17:375--407, December 2010.

Digital Library

[18]

A. Mockus, R. T. Fielding, and J. Herbsleb. A case study of open source software development: the apache server. In Proceedings of the 22nd International Conference on Software Engineering (ICSE 2000), pages 263--272. ACM Press, 2000.

Digital Library

[19]

P. Mohagheghi and R. Conradi. An empirical investigation of software reuse benefits in a large telecom product. ACM Transactions on Software Engineering and Methodology, 17:13:1--13:31, June 2008.

Digital Library

[20]

P. Mohagheghi, R. Conradi, O. M. Killi, and H. Schwarz. An empirical study of software reuse vs. defect-density and stability. In Proceedings of the 26th International Conference on Software Engineering, ICSE '04, pages 282--292, Washington, DC, USA, 2004. IEEE Computer Society.

Digital Library

[21]

N. Nagappan, T. Ball, and A. Zeller. Mining metrics to predict component failures. In Proceedings of the 28th international conference on Software engineering, ICSE '06, pages 452--461, New York, NY, USA, 2006.

Digital Library

[22]

N. Nagappan, A. Zeller, T. Zimmermann, K. Herzig, and B. Murphy. Change bursts as defect predictors. In ISSRE, pages 309--318, 2010.

Digital Library

[23]

J. W. Paulson, G. Succi, and A. Eberlein. An empirical study of open-source and closed-source software products. IEEE Transactions on Software Engineering, 30:246--256, 2004.

Digital Library

[24]

K. Pohl, G. Böckle, and F. J. v. d. Linden. Software Product Line Engineering: Foundations, Principles and Techniques. Springer-Verlag New York, Inc., Secaucus, NJ, USA, 2005.

Digital Library

[25]

C. Rahmani, A. Azadmanesh, and L. Najjar. A comparative analysis of open source software reliability. Journal of Software, 5:1384--1394, December 2010.

[26]

D. M. Weiss and C. T. R. Lai. Software product-line engineering: a family-based software development process. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA, 1999.

Digital Library

[27]

T. Zimmermann, N. Nagappan, and A. Zeller. Predicting Bugs from History, chapter Predicting Bugs from History, pages 69--88. Springer, February 2008.

[28]

T. Zimmermann, R. Premraj, and A. Zeller. Predicting defects for eclipse. In Proceedings of the Third International Workshop on Predictor Models in Software Engineering, May 2007.

Digital Library

Cited By

Vranković AGalinac Grbac TCar Ž(2018)Software structure evolution and relation to subgraph defectivenessIET Software10.1049/iet-sen.2018.5060Online publication date: 11-Dec-2018
https://doi.org/10.1049/iet-sen.2018.5060
Joshi SDeshpande BPunnekkat SGorthi RSarkar SMedvidovic NKulkarni VKumar AJoshi PInverardi PSureka ASharma R(2017)Do Software Reliability Prediction Models Meet Industrial Perceptions?Proceedings of the 10th Innovations in Software Engineering Conference10.1145/3021460.3021467(66-73)Online publication date: 5-Feb-2017
https://dl.acm.org/doi/10.1145/3021460.3021467
Echeverría JPérez FAbellanas APanach JCetina CPastor ÓGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Evaluating Bug-Fixing in Software Product LinesProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962635(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962635
Show More Cited By

Index Terms

Empirical evaluation of reliability improvement in an evolving software product line
1. General and reference
  1. Cross-computing tools and techniques
    1. Metrics

Recommendations

Toward Compositional Software Product Lines

Software product lines (SPLs) present the first successful approach to intraorganizational reuse. However, research shows that successful product lines suffer over time from increasing dependencies between the software assets that make up the product ...
Integrating Software Product Lines: A Study of Reuse versus Stability
COMPSAC '12: Proceedings of the 2012 IEEE 36th Annual Computer Software and Applications Conference

To achieve large-scale reuse and accelerate time-to-market, integration of multiple software product lines (SPLs) is becoming a trend. The integration of independently-developed SPLs enables the derivation of new products on demand in a particular ...
A concrete product derivation in software product line engineering: a practical approach

Software Product Lines enable the development of a perfect family of products by reusing shared assets in a systematic manner. Product derivation is a critical activity in software product line engineering and one of the most pressing issues that a ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MSR '11: Proceedings of the 8th Working Conference on Mining Software Repositories

May 2011

260 pages

ISBN:9781450305747

DOI:10.1145/1985441

General Chair:
Arie van Deursen
Delft University of Technology, The Netherlands
,
Program Chairs:
Tao Xie
North Carolina State University, USA
,
Thomas Zimmermann
Microsoft Research, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 May 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE11

Sponsor:

SIGSOFT

ICSE11: International Conference on Software Engineering

May 21 - 22, 2011

HI, Waikiki, Honolulu, USA

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
308
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Vranković AGalinac Grbac TCar Ž(2018)Software structure evolution and relation to subgraph defectivenessIET Software10.1049/iet-sen.2018.5060Online publication date: 11-Dec-2018
https://doi.org/10.1049/iet-sen.2018.5060
Joshi SDeshpande BPunnekkat SGorthi RSarkar SMedvidovic NKulkarni VKumar AJoshi PInverardi PSureka ASharma R(2017)Do Software Reliability Prediction Models Meet Industrial Perceptions?Proceedings of the 10th Innovations in Software Engineering Conference10.1145/3021460.3021467(66-73)Online publication date: 5-Feb-2017
https://dl.acm.org/doi/10.1145/3021460.3021467
Echeverría JPérez FAbellanas APanach JCetina CPastor ÓGenero MJedlitschka AJørgensen MScanniello GSampath SCaivano DPort D(2016)Evaluating Bug-Fixing in Software Product LinesProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement10.1145/2961111.2962635(1-6)Online publication date: 8-Sep-2016
https://dl.acm.org/doi/10.1145/2961111.2962635
Al-Hajjaji MBenduhn FThüm TLeich TSaake G(2016)Mutation Operators for Preprocessor-Based VariabilityProceedings of the 10th International Workshop on Variability Modelling of Software-Intensive Systems10.1145/2866614.2866626(81-88)Online publication date: 27-Jan-2016
https://dl.acm.org/doi/10.1145/2866614.2866626
Devine TGoseva-Popstojanova KKrishnan SLutz R(2016)Assessment and cross-product prediction of software product line qualityAutomated Software Engineering10.1007/s10515-014-0160-423:2(253-302)Online publication date: 1-Jun-2016
https://dl.acm.org/doi/10.1007/s10515-014-0160-4
Petrić JGrbac TShepperd MHall TMyrtveit I(2014)Software structure evolution and relation to system defectivenessProceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering10.1145/2601248.2601287(1-10)Online publication date: 13-May-2014
https://dl.acm.org/doi/10.1145/2601248.2601287
JUNG WLEE EWU C(2012)A Survey on Mining Software RepositoriesIEICE Transactions on Information and Systems10.1587/transinf.E95.D.1384E95.D:5(1384-1406)Online publication date: 2012
https://doi.org/10.1587/transinf.E95.D.1384
Devine TGoseva-Popstajanova KKrishnan SLutz RLi J(2012)An Empirical Study of Pre-release Software Faults in an Industrial Product LineProceedings of the 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation10.1109/ICST.2012.98(181-190)Online publication date: 17-Apr-2012
https://dl.acm.org/doi/10.1109/ICST.2012.98
Krishnan SStrasburg CLutz RGoševa-Popstojanova KMenzies T(2011)Are change metrics good predictors for an evolving software product line?Proceedings of the 7th International Conference on Predictive Models in Software Engineering10.1145/2020390.2020397(1-10)Online publication date: 20-Sep-2011
https://dl.acm.org/doi/10.1145/2020390.2020397

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents