research-article

Open access

FuzzTastic: a fine-grained, fuzzer-agnostic coverage analyzer

Authors:

Thomas Hutzelmann,

Sebastian Banescu,

Alexander Pretschner,

Marcel BöhmeAuthors Info & Claims

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

Pages 75 - 79

https://doi.org/10.1145/3510454.3516847

Published: 19 October 2022 Publication History

Abstract

Performing sound and fair fuzzer evaluations can be challenging, not only because of the randomness involved in fuzzing, but also due to the large number of fuzz tests generated. Existing evaluations use code coverage as a proxy measure for fuzzing effectiveness. Yet, instead of considering coverage of all generated fuzz inputs, they only consider the inputs stored in the fuzzer queue. However, as we show in this paper, this approach can lead to biased assessments due to path collisions. Therefore, we developed FuzzTastic, a fuzzeragnostic coverage analyzer that allows practitioners and researchers to perform uniform fuzzer evaluations that are not affected by such collisions. In addition, its time-stamped coverage-probing approach enables frequency-based coverage analysis to identify barely tested source code and to visualize fuzzing progress over time and across code. To foster further studies in this field, we make FuzzTastic, together with a benchmark dataset worth ~12 CPU-years of fuzzing, publicly available; the demo video can be found at https://youtu.be/Lm-eBx0aePA.

References

[1]

[n. d.]. American Fuzzy Lop (AFL). https://lcamtuf.coredump.cx/afl/. Accessed: 2021-09-20.

[2]

[n. d.]. Clang Documentation: Source-based Code Coverage. https://clang.llvm.org/docs/SourceBasedCodeCoverage.html. Accessed: 2021-09-20.

[3]

[n. d.]. Honggfuzz: Security-oriented Software Fuzzer. https://honggfuzz.dev/. Accessed: 2021-10-06.

[4]

Marcel Böhme and Brandon Falk. 2020. Fuzzing: On the Exponential Cost of Vulnerability Discovery. In Proceedings of the Joint Meeting European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 713--724.

Digital Library

[5]

Marcel Böhme, Van-Thuan Pham, and Abhik Roychoudhury. 2019. Coverage-Based Greybox Fuzzing as Markov Chain. IEEE Transactions on Software Engineering 45, 5 (may 2019), 489--506.

[6]

Hudson Borges and Marco Tulio Valente. 2018. What's in a Github Star? Understanding Repository Starring Practices in a Social Coding Platform. Journal of Systems and Software 146 (2018), 112--129.

[7]

Jaeseung Choi, Joonun Jang, Choongwoo Han, and Sang Kil Cha. 2019. Grey-Box Concolic Testing on Binary Code. In Proceedings of the International Conference on Software Engineering, Vol. 2019-May. IEEE, 736--747.

Digital Library

[8]

Andrea Fioraldi, Dominik Maier, Heiko Eißfeldt, and Marc Heuse. 2020. AFL++: Combining Incremental Steps of Fuzzing Research. In Proceedings of the USENIX Workshop on Offensive Technologies.

[9]

Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. 2018. CollAFL: Path Sensitive Fuzzing. In Proceedings of the IEEE Symposium on Security and Privacy, Vol. 2018-May. IEEE, 679--696.

[10]

Ahmad Hazimeh, Adrian Herrera, and Mathias Payer. 2021. Magma: A Ground-Truth Fuzzing Benchmark. Proceedings of the International Conference on Measurement and Modeling of Computer Systems 4, 3 (2021), 81--82.

Digital Library

[11]

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. 2018. Evaluating Fuzz Testing. In Proceedings of the Conference on Computer and Communications Security. ACM, New York, NY, USA, 2123--2138.

Digital Library

[12]

Caroline Lemieux and Koushik Sen. 2018. FairFuzz: A Targeted Mutation Strategy for Increasing Greybox Fuzz Testing Coverage. In Proceedings of the International Conference on Automated Software Engineering. ACM, New York, NY, USA, 475--485.

Digital Library

[13]

Chenyang Lyu, Shouling Ji, Chao Zhang, Yuwei Li, Wei Han Lee, Yu Song, and Raheem Beyah. 2019. MOPT: Optimized Mutation Scheduling for Fuzzers. In Proceedings of the USENIX Security Symposium.

[14]

Valentin J. M. Manès, HyungSeok Han, Choongwoo Han, sang kil Cha, Manuel Egele, Edward J. Schwartz, and Maverick Woo. 2019. The Art, Science, and Engineering of Fuzzing: A Survey. IEEE Transactions on Software Engineering (2019), 2312--2331.

[15]

Jonathan Metzman, László Szekeres, Laurent Simon, Read Sprabery, and Abhishek Arya. 2021. FuzzBench: An Open Fuzzer Benchmarking Platform and Service. In Proceedings of the Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1393--1403.

Digital Library

[16]

Barton P. Miller, Louis Fredriksen, and Bryan So. 1990. An Empirical Study of the Reliability of Unix Utilities. Commun. ACM 33, 12 (dec 1990), 32--44.

Digital Library

[17]

Mathias Payer. 2019. The Fuzzing Hype-train: How Random Testing Triggers Thousands of Crashes. IEEE Security and Privacy Magazine 17, 1 (jan 2019), 78--82.

[18]

Van-Thuan Pham, Marcel Böhme, Andrew Edward Santosa, Alexandru Razvan Caciulescu, and Abhik Roychoudhury. 2020. Smart Greybox Fuzzing. IEEE Transactions on Software Engineering (2020), 1980--1997.

[19]

Kostya Serebryany. 2017. OSS-Fuzz: Google's Continuous Fuzzing Service for Open-Source Software. USENIX Association, Vancouver, BC.

[20]

Jinghan Wang, Yue Duan, Wei Song, Heng Yin, and Chengyu Song. 2019. Be Sensitive and Collaborative: Analyzing Impact of Coverage Metrics in Grey-box Fuzzing. Proceedings of the International Symposium on Research in Attacks, Intrusions and Defenses (2019), 1--15.

[21]

Elaine J. Weyuker. 1986. Axiomatizing Software Test Data Adequacy. IEEE Transactions on Software Engineering SE-12, 12 (dec 1986), 1128--1138.

Digital Library

Cited By

Sarafov VMarkvica DBerlakovich FBernad MBrunthaler SBöhme MNoller YSzekeres L(2024)Understanding and Improving Coverage Tracking with AFL++ (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685537(80-89)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685537
Schloegel MBars NSchiller NBernhard LScharnowski TCrump AAle-Ebrahim ABissantz NMuench MHolz T(2024)SoK: Prudent Evaluation Practices for Fuzzing2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00137(1974-1993)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00137
Böhme MSzekeres LMetzman JDwyer MDamian DZeller A(2022)On the reliability of coverage-based fuzzer benchmarkingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510230(1621-1633)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510230

Index Terms

FuzzTastic: a fine-grained, fuzzer-agnostic coverage analyzer
1. Security and privacy
  1. Software and application security
    1. Software security engineering

Recommendations

FuzzBench: an open fuzzer benchmarking platform and service
ESEC/FSE 2021: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Fuzzing is a key tool used to reduce bugs in production software. At Google, fuzzing has uncovered tens of thousands of bugs. Fuzzing is also a popular subject of academic research. In 2020 alone, over 120 papers were published on the topic of improving,...
InstruGuard: find and fix instrumentation errors for coverage-based greybox fuzzing
ASE '21: Proceedings of the 36th IEEE/ACM International Conference on Automated Software Engineering

As one of the most successful methods at vulnerability discovery, coverage-based greybox fuzzing relies on the lightweight compile-time instrumentation to achieve the finegrained coverage feedback of the target program. Researchers improve it by ...
FUDGE: fuzz driver generation at scale
ESEC/FSE 2019: Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering

At Google we have found tens of thousands of security and robustness bugs by fuzzing C and C++ libraries. To fuzz a library, a fuzzer requires a fuzz driver—which exercises some library code—to which it can pass inputs. Unfortunately, writing fuzz ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICSE '22: Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings

May 2022

394 pages

ISBN:9781450392235

DOI:10.1145/3510454

General Chair:
Matthew B Dwyer
University of Virginia

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGSOFT: ACM Special Interest Group on Software Engineering

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2022

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICSE '22

Sponsor:

SIGSOFT

ICSE '22: 44th International Conference on Software Engineering

May 21 - 29, 2022

Pennsylvania, Pittsburgh

Acceptance Rates

Overall Acceptance Rate 276 of 1,856 submissions, 15%

Upcoming Conference

ICSE 2025

2025 IEEE/ACM 46th International Conference on Software Engineering

April 26 - May 3, 2025

Ottawa , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
312
Total Downloads

Downloads (Last 12 months)143
Downloads (Last 6 weeks)16

Reflects downloads up to 19 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Sarafov VMarkvica DBerlakovich FBernad MBrunthaler SBöhme MNoller YSzekeres L(2024)Understanding and Improving Coverage Tracking with AFL++ (Registered Report)Proceedings of the 3rd ACM International Fuzzing Workshop10.1145/3678722.3685537(80-89)Online publication date: 13-Sep-2024
https://dl.acm.org/doi/10.1145/3678722.3685537
Schloegel MBars NSchiller NBernhard LScharnowski TCrump AAle-Ebrahim ABissantz NMuench MHolz T(2024)SoK: Prudent Evaluation Practices for Fuzzing2024 IEEE Symposium on Security and Privacy (SP)10.1109/SP54263.2024.00137(1974-1993)Online publication date: 19-May-2024
https://doi.org/10.1109/SP54263.2024.00137
Böhme MSzekeres LMetzman JDwyer MDamian DZeller A(2022)On the reliability of coverage-based fuzzer benchmarkingProceedings of the 44th International Conference on Software Engineering10.1145/3510003.3510230(1621-1633)Online publication date: 21-May-2022
https://dl.acm.org/doi/10.1145/3510003.3510230

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents