Marius Smytzek

Generating Test Inputs and Interactions with the Fandango Fuzzer: How to Test Complex Systems Automatically and Exhaustively

Juli 2026 | FSE 2026

Jose Antonio Zamudio Amaya, Alexander Liggesmeyer, Marius Smytzek, Andreas Zeller

Abstract:

Even in the age of AI, generating comprehensive test inputs for software systems remains a challenge, particularly for interactive and reactive systems, which require entire interaction sequences to reach the desired states or to cover missing behavior. While modern fuzzers are effective at generating random inputs that test the robustness of input processors, they still struggle to produce or mutate complex inputs and interactions. LLM-based systems, on the other hand, can generate small example inputs, but fail to systematically explore the space of inputs and interactions as would be required for comprehensive testing.

In this tutorial, we introduce test generation with FANDANGO, a modern specification-based generator of test inputs and interactions. Based on a spec file that defines the structure and properties of the program input, fandango produces inputs that are syntactically and semantically correct, and systematically cover the input space of the program under test. In three interactive sessions, we (1) introduce language-based testing, (2) demonstrate how to use constraints to define desired properties, and then (3) delve into full-fledged protocol testing, producing entire interaction sequences between fandango and the network components under test.

Participants are expected to bring a basic understanding of software testing; knowledge of python is a plus, but not required. At the end of the tutorial, they will (1) be able to specify and test formats for inputs and interactions, (2) produce comprehensive test suites using language-based techniques, and (3) understand the trade-offs and practical applications of language-based testing.

PYURIFY: Purifying Python Tests for Precise Fault Localization

Juli 2026 | FSE 2026

Marius Smytzek, Andreas Zeller

Abstract:

Statistical fault localization (SFL) techniques use test coverage to rank suspicious program elements. However, they often suffer from low precision when tests contain multiple assertions or unrelated code that obscures the relationship between test outcomes and faulty code. Test case purification addresses this problem by transforming complex test cases into simplified, single-assertion variants. We present PYURIFY, a tool that implements test case purification for Python through two steps: first, atomization splits each test with multiple assertions into separate single-assertion tests; second, dynamic program slicing removes code irrelevant to each assertion. This purification process produces cleaner test spectra, enabling SFL techniques to pinpoint faulty code more accurately. PYURIFY provides both a command-line interface and a Python API for seamless integration with existing testing and fault localization workflows. Our evaluation of real Python bugs demonstrates that test case purification consistently improves fault localization accuracy.

Constraint-Driven Fuzzing at Scale with FANDANGO

Juli 2026 | ICST 2026

José Antonio Zamudio Amaya, Marius Smytzek, Alexander Liggesmeyer, Valentin Huber, Andreas Zeller

Abstract:

By combining grammars with semantic constraints, language-based testing enables the generation of complex, valid test inputs. We present FANDANGO, a test generator that replaces symbolic solving with genetic algorithms, allowing constraints to be written directly in Python rather than in a limited function-solver library. Compared to the state-of-the-art, this approach delivers input generation speeds one to three orders of magnitude higher while maintaining near-perfect validity. FANDANGO is easy to use and supports a wide range of input formats. This paper emphasizes the public release: packaging, command-line, and Python APIs, reproducible examples, and early adoption in teaching and industry. A screencast is available at: https://youtu.be/7W9CaHEC1bM

Combining Input Constraints with Execution Goals

Juli 2026 | ICST 2026

Leon Bettscheider, Marius Smytzek, Andreas Zeller

Abstract:

Recent advances in blackbox test generation (fuzzing) allow combining syntax specifications (grammars) with semantic constraints over grammar elements. These constraints not only help in producing valid inputs (“ ⟨ checksum ⟩ should be a Luhn checksum over ⟨ number ⟩ ”) but also allow specifying testing goals (“ payload should be as short as possible”). Such constraints give testers unprecedented control over the generated inputs. However, constraint-based fuzzers like ISLa or Fandango have only been limited to input features and blackbox fuzzing. This paper shows how to extend this concept with execution constraints relating to the program under test. We introduce a set of primitives that check for specific execution features such as coverage of specific locations, memory usage, execution time, and more. Testers can then use these to specify additional testing goals: “I want an input that is (1) valid, (2) as short as possible, and (3) results in a maximum of code coverage.” We implement our approach on top of the Fandango fuzzer and find that its evolutionary algorithm efficiently finds solutions that satisfy input and execution constraints. We demonstrate that combining input and execution constraints enables us to specify and implement approaches such as directed, performance, and coverage-guided fuzzing, effectively covering the entire range of testing goals from whitebox to blackbox fuzzing in a unified framework.

Search-Based Generation of Complex Inputs with FANDANGO

Juli 2026 | SSBSE 2026

Jose Antonio Zamudio Amaya, Marius Smytzek, Andreas Zeller

Abstract:

Generating complex, semantically valid test inputs remains a central challenge in search-based software testing. FANDANGO addresses this challenge by combining context-free grammars with fully specified Python constraints and using evolutionary search to satisfy them. Compared with the symbolic state of the art, FANDANGO achieves speedups of up to three orders of magnitude while maintaining 100% input validity and equal or higher grammar coverage.FANDANGO is under active development: recent work explores whitebox fuzzing and stateful fuzzing through interaction grammars that model multi-turn protocol behavior. With more than 100,000 PyPI downloads, adoption in industry (including Volkswagen and Bosch), and coverage in the science press, FANDANGO has established itself as a practical input generator.

Denoising Fault Localization with Test Line Proximity

Juni 2026 | FSE 2026

Marius Smytzek, Andreas Zeller

Abstract:

When a program fails, statistical fault localization (SFL) provides important debugging hints by identifying the locations whose execution most correlates with failures. However, such correlations can be weakened if a test contains both passing and failing assertions, creating ambiguous and misleading associations. Likewise, if multiple lines correlate with the same strength, SFL provides little guidance to disambiguate between them.

This paper proposes a novel proximity-based weighting scheme for SFL that assigns different weights to locations in the test subject based on temporal proximity to a failure. The more recently a subject line is executed before the test fails, the higher its weight. We operationalize a known heuristic into a lightweight statistical form compatible with existing SFL formulas. Our approach applies to any test, from simple single-line tests (where it preserves SFL behavior), to single-assertion tests with multiple lines (where it benefits from temporal proximity), to complex multi-assertion tests (where it provides the most benefit by distinguishing failing from passing assertions). Once computed, the weights can be integrated into any existing SFL technique.

Our evaluation of proximity-weighted fault localization on 310 real-world programs shows that it consistently outperforms fault localization techniques across all test types. Proximity-weighted fault localization shows per-subject relative improvements of 200%–400%, meaning that, for a typical subject, it provides 3 to 5 times the baseline effectiveness. These improvements represent substantial gains over baseline techniques. Our approach can be integrated into existing fault localization techniques to improve performance, making it a valuable addition to automated debugging.

How do Execution Features Improve Statistical Fault Localization? An Empirical Study

Juni 2026 | ICSME 2026

Marius Smytzek, Andreas Zeller

Abstract:

Automated fault localization helps developers find faults in large code bases. Statistical fault localization (SFL) ranks suspicious lines from pass/fail spectra, but line execution alone misses information like data-flow, values, or branch conditions that explain why a failure occurs.

This study evaluates whether augmenting SFL with execution features improves localization accuracy and developer-oriented inspection effort. We extract execution features with EFDD for all Tests4Py subjects, train per-subject random forests, map importances to source lines, and combine the resulting weights with established SFL formulas. The evaluation measures reference-patch accuracy, line- and function-level effort, robustness, and feasibility using a confounder-adjusted mixed-effects model, corroborated by paired statistical tests and outcome-neutral quality checks.

How Execution Features Relate to Failures: An Empirical Study and Diagnosis Approach

Dezember 2025 | TOSEM

Marius Smytzek, Martin Eberlein, Lars Grunske, Andreas Zeller

Abstract:

Fault localization aims to identify code regions responsible for failures. Traditional techniques primarily correlate statement execution with failures; however, program behavior involves diverse execution features, including variable values, branch conditions, and definition-use pairs, which can provide richer diagnostic insights.

This paper comprehensively investigates execution features for fault understanding, addressing two complementary goals. First, we conduct an empirical study of 310 bugs across 20 projects, analyzing 17 execution features and assessing their correlation with failure outcomes. Our findings suggest that fault localization benefits from a broader range of execution features: (1) Scalar pairs exhibit the strongest correlation with failures; (2) Beyond line executions, def-use pairs and functions executed are key indicators for fault localization; and (3) Combining multiple features enhances effectiveness compared to relying on individual features.

Second, building on these insights, we introduce a debugging approach that learns relevant features from labeled test outcomes. The approach extracts fine-grained execution features and trains a decision tree to differentiate passing and failing runs. The trained model generates fault diagnoses that explain the underlying causes of failures.

Our evaluation demonstrates that the generated diagnoses achieve high predictive accuracy. These interpretable diagnoses empower developers to debug software efficiently by providing deeper insights into failures.

BASHIRI: Learning Failure Oracles from Execution Features

November 2025 | ASE 2025

Marius Smytzek, Martin Eberlein, Tural Mammadov, Lars Grunske, Andreas Zeller

Abstract:

Program fixes must preserve passing tests while fixing failing ones. Validating these properties requires test oracles that distinguish passing from failing runs.We introduce BASHIRI, a tool that learns failure oracles from test suites with labeled outcomes using execution features. BASHIRI leverages execution-feature-driven debugging to collect program execution features and trains interpretable models as testing oracles. Our evaluation shows that BASHIRI predicts test outcomes with 95% accuracy, effectively identifying failing runs. BASHIRI is available as an open-source tool at https://github.com/smythi93/bashiri

jAST: Analyzing and Modifying Java ASTs with Python

Juli 2025 | FSE 2025

Marius Smytzek, Martin Eberlein, Lars Grunske, Andreas Zeller

Abstract:

Analyzing and modifying source code at the Abstract Syntax Tree (AST) level is fundamental to numerous software engineering tasks, including program analysis, instrumentation, and transformation. While Java-specific tools exist, they often operate at the bytecode level or are tightly coupled to the Java ecosystem, limiting their flexibility and accessibility. In this paper, we present jAST, a Python-based tool that generates and manipulates ASTs for Java programs. By leveraging Python's simplicity and extensive ecosystem, jAST enables precise source-level analysis, seamless integration with Python workflows, and support for advanced tasks such as feature extraction for debugging and learning-based failure analysis. jAST is an open-source tool offering researchers, educators, and practitioners an extensible framework for working with Java ASTs.

FANDANGO: Evolving Language-Based Testing

Juni 2025 | ISSTA 2025

José Antonio Zamudio Amaya, Marius Smytzek, Andreas Zeller

Abstract:

Language-based fuzzers leverage formal input specifications (languages) to generate arbitrarily large and diverse sets of valid inputs for a program under test. Modern language-based test generators combine grammars and constraints to satisfy syntactic and semantic input constraints. ISLA, the leading input generator in that space, uses symbolic constraint solving to solve input constraints. Using solvers places ISLA among the most precise fuzzers but also makes it slow.

In this paper, we explore search-based testing as an alternative to symbolic constraint solving. We employ a genetic algorithm that iteratively generates candidate inputs from an input specification, evaluates them against defined constraints, evolving a population of inputs through syntactically valid mutations and retaining those with superior fitness until the semantic input constraints are met. This evolutionary procedure, analogous to natural genetic evolution, leads to progressively improved inputs that cover both semantics and syntax. This change boosts the efficiency of language-based testing: In our experiments, compared to ISLA, our search-based FANDANGO prototype is faster by one to three orders of magnitude without sacrificing precision.

The search-based approach no longer restricts constraints to constraint solvers' (miniature) languages. In FANDANGO, constraints can use the whole Python language and library. This expressiveness gives testers unprecedented flexibility in shaping test inputs. It allows them to state arbitrary goals for test generation: ''Please produce 1,000 valid test inputs where the voltage field follows a Gaussian distribution but never exceeds 20 mV.''

FixKit: A Program Repair Collection for Python

Oktober 2024 | ASE 2024

Marius Smytzek, Martin Eberlein, Kai Werk, Lars Grunske, Andreas Zeller

Abstract:

In recent years, automatic program repair has gained much attention in the research community. Generally, program repair approaches consider a faulty program and a test suite that captures the program's intended behavior. The goal is automatically generating a patch that corrects the fault by identifying the faulty code locations, suggesting a candidate fix, and validating it against the provided tests. However, most existing program repair tools focus on Java or C programs, while Python, one of the most popular programming languages, lacks approaches that work on it.

We present FixKit, a collection of five program repair approaches for Python programs. Moreover, our framework allows for easy integration of new repair approaches and swapping individual components, such as fault localization. Our framework enables researchers to compare and investigate various repair, fault localization effortlessly, and validation approaches on a common set of techniques.

Tests4Py: A Benchmark for System Testing

Juli 2024 | FSE 2024

Marius Smytzek, Martin Eberlein, Batuhan Serçe, Lars Grunske, Andreas Zeller

Abstract:

Benchmarks are among the main drivers of progress in software engineering research. However, many current benchmarks are limited by inadequate system oracles and sparse unit tests. Our Tests4Py benchmark, derived from the BugsInPy benchmark, addresses these limitations. It includes 73 bugs from seven real-world Python applications and six bugs from example programs. Each subject in Tests4Py is equipped with an oracle for verifying functional correctness and supports both system and unit test generation. This allows for comprehensive qualitative studies and extensive evaluations, making Tests4Py a cutting-edge benchmark for research in test generation, debugging, and automatic program repair.

Semantic Debugging

November 2023 | ESEC/FSE 2023

Martin Eberlein, Marius Smytzek, Dominic Steinhöfel, Lars Grunske, Andreas Zeller

Abstract:

Why does my program fail? We present a novel and general technique to automatically determine failure causes and conditions, using logical properties over input elements: “The program fails if and only if int(<length>) > len(<payload>) holds—that is, the given <length> is larger than the <payload> length.” Our AVICENNA prototype uses modern techniques for inferring properties of passing and failing inputs and validating and refining hypotheses by having a constraint solver generate supporting test cases to obtain such diagnoses. As a result, AVICENNA produces crisp and expressive diagnoses even for complex failure conditions, considerably improving over the state of the art with diagnoses close to those of human experts.

From Input to Failure: Explaining Program Behavior via Cause-Effect Chains

Juli 2023 | ICSE 2023

Marius Smytzek

Abstract:

Debugging a fault in a program is an error-prone and resource-intensive process that requires considerable work. My doctoral research aims at supporting developers during this process by integrating test generation as a feedback loop into a novel fault diagnosis to narrow down the causality by validating or disproving suggested hypotheses. I will combine input, output, and state to detect relevant relations for an immersive fault diagnosis. Further, I want to introduce an approach for a targeted test that leverages statistical fault localization to extract oracles based on execution features to identify failing tests.

SFLKit: A Workbench for Statistical Fault Localization

November 2022 | ESEC/FSE 2022

Marius Smytzek, Andreas Zeller

Abstract:

Statistical fault localization aims at detecting execution features that correlate with failures, such as whether individual lines are part of the execution. We introduce SFLKit, an out-of-the-box workbench for statistical fault localization. The framework provides straightforward access to the fundamental concepts of statistical fault localization. It supports five predicate types, four coverage-inspired spectra, like lines, and 44 similarity coefficients, e.g., TARANTULA or OCHIAI, for statistical program analysis.

SFLKit separates the execution of tests from the analysis of the results and is therefore independent of the used testing framework. It leverages program instrumentation to enable the logging of events and derives the predicates and spectra from these logs. This instrumentation allows for introducing multiple programming languages and the extension of new concepts in statistical fault localization. Currently, SFLKit supports the instrumentation of Python programs. It is highly configurable, requiring only the logging of the required events.