Mutation Testing
We intentionally inject bugs into our plugins to verify the test suite actually catches them. This isn't about making plugins fail—it's about proving our tests actually work.
What is Mutation Testing?
Mutation testing systematically injects faults (“mutations”) into source code to verify that the test suite detects them. If a test suite passes despite a fault being present, you've found a place where detection coverage can be strengthened.
Think of it as testing the tests. A comprehensive test suite should catch every realistic fault we can inject.
Campaign: BwsPressure v1.0.0
- Date:
- January 30, 2026
- Target:
- Pressure DSP pipeline
- Mutations completed:
- 2
- Findings:
- 2 identified, resolved
- Campaign:
- Ongoing — additional mutations scheduled
Findings & Resolutions
Finding #1: DC Offset Detection (Resolved)
Mutation: Injected a DC offset into the output buffer to test whether the framework's buffer validation would detect it.
Result: The stress test suite passed without flagging the offset. Buffer validation covered NaN, Inf, and clipping but did not yet include DC offset measurement.
Resolution: DC offset detection was added to the BPTS framework with a −60 dBFS threshold. All plugins were re-validated against the updated suite.
Resolved in BPTS v2.2.0 · February 2026
Finding #2: Sample-Level Discontinuity Detection (In Progress)
Mutation: Injected a sharp gain discontinuity into the output buffer to test whether the framework would detect sample-level anomalies.
Result: The artifact and stress test suites passed without flagging the discontinuity. The framework checked for parameter-induced artifacts but did not yet include sample-by-sample gradient analysis.
Resolution: A discontinuity detector with a 6 dB inter-sample threshold is being integrated to extend coverage to transient-level anomalies across all test categories.
In progress · Targeted for BPTS v2.3.0
Mutation Matrix
| # | Mutation | Test Suite | Status |
|---|---|---|---|
| 1 | DC Offset Injection | Artifact Tests | Resolved |
| 2 | Gain Spike | Artifact Tests | In Progress |
| 3 | Break Oversampling | Frequency Tests | Scheduled |
| 4 | Parameter Range | Characterization | Scheduled |
| 5 | APVTS Sync | GUI Tests | Scheduled |
| 6 | NaN Injection | BOL Gate | Scheduled |
| 7 | Latency Report | Integration Tests | Scheduled |
| 8 | Undo Manager | Integration Tests | Scheduled |
| 9 | Memory Leak | Marathon Tests | Scheduled |
| 10 | RT Violation | Artifact Tests | Scheduled |
Why We Publish Framework Findings
Publishing the engineering process—what we test, what we find, and what we improve—builds trust through evidence, not claims.
These findings are about the test framework, not the plugins themselves. Every plugin passed its full test suite. Mutation testing strengthened the framework by revealing where detection coverage could be deeper.
If you find a bug our tests missed, we genuinely want to know. That's how we improve the framework for every plugin in the catalog.
Case Study: DC Offset Detection (January 2026)
Mutation testing revealed a detection gap in buffer validation—the framework didn't yet measure DC offset. We added the detection within 48 hours, re-validated all plugins, and updated the testing standard. This is why we publish methodology: finding issues and resolving them quickly is the process working as intended.