Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
target/
.git/
.DS_Store
mutation.db
benchmark-results/
*.log

48 changes: 46 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -141,9 +141,12 @@ When `--sqlite` is used, the `mutate` command prints a `run_id` that you pass to
| `--folder PATH` | `-f` | | Folder containing mutants (alternative to `--sqlite` / `--run-id`). |
| `--timeout SECONDS` | `-t` | `300` | Timeout in seconds for each mutant's test run. |
| `--jobs N` | `-j` | `0` | Number of parallel jobs passed to the compiler (e.g. `make -j N`). `0` uses the system default. |
| `--parallel N` | `-P` | `1` | Number of mutants to verify concurrently. Values larger than the number of mutants are capped automatically. |
| `--setup-command CMD` | | | Command run once in every isolated parallel worker before its baseline check. Useful for configuring a fresh `build/` directory. |
| `--keep-worktrees` | | | Preserve temporary parallel worker worktrees for debugging instead of removing them. |
| `--survival-threshold RATE` | | `0.75` | Maximum acceptable mutant survival rate (e.g. `0.3` = 30%). The run exits with an error if the threshold is exceeded. |
| `--min-score RATE` | | | CI gate: fail with a non-zero exit code if the final mutation score (killed / total) is below this value (e.g. `0.8` = 80%). Aggregated across all analyzed folders. When unset, the score is not enforced. |
| `--surviving` | | | Only analyze mutants that survived a previous run. Requires `--run-id`. |
| `--survivors-only` | | | Only analyze mutants that survived a previous run. Requires `--run-id`. |

### Examples

Expand All @@ -163,7 +166,7 @@ bcore-mutation analyze --sqlite --run-id=1 --file-path="src/net_processing.cpp"

**Retry only survivors from a previous run:**
```bash
bcore-mutation analyze --sqlite --run-id=1 --surviving \
bcore-mutation analyze --sqlite --run-id=1 --survivors-only \
-c "cmake --build build && ./build/test/functional/wallet_test.py"
```

Expand All @@ -183,6 +186,28 @@ bcore-mutation analyze --sqlite --run-id=1 -t 120 -j 8 \
-c "cmake --build build && ./build/test/functional/wallet_test.py"
```

**Verify mutants with three parallel workers:**
```bash
bcore-mutation analyze --sqlite --run-id=1 --parallel 3 --jobs 4 \
--setup-command "cmake -B build -DENABLE_IPC=OFF && cmake --build build -j4" \
-c "cmake --build build -j4 && ./build/bin/test_bitcoin"
```

Parallel workers are assigned mutants dynamically, so a worker receives the
next pending mutant as soon as it finishes its current one. Each worker uses a
detached Git worktree and its own `build/` directory; the checkout from which
`bcore-mutation` was started is not modified. A custom test command runs from
the worker directory. If the command references a sibling path such as
`../qa-assets`, parallel mode links the matching sibling from the original
checkout's parent into the temporary worker root, so existing Bitcoin Core fuzz
corpus commands keep working without copying the corpus. Fresh worktrees do not
contain an existing build, so provide `--setup-command` unless the test command
configures the build itself.

`--parallel` controls concurrent mutants while `--jobs` controls parallel jobs
inside each build. For example, `--parallel 3 --jobs 4` can run approximately
12 compiler jobs and also requires disk space for three build trees.

**Set a survival rate threshold:**
```bash
bcore-mutation analyze --sqlite --run-id=1 --survival-threshold=0.2 \
Expand All @@ -197,6 +222,25 @@ bcore-mutation analyze --sqlite --run-id=1 --survival-threshold=0.2 \
cargo test
```

## Benchmarking Parallel Analysis

The `benchmark/` directory contains a Docker-based harness that runs the same
SQLite mutation run with multiple `--parallel` / `--jobs` configurations and
generates CSV results plus timing and speedup charts.

```bash
bash benchmark/run-in-docker.sh /path/to/bitcoin \
--db mutation.db \
--run-id 123 \
--repeats 3 \
--case sequential:1:3 \
--case parallel-3x3:3:3 \
--command "cmake --build build -j3 && ./build/bin/test_bitcoin"
```

See `benchmark/README.md` for the fuzzing-oriented example that keeps
`../qa-assets` available inside Docker.

## Contributing

1. Fork the repository.
Expand Down
25 changes: 25 additions & 0 deletions benchmark/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
FROM rust:1-bookworm

RUN apt-get update && apt-get install -y --no-install-recommends \
bash \
build-essential \
ca-certificates \
clang \
cmake \
git \
libboost-dev \
libevent-dev \
libsqlite3-dev \
ninja-build \
pkg-config \
python3 \
python3-matplotlib \
sqlite3 \
&& rm -rf /var/lib/apt/lists/*

WORKDIR /opt/bcore-mutation
COPY . /opt/bcore-mutation
RUN cargo install --path /opt/bcore-mutation

ENTRYPOINT ["python3", "/opt/bcore-mutation/benchmark/benchmark_parallel.py"]

68 changes: 68 additions & 0 deletions benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Parallel Benchmark Harness

This harness compares `bcore-mutation analyze --parallel 1` with one or more
parallel configurations inside Docker. It records wall-clock time, keeps logs,
and generates charts.

## Build and Run

From this repository:

```bash
bash benchmark/run-in-docker.sh /path/to/bitcoin \
--db mutation.db \
--run-id 123 \
--repeats 3 \
--timeout 900 \
--case sequential:1:3 \
--case parallel-2x4:2:4 \
--case parallel-3x3:3:3 \
--case parallel-4x2:4:2 \
--setup-command "cmake -B build_corecheck -DBUILD_FOR_FUZZING=ON && cmake --build build_corecheck -j3" \
--command "FUZZ=coin_grinder_is_optimal ./build_corecheck/bin/fuzz ../qa-assets/fuzz_corpora/coin_grinder_is_optimal"
```

The Docker runner mounts the subject repository's parent directory at `/bench`.
That matters for Bitcoin Core fuzz benchmarks because a sibling directory such
as `../qa-assets` remains visible inside the container.

By default the runner limits Docker to 10 CPUs. Override it with:

```bash
BCORE_BENCH_CPUS=8 bash benchmark/run-in-docker.sh /path/to/bitcoin ...
```

## Outputs

Results are written to `benchmark-results/` by default:

- `results.csv`: one row per case/repeat.
- `summary.md`: median timing and speedup table.
- `time-by-case.png`: median wall-clock seconds.
- `speedup-by-case.png`: speedup relative to the `sequential` case.
- `logs/*.stdout.log` and `logs/*.stderr.log`: raw analyze output.
- `db/*.db`: copied SQLite database used for each run.

Each run uses a copy of the input SQLite database, so the original DB is not
modified by the benchmark.

## Case Format

Cases use this format:

```text
LABEL:PARALLEL:JOBS
```

For a 10 CPU machine, useful starting points are:

```text
sequential:1:3
parallel-2x4:2:4
parallel-3x3:3:3
parallel-4x2:4:2
```

`PARALLEL * JOBS` is the approximate maximum compiler/test parallelism. For
example, `parallel-3x3` can run about 9 build jobs across 3 active mutants.

Binary file not shown.
Loading
Loading