Skip to content
Open
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
52 changes: 51 additions & 1 deletion docs/nvbench_compare_robust.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,14 +64,25 @@ nvbench-compare-robust --display explain reference.json compare.json
nvbench-compare-legacy reference.json compare.json
```

Plot the comparison summary, or plot timings along a positive numeric axis. Add
Plot the comparison summary, or plot timings along a positive numeric axis. By
default, plotting uses Matplotlib's interactive `plt.show()` behavior. Add
`--dark` to the summary plot when it should use a dark theme:

```bash
nvbench-compare-robust --plot --dark reference.json compare.json
nvbench-compare-robust --plot-along "Elements{io}" reference.json compare.json
```

Save plots to files when running in CI, remote shells, or scripted workflows:

```bash
nvbench-compare-robust --plot --plot-output compare.png reference.json compare.json
nvbench-compare-robust \
--plot-along "Elements{io}" \
--plot-along-output "plots/{benchmark}-device{device}-{axis}.png" \
reference.json compare.json
```

Generate Python code with bulk sample/frequency filenames for every displayed
row:

Expand Down Expand Up @@ -646,3 +657,42 @@ fraction: use `--threshold-diff 5` for a 5% threshold.

This option affects table output. It does not change summary counters or the
data used by `--plot-along`.

### `--plot-output PATH`

Save the summary plot generated by `--plot` to `PATH` and do not call
`plt.show()`. When this option is omitted, `--plot` keeps the default
interactive behavior.

For directory comparisons that produce more than one summary plot, a single
`--plot-output PATH` would overwrite earlier output. In that case the command
fails instead of writing multiple plots to the same path.

### `--plot-along-output PATH_OR_TEMPLATE`

Save plots generated by `--plot-along` to files and do not call `plt.show()`.
Because `--plot-along` may produce one plot per benchmark/device pair, this
option accepts filename templates with these fields:

- `{benchmark}`: benchmark name
- `{device}`: compare-device id
- `{axis}`: selected plot axis name
- `{pair}`: zero-based positional device-pair index

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

{pair} is the zero-based index of (reference_device_id, compare_device_id) pairs that specification of --reference-devices/--compare-devices implies as specified. For example, specifying --reference-devices 0,1 --compare-devices 0,0, the pair (0, 0) would have index 0, and the pair (1, 0) would have index 1.


Field values are sanitized before substitution so benchmark or axis names from
input JSON cannot introduce path separators. Directory structure should be
written literally in the template, as in `plots/{benchmark}.png`.

For example:

```bash
nvbench-compare-robust \
--plot-along "Elements{io}" \
--plot-along-output "plots/{benchmark}-pair{pair}-{axis}.png" \
reference.json compare.json
```

A plain path without template fields is valid only when one plot is generated
across the whole command. If multiple plot-along figures, including figures
from different JSON file pairs in directory mode, would write the same path, the
command fails instead of silently overwriting earlier output.
Comment thread
oleksandr-pavlyk marked this conversation as resolved.
Outdated
Loading
Loading