Skip to content

crun: avoid forced CRIU debug logging for normal TClone requests - #16

Open
yiying-zhang wants to merge 70 commits into
mainfrom
perf/tfork-runtime-log-level
Open

crun: avoid forced CRIU debug logging for normal TClone requests#16
yiying-zhang wants to merge 70 commits into
mainfrom
perf/tfork-runtime-log-level

Conversation

@yiying-zhang

Copy link
Copy Markdown
Collaborator

Summary

  • make TClone's CRIU log level follow crun's runtime verbosity;
  • keep normal requests at error level, warnings at warning level, and debug
    requests at full CRIU debug level;
  • add CRIU_TFORK_LOG_LEVEL=0..4 as an explicit diagnostic and exact-binary
    benchmark override;
  • leave checkpoint and ordinary restore logging unchanged.

Why

The TClone-specific crun path unconditionally selected CRIU debug level 4.
CRIU formats and writes detailed task, VMA, parasite, cgroup, image, and
restore traces throughout the operation. The output grows almost linearly
with process count: about 1.66 MB / 24,114 lines at 100 processes and 3.30 MB /
47,825 lines at 200.

This is useful on demand, but it should not be synchronous work in every
normal production fork.

Performance

The exact same candidate binary was used for both arms. Only
CRIU_TFORK_LOG_LEVEL=4 versus 1 changed; the independent Phase B sibling
optimization was forced off.

Processes Debug median Error median Paired median saving Wins
1 484.770 ms 468.237 ms 6.156 ms 3/5
4 562.424 ms 555.582 ms 7.378 ms 3/5
25 687.619 ms 643.021 ms 54.786 ms 5/5
50 864.009 ms 807.913 ms 61.850 ms 4/5
100 1,163.351 ms 1,036.457 ms 109.387 ms 10/10
200 1,792.382 ms 1,560.492 ms 236.800 ms 5/5

The median-fit slope falls from 6.402 to 5.286 ms/process, removing
1.116 ms/process. A combined test with eight-helper Phase B sibling creation
saved 195.529 ms at 100 processes and 420.795 ms at 200, showing that the two
changes are complementary.

Validation

  • isolated remote build succeeded;
  • crun tests: 503 total, 325 pass, 178 environment skips, 0 failures;
  • 70/70 primary scaling arms passed exact process-count, parent-tree,
    sentinel, filesystem-isolation, and source-survival checks;
  • an independent 100-process repeat also passed all 10 arms;
  • combined logging/parallel validation passed all 20 arms;
  • a no-override Podman/TClone smoke test selected the same error-only path;
  • the explicit level-4 override retained the full trace.

Operational note

Podman's custom TClone conmon launcher currently does not propagate generic
Podman --runtime-flag values. Use CRIU_TFORK_LOG_LEVEL=4 when full tracing
is needed through that path. A separate pre-existing issue causes failed-clone
rollback to delete the bundle containing the detailed CRIU log after reporting
its path; this change does not alter rollback, but that diagnostic should be
copied to a stable error record before bundle deletion.

yiying-zhang and others added 30 commits July 10, 2026 23:44
Stabilize live tfork source freezing
yiying-zhang and others added 30 commits July 20, 2026 03:55
…-main

Stabilize single-copy tfork clone path
Rewrite tclone setup for Gensee Crate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants