Skip to content

criu: parallelize independent root siblings during TClone restore - #15

Open
yiying-zhang wants to merge 70 commits into
mainfrom
perf/tfork-parallel-siblings
Open

criu: parallelize independent root siblings during TClone restore#15
yiying-zhang wants to merge 70 commits into
mainfrom
perf/tfork-parallel-siblings

Conversation

@yiying-zhang

Copy link
Copy Markdown
Collaborator

Summary

  • add an opt-in CRIU_TFORK_PARALLEL_SIBLINGS=N restore path;
  • distribute eligible root children across a bounded temporary helper pool;
  • use clone3(set_tid) plus syscall-only CLONE_PARENT to preserve requested
    PIDs and the restored parent graph;
  • retain the serial path for small trees and unsupported clone semantics;
  • make 0 an explicit off value for exact-binary ablation.

Why

At 100 processes, profiling measured about 166.96 ms per run in
fork_with_pid(), including 155.75 ms in clone3. Kernel sampling assigned
most sampled copy_process() time to dup_mmap(). CRIU currently launches
root siblings serially even though their Linux task creation is independent.

This change overlaps that real kernel and restore initialization work instead
of optimizing another small image-decoding or procfs leaf.

Safety scope

The fast path is disabled by default and requires:

  • active TClone restore at the process-tree root;
  • clone3(set_tid) support;
  • at least 32 children in the relevant session group;
  • no child with CLONE_PARENT, CLONE_NEWPID, CLONE_VM, or
    CLONE_THREAD requirements.

Worker count is capped at 16. Nested restore tasks stay serial, so wide nested
trees cannot recursively multiply helper pools. The restore root blocks
SIGCHLD, reaps every temporary helper, and fails restore if any helper fails.
An ineligible group falls back before any parallel work begins.

Performance

The A/B used the exact same Podman, crun, libcriu, and CRIU files. Only
CRIU_TFORK_PARALLEL_SIBLINGS=0 versus 8 changed. Every run used a fresh
source, alternating AB/BA order, and a one-second idle cooldown.

Processes Serial median Parallel median Paired median saving Wins
1 506.682 ms 506.380 ms -21.938 ms 1/5
4 546.294 ms 546.700 ms 7.864 ms 3/5
25 678.479 ms 684.028 ms -10.993 ms 2/5
50 822.954 ms 770.983 ms 80.073 ms 5/5
100 1,127.787 ms 1,030.824 ms 90.810 ms 10/10
200 1,762.010 ms 1,547.472 ms 236.945 ms 5/5

At 100 processes, the internal Phase-B-marker-to-clone-tree median fell from
343.010 to 227.406 ms, with 118.879 ms paired-median saving. A median fit
over all six points changes from 6.231 to 5.117 ms/process for this synthetic
workload.

Validation

  • full CRIU build succeeds;
  • criu check --all reports Looks good.;
  • all final 50/100/200 process candidate samples won;
  • every recorded source and child had the exact expected process count and
    flat parent graph;
  • sentinel and source/child filesystem isolation checks passed;
  • a 161-process nested tree preserved 40 direct parents and 120 grandchildren;
  • 40 independent session leaders preserved their PID, PPID, and SID
    relationships;
  • the source remained running and unchanged after every recorded fork.

Known limitations

The branch is intentionally opt-in pending nested PID-namespace,
shared-VM/thread-group, helper-failure-injection, concurrency, and long-soak
coverage. Three exploratory runs also hit the pre-existing 60-second
event-readiness timeout—two serial and one parallel. Transactional rollback
cleaned each unpublished child. This branch does not attempt to hide or retry
that separate reliability issue.

yiying-zhang and others added 30 commits July 10, 2026 23:44
Stabilize live tfork source freezing
yiying-zhang and others added 30 commits July 20, 2026 03:55
…-main

Stabilize single-copy tfork clone path
Rewrite tclone setup for Gensee Crate
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants