SCHED-1918: Improve multi-node startup with concurrent Pyxis cache reads

[aydinmirac]

Summary

Warm-cache users currently take the same exclusive flock used for image creation. On large multi-node jobs, this turns independent cache readers into a sequential queue even though the completed SquashFS image is immutable.

This change adds a lock-free warm-cache fast path while preserving exclusive coordination for cold-cache publication.

Changes

• Return completed cached images without acquiring flock.
• Use a stable, digest-scoped lock for cache misses.
• Allow one cold-cache producer to hold the exclusive lock.
• Release waiting readers together using a shared lock after publication.
• Use node-specific temporary files so cleanup cannot affect another node.
• Publish images atomically and retry if a producer exits without publishing.
• Avoid unlinking active lock files, which could allow callers to lock different inodes.

Validation

Automated tests cover concurrent warm readers, single-producer cold-cache behavior, node-specific cleanup and failed imports. End-to-end tests were performed using real Slurm, Pyxis and Enroot deployments:

Environment	Original	Patched	Improvement
8 CPU nodes, warm-cache total	11,124 ms	8,138 ms average	2,986 ms / 26.8%
8 CPU nodes, readiness spread	3,183 ms	451 ms average	85.8% smaller
4 H100 nodes, 12 GB PyTorch image, warm-cache total	38,201 ms average	37,179 ms average	1,022 ms / 2.7%
4 H100 nodes, readiness spread	4,716 ms	4,256 ms	9.8% smaller

The larger CPU improvement is expected. That test had eight lock contenders, while the GPU test had only four, so fewer readers accumulated behind the exclusive lock. The GPU test also has an approximately 27 second baseline from digest lookup, Slurm/Pyxis startup, SquashFS mounting, Enroot setup and NVIDIA device/library injection. These costs are outside this patch and dilute the percentage improvement. The patch targets the additional delay accumulated by queued readers, which becomes more significant as node count increases.

Cold-cache reconstruction of the 12 GB image completed successfully, and subsequent warm-cache launches remained stable.

An exact replay of the original and patched Bash importers was also performed against the shared VirtioFS cache using a fake digest lookup. At 64 concurrent callers:

• Original importer: 37,250 ms
• Patched importer: 175 ms
• Improvement: 99.5% / approximately 213×
• Failures: 0

The patched importer was successfully replayed with 1,000 concurrent callers. A lighter lock-path benchmark also completed 5,000 concurrent cache reads distributed across four workers in approximately 150 ms, with zero failures and zero lock wait.

Scope and limitations

The 5,000-caller benchmark is a synthetic contention test, not an end-to-end 5,000-node launch. It validates the locking algorithm but does not reproduce 5,000 independent Slurm daemons, filesystem clients, container mounts, network paths or GPU initialization. Production-scale testing is still required to measure absolute startup time on a real 5,000-node cluster.