Skip to content

Add opt-in macOS cleanup for delivered Python processes - #18

Open
mandu5 wants to merge 7 commits into
deeplethe:devfrom
mandu5:fix/delivered-process-lifetime-dev
Open

mandu5 wants to merge 7 commits into
deeplethe:devfrom
mandu5:fix/delivered-process-lifetime-dev

Conversation

@mandu5

@mandu5 mandu5 commented Oct 17, 2026 •

Copy link
Copy Markdown

A detached child of an oncall_incident responder can survive its parent's return and access the simulation during later incidents. This adds an opt-in ctx["run_delivered"] callback that cleans up identifiable descendants before returning or raising. It requires macOS 15.4+ with validated original-parent identity support; unsupported hosts stop before delivered code runs. Existing graders and the container/UID launch path remain unchanged.

Cases are distributed separately. The included one-line patch enables the callback in the pinned public oncall grader; adding the harness alone does not enable cleanup. Do not apply this macOS case patch to Linux runners as-is.

Validation

Fresh paired replay of commit ea68c1d1a37a3898bf47079d125d4e5b7e144be0 against dev at 03c1344d4aba7ba26b6f988fda014ce04f05bb82, using public dataset 10fee70cdb7e4010ea9b301359550d930b0de267:

Condition Cases before / after KPI before / after Observed requests reaching later incidents
Untouched starter 184 / 184 -565 / -565 0 / 0
Same starter + identical bounded detached-child probe 184 / 184 -565 / -565 60 / 0

All 184 per-case detail entries match in each comparison. Submitted inputs match by hash; both probe processes spawned; errors, timeouts, incidents, model calls and spend were zero. Harness files match the exact commits and remained unchanged during replay. Request evidence is an observer snapshot, not a complete service trace. Fresh source/result hashes and verification receipt.

  • Public-fork GitHub Actions: macOS 15.7.9 arm64, Python 3.9.13 and 3.13.15, 12 tests passed per interpreter. A separate macOS 14.8.9 job verifies rejection without a delivered side effect. Local Python 3.9.6 also passes all 12 tests.
  • Hosted CI exposed the older-kernel gap: the original-parent field is reserved/zero on macOS 14. An orphan that execs before discovery can lose its original parent UID. The fix rejects that missing capability and checks the stopped interpreter's original-parent generation before resuming it. The regression forces orphaning and exec before first discovery; it retains the immediate no-live-child assertion. Preserved failing diagnostic run.
  • App typecheck/build and all 33 app tests, zero skipped, passed with a disposable PostgreSQL14 cluster. The app source is unchanged by this follow-up. Selecting /usr/bin/python3 resolved the earlier local SQLite-view test environment failure.
  • The site smoke passed real registration, session creation, file loading, submission and grading for the patched public oncall case (25s, 184 incidents, KPI -565). Its separate PostgreSQL/S3-emulator services were stopped; no existing database or model key was used. Preserved site receipt.
  • Corrected pinned reproduction package, docs/process-lifetime.md, and docs/evaluation/ retain commands, source pins and earlier experimental snapshots.

Scope

This is bounded lifecycle cleanup, not a complete OS security boundary. An intermediate process that exits before observation can hide descendants; identity-to-signal races and forcible supervisor termination remain limitations. Apple exposes the identity structure in a private interface, so runtime capability and parent-generation checks are required.

The site smoke reported environment scrubbing without a file sandbox; it is lifecycle integration evidence, not full file/OS isolation. Production object storage, other cases/OSes and upstream CI approval remain separate checks. No score gain or research novelty is claimed.

@mandu5
mandu5 marked this pull request as ready for review October 17, 2026 05:39
@mandu5
mandu5 marked this pull request as draft October 17, 2026 05:54
@mandu5
mandu5 marked this pull request as ready for review October 17, 2026 06:06

@WaylandYang WaylandYang left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this. The mechanism is sound and the evidence is unusually thorough. I checked out the branch and ran it here:

  • macOS 26.7.1 / Python 3.9.6: all 12 tests in tests/test_process_lifetime.py pass, including the orphan-then-exec case.
  • The struct layout matches Apple's proc_uniqidentifierinfo in xnu-11417.101.15 (p_orig_ppidversion as the fifth field, reserved before 15.4), so the capability gate is right.
  • The existing harness check (compileall + run.py --version) passes on the branch, the sandbox wrapper execs Python so the SIGSTOP handshake works under sandbox-exec, and nothing on dev since your base touches isolate.py/run.py.

Requesting changes for four things, the first being the one that blocks a merge:

  1. It breaks grading on the production runner. run_delivered raises ProcessCleanupError anywhere that is not macOS, and the oncall grader only catches TimeoutExpired, so a case that opts in aborts the whole grade inside the Docker runner (Linux). That is where FDE Gym actually runs cases, and #21 just made a container per run the direction. Please fall back to subprocess.run when the host is not macOS (or when delivered code already runs as another user / in a container, where the container's teardown ends orphans anyway), and keep fail-closed only for macOS hosts older than 15.4. If you would rather keep the callback strictly macOS-only, then it needs a name that says so and the case patch must not be the thing that selects it.

  2. Keep the PR to the harness change. The three docs/evaluation/*.json receipts, tools/replay_process_lifetime.py and the patches/ directory are your experiment record, not repository code, and they open three new top-level directories. CONTRIBUTING asks for one change per pull request. Please drop them here; the PR description and the release links already preserve the evidence, and a short note in docs/process-lifetime.md can point at them. The one-line case change belongs in a PR to the examples dataset, not in this repository.

  3. The new workflow lists your own branch under push.branches; please remove it. Keeping the macOS jobs in a separate workflow is fine.

  4. Module-level import in isolate.py. from process_lifetime import run_delivered works only because every importer has harness/ on sys.path. Since run.py is the only place that needs it, import it there (where isolate itself is imported lazily) and leave isolate.py as it is.

Also: the tests/ directory is new to the repository and the existing harness CI job does not run it. That is fine for now, but say in CONTRIBUTING.md how to run it, next to the compileall line.


class _Processes:
def __init__(self):
if sys.platform != "darwin":

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the blocking one. On the Docker runner this raises for any case that opted in, and the grader does not catch it, so the grade aborts. Please fall back to subprocess.run off macOS (and in the container / other-UID path) instead of failing closed; keep the hard failure for macOS < 15.4 only.

Comment thread harness/run.py
"facts": asked_facts(run_dir) if run_dir else set(),
"run_dir": run_dir, "services": {}}
"run_dir": run_dir, "services": {},
"run_delivered": isolate.run_delivered}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the callback stays macOS-only, this key should say so (run_delivered_macos?), or the fallback should live here so that ctx["run_delivered"] is always safe for a grader to call.

Comment thread harness/isolate.py
import sys
import tempfile

from process_lifetime import run_delivered

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please move this import into run.py next to the lazy import isolate there. isolate is imported from bench.py and npc_server.py too and should not grow a dependency it does not use.

- tests/test_process_lifetime.py
- .github/workflows/process-lifetime.yml
push:
branches: [main, dev, fix/delivered-process-lifetime-dev]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Drop fix/delivered-process-lifetime-dev from the push branches; main and dev plus pull_request is enough.

@@ -0,0 +1,94 @@
"""Bounded experiment driver. Official harness/case files remain byte-identical."""

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The experiment driver, the docs/evaluation/*.json receipts and patches/ should come out of this PR (see the review summary). The release assets you linked already keep them reproducible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants