Master - #26
Open
Alok-Ethara wants to merge 254 commits into
Open
Conversation
…I migration Recovered from dangling checkpoint e7ebcf9 after a stray checkout dropped the work from the worktree. Adds: - environment/_mutable_store.py: in-process mutable record store (Tables + Documents, snapshot/restore, drift log, RLock per store). - environment/admin_plane.py: gated FastAPI router with IP allowlist, one-shot response interceptor, /admin/data/, /admin/inject/, /admin/snapshot, /admin/drift/log endpoints. - src/utils/drift_director.py: host-side thread polling /audit and firing schedule/triggers/one_shot events; trigger kinds audit.first_call_on, audit.after, audit.count_at_least, file.created, time.elapsed, all_of/any_of composites. - 101 <api>_data.py modules migrated to register Tables/Documents with _mutable_store; matching server.py wires install_admin_plane. - scripts/migrate_to_drift_plane.py + verify_applied.py. - tests/test_drift_plane_smoke.py (6 tests). - environment/MIGRATION_RECIPE.md. - Wiring in eval/run_batch.py, src/utils/mock_stack.py, src/utils/task_parser.py, environment/tracking_middleware.py to surface drift_script_path, pass MOCK_ADMIN_ENABLED env, start/stop the DriftDirector per task, and exclude /admin from audit. - input/brian-henderson__google-classroom__riya-jha_2/ reference task.
Caught during verification: data modules for these 2 APIs were migrated to the mutable store, but their server.py never got the matching install_admin_plane(app, store=...) call. All other 99 APIs already had it. Brings the count to 101/101 wired.
- Introduced `test_weights.json` to define weights for test cases in the alden-croft task. - Created a new log file `alden-croft_provtest_verify.log` to capture detailed execution logs. - Implemented `_load_provided_tests` function to load test code and weights from the task directory. - Updated task loading functions to include test code and weights in the task dictionary. - Enhanced the test executor to handle hand-authored test suites without requiring pytest fixtures.
…1 verifier - litellm_sidecar.py: route opus via native Bedrock Invoke (drop converse/ infix) + output_config.effort:high so thinking blocks stream as parseable Anthropic SSE (fixes 1-turn 'request ended without sending any chunks'). - runner.py: anthropic branch presents recognized claude-opus-4-6 id + api=anthropic-messages to activate extended thinking and round-trip signed thinking blocks; gpt-5.5 stays openai-completions. - grading.py: council evidence parity - exclude binary deliverables and sort report/flagged first so all judges receive identical evidence within their budgets (per_member_user_chars now equal; eliminates 'report.md missing' hallucination from truncated judges). - linear-api/linear_data.py: guard empty sortOrder to prevent import crash. - amanda_hayes_01 verifier (test_outputs.py + test_weights.json): authored 6-check suite; fix inverted polarity on no_* guardrail tests (==0). - test_judge_budget_invariant.py: tighten chars_per_token_floor for non-Sonnet judges (Bedrock ctx guard). - SIX_CHECK_VERIFICATION_REPORT.md: full evidence report for both models.
Oracle gate 3 required persisted GPT evidence (run_1 predated the parity fix). Fresh GPT-5.5 run_2 (batch b096d8da4725) completed end-to-end with a report.md + binary JPEGs present (the exact gate-3 trigger config): - CHECK 6: 3 judges, 26/26 verdicts, truncation_affected_by_judge=[0,0,0], ZERO 'report.md missing'/'jpeg corrupted' hallucinations (vs 48+8 pre-fix). All 3 judges cite report/flagged/grocery evidence. Char asymmetry now is legitimate budget trimming of low-priority trailing evidence only. - CHECK 1: independent HTTP probe - notion 11/plaid 1/trello 10 business calls, all 5 APIs HTTP 200, quickbooks distractor 0. - CHECK 2-5 re-confirmed: import_error null; ctrf 28P/3F; 20/20 thinking blocks; cache_read 88.6% usage_source=litellm. Report updated to cite run_2 as authoritative for the GPT path.
… account rotation
…erbatim publishing
- pull-based turn engine (src/utils/turn_source.py: StaticTurnSource / HumanTurnSource) + AgentTaskSpec.turn_source; runner loop pulls per-turn - --interactive wires HumanTurnSource on the native+inject path: human paces each turn (Enter=accept scripted / type=override / /exit=end), the agent's prior reply is echoed, inject silent mutations still fire at boundaries, scoring unchanged (Channel A test_outputs.py + optional --judge-council) - script/session_to_prompts.py promotes a session's turn_timeline.jsonl into a static prompts.txt (SFT session -> reusable static task) - regrade _find_prompt_path prompts.txt/PROMPT.md fallback - Skoll output.json fixes: platform 'Linux', task.yaml system_prompt retention - docs/MULTITURN.md
…hecks - task_parser: prompts.txt now takes priority over the prompt.txt T0 mirror. Every corpus task ships both, so the old order emptied turn_messages and silently degraded every multi-turn run to a single turn with no injections. - inject_director: _patch_row accepts the explicit-admin caller's pk so admin-block patches work on domain-keyed stores (order_id etc.) whose rows expose no id/pk column. - preflight_task: count data/ files recursively (corpus stages under data/home/**) and treat distractor_apis: auto as the sentinel instead of iterating it as a char list (a-api/u-api/t-api/o-api).
- script/compile_declarative_task.py: compile four data files
(metadata.json, prompt.txt, stages.toml, rubrics.json) into a native task
bundle; positional turn/injection bookkeeping, service-name normalization,
weight vocabulary enforcement, checks compiled to a hermetic pytest suite
(regex_any matched with IGNORECASE against normalized values).
- declarative/: authoring area + README. input/ is gitignored, so these
sources are the version-controlled representation of compiled tasks.
- standard_task_template: the reference exemplar every new task copies —
overlay-seeded world, admin-block silent upsert, filesystem drop,
turn-only stage, anchored checks, status-gated violation detectors.
Hardened via a 35-agent adversarial review (13 confirmed findings fixed).
- davis_hitl_catering_invoice: multi-turn task authored for interactive
Mode-2 (human-in-the-loop SFT) with a re-verification branch point.
- docs/TASK_STANDARD.md: the authoring standard (12 rules + definition of
done). docs/MULTITURN.md: prompts.txt-priority note, declarative authoring
section, working OAuth commands, current example task.
- .gitignore: input/ (corpus is untracked; declarative sources are the
system of record) and declarative/_build/ (regenerable compiler output).
run_background opened the per-turn log with mode 'w', so each turn's relaunch truncated agent.log and only the last turn survived. Add a keyword-only append flag and pass append=turn_index > 0 from the turn loop (turn 0 still truncates so retried run_N dirs start clean). Also close each turn's log handle after the process exits (close_proc_log previously had no callers, leaking one handle per turn). Regression test pins the aggregation and the turn-0 truncation.
The inject pipeline logged 'applied: N silent op(s)' counting attempts, discarded per-op outcomes, and echoed request values as 'after' without verification — so tasks whose mutations never landed (wrong service name, wrong storage object) survived seven runs looking healthy. inject_director: stage log now reports applied/FAILED counts from real outcomes (WARNING on any failure); timeline records applied_ops/failed_ops; seed() returns its outcomes; new is_defect() classifier (seed-time fs ops without a workspace copy hook stay benign); every successful write is re-read through the audit-invisible admin plane and a 2xx whose values are absent on read-back becomes a failure; fixed the REST-path status clobber (int HTTP code now in 'http', status is the semantic string). run_batch: injection_defects accumulator feeds injection_ok/ injection_defects into score.json (happy, stub and last-resort writers) and pass_summary (mirrored in rebuild/backfill scripts); opt-in WCB_STRICT_INJECTION=1 promotes defects to a run error before grading. regrade preserves the injection keys; repackage warns on stderr when bundling a defective run; preflight warns on bare REST-form mutation ops (fuzzy resolution); TUI colors error/no-match/missing_src red. Verified: broken task (davis_lab_worry) flags 7 defects at every layer including 5 previously unknown missing seed files; known-good task (davis_hitl_catering_invoice) scores its exact baseline with all injections read-back-verified and zero warnings.
Multi-turn tasks now ship a prompts.json/prompts.txt PAIR: the finalized
json schema (task_id/persona/timezone/turn_count + turns[{turn,
timestamp, message}]) drives the trajectory, and the authored txt is
published verbatim in the output bundle as the client-facing contract.
A prompts.json without its companion txt fails loudly at load and in
preflight; pair-content drift (count or text) warns but does not block
(pair correctness is owned by the task team). Legacy txt-only tasks are
untouched.
parse_prompts_json validates loudly at load: contiguous T0..TN labels
matching position, turn_count agreement, non-empty messages, ISO-8601
chronological timestamps (draft day/time shape still accepted). The
loader prefers prompts.json over all txt forms and stores schedule
metadata as task[turn_schedule] (never sent to the agent, matching the
txt format where header labels are discarded). Downstream readers
updated: testgen cache key hashes prompts.json only when present (legacy
cache keys unchanged), task-root copy, regrade judge-prompt candidate,
preflight structure + turn checks, and the bundler publishes the
authored txt verbatim for pairs (rendering json to canonical
TURN-delimited text with timestamp-derived Day/time headers only as the
legacy fallback).
Runs also capture the task's inject/ spec verbatim: run_N/inject/ next
to inject_timeline.jsonl (replace-not-merge on retries), a copy_inject
stager into the bundle root, and reconstruct_input_from_bundle recovers
it. run_batch warns when an inject stage targets a turn beyond the
schedule (previously a silent no-op).
Verified e2e on the catering task converted to the finalized schema:
3 turns fed from JSON, injection applied and read-back verified,
injection_ok true, inject copies byte-identical, bundle PROMPT.md
carries the authored companion txt verbatim, combined reward at the
0.8077 baseline.
Ops whose path targets /admin/* are now replayed verbatim against the environment's admin plane instead of being routed through fuzzy row resolution: POST /admin/data/<table>+row -> store upsert with read-back verification, PATCH /admin/data/<table>/<pk> -> patch with before/after capture, other admin POSTs (e.g. /admin/inject/raw) replayed with per-op results checked. environment/ is the task team's source of truth and the admin plane's endpoints are part of that surface; the applier previously rejected valid calls against it (creates could not be expressed at all). Row lookups now honor each table's declared primary_key from /admin/tables (cached per api) instead of hardcoding id/pk — fixes every non-lowercase-id store (quickbooks/salesforce Id, monday item_id, servicenow sys_id, algolia objectID, ...) at all three lookup sites. inject_validator learns the same shapes: bare admin-path ops classify as upsert/patch/admin_raw (creates no longer rejected for zero fields, and their row ids fold into the seed snapshot), bare-PATCH targets are checked via the path pk, and seed-id collection is pk-name/case insensitive (was false-flagging quickbooks rows keyed by Id). Tests: three stubs updated to the 4-tuple _resolve_target signature the schema-acceptance commit introduced (pre-existing failures). Adds RUN_JOURNAL.md + tasks_multiturn_batch1.txt tracking the kensei-dataset multi-turn pilot these fixes were diagnosed against.
…anded Two defects surfaced while making silent drift land for raw-REST ops: 1. okta `_set_user_status` mutated a deep-copied row (store reads copy) and never persisted, so suspend/activate/deactivate returned 200 but the store stayed unchanged — the agent's reads AND injected drift silently no-op'd. Fix: patch the users table so lifecycle transitions actually stick. 2. `apply_as_api` trusted a 2xx from the replayed endpoint without checking the store changed, so a non-persisting endpoint reported a false "applied". Fix: the admin-plane replay fingerprints the store before/after and returns `changed`; the injector marks `applied`/`verified` only when changed=True, else falls through to an honest "unresolved". Verified end-to-end: okta 00u00055vs ACTIVE->SUSPENDED in the live store and the after_injection snapshot; new tests cover changed=True (real persist) and changed=False (copy-bug shape). Full injection/drift/smoke suite green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
13 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.