Skip to content

feat: production FQL actor, critic and offline transition training - #293

Draft
felixmaximilian wants to merge 8 commits into
mainfrom
feat/fql-production
Draft

felixmaximilian wants to merge 8 commits into
mainfrom
feat/fql-production

Conversation

@felixmaximilian

@felixmaximilian felixmaximilian commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Scope

Opt-in production FQL implementation on main-based branch: official low-dimensional architecture with a flow behavior prior, a separate one-step actor, twin critics and target critics. It adds explicit-noise losses, isolated gradients, pre-optimizer target EMA, Lightning training/checkpoint contracts, normalization, and an explicitly labeled transition datamodule. FM/VQ/RTC defaults are unchanged. This is H1 state-based FQL, not an image/H50/RTC/D12 serving policy.

Validation

  • Numerical compatibility: same-point forward/loss/all-gradient checks and persistent common-gradient Adam/counters/EMA checks pass. The original strict chained numerical diagnostics still fail near Adam epsilon and remain disclosed; this is not a bitwise JAX trajectory claim.
  • Transferred behavior: 24 accepted checkpoints, 1,200 paired episodes per backend. Torch 95.42% versus JAX 95.92%; difference -0.50 percentage points, 95% interval [-1.25,+0.0833], within the frozen ±2-point margin.
  • Native training: eight analytic-initialized production seeds × 1M updates; 1,155/1,200 fixed-checkpoint successes = 96.25% ±1.63pp sample SD. Accepted reference = 96.333%; paired difference -0.0833pp, interval [-2.0833,+2.1667]pp. All three frozen gates pass. The independent benchmark audit verifies 80 checkpoint loads, actual optimizer counters, and 4,400 raw evaluation episodes.
  • Local focused validation after the CI typing fix: 47 FQL tests pass; formatting, Ruff, and targeted type checks pass. Full CI passes at commit 5966718 (sync, check, full test suite).

Evidence and scope limits

Benchmark-side receipts and frozen reports are in yaak-ai/rmind-bench#7; the independent audit is in yaak-ai/rmind-bench#4. The benchmark imports the actual production source rather than a duplicate objective. Task-unaware OGBench play is not labeled failed truck experience. Image conditioning, action chunks, RTC, and D12 rollouts require a separate architecture/data/closed-loop gate and PR. No D12 training, deployment or actuation is authorized by this PR.

@felixmaximilian

Copy link
Copy Markdown
Contributor Author

Independent source-contract review at def7325: no remaining blocker found for the explicitly opt-in single-transition/state FQL scope. Compared gradient routing and old-critic EMA to pinned actual upstream. Findings fixed: NaN/unstable episode IDs bypassing disjointness; coercible boolean config; overflowing physical action spans; nonidentity tiny-action normalization; complex input silently cast to real. Independent26CPUtests pass. Existing dataset adapter deliberately requires explicit rewards/masks/next states and does not infer human failure labels; exact episode transition/timeout semantics remain a benchmark-adapter gate. Generic shuffled Lightning loader is not upstream with-replacement training. This is source-contract approval ONLY: strict chained numerical oracle currently retains a diagnosed near-zero-gradient Adam failure, and transferred/native8-seed G2 quality gates remain pending. No G2-complete claim, merge authorization or D12 training/deployment approval.

@felixmaximilian

Copy link
Copy Markdown
Contributor Author

Incremental independent source review now covers0cbc3f46f373110fd785398573b9a6a8ca2e76d8. The actual clipping derivative discrepancy at exact±1 is corrected in actor-Q loss (matching JAX half tie), and physical endpoint roundoff is clamped only after validating true physical bounds. Independently28CPUtests pass. This supersedes previous commit-scoped source review only; amended full numerical oracle and transferred/native8-seed quality remain pending. No merge/deployment or G2-complete claim.

@felixmaximilian

Copy link
Copy Markdown
Contributor Author

Independent incremental source-contract review: approved engineering scope at d91597c. Shared Adam factory retains single_tensor default, exposes optional foreach, rejects fused. Requested and verified checkpoint resume guards bind constructor execution/LR to saved contract and actual optimizer param-group settings (with explicit legacy single-tensor compatibility). Independently ran tests/test_fql.py: 35 passed. Numerical/closed-loop validation for the chosen native runtime/mode remains a separate G2 gate; this is not D12 training or deployment approval.

@felixmaximilian

Copy link
Copy Markdown
Contributor Author

Independent final incremental engineering review: exact e608fac approved in scoped source review; 47 CPU tests independently pass. Shared gradient validator preserves every existing finite-gradient predicate, including any present target-critic gradients, and runs before EMA/Adam; only per-device host synchronization is aggregated. No training math or safety guard was removed. Closed-loop transferred-checkpoint G2-B separately passes (95.4167% vs95.9167%, paired95% CI difference [-1.25,+0.0833]pp inside frozen±2pp); native eight-seed G2-C remains pending, so no D12 training/deployment approval.

@felixmaximilian
felixmaximilian marked this pull request as ready for review October 19, 2026 20:13
@felixmaximilian
felixmaximilian marked this pull request as draft October 20, 2026 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant