Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions tools/loop-context/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,14 @@ cat run.json | loop-context --check
| `--stagnation <n>` | 3 | Escalate when the same error repeats N× in a row |
| `--no-progress <n>` | 5 | Escalate after N consecutive failures |
| `--token-budget <n>` | none | Escalate when cumulative tokens reach the cap |
| `--similarity-threshold <f>` | 0.85 | How alike two errors (or actions) must be to count as a repeat. A fraction in (0, 1], **not a percentage**: `0.95`, not `95` |
| `--window <n>` | 5 | Attempts kept when pruning |
| `--max-trace-lines <n>` | 8 | Stack-trace lines kept when pruning |

Exit codes: `0` continue · `2` escalate · `1` error.

A value that would switch a rule off is refused with exit `1` instead of being accepted. That covers a threshold of `0` or a fraction like `1.5`, and a similarity threshold above `1`: similarity is at most 1.0, so `95` would mean no two errors ever match and the stagnation rule never fires.

## Resolving the token budget from a pattern

Typing `--token-budget <n>` by hand means guessing. [`loop-cost`](../loop-cost) already
Expand Down Expand Up @@ -179,6 +182,8 @@ if (decision.escalate) escalateToHuman(decision.reason);
else runNextIteration(buildContextInjection(ledger));
```

`checkCircuitBreaker`, `pruneLedger`, `summarizeAttempts` and `buildContextInjection` throw on a config that would switch a rule off (a non-positive or fractional count, or a `similarityThreshold` outside (0, 1]), rather than quietly never escalating. Call `validateBreakerConfig` / `validatePruneConfig` to check a config up front.

## Where it fits

This is the **Memory / State** primitive of loop engineering made dynamic: `STATE.md` stores state statically; `loop-context` manages it across iterations. See [docs/primitives.md](../../docs/primitives.md) and the [operating & safety](../../docs/) guides.
Expand Down
21 changes: 13 additions & 8 deletions tools/loop-context/dist/cli.js
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#!/usr/bin/env node
import { readFile } from 'node:fs/promises';
import { spawn } from 'node:child_process';
import { buildContextInjection, checkCircuitBreaker, pruneLedger, summarizeAttempts, DEFAULT_BREAKER, DEFAULT_PRUNE, } from './context-manager.js';
import { buildContextInjection, checkCircuitBreaker, pruneLedger, summarizeAttempts, DEFAULT_BREAKER, DEFAULT_PRUNE, assertSimilarityThreshold, } from './context-manager.js';
import { resolveTokenBudgetFromPattern, resolveDailyBudgetFromPattern, } from './budget-resolver.js';
import { recordDailySpend } from './daily-spend.js';
/** Reject NaN/0/floats so a bad flag cannot silently disable the breaker. */
Expand All @@ -15,14 +15,18 @@ function parsePositiveIntFlag(raw, flag) {
}
return n;
}
function parsePositiveFloatFlag(raw, flag) {
/**
* A similarity threshold is a fraction in (0, 1]. "95" (meant as 95%) used to
* be accepted and silently switched off the stagnation rule, because no two
* errors are ever more than 1.0 similar.
*/
function parseFractionFlag(raw, flag) {
if (raw === undefined || raw === '') {
throw new Error(`${flag} requires a positive number value.`);
throw new Error(`${flag} requires a value between 0 and 1 (e.g. 0.85).`);
}
const n = Number(raw);
if (Number.isNaN(n) || n <= 0) {
throw new Error(`${flag} must be a positive number; got "${raw}".`);
}
// Report "abc" rather than NaN when the value is not a number at all.
assertSimilarityThreshold(Number.isNaN(n) ? raw : n, flag);
return n;
}
function parseArgs(argv) {
Expand Down Expand Up @@ -91,7 +95,7 @@ function parseArgs(argv) {
else if (a === '--max-trace-lines')
prune.maxTraceLines = parsePositiveIntFlag(argv[++i], '--max-trace-lines');
else if (a === '--similarity-threshold') {
const val = parsePositiveFloatFlag(argv[++i], '--similarity-threshold');
const val = parseFractionFlag(argv[++i], '--similarity-threshold');
breaker.similarityThreshold = val;
prune.similarityThreshold = val;
}
Expand Down Expand Up @@ -167,7 +171,8 @@ Options:
--on-exceed <script> On escalate, pipe the decision as JSON to this
script's stdin (fire-and-forget; its exit code
is not checked and does not change --check's own).
--similarity-threshold <f> Float 0.0-1.0 to cluster similar errors (default: ${DEFAULT_BREAKER.similarityThreshold})
--similarity-threshold <f> Fraction in (0, 1] to cluster similar errors (default: ${DEFAULT_BREAKER.similarityThreshold}).
A fraction, not a percentage: 0.95, not 95
--window <n> Attempts kept when pruning (default: ${DEFAULT_PRUNE.window})
--max-trace-lines <n> Stack-trace lines kept (default: ${DEFAULT_PRUNE.maxTraceLines})
-h, --help This help
Expand Down
5 changes: 5 additions & 0 deletions tools/loop-context/dist/context-manager.d.ts
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,11 @@ export interface PruneConfig {
}
export declare const DEFAULT_BREAKER: CircuitBreakerConfig;
export declare const DEFAULT_PRUNE: PruneConfig;
/** Throw unless value is a similarity threshold: a fraction in (0, 1]. */
export declare function assertSimilarityThreshold(value: unknown, label?: string): asserts value is number;
/** Throw if any rule in the config could never fire (or would fire on nothing). */
export declare function validateBreakerConfig(config: CircuitBreakerConfig): void;
export declare function validatePruneConfig(config: PruneConfig): void;
/**
* Reduce a raw error / stack trace to a stable signature so that "the same
* error" can be recognized across iterations even when volatile details
Expand Down
44 changes: 44 additions & 0 deletions tools/loop-context/dist/context-manager.js
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,47 @@ export const DEFAULT_PRUNE = {
window: 5,
similarityThreshold: 0.85,
};
// ── Config validation ──────────────────────────────────────────────
//
// Every breaker rule compares a count or a similarity against a threshold, so
// an out-of-range value does not error on its own: it quietly turns the rule
// off. calculateSimilarity() tops out at 1.0, so a similarityThreshold of 95
// (meant as 95%) means no two errors ever match and stagnation never fires;
// NaN makes every comparison false. A breaker that silently stops breaking is
// worse than one that refuses to start, so bad configs throw.
/** Throw unless value is a similarity threshold: a fraction in (0, 1]. */
export function assertSimilarityThreshold(value, label = 'similarityThreshold') {
// NaN fails both comparisons and Infinity fails the second, so this range check covers them.
if (typeof value === 'number' && value > 0 && value <= 1)
return;
let why = '';
if (typeof value === 'number' && value > 1) {
why = ' Similarity is at most 1.0, so no two errors would ever match and the stagnation rule would never fire.';
if (Number.isInteger(value) && value <= 100)
why += ` If you meant ${value}%, use ${value / 100}.`;
}
throw new Error(`${label} must be a fraction greater than 0 and at most 1 (default ${DEFAULT_BREAKER.similarityThreshold}); got ${String(value)}.${why}`);
}
function assertPositiveInteger(value, label) {
if (typeof value === 'number' && Number.isInteger(value) && value >= 1)
return;
throw new Error(`${label} must be a positive integer; got ${String(value)}. An invalid value would switch this rule off.`);
}
/** Throw if any rule in the config could never fire (or would fire on nothing). */
export function validateBreakerConfig(config) {
assertPositiveInteger(config.maxIterations, 'maxIterations');
assertPositiveInteger(config.stagnationThreshold, 'stagnationThreshold');
assertPositiveInteger(config.frustrationThreshold, 'frustrationThreshold');
assertPositiveInteger(config.noProgressThreshold, 'noProgressThreshold');
if (config.tokenBudget !== undefined)
assertPositiveInteger(config.tokenBudget, 'tokenBudget');
assertSimilarityThreshold(config.similarityThreshold);
}
export function validatePruneConfig(config) {
assertPositiveInteger(config.maxTraceLines, 'maxTraceLines');
assertPositiveInteger(config.window, 'window');
assertSimilarityThreshold(config.similarityThreshold);
}
// ── Error normalization ────────────────────────────────────────────
/**
* Reduce a raw error / stack trace to a stable signature so that "the same
Expand Down Expand Up @@ -97,6 +138,7 @@ function trailingFailureRun(attempts) {
* actionable one when several conditions hold.
*/
export function checkCircuitBreaker(ledger, config = DEFAULT_BREAKER) {
validateBreakerConfig(config);
const iterations = ledger.attempts.length;
const tokensUsed = totalTokens(ledger);
const base = { iterations, tokensUsed };
Expand Down Expand Up @@ -197,6 +239,7 @@ export function pruneStackTrace(trace, maxLines) {
* returns a new object.
*/
export function pruneLedger(ledger, config = DEFAULT_PRUNE) {
validatePruneConfig(config);
const recent = ledger.attempts.slice(-config.window);
const collapsed = [];
for (const attempt of recent) {
Expand All @@ -223,6 +266,7 @@ export function pruneLedger(ledger, config = DEFAULT_PRUNE) {
}
/** Deterministic factual rollup of the whole run — no LLM required. */
export function summarizeAttempts(ledger, similarityThreshold = 0.85) {
assertSimilarityThreshold(similarityThreshold);
const groups = new Map();
const actions = new Set();
let successes = 0;
Expand Down
20 changes: 13 additions & 7 deletions tools/loop-context/src/cli.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import {
summarizeAttempts,
DEFAULT_BREAKER,
DEFAULT_PRUNE,
assertSimilarityThreshold,
type Ledger,
type CircuitBreakerConfig,
type PruneConfig,
Expand Down Expand Up @@ -52,14 +53,18 @@ function parsePositiveIntFlag(raw: string | undefined, flag: string): number {
return n;
}

function parsePositiveFloatFlag(raw: string | undefined, flag: string): number {
/**
* A similarity threshold is a fraction in (0, 1]. "95" (meant as 95%) used to
* be accepted and silently switched off the stagnation rule, because no two
* errors are ever more than 1.0 similar.
*/
function parseFractionFlag(raw: string | undefined, flag: string): number {
if (raw === undefined || raw === '') {
throw new Error(`${flag} requires a positive number value.`);
throw new Error(`${flag} requires a value between 0 and 1 (e.g. 0.85).`);
}
const n = Number(raw);
if (Number.isNaN(n) || n <= 0) {
throw new Error(`${flag} must be a positive number; got "${raw}".`);
}
// Report "abc" rather than NaN when the value is not a number at all.
assertSimilarityThreshold(Number.isNaN(n) ? raw : n, flag);
return n;
}

Expand Down Expand Up @@ -109,7 +114,7 @@ function parseArgs(argv: string[]): Args {
else if (a === '--window') prune.window = parsePositiveIntFlag(argv[++i], '--window');
else if (a === '--max-trace-lines') prune.maxTraceLines = parsePositiveIntFlag(argv[++i], '--max-trace-lines');
else if (a === '--similarity-threshold') {
const val = parsePositiveFloatFlag(argv[++i], '--similarity-threshold');
const val = parseFractionFlag(argv[++i], '--similarity-threshold');
breaker.similarityThreshold = val;
prune.similarityThreshold = val;
}
Expand Down Expand Up @@ -189,7 +194,8 @@ Options:
--on-exceed <script> On escalate, pipe the decision as JSON to this
script's stdin (fire-and-forget; its exit code
is not checked and does not change --check's own).
--similarity-threshold <f> Float 0.0-1.0 to cluster similar errors (default: ${DEFAULT_BREAKER.similarityThreshold})
--similarity-threshold <f> Fraction in (0, 1] to cluster similar errors (default: ${DEFAULT_BREAKER.similarityThreshold}).
A fraction, not a percentage: 0.95, not 95
--window <n> Attempts kept when pruning (default: ${DEFAULT_PRUNE.window})
--max-trace-lines <n> Stack-trace lines kept (default: ${DEFAULT_PRUNE.maxTraceLines})
-h, --help This help
Expand Down
47 changes: 47 additions & 0 deletions tools/loop-context/src/context-manager.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,50 @@ export const DEFAULT_PRUNE: PruneConfig = {
similarityThreshold: 0.85,
};

// ── Config validation ──────────────────────────────────────────────
//
// Every breaker rule compares a count or a similarity against a threshold, so
// an out-of-range value does not error on its own: it quietly turns the rule
// off. calculateSimilarity() tops out at 1.0, so a similarityThreshold of 95
// (meant as 95%) means no two errors ever match and stagnation never fires;
// NaN makes every comparison false. A breaker that silently stops breaking is
// worse than one that refuses to start, so bad configs throw.

/** Throw unless value is a similarity threshold: a fraction in (0, 1]. */
export function assertSimilarityThreshold(value: unknown, label = 'similarityThreshold'): asserts value is number {
// NaN fails both comparisons and Infinity fails the second, so this range check covers them.
if (typeof value === 'number' && value > 0 && value <= 1) return;
let why = '';
if (typeof value === 'number' && value > 1) {
why = ' Similarity is at most 1.0, so no two errors would ever match and the stagnation rule would never fire.';
if (Number.isInteger(value) && value <= 100) why += ` If you meant ${value}%, use ${value / 100}.`;
}
throw new Error(
`${label} must be a fraction greater than 0 and at most 1 (default ${DEFAULT_BREAKER.similarityThreshold}); got ${String(value)}.${why}`,
);
}

function assertPositiveInteger(value: unknown, label: string): void {
if (typeof value === 'number' && Number.isInteger(value) && value >= 1) return;
throw new Error(`${label} must be a positive integer; got ${String(value)}. An invalid value would switch this rule off.`);
}

/** Throw if any rule in the config could never fire (or would fire on nothing). */
export function validateBreakerConfig(config: CircuitBreakerConfig): void {
assertPositiveInteger(config.maxIterations, 'maxIterations');
assertPositiveInteger(config.stagnationThreshold, 'stagnationThreshold');
assertPositiveInteger(config.frustrationThreshold, 'frustrationThreshold');
assertPositiveInteger(config.noProgressThreshold, 'noProgressThreshold');
if (config.tokenBudget !== undefined) assertPositiveInteger(config.tokenBudget, 'tokenBudget');
assertSimilarityThreshold(config.similarityThreshold);
}

export function validatePruneConfig(config: PruneConfig): void {
assertPositiveInteger(config.maxTraceLines, 'maxTraceLines');
assertPositiveInteger(config.window, 'window');
assertSimilarityThreshold(config.similarityThreshold);
}

// ── Error normalization ────────────────────────────────────────────

/**
Expand Down Expand Up @@ -181,6 +225,7 @@ export function checkCircuitBreaker(
ledger: Ledger,
config: CircuitBreakerConfig = DEFAULT_BREAKER,
): BreakerDecision {
validateBreakerConfig(config);
const iterations = ledger.attempts.length;
const tokensUsed = totalTokens(ledger);
const base = { iterations, tokensUsed };
Expand Down Expand Up @@ -286,6 +331,7 @@ export function pruneStackTrace(trace: string, maxLines: number): string {
* returns a new object.
*/
export function pruneLedger(ledger: Ledger, config: PruneConfig = DEFAULT_PRUNE): Ledger {
validatePruneConfig(config);
const recent = ledger.attempts.slice(-config.window);

const collapsed: Attempt[] = [];
Expand Down Expand Up @@ -337,6 +383,7 @@ export interface AttemptSummary {

/** Deterministic factual rollup of the whole run — no LLM required. */
export function summarizeAttempts(ledger: Ledger, similarityThreshold: number = 0.85): AttemptSummary {
assertSimilarityThreshold(similarityThreshold);
const groups = new Map<string, ErrorGroup>();
const actions = new Set<string>();
let successes = 0;
Expand Down
35 changes: 35 additions & 0 deletions tools/loop-context/test/cli.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,41 @@ test('cli rejects invalid --max-iterations values', () => {
assert.match(r.stderr, /--max-iterations must be a positive integer/);
});

test('cli rejects a percentage-style --similarity-threshold instead of disabling stagnation', () => {
// 95 was accepted and meant no two errors could ever match: the same failure
// repeated forever came back CONTINUE with exit 0.
for (const value of ['95', '100', '1.5', 'Infinity']) {
const r = runCli(['--check', '--similarity-threshold', value, '--json']);
assert.equal(r.status, 1, value);
assert.equal(r.stdout, '', `${value}: no decision is printed`);
assert.match(r.stderr, /--similarity-threshold must be a fraction greater than 0 and at most 1/, value);
assert.match(r.stderr, /stagnation rule would never fire/, value);
}
assert.match(runCli(['--check', '--similarity-threshold', '95']).stderr, /If you meant 95%, use 0\.95/);
});

test('cli rejects zero, negative and non-numeric --similarity-threshold', () => {
for (const value of ['0', '-0.5', 'abc']) {
const r = runCli(['--check', '--similarity-threshold', value, '--json']);
assert.equal(r.status, 1, value);
assert.ok(r.stderr.includes(`got ${value}.`), value);
}
});

test('cli accepts --similarity-threshold across (0, 1] and still trips stagnation', () => {
for (const value of ['1', '0.95', '0.5', '0.01']) {
const r = runCli(['--check', '--similarity-threshold', value, '--json']);
assert.equal(r.status, 2, value);
assert.equal(JSON.parse(r.stdout).trigger, 'stagnation', value);
}
});

test('cli validates --similarity-threshold for --prune too', () => {
const r = runCli(['--prune', '--similarity-threshold', '85']);
assert.equal(r.status, 1);
assert.match(r.stderr, /If you meant 85%, use 0\.85/);
});

test('cli rejects missing numeric flag values', () => {
const r = runCli(['--check', '--stagnation']);
assert.equal(r.status, 1);
Expand Down
Loading
Loading