Expand Memory.swift evals with metadata-aware smoke coverage and grouped gates

.agents/skills/memory-eval-design/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: memory-eval-design
+description: Design and implement Memory.swift evaluations as durable quality gates. Use when adding or changing eval datasets, metrics, scenario rows, baseline gates, or MemoryEvalCLI scoring for storage, recall, query expansion, and agent-memory behavior.
+---
+
+# Memory Eval Design
+
+Use this skill when the task is to add, reshape, or promote Memory.swift
+evaluations. Pair it with `memory-evals` when you need to run existing suites.
+
+This skill is based on Apple Evaluations guidance: treat evaluations as a
+repeatable specification, define measurable criteria before tuning behavior,
+use datasets with golden, edge, adversarial, and known-failure coverage, prefer
+code-based metrics for objective checks, reserve model-as-judge for subjective
+quality, inspect both aggregate and per-sample results, and protect against
+overfitting with fresh/held-out data.
+
+## Start Here
+
+Read these before editing:
+
+- `AGENTS.md`
+- `Evals/README.md`
+- the target dataset `manifest.json`
+- the relevant baseline in `Evals/baselines/`
+- `Sources/MemoryEvalCLI/MemoryEvalCLI.swift` when adding fields, metrics, or report output
+
+Keep generated or bulky exploratory data under `Explorations/Evals/` unless the
+dataset README or baseline manifest makes it part of the repo contract.
+
+## Turn Behavior Into Metrics
+
+Before adding rows or code, write down the behavior as measurable criteria:
+
+- **Subject under test**: storage extraction, recall/ranking, query expansion,
+  agent capture, prepared context, maintenance, or tool/bridge behavior.
+- **Expected output**: exact kind/status/key, relevant document IDs, required
+  text/facets/entities/topics, packed context contents, or proposal count.
+- **Must-not behavior**: false writes, assistant text contamination, benchmark
+  answer leakage, stale/superseded memories, irrelevant context, or over-budget
+  packing.
+- **Metric type**: binary pass rate for objective checks, numeric score for rank
+  or latency, model-as-judge only when subjective quality is unavoidable.
+- **Aggregation**: mean/pass rate for binary metrics, MRR/nDCG/recall for ranked
+  retrieval, median or p95 for latency-sensitive dimensions.
+
+If the criterion cannot be expressed as a stable metric, do not promote it into
+a release gate yet. Add it as exploratory diagnostics or a review queue item.
+
+## Choose The Dataset Surface
+
+- `Evals/memory_schema_gold_v2`: canonical write-path kind/status/facet/entity/topic/update behavior.
+- `Evals/agent_memory_gold_v1`: public agent workflow behavior: no-write, capture/extract, update lifecycle, recall, context prep, and maintenance.
+- `Evals/general_v2`: broad retrieval regression gate.
+- `Evals/longmemeval_v2`: long-horizon conversational recall benchmark.
+- `Evals/query_expansion_gold_v1`: query-expansion coverage and no-harm checks.
+- focused LongMemEval slices: use only as targeted regression/debug gates and
+  pair wins with a broader suite before promoting runtime changes.
+
+For new coverage, prefer extending the smallest relevant existing suite. Create
+a new committed dataset only when the behavior has a distinct purpose, manifest,
+and gate plan.
+
+## Dataset Design Checklist
+
+For each new or edited dataset, ensure:
+
+- clear `manifest.json` purpose, provenance, synthetic status, and review status
+- unique IDs and one dominant behavior per row
+- categories that cover golden path, edge cases, adversarial/refusal cases, and
+  known regressions
+- variation in input length, phrasing, ambiguity, entity/date density, and
+  user/profile style
+- at least one hard or negative row for any behavior likely to false-positive
+- no benchmark-specific rescue phrases, answer strings, IDs, or named facts in
+  production runtime logic
+- expected labels are auditable from the input and not hidden in the prompt
+- holdout or pressure rows remain separate until they are stable enough for a
+  release gate
+
+## Implementation Pattern
+
+1. Add or update JSONL rows with `apply_patch`.
+2. If new fields are needed, extend the Decodable structs in
+   `MemoryEvalCLI.swift`.
+3. Score the field in the suite runner and include per-case evidence in
+   `caseResults`.
+4. Export aggregate metrics through `reducedMetrics(from:)` if a baseline may
+   gate the value.
+5. Add the metric to console and Markdown summaries so failures are visible
+   without reading raw JSON.
+6. Update `Evals/README.md` only for durable schema/workflow changes.
+7. Update `Evals/baselines/*.json` only after fresh no-cache reports support
+   the new threshold.
+
+Do not hard-code baseline thresholds outside the manifests.
+
+## Validation
+
+For dataset or scoring changes, run:
+
+```bash
+swift run memory_eval validate-datasets --strict
+python3 Scripts/check_benchmark_leakage.py
+git diff --check
+```
+
+For code changes, also run:
+
+```bash
+swift test
+```
+
+For release-gate changes, run the affected suite with deterministic settings:
+
+```bash
+swift run --traits CoreMLEmbedding memory_eval run \
+  --profile coreml_default \
+  --dataset-root ./Evals/<dataset> \
+  --no-cache \
+  --no-index-cache
+```
+
+Then gate fresh reports. If using `current.json`, provide all required run JSONs;
+for a focused check, create a temporary baseline containing only the changed
+requirements and state clearly that it is not the full release gate.
+
+## Report Back
+
+Summarize:
+
+- changed dataset roots and metric fields
+- fresh report JSON/Markdown paths
+- old vs new headline metrics when relevant
+- whether baseline manifests changed
+- validation commands and any suites not rerun

.agents/skills/memory-synthetic-datasets/SKILL.md

@@ -0,0 +1,167 @@
+---
+name: memory-synthetic-datasets
+description: Generate, review, and promote synthetic Memory.swift evaluation datasets. Use when creating synthetic storage, recall, query-expansion, or agent-memory samples; expanding seed cases; auditing generated JSONL; or preparing synthetic data for Memory.swift eval gates.
+---
+
+# Memory Synthetic Datasets
+
+Use this skill when the user wants new synthetic evaluation data or wants to
+expand a small seed set into a larger Memory.swift eval dataset.
+
+This skill follows Apple Evaluations dataset guidance: start from high-quality
+manual seeds, expand by category instead of asking for generic diversity,
+validate generated samples programmatically and manually, watch for duplicates
+and distribution skew, keep hard/refusal cases represented, and promote only
+reviewed data into durable gates.
+
+## Start Here
+
+Read before generating or promoting data:
+
+- `AGENTS.md`
+- `Evals/README.md`
+- `.agents/skills/memory-evals/SKILL.md` if you will run reports
+- the target dataset `manifest.json`
+- generator scripts relevant to the target suite
+
+Useful scripts:
+
+- `Scripts/generate_eval_data_minimax.py`
+- `Scripts/generate_eval_data_codex.py`
+- `Scripts/generate_agent_memory_scenarios.py`
+- `Scripts/tag_eval_data_codex.py`
+- `Scripts/tag_eval_data_minimax.py`
+- `Scripts/build_audit_packet.py`
+- `Scripts/audit_eval_data.py`
+- `Scripts/merge_audit_results.py`
+- `Scripts/merge_eval_corpora.py`
+
+## Decide The Target
+
+Before generation, define:
+
+- dataset root and suite files (`storage_cases.jsonl`, `recall_documents.jsonl`,
+  `recall_queries.jsonl`, `query_expansion_cases.jsonl`, or `scenarios.jsonl`)
+- whether data is exploratory, pressure/held-out, or release-gate intended
+- target count and counts per category
+- seed source and whether seeds are human-written, synthetic, external, or mixed
+- required labels and fields for every row
+- validation and review plan
+
+Default new generation goes under `Explorations/Evals/<name>/`. Do not overwrite
+curated committed datasets unless the user explicitly asks for promotion or
+regeneration.
+
+## Seed Matrix
+
+Create a small hand-written seed matrix before bulk generation. For Memory.swift,
+use axes such as:
+
+- memory kind: profile, fact, decision, commitment, episode, procedure, handoff
+- lifecycle: append, dedupe, replace active, supersede, resolve, archive
+- retrieval shape: exact lookup, paraphrase, temporal, aggregate, multi-evidence,
+  contextual ellipsis, entity/topic-heavy, procedure-like
+- agent behavior: no-write, capture, update, prepared context, maintenance,
+  assistant-text contamination, tool/bridge behavior
+- difficulty: easy, medium, hard, adversarial/refusal
+- phrasing: terse, conversational, typo/noisy, ambiguous, long-context
+
+Keep at least 20-30 percent of seeds genuinely hard or negative. Include rows
+where correct behavior is to refuse, write nothing, mark ambiguity, or preserve
+only partial evidence.
+
+## Generation Strategy
+
+Generate by category or by cross-product slices, not one broad prompt. A focused
+prompt gives better control over representation and makes review easier.
+
+For each generation batch:
+
+- include the exact JSONL schema and accepted enum values
+- state the category, difficulty, and required variation
+- require labels to be justified by visible input text
+- prohibit benchmark IDs, exact answer phrases from external benchmarks, and
+  hidden labels in prompts
+- ask for commercially safe synthetic content unless an external source is
+  explicitly allowed
+- ask for unique IDs with a stable prefix
+- keep expected outputs concise and auditable
+
+When using repo generators, prefer local exploratory roots first:
+
+```bash
+python3 Scripts/generate_eval_data_minimax.py \
+  --dataset-root ./Explorations/Evals/<name> \
+  --dataset-mode tech \
+  --overwrite
+```
+
+For agent-memory scenarios:
+
+```bash
+python3 Scripts/generate_agent_memory_scenarios.py \
+  --dataset-root ./Explorations/Evals/<name> \
+  --overwrite
+```
+
+Use `--backend minimax` or Codex/Minimax taggers only when credentials and cost
+are acceptable. Review generated rows before promoting them.
+
+## Validation Layers
+
+Run programmatic checks:
+
+```bash
+swift run memory_eval validate-datasets --strict ./Explorations/Evals/<name>
+python3 Scripts/check_benchmark_leakage.py
+```
+
+Inspect distribution:
+
+- row counts by category/source family/difficulty
+- duplicate IDs and near-duplicate prompts
+- missing required fields
+- empty or unauditable expected outputs
+- over-representation of easy happy-path rows
+- answer leakage in user prompts, documents, or expected text
+
+Do manual review before relying on generated rows. Sample at least 5-10 percent,
+and always review all hard, adversarial, no-write, refusal, and newly introduced
+schema-field rows.
+
+If the dataset is large or generated by a model, build an audit packet:
+
+```bash
+python3 Scripts/build_audit_packet.py --dataset-root ./Explorations/Evals/<name>
+python3 Scripts/audit_eval_data.py --packet ./Evals/_audit/<name>/packet.jsonl --backend opencode
+python3 Scripts/merge_audit_results.py --packet ./Evals/_audit/<name>/packet.jsonl
+```
+
+Keep review queues and draft sidecars out of release-gate roots unless they are
+part of the documented workflow.
+
+## Promotion Workflow
+
+Promote only after validation and review:
+
+1. Copy selected rows into the committed dataset with `apply_patch`.
+2. Update the dataset `manifest.json` provenance, synthetic status, and review
+   status.
+3. Run `validate-datasets --strict`.
+4. Run the affected eval suite with `--no-cache --no-index-cache`.
+5. Compare/gate against the relevant baseline.
+6. Update `Evals/baselines/*.json` only when the gate change is intentional and
+   supported by fresh reports.
+
+Never tune production retrieval or extraction logic to exact synthetic rows.
+Treat synthetic data as coverage pressure; keep product logic generic.
+
+## Report Back
+
+Include:
+
+- generated root and files changed
+- counts by category and difficulty
+- review method and remaining review gaps
+- validation commands and report paths
+- whether anything was promoted into a committed gate

Evals/README.md

@@ -8,6 +8,23 @@ Each dataset folder can also include a `manifest.json` describing provenance, wh
 
 The default committed suite should remain commercially safe. If you want to experiment with stronger non-commercial-only eval sets, keep them under `Evals/local_nc/` so they stay out of the public default workflow.
 
+## Shared Row Metadata
+
+Runnable row files may include metadata fields so reports can slice metrics by
+coverage area and provenance:
+
+- `case_category`: `golden`, `edge`, `adversarial`, `known_regression`, `holdout`, or `synthetic_expansion`.
+- `source_family`: free-form stable family label such as `profile_update`, `no_write`, `maintenance`, or `context_preparation`.
+- `difficulty`: `easy`, `medium`, or `hard`.
+- `generation_method`: `seed`, `template`, `synthetic_seed`, `model_generated`, `converted`, or `external`.
+- `review_status`: `curated`, `reviewed`, or `needs_review`.
+- `synthetic_status`: `human`, `synthetic`, `external`, or `hybrid`.
+
+If a dataset manifest sets `metadata_required: true`, every storage case,
+recall query, query-expansion case, and agent-memory scenario must include all
+six fields. When present, enum values are validated by
+`swift run memory_eval validate-datasets --strict`.
+
 ## Files
 
 ### `storage_cases.jsonl`
@@ -22,7 +39,10 @@ One JSON object per line. The primary supported shape is the canonical memory sc
   "expected_status": "active",
   "expected_facets": ["preference", "project", "fact_about_user"],
   "required_entities": ["zac", "memory.swift", "zed"],
-  "required_topics": ["memory.swift work"]
+  "required_topics": ["memory.swift work"],
+  "expected_subject": "user",
+  "required_evidence_roles": ["user"],
+  "forbidden_text_contains": ["assistant refusal"]
 }
 ```
 
@@ -35,6 +55,9 @@ Fields:
 - `expected_facets` (canonical): expected fixed facet tags.
 - `required_entities` (canonical): normalized entity values that must be extracted.
 - `required_topics` (canonical): normalized topic phrases that must be extracted.
+- `expected_subject` (canonical, optional): expected subject classification such as `user`.
+- `required_evidence_roles` (canonical, optional): evidence roles that must survive into the stored memory.
+- `forbidden_text_contains` (canonical, optional): substrings that must not be copied into the stored memory text.
 - `expected_update_behavior` (canonical, optional): `replace_active`, `merge_status`, `append`, `dedupe`, or `supersede`.
 - `canonical_key` (canonical, optional): explicit canonical-key override for update-behavior cases.
 - `setup_memories` (canonical, optional): seed memories inserted before the evaluated write.
@@ -78,6 +101,43 @@ Fields:
 - `relevant_document_ids` (required): IDs from `recall_documents.jsonl`.
 - `memory_types` (optional, legacy metadata): preserved for dataset provenance, but no longer used as an active retrieval filter.
 
+### `scenarios.jsonl`
+One agent-memory scenario per line. These rows exercise the public agent workflow surface: capture/extract, ingest/update, recall, prepared context, and maintenance:
+
+```json
+{
+  "id": "profile-location-embedded-question",
+  "workflow": "capture",
+  "messages": [
+    {"role": "user", "content": "i live in sf, what's a fun thing to do tonight there?"},
+    {"role": "assistant", "content": "I don't have real-time suggestions."}
+  ],
+  "expected_write_count": 1,
+  "expected_memories": [
+    {
+      "kind": "profile",
+      "status": "active",
+      "canonical_key": "profile:user:location",
+      "text_contains": ["San Francisco"],
+      "subject": "user",
+      "required_evidence_roles": ["user"],
+      "required_evidence_source_ids": ["profile-location-embedded-question"],
+      "forbidden_text_contains": ["real-time suggestions"]
+    }
+  ]
+}
+```
+
+Fields:
+- `workflow` (optional): `extract_ingest` for the legacy two-step path or `capture` for `MemoryIndex.capture`.
+- `messages` (required): conversation messages used for extraction or capture.
+- `setup_memories` (optional): seed memories inserted before the evaluated workflow.
+- `setup_context_hints` (optional): durable hints inserted before context-preparation checks.
+- `expected_write_count` / `expected_memories`: write-count and matched-memory expectations. Expected memories may assert kind, status, canonical key, contained text, facets, entities, topics, subject, evidence roles/source IDs, and forbidden copied text.
+- `recall_queries` (optional): recall assertions over the scenario index after writes.
+- `context_expectations` (optional): calls `prepareContext` and checks required/forbidden context text, required hints, untrusted framing, reference limits, and token budgets.
+- `maintenance_expectation` (optional): records recall signals, runs maintenance in preview mode, and checks proposal text/count plus forbidden proposal text.
+
 ### `review_queue.jsonl`
 Optional sidecar generated by `Scripts/tag_eval_data_codex.py` or `Scripts/tag_eval_data_minimax.py`: