feat: Phase 2 — validated streaming

[planetf1]

Context

mellea/stdlib/streaming.py (landing in PR #942, closing #901) provides Phase 1 streaming validation — a call-site ChunkingStrategy with three built-in chunkers (SentenceChunker, WordChunker, ParagraphChunker) and an orchestrator (stream_with_chunking). This is a scoped pragmatic choice. Phase 2 refines that foundation and extends it, broken into the sub-issues below.

Consolidated design summary

Motivation

Phase 1 collapses two semantic concerns onto the call site: how to chunk the stream (a property of the output type) and how to validate chunks (a property of the requirement). These are independent and belong in different owners.

• Type semantics. What counts as a complete chunk of this kind of output? JSON value, prose sentence, code statement, audio segment, image region. Invariant across requirements.
• Constraint semantics. What makes a particular output acceptable? Max three sentences, matches schema X, no hallucinated entities. Invariant across outputs of the same type.

Under Phase 1, both are author-written at the call site. Phase 2 separates them: chunking becomes a per-requirement concern, and the streaming parsed representation becomes its own capability.

Motivating output types (all in scope)

Phase 1 chunkers cover prose only (sentence/word/paragraph, all operating on accumulated_text: str). Phase 2 must support at least these output types, each with genuinely different chunk semantics:

• Prose — sentence, word, paragraph boundaries. Already covered by Phase 1 chunkers; Phase 2 should subsume them.
• Structured text — JSON values, YAML documents, code statements/blocks. Chunk boundary is "one complete parseable unit."
• Multi-modal streams — audio (silence-delimited segments, fixed windows, VAD-detected utterances), image (region or tile boundaries), potentially video. Chunk boundary is inherently non-string.

Multi-modal is first-class motivation for this work, not deferred scope. Epic #891 explicitly names the audio case ("Audio that goes wrong in the first few seconds can't be caught until the full clip is done"), and the ChunkingStrategy.split(accumulated_text: str) -> list[str] signature in Phase 1 forecloses on multi-modal by design — that foreclosure is what this issue exists to address.

Sub-issues

Phase 2 is broken into four sub-issues (an earlier version of this issue proposed moving chunking onto the MOT; the direction has since changed — chunking is handled per-Requirement, and stream_parsed_repr is a separate concern from chunking):

• refactor: replace stream_with_chunking with single-task stream() #1440 — Refactor: single-task stream() primitive (blocks all others) — replace the two-task stream_with_chunking + StreamChunkingResult with a single stream()/Streamer consumed by a plain async for. Includes ModelOutputThunk.__aiter__, event emission via the STREAMING_EVENT hook, an events-consumption example, and factoring chunk-boundary state into a Chunker object. A working POC exists on PR feat: validated streaming refactor POC + Phase 2 epic breakdown #1409.
• feat: per-requirement chunking strategies #1441 — Per-validator chunking — each Requirement carries its own ChunkingStrategy so different validators can validate at different granularities. Open: where the per-validator chunk state lives.
• feat: allow streaming for sampling results #403 — Sampling integration — give stream() the instruct-validate-repair retry/repair behavior SamplingStrategy provides for the non-streaming path.
• feat(core): stream_parsed_repr — streaming parsed representation on the MOT #1442 — stream_parsed_repr — a streaming counterpart to parsed_repr: expose the MOT's typed parsed representation incrementally, separate from chunking and validation. Multi-modal folds in here.

#1440 blocks the rest; #1441, #403, and #1442 depend only on it and can proceed in parallel.

Open design questions (for comments)

1. Signature and generator shape. Is stream_parsed_repr an async generator on the MOT? Does it share a queue with the raw astream(), or run in parallel? For multi-modal, does the signature accept bytes / frames / tensors rather than str?
2. Chunking boundary authority. Which component decides what a "complete chunk" is — the MOT's parser, a pluggable chunk-boundary predicate, or both? Answer likely differs between text and multi-modal.
3. Backpressure. If parsed chunks are slower to produce than raw tokens/frames, where does the buffering live?
4. Backwards compatibility. How to migrate from the Phase 1 call sites?
5. Typed output. Does stream_parsed_repr yield values of type S (the MOT's parsed_repr type parameter), or a richer container that carries partial-parse state? Multi-modal MOTs may need the latter.
6. Error handling. What happens if the MOT's parser fails on a partial stream — surface immediately, wait for more data, or fall back to raw chunking?
7. Testability. Each MOT type's stream_parsed_repr needs verification against its non-streaming parsed_repr. Shape of the shared test harness? Multi-modal test fixtures are their own problem.
8. Agent authoring. If we want new MOT types to be agent-writable (see PR feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942 discussion), what contracts are needed on the MOT base class? Clear APIs and guidelines are the substrate; a skill can sit on top where the framework supports one.

Broader scope (to discuss)

The PR #942 thread raised a parallel observation. Both stream_validate authoring and (future) stream_parsed_repr authoring have deterministic checks against a non-streaming counterpart (validate() and parsed_repr respectively), which makes them plausible candidates for agent-friendly extension patterns — potentially skills in frameworks that support them. Worth discussing whether this issue should cover just stream_parsed_repr or also the broader agent-authoring story, or whether the latter warrants its own separate issue.

Dependencies

• Phase 1 streaming validation must be complete: feat(stdlib): implement stream_with_chunking() with per-chunk validation #901 (PR feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942, in review), feat(stdlib): add standard streaming event types #902 (deferred event types and orchestrator-level OTEL).
• May interact with Epic: ModelOutputThunk structural cleanup #909 (MOT structural cleanup) — worth coordinating to avoid base.py collision.

References

• Parent epic: feat: streaming validation — per-chunk requirement checking with early exit #891
• Phase 1 PR: feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942
• Discussion origin: PR feat(stdlib): add stream_with_chunking() with per-chunk validation (#901) #942 review comment thread on mellea/stdlib/__init__.py and mellea/stdlib/chunking.py

[planetf1]

Concrete Phase 1 data point worth recording here, from the current PR #942 review cycle.

When a streaming delta produces multiple chunks and chunk N fails validation, the orchestrator has to maintain two separate views of the text:

• accumulated — the full raw text, needed by the external ChunkingStrategy to maintain its split state across calls (split(accumulated) is how the chunker tracks where it left off)
• emitted_text — only the chunks that actually passed validation and reached the consumer

These diverge on early exit, and we had to introduce a second tracking variable to bridge them. The fix is clean enough, but the reason it's needed at all is that chunking is outside the MOT — the chunker needs the full accumulation to do its job, while full_text should only reflect what was validated. If the MOT owned the chunker, both views would be internal and there'd be no split to paper over.

Small evidence, but it's a concrete instance of the "two semantic concerns collapsed onto the call site" argument in the issue description above.

[jakelorocco]

@planetf1, can this be closed now?

[ajbozarth]

Coordination note from #909 design research (Discussion #1197): #909's Area 3 proposes splitting MOT computation logistics into a _GenerationState sub-object. That shape is intentionally designed to accommodate this issue's stream_parsed_repr parsed-repr stream buffer as a future field, so #1013 benefits if Area 3 lands first.

Not blocking either direction; flagging so they don't collide later.