Compaction

[nrfulton]

Compacting Context

Ramon et al. have proposed leading a push on compaction during this/next sprint.

Here are some notes from our design meeting.

Changes to make to Mellea's Standard Library

1. Add a Compactor protocol to stdlib

class Compactor(Protocol):
    def compact(backend: Backend, ctx: T <: Context) -> T:

We will add a mellea.stdlib.context module that has a compactor.py exposing the following protocol". Everything in mellea/stdlib/context.py will have to be refactored to live in that package, and __init__.py will have to expose the stuff currently in context.py so that this is minimally breaking.

Some important notes:

1. In this case, a generic compactor needs to be agnostic to the internal representation of a Context.
2. Compactors should never mutate an existing context. Figure out if we can do this with frozen.
3. The compaction should only copy over data that will be retained, and should copy over data where available.

2. Refactor existing "compaction" to use the Compactor Protocol

There is a window= in ChatHistory that needs to be reimplemented using a compactor.

class WindowCompactor(Compactor):
    def __init__(..., window_size:int=...):
        ...


class ChatContext(Context):
    ....

    def __init__(self, *, compactor: Compactor | None, window_size: int | None = None):
        if compactor is not None and window_size is not None:
		raise Exception("one or the other...")
        if compactor is None:
            window_size = 5 if window_size is None else window_size
            compactor = WindowCompactor(size=window_size)


    def add(self) -> ChatContext:
        calls compactor.compact()?
    
    def view_for_generation():
        # remove window management from call_for_generation, since now compaction is "real" as opposed to "virtual"

3. Document use Patterns (needs to be documented well -- not unregulated slop)

There are two use patterns, both of which are legitimate.

Pattern 1. In the ctx.add method (so the subclass of context that defines your app's special context knows what it means to compact itself)

ctx_0 # initial context
mot_1, ctx_1 = await backend.generate_from_context(backend, ctx_0, my_component)
# more code happens...
mot_i, ctx_i = await backend.generate_from_context(backend, ctx_{i-1}, yet_another_component)

Pattern 2. manually: we use a context without any compaction (or maybe with compaction, doesn't matter) but we want to do our own stuff

my_compactor = MyCompactor(...)


ctx_0 # initial context
mot_1, ctx_1 = await backend.generate_from_context(backend, ctx_0, my_component)
ctx_1 = my_compactor.compact(ctx_1)
# more code happens...
mot_i, ctx_i = await backend.generate_from_context(backend, ctx_{i-1}, yet_another_component)

One example of why you might want the manual compaction: if compaction is provided as a tool to the model.

Implement some simple examples and build up to react. Put the progression in docs/rewrite. Docs should be LLM-friendly.

Etc.

From here, it's not clear how to proceed. There are two different ways of defining compaction:

1. It's part of a context type
   -> not in stdlib unless that context type is in stdlib.
2. It's specific to a context type but there are multiple options for a given context type, but those compaction strategies won't for a generic context type
   -> these got in stdlib iff (1) that context type is defined in stdlib (simple, chat) and it's somehow essential
3. It's independent of a context type
   -> probably goes into a separate mellea-compactors or something.

There are probably middle grounds as well.

[ramon-astudillo]

@nrfulton just to confirm from our meeting that we agreed on (2) for "Etc."

We also talked about a react specific Compactor for react to live in the folder

https://github.com/generative-computing/mellea/tree/main/mellea/stdlib/frameworks/

cc @yelkurdi