Repository navigation
Cap the semantic cache with approximated LRU eviction - #59
Draft
tchalikanti1705 wants to merge 12 commits into
Draft
tchalikanti1705 wants to merge 12 commits into
tchalikanti1705 wants to merge 12 commits into
Conversation
Item carries ExpireUnix (0 means no expiry) and Set takes it as an argument, plumbed through store.VSet. Every caller passes 0 for now, so nothing changes behavior yet; the semantic cache starts using it once search and the sweeper understand it. SweepExpired removes every item whose deadline has passed, for the sweeper to call.
Search takes the current Unix time and ignores items whose deadline has passed (0 disables the check, which is what every non-store caller passes). An expired entry can no longer outrank a live one or win top-1 at all; it stays in memory until the sweeper reclaims it. store.VSearch passes the store clock through.
The sweeper now walks into vector collections and removes items whose deadline has passed, dropping the key when a collection empties, the same as VDel does. This is what reclaims a cache entry that expires and is never queried again; lazy expiry alone can never reach it.
VecRecord gains ExpireUnix. Old snapshots simply lack the field and gob decodes it as 0, so pre-upgrade entries load as never expiring, and old binaries reading new snapshots ignore the extra field. Items already dead at save or load time are dropped instead of being written out or resurrected, and a collection that comes back empty creates no key.
SCACHE.SET puts the deadline on the vector item, so search skips expired entries and the sweeper reclaims them without anyone querying. SCACHE.GET still honors the deadline in the meta JSON, but checks it before the threshold now: entries from pre-upgrade snapshots have no item deadline, and the old order meant one that expired and then scored below the threshold was never dropped.
Covers the acceptance list from Rohit-Dnath#55: search skips an expired item the moment its deadline passes and a live one wins the slot; the sweeper reclaims expired entries (and all-expired keys) with the clock advanced, not slept; snapshots carry deadlines, drop dead items at save and load, and load pre-upgrade records as never expiring; and over the wire, a legacy meta-only entry scoring below the threshold is reclaimed on lookup instead of surviving forever. The wire test runs against a fake embeddings provider with fixed vectors so scores are exact.
A huge TTL overflowed secs multiplied by time.Second into a wrapped, possibly past deadline; now that the sweeper acts on deadlines, that would silently delete the entry. A non-positive TTL cached an entry already dead on arrival. Both are rejected up front with the same guard and error shape as SETEX. Also cover the save path for a collection whose items have all expired: it writes no record.
Each item records when it was last inserted or hit. store.VSet stamps the insert so a brand-new entry is never the LRU pick, and a semantic cache hit refreshes its entry through the new VTouch, so hot entries outlive a cap. Not persisted: snapshot-loaded items start at zero and rank oldest until used again.
EvictLRU removes the oldest-accessed of a small sample until the collection fits, Redis-style approximated LRU with no ordered structure. VEnforceCap sweeps expired items first, since reclaiming a dead entry is always better than evicting a live one, then applies the LRU pass.
SCACHE.SET enforces a max-entries bound after each insert, so sustained distinct prompts hold the cache at the cap instead of growing without limit. Configurable via --scache-max / RAMEN_SCACHE_MAX, default 10000 (at that size a lookup costs ~6ms, a few percent of the LLM call it saves); 0 explicitly disables the bound. Plain VSET collections are not capped.
Whole seconds tie for every access inside the same second, which made the LRU victim arbitrary during a burst. UnixNano keeps insertion and hit order exact; the expiry deadline stays in seconds to match TTL.
EvictLRU and VEnforceCap unit tests use few enough items to sit inside one sample, so eviction order is exact and deterministic. The wire test holds a cap of 2 while distinct prompts arrive: VCARD stays at the cap, the least-recently-used entry is the one evicted, a hit protects its entry, and replacing an existing prompt evicts nobody. Expired entries are reclaimed before any live one is evicted.
tchalikanti1705
force-pushed
the
cap-scache-collection
branch
from
July 19, 2026 02:01
ceae622 to
1ef2bb7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Bounds
__scache__to a configurable max, evicting approximately the least-recently-used entry once distinct prompts push past it. Part 2 of #55 — together with #58 this closes it.Stacked on #58 (it builds on the expiry plumbing there), so this diff includes those commits until #58 merges; the new work is the last 5 commits. Marked draft until #58 lands — happy to rebase it the moment that merges.
Related Issue
Part 2 of #55; closes #55 once both land.
Changes
SCACHE.SETstamps the insert and a cache hit refreshes its entry via the newVTouch, so hot entries outlive the capEvictLRUremoves the oldest of a small sample until the collection fits (Redis-style approximated LRU, no ordered structure);VEnforceCapsweeps expired items first, since reclaiming a dead entry always beats evicting a live oneSCACHE.SETenforces the cap after each insert; plainVSETcollections are never capped--scache-max/RAMEN_SCACHE_MAX, default 10000 (at that size a lookup is ~6ms, a few percent of the LLM call it saves — happy to change the number if you want a different default);0explicitly disables the boundVerified live: with
--scache-max 3, five distinct prompts holdVCARD __scache__at exactly 3.Checklist
go test ./...)go fmt ./...)