Add DRY sampler stage (SS_DRY) - #278
Open
sashko-zakharchuk wants to merge 2 commits into
Open
Conversation
sashko-zakharchuk
force-pushed
the
dry-sampler
branch
from
August 9, 2026 13:55
08ff71f to
3d396c1
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the DRY (Don't Repeat Yourself) repetition penalty as
SS_DRY, thedry_multiplierhalfof #270 (the
logit_biashalf is #274). TabbyAPI currently warns "Ignoring sampler params notsupported by the exllamav3 backend: dry_multiplier"; I'll open the companion TabbyAPI PR to pass
the parameters through once this lands.
SS_DRYpenalizes a token that would extend a sequence already present in the context bymultiplier * base ** (match_length - allowed_length), so verbatim loops get exponentiallyexpensive while short incidental repeats are untouched. It is modeled on
SS_RepP: it runsamong the first steps on raw logits, participates in the fused-sampler fast path as a leading
step, no-ops when
dry_multiplier == 0ordry_base < 1, and is wired intoComboSamplerasdry_*. The scan is pure torch on whatever device holdspast_ids, with no host-device sync,so there is no extension change.
The matching semantics and the float32 exponent-overflow clamp follow llama.cpp's
llama_sampler_dry_apply(the koboldcpp implementation by pi6am, scheme by p-e-w), includingnever penalizing breaker tokens and the
base = 1.75/allowed_length = 2defaults. Theparameter surface follows exllamav2:
dry_multiplier,dry_base,dry_allowed_length,dry_range(0 = whole context, where llama.cpp's equivalent defaults to 64), anddry_sequence_breakersas token IDs (TabbyAPI's breaker strings resolve to IDs with theexported
dry_sequence_breaker_tokens). Left as None, the breaker set derives from thetokenizer at sampling time with exllamav2's charset rule
(
get_dry_default_sequence_breaker_tokens).Divergences, also stated in the docstrings:
occurrence count and caps matches at
dry_max_ngram = 20). This port follows llama.cppinstead, since that is what circulating DRY presets are tuned against. If you would rather
match the exl2 behavior, I can rework it.
sequences out of partially overlapping tokens; that machinery is not ported.
dry_allowed_lengthplus llama.cpp's exponentclamp exceeds 2048 (
dry_basebelow ~1.044 at the defaultdry_allowed_length), a verbatimrepeat longer than the cap is penalized as a 2048-token repeat. The cap is what keeps the
scan bounded for bases near 1.
Cost, measured on an RTX 5050 with a 128k vocabulary: 0.5 ms per sampled token at 1k-8k token
windows with the default
dry_base = 1.75, 6.6 ms at a 128k-token window. A small base widensthe scan up to the 2048 cap (
dry_base = 1.1gives 2.6 ms at 8k, 37 ms at 128k). Transientsare chunked, ~200 MiB worst case. The docstring says to cap
dry_rangefor very long contexts.Tests: ten deterministic cases (window bounds, breaker capping and exclusion, batch rows, the
float32 exponent-clamp boundary), a randomized differential against a brute-force oracle on
both CPU and CUDA
past_ids(200 cases each, covering chunk-boundary crossings andout-of-vocabulary IDs), default-breaker derivation through a stub tokenizer, and fused-collapse
assertions. Outputs were also cross-checked against the DRY port in vllm#50584: 1200/1200
randomized cases match. Sampler suite: 151 passed vs 137 on master, and clean with
EXL3_FUSED_SAMPLER=0.