Skip to content

Add DRY sampler stage (SS_DRY) - #278

Open
sashko-zakharchuk wants to merge 2 commits into
turboderp-org:masterfrom
sashko-zakharchuk:dry-sampler
Open

Add DRY sampler stage (SS_DRY)#278
sashko-zakharchuk wants to merge 2 commits into
turboderp-org:masterfrom
sashko-zakharchuk:dry-sampler

Conversation

@sashko-zakharchuk

Copy link
Copy Markdown
Contributor

Adds the DRY (Don't Repeat Yourself) repetition penalty as SS_DRY, the dry_multiplier half
of #270 (the logit_bias half is #274). TabbyAPI currently warns "Ignoring sampler params not
supported by the exllamav3 backend: dry_multiplier"; I'll open the companion TabbyAPI PR to pass
the parameters through once this lands.

SS_DRY penalizes a token that would extend a sequence already present in the context by
multiplier * base ** (match_length - allowed_length), so verbatim loops get exponentially
expensive while short incidental repeats are untouched. It is modeled on SS_RepP: it runs
among the first steps on raw logits, participates in the fused-sampler fast path as a leading
step, no-ops when dry_multiplier == 0 or dry_base < 1, and is wired into ComboSampler as
dry_*. The scan is pure torch on whatever device holds past_ids, with no host-device sync,
so there is no extension change.

The matching semantics and the float32 exponent-overflow clamp follow llama.cpp's
llama_sampler_dry_apply (the koboldcpp implementation by pi6am, scheme by p-e-w), including
never penalizing breaker tokens and the base = 1.75 / allowed_length = 2 defaults. The
parameter surface follows exllamav2: dry_multiplier, dry_base, dry_allowed_length,
dry_range (0 = whole context, where llama.cpp's equivalent defaults to 64), and
dry_sequence_breakers as token IDs (TabbyAPI's breaker strings resolve to IDs with the
exported dry_sequence_breaker_tokens). Left as None, the breaker set derives from the
tokenizer at sampling time with exllamav2's charset rule
(get_dry_default_sequence_breaker_tokens).

Divergences, also stated in the docstrings:

  • exllamav2's own DRY is a different algorithm (an incremental trie that weights the penalty by
    occurrence count and caps matches at dry_max_ngram = 20). This port follows llama.cpp
    instead, since that is what circulating DRY presets are tuned against. If you would rather
    match the exl2 behavior, I can rework it.
  • Sequence breakers are single tokens. llama.cpp additionally builds multi-token restart
    sequences out of partially overlapping tokens; that machinery is not ported.
  • Match lengths are capped at 2048 tokens. Where dry_allowed_length plus llama.cpp's exponent
    clamp exceeds 2048 (dry_base below ~1.044 at the default dry_allowed_length), a verbatim
    repeat longer than the cap is penalized as a 2048-token repeat. The cap is what keeps the
    scan bounded for bases near 1.

Cost, measured on an RTX 5050 with a 128k vocabulary: 0.5 ms per sampled token at 1k-8k token
windows with the default dry_base = 1.75, 6.6 ms at a 128k-token window. A small base widens
the scan up to the 2048 cap (dry_base = 1.1 gives 2.6 ms at 8k, 37 ms at 128k). Transients
are chunked, ~200 MiB worst case. The docstring says to cap dry_range for very long contexts.

Tests: ten deterministic cases (window bounds, breaker capping and exclusion, batch rows, the
float32 exponent-clamp boundary), a randomized differential against a brute-force oracle on
both CPU and CUDA past_ids (200 cases each, covering chunk-boundary crossings and
out-of-vocabulary IDs), default-breaker derivation through a stub tokenizer, and fused-collapse
assertions. Outputs were also cross-checked against the DRY port in vllm#50584: 1200/1200
randomized cases match. Sampler suite: 151 passed vs 137 on master, and clean with
EXL3_FUSED_SAMPLER=0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant