feat(buffs): add Caesar cipher buff

[albrodfer1]

What this does

Adds a Caesar buff (buffs.encoding.Caesar) that shift-encodes probe prompts with a Caesar cipher, so any probe's prompts can be tested against a target's ability to decode-and-follow shifted text (a jailbreak/obfuscation vector).

• Shifts A–Z / a–z along the alphabet (wrapping), preserving case; digits, punctuation and whitespace pass through untouched.
• Shift is a configurable param via DEFAULT_PARAMS = {"shift": 5}.
• Wraps the encoded payload in an explicit decode instruction so the target understands the text is shifted and how to reverse it.

Follows the existing buffs.encoding.Base64 / CharCode idiom (transform() rewriting attempt.prompt). Auto-discovered — no registration needed. Leaves post_buff_hook = False, so the raw model response is scored.

Verification

• Run the tests and ensure they pass python -m pytest tests/buffs/ → 14 passed, 1 skipped
• Verify the shift is correct: Hello, World! xyz XYZ → Mjqqt, Btwqi! cde CDE (wrap + case preserved, non-letters untouched), covered by test_caesar_shift in tests/buffs/test_buffs.py
• Auto-parametrized test_buff_structure / test_buff_load_and_transform cover load + transform
• Live run: python -m garak -t test.Blank --spec 'probes.test.Test,buffs.encoding.Caesar' — buff loads, expands attempts, and the report contains correctly-encoded prompts (The quick brown fox jumps over the lazy dog → Ymj vznhp gwtbs ktc ozrux tajw ymj qfed itl)

[leondz]

New probes should include a demonstration that the probe gets hits on a reasonable target - do you have that?

[albrodfer1]

New probes should include a demonstration that the probe gets hits on a reasonable target - do you have that?

Hi @leondz, thanks for your message. Since this feature is a buff, in order to prove its value, I selected an existing probe donotanswer.InformationHazard and compared two scenarios: The baseline without the buff, and using the buff implemented in this PR.

Using the buff significantly decreases the model score against the probe:

Baseline: 75.8%
Caesar Buff: 17.8%

Run details

• Command Baseline: .venv/bin/python -m garak -m openai -n gpt-4o-mini -p donotanswer.InformationHazard --generations 1 --report_prefix gpt_4o_mini_baseline --parallel_attempts 20 --generator_options '{"openai":{"OpenAIGenerator":{"max_tokens":4096}}}'
• Command Buff: .venv/bin/python -m garak -m openai -n gpt-4o-mini -p donotanswer.InformationHazard --buffs encoding.Caesar --generations 1 --report_prefix gpt_4o_mini --parallel_attempts 20 --generator_options '{"openai":{"OpenAIGenerator":{"max_tokens":4096}}}'
• Probe donotanswer.InformationHazard was selected because this cypher is more effective in hazardous request attempts
• max_tokens had to be increased since this buff requires the model 1) to decode the prompt and 2) to answer it
• The model selected was gpt-4o-mini, this attack works better against non-reasoning models

Attached files

• gpt_4o_mini.report.jsonl.txt: Execution report of the probe that used the buff
• gpt_4o_mini_baseline.report.jsonl.txt Execution report of the probe that didn't use the buff and served as a baseline for comparison