Skip to content

Add indirect prompt injection probe, tool messages - #2058

Draft
erickgalinkin wants to merge 9 commits into
NVIDIA:mainfrom
erickgalinkin:feature/ipi
Draft

Add indirect prompt injection probe, tool messages#2058
erickgalinkin wants to merge 9 commits into
NVIDIA:mainfrom
erickgalinkin:feature/ipi

Conversation

@erickgalinkin

Copy link
Copy Markdown
Collaborator

Draft PR for discussion. Adds "tool" role to messages, necessary tool calling params. Will need to adapt a bit more.

Have not verified yet.

@ABeltramo

Copy link
Copy Markdown
Contributor

We are exploring the same problem from a slightly different angle over on our fork at RedHat, so I wanted to flag it early to avoid duplicated effort and hopefully converge.

At a high level, our approach is replay-based rather than in-tree construction. We are producing harmful conversations + tools as part of a different process (asago artifact generator) so a payload can be injected anywhere in the dialogue, including as a tool-call result. Garak then replays each conversation unchanged against the target and judges the response with an LLM-as-a-judge.

Concretely, we've been building out:

  • attempt.py: same "tool" role addition you have. One difference worth comparing notes on: rather than adding tool_calls/name as new fields on Message/Turn, we route the tool metadata (tool_calls / tool_call_id / name) through Message.notes so the dataclass schema stays unchanged and tool turns round-trip cleanly through Conversation.from_openai. Curious which direction you'd prefer upstream; happy to align to yours.
  • generators/openai.py (OpenAICompatible chat path) - serialize tool turns outbound, forward per-conversation tools/tool_choice, and capture the model's response tool_calls so a detector can inspect them.
  • A judge detector: scores each response against a per-conversation, natural-language success description and the emitted tool calls, so an attack that succeeds via a harmful tool call (benign-looking text) is still caught.

Leaving aside the actual attack vector/probe I think it would be nice to get a solid tool foundation both in Generators and Detectors so that we can all build on top..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Progress

Development

Successfully merging this pull request may close these issues.

feature: First-class support for Tool messages

3 participants