-
-
Notifications
You must be signed in to change notification settings - Fork 921
feat(core): add execution validators #1219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| "@voltagent/core": minor | ||
| "@voltagent/server-core": patch | ||
| --- | ||
|
|
||
| feat(core): add pre-execution validators for tools and workflows | ||
|
|
||
| Agents and workflows can now define `executionValidators` that run before tool execution or | ||
| workflow step execution. Validators can return `false` or `{ pass: false }` to block the run with | ||
| an `ExecutionValidationError`, including a custom message, code, HTTP status, and metadata. They | ||
| can run synchronously or asynchronously. | ||
|
|
||
| ```ts | ||
| const agent = new Agent({ | ||
| name: "Policy Controlled Assistant", | ||
| instructions: "You enforce tenant policy.", | ||
| model: "openai/gpt-4o", | ||
| tools: [updateRecordTool], | ||
| executionValidators: { | ||
| tools: [ | ||
| async ({ toolName, operationContext }) => { | ||
| const tenant = operationContext?.requestHeaders?.["x-tenant-id"]; | ||
| const allowed = await checkTenantToolAccess({ tenant, toolName }); | ||
|
|
||
| if (toolName === "update_record" && !allowed) { | ||
| return { | ||
| pass: false, | ||
| message: "This tenant cannot update records.", | ||
| code: "TOOL_TENANT_DENIED", | ||
| httpStatus: 403, | ||
| }; | ||
| } | ||
| }, | ||
| ], | ||
| }, | ||
| }); | ||
| ``` | ||
|
|
||
| Server-core direct tool and workflow handlers now preserve `ClientHTTPError` details so blocked | ||
| executions can return the validator's status and code instead of a generic 500. | ||
|
|
||
| Fixes #1213 | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -48,6 +48,11 @@ import { | |
| validateUIMessages, | ||
| } from "ai"; | ||
| import { z } from "zod"; | ||
| import { | ||
| type AgentExecutionValidators, | ||
| type ToolExecutionValidationContext, | ||
| runExecutionValidators, | ||
| } from "../execution-validation"; | ||
| import { LogEvents, LoggerProxy } from "../logger"; | ||
| import { ActionType, buildAgentLogMessage } from "../logger/message-builder"; | ||
| import { Memory } from "../memory"; | ||
|
|
@@ -97,6 +102,7 @@ import { | |
| createVoltAgentError, | ||
| isBailError, | ||
| isClientHTTPError, | ||
| isExecutionValidationError, | ||
| isMiddlewareAbortError, | ||
| isToolDeniedError, | ||
| isVoltAgentError, | ||
|
|
@@ -279,6 +285,20 @@ const firstDefined = <T>(...values: Array<T | null | undefined>): T | undefined | |
| return undefined; | ||
| }; | ||
|
|
||
| const normalizeAgentExecutionValidators = ( | ||
| validators?: AgentExecutionValidators, | ||
| ): AgentExecutionValidators | undefined => { | ||
| const tools = validators?.tools?.filter((validator) => typeof validator === "function") ?? []; | ||
| return tools.length > 0 ? { tools: [...tools] } : undefined; | ||
| }; | ||
|
|
||
| const mergeAgentExecutionValidators = ( | ||
| ...configs: Array<AgentExecutionValidators | undefined> | ||
| ): AgentExecutionValidators | undefined => { | ||
| const tools = configs.flatMap((config) => config?.tools ?? []); | ||
| return tools.length > 0 ? { tools } : undefined; | ||
| }; | ||
|
|
||
| type OpenRouterUsageCost = { | ||
| cost?: number; | ||
| isByok?: boolean; | ||
|
|
@@ -902,6 +922,9 @@ export interface BaseGenerationOptions<TProviderOptions extends ProviderOptions | |
| // Hooks (can override agent hooks) | ||
| hooks?: AgentHooks; | ||
|
|
||
| // Execution validators (can add per-call validators) | ||
| executionValidators?: AgentExecutionValidators; | ||
|
|
||
|
Comment on lines
+925
to
+927
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🛠️ Refactor suggestion | 🟠 Major Strip
Proposed fix parentOperationContext,
hooks,
+ executionValidators: _executionValidators,
feedback: _feedback,
maxSteps: userMaxSteps,Apply the same destructuring exclusion in the Also applies to: 1337-1356, 1956-1976, 2860-2879, 3234-3254 🤖 Prompt for AI Agents |
||
| // Guardrails (can override agent-level guardrails) | ||
| inputGuardrails?: InputGuardrail[]; | ||
| outputGuardrails?: OutputGuardrail<any>[]; | ||
|
|
@@ -1004,6 +1027,7 @@ export class Agent { | |
| private readonly prompts?: PromptHelper; | ||
| private readonly evalConfig?: AgentEvalConfig; | ||
| private readonly feedbackOptions?: AgentFeedbackOptions | boolean; | ||
| private readonly executionValidators?: AgentExecutionValidators; | ||
| private readonly inputGuardrails: NormalizedInputGuardrail[]; | ||
| private readonly outputGuardrails: NormalizedOutputGuardrail[]; | ||
| private readonly inputMiddlewares: NormalizedInputMiddleware[]; | ||
|
|
@@ -1048,6 +1072,7 @@ export class Agent { | |
| this.voltOpsClient = options.voltOpsClient; | ||
| this.evalConfig = options.eval; | ||
| this.feedbackOptions = options.feedback; | ||
| this.executionValidators = normalizeAgentExecutionValidators(options.executionValidators); | ||
| this.inputGuardrails = normalizeInputGuardrailList(options.inputGuardrails || []); | ||
| this.outputGuardrails = normalizeOutputGuardrailList(options.outputGuardrails || []); | ||
| this.inputMiddlewares = normalizeInputMiddlewareList(options.inputMiddlewares || []); | ||
|
|
@@ -1140,6 +1165,53 @@ export class Agent { | |
| // Public API Methods | ||
| // ============================================================================ | ||
|
|
||
| async validateToolExecution({ | ||
| tool, | ||
| args, | ||
| options, | ||
| toolCallId, | ||
| messages, | ||
| operationContext, | ||
| }: { | ||
| tool: Tool<any, any> | ProviderTool; | ||
| args: unknown; | ||
| options?: ToolExecuteOptions; | ||
| toolCallId?: string; | ||
| messages?: unknown[]; | ||
| operationContext?: OperationContext; | ||
| }): Promise<void> { | ||
| const validators = options?.executionValidators?.tools ?? this.executionValidators?.tools; | ||
| if (!validators || validators.length === 0) { | ||
| return; | ||
| } | ||
|
Comment on lines
+1183
to
+1186
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Resolve validators from the operation context first.
Proposed fix- const validators = options?.executionValidators?.tools ?? this.executionValidators?.tools;
+ const validators =
+ operationContext?.executionValidators?.tools ??
+ mergeAgentExecutionValidators(this.executionValidators, options?.executionValidators)?.tools;
if (!validators || validators.length === 0) {
return;
}🤖 Prompt for AI Agents |
||
|
|
||
| const resolvedToolCallId = toolCallId ?? options?.toolContext?.callId ?? randomUUID(); | ||
| const resolvedOperationContext = | ||
| operationContext ?? | ||
| (options?.operationId && options.context && options.systemContext | ||
| ? (options as OperationContext) | ||
| : undefined); | ||
| const context: ToolExecutionValidationContext = { | ||
| type: "tool", | ||
| agent: this, | ||
| tool, | ||
| toolName: tool.name, | ||
| args, | ||
| options, | ||
| operationContext: resolvedOperationContext, | ||
| toolCallId: resolvedToolCallId, | ||
| messages: messages ?? options?.toolContext?.messages ?? [], | ||
| timestamp: new Date(), | ||
| }; | ||
|
|
||
| await runExecutionValidators( | ||
| validators, | ||
| context, | ||
| `Tool ${tool.name} execution blocked by validation.`, | ||
| "TOOL_VALIDATION_FAILED", | ||
| ); | ||
| } | ||
|
|
||
| /** | ||
| * Generate text response | ||
| */ | ||
|
|
@@ -4071,6 +4143,11 @@ export class Agent { | |
| operationId, | ||
| context, | ||
| requestHeaders: options?.requestHeaders ?? options?.parentOperationContext?.requestHeaders, | ||
| executionValidators: mergeAgentExecutionValidators( | ||
| options?.parentOperationContext?.executionValidators, | ||
| this.executionValidators, | ||
| options?.executionValidators, | ||
| ), | ||
| systemContext, | ||
| isActive: true, | ||
| logger, | ||
|
|
@@ -6351,7 +6428,7 @@ export class Agent { | |
| options: executionOptions, | ||
| }); | ||
|
|
||
| if (isToolDeniedError(errorValue)) { | ||
| if (isToolDeniedError(errorValue) || isExecutionValidationError(errorValue)) { | ||
| oc.abortController.abort(errorValue); | ||
| } | ||
|
|
||
|
|
@@ -6367,6 +6444,14 @@ export class Agent { | |
| return async function* (this: Agent): AsyncGenerator<any, void, void> { | ||
| try { | ||
| await oc.traceContext.withSpan(toolSpan, async () => { | ||
| await this.validateToolExecution({ | ||
| tool: tool as Tool<any, any>, | ||
| args, | ||
| options: executionOptions, | ||
| toolCallId, | ||
| messages, | ||
| operationContext: oc, | ||
| }); | ||
| await runToolStartHooks(); | ||
| }); | ||
|
|
||
|
|
@@ -6426,6 +6511,14 @@ export class Agent { | |
|
|
||
| return oc.traceContext.withSpan(toolSpan, async () => { | ||
| try { | ||
| await this.validateToolExecution({ | ||
| tool: tool as Tool<any, any>, | ||
| args, | ||
| options: executionOptions, | ||
| toolCallId, | ||
| messages, | ||
| operationContext: oc, | ||
| }); | ||
| // Call tool start hook - can throw ToolDeniedError | ||
| await runToolStartHooks(); | ||
|
|
||
|
|
@@ -6946,6 +7039,15 @@ export class Agent { | |
| executionOptions.toolContext?.callId ?? randomUUID(), | ||
| ); | ||
|
|
||
| await this.validateToolExecution({ | ||
| tool, | ||
| args, | ||
| options: executionOptions, | ||
| toolCallId: executionOptions.toolContext?.callId, | ||
| messages: executionOptions.toolContext?.messages ?? [], | ||
| operationContext: oc, | ||
| }); | ||
|
|
||
| const tools: Record<string, any> = { | ||
| [tool.name]: tool, | ||
| }; | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would it make more sense to merge this into the GuardRails primitive? They seem like a very similar in concept?
We could add an execution guardrail for tools and steps.
Something like:
This could then be used to implement things like PreToolUse hooks like in claude etc. aka permissions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@omeraplak i might take a stab at integrating into Guardrails this week vs this implemenation, just ping me on discord to chat or we can discuss here direct..
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep honestly I think your approach makes a lot of sense. Also pretty sure we can merge your PR 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, I'll take a stab here this weekend or next week!