Skip to content

api-reference.md

Latest commit

 

History

History
1645 lines (1251 loc) · 59.3 KB

api-reference.md

File metadata and controls

1645 lines (1251 loc) · 59.3 KB
Raw

API Reference

Complete API documentation for @gleanwork/mcp-server-tester.

Entry points

Import from Contents Stability
@gleanwork/mcp-server-tester Fixtures, matchers and validators, the MCP client, config, datasets with runEvalDataset and runEvalCase, judges, conformance, Agent Skills Stable
@gleanwork/mcp-server-tester/fixtures/mcp test and assertions with the MCP fixtures and matchers Stable
@gleanwork/mcp-server-tester/fixtures/mcpAuth Auth fixtures Stable
@gleanwork/mcp-server-tester/reporters/mcpReporter The MCP reporter Stable
@gleanwork/mcp-server-tester/evals The evaluation framework: eval configs, evals and batches, extension definition types, metrics, result stores, comparisons, tool optimizations Stable, except the environment interfaces (below)
@gleanwork/mcp-server-tester/auth Low-level OAuth: discovery, token storage, client credentials Stable
@gleanwork/mcp-server-tester/experimental/clients Desktop-run metadata types, Cowork settings and audit, client plugins Experimental: may change between minor versions
@gleanwork/mcp-server-tester/types The root's shared types on their own, without runtime code Stable

The ./evals, ./auth and ./experimental/clients subpaths are ESM only, and CommonJS code cannot require them. The ESM root and those three subpaths share one copy of the library, so anything registered through one is visible through the others. The CommonJS root is a separate copy; don't mix it with ESM imports of the subpaths. Optional desktop-client fields on root result types (such as EvalCaseResult.clientMetadata) are typed from ./experimental/clients and share its stability.

The environment interfaces in ./evals (EnvironmentDefinition, Environment, EnvironmentContext, EnvironmentKeep, Machine, WorkerChannel, the Shard* types, ShardTokens, TrialKey, machineEnvironment and dockerEnvironment) can change between 2.0 prereleases, until the mst.shard/v1 contract is frozen (ADR 0004).

Table of Contents

Fixtures

mcpPlugins option

Plugins whose extensions this project's tests use, such as a judge referenced as toPassToolJudge({ judge: 'acme/judge/completeness' }). Set it in a project's use block or with test.use({ mcpPlugins: [acme] }). The mcp and mcpClient fixtures install them. See Plugins.

installPlugins(plugins)

Install plugin objects for code that calls validators or matchers outside runEvalDataset, runEvalCase, an eval, or the mcp fixture (all of which install the plugins you pass them). Validates every plugin first; installing the same plugin again is a no-op, and a different plugin claiming a loaded namespace throws. Returns the validated plugins.

mcpClient: Client

Raw MCP SDK client from @modelcontextprotocol/client (MCP TypeScript SDK v2).

test('use raw client', async ({ mcpClient }) => {
  const tools = await mcpClient.listTools();
  const result = await mcpClient.callTool({ name: 'tool_name', arguments: { ... } });
});

mcp: MCPFixtureApi

High-level test API with helper methods.

/**
 * High-level API for interacting with MCP servers in tests
 *
 * This interface wraps the raw MCP Client with test-friendly methods
 */
export interface MCPFixtureApi extends MCPFixtureExtensions {
  /**
   * The underlying MCP client (for advanced usage)
   */
  client: Client;

  /**
   * Authentication type used for this test session
   */
  authType: AuthType;

  /**
   * Playwright project name for this test session
   */
  project?: string;

  /**
   * The protocol this connection requested and negotiated, e.g.
   * `{ requested: '2026-07-28', negotiated: '2026-07-28', era: 'modern' }`.
   * Use it to skip era-specific tests:
   * `test.skip(mcp.protocol.era !== 'modern')`.
   */
  readonly protocol: MCPProtocolInfo;

  /**
   * Lists all available tools from the MCP server
   *
   * @returns Array of tool definitions
   */
  listTools(): Promise<Array<Tool>>;

  /**
   * Calls a tool on the MCP server
   *
   * @param name - Tool name
   * @param args - Tool arguments
   * @returns Tool call result
   */
  callTool<TArgs extends Record<string, unknown> = Record<string, unknown>>(
    name: string,
    args: TArgs
  ): Promise<CallToolResult>;

  /**
   * Gets information about the connected server
   */
  getServerInfo(): {
    name?: string;
    version?: string;
  } | null;
}

Methods

listTools()

List all tools available from the MCP server.

Returns: Promise<Array<Tool>>

const tools = await mcp.listTools();
console.log(tools.map((t) => t.name));
callTool<TArgs>(name, args)

Call a tool by name with arguments.

Parameters:

  • name: string - Tool name
  • args: TArgs - Tool arguments

Returns: Promise<CallToolResult>

const result = await mcp.callTool('get_weather', { city: 'London' });
getServerInfo()

Get server information (name, version).

Returns: { name?: string; version?: string } | null

const info = mcp.getServerInfo();
console.log(info?.name, info?.version);
protocol

The protocol the connection requested and negotiated: { requested, negotiated, era }, e.g. { requested: '2026-07-28', negotiated: '2026-07-28', era: 'modern' }. See Protocol Versions.

discover()

The server/discover result on 2026-07-28 connections, null on legacy connections.

Returns: Promise<DiscoverResult | null>

listResources() / readResource(uri)

resources/list (all pages) and resources/read.

Returns: Promise<Resource[]> / Promise<ReadResourceResult>

request(method, params, resultSchema)

Send any request, such as an extension method, and validate the result against a Standard Schema (for example a Zod schema).

const result = await mcp.request(
  'skills/list',
  {},
  z.object({ skills: z.array(z.object({ uri: z.string() })) })
);
skills

Agent Skills (SEP-2640) helpers: supported(), settings(), list(), get(uri), and read(uri, { entry?, verify? }), which verifies digest, size, and frontmatter against the skill's entry. See Agent Skills.

const [entry] = await mcp.skills.list();
const skill = await mcp.skills.read(entry!.uri);
expect(skill.verified).toBe(true);

createFixtureExtensions(client)

Builds discover, listResources, readResource, request, and skills for a client. Use it (with getProtocolInfo(client) for protocol) when you implement MCPFixtureApi yourself.

createMCPFixture(client, testInfo?, options?)

Creates an MCPFixtureApi wrapper around a raw MCP Client. Use this when you need manual fixture setup — for example in custom fixture hierarchies, non-Playwright test runners (Vitest, Jest), or when composing with other lifecycle logic.

For the standard Playwright use case, prefer importing test and mcp from @gleanwork/mcp-server-tester/fixtures/mcp, which wires this up automatically.

Parameters:

  • client: Client — MCP client created via createMCPClientForConfig(). A client created another way works for calls, but mcp.protocol reports the default requested setting, and conformance probes and wire-level checks skip, because MST didn't record how it connected.
  • testInfo?: TestInfo — Optional Playwright TestInfo. When provided, operations are wrapped in test.step() and attachments are created for the MCP reporter
  • options?: MCPFixtureOptions — Optional configuration

MCPFixtureOptions:

Field Type Default Description
authType 'oauth' | 'api-token' | 'none' 'none' Authentication type for this session
project string — Playwright project name (for filtering/grouping in the reporter)
callTimeoutMs number 30000 Timeout in milliseconds for MCP operations

Returns: MCPFixtureApi

import {
  createMCPFixture,
  createMCPClientForConfig,
  closeMCPClient,
} from '@gleanwork/mcp-server-tester';
import { test as base } from '@playwright/test';
import type { MCPFixtureApi } from '@gleanwork/mcp-server-tester';

const test = base.extend<{ mcp: MCPFixtureApi }>({
  mcp: async ({}, use, testInfo) => {
    const client = await createMCPClientForConfig(config);
    const api = createMCPFixture(client, testInfo, { authType: 'api-token' });
    await use(api);
    await closeMCPClient(client);
  },
});

// Non-Playwright usage (no reporter attachments)
const client = await createMCPClientForConfig(config);
const api = createMCPFixture(client);
const tools = await api.listTools();

Authentication

For comprehensive authentication documentation, see the Authentication Guide.

Token Utilities

import {
  createTokenAuthHeaders,
  validateAccessToken,
  isTokenExpired,
  isTokenExpiringSoon,
} from '@gleanwork/mcp-server-tester';

createTokenAuthHeaders(accessToken, tokenType?)

Create HTTP headers with Authorization header.

Parameters:

  • accessToken: string - Access token
  • tokenType?: string - Token type (default: 'Bearer')

Returns: Record<string, string>

const headers = createTokenAuthHeaders(process.env.MCP_ACCESS_TOKEN);
// { Authorization: 'Bearer eyJ...' }

validateAccessToken(accessToken)

Validate that an access token is present and non-empty.

Parameters:

  • accessToken: string | undefined - Token to validate

Throws: Error if token is missing or empty

isTokenExpired(accessToken)

Check if a JWT token appears to be expired.

Parameters:

  • accessToken: string - JWT token

Returns: boolean

isTokenExpiringSoon(expiresAt, bufferMs?)

Check if a token will expire within the buffer time.

Parameters:

  • expiresAt: number | undefined - Expiration timestamp in milliseconds
  • bufferMs?: number - Buffer time (default: 60000 = 1 minute)

Returns: boolean

OAuth Client Provider

import { PlaywrightOAuthClientProvider } from '@gleanwork/mcp-server-tester';

Implements the MCP SDK's OAuthClientProvider interface with file-based storage.

const provider = new PlaywrightOAuthClientProvider({
  storagePath: 'playwright/.auth/mcp-oauth-state.json',
  redirectUri: 'http://localhost:3000/oauth/callback',
  clientId: process.env.MCP_OAUTH_CLIENT_ID,
  clientSecret: process.env.MCP_OAUTH_CLIENT_SECRET,
});

Auth Fixture

import { test } from '@gleanwork/mcp-server-tester/fixtures/mcpAuth';

test('uses auth provider', async ({ mcpAuthProvider }) => {
  // mcpAuthProvider is configured from environment variables
});

Auth Configuration Types

interface MCPAuthConfig {
  accessToken?: string;
  oauth?: MCPOAuthConfig;
}

interface MCPOAuthConfig {
  serverUrl: string;
  scopes?: string[];
  resource?: string;
  authStatePath?: string;
  clientId?: string;
  clientSecret?: string;
  redirectUri?: string;
}

Eval Functions

loadEvalDataset(path, options?)

Load an eval dataset from a JSON file.

Parameters:

  • path: string - Path to dataset JSON file
  • options?: object
    • validate?: boolean - Validate the dataset against the schema (default: true)

Returns: Promise<EvalDataset>

const dataset = await loadEvalDataset('./data/evals.json');

runEvalDataset(options, context)

Run an eval dataset. Assertions are defined per-case in the dataset's assertions blocks.

Parameters:

  • options: EvalRunnerOptions
    • dataset: EvalDataset - Dataset to run
    • plugins?: readonly Plugin[] - Plugins whose extensions (for example acme/judge/completeness judges) the cases use
    • stopOnFailure?: boolean - Stop on first failure (default: false)
    • onCaseComplete?: (result: EvalCaseResult) => void - Callback after each case completes
    • onTrialComplete?: (result: EvalCaseResult, trial: number) => void - Callback after each trial of a case, with the trial's result and its number from 0, before the case's onCaseComplete
    • concurrency?: number - Max parallel cases (default: 1 = sequential)
    • client?: string, model?: string, clientOptions?: ClientOptions - The client cases run on, its model and options; a case's own fields change them. Outside an eval, cases run on mst, on the test's MCP connection (default client: 'mst')
    • defaultTrials?: number - Default trial count for client cases (default: 1)
    • defaultPassThreshold?: number - Default passThreshold for client cases that don't set one (default: 1)
    • defaultJudgeReps?: number - Default judge evaluation count per case (default: 1)
    • filterTags?: string[] - Only run cases whose tags contain at least one match
    • saveResultsTo?: string - Save run results to file for baseline comparison
    • omitResponsesFromBaseline?: boolean - Strip responses from saved baseline (default: true)
    • baselineResultsFrom?: string - Load baseline file for regression detection
    • toolOverrides?: ToolOverrideVariant - Runtime tool metadata overrides for tool optimizations
    • judgeModel?: string - Judge model identifier recorded in run metadata
    • reporting?: 'playwright' | 'none' - 'none' when the caller reports results itself (evals do): no suggestion to pass testInfo (default: 'playwright')
  • context: EvalContext
    • mcp: MCPFixtureApi - MCP fixture API
    • testInfo?: TestInfo - Playwright test info (required for snapshot support)
    • expect?: ExpectType - Playwright expect function (required for snapshot support)

Returns: Promise<EvalRunnerResult>

const result = await runEvalDataset(
  { dataset }, // options — what to run and how
  { mcp, testInfo } // context — Playwright fixtures from your test
);

console.log(`Passed: ${result.passed}/${result.total}`);

Runtime tool overrides let you test alternate tool descriptions or input schemas without editing the eval dataset or MCP server source. Tool names are canonical server tool names; v1 does not support renames.

const variant = {
  id: 'search-description-v2',
  tools: {
    search: {
      description:
        'Search internal company documents, policies, wiki pages, and announcements.',
      inputSchema: {
        type: 'object',
        properties: {
          query: {
            type: 'string',
            description: 'Natural language document or policy query.',
          },
        },
        required: ['query'],
      },
    },
  },
};

const baseline = await runEvalDataset(
  { dataset, defaultTrials: 10 },
  { mcp, testInfo }
);

const candidate = await runEvalDataset(
  {
    dataset,
    defaultTrials: 10,
    toolOverrides: variant,
  },
  { mcp, testInfo }
);

console.log(candidate.metadata?.toolVariantId);

Use compareEvalRuns() to summarize the completed baseline and candidate runs:

import { compareEvalRuns } from '@gleanwork/mcp-server-tester/evals';

const comparison = compareEvalRuns({
  baseline,
  candidate,
  labels: {
    baseline: 'baseline',
    candidate: variant.id,
  },
});

console.log(`Pass-rate delta: ${comparison.deltaPassRate}`);
console.log(`Improved cases: ${comparison.improvedCases.length}`);
console.log(`Regressed cases: ${comparison.regressedCases.length}`);
interface ToolOverrideVariant {
  id: string;
  description?: string;
  tools: Record<
    string,
    {
      description?: string;
      inputSchema?: Record<string, unknown>;
    }
  >;
}

compareEvalRuns(options)

Compare two completed eval runs. This is a pure utility: it does not run evals, read or write baselines, call LLMs, or mutate datasets.

Parameters:

  • options: CompareEvalRunsOptions
    • baseline: EvalRunnerResult - Baseline run result
    • candidate: EvalRunnerResult - Candidate run result
    • labels?: { baseline?: string; candidate?: string } - Optional display labels

Returns: EvalRunComparisonResult

const comparison = compareEvalRuns({
  baseline,
  candidate,
});

The result includes pass-rate deltas, optional tool precision/recall/F1 deltas, and case buckets:

  • improvedCases - failed in baseline, passed in candidate
  • regressedCases - passed in baseline, failed in candidate
  • unchangedPasses - passed in both runs
  • unchangedFailures - failed in both runs
  • missingFromBaseline - case exists only in candidate
  • missingFromCandidate - case exists only in baseline

It also has warnings: string[], which flags runs that negotiated different MCP protocol eras or revisions (from metadata.protocol).

External Result Storage

External result storage persists eval runs, reporter runs, and comparison artifacts as JSON. GCS is the first built-in cloud provider.

type StoredArtifactKind =
  | 'eval-runner-result'
  | 'eval-run-summary'
  | 'eval-run-comparison';

interface EvalResultStore {
  saveArtifact<T>(artifact: StoredEvalArtifact<T>): Promise<void>;
  loadArtifact<T>(
    kind: StoredArtifactKind,
    id: string
  ): Promise<StoredEvalArtifact<T>>;
  loadLatestArtifact<T>(
    kind: StoredArtifactKind
  ): Promise<StoredEvalArtifact<T> | null>;
  listArtifacts(
    kind: StoredArtifactKind,
    options?: { limit?: number }
  ): Promise<StoredArtifactSummary[]>;
}

Create a store from config:

import { createEvalResultStore } from '@gleanwork/mcp-server-tester/evals';

const store = createEvalResultStore({
  provider: 'gcs',
  bucket: 'my-mcp-eval-results',
  prefix: 'my-server/main',
});

runEvalDataset() accepts store-backed baseline references in addition to local file paths:

await runEvalDataset(
  {
    dataset,
    resultStore: store,
    baselineResultsFrom: { store: true, ref: 'latest' },
    saveResultsTo: { store: true, ref: { id: 'candidate-run' } },
  },
  { mcp, testInfo }
);

Stored runs can be used with compareEvalRuns():

import {
  compareEvalRuns,
  loadStoredEvalRunnerResult,
  saveEvalRunComparison,
} from '@gleanwork/mcp-server-tester/evals';

const baseline = await loadStoredEvalRunnerResult(store, { id: 'baseline' });
const candidate = await loadStoredEvalRunnerResult(store, { id: 'candidate' });
const comparison = compareEvalRuns({
  baseline: baseline.data,
  candidate: candidate.data,
});

await saveEvalRunComparison({ store, comparison, id: 'candidate-comparison' });

Result Structure:

/**
 * Overall result of running an eval dataset
 */
export interface EvalRunnerResult {
  /**
   * Total number of cases
   */
  total: number;

  /**
   * Number of passing cases
   */
  passed: number;

  /**
   * Number of failing cases
   */
  failed: number;

  /**
   * Individual case results
   */
  caseResults: Array<EvalCaseResult>;

  /**
   * Overall execution time in milliseconds
   */
  durationMs: number;

  /**
   * Difference between current pass rate and baseline pass rate.
   * Positive = improvement, negative = regression.
   * Only present when `baselineResultsFrom` was provided.
   */
  deltaPassRate?: number;

  /**
   * Number of cases that regressed: passed in baseline, failed now.
   * Only present when `baselineResultsFrom` was provided.
   */
  regressions?: number;

  /**
   * Number of cases that improved: failed in baseline, passed now.
   * Only present when `baselineResultsFrom` was provided.
   */
  improvements?: number;

  /**
   * Average tool precision across all client cases that have a
   * `toolsTriggered` assertion (precision = fraction of called tools
   * that were expected). Only present when at least one such case ran.
   */
  datasetToolPrecision?: number;

  /**
   * Average tool recall across all client cases that have a
   * `toolsTriggered` assertion (recall = fraction of required tools
   * that were actually called). Only present when at least one such case ran.
   */
  datasetToolRecall?: number;

  /**
   * Harmonic mean of `datasetToolPrecision` and `datasetToolRecall`.
   * Only present when at least one case contributes precision/recall data.
   */
  datasetToolF1?: number;

  /**
   * Optimization tracking metadata captured at run time.
   */
  metadata?: EvalRunMetadata;

  /**
   * Aggregate token usage from every client case's model calls.
   */
  totalClientUsage?: UsageMetrics;

  /**
   * Aggregate token usage of judges across all cases, from judges that report it.
   */
  totalJudgeUsage?: Partial<UsageMetrics>;
}

runToolOptimization(options, context) / runToolOptimization(evalOptions)

Run a tool optimization: establish a baseline, give each candidate its tool metadata (tools), compare against the baseline, rank by a metric, guard against regressions, and emit a structured improvement proposal. This is the high-level API that wraps the manual baseline → candidate → compareEvalRuns loop.

Parameters:

  • options: ToolOptimizationOptions
    • dataset: EvalDataset - The dataset to run (never mutated)
    • variants?: ToolOverrideVariant[] - Static candidates tried in round 0
    • proposeVariants?: (ctx: ProposeVariantsContext) => Promise<ToolOverrideVariant[]> - Callback returning the next candidates from prior-round evidence; return [] to stop
    • metric?: 'passRate' | 'toolF1' | 'toolPrecision' | 'toolRecall' - Ranking metric (default 'passRate')
    • maxRounds?: number - Round budget (default 1)
    • minImprovement?: number - Stop when a round's best gain over the best so far is below this (default 0); with better: 'lower' a gain is a decrease
    • allowRegressions?: boolean - Allow winners that break cases (default false)
    • regressionCheck?: 'significant' | 'any-case' - How breakage is judged on regression cases. 'significant' (default): they got worse as a group (exact paired sign-flip test, p < 0.025) or one did on its own (Fisher's exact test, Holm-corrected at 0.05). 'any-case': any case that passes with the baseline fails with the candidate. Either way, recommending needs a clear improvement: p < 0.025 / variants tried. See How variants are judged
    • regressionTag?: string - Tag marking regression cases (default 'regression'). When no case has it, the baseline runs once more, only to group cases
    • heldOutTag?: string - Tag marking held-out cases: left out of ranking and hidden from proposeVariants (default 'held-out')
    • Plus runEvalDataset passthrough: client, model, clientOptions, defaultTrials, defaultJudgeReps, concurrency, filterTags, schemas, judgeModel
  • context: EvalContext - { mcp, testInfo? } from your test

Returns: ToolOptimizationResult

  • baseline - The baseline run: no tool variant on a dataset; on an eval, the base variant as configured (candidates replace its tools)
  • grouping - 'declared' (from regressionTag) or 'grouping-run'; groupingBaseline holds the extra run when one was needed
  • rounds - Every round's candidates with per-candidate result, comparison, metricValue and metricDelta (without held-out cases), measurement (per-group pass rates, changes with p-values and verdicts, and brokenCaseIds), improvement and fixes (the clearly-better call, adjusted for every variant tried), disqualified
  • winner - Best non-disqualified candidate across all rounds
  • proposal - VariantImprovementProposal with recommendation: 'apply' | 'reject' | 'inconclusive', metric values, toolChanges, and improved/regressed case ids
  • reason - Why the optimization stopped: 'no-variants' | 'no-improvement' | 'max-rounds' | 'threshold-met'
import { runToolOptimization } from '@gleanwork/mcp-server-tester/evals';

const result = await runToolOptimization(
  {
    dataset,
    variants: [variant],
    metric: 'passRate',
    defaultTrials: 10,
  },
  { mcp, testInfo }
);

if (result.proposal?.recommendation === 'apply') {
  console.log(result.winner?.variant.id, result.proposal.delta);
}

On an eval config. Pass evalConfig instead of a dataset and context, and the candidates run as the eval config's variants, on any client that can show tool metadata: in-process for mst, through MST's tool proxy for plugin clients, claude-code and cowork. The base variant (evalConfig.baseVariant, or the eval config's baseline) is the baseline; each candidate runs as a copy of it with the candidate's tool metadata as its tools.

const result = await runToolOptimization({
  evalConfig: { configPath: './eval.json', baseVariant: 'control' },
  variants: [conciseSearch, verboseSearch],
  metric: 'input_tokens_mean', // passRate, trialPassRate, or any numeric variant metric
  better: 'lower',
});
  • metric - passRate (default), trialPassRate, or any numeric key of a variant's metrics. A candidate that reports no value for it is disqualified, with metricUnavailable: true.
  • better - 'higher' (default) or 'lower', for tokens, cost or time. Ranking, minImprovement and the recommendation follow it; metricDelta and proposal.delta stay candidate minus baseline.
  • evalConfig also takes rootDir, pluginPaths, plugins and secretsFile, as runEval does. Static variants run in one run with the baseline; later rounds run as their own runs. Variant ids must be unique and differ from the base variant's name. Each run is stored like any other run of the eval config.

A candidate that breaks cases that work today is disqualified from winning unless allowRegressions: true; the best candidate tried is still surfaced in proposal with recommendation: 'reject' so an agent can see what broke. See MCP Client Simulation for the full agent-loop example.

runEvalCase(evalCase, context, options?)

Run a single eval case. Useful when you want fine-grained control over individual cases outside of a dataset, or when building custom eval orchestration.

Parameters:

  • evalCase: EvalCase - The eval case to run
  • context: EvalContext
    • mcp: MCPFixtureApi - MCP fixture API
    • testInfo?: TestInfo - Playwright test info (for reporter integration)
    • expect?: Expect - Playwright expect (for snapshot support)
  • options?: EvalCaseOptions
    • plugins?: readonly Plugin[] - Plugins whose extensions the case uses
    • datasetName?: string - Dataset name for the result (default: 'single-case')
    • client?: string, model?: string, clientOptions?: ClientOptions - The client the case runs on, as for runEvalDataset

Returns: Promise<EvalCaseResult>

import { runEvalCase } from '@gleanwork/mcp-server-tester';

test('single eval case', async ({ mcp }, testInfo) => {
  const result = await runEvalCase(
    {
      id: 'search-check',
      input: 'Find the planning docs',
      assertions: { toolsTriggered: { calls: [{ name: 'search' }] } },
    },
    { mcp, testInfo },
    { client: 'mst', model: 'claude-haiku-4-5' }
  );

  expect(result.pass).toBe(true);
});

When evalCase.trials > 1, the case is run multiple times and result.passRate is populated with the fraction of passing trials.

Programmatic Validators

Pure validation functions that power both Playwright matchers and the eval runner. Each returns a ValidationResult with pass, message, and optional details. Use these when you need validation logic outside of Playwright's expect() — for example in Vitest/Jest tests, eval datasets, or custom pipelines.

import { validateText, validateSchema } from '@gleanwork/mcp-server-tester';

interface ValidationResult {
  pass: boolean;
  message: string;
  details?: Record<string, unknown>;
  metrics?: { precision?: number; recall?: number };
}

validateText(response, expected, options?)

Checks that the response contains all expected text substrings.

Parameters:

  • response: unknown — The response to validate
  • expected: string | string[] — Substring(s) to find
  • options?: TextValidatorOptions — { caseSensitive?: boolean } (default: true)
const result = validateText(response, ['temperature', 'conditions']);
const result2 = validateText(response, 'hello', { caseSensitive: false });

validatePattern(response, patterns, options?)

Checks that the response matches all expected regex patterns. Each pattern check starts at index zero and leaves the supplied RegExp object's lastIndex unchanged. You can safely reuse global and sticky patterns.

Parameters:

  • response: unknown — The response to validate
  • patterns: string | RegExp | (string | RegExp)[] — Pattern(s) to match
  • options?: PatternValidatorOptions — { caseSensitive?: boolean } (default: true)
const result = validatePattern(response, /temperature: \d+/);
const result2 = validatePattern(response, ['\\d+ degrees', /humidity: \d+%/]);

validateError(response, expected?)

Checks that the response is (or is not) an error, optionally with a specific message.

Parameters:

  • response: unknown — The response to validate
  • expected?: boolean | string | string[] — true = expect any error, false = expect no error, string = expect error containing text (default: true)
const result = validateError(response, true); // any error
const result2 = validateError(response, false); // no error
const result3 = validateError(response, 'not found'); // error with message

validateSize(response, options)

Checks that the response size in bytes is within bounds.

Parameters:

  • response: unknown — The response to validate
  • options: SizeValidatorOptions — { minBytes?: number; maxBytes?: number } (at least one required)
const result = validateSize(response, { maxBytes: 10_000 });
const result2 = validateSize(response, { minBytes: 100, maxBytes: 50_000 });

validateSchema(response, schema, options?)

Validates the response against a Zod schema. Automatically parses JSON text responses.

Parameters:

  • response: unknown — The response to validate
  • schema: ZodType — Zod schema to validate against
  • options?: SchemaValidatorOptions — { strict?: boolean } (default: false)
import { z } from 'zod';

const WeatherSchema = z.object({
  temperature: z.number(),
  conditions: z.string(),
});

const result = validateSchema(response, WeatherSchema);

validateResponse(actual, expected)

Deep equality comparison using JSON serialization.

Parameters:

  • actual: unknown — The actual response
  • expected: unknown — The expected response
const result = validateResponse(response, { status: 'ok', count: 42 });

validateToolCalls(response, assertion)

Validates the tool calls in a client case's trace. Only applicable to client cases.

Parameters:

  • response: unknown — Must be an MstClientSimulationResult
  • assertion: ToolCallAssertion — Expected tool call specification
import type { ToolCallAssertion } from '@gleanwork/mcp-server-tester';

const assertion: ToolCallAssertion = {
  calls: [{ name: 'search', required: true }],
  order: 'any',
  exclusive: false,
};

const result = validateToolCalls(simulationResult, assertion);
// result.metrics contains { precision, recall }

validateToolCallCount(response, options)

Validates the number of tool calls in a client case's trace. Only applicable to client cases.

Parameters:

  • response: unknown — Must be an MstClientSimulationResult
  • options: ToolCallCountOptions — { min?: number; max?: number; exact?: number }
const result = validateToolCallCount(simulationResult, { min: 1, max: 3 });

validateJudge(response, config, run?) (async)

Evaluates a response with a judge: the built-in rubric LLM judge or a plugin judge. Returns a Promise<ValidationResult>. A judge that can't score the response (an unknown judge, invalid options, an API error, a score outside 0 to 1) fails with details.error set. A judge that skips passes with details.skipped set.

Parameters:

  • response: unknown — The response to evaluate
  • config: JudgeValidatorConfig — Judge configuration
  • run?: JudgeRun — { evalCase?, clientResponse?, evidence? }, from which the judge's { case, trial } input is built (see Judge contract). Without it, the case is empty except for expected.answer (the reference).

JudgeValidatorConfig:

Field Type Default Description
judge string 'rubric' The built-in rubric, or a plugin judge as <namespace>/judge/<name>
rubric RubricSpec — Shorthand for the rubric judge (required unless judge is set)
reference unknown — Reference response to compare against
threshold number 0.7 Minimum mean score to pass (0–1)
reps number 1 Times the judge scores the response, for every judge (scores averaged)
options Record<string, unknown> — The judge's own options; without it, a named judge gets its other flat fields
provider ProviderKind 'anthropic' The rubric judge's LLM provider (also model, temperature, and so on) 
const result = await validateJudge(response, {
  rubric: 'Does the response accurately describe the weather?',
  threshold: 0.8,
});

validateSnapshot(response, name, options) (async)

Compares a response's text, after sanitizing, with a named snapshot in a SnapshotStore. toMatchToolSnapshot and eval snapshot assertions both use it. Throws when a sanitizer is invalid, which is a configuration error, not a mismatch.

Parameters:

  • response: unknown — The response to compare
  • name: string — Snapshot name
  • options.store: SnapshotStore — Where snapshots live: playwrightSnapshotStore(expect) inside a Playwright test, or your own { match(name, content, { negated }) } that resolves to { pass, message }
  • options.sanitizers?: SnapshotSanitizer[] — Applied before comparison (see toMatchToolSnapshot)
  • options.negated?: boolean — Compare for a .not assertion: pass is still "matches", but the store must not write snapshots, and a missing snapshot counts as a match. playwrightSnapshotStore uses Playwright's own .not.toMatchSnapshot.
import {
  validateSnapshot,
  playwrightSnapshotStore,
} from '@gleanwork/mcp-server-tester';

const result = await validateSnapshot(response, 'weather', {
  store: playwrightSnapshotStore(expect),
  sanitizers: ['uuid', 'iso-date'],
});

validatePredicate(response, predicate, description?) (async)

Runs a custom predicate, which receives the response and its extracted text. description names the predicate in default messages (default 'custom predicate'). A predicate that throws fails with details.error set, so you can tell a crash from a false.

const result = await validatePredicate(
  response,
  (_raw, text) => text.includes('temperature'),
  'mentions temperature'
);

Playwright Matchers

Custom Playwright matchers for writing inline assertions against MCP tool responses. Import assertions from the package or its fixtures:

import { expect } from '@gleanwork/mcp-server-tester';
// or, when using fixtures:
import { test, expect } from '@gleanwork/mcp-server-tester/fixtures/mcp';

toMatchToolResponse(expected)

Assert that the tool response exactly deep-equals the expected value.

test('exact response', async ({ mcp }) => {
  const result = await mcp.callTool('calculate', { a: 2, b: 3 });
  expect(result).toMatchToolResponse({ result: 5 });
});

toContainToolText(text | text[])

Assert that the tool response text contains the given substring(s).

test('text contains', async ({ mcp }) => {
  const result = await mcp.callTool('get_weather', { city: 'London' });
  expect(result).toContainToolText('temperature');
  expect(result).toContainToolText(['London', 'temperature', 'humidity']);
});

toMatchToolPattern(pattern | pattern[])

Assert that the tool response text matches the given regex pattern(s).

test('pattern match', async ({ mcp }) => {
  const result = await mcp.callTool('get_weather', { city: 'London' });
  expect(result).toMatchToolPattern('Temperature: \\d+°[CF]');
  expect(result).toMatchToolPattern(['^## Weather', '\\d{4}-\\d{2}-\\d{2}']);
});

toMatchToolSchema(schema)

Assert that the tool response validates against a Zod schema.

import { z } from 'zod';

const WeatherSchema = z.object({
  city: z.string(),
  temperature: z.number(),
  conditions: z.string(),
});

test('schema validation', async ({ mcp }) => {
  const result = await mcp.callTool('get_weather', { city: 'London' });
  expect(result).toMatchToolSchema(WeatherSchema);
});

toMatchToolSnapshot(name, sanitizers?)

Assert that the tool response matches a saved Playwright snapshot. Use sanitizers to normalize variable fields (timestamps, UUIDs, etc.) before comparison.

test('snapshot', async ({ mcp }, testInfo) => {
  const result = await mcp.callTool('help', {});
  expect(result).toMatchToolSnapshot('help-output');
});

// With sanitizers
expect(result).toMatchToolSnapshot('user-profile', ['uuid', 'iso-date']);

toBeToolError(expected?)

Assert that the tool response is an error (or is not an error when negated). Optionally assert on the error message.

test('error handling', async ({ mcp }) => {
  const result = await mcp.callTool('nonexistent_tool', {});
  expect(result).toBeToolError();

  // Assert specific error message substring
  expect(result).toBeToolError('not found');

  // Assert response is NOT an error
  const good = await mcp.callTool('get_weather', { city: 'London' });
  expect(good).not.toBeToolError();
});

toPassToolJudge(rubric, options?)

Assert that the tool response passes a judge: the built-in rubric judge (an LLM call) or, with { judge: '<namespace>/judge/<name>' }, a plugin judge. The matcher takes passingThreshold (default 0.7), reference, reps, provider, model, judge and options.

test('semantic quality', async ({ mcp }) => {
  const result = await mcp.callTool('search_docs', { query: 'authentication' });
  await expect(result).toPassToolJudge(
    {
      text: 'The results should be relevant to the query about authentication. Score 0-1.',
    },
    { passingThreshold: 0.7 }
  );
});

toHaveToolResponseSize(options)

Assert that the tool response size is within specified byte bounds.

test('response size', async ({ mcp }) => {
  const result = await mcp.callTool('list_files', {});
  expect(result).toHaveToolResponseSize({ minBytes: 10, maxBytes: 50000 });
});

toSatisfyToolPredicate(fn, desc?)

Assert that the tool response satisfies a custom predicate function.

test('custom predicate', async ({ mcp }) => {
  const result = await mcp.callTool('list_files', {});
  expect(result).toSatisfyToolPredicate(
    (r) => Array.isArray(r.content) && r.content.length > 0,
    'response should contain at least one file'
  );
});

toHaveToolCalls(assertion) (client cases only)

Assert that the client made specific tool calls when given a natural language prompt. Only meaningful for client cases.

test('tool discovery', async ({ mcp }) => {
  const result = await mcp.callTool('search', { query: 'find recent docs' });
  expect(result).toHaveToolCalls({
    calls: [{ name: 'search', required: true }],
    order: 'any',
    exclusive: false,
  });
});

toHaveToolCallCount(options) (client cases only)

Assert that the client made a specific number of tool calls. Only meaningful for client cases.

test('call count', async ({ mcp }) => {
  const result = await mcp.callTool('search', { query: 'find docs' });
  expect(result).toHaveToolCallCount({ min: 1, max: 5 });
});

Text Utilities

extractText(response)

Extract text content from various MCP response formats.

Parameters:

  • response: CallToolResult - MCP tool call result

Returns: string

const result = await mcp.callTool('get_info', {});
const text = extractText(result);

normalizeWhitespace(text)

Normalize whitespace for consistent comparison.

Parameters:

  • text: string - Text to normalize

Returns: string

const normalized = normalizeWhitespace('  hello\n\n  world  ');
// Returns: "hello world"

Judge Functions

createJudge(config?)

Create an LLM judge for semantic evaluation of tool responses.

Parameters:

  • config?: JudgeConfig (all fields optional)

    • provider?: 'anthropic' | 'vertex-anthropic' | 'anthropic-agent-sdk' | 'openai' | 'google' - LLM provider (default: 'anthropic'). Each loads its SDK on first use; install the one you use:

      Provider Package
      anthropic @anthropic-ai/sdk
      vertex-anthropic @anthropic-ai/vertex-sdk
      anthropic-agent-sdk @anthropic-ai/claude-agent-sdk
      openai openai
      google @google/generative-ai

    • model?: string - Model name (default: 'claude-sonnet-4-6')

    • temperature?: number - Temperature 0–1 (default: 0.0)

    • maxTokens?: number - Maximum tokens for response (default: 4096)

    • maxBudgetUsd?: number - Maximum budget in USD (default: 0.10)

    • effort?: string - Reasoning effort, such as 'high' or 'xhigh'. OpenAI only: the judge calls the Responses API with this effort, maxTokens as its output budget (reasoning included) and no temperature, which reasoning models reject

    • maxToolOutputSize?: number - Fail if response exceeds this byte count

Returns: Judge

Default (Claude):

import { createJudge } from '@gleanwork/mcp-server-tester';

const judge = createJudge();
// Requires: ANTHROPIC_API_KEY environment variable, or a gateway credential
// (see LLM Gateways: ./llm-gateways.md)

With configuration:

const judge = createJudge({
  provider: 'openai',
  model: 'gpt-4o',
  temperature: 0.0,
});
// Requires: OPENAI_API_KEY environment variable, or a gateway credential

createJudgeCompletion(config?)

The provider call createJudge is built on, without its prompt or parser: a plugin judge that brings its own prompt and reads its own answer uses it to reach models the way MST's judges do, with the same providers, credentials and LLM gateway settings. It takes the same JudgeConfig and throws, as createJudge does, for an unknown provider or a missing credential. preflightJudge(config) checks the same without calling a model, for a plugin judge's preflight.

import { createJudgeCompletion } from '@gleanwork/mcp-server-tester';

const complete = createJudgeCompletion({
  provider: 'openai',
  model: 'gpt-5.6-luna',
  effort: 'xhigh',
  maxTokens: 64000,
});
const { text, usage, truncated } = await complete({
  system: 'Answer in JSON.',
  prompt: 'Rate each citation.',
});

LLM Client Diagnostic Utilities

The following utilities are available for checking whether optional LLM provider packages are installed. They are useful for debugging provider configuration issues but are not part of the typical test-writing path.

isProviderAvailable(provider)

Check whether the npm package required for a given mst client provider is installed in the current environment.

import { isProviderAvailable } from '@gleanwork/mcp-server-tester/evals';

if (!isProviderAvailable('anthropic')) {
  console.warn('Install @anthropic-ai/sdk to use the anthropic provider');
}

getMissingDependencyMessage(provider)

Return a human-readable message describing the missing dependency for a provider, suitable for displaying in error output or test skip conditions.

import { getMissingDependencyMessage } from '@gleanwork/mcp-server-tester/evals';

const message = getMissingDependencyMessage('openai');
// e.g. "Provider 'openai' requires the 'openai' package. Run: npm install openai"

See LLM Client Guide for full details on configuring the mst client.

Conformance Functions

runConformanceChecks(mcp, options?)

Run MCP protocol conformance checks.

Parameters:

  • mcp: MCPFixtureApi - MCP fixture API
  • options?: object
    • requiredTools?: string[] - Tools that must be present
    • validateSchemas?: boolean - Validate tool input schemas (default: true)
    • checkServerInfo?, checkResources?, checkPrompts?: boolean - Toggle those checks (default: true)
    • probe?: boolean - Allow raw probe requests for 2026-07-28 rejection rules (default: true)
    • skills?: { maxSkills?: number; verifyFiles?: 'skill-md' | 'all'; maxPages?: number } | false - Tune or disable the skills checks (defaults: 25 skills, 'skill-md', 64 pages)
  • testInfo?: TestInfo - Attach results to the MCP reporter

Checks are selected by the negotiated era: legacy connections run the core checks; 2026-07-28 connections also run the modern checks; servers that declare the skills extension get the skills checks in every era.

Returns: Promise<MCPConformanceResult>

const result = await runConformanceChecks(mcp, {
  requiredTools: ['get_weather', 'search_docs'],
  validateSchemas: true,
});

expect(result.pass).toBe(true);

Result Structure:

interface MCPConformanceResult {
  pass: boolean; // every non-skipped 'must' check passed
  checks: Array<{
    name: string;
    pass: boolean;
    message: string;
    severity?: 'must' | 'should'; // failing 'should' checks are warnings
    skipped?: boolean; // not applicable here; message says why
    specVersion?: string; // e.g. '2026-07-28'
    specRef?: string; // spec section the check enforces
  }>;
  protocol: MCPProtocolInfo; // { requested, negotiated, era }
  raw: MCPConformanceRaw; // serverInfo, capabilities, tools, resources, prompts
}

runCrossEraChecks(config, options?, testInfo?)

Connect to the same server once per protocol and check it serves the same tools, tool definitions, resources, prompts, skills, and capabilities in each, and that protocol: 'auto' picks the modern era.

Parameters:

  • config: MCPConfig
  • options?: { protocols?: ProtocolSetting[]; checkAuto?: boolean; clientOptions?: Omit<CreateMCPClientOptions, 'protocol'> } - protocols defaults to ['legacy', '2026-07-28']
  • testInfo?: TestInfo

Returns: Promise<MCPCrossEraResult> with pass, checks, and connections.

Protocol Helpers

protocolMatrix(project, protocols)

Expand one Playwright project into one project per protocol, named <name>@<protocol>, with mcpProtocol and mcpConfig.protocol set.

projects: [
  ...protocolMatrix({ name: 'docs', use: { mcpConfig } }, [
    'legacy',
    '2026-07-28',
  ]),
];

getProtocolInfo(client)

{ requested, negotiated, era } for a client created by MST.

eraOfRevision(revision) / isProtocolRevision(value)

Classify a dated revision as 'legacy' or 'modern', and check a string is a YYYY-MM-DD revision.

DEFAULT_PROTOCOL_SETTING / ProtocolMatrixEntry<T>

DEFAULT_PROTOCOL_SETTING is 'legacy', the protocol used when none is set. ProtocolMatrixEntry<T> is the type of each project protocolMatrix() returns.

LEGACY_PROTOCOL_VERSIONS / MODERN_PROTOCOL_VERSIONS / FIRST_MODERN_PROTOCOL_VERSION

The dated revisions MST speaks, in each era, and the first modern one (2026-07-28). Pass them to protocolMatrix() to run a project across every revision:

import {
  LEGACY_PROTOCOL_VERSIONS,
  MODERN_PROTOCOL_VERSIONS,
  protocolMatrix,
} from '@gleanwork/mcp-server-tester';

const projects = protocolMatrix(project, [
  ...LEGACY_PROTOCOL_VERSIONS,
  ...MODERN_PROTOCOL_VERSIONS,
]);

Tool Call Helpers

callToolNormalized(client, params, options?)

What mcp.callTool() uses: calls a tool on a raw SDK Client and turns a JSON-RPC error sent by the server (such as -32602 for an unknown tool) into an isError: true result whose text is MCP error <code>: <message>. Local SDK errors (timeouts, closed connections, auth) still reject.

getToolProtocolError(result)

The { code, message, data? } of the protocol error a result was made from, or null for results the server returned.

Skills Functions

mcp.skills wraps these; use them directly with a raw SDK Client. See Agent Skills.

Export Purpose
getSkillsExtension(client) The server's io.modelcontextprotocol/skills settings, or null when it doesn't declare the extension
SKILLS_EXTENSION_ID 'io.modelcontextprotocol/skills', the extension's capability key
listSkills(client, { maxPages? }) All skills/list entries, following nextCursor (default 64 pages)
getSkill(client, uri) One entry from skills/get
readSkillFile(client, uri) Read a skill file with resources/read: { uri, text?, bytes, mimeType? }
verifySkillFile(entry, file) Problems (digest, size, SKILL.md frontmatter) found checking a read file against its entry
validateSkillEntry(entry) SEP-2640 entry problems, each with severity: 'must' | 'should'
parseSkillFrontmatter(markdown) The YAML frontmatter of a SKILL.md as an object, or null without one
SkillEntrySchema Zod schema for the wire shape of an entry (use validateSkillEntry() for the SEP rules)

Type Definitions

EvalAssertions

/**
 * A case's assertions
 *
 * Mirrors the Playwright matcher API for consistency.
 */
export interface EvalAssertions {
  /**
   * Text substring(s) the client's answer must contain (toContainToolText)
   */
  containsText?: string | string[];

  /**
   * Regex pattern(s) the client's answer must match (toMatchToolPattern)
   */
  matchesPattern?: string | string[];

  /**
   * Asserts which tools the client called. Needs structured tool evidence
   * (a client that reports what it called).
   */
  toolsTriggered?: {
    /** Expected tool calls */
    calls: Array<{
      /** Tool or explicitly selected client event name. */
      name: string;
      kind?: TraceEvent['kind'];
      source?: TraceEvent['source'];
      server?: string;
      /** Expected arguments (partial match — extra keys are allowed) */
      arguments?: Record<string, unknown>;
      /** Whether this call MUST have been made (default: true) */
      required?: boolean;
    }>;
    /**
     * 'strict': calls must appear in the exact order listed
     * 'any': calls can appear in any order (default)
     */
    order?: 'strict' | 'any';
    /** If true, no tool calls outside the `calls` list are allowed */
    exclusive?: boolean;
  };

  /**
   * Asserts the number of tool calls the client made. Needs structured tool
   * evidence.
   */
  toolCallCount?: {
    /** Minimum number of tool calls */
    min?: number;
    /** Maximum number of tool calls */
    max?: number;
    /** Exact number of tool calls */
    exact?: number;
  };
}

CaseJudge

A case's judges entries, written like an eval config's judges.

/**
 * One of a case's judges, written like an eval config's: a reference
 * (`acme/judge/completeness`) or `{ "type": <reference>, ...options }`.
 */
export type CaseJudge = string | CaseJudgeConfig;

/** How a judge grades: the settings a case judge and a judge request share. */
interface JudgeSettings {
  /** The judge's options, validated by its schema. */
  options?: Record<string, unknown>;
  /** Other flat fields are the judge's options too. */
  [key: string]: unknown;
  /** The rubric judge's rubric: a built-in name or a custom `{ text }`. */
  rubric?: BuiltInRubric | { text: string };
  /** Reference answer to compare against. Default: the case's `expected.answer`. */
  reference?: unknown;
  /** Score threshold for passing (0-1, default: 0.7) */
  threshold?: number;
  /** Number of judge evaluations. Overrides EvalCase.judgeReps. */
  reps?: number;
  /** The rubric judge's provider. @default 'anthropic' */
  provider?: ProviderKind;
  /** Model override (e.g., 'claude-opus-4-20250514') */
  model?: string;
  /** Environment variable name for API key */
  apiKeyEnvVar?: string;
  /** Max tokens for judge response */
  maxTokens?: number;
  /** Temperature for judge LLM (0-1) */
  temperature?: number;
  /** Max budget in USD per evaluation */
  maxBudgetUsd?: number;
  /** Fail if response exceeds this size in bytes before judging */
  maxToolOutputSize?: number;
}

/**
 * A case judge with its settings: `type` names the judge (the built-in
 * `rubric`, or `<namespace>/judge/<name>` from a plugin); `threshold`,
 * `reference` and `reps` say how it grades; other flat fields (or
 * `options`) are the judge's own options.
 */
export interface CaseJudgeConfig extends JudgeSettings {
  type: string;
}

EvalCase

/**
 * A single eval case: an input the client under test acts on, and what to
 * assert about what it did. The case runs on the `client`, `model` and
 * `clientOptions` it inherits from the eval (or the run), changed by its own.
 *
 * Direct tool calls aren't cases: write them as Playwright tests with
 * `mcp.callTool()` and the matchers.
 */
export interface EvalCase extends ClientFields {
  /**
   * Unique identifier for this test case
   */
  id: string;

  /**
   * Human-readable description of what this test case validates
   */
  description?: string;

  /**
   * The user's request the client acts on, sent as its prompt.
   *
   * @example "Get the weather for London and tell me if I need an umbrella"
   */
  input: string;

  /** Additional metadata for this test case. */
  metadata?: Record<string, unknown>;

  /**
   * Number of trials (independent runs) of this case. When > 1,
   * `EvalCaseResult.passRate` is the share of trials that passed, and
   * `pass` is decided by `passThreshold`.
   * @default 1
   */
  trials?: number;

  /**
   * Share of trials (0–1) that must pass for the case to pass.
   * @default 1.0 (every trial)
   */
  passThreshold?: number;

  /**
   * Number of times to run each of the case's judges. Scores are averaged;
   * the mean must meet the threshold to pass. Reduces judge variance caused
   * by non-determinism. A judge's own `reps` overrides this value.
   * @default 1
   */
  judgeReps?: number;

  /**
   * What the case expects, for graders: `answer` (the reference answer,
   * passed to judges as `reference` unless a judge sets its own),
   * `criteria` (rubric criteria keyed by name), and any other ground truth.
   * Judges read it as `case.expected`.
   */
  expected?: {
    answer?: unknown;
    criteria?: Record<string, string>;
    [key: string]: unknown;
  };

  /**
   * Arbitrary string labels for this case.
   * Use for filtering eval runs with `EvalRunnerOptions.filterTags`
   * and for slicing results by category.
   *
   * @example ['tool-finding', 'multi-hop', 'search']
   */
  tags?: string[];

  /**
   * Assertions (code graders) each trial must pass. All of them run.
   *
   * @example
   * ```json
   * {
   *   "id": "weather-london",
   *   "toolName": "get_weather",
   *   "args": { "city": "London" },
   *   "assertions": {
   *     "containsText": ["temperature", "conditions"],
   *     "schema": "WeatherResponse",
   *     "responseSize": { "maxBytes": 10000 },
   *     "isError": false
   *   }
   * }
   * ```
   */
  assertions?: EvalAssertions;

  /**
   * Judges (model graders) that score each trial. Each must pass. They run
   * on top of the eval config's `judges`; when the case and the eval config
   * list the same judge, the case's settings win.
   *
   * @example
   * ```json
   * "judges": [
   *   { "type": "rubric", "rubric": "correctness", "threshold": 0.8 },
   *   "acme/judge/completeness"
   * ]
   * ```
   */
  judges?: CaseJudge[];
}

Next Steps