Conversation
Add the data model, runtime, and API for synthetic worlds: a replica of an agent's tool set that evals run against instead of a real backend. - SyntheticWorld (project child) owns SyntheticTool (code-tool artifacts that replace a real tool id) and SyntheticFixture (opaque data plus a timezone-aware frozen time), with a shared lib/ on the sandbox path. - EvalInput.synthetic_environment names the world and fixture. The eval runner creates one instance per job, swaps bound tools through the registry under the real tool id, records the instance on the trace, and drops copies the run left unchanged. Retention follows the trace. - The trace key gains a variant slot so fixtures never share generations. - Scorers can declare synthetic_instance; judges see ids without paths. - Minimal CRUD API for worlds, tools, and fixtures on the desktop server. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019jkErL54oXyYL35E13SRDM
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
📊 Coverage ReportOverall Coverage: 93% Diff: origin/main...HEAD
Summary
Line-by-lineView line-by-line diff coverageapp/desktop/studio_server/synthetic_world_api.pyLines 143-151 143
144 def _tool_from_id(world: SyntheticWorld, tool_id: str) -> SyntheticTool:
145 tool = SyntheticTool.from_id_and_parent_path(tool_id, world.path)
146 if tool is None:
! 147 raise HTTPException(status_code=404, detail="Synthetic tool not found")
148 return tool
149
150
151 def _require_local_files(world: SyntheticWorld) -> None:Lines 294-303 294 world = _world_from_id(project_id, world_id)
295 try:
296 for field, value in request.model_dump(exclude_unset=True).items():
297 setattr(world, field, value)
! 298 except (ValueError, PydanticValidationError) as e:
! 299 raise HTTPException(status_code=400, detail=str(e))
300 world.save_to_file()
301 return _world_response(world)
302
303 @app.delete(Lines 327-336 327 project = project_from_id(project_id)
328 world = _world_from_id(project_id, world_id)
329 try:
330 warnings = world.validate_bindings(project)
! 331 except ValueError as e:
! 332 raise HTTPException(status_code=400, detail=str(e))
333 return SyntheticWorldValidationResponse(warnings=warnings)
334
335 # ---- tools ----Lines 413-422 413 tool = _tool_from_id(_world_from_id(project_id, world_id), tool_id)
414 try:
415 for field, value in request.model_dump(exclude_unset=True).items():
416 setattr(tool, field, value)
! 417 except (ValueError, PydanticValidationError) as e:
! 418 raise HTTPException(status_code=400, detail=str(e))
419 tool.save_to_file()
420 return _tool_response(tool)
421
422 @app.delete(libs/core/kiln_ai/adapters/eval/eval_runner.pyLines 472-480 472 generation actually left. The context is reset in `finally` both times: worker
473 tasks are reused across jobs.
474 """
475 if job.task_run_config is None:
! 476 raise ValueError("A task_run_eval job requires a run config")
477 world = self._resolve_synthetic_world(environment)
478 launcher = self._synthetic_launcher_override or launcher_for_world(world)
479 bindings = world.bindings()
480 variant = synthetic_fingerprint(Lines 509-517 509
510 trace, _ = await self._trace_index.get_or_create(key, generate)
511 instance = trace.synthetic_instance
512 if instance is None:
! 513 raise ValueError(
514 f"Eval trace {trace.id} was generated for a synthetic environment but "
515 "records no synthetic instance"
516 )
517 if world.strict and not instance.valid:Lines 554-562 554 """The world an input names, or an error: never a silent fallback. The launch
555 config is the launcher's to validate, which it does at launch."""
556 project = self.task.parent_project()
557 if project is None:
! 558 raise ValueError(
559 "Synthetic environments require the task to belong to a project"
560 )
561 world = SyntheticWorld.from_id_and_parent_path(
562 environment.world_id, project.pathLines 571-580 571 pending, self._unchanged_instances = self._unchanged_instances, []
572 for launcher, instance in pending:
573 try:
574 await launcher.release(instance)
! 575 except Exception as e:
! 576 logger.warning(
577 "Releasing unchanged synthetic instance %s failed: %s",
578 instance.instance_id,
579 e,
580 )Lines 592-606 592 try:
593 launchers[world.launcher] = self._synthetic_launcher_override or (
594 launchers.get(world.launcher) or launcher_for_world(world)
595 )
! 596 except Exception as e:
! 597 logger.warning("No launcher for world %s: %s", world.name, e)
598 for launcher in launchers.values():
599 try:
600 await launcher.prune(live)
! 601 except Exception as e:
! 602 logger.warning(
603 "Pruning synthetic instances failed: %s", e, exc_info=True
604 )
605
606 async def _resolve_trace(libs/core/kiln_ai/datamodel/project.pyLines 78-83 78 def code_tools(self, readonly: bool = False) -> list[CodeTool]:
79 return super().code_tools(readonly=readonly) # type: ignore
80
81 def synthetic_worlds(self, readonly: bool = False) -> list[SyntheticWorld]:
! 82 return super().synthetic_worlds(readonly=readonly) # type: ignorelibs/core/kiln_ai/datamodel/synthetic_world.pyLines 43-51 43 ToolId,
44 )
45
46 if TYPE_CHECKING:
! 47 from kiln_ai.datamodel.project import Project
48
49 WORLD_LIB_DIRNAME = "lib"
50 LOCAL_FILES_LAUNCHER = "local_files"Lines 299-311 299 description="An external tool server (MCP) whose whole tool surface a launched instance serves. While an instance with a connection is active, every tool of that server resolves to the same-named tool on the instance. Per-tool bindings still apply to anything else.",
300 )
301
302 def tools(self, readonly: bool = False) -> list[SyntheticTool]:
! 303 return super().tools(readonly=readonly) # type: ignore
304
305 def world_dir(self) -> Path:
306 if self.path is None:
! 307 raise ValueError("World must be saved before accessing its directory")
308 return self.path.parent
309
310 def lib_dir(self) -> Path:
311 return self.world_dir() / WORLD_LIB_DIRNAMELines 380-389 380 )
381 if real.parameters_schema != synthetic.parameters_schema:
382 warnings.append(f"{real_id}: parameters schema differs")
383 elif real_id.startswith(KILN_TASK_TOOL_ID_PREFIX):
! 384 server_id = kiln_task_server_id_from_tool_id(real_id)
! 385 server = next(
386 (
387 s
388 for s in project.external_tool_servers(readonly=True)
389 if s.id == server_idLines 389-402 389 if s.id == server_id
390 ),
391 None,
392 )
! 393 if server is None:
! 394 warnings.append(f"{real_id}: Kiln task tool server not found")
! 395 continue
! 396 real_name = server.properties.get("name")
! 397 if real_name != synthetic.tool_function_name:
! 398 warnings.append(
399 f"{real_id}: function name mismatch (real '{real_name}', synthetic '{synthetic.tool_function_name}')"
400 )
401 return warningslibs/core/kiln_ai/run_context.pyLines 12-20 12 from dataclasses import dataclass, field
13 from typing import TYPE_CHECKING
14
15 if TYPE_CHECKING:
! 16 from kiln_ai.datamodel.synthetic_world import (
17 SyntheticInstance,
18 SyntheticTool,
19 SyntheticWorld,
20 )libs/core/kiln_ai/sandbox/synthetic_env.pyLines 93-103 93 name = metadata_env_name(str(key))
94 if name not in _RESERVED:
95 os.environ[name] = str(value)
96 elif isinstance(value, bool):
! 97 name = metadata_env_name(str(key))
! 98 if name not in _RESERVED:
! 99 os.environ[name] = "true" if value else "false"
100 lib_path = instance.get("world_lib_path")
101 if lib_path and os.path.isdir(lib_path) and lib_path not in sys.path:
102 sys.path.insert(0, lib_path)Lines 114-121 114 raw = os.environ.get(env_name)
115 empty = None if key == "connection" else {}
116 try:
117 result[key] = json.loads(raw) if raw else empty
! 118 except json.JSONDecodeError:
! 119 result[key] = empty
120 return resultlibs/core/kiln_ai/synthetic_worlds/launcher.pyLines 72-80 72 self, world: SyntheticWorld, config: dict[str, JsonValue]
73 ) -> str | None:
74 """Identity of the content a launch of *config* would start from, without
75 launching. Folded into the trace fingerprint: change it and traces regenerate."""
! 76 ...
77
78 async def launch(
79 self, world: SyntheticWorld, config: dict[str, JsonValue]
80 ) -> SyntheticInstance: ...Lines 137-145 137 if not manifest.is_file():
138 return {}
139 loaded = yaml.safe_load(manifest.read_text(encoding="utf-8")) or {}
140 if not isinstance(loaded, dict):
! 141 raise ValueError(f"{manifest} must hold a mapping")
142 return {str(k): _jsonable(v) for k, v in loaded.items()}
143
144
145 def _jsonable(value: object) -> JsonValue:Lines 143-155 143
144
145 def _jsonable(value: object) -> JsonValue:
146 if isinstance(value, datetime):
! 147 return value.isoformat()
148 if isinstance(value, dict):
! 149 return {str(k): _jsonable(v) for k, v in value.items()}
150 if isinstance(value, list):
! 151 return [_jsonable(v) for v in value]
152 return value # type: ignore[return-value]
153
154
155 class LocalFilesLauncher:Lines 198-206 198 async def launch(
199 self, world: SyntheticWorld, config: dict[str, JsonValue]
200 ) -> SyntheticInstance:
201 if world.id is None:
! 202 raise ValueError("World must be saved before launching an instance")
203 fixture_id = config.get("fixture_id")
204 if not isinstance(fixture_id, str) or not fixture_id:
205 raise ValueError(
206 f"World '{world.name}' uses the local files launcher, which needs a "Lines 293-301 293 now = time.time()
294 candidates: list[tuple[Path, float, int]] = []
295 for entry in self._root.iterdir():
296 if not entry.is_dir():
! 297 continue
298 marker = entry / ACTIVE_MARKER
299 if marker.exists() and now - marker.stat().st_mtime < ACTIVE_GRACE_SECONDS:
300 continue
301 if entry.resolve() not in live_paths:Lines 374-381 374 for dirpath, _, filenames in os.walk(root):
375 for name in filenames:
376 try:
377 total += os.path.getsize(os.path.join(dirpath, name))
! 378 except OSError:
! 379 continue
380 return totallibs/core/kiln_ai/tools/base_tool.pyLines 7-15 7 from kiln_ai.datamodel.json_schema import validate_schema_dict
8 from kiln_ai.datamodel.tool_id import KilnBuiltInToolId, ToolId
9
10 if TYPE_CHECKING:
! 11 from kiln_ai.datamodel.synthetic_world import SyntheticInstance
12
13
14 class ToolFunction(TypedDict):
15 """Typed dict for the function definition within a tool call definition."""libs/core/kiln_ai/tools/kiln_task_tool.pyLines 66-74 66 self, context: ToolCallContext | None = None, **kwargs
67 ) -> KilnTaskToolResult:
68 """Execute the wrapped Kiln task with the given parameters and calling context."""
69 if context is None:
! 70 synthetic_ctx = get_synthetic_instance()
71 context = ToolCallContext(
72 allow_saving=False,
73 synthetic_instance=synthetic_ctx.instance
74 if synthetic_ctx is not Nonelibs/core/kiln_ai/tools/synthetic_tool.pyLines 54-62 54 self._impl = PythonCodeTool(synthetic_tool, project, task)
55
56 @property
57 def synthetic_tool(self) -> SyntheticTool:
! 58 return self._synthetic_tool
59
60 async def id(self) -> ToolId:
61 return self._real_tool_idLines 86-94 86 connection stay in memory with it.
87 """
88 connection = instance.connection
89 if connection is None:
! 90 raise ValueError(
91 f"Synthetic instance {instance.instance_id} has no connection; it cannot "
92 "stand in for a tool server"
93 )
94 server_id = f"syn{instance.instance_id}"[:12].replace("-", "0")Lines 93-101 93 )
94 server_id = f"syn{instance.instance_id}"[:12].replace("-", "0")
95 if connection.transport == "stdio":
96 if not connection.command:
! 97 raise ValueError(
98 f"Synthetic instance {instance.instance_id} declares a stdio "
99 "connection without a command"
100 )
101 local: LocalServerProperties = {Lines 110-118 110 type=ToolServerType.local_mcp,
111 properties=local,
112 )
113 if not connection.url:
! 114 raise ValueError(
115 f"Synthetic instance {instance.instance_id} declares an HTTP connection "
116 "without a url"
117 )
118 remote: RemoteServerProperties = {Lines 144-153 144 self._impl = MCPServerTool(self._server, tool_name)
145
146 @property
147 def connection(self) -> SyntheticInstanceConnection:
! 148 assert self._instance.connection is not None
! 149 return self._instance.connection
150
151 @property
152 def tool_server(self) -> ExternalToolServer:
153 return self._serverLines 158-171 158 async def name(self) -> str:
159 return await self._impl.name()
160
161 async def description(self) -> str:
! 162 return await self._impl.description()
163
164 async def toolcall_definition(self) -> ToolCallDefinition:
! 165 return await self._impl.toolcall_definition()
166
167 async def run(
168 self, context: ToolCallContext | None = None, **kwargs: Any
169 ) -> ToolCallResult:
! 170 return await self._impl.run(context, **kwargs)libs/core/kiln_ai/tools/tool_registry.pyLines 71-79 71 f"{synthetic_ctx.world.replaces_tool_server_id}, but instance "
72 f"{synthetic_ctx.instance.instance_id} has no connection to serve it"
73 )
74 if project is None:
! 75 raise ValueError(
76 f"Unable to resolve synthetic server tool for {tool_id}: requires a parent project/task."
77 )
78 server_id, tool_name = mcp_server_and_tool_name_from_id(tool_id)
79 real_server = next(Lines 225-233 225 elif tool_id.startswith(SYNTHETIC_TOOL_ID_PREFIX):
226 # Direct resolution, for tests and world tooling. Runs the synthetic tool under
227 # its own id; the eval-time swap above is what presents it under the real id.
228 if project is None:
! 229 raise ValueError(
230 f"Unable to resolve tool from id: {tool_id}. Requires a parent project/task."
231 )
232 world_id, synthetic_tool_id = synthetic_world_and_tool_ids_from_id(tool_id)
|
A second judge that reuses a generation can still be reading the instance copy while the first job finalizes it. finalize now only marks the record unchanged; the runner destroys those copies once the job runner completes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019jkErL54oXyYL35E13SRDM
2 tasks
…nch configs Fixtures are no longer Kiln objects. An eval input's synthetic_environment is a world id plus an opaque launch config; the world names a launcher, which turns that config into an instance and releases it. The local-files launcher treats a fixture as a directory under the world with an optional fixture.yaml of facts it reports on the instance (frozen_time, for one). Instances carry a path or an endpoint plus launcher-reported metadata, which the sandbox exports as KILN_SYNTHETIC_<KEY>. Fixture endpoints become directory operations on the local launcher. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019jkErL54oXyYL35E13SRDM
Adjusts the launcher contract and instance record so a hosted world framework can implement it without changing Kiln again: - Launchers report a content_version(world, config) before launch; it replaces the author-only framework_content_hash in the trace fingerprint, so a regenerated fixture (or a framework's new world version) regenerates traces. The local launcher digests the fixture bytes (cached by stat signature) plus the world's declared content_version. - SyntheticInstance.endpoint becomes a structured connection (transport, url, headers, command, args, env). Credentials are excluded from serialization, so they never reach the trace, an API response, or a judge prompt; only the in-memory record handed to tools carries them. - Instances gain changes (what the launcher recorded at finalize, handed to code scorers, not to judges) and a valid/invalid_reason verdict. Finalize now runs after generation and before grading, inside the trace lock, so the persisted record is settled before any judge reads it. Strict worlds skip grading an invalid run (SkippedReason.synthetic_instance_invalid). - A world may name replaces_tool_server_id: while an instance with a connection is active, every tool of that MCP server resolves to the same- named tool on the instance, under the real id, via an in-memory server config. Per-tool bindings still cover everything else. - Retention is documented as launcher policy rather than a Kiln guarantee. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019jkErL54oXyYL35E13SRDM
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
TLDR: Evals for agents with stateful tools need a replica of the tool set that they can read and write without a real backend, so this PR adds synthetic worlds: a world data model, a launcher hook that turns an opaque launch config into an isolated instance per eval job, a registry-level tool swap, and a minimal API. Targets
main.Summary
Before this PR, an eval input could only run against the tools that the run config names. A test of write behavior had to hit a shared backend, and a pinned reference fact drifted when anyone changed that backend. After this PR, an eval input can name a synthetic world and a launch config. The world's launcher turns that config into an isolated instance for the job, the runner replaces the bound real tools with the world's synthetic tools, records the instance on the trace, and gives graders access to the state the run left behind.
Implementation
datamodel/synthetic_world.py:SyntheticWorldis a project child with alaunchername andlauncher_config. It ownsSyntheticTool(a code-tool artifact withreplaces_tool_id) and alib/directory for shared world code.SyntheticEnvironment(world id plus an opaqueconfig) is the reference anEvalInputcarries.SyntheticInstance(apathor a structuredconnection, launcher-reportedmetadata, the launcher'scontent_version, and after finalizechangesand avalidverdict) is the record a trace carries. Connection credentials are excluded from serialization, so they never reach the trace, an API response, or a judge prompt. A world may also namereplaces_tool_server_id: every tool of that MCP server then resolves to the same-named tool on the launched instance, for frameworks that serve a whole tool surface over MCP.datamodel/code_tool.py: the code-tool fields, validators, and sibling-file storage move toCodeToolBase, soCodeToolandSyntheticToolshare one implementation.datamodel/tool_id.py: the newkiln_tool::synthetic::<world_id>::<tool_id>form. A run config or an allowlist rejects it. The registry swap is the only way to reach a synthetic tool.run_context.pyandtools/tool_registry.py: the runner sets a job-scopedsynthetic_instancecontext.tool_from_id_and_projectreturns aSyntheticToolProxyfor a bound id. The proxy answers with the real tool id and the real function name, and runs the synthetic code. A world withstrict=Truerejects an unbound tool id, except a built-in.tools/base_tool.pyand the callers that build aToolCallContext: the context carries the instance. The sandbox child exports it asKILN_SYNTHETIC_*environment variables, puts the worldlib/onsys.path, and offerskiln.synthetic_instance().adapters/eval/eval_runner.py: after the skip checks, a job with a synthetic environment resolves the world and its launcher, launches the instance inside the trace lock, generates, and then grades with the context rebuilt from the trace record. A run that leaves a file-backed instance byte-identical to its source releases the copy after the run, and later readers use the source.adapters/eval/trace_index.pyandEvalItemSource.variant: the trace key gains a fourth slot. The synthetic fingerprint (world, canonical launch config, engine hash) goes in that slot, so two configs never share one generation. A record with no variant maps to the empty string, so existing traces keep their keys.synthetic_worlds/launcher.py: theSyntheticWorldLauncherprotocol (content_version,launch,finalize,release,prune), a registry keyed by the world'slaunchername, andLocalFilesLauncher, which treats a fixture as a directory under the world (fixtures/<fixture_id>/, with an optionalfixture.yamlof facts reported on the instance) and copies it under the Kiln cache. Itscontent_versiondigests the fixture bytes plus the world's declared version, so a regenerated fixture regenerates traces.finalizeruns after generation and before grading, inside the trace lock, so the persisted record is settled before any judge reads it; a strict world skips grading a run the launcher marked invalid (synthetic_instance_invalid). Retention is launcher policy. For the local launcher it follows the trace:prunedeletes only copies that no surviving trace references, then applies a size cap from the newsynthetic_instance_cache_max_gbconfig key. An evicted copy makes a state-reading scorer skip withsynthetic_instance_unavailable. It never triggers a new generation, so every judge scores the same output.EvalTaskInput.synthetic_instancegives judges the ids, the launch config and the reported metadata, with no paths or endpoints, because that model is an API request body. A code-eval scorer can declaresynthetic_instanceand receives the full record through the sandbox inputs. Sandboxed code also seesKILN_SYNTHETIC_*env vars, including one per scalar metadata entry such asKILN_SYNTHETIC_FROZEN_TIME.app/desktop/studio_server/synthetic_world_api.py: CRUD for worlds and tools, a bindings validation endpoint, and directory operations for the local launcher's fixtures (create, upload a data file, list, delete). The TS schema is regenerated.flowchart LR A[EvalInput.synthetic_environment] -->|world + launch config| B[EvalRunner] B -->|launch| C[Launcher: instance] B -->|set context| D[tool_from_id_and_project] D -->|bound id| E[SyntheticToolProxy: real id, synthetic code] D -->|unbound id| F[Real tool] B -->|record instance| G[TaskRun.synthetic_instance] G --> H[Judges: ids only] G --> I[Code scorers: full record]Before you merge
mainskips multi-turn V2 eval inputs, so this PR serves single-turn inputs now. The lifecycle hook encloses the point where the multi-turn lane dispatches, so that lane gets the same behavior when it merges. The multi-turn work and this PR must use the samevariantslot onEvalItemSource.package_project.pydoes not export synthetic worlds. This matches code tools today.app/web_ui/src/lib/generate_schema.shif you add endpoints on top of this branch.Example: a scorer that checks a write
Related Issues
None.
Contributor License Agreement
Left for the PR author to complete.
Checklists
uv run ./checks.sh --agent-modegreen)🤖 Generated with Claude Code
https://claude.ai/code/session_019jkErL54oXyYL35E13SRDM