AI-1167: shared code support for SQL transformation tooling - #499
AI-1167: shared code support for SQL transformation tooling#499Matovidlo wants to merge 22 commits into
Conversation
|
@claude review |
There was a problem hiding this comment.
Pull request overview
Adds a new RFC documenting how the Keboola MCP server should support “shared code” for transformations, including discovery, CRUD workflows, and transformation linkage fields.
Changes:
- Introduces
feature_spec/shared_code_support/RFC.mddescribing shared code behavior, tool/API shape, and required model/tooling updates - Specifies a new read-only
get_shared_codestool plus extensions to existing config/transformation tools for shared code linkage - Documents expected workflows and a proposed test plan for the feature
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Describes the full scope of changes needed to give the MCP server awareness of Keboola's shared code feature: a new get_shared_codes discovery tool, shared code CRUD via existing generic tools (with a row_id param extension to add_config_row), shared_code_id / shared_code_row_ids linkage params on create/update transformation tools, and a system prompt section that guides the LLM on when and how to use shared code. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
a7d2ee1 to
e33f6cc
Compare
There was a problem hiding this comment.
Pull request overview
Adds a new RFC documenting how the MCP server should support Keboola “shared code” for transformations (discovery, CRUD via existing tools, and transformation linkage fields).
Changes:
- Introduces
feature_spec/shared_code_support/RFC.mddescribing shared code behavior, tool/API changes, and implementation/testing plan. - Specifies a new read-only discovery tool (
get_shared_codes) and required extensions to existing config/transformation tools and models. - Documents required system-prompt guidance for shared code discovery/creation and correct Mustache/linkage usage.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Implements RFC at feature_spec/shared_code_support/RFC.md: - New read-only `get_shared_codes` tool for discovering `keboola.shared-code` libraries and their reusable snippets, with optional filter by transformation component ID. - `add_config_row` accepts optional `row_id` forwarded as SAPI `rowId`, so Mustache placeholder keys (e.g. `dumpfiles`) can be set on shared-code rows at creation time. Storage client `configuration_row_create` updated to forward the parameter (was missing). - `create_sql_transformation` / `update_sql_transformation` accept `shared_code_id` + `shared_code_row_ids` and write them at the configuration root (alongside `parameters` and `storage`). - `update_sql_transformation` dispatches new `TfSetSharedCode` and `TfRemoveSharedCode` parameter-update ops against the configuration root. - `create_config` / `update_config` accept `shared_code_id` + `shared_code_row_ids` for Python / R / DuckDB transformations. - `TransformationConfiguration` model gains optional `shared_code_id` / `shared_code_row_ids` fields so existing linkage survives round-trips. - New `SharedCodeRow` / `SharedCodeConfig` / `GetSharedCodesOutput` output models. - System prompt: comprehensive Shared Code section teaching the LLM discovery, parent-config creation, row addition with Mustache keys, referencing from transformations, and validation rules (case sensitivity, bidirectional script/array consistency). - Unit tests: 16 new cases covering rowId forwarding, root-field persistence at create time, set/remove dispatch, parameter-only round-trip preservation, `create_config`/`update_config` shared-code flows, and `get_shared_codes` happy path + filter + unknown-filter rejection. - Integration tests: 7 new live-stack tests with a session-scoped `shared_code_parent_factory` fixture for cleanup. - TOOLS.md regenerated. - Version bumped 1.60.0 -> 1.61.0; uv.lock synced. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tionId Found while running the live demo on a real BigQuery project: the previous implementation persisted shared-code configs in a form the Keboola UI and runtime expansion cannot consume. Root causes: - `create_config` and `add_config_row` always wrap the user-provided `parameters` under a `"parameters"` key in the configuration body. For `keboola.shared-code` parent libraries the platform expects `componentId` at the configuration root; for shared-code rows it expects `code_content` at the row configuration root. Nesting under `parameters` makes the data invisible to the platform's resolver. - `create_config` had no way to specify the configuration ID, so the SAPI auto-assigned a UUID. The UI and runtime look up shared-code libraries by the conventional `shared-codes.<transformation-component-id>` ID, so auto-assigned IDs are unusable. Fixes: - `clients/storage.py:configuration_create` accepts an optional `configuration_id` and forwards it as the SAPI `configurationId` form field (mirrors the `row_id` -> `rowId` fix from the initial PR). - `tools/components/tools.py:create_config` exposes `configuration_id` and special-cases `component_id == "keboola.shared-code"` to write the user-provided `parameters` dict at the configuration root rather than nesting it. - `tools/components/tools.py:add_config_row` applies the same flat-body special case for shared-code rows. - `tools/components/tools.py:get_shared_codes` reads `componentId` / `code_content` from the configuration root first, with a fallback to `parameters.<field>` so legacy configs created by the wrapped wire format remain discoverable until they are migrated. - System prompt: documents that `configuration_id="shared-codes.<...>"` is required when creating a parent library, and explains the flat-body unwrapping. - Unit tests: 3 new cases pinning the flat-body create_config / add_config_row behaviour and the root-vs-parameters fallback in get_shared_codes; existing `test_create_config` updated for the new `configuration_id=None` kwarg. - TOOLS.md regenerated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without an explicit `include=rows` parameter the Storage API's
GET /components/{id}/configs/{cid} endpoint omits the configuration's row
data. This caused `get_shared_codes` to return every library with an empty
`rows` list — visible immediately on the live demo project.
- `clients/storage.py:configuration_detail` accepts an optional `include`
parameter and forwards it as `?include=<csv>` on the query string.
- `tools/components/tools.py:get_shared_codes` requests `include=['rows']`
so rows are populated.
- Mock signatures in the unit tests updated to accept the new kwarg.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…n detail fetch
Live demo against a real BigQuery project surfaced four follow-up issues with
the previous AI-1167 implementation:
A) Mustache placeholders never substituted at runtime
The Keboola platform's runtime expansion does NOT do text substitution of
`{{rowId}}` inside the user's surrounding SQL. It expands ONLY a dedicated
marker code block whose entire script is the placeholder
(`Shared Code (<shared_code_id>-<row_id>)`, script `["{{rowId}}"]`).
`create_sql_transformation` / `update_sql_transformation` now automatically
append/sync these marker code blocks whenever `shared_code_id` +
`shared_code_row_ids` are set, mirroring what the Keboola UI emits. Without
the markers the snippet is silently skipped and the user-authored SQL fails
(or runs wrong) at job time.
B) `get_configs` returned `configuration_rows: null` for shared-code parents
The detail fetch did not pass `include=rows` to the Storage API, so row
data was always omitted. `get_configs` now requests `include=['rows']`
in its concurrent per-config detail fetch path.
C) `add_config_row` response did not surface the actual row ID assigned by SAPI
Added `configuration_row_id` to `ConfigToolOutput`; the row-create tool now
populates it from the API response and logs a warning when a requested
`row_id` does not match the assigned id (which would indicate SAPI rejected
or transformed the form value).
D) System prompt incomplete on the substitution semantics
The "Shared Code" section now explicitly states (1) the LLM does not need
to author marker blocks — the SQL transformation tools emit them — and
(2) shared-code row content must be a complete, independently executable
statement, because the runtime substitutes each placeholder with the row's
`code_content` and runs the result as its own query. Examples added.
New helpers in `tools/components/utils.py`:
- `shared_code_marker_code_name(sid, rid)`
- `is_shared_code_marker(name, script)`
- `build_shared_code_marker_codes(sid, row_ids)` (Pydantic / create-path)
- `apply_shared_code_markers(tf_cfg, sid, row_ids)` (Pydantic / create-path)
- `sync_shared_code_markers_in_dict(updated_configuration)` (dict / update-path)
Tests:
- `test_create_sql_transformation_emits_shared_code_marker_blocks[snowflake/bigquery]`
- `test_update_sql_transformation_set_then_remove_shared_code_syncs_markers`
- `test_add_config_row_surfaces_assigned_row_id`
- Existing `test_get_configs_detail*` updated for the `include=['rows']` call.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… + minimize prompt
The shared-code workflow is now strictly: pass `shared_code_id` + `shared_code_row_ids`
to the transformation create/update tool; the tool emits the UI-canonical `Shared Code (...)`
marker code blocks; the platform substitutes them at runtime. Inline placeholders are not
substituted (per Keboola dev docs — substitution operates on a script ARRAY ELEMENT, not
text within a string), so the tool now hard-rejects configurations that try to use a
`{{ rowId }}` placeholder without the corresponding root linkage.
Enforcement (new):
- `utils.validate_shared_code_linkage(parameters, shared_code_id, shared_code_row_ids)`
scans `parameters.blocks[*].codes[*].script` for `{{ rowId }}` placeholders and raises
ValueError when:
* a placeholder is referenced but `shared_code_id` is empty, or
* a referenced row is not in `shared_code_row_ids`.
- Wired into `create_sql_transformation`, `update_sql_transformation_internal`, and the
`create_config` / `update_config` paths for transformation components (Python/R/etc.).
Symmetry (new):
- `create_config` / `update_config` now auto-emit `Shared Code (<sid>-<rid>)` marker code
blocks via `sync_shared_code_markers_in_dict` whenever the target `component_id` is a
transformation backend that supports shared code, matching what
`create_sql_transformation` already does. The LLM/agent only ever supplies the linkage
fields — it never has to author the marker blocks itself.
System prompt:
- The Shared Code section is rewritten and shortened from 103 lines to 32. The canonical
workflow is presented as a single table, plus three hard rules (complete-statement row
content, linkage-required placeholders, case-sensitive row IDs).
Tests:
- `test_create_sql_transformation_rejects_placeholder_without_linkage`
- `test_create_sql_transformation_rejects_placeholder_missing_from_row_ids`
- `test_create_config_emits_markers_for_python_transformation`
All gates green (97 → 104 component tests, flake8/black OK).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If a transformation already has a code block whose script is exactly
`["{{ rowId }}"]` (the platform's only substitutable form), do not also emit
a Shared Code (...) marker for that row — emitting one would execute the
snippet twice. Inline placeholders inside other SQL strings are ignored
because the platform does not substitute those, so the marker is still needed.
Bump version to 1.61.1; update prompt to document the skip-if-already-referenced
behavior; add 151 lines of tests covering pure-placeholder detection, marker
de-duplication on create/update, and inline-placeholder ignore.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
…ng caveat The RFC was correct at the high level but missed several wire-format and runtime-substitution details that only surfaced during live testing. The running prompt and TOOLS.md already documented the as-built state; the RFC did not. RFC updates (deltas keyed to commit SHAs in a new "Implementation Deltas" section at the bottom): - Flat-body wire format for `keboola.shared-code` (componentId / code_content at config / row root, not under "parameters"); conventional configurationId is required, not optional (67b893c). - `configuration_detail` needs `include=['rows']` for get_shared_codes to see any rows (30faec4). - Runtime substitutes `{{ rowId }}` only when it is the sole array element of a script; tools auto-emit UI-canonical `Shared Code (sid-rid)` marker blocks on every create/update path, for SQL and Python/R/DuckDB alike (91c112d, aa8fc63). - Placeholder without matching root linkage is now a hard ValueError (aa8fc63). - Auto-emit is skipped when the user already has a pure `["{{ rowId }}"]` block, to avoid running the snippet twice (31cee8b). - New caveat §6d: auto-emit position is END of block, so callers must reorder when user code depends on the shared snippet's side-effect (SET vars, temp tables/views). Tracked as a follow-up. Prompt update: add a "Marker ordering" hard-rule bullet pointing to the same caveat with the concrete remove_code + add_code(position="start") workaround. No source changes; tests unaffected. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 10 out of 11 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
src/keboola_mcp_server/tools/components/tools.py:1530
- Shared-code rows are now created with
code_contentat the row configuration root, butupdate_config_rowstill appliesparameter_updatesunder a nestedparametersobject. As a result, the documented “edit snippet” flow will writeparameters.code_contentwhile leaving the canonical rootcode_content(whichget_shared_codesand the runtime read first) unchanged. Add the same shared-code flat-body special case toupdate_config_row_internalso updates to shared-code rows mutate the root payload.
# `keboola.shared-code` rows use a flat configuration body — the platform reads
# `code_content` at the row configuration root, not under `parameters`.
if component_id == SHARED_CODE_COMPONENT_ID:
configuration_payload: dict[str, Any] = dict(parameters or {})
else:
configuration_payload = {'storage': storage_cfg, 'parameters': parameters}
src/keboola_mcp_server/tools/components/tools.py:1348
- The documentation says shared-code parent libraries require a conventional
configuration_id, but the tool still allowscomponent_id="keboola.shared-code"with an emptyconfiguration_id, which creates an auto-assigned config ID that the UI/runtime will not discover. Reject shared-code parent creation unlessconfiguration_idis provided (and ideally matches the expectedshared-codes.<transformation-component-id>value).
# `keboola.shared-code` parent libraries use a flat configuration body — the platform's UI
# and runtime expansion read `componentId` at the configuration root, not under `parameters`.
# For every other component the generic wrapper applies.
if component_id == SHARED_CODE_COMPONENT_ID:
configuration_payload: dict[str, Any] = dict(parameters or {})
else:
configuration_payload = {'storage': storage_cfg, 'parameters': parameters}
…dd-support-for-shared-codes-to-sql_transformation-tooling # Conflicts: # TOOLS.md # pyproject.toml # src/keboola_mcp_server/tools/components/tools.py # src/keboola_mcp_server/tools/components/utils.py # tests/tools/components/test_tools.py # uv.lock
…tool list; black) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… ordering
- split the shared-code rules into tool-enforced rejections (pure-placeholder linkage,
row_ids-require-id, case sensitivity) vs runtime requirements (snippet completeness,
pure-element-only substitution), since the tool does not validate snippet completeness
or reject inline {{ rowId }} occurrences
- fix the marker-ordering guidance: the sync now strips and re-appends all marker blocks,
so reordering an auto-emitted marker does not persist. Instruct authoring a non-marker
pure ["{{ rowId }}"] block at the desired position, which the de-dup logic respects.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The set_shared_code parameter_update wrote an empty shared_code_id into the configuration root, leaving an inconsistent state (row IDs set but no valid library ID). It now raises and directs callers to use remove_shared_code to clear linkage. Adds a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ate log - create_config now rejects a keboola.shared-code configuration_id that doesn't match the conventional shared-codes.<componentId-suffix> derived from parameters.componentId, so a typo can no longer create a library the UI/runtime expansion won't resolve - add_config_row log now reports "configuration row <id>" instead of "new configuration", including the assigned row id, so row-linkage debugging isn't misled - add regression test for the non-conventional-ID rejection Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…older
is_shared_code_marker used a loose startswith('{{')/endswith('}}') check that would
misclassify a multi-placeholder script element as a marker and strip user-authored code
during marker sync. It now requires a full match against PURE_SHARED_CODE_PLACEHOLDER_RE.
Adds a parametrized regression test.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…odes A keboola.shared-code parent missing componentId (at the root and under parameters) has an ambiguous backend type; get_shared_codes now logs a warning and skips it instead of emitting a SharedCodeConfig with an empty transformation_component_id that breaks filtering/consumption. Adds a regression test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The live-stack Integration Tests assert the exact set of registered tools; the new
get_shared_codes tool was missing from that set, so the suite failed with 'Unexpected new
tools: {get_shared_codes}'.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…dd-support-for-shared-codes-to-sql_transformation-tooling # Conflicts: # pyproject.toml # uv.lock
| target_component_id = str((parameters or {}).get('componentId') or '') | ||
| expected_configuration_id = ( | ||
| f'shared-codes.{target_component_id.split(".", 1)[-1]}' if target_component_id else '' | ||
| ) |
There was a problem hiding this comment.
Fixed ✅ in commit 19d78ac
create_config now rejects a keboola.shared-code parent when parameters.componentId is empty, before calling SAPI — such a config has no backend association and would later be skipped by get_shared_codes. (The conventional-ID derivation and match check now always run, since componentId is guaranteed present.) Covered by new test test_create_config_shared_code_parent_requires_component_id.
| updated_root = update_params(configuration_payload, parameter_updates) | ||
| siblings = {key: updated_root[key] for key in ('storage', 'processors') if key in updated_root} | ||
| flat_body = {key: value for key, value in updated_root.items() if key not in siblings} | ||
| validated_flat = validate_row_parameters_configuration( | ||
| component=component, | ||
| parameters=flat_body, | ||
| initial_message='Applying the "parameter_updates" resulted in an invalid row configuration.', | ||
| configuration_id=configuration_id, | ||
| configuration_row_id=configuration_row_id, | ||
| ) | ||
| configuration_payload = {**validated_flat, **siblings} |
There was a problem hiding this comment.
Fixed ✅ in commit 19d78ac
The shared-code row update path now flattens a legacy nested parameters wrapper into the flat root before validation: it pops flat_body['parameters'] and merges it under the root (root wins). Because validate_row_parameters_configuration unwraps a parameters key and returns only its contents, without this a flat-root code_content update would be silently dropped for legacy wrapper-created rows. Covered by new test test_update_config_row_shared_code_flattens_legacy_parameters_wrapper.
…y row wrappers - create_config now rejects a keboola.shared-code parent created without parameters.componentId (it would have no backend association and get_shared_codes would later skip it) - update_config_row now flattens a legacy nested `parameters` wrapper into the row root before validation, so a flat-root code_content update sticks instead of being dropped by the validator (which unwraps and returns only the `parameters` contents) - add regression tests for both Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Description
Linear: AI-1167
Change Type
Summary
Implements the RFC at
feature_spec/shared_code_support/RFC.md(commite33f6ccb). The MCP server can now discover, create, link and unlink Keboola shared code libraries — reusable SQL/Python/R snippets referenced from transformations via Mustache placeholders ({{ rowId }}).New surface (LLM-visible):
get_shared_codes(transformation_component_ids=[...])— read-only discovery tool; listskeboola.shared-codeparent libraries and their rows.add_config_row(..., row_id="<mustache-key>")— optional explicit row ID forwarded as SAPIrowId. Required so the LLM can set Mustache placeholder keys on shared-code rows at creation time.create_sql_transformation(..., shared_code_id="", shared_code_row_ids=[])— links a new SQL transformation to a shared-code library at creation time.update_sql_transformation(..., parameter_updates=[...])— two new discriminated ops:set_shared_code(replaces linkage) andremove_shared_code(clears linkage). Both patch the configuration root, alongsideparameters/storage.create_config(..., shared_code_id="", shared_code_row_ids=[])/update_config(..., shared_code_id=..., shared_code_row_ids=...)— same fields for Python/R/DuckDB transformations created via the generic config tools.Internal:
TransformationConfigurationmodel gains optionalshared_code_id/shared_code_row_idsfields so existing linkage survives deserialize/re-serialize cycles.configuration_row_createstorage-client method forwardsrowIdto SAPI (was missing).Testing
Streamable-HTTPtransports)Optional testing
canary-orionMCP (Streamable-HTTP)canary-orioncanary-orionAutomated coverage:
tests/tools/components/test_tools.py): 16 new cases — rowId forwarding, root-field persistence at create,TfSetSharedCode/TfRemoveSharedCodedispatch, parameter-only update preservation,create_config/update_configset/clear/preserve flows,get_shared_codeshappy path + filter + unknown-filter rejection.integtests/tools/components/test_tools.py): 7 new live-stack tests with ashared_code_parent_factoryfixture for cleanup. Covers discovery (empty + filter), rowId forwarding to SAPI, create transformation with shared code, set/remove round-trip viaupdate_sql_transformation, parameter-only update preservation (regression guard),update_configset/clear/preserve.toxclean: flake8 OK, black OK, 991 unit tests pass, 37 integtests collect.Checklist
🤖 Generated with Claude Code