Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 6 additions & 2 deletions TOOLS.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,9 @@ EXAMPLES:

Creates a root component configuration using the specified name, component ID, configuration JSON, and description.
Not for SQL transformations (`keboola.snowflake-transformation` / `keboola.google-bigquery-transformation`),
data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE).
data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). This IS the tool for
Python (`keboola.python-transformation-v2`), R (`keboola.r-transformation-v2`) and DuckDB
(`keboola.duckdb-transformation`) transformations.

BEFORE CALLING - REQUIRED STEPS:
1. Call `get_components([component_id])` to retrieve the component's `configuration_schema`.
Expand Down Expand Up @@ -397,6 +399,8 @@ USAGE:
- This is THE tool for creating `keboola.snowflake-transformation` and `keboola.google-bigquery-transformation`
components (do NOT use `create_config` for these); the transformation ID is derived automatically from the
workspace SQL dialect.
- Snowflake/BigQuery only. For Python, R, or DuckDB transformations, use `create_config` with the appropriate
`component_id` instead — this tool cannot create them.

EXAMPLES:
- user_input: `Can you create a new transformation out of this sql query?`
Expand Down Expand Up @@ -1379,7 +1383,7 @@ the operations you want to perform. All other fields will remain unchanged.
Use this for modifying SQL transformations created with create_sql_transformation.

WHEN TO USE:
- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R transformations
- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R/DuckDB transformations
- Modifying SQL queries in transformation (add/edit/remove SQL statements)
- Updating transformation block or code block names
- Changing input/output table mappings for the transformation
Expand Down
99 changes: 99 additions & 0 deletions feature_spec/duckdb_transformation_support/RFC.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
# RFC: DuckDB transformation support

Linear: [AI-3112](https://linear.app/keboola/issue/AI-3112/kai-supporting-creation-of-duckdb-transformations)

## Problem

Keboola added `keboola.duckdb-transformation` as a third transformation backend alongside
Snowflake/BigQuery SQL transformations and Python/R transformations. The MCP server has no
awareness of it:

- `create_sql_transformation` / `update_sql_transformation` are hard-anchored to whichever
backend the project's **workspace** uses. The component ID is derived via
`WorkspaceManager.get_sql_dialect()` → `get_sql_transformation_id_from_sql_dialect()`
(`src/keboola_mcp_server/tools/components/utils.py:281-299`), which only maps `'snowflake'`
and `'bigquery'` and raises `ValueError` otherwise. Workspaces themselves
(`src/keboola_mcp_server/workspace.py:714-813`) only support those two backends. DuckDB has
no cloud workspace and was never going to plug into this path.
- `create_config`'s docstring (`tools.py:1121-1138`) tells the caller to use `create_sql_transformation`
for Snowflake/BigQuery and says nothing about Python, R, or DuckDB — callers currently infer
by precedent that non-SQL transformation types go through `create_config`.
- `update_sql_transformation_internal`'s 404 error message (`tools.py:996-1000`) already hints
"if this is a Python or R transformation, use `update_config`..." — DuckDB hits the same 404
and needs the same hint, or the guidance is incomplete/misleading.
- `FOLDER_SUPPORTING_COMPONENT_IDS` (`utils.py:74`) only contains the Python/R component IDs, so
`update_config`'s folder metadata handling (`tools.py:1551-1555`) silently skips folder
management for DuckDB configs even though the UI supports organizing DuckDB transformations
into folders (per the DuckDB transformation docs).

Visible symptom: asking Kai to create or update a DuckDB transformation either fails outright
(if routed through the SQL-transformation tools, which will raise on non-snowflake/bigquery
workspaces) or succeeds via `create_config` by accident, without folder support and without any
tool guidance steering the model there in the first place.

## Required Behavior

| Scenario | Required behavior |
| --- | --- |
| User asks to create/update a `keboola.duckdb-transformation` config | Routed through `create_config` / `update_config`, same as Python/R today — **not** through `create_sql_transformation`/`update_sql_transformation`. |
| User asks for a transformation but doesn't specify backend, and the workspace default isn't what's implied | Kai should recognize DuckDB and Python as the two non-workspace-backed alternatives and ask the user which one, rather than silently guessing. (Prompt-level guidance change, not new code — see Scope.) |
| DuckDB config created/updated via `update_config` | Folder metadata (`folder` param) is applied, same as Python/R — i.e. `FOLDER_SUPPORTING_COMPONENT_IDS` includes the DuckDB component ID. |
| Caller mistakenly calls `update_sql_transformation` on a DuckDB config | 404 error message mentions DuckDB as a valid alternative alongside Python/R, so the caller self-corrects to `update_config`. |
| DuckDB-specific sync actions (`syntax_check`, `lineage_visualization`, `execution_plan_visualization`, `expected_input_tables`) | No new code — `run_sync_action` (`tools.py:1960-2030`) already dispatches generically to whatever sync action a component declares; this works automatically once the DuckDB component's API metadata lists them. |
| DuckDB-specific config parameters (`backend_size`, `timeout`, `duckdb_version`, `use_parquet`, `infer_input_table_data_types`, etc.) | No new code — `create_config`/`update_config` already treat `parameters` as an opaque JSON blob validated against the component's `configuration_schema` fetched via `get_components`/`get_config_examples`. |

## Resolution Strategy

Minimal, additive change mirroring the existing Python/R pattern — no new tools, no workspace
changes:

1. **`utils.py:68`** — add `DUCKDB_TRANSFORMATION_ID = 'keboola.duckdb-transformation'` next to
`PYTHON_TRANSFORMATION_ID` / `R_TRANSFORMATION_ID`.
2. **`utils.py:74`** — add `DUCKDB_TRANSFORMATION_ID` to `FOLDER_SUPPORTING_COMPONENT_IDS`.
3. **`tools.py:1121-1138` (`create_config` docstring)** — extend the "Not for SQL transformations"
line and "WHEN NOT TO USE" bullet to name Python/R/DuckDB explicitly as transformation types
that *do* go through `create_config`, so the model doesn't have to infer this from precedent.
4. **`tools.py:996-1000` (`update_sql_transformation_internal` `ToolError`)** — extend the message
to say "...Python, R, or DuckDB transformation, use `update_config` with component_id
`keboola.python-transformation-v2`, `keboola.r-transformation-v2`, or
`keboola.duckdb-transformation`...".
5. **`create_sql_transformation`/`update_sql_transformation` docstrings (`tools.py:440-473`,
`563-...`)** — add one line clarifying these tools only ever produce Snowflake/BigQuery
transformations (component ID derived from the workspace backend) and that DuckDB is handled
via `create_config`/`update_config`.

No changes to `workspace.py`, `sql_utils.py`, or the sync-action dispatch — those are already
either backend-specific by design (workspace = Snowflake/BigQuery only, intentionally out of
scope for DuckDB) or already generic enough to cover DuckDB for free.

## Scope

In scope:

- The five code changes above (constants, docstrings, error message).
- Unit tests: `DUCKDB_TRANSFORMATION_ID` present in `FOLDER_SUPPORTING_COMPONENT_IDS`; folder
metadata applied on `update_config` for a DuckDB component ID (extend the existing
parametrized Python/R folder test with a DuckDB case rather than adding a new test function);
updated error-message assertion in the `update_sql_transformation` 404 test.
- Version bump (minor — new tool-facing guidance/behavior) + `uv lock` + `TOOLS.md` regen.

Out of scope (tracked separately, each with its own RFC per CONTRIBUTING.md):

- Consolidating `create_sql_transformation`/`create_config`/`update_config` into fewer, more
generic tools, and moving the guidance currently duplicated across tool docstrings into
`project_system_prompt.md` (or a future Agent Skill). Tracked in a follow-up Linear issue,
explicitly scoped to build on top of this change.
- Any DuckDB-specific workspace/backend support for `query_data` — DuckDB is not a workspace
backend and this RFC does not add one.
- New dedicated sync-action tools — `run_sync_action` already covers this generically.

## Testing / Verification

1. `tox` — pytest, black, isort, flake8, check-tools-docs all exit 0.
2. Unit tests in `tests/tools/components/test_utils.py` / `test_tools.py`: extend existing
parametrized folder-metadata and error-message tests with a DuckDB axis, rather than adding
new test functions.
3. Manual E2E via local `.mcp.json` against a project with a DuckDB transformation: confirm
`create_config`/`update_config` with `component_id='keboola.duckdb-transformation'` succeeds,
folder metadata is applied, and `run_sync_action` executes `syntax_check` successfully.
4. `tox -e check-tools-docs` to regenerate `TOOLS.md` after docstring changes.
14 changes: 9 additions & 5 deletions src/keboola_mcp_server/tools/components/tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,8 @@ async def create_sql_transformation(
- This is THE tool for creating `keboola.snowflake-transformation` and `keboola.google-bigquery-transformation`
components (do NOT use `create_config` for these); the transformation ID is derived automatically from the
workspace SQL dialect.
- Snowflake/BigQuery only. For Python, R, or DuckDB transformations, use `create_config` with the appropriate
`component_id` instead — this tool cannot create them.

EXAMPLES:
- user_input: `Can you create a new transformation out of this sql query?`
Expand Down Expand Up @@ -673,7 +675,7 @@ async def update_sql_transformation(
Use this for modifying SQL transformations created with create_sql_transformation.

WHEN TO USE:
- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R transformations
- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R/DuckDB transformations
- Modifying SQL queries in transformation (add/edit/remove SQL statements)
- Updating transformation block or code block names
- Changing input/output table mappings for the transformation
Expand Down Expand Up @@ -995,9 +997,9 @@ async def update_sql_transformation_internal(
if e.response.status_code == 404:
raise ToolError(
f"Configuration '{configuration_id}' was not found under SQL transformation component "
f"'{sql_transformation_id}'. If this is a Python or R transformation, use 'update_config' "
f"with component_id 'keboola.python-transformation-v2' or 'keboola.r-transformation-v2' "
f"instead of 'update_sql_transformation'."
f"'{sql_transformation_id}'. If this is a Python, R, or DuckDB transformation, use 'update_config' "
f"with component_id 'keboola.python-transformation-v2', 'keboola.r-transformation-v2', or "
f"'keboola.duckdb-transformation' instead of 'update_sql_transformation'."
) from e
raise
api_component = await fetch_component(client=client, component_id=sql_transformation_id)
Expand Down Expand Up @@ -1121,7 +1123,9 @@ async def create_config(
"""
Creates a root component configuration using the specified name, component ID, configuration JSON, and description.
Not for SQL transformations (`keboola.snowflake-transformation` / `keboola.google-bigquery-transformation`),
data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE).
data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). This IS the tool for
Python (`keboola.python-transformation-v2`), R (`keboola.r-transformation-v2`) and DuckDB
(`keboola.duckdb-transformation`) transformations.

BEFORE CALLING - REQUIRED STEPS:
1. Call `get_components([component_id])` to retrieve the component's `configuration_schema`.
Expand Down
5 changes: 4 additions & 1 deletion src/keboola_mcp_server/tools/components/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,12 +67,15 @@
BIGQUERY_TRANSFORMATION_ID = 'keboola.google-bigquery-transformation'
PYTHON_TRANSFORMATION_ID = 'keboola.python-transformation-v2'
R_TRANSFORMATION_ID = 'keboola.r-transformation-v2'
DUCKDB_TRANSFORMATION_ID = 'keboola.duckdb-transformation'
VARIABLES_COMPONENT_ID = 'keboola.variables'

# Component IDs for which update_config actively manages folder metadata (set/clear/hint).
# For all other components the folder parameter is accepted but silently skipped to avoid
# unnecessary API calls on components where folder organisation is not expected.
FOLDER_SUPPORTING_COMPONENT_IDS: frozenset[str] = frozenset({PYTHON_TRANSFORMATION_ID, R_TRANSFORMATION_ID})
FOLDER_SUPPORTING_COMPONENT_IDS: frozenset[str] = frozenset(
{PYTHON_TRANSFORMATION_ID, R_TRANSFORMATION_ID, DUCKDB_TRANSFORMATION_ID}
)


# ============================================================================
Expand Down
30 changes: 21 additions & 9 deletions tests/tools/components/test_tools.py
Original file line number Diff line number Diff line change
Expand Up @@ -1314,7 +1314,7 @@ async def test_update_sql_transformation_wrong_component_type(
) -> None:
"""
update_sql_transformation should raise ToolError with actionable guidance when the
configuration belongs to a Python/R transformation (Storage returns 404 for the SQL
configuration belongs to a Python/R/DuckDB transformation (Storage returns 404 for the SQL
component + config-ID combination).
"""
context = mcp_context_components_configs
Expand Down Expand Up @@ -1342,6 +1342,8 @@ async def test_update_sql_transformation_wrong_component_type(
assert 'keboola.snowflake-transformation' in error_msg
assert 'update_config' in error_msg
assert 'keboola.python-transformation-v2' in error_msg
assert 'keboola.r-transformation-v2' in error_msg
assert 'keboola.duckdb-transformation' in error_msg


@pytest.mark.asyncio
Expand Down Expand Up @@ -1655,14 +1657,23 @@ async def test_update_config(


@pytest.mark.parametrize(
('folder', 'cfg_count', 'cfg_folders', 'expect_folder_metadata', 'expect_folder_delete', 'expect_hint'),
(
'component_id',
'folder',
'cfg_count',
'cfg_folders',
'expect_folder_metadata',
'expect_folder_delete',
'expect_hint',
),
[
('Analytics', 0, [], True, False, False),
(' Analytics ', 0, [], True, False, False),
(None, 5, [], False, False, False),
(None, 25, ['Analytics'], False, False, True),
(None, 25, [], False, False, True),
('', 5, [], False, True, False),
('keboola.python-transformation-v2', 'Analytics', 0, [], True, False, False),
('keboola.python-transformation-v2', ' Analytics ', 0, [], True, False, False),
('keboola.python-transformation-v2', None, 5, [], False, False, False),
('keboola.python-transformation-v2', None, 25, ['Analytics'], False, False, True),
('keboola.python-transformation-v2', None, 25, [], False, False, True),
('keboola.python-transformation-v2', '', 5, [], False, True, False),
('keboola.duckdb-transformation', 'Analytics', 0, [], True, False, False),
],
ids=[
'folder_provided',
Expand All @@ -1671,13 +1682,15 @@ async def test_update_config(
'no_folder_many_with_folders',
'no_folder_many_no_folders',
'folder_empty_deletes',
'duckdb_folder_provided',
],
)
@pytest.mark.asyncio
async def test_update_config_folder(
mocker: MockerFixture,
mcp_context_components_configs: Context,
mock_component: dict[str, Any],
component_id: str,
folder: Any,
cfg_count: int,
cfg_folders: list[str],
Expand All @@ -1688,7 +1701,6 @@ async def test_update_config_folder(
"""Test folder metadata is set/cleared and folder hint is returned by update_config."""
context = mcp_context_components_configs
keboola_client = KeboolaClient.from_state(context.session.state)
component_id = 'keboola.python-transformation-v2'
mock_component['id'] = component_id
configuration_id = 'cfg-folder-test'
existing = {
Expand Down
Loading