diff --git a/TOOLS.md b/TOOLS.md index fb0861cf4..9084b7716 100644 --- a/TOOLS.md +++ b/TOOLS.md @@ -211,7 +211,9 @@ EXAMPLES: Creates a root component configuration using the specified name, component ID, configuration JSON, and description. Not for SQL transformations (`keboola.snowflake-transformation` / `keboola.google-bigquery-transformation`), -data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). +data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). This IS the tool for +Python (`keboola.python-transformation-v2`), R (`keboola.r-transformation-v2`) and DuckDB +(`keboola.duckdb-transformation`) transformations. BEFORE CALLING - REQUIRED STEPS: 1. Call `get_components([component_id])` to retrieve the component's `configuration_schema`. @@ -397,6 +399,8 @@ USAGE: - This is THE tool for creating `keboola.snowflake-transformation` and `keboola.google-bigquery-transformation` components (do NOT use `create_config` for these); the transformation ID is derived automatically from the workspace SQL dialect. +- Snowflake/BigQuery only. For Python, R, or DuckDB transformations, use `create_config` with the appropriate + `component_id` instead — this tool cannot create them. EXAMPLES: - user_input: `Can you create a new transformation out of this sql query?` @@ -1379,7 +1383,7 @@ the operations you want to perform. All other fields will remain unchanged. Use this for modifying SQL transformations created with create_sql_transformation. WHEN TO USE: -- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R transformations +- SQL transformations only (Snowflake/BigQuery); use update_config for Python/R/DuckDB transformations - Modifying SQL queries in transformation (add/edit/remove SQL statements) - Updating transformation block or code block names - Changing input/output table mappings for the transformation diff --git a/feature_spec/duckdb_transformation_support/RFC.md b/feature_spec/duckdb_transformation_support/RFC.md new file mode 100644 index 000000000..816ca7172 --- /dev/null +++ b/feature_spec/duckdb_transformation_support/RFC.md @@ -0,0 +1,99 @@ +# RFC: DuckDB transformation support + +Linear: [AI-3112](https://linear.app/keboola/issue/AI-3112/kai-supporting-creation-of-duckdb-transformations) + +## Problem + +Keboola added `keboola.duckdb-transformation` as a third transformation backend alongside +Snowflake/BigQuery SQL transformations and Python/R transformations. The MCP server has no +awareness of it: + +- `create_sql_transformation` / `update_sql_transformation` are hard-anchored to whichever + backend the project's **workspace** uses. The component ID is derived via + `WorkspaceManager.get_sql_dialect()` → `get_sql_transformation_id_from_sql_dialect()` + (`src/keboola_mcp_server/tools/components/utils.py:281-299`), which only maps `'snowflake'` + and `'bigquery'` and raises `ValueError` otherwise. Workspaces themselves + (`src/keboola_mcp_server/workspace.py:714-813`) only support those two backends. DuckDB has + no cloud workspace and was never going to plug into this path. +- `create_config`'s docstring (`tools.py:1121-1138`) tells the caller to use `create_sql_transformation` + for Snowflake/BigQuery and says nothing about Python, R, or DuckDB — callers currently infer + by precedent that non-SQL transformation types go through `create_config`. +- `update_sql_transformation_internal`'s 404 error message (`tools.py:996-1000`) already hints + "if this is a Python or R transformation, use `update_config`..." — DuckDB hits the same 404 + and needs the same hint, or the guidance is incomplete/misleading. +- `FOLDER_SUPPORTING_COMPONENT_IDS` (`utils.py:74`) only contains the Python/R component IDs, so + `update_config`'s folder metadata handling (`tools.py:1551-1555`) silently skips folder + management for DuckDB configs even though the UI supports organizing DuckDB transformations + into folders (per the DuckDB transformation docs). + +Visible symptom: asking Kai to create or update a DuckDB transformation either fails outright +(if routed through the SQL-transformation tools, which will raise on non-snowflake/bigquery +workspaces) or succeeds via `create_config` by accident, without folder support and without any +tool guidance steering the model there in the first place. + +## Required Behavior + +| Scenario | Required behavior | +| --- | --- | +| User asks to create/update a `keboola.duckdb-transformation` config | Routed through `create_config` / `update_config`, same as Python/R today — **not** through `create_sql_transformation`/`update_sql_transformation`. | +| User asks for a transformation but doesn't specify backend, and the workspace default isn't what's implied | Kai should recognize DuckDB and Python as the two non-workspace-backed alternatives and ask the user which one, rather than silently guessing. (Prompt-level guidance change, not new code — see Scope.) | +| DuckDB config created/updated via `update_config` | Folder metadata (`folder` param) is applied, same as Python/R — i.e. `FOLDER_SUPPORTING_COMPONENT_IDS` includes the DuckDB component ID. | +| Caller mistakenly calls `update_sql_transformation` on a DuckDB config | 404 error message mentions DuckDB as a valid alternative alongside Python/R, so the caller self-corrects to `update_config`. | +| DuckDB-specific sync actions (`syntax_check`, `lineage_visualization`, `execution_plan_visualization`, `expected_input_tables`) | No new code — `run_sync_action` (`tools.py:1960-2030`) already dispatches generically to whatever sync action a component declares; this works automatically once the DuckDB component's API metadata lists them. | +| DuckDB-specific config parameters (`backend_size`, `timeout`, `duckdb_version`, `use_parquet`, `infer_input_table_data_types`, etc.) | No new code — `create_config`/`update_config` already treat `parameters` as an opaque JSON blob validated against the component's `configuration_schema` fetched via `get_components`/`get_config_examples`. | + +## Resolution Strategy + +Minimal, additive change mirroring the existing Python/R pattern — no new tools, no workspace +changes: + +1. **`utils.py:68`** — add `DUCKDB_TRANSFORMATION_ID = 'keboola.duckdb-transformation'` next to + `PYTHON_TRANSFORMATION_ID` / `R_TRANSFORMATION_ID`. +2. **`utils.py:74`** — add `DUCKDB_TRANSFORMATION_ID` to `FOLDER_SUPPORTING_COMPONENT_IDS`. +3. **`tools.py:1121-1138` (`create_config` docstring)** — extend the "Not for SQL transformations" + line and "WHEN NOT TO USE" bullet to name Python/R/DuckDB explicitly as transformation types + that *do* go through `create_config`, so the model doesn't have to infer this from precedent. +4. **`tools.py:996-1000` (`update_sql_transformation_internal` `ToolError`)** — extend the message + to say "...Python, R, or DuckDB transformation, use `update_config` with component_id + `keboola.python-transformation-v2`, `keboola.r-transformation-v2`, or + `keboola.duckdb-transformation`...". +5. **`create_sql_transformation`/`update_sql_transformation` docstrings (`tools.py:440-473`, + `563-...`)** — add one line clarifying these tools only ever produce Snowflake/BigQuery + transformations (component ID derived from the workspace backend) and that DuckDB is handled + via `create_config`/`update_config`. + +No changes to `workspace.py`, `sql_utils.py`, or the sync-action dispatch — those are already +either backend-specific by design (workspace = Snowflake/BigQuery only, intentionally out of +scope for DuckDB) or already generic enough to cover DuckDB for free. + +## Scope + +In scope: + +- The five code changes above (constants, docstrings, error message). +- Unit tests: `DUCKDB_TRANSFORMATION_ID` present in `FOLDER_SUPPORTING_COMPONENT_IDS`; folder + metadata applied on `update_config` for a DuckDB component ID (extend the existing + parametrized Python/R folder test with a DuckDB case rather than adding a new test function); + updated error-message assertion in the `update_sql_transformation` 404 test. +- Version bump (minor — new tool-facing guidance/behavior) + `uv lock` + `TOOLS.md` regen. + +Out of scope (tracked separately, each with its own RFC per CONTRIBUTING.md): + +- Consolidating `create_sql_transformation`/`create_config`/`update_config` into fewer, more + generic tools, and moving the guidance currently duplicated across tool docstrings into + `project_system_prompt.md` (or a future Agent Skill). Tracked in a follow-up Linear issue, + explicitly scoped to build on top of this change. +- Any DuckDB-specific workspace/backend support for `query_data` — DuckDB is not a workspace + backend and this RFC does not add one. +- New dedicated sync-action tools — `run_sync_action` already covers this generically. + +## Testing / Verification + +1. `tox` — pytest, black, isort, flake8, check-tools-docs all exit 0. +2. Unit tests in `tests/tools/components/test_utils.py` / `test_tools.py`: extend existing + parametrized folder-metadata and error-message tests with a DuckDB axis, rather than adding + new test functions. +3. Manual E2E via local `.mcp.json` against a project with a DuckDB transformation: confirm + `create_config`/`update_config` with `component_id='keboola.duckdb-transformation'` succeeds, + folder metadata is applied, and `run_sync_action` executes `syntax_check` successfully. +4. `tox -e check-tools-docs` to regenerate `TOOLS.md` after docstring changes. diff --git a/src/keboola_mcp_server/tools/components/tools.py b/src/keboola_mcp_server/tools/components/tools.py index cc5d2a998..210c6f0c1 100644 --- a/src/keboola_mcp_server/tools/components/tools.py +++ b/src/keboola_mcp_server/tools/components/tools.py @@ -463,6 +463,8 @@ async def create_sql_transformation( - This is THE tool for creating `keboola.snowflake-transformation` and `keboola.google-bigquery-transformation` components (do NOT use `create_config` for these); the transformation ID is derived automatically from the workspace SQL dialect. + - Snowflake/BigQuery only. For Python, R, or DuckDB transformations, use `create_config` with the appropriate + `component_id` instead — this tool cannot create them. EXAMPLES: - user_input: `Can you create a new transformation out of this sql query?` @@ -673,7 +675,7 @@ async def update_sql_transformation( Use this for modifying SQL transformations created with create_sql_transformation. WHEN TO USE: - - SQL transformations only (Snowflake/BigQuery); use update_config for Python/R transformations + - SQL transformations only (Snowflake/BigQuery); use update_config for Python/R/DuckDB transformations - Modifying SQL queries in transformation (add/edit/remove SQL statements) - Updating transformation block or code block names - Changing input/output table mappings for the transformation @@ -995,9 +997,9 @@ async def update_sql_transformation_internal( if e.response.status_code == 404: raise ToolError( f"Configuration '{configuration_id}' was not found under SQL transformation component " - f"'{sql_transformation_id}'. If this is a Python or R transformation, use 'update_config' " - f"with component_id 'keboola.python-transformation-v2' or 'keboola.r-transformation-v2' " - f"instead of 'update_sql_transformation'." + f"'{sql_transformation_id}'. If this is a Python, R, or DuckDB transformation, use 'update_config' " + f"with component_id 'keboola.python-transformation-v2', 'keboola.r-transformation-v2', or " + f"'keboola.duckdb-transformation' instead of 'update_sql_transformation'." ) from e raise api_component = await fetch_component(client=client, component_id=sql_transformation_id) @@ -1121,7 +1123,9 @@ async def create_config( """ Creates a root component configuration using the specified name, component ID, configuration JSON, and description. Not for SQL transformations (`keboola.snowflake-transformation` / `keboola.google-bigquery-transformation`), - data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). + data apps (`keboola.data-apps`) or flows — use the dedicated tools (see WHEN NOT TO USE). This IS the tool for + Python (`keboola.python-transformation-v2`), R (`keboola.r-transformation-v2`) and DuckDB + (`keboola.duckdb-transformation`) transformations. BEFORE CALLING - REQUIRED STEPS: 1. Call `get_components([component_id])` to retrieve the component's `configuration_schema`. diff --git a/src/keboola_mcp_server/tools/components/utils.py b/src/keboola_mcp_server/tools/components/utils.py index 7bcad1e6c..4ad2beb51 100644 --- a/src/keboola_mcp_server/tools/components/utils.py +++ b/src/keboola_mcp_server/tools/components/utils.py @@ -67,12 +67,15 @@ BIGQUERY_TRANSFORMATION_ID = 'keboola.google-bigquery-transformation' PYTHON_TRANSFORMATION_ID = 'keboola.python-transformation-v2' R_TRANSFORMATION_ID = 'keboola.r-transformation-v2' +DUCKDB_TRANSFORMATION_ID = 'keboola.duckdb-transformation' VARIABLES_COMPONENT_ID = 'keboola.variables' # Component IDs for which update_config actively manages folder metadata (set/clear/hint). # For all other components the folder parameter is accepted but silently skipped to avoid # unnecessary API calls on components where folder organisation is not expected. -FOLDER_SUPPORTING_COMPONENT_IDS: frozenset[str] = frozenset({PYTHON_TRANSFORMATION_ID, R_TRANSFORMATION_ID}) +FOLDER_SUPPORTING_COMPONENT_IDS: frozenset[str] = frozenset( + {PYTHON_TRANSFORMATION_ID, R_TRANSFORMATION_ID, DUCKDB_TRANSFORMATION_ID} +) # ============================================================================ diff --git a/tests/tools/components/test_tools.py b/tests/tools/components/test_tools.py index c14f501c4..26022b94b 100644 --- a/tests/tools/components/test_tools.py +++ b/tests/tools/components/test_tools.py @@ -1314,7 +1314,7 @@ async def test_update_sql_transformation_wrong_component_type( ) -> None: """ update_sql_transformation should raise ToolError with actionable guidance when the - configuration belongs to a Python/R transformation (Storage returns 404 for the SQL + configuration belongs to a Python/R/DuckDB transformation (Storage returns 404 for the SQL component + config-ID combination). """ context = mcp_context_components_configs @@ -1342,6 +1342,8 @@ async def test_update_sql_transformation_wrong_component_type( assert 'keboola.snowflake-transformation' in error_msg assert 'update_config' in error_msg assert 'keboola.python-transformation-v2' in error_msg + assert 'keboola.r-transformation-v2' in error_msg + assert 'keboola.duckdb-transformation' in error_msg @pytest.mark.asyncio @@ -1655,14 +1657,23 @@ async def test_update_config( @pytest.mark.parametrize( - ('folder', 'cfg_count', 'cfg_folders', 'expect_folder_metadata', 'expect_folder_delete', 'expect_hint'), + ( + 'component_id', + 'folder', + 'cfg_count', + 'cfg_folders', + 'expect_folder_metadata', + 'expect_folder_delete', + 'expect_hint', + ), [ - ('Analytics', 0, [], True, False, False), - (' Analytics ', 0, [], True, False, False), - (None, 5, [], False, False, False), - (None, 25, ['Analytics'], False, False, True), - (None, 25, [], False, False, True), - ('', 5, [], False, True, False), + ('keboola.python-transformation-v2', 'Analytics', 0, [], True, False, False), + ('keboola.python-transformation-v2', ' Analytics ', 0, [], True, False, False), + ('keboola.python-transformation-v2', None, 5, [], False, False, False), + ('keboola.python-transformation-v2', None, 25, ['Analytics'], False, False, True), + ('keboola.python-transformation-v2', None, 25, [], False, False, True), + ('keboola.python-transformation-v2', '', 5, [], False, True, False), + ('keboola.duckdb-transformation', 'Analytics', 0, [], True, False, False), ], ids=[ 'folder_provided', @@ -1671,6 +1682,7 @@ async def test_update_config( 'no_folder_many_with_folders', 'no_folder_many_no_folders', 'folder_empty_deletes', + 'duckdb_folder_provided', ], ) @pytest.mark.asyncio @@ -1678,6 +1690,7 @@ async def test_update_config_folder( mocker: MockerFixture, mcp_context_components_configs: Context, mock_component: dict[str, Any], + component_id: str, folder: Any, cfg_count: int, cfg_folders: list[str], @@ -1688,7 +1701,6 @@ async def test_update_config_folder( """Test folder metadata is set/cleared and folder hint is returned by update_config.""" context = mcp_context_components_configs keboola_client = KeboolaClient.from_state(context.session.state) - component_id = 'keboola.python-transformation-v2' mock_component['id'] = component_id configuration_id = 'cfg-folder-test' existing = {