Skip to content

feat(AI-3112): support keboola.duckdb-transformation via generic config tools - #652

Open
Matovidlo wants to merge 1 commit into
mainfrom
martinvasko-ai-3112-kai-supporting-creation-of-duckdb-transformations
Open

feat(AI-3112): support keboola.duckdb-transformation via generic config tools#652
Matovidlo wants to merge 1 commit into
mainfrom
martinvasko-ai-3112-kai-supporting-creation-of-duckdb-transformations

Conversation

@Matovidlo

Copy link
Copy Markdown
Contributor

Description

Linear: AI-3112

Change Type

  • Major (breaking changes, significant new features)
  • Minor (new features, enhancements, backward compatible)
  • Patch (bug fixes, small improvements, no new features)

Summary

Adds keboola.duckdb-transformation as a supported transformation type, routed through the
existing generic create_config/update_config tools — the same path Python/R transformations
already use. create_sql_transformation/update_sql_transformation stay Snowflake/BigQuery-only,
since their component ID is derived from the project's workspace SQL dialect and DuckDB has no
cloud workspace backend.

Changes:

  • utils.py: new DUCKDB_TRANSFORMATION_ID constant, added to FOLDER_SUPPORTING_COMPONENT_IDS
    so folder metadata management works the same as for Python/R.
  • tools.py: docstrings for create_config, create_sql_transformation, and
    update_sql_transformation, plus the update_sql_transformation 404 error message, updated to
    mention DuckDB alongside Python/R so the model routes DuckDB requests correctly.
  • No new tools, no workspace changes. DuckDB-specific sync actions (syntax_check,
    lineage_visualization, execution_plan_visualization, expected_input_tables) and config
    parameters (backend_size, timeout, duckdb_version, etc.) require no new code — they're
    already handled generically by run_sync_action and the opaque-JSON parameters field.

RFC: feature_spec/duckdb_transformation_support/RFC.md

Follow-up (separate Linear issue, own RFC, not in this PR): AI-3662
consolidating the per-backend transformation tools and moving shared guidance into
project_system_prompt.md.

Testing

  • Tested with Cursor AI desktop (Streamable-HTTP transports)

Optional testing

  • Tested with Cursor AI desktop (all transports)
  • Tested with claude.ai web and canary-orion MCP (Streamable-HTTP)
  • Tested with In Platform Agent on canary-orion
  • Tested with RO chat on canary-orion

Checklist

  • Self-review completed
  • Unit tests added/updated (if applicable)
  • Integration tests added/updated (if applicable)
  • Project version bumped according to the change type
  • Documentation updated (if applicable)

@linear-code

linear-code Bot commented Aug 3, 2026

Copy link
Copy Markdown

AI-3112

@Matovidlo

Copy link
Copy Markdown
Contributor Author

@claude review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds DuckDB transformation support (keboola.duckdb-transformation) by routing it through the existing generic configuration tools, aligning it with the existing Python/R transformation flow (and keeping the SQL-transformation tools Snowflake/BigQuery-only).

Changes:

  • Added DUCKDB_TRANSFORMATION_ID and enabled folder metadata handling for DuckDB transformations via update_config.
  • Updated tool docstrings and the update_sql_transformation 404 guidance to explicitly point DuckDB users to create_config/update_config.
  • Extended unit tests to cover DuckDB folder behavior and updated error-message expectations; bumped project version to 1.75.0 (plus regenerated docs/lock updates).

Reviewed changes

Copilot reviewed 6 out of 7 changed files in this pull request and generated no comments.

Show a summary per file
File Description
src/keboola_mcp_server/tools/components/utils.py Adds DuckDB transformation component ID constant and includes it in folder-support set.
src/keboola_mcp_server/tools/components/tools.py Updates docstrings and improves 404 guidance to steer DuckDB to generic config tools.
tests/tools/components/test_tools.py Extends existing parametrized tests to cover DuckDB folder handling and error message content.
TOOLS.md Regenerated tool documentation reflecting updated docstrings/guidance.
feature_spec/duckdb_transformation_support/RFC.md Adds an RFC documenting the motivation, scope, and verification plan for DuckDB support.
pyproject.toml Version bump to 1.75.0.
uv.lock Lockfile updated to reflect the version bump.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@Matovidlo
Matovidlo force-pushed the martinvasko-ai-3112-kai-supporting-creation-of-duckdb-transformations branch from cac143a to 5b18e80 Compare August 3, 2026 12:13
@Matovidlo

Copy link
Copy Markdown
Contributor Author

Rebased onto the AI-3664 fix branch (martinvasko-ai-3664-migrate-flow-integration-tests-off-legacy-orchestrator) as an interim step to unblock Integration Tests, which were failing due to the now-disabled legacy Orchestrator flow API (see #653). This PR's diff will temporarily include #653's commits until that PR merges to main, at which point I'll rebase this branch onto main to drop the duplication.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Suppressed comments (2)

integtests/tools/flow/test_tools.py:833

  • The cleanup block swallows deletion exceptions but logs without traceback, which makes integration-test failures much harder to debug. Use structured logging with exc_info=True (or LOG.exception) when catching and continuing.
            except Exception as e:
                LOG.warning(f'Failed to delete {flow_type} flow {flow_id}: {e}')

integtests/tools/flow/test_tools.py:424

  • get_flows() lists all flows in the project (it does not filter to only those created by these fixtures), so asserting exact equality of IDs will fail in projects that already have flows. This test should assert that the two fixture-created flow IDs are present (subset), not that they are the only results.
    assert {f.configuration_id for f in flows.flows} == {
        initial_cf.configuration_id,
        initial_cf2.configuration_id,
    }

Add keboola.duckdb-transformation alongside Python/R as a generic-config-managed
transformation type (create_sql_transformation/update_sql_transformation remain
Snowflake/BigQuery-only, since their component ID is derived from the workspace
SQL dialect and DuckDB has no workspace backend).

- utils.py: add DUCKDB_TRANSFORMATION_ID, include it in FOLDER_SUPPORTING_COMPONENT_IDS
  so folder metadata works the same as Python/R.
- tools.py: update create_config/create_sql_transformation/update_sql_transformation
  docstrings and the update_sql_transformation 404 error message to mention DuckDB.
- Extend existing parametrized folder and 404-error tests with DuckDB cases.

RFC: feature_spec/duckdb_transformation_support/RFC.md
@Matovidlo
Matovidlo marked this pull request as ready for review August 10, 2026 09:13
@Matovidlo
Matovidlo requested a review from a team as a code owner August 10, 2026 09:13
@Matovidlo
Matovidlo requested review from tomasfejfar and removed request for a team August 10, 2026 09:13

@keboola-pr-reviewer-bot keboola-pr-reviewer-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: auto_approve (risk 2/5) · profile keboola-mcp-server

Additive DuckDB component-ID support via existing generic config tools, with linked RFC, version bump, and extended tests — safe to auto-approve.

Impact flags: scope creep — see Check Run summary.

Concerns:

  • integtests/tools/flow/test_tools.py: Unrelated flow-test refactor bundled into a DuckDB PR; drops legacy Orchestrator coverage.

@Matovidlo
Matovidlo force-pushed the martinvasko-ai-3112-kai-supporting-creation-of-duckdb-transformations branch from 5b18e80 to 4fab217 Compare August 10, 2026 09:20
@keboola-pr-reviewer-bot
keboola-pr-reviewer-bot dismissed their stale review August 10, 2026 09:20

Dismissing prior approval — a new commit was pushed and this review was for an earlier SHA. Run @keboola-pr-reviewer-bot review to get a fresh verdict.

@keboola-pr-reviewer-bot

Copy link
Copy Markdown

New commit on 4fab217 — dismissed 1 stale bot approval. Comment @keboola-pr-reviewer-bot review when you want a fresh review.

@tomasfejfar

Copy link
Copy Markdown
Contributor

👀

@tomasfejfar tomasfejfar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't do the same for update.

Also, while it works, it's not great. DuckDB transformation is essentially a SQL transformation and should be treated as such. Limitations of our tool shouldn't dictate that it goes to code-like transformations. But I wouldn't block on that alone. The missing update is the blocker.

@tomasfejfar

Copy link
Copy Markdown
Contributor

Also:

Friction summary: get_components and get_config_examples — the two tools the system prompt explicitly directs you to consult before calling create_config — return nothing usable for keboola.duckdb-transformation. The only way to actually learn the config shape was docs_query, and even that needed a second, more targeted question after the first answer resolved to the wrong (but similarly-named) component. A user following the documented discovery steps literally would hit two dead ends before finding a working path.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants