Skip to content

feat: add configuration JSON schema and sync it from the repo (CFTL-530) - #33

Open
matyas-jirat-keboola wants to merge 4 commits into
masterfrom
feat/CFTL-530-add-configuration-schema
Open

feat: add configuration JSON schema and sync it from the repo (CFTL-530)#33
matyas-jirat-keboola wants to merge 4 commits into
masterfrom
feat/CFTL-530-add-configuration-schema

Conversation

@matyas-jirat-keboola

@matyas-jirat-keboola matyas-jirat-keboola commented Aug 6, 2026

Copy link
Copy Markdown

Adds the configurationSchema for this component and makes the repository its source of truth.

Part of the KAI JSON-schema coverage effort (CFTL-530 / CFTL-505 / AI-2584).

Why

configurationSchema in the Developer Portal is {} today (app version 52). Two consequences:

  • The Keboola MCP server's validation.py skips validation entirely when a component has no
    schema, so configurations written by the in-platform AI assistant and by MCP clients reach
    Storage unchecked.
  • The configuration page is a raw JSON editor, so there is nothing guiding a human either.

This component has no custom UI route, so the schema also becomes the config form — see
User-visible change below.

What changed

File Purpose
component_config/configSchema.json The draft-07 schema, describing the contents of parameters only
scripts/developer_portal/update_properties.sh Pushes repo-owned portal properties on a semantic-tag deploy
.github/workflows/push.yml Runs that script from the existing deploy job
README.md Documents the script, and that editing the repo file is the durable way to change the live schema

Merging alone changes nothing in the portal — the sync runs on the next semantic-tag release.

Blast radius of the new sync step

The repo had no property sync at all, so this introduces one. It is deliberately narrow:

  • It pushes only configurationSchema. Descriptions, documentationUrl/sourceCodeUrl,
    actions, uiOptions, encryption, defaultBucket and everything else are untouched and stay
    portal-owned. Adding any of those later needs a repo-vs-portal diff first, because the repo file
    would start overwriting whatever is live.
  • It refuses to push an empty {} / [] document and fails the job instead. The usual
    [ -n "$value" ] guard passes for a 2-byte {} placeholder, which silently wipes a populated
    live property on the next release — that is a real failure mode elsewhere in the org, not a
    hypothetical.
  • It does not copy the Python-oriented fn_actions_md_update.sh from other repos. That script
    greps src/component.py, which does not exist here, and under set -e it would abort the deploy.

Behaviour verified locally with docker stubbed: happy path pushes, {} exits 1 before calling the
portal, a missing file is skipped without failing.

How the schema was derived

From src/Keboola/DynamoDbExtractor/Config/ConfigDefinition.php — the component's own Symfony
config tree — cross-read against Extractor.php, Exporter.php, both ReadingAdapters and
CONFIG.md. Notable points:

  • db.endpoint, db.accessKeyId, db.#secretAccessKey, db.regionName are all
    isRequired()->cannotBeEmpty()minLength: 1.
  • Per export, id, name, table, incremental and mapping are required; mapping is
    cannotBeEmpty()minProperties: 1.
  • id and limit are integer only — Symfony's integerNode rejects "1", so no string union.
  • The scan/query split is encoded as one if/then/else inside exports.items, mirroring
    QUERY_INVALID_NODES and SCAN_INVALID_NODES: dateFilter is rejected in query mode, and the
    four query-only nodes are rejected in scan mode.
  • dateFilter requires field + format + value together, matching
    Extractor::validateDateFilter.
  • #secretAccessKey is format: password; the examples use an obvious placeholder, never a
    realistic-looking ciphertext.
  • No default keys anywhere and no additionalProperties: false, so unknown/legacy keys in stored
    configurations keep validating and nothing gets silently written into existing configs.

Validation

  • jsonschema.Draft7Validator.check_schema — passes.
  • 17 real configurations from tests/functional — every one accepted (or rejected, for the two
    the component itself rejects) by both jsonschema and the MCP server's
    KeboolaParametersValidator, with 0 undeclared keys. Coverage includes scan, query, query
    with the $ placeholder syntax, secondary-index query, index scan, dateFilter, limit,
    child-table mapping and the native-types manifest case.
  • 18 negative tests — each rejected by both validators: query without keyConditionExpression;
    query carrying dateFilter; scan carrying each of the four query-only nodes; mode omitted while
    carrying a query-only node; dateFilter missing a part; missing incremental/mapping; empty
    mapping; missing #secretAccessKey; empty endpoint; mode outside the enum; limit as a
    string; no exports; empty exports.

User-visible change — please read before releasing

Adding a schema turns the configuration page from a raw JSON editor into a generated form. Driven
through the real Keboola RJSF editor, the form renders correctly:

  • Field order, tooltips and placeholders as authored; secret masked; id/limit typed as numbers.
  • mode, enabled and incremental render as selects with the intended titles (Scan/Query,
    Enabled/Disabled, Full Load/Incremental Load).
  • mapping, expressionAttributeNames and expressionAttributeValues (type: "object" +
    format: "editor") render as editable CodeMirror JSON editors and emit parsed objects.
  • Mode gating works both ways: index and dateFilter show only in Scan mode; indexName
    (Secondary Index), keyConditionExpression and the two expression-attribute editors show only in
    Query mode. A legacy scan config with no mode key still shows the scan-only fields — the editor
    fills the scan default on load.
  • Primary Key renders as a creatable tag picker (type each destination column and press Enter),
    not a stack of add-a-row text inputs. It still stores the same string[].
  • Field labels chosen for clarity in the form: nameOutput Table Name, indexIndex
    (Scan only), valueFrom Date.

Two things to know before a release:

  • The three code-editor fields render without a title label — a bare editor box. This is an
    editor-side limitation (the app's CodeEditorField bypasses the field template); the schema's
    title is correct and is what Kai/MCP read, the form just doesn't paint it. Not fixable from the
    schema.
  • Switching an export's mode after entering the other mode's fields leaves the now-hidden values in
    the config. The schema still rejects such a config (Node "…" is not allowed for … export.), and
    so does the component, but the editor does not surface that error prominently. Clearing hidden
    values on mode switch is editor behaviour, out of scope for this schema.

Sanity check before cutting a release: open a real configuration, press Ctrl+D,
paste this schema and confirm a populated mapping survives a save untouched.

Known limitations

  • Not cross-checked against production configurations. Validation used the 17 configurations in
    this repo, which cover every documented shape, but not customer data. The one place that would
    bite if a real config is shaped unexpectedly is keyConditionExpression being required in query
    mode — the component does not enforce it, only DynamoDB does, so a stored query export lacking it
    would start being rejected on edit. Every other required entry is enforced by the component.
  • db and exports are required at the root even though Symfony tolerates an absent db node;
    both are unconditionally needed for a configuration that can run.
  • The portal's longDescription still says the component "must be configured manually" and that you
    "must be familiar with JSON". That is portal-owned and not synced by this PR; worth updating
    separately once the form behaviour above is settled.

🤖 Generated with Claude Code

Adds the draft-07 `configurationSchema` for the `parameters` object and the CI
step that pushes it to the Developer Portal on a semantic-tag release, so the
repository is the source of truth for it.

The component previously had no schema (`configurationSchema` was `{}`), so the
Keboola MCP server skipped validation entirely and wrote agent-authored configs
to Storage unchecked, and the configuration page was a raw JSON editor.

The schema is derived from `Config/ConfigDefinition.php` and validated against
every runnable configuration in `tests/functional`, through both `jsonschema`
and the MCP server's own `KeboolaParametersValidator`.

`scripts/developer_portal/update_properties.sh` deliberately pushes only
`configurationSchema`; every other portal property stays portal-owned. It
refuses to push an empty `{}` document rather than silently clearing a live
property.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@linear-code

linear-code Bot commented Aug 6, 2026

Copy link
Copy Markdown

CFTL-530

matyas-jirat-keboola and others added 2 commits August 6, 2026 15:38
Root-level `examples` are not rendered by the configuration form, so they were
dead weight in a schema whose job is to generate that form. Configuration
examples are served to the AI assistant from the AI service's
`rootConfigurationExamples` / `rowConfigurationExamples`, not from the schema.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dateFilter is scan-only — the component rejects it on a query export
(QUERY_INVALID_NODES). The schema already invalidated that combination,
but the form still rendered the Date Filter fields in query mode. Gate it
with options.dependencies {"mode": "scan"}, mirroring the query-only fields.

Verified in the real RJSF editor: hidden in query, shown in scan, and still
shown for a legacy scan config with no explicit mode key (the editor fills
the scan default on load).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@keboola-pr-reviewer-bot keboola-pr-reviewer-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verdict: needs_human (risk 3/5) · profile component-factory

New configurationSchema imposes required constraints unverified against production configs, so a human should confirm no live config breaks.

Impact flags: possible rollback re-introduction — see Check Run summary.

Concerns:

  • component_config/configSchema.json: Requires keyConditionExpression in query mode; component doesn't enforce it, unverified vs production configs
  • component_config/configSchema.json: New required sets on a live customer-facing component could reject existing configs on edit

Suggested reviewers: @keboola/component-factory

Reverse-engineering follow-up from reviewing the live form:

- index: gate to scan mode (options.dependencies mode:scan) and retitle
  "Index". It only affects Scan reads; it was showing in Query mode next to
  Secondary Index, which was confusing. Query mode now shows only Secondary
  Index (indexName).
- primaryKey: render as a creatable tag picker (uniqueItems + options.tags)
  instead of an add-a-row array of text inputs. Emits the same string[].
- name: retitle "Output Table Name" with a clearer description — it names the
  output Storage table.
- value/keyConditionExpression: clearer titles/descriptions ("From Date", key
  condition wording).

Validation unchanged: every functional-test config still validates through
jsonschema and the MCP KeboolaParametersValidator; the 6 primaryKey fixtures
have no duplicate columns so uniqueItems rejects nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants