feat: add configuration JSON schema and sync it from the repo (CFTL-530) - #33
Open
matyas-jirat-keboola wants to merge 4 commits into
Open
feat: add configuration JSON schema and sync it from the repo (CFTL-530)#33matyas-jirat-keboola wants to merge 4 commits into
matyas-jirat-keboola wants to merge 4 commits into
Conversation
Adds the draft-07 `configurationSchema` for the `parameters` object and the CI
step that pushes it to the Developer Portal on a semantic-tag release, so the
repository is the source of truth for it.
The component previously had no schema (`configurationSchema` was `{}`), so the
Keboola MCP server skipped validation entirely and wrote agent-authored configs
to Storage unchecked, and the configuration page was a raw JSON editor.
The schema is derived from `Config/ConfigDefinition.php` and validated against
every runnable configuration in `tests/functional`, through both `jsonschema`
and the MCP server's own `KeboolaParametersValidator`.
`scripts/developer_portal/update_properties.sh` deliberately pushes only
`configurationSchema`; every other portal property stays portal-owned. It
refuses to push an empty `{}` document rather than silently clearing a live
property.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Root-level `examples` are not rendered by the configuration form, so they were dead weight in a schema whose job is to generate that form. Configuration examples are served to the AI assistant from the AI service's `rootConfigurationExamples` / `rowConfigurationExamples`, not from the schema. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
dateFilter is scan-only — the component rejects it on a query export
(QUERY_INVALID_NODES). The schema already invalidated that combination,
but the form still rendered the Date Filter fields in query mode. Gate it
with options.dependencies {"mode": "scan"}, mirroring the query-only fields.
Verified in the real RJSF editor: hidden in query, shown in scan, and still
shown for a legacy scan config with no explicit mode key (the editor fills
the scan default on load).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
keboola-pr-reviewer-bot
left a comment
There was a problem hiding this comment.
Verdict: needs_human (risk 3/5) · profile component-factory
New configurationSchema imposes required constraints unverified against production configs, so a human should confirm no live config breaks.
Impact flags: possible rollback re-introduction — see Check Run summary.
Concerns:
component_config/configSchema.json: Requires keyConditionExpression in query mode; component doesn't enforce it, unverified vs production configscomponent_config/configSchema.json: New required sets on a live customer-facing component could reject existing configs on edit
Suggested reviewers: @keboola/component-factory
Reverse-engineering follow-up from reviewing the live form:
- index: gate to scan mode (options.dependencies mode:scan) and retitle
"Index". It only affects Scan reads; it was showing in Query mode next to
Secondary Index, which was confusing. Query mode now shows only Secondary
Index (indexName).
- primaryKey: render as a creatable tag picker (uniqueItems + options.tags)
instead of an add-a-row array of text inputs. Emits the same string[].
- name: retitle "Output Table Name" with a clearer description — it names the
output Storage table.
- value/keyConditionExpression: clearer titles/descriptions ("From Date", key
condition wording).
Validation unchanged: every functional-test config still validates through
jsonschema and the MCP KeboolaParametersValidator; the 6 primaryKey fixtures
have no duplicate columns so uniqueItems rejects nothing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the
configurationSchemafor this component and makes the repository its source of truth.Part of the KAI JSON-schema coverage effort (CFTL-530 / CFTL-505 / AI-2584).
Why
configurationSchemain the Developer Portal is{}today (app version 52). Two consequences:validation.pyskips validation entirely when a component has noschema, so configurations written by the in-platform AI assistant and by MCP clients reach
Storage unchecked.
This component has no custom UI route, so the schema also becomes the config form — see
User-visible change below.
What changed
component_config/configSchema.jsonparametersonlyscripts/developer_portal/update_properties.sh.github/workflows/push.ymldeployjobREADME.mdMerging alone changes nothing in the portal — the sync runs on the next semantic-tag release.
Blast radius of the new sync step
The repo had no property sync at all, so this introduces one. It is deliberately narrow:
configurationSchema. Descriptions,documentationUrl/sourceCodeUrl,actions,uiOptions,encryption,defaultBucketand everything else are untouched and stayportal-owned. Adding any of those later needs a repo-vs-portal diff first, because the repo file
would start overwriting whatever is live.
{}/[]document and fails the job instead. The usual[ -n "$value" ]guard passes for a 2-byte{}placeholder, which silently wipes a populatedlive property on the next release — that is a real failure mode elsewhere in the org, not a
hypothetical.
fn_actions_md_update.shfrom other repos. That scriptgreps
src/component.py, which does not exist here, and underset -eit would abort the deploy.Behaviour verified locally with
dockerstubbed: happy path pushes,{}exits 1 before calling theportal, a missing file is skipped without failing.
How the schema was derived
From
src/Keboola/DynamoDbExtractor/Config/ConfigDefinition.php— the component's own Symfonyconfig tree — cross-read against
Extractor.php,Exporter.php, bothReadingAdapters andCONFIG.md. Notable points:db.endpoint,db.accessKeyId,db.#secretAccessKey,db.regionNameare allisRequired()->cannotBeEmpty()→minLength: 1.id,name,table,incrementalandmappingare required;mappingiscannotBeEmpty()→minProperties: 1.idandlimitareintegeronly — Symfony'sintegerNoderejects"1", so no string union.if/then/elseinsideexports.items, mirroringQUERY_INVALID_NODESandSCAN_INVALID_NODES:dateFilteris rejected in query mode, and thefour query-only nodes are rejected in scan mode.
dateFilterrequiresfield+format+valuetogether, matchingExtractor::validateDateFilter.#secretAccessKeyisformat: password; the examples use an obvious placeholder, never arealistic-looking ciphertext.
defaultkeys anywhere and noadditionalProperties: false, so unknown/legacy keys in storedconfigurations keep validating and nothing gets silently written into existing configs.
Validation
jsonschema.Draft7Validator.check_schema— passes.tests/functional— every one accepted (or rejected, for the twothe component itself rejects) by both
jsonschemaand the MCP server'sKeboolaParametersValidator, with 0 undeclared keys. Coverage includes scan, query, querywith the
$placeholder syntax, secondary-index query, index scan,dateFilter,limit,child-table
mappingand the native-types manifest case.keyConditionExpression;query carrying
dateFilter; scan carrying each of the four query-only nodes;modeomitted whilecarrying a query-only node;
dateFiltermissing a part; missingincremental/mapping; emptymapping; missing#secretAccessKey; emptyendpoint;modeoutside the enum;limitas astring; no
exports; emptyexports.User-visible change — please read before releasing
Adding a schema turns the configuration page from a raw JSON editor into a generated form. Driven
through the real Keboola RJSF editor, the form renders correctly:
id/limittyped as numbers.mode,enabledandincrementalrender as selects with the intended titles (Scan/Query,Enabled/Disabled, Full Load/Incremental Load).
mapping,expressionAttributeNamesandexpressionAttributeValues(type: "object"+format: "editor") render as editable CodeMirror JSON editors and emit parsed objects.indexanddateFiltershow only in Scan mode;indexName(Secondary Index),
keyConditionExpressionand the two expression-attribute editors show only inQuery mode. A legacy scan config with no
modekey still shows the scan-only fields — the editorfills the
scandefault on load.not a stack of add-a-row text inputs. It still stores the same
string[].name→ Output Table Name,index→ Index(Scan only),
value→ From Date.Two things to know before a release:
editor-side limitation (the app's
CodeEditorFieldbypasses the field template); the schema'stitleis correct and is what Kai/MCP read, the form just doesn't paint it. Not fixable from theschema.
the config. The schema still rejects such a config (
Node "…" is not allowed for … export.), andso does the component, but the editor does not surface that error prominently. Clearing hidden
values on mode switch is editor behaviour, out of scope for this schema.
Sanity check before cutting a release: open a real configuration, press Ctrl+D,
paste this schema and confirm a populated
mappingsurvives a save untouched.Known limitations
this repo, which cover every documented shape, but not customer data. The one place that would
bite if a real config is shaped unexpectedly is
keyConditionExpressionbeing required in querymode — the component does not enforce it, only DynamoDB does, so a stored query export lacking it
would start being rejected on edit. Every other
requiredentry is enforced by the component.dbandexportsare required at the root even though Symfony tolerates an absentdbnode;both are unconditionally needed for a configuration that can run.
longDescriptionstill says the component "must be configured manually" and that you"must be familiar with JSON". That is portal-owned and not synced by this PR; worth updating
separately once the form behaviour above is settled.
🤖 Generated with Claude Code