Skip to content

PRDCT-554: Variables under Components — one home, four override treatments collapsed - #1051

Open
Iamfle4ka wants to merge 10 commits into
mainfrom
PRDCT-554-variables-api-recut
Open

PRDCT-554: Variables under Components — one home, four override treatments collapsed#1051
Iamfle4ka wants to merge 10 commits into
mainfrom
PRDCT-554-variables-api-recut

Conversation

@Iamfle4ka

@Iamfle4ka Iamfle4ka commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

What & why

Re-cut of phase-2 batch 4 (was #1033, stacked on the big-bang #1027) as a standalone per-unit PR off main, per the 2026-07-27 pivot. Same content/placement as #1033, but with no dependency on #1027's dots-UI / temp "Developer Docs" nav group — so it can be reviewed and merged on its own.

Moves the 2 Variables API pages under the canonical Variables page:

  • /transformations/variables/api/ + /transformations/variables/api/tutorial/
  • Old /integrate/variables/* URLs keep working via redirect_from.
  • Nav: Variables becomes a branch with a "Variables API" child.

Verification

  • npm run build clean — 258 pages.
  • Live-verified in demo project 264 (GCP EU): a keboola.variables config + {{var}} placeholder resolves at runtime and the value lands in the output data — job 94591978, status: success (default-values path). The inline variableValuesData override path is verified against keboola-sdk-go source.
  • node scripts/audit-phase2.mjs: 0 missing images. Broken internal links are all forward-refs from these pages to sibling batches not yet on main (/storage/api, /components/extractors/generic-extractor, /integrate/jobs, /extend/*) — they heal as those units land; acceptable during the incremental merge.

Supersedes

Replaces the stacked #1033 (which will be closed together with the #1027 big-bang stack once the per-unit queue is drained).

🤖 Generated with Claude Code

Placement / dedup audit (2026-08-03)

Ran the placement/dedup pass. Unlike Generic Extractor, this is not a clean stub-replace —
the nav already disambiguates (Variables & Shared CodeVariables APITutorial), so
there's no twin-entry problem. What's left is genuinely owner/SME decisions, so I applied
only one safe fix and am surfacing the rest rather than pre-deciding:

Applied: retitled the API page's frontmatter VariablesVariables API (it collided with
the existing Variables page's title; nav already calls it "Variables API").

Open decisions (for Jordan / Matias — not touched):

  1. Flow-variable overrides are documented in 3 places, two of them stale:
    • flows/index.md## Variables — the current, canonical treatment (conditional-flows UI + JSON).
    • transformations/variables/index.md### Flow Usageoutdated (old orchestration screenshots).
    • this PR's …/api/## Orchestrator Integrationoutdated term ("orchestration" ×22, pre-conditional-flows).
      → Recommendation: make flows/index.md canonical; trim the Variables "Flow Usage" section to a
      link, and reframe the API page's "Orchestrator Integration" to "Flows" terminology + link. Touches
      the Flows section, so worth a joint call.
  2. Shared Code is covered twice — UI walkthrough in transformations/variables/index.md and API in
    this page. Complementary, no dedicated Shared Code home (the nav group is "Variables & Shared Code").
    → Recommendation: cross-link rather than merge; low urgency.
  3. Does Variables belong under Transformations at all? It's cross-cutting (components + flows +
    transformations). Jordan raised this on 07-30 → Matias's call. Left as-is pending that decision.

@linear-code

linear-code Bot commented Jul 28, 2026

Copy link
Copy Markdown

PRDCT-554

@vercel

vercel Bot commented Jul 28, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
connection-docs Ready Ready Preview Aug 7, 2026 11:59pm

Request Review

@Iamfle4ka Iamfle4ka added the dev-docs-migration developers.keboola.com → help.keboola.com migration label Jul 28, 2026
Iamfle4ka pushed a commit that referenced this pull request Aug 3, 2026
The map still routed the 42 dev /extend/** pages to /components/development/**,
a section that was never built: Jordan approved /extend/ as its own top-level
section instead (#1046, merged 2026-07-29, commit 8504c9d). Every one of those
42 targets 404'd, so deleting the dev pages would have orphaned their URLs.

#1046 preserved the dev path structure 1:1, so these become identity redirects
across domains (developers.keboola.com/extend/X/ -> help.keboola.com/extend/X/).
Verified: all 42 targets now resolve in a production build; conservation failures
drop from 162 to 117, and the remaining 117 are units whose help pages are not on
main yet (#1023, #1051-#1054, artifacts, jobs, overview/api, encryption, CLI).

The NEW-sections table entry is marked superseded rather than deleted so the
provenance of the change stays readable.

Left untouched: the 3 rows still targeting /components/development/ (artifacts x2,
overview/encryption). They are flagged `unsure` and are genuine open owner calls.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Iamfle4ka

Copy link
Copy Markdown
Collaborator Author

Owner decisions from the placement audit — resolved

Asked Matyáš. Two of the three open items are settled.

Decision 3 — does Variables belong under Transformations? → no, Components

Variables are a base component (keboola.variables), with no constraint on location. Target section is Components.

Not executed here, deliberately. The migrated …/api/ pages are children of the native Variables & Shared Code page, so the whole branch has to move together — that is a SECTION-MAP call, not something a migration PR should do. Recorded in SECTION-MAP.md as three MOVES rows (/transformations/variables/ + the two API children, conf: owner (Matyáš, 2026-08-04)), with the section counts adjusted: Transformations 22 → 19, Components 223 → 226. The owner call settled the section; the exact target slug is our proposal.

Decision 1 — flow-variable overrides in three places → first half applied

/flows/ is canonical; our copy here was stale (pre-conditional-flows orchestration screenshots). Pushed in 6df28b9c: the ### Flow Usage section in transformations/variables/index.md is trimmed to a cross-link.

Worth flagging what the fact-check caught, because my first attempt was wrong: I originally linked /flows/#variables, which is the Set Variable UI — a different mechanism from the one the sentence described. Corrected to name both real routes with the anchors that actually document them:

Conservation: every removed paragraph, screenshot and JSON block survives in flows/index.md, flows/flows-legacy/index.md, or this PR's own …/api/ page. The one sample with no home elsewhere was the {"config": "6939", "variableValuesData": {…}} snippet — and it was wrong: it omits componentId, which the job API requires (job-queue-internal-api-php-clientNewJobDefinition.php marks it isRequired()->cannotBeEmpty()). Not restored.

Also dropped orchestration-parameters.png and task-parameters.png (both copies each) — referenced only by the removed section. /flows/task-parameters.png is a different file (different checksum) and is untouched.

Still open

  • Decision 1, second half — reframing this PR's ## Orchestrator Integration into Flows terminology is not in this commit. Of the 22 orchestrat* occurrences on that page, some are the literal component id keboola.orchestrator and real API calls, which must stay; it needs a legacy-term-vs-valid-id pass, not a rename. Filed separately.
  • Decision 2 (Shared Code covered twice) — unchanged, still cross-link-not-merge, low urgency.

Verification

npm run build clean (258 pages). audit-phase2: 0 missing images, both new anchors resolve, no new findings. The 17 broken internal links are the same forward-refs to sibling batches documented in the PR body.

One finding for the Flows owner, not for this PR

flows/index.md:43-52 (Control Task Execution) reuses the legacy Task Parameters screenshot and the “Set advanced parameters” label verbatim from flows-legacy, and its bullet list was only partly updated for Conditional Flows. Since this page now cross-links that mechanism, the label and screenshot are worth confirming against the current builder.

Nikita and others added 5 commits August 5, 2026 01:42
Re-cut of phase-2 batch 4 (was #1033, stacked on the big-bang #1027) as a
standalone per-unit PR off main, per the 2026-07-27 pivot — no dependency on
#1027's dots-UI / temp nav group.

Moves the 2 Variables API pages under the canonical Variables page at
/transformations/variables/api/ (+ /tutorial/). Old /integrate/variables/*
URLs keep working via redirect_from. Nav: Variables becomes a branch with a
"Variables API" child.

Build clean (258 pp); audit 0 missing images. Broken internal links are all
forward-refs from these pages to sibling batches not yet on main (/storage/api,
/components/extractors/generic-extractor, /integrate/jobs, /extend/*) — they
heal as those units land; acceptable during the incremental merge.

Supersedes the stacked #1033 (closed with the #1027 stack once the queue drains).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Fact-checked against the Queue API + config-resolver source (keboola-sdk-go,
job-queue-*-client, configuration-variables-resolver). Migration was drift-free;
fixes are pre-existing dev-source defects:

- CRITICAL: the runtime job parameter was misspelled `variablesValuesId` (extra
  "s") in the prose (×5) — the real Queue API field is `variableValuesId`
  (the code examples already used the correct form). A reader copying the prose
  spelling into a job body gets it silently ignored.
- MAJOR: two invalid JSON examples (missing comma after the parameters object;
  unterminated `"config"` string) → fixed so both parse.
- Removed the stale "preview feature" banner — Variables are GA (the canonical
  /transformations/variables/ page carries no such caveat; keboola.variables is
  first-class in config resolution).
- Softened the absolute "type is always string" claim — it contradicted the
  tutorial (which uses type:int); allowed types flagged for owner-verify.

Build clean (258 pp); audit 0 missing images (the 4 assets are present in
public/). Remaining broken links are forward-refs to sibling not-yet-merged units.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Followability pass (guide-tester). Two flags run down first: the component id
`ex-generic-v2` was flagged as possibly-wrong but is VERIFIED correct against the
live component index (keboola.ex-generic-v2 is absent) — left unchanged; and the
variable `type` question was resolved against configuration-variables-resolver
(free-form scalar, no enum; value substituted as text).

Fixes:
- Tutorial "Check the job result" was screenshot-only → added the job API: the
  run-job returns id + status; poll GET $JOB_QUEUE_HOST/jobs/{jobId} (status
  processing → success / error). Screenshots kept as a supplement.
- index: clarified `type` is a free-form descriptive label (string/int/…), used by
  the UI; the resolver substitutes as text — so the tutorial's `type: int` is valid
  and index/tutorial no longer appear to disagree.
- Fixed two broken in-page anchors (verified against the generated heading IDs):
  step-1 …variables→variable, step-2 …variable→variables.
- Unified the job-queue API-reference link format with the index.
- Tutorial: dropped a trailing-dash typo; noted the Storage token needs
  config-management rights (a read-only token fails at create-config).

Build clean (258 pp); audit 0 missing images; the fixed anchors resolve.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…les page

Both the existing transformations/variables page and this new API reference were
titled "Variables" (near-sibling slugs). Align the API page's frontmatter title
with its nav label ("Variables API") and its actual content. Placement/dedup of the
overlapping Flow-usage and Shared-Code content is a decision for the owner/SME —
surfaced in the PR body, not pre-decided here.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Closes the first half of the placement/dedup audit's open decision 1,
per Matyáš: /flows/ is canonical for flow-driven variable values and our
copy here was stale (pre-conditional-flows orchestration screenshots).

The removed prose, screenshots and JSON are all covered by flows/index.md
and flows/flows-legacy/index.md. The one sample with no home elsewhere —
{"config": "6939", "variableValuesData": {…}} — was wrong anyway: it omits
componentId, which the job API requires (job-queue-internal-api-php-client
NewJobDefinition.php marks it isRequired()->cannotBeEmpty()). Not restored.

Both flow routes are named, with the anchors that actually document them:
name-matching merge → #how-variables-reach-component-jobs, per-task
override → #control-task-execution. An earlier draft pointed at
/flows/#variables, which is the Set-Variable UI — a different mechanism.

Also drops orchestration-parameters.png and task-parameters.png (both
copies each) — referenced only by the removed section. /flows/task-parameters.png
is a different file and is untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…ed Code split

Jordan's 07-30 note was that the nav reads as two copies of one feature. The cause
was that nothing linked the two pages in either direction — the new API page's only
internal links were its own self-anchors.

- Both pages now open with a one-line pointer at the other (UI <-> API).
- The API page's `## Shared Code` becomes `## Shared Code via the API` and states
  what it covers; the UI chapter gains the reverse pointer. The two chapters cover
  different surfaces (UI workflow vs. the create-configuration call against
  `keboola.shared-code`), so neither is redundant — they just read that way unlinked.
- The native `#shared-code` heading is deliberately unchanged: two links in
  transformations/index.md target that anchor. Only the API page's heading moved,
  and nothing links to its old anchor.

Verified: build clean (308 pages), both `#shared-code` and `#shared-code-via-the-api`
present in the built HTML, broken-link count unchanged (54, same 8 cross-unit
danglers as before — no new ones).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Nikita and others added 2 commits August 5, 2026 18:02
…verride treatments

Owner decision 2026-08-05 (Jordan's option on the 07-30 call): Variables lives under
Flows. `/transformations/variables/` becomes `/flows/variables/`, with the API pages as
its children; the old URLs keep working via `redirect_from`.

The override of a configuration variable from a flow was documented in FOUR places:
`transformations/variables` "Flow Usage", `flows/index.md` "## Variables",
`flows/index.md` "Control Task Execution" (the only one with current UI), and the API
page's "Orchestrator Integration". They collapse into one section on the new page.

Corrected while merging them, against the live flow schemas:

- Conditional flows are `keboola.flow`; `keboola.orchestrator` is legacy. The API page's
  "Orchestrator Integration" documented the legacy component as if it were current. It is
  now "Driving Variables from a Flow", split into Conditional Flows (variable tasks,
  merge-by-name, `variableOverrides` tri-state) and Legacy Flows (the existing task
  payload, labelled legacy with a migration-guide link).
- The evaluation-sequence rules and the surrounding prose said "orchestration" throughout.

Shared Code is a different feature — it substitutes code, not values, and only for
transformations — so it moves out to `/transformations/shared-code/`, UI plus the API
section lifted off the Variables API page. Its 17 images move with it.

Also: the Jekyll-era escape `{{ "{{ multiplier " }}}}` rendered literally on the page;
`flows/index.md` drops from 391 to 227 lines; `workspace/sql-editor` linked variables
through an absolute help.keboola.com URL.

Verified: build clean, 309 pages. audit-phase2 broken links 54 → 54, missing images 0,
one old-docs smell fixed (156 → 155 total). #1073's anchor checker reports no new broken
anchors from these pages — the two on the API page are the `config-file` headings #1069
repairs. All five old URLs redirect, no duplicate heading ids.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two of these are defects my own rewrite introduced:

- `actionParameters` was bound to `keboola.orchestrator` by the new "Legacy Flows"
  heading. That property belongs to the *older* `orchestrator` component
  (syrup.keboola.com/orchestrator); `keboola.orchestrator` stores tasks as
  `task: {componentId, configId, mode, …}`, which the page's own Step 5 example shows.
  Claim dropped rather than restated.
- Renaming "orchestration" → "flow" in the evaluation-sequence rules silently asserted
  legacy-orchestrator precedence for `keboola.flow`, where neither task-level
  `variableValuesId` nor run-level overrides are declared. Those bullets are now scoped
  to legacy flows explicitly.
- The exception list read as though a `keboola.flow` configuration may carry placeholders
  while the next sentence said it may not; it now names both flow components.

Pre-existing, but on pages this PR makes canonical:

- "If you omit *thirdVar*, it will be replaced by an empty string" is wrong.
  `configuration-variables-resolver` collects every unresolved placeholder and throws
  `Missing values for placeholders: …`. It also contradicted the new UI page, which
  states the error correctly. Replaced with both real error strings.
- The static flow-variable example used `"value": 3600`; the live `keboola.flow` schema
  declares `value` as a string, so an unquoted number fails validation. Same for the
  `variableValuesData` example and the tutorial's `"value": 1`.
- **Tutorial blocker:** the run-job body inlined `'$VARIABLES_VALUES'` unquoted, so bash
  word-split a multi-line value — curl got a JSON fragment truncated at
  `"values":` plus 12 stray argv words, and only zsh happened to work. Now a heredoc on
  stdin: verified ARGC=10 and a byte-identical, `json.load`-valid body in both shells.
- The tutorial said default values live in "additional configurations"; they are
  configuration rows of the same variable configuration, referenced by
  `variables_values_id`.

Also: the legacy task-payload claim is inherited from developers.keboola.com and is
absent from every public orchestrator schema, so it now carries a caution saying so
rather than reading as current behaviour. And `PageTitle.astro` still keyed its emoji off
`transformations/variables`, so the icon vanished with the move.

The two remaining broken anchors on the API page (`config-file/#input-mapping--basic`)
resolve once #1069 lands — it converts those `---` headings to em dashes, which slug to
exactly the two-dash form linked here.

Verified: build clean, 309 pages, audit unchanged at 54 broken links / 0 missing images /
155 total, no new broken anchors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Iamfle4ka Iamfle4ka changed the title PRDCT-554: Variables API under Transformations (re-cut off main) PRDCT-554: Variables under Flows — one home, four override treatments collapsed Aug 5, 2026
@Iamfle4ka

Copy link
Copy Markdown
Collaborator Author

Placement decided: Flows (owner, 2026-08-05)

Jordan's option from the 07-30 call. /transformations/variables//flows/variables/, with the API pages as its children; all five old URLs keep working via redirect_from (verified in dist).

The loop this closes

Overriding a configuration variable from a flow was documented in four places, two of them stale:

Where State
transformations/variables › Flow Usage stale (2019 Orchestrations screenshots)
flows/index.md## Variables current
flows/index.md › Control Task Execution the only one with current UI
variables/api › Orchestrator Integration documented the legacy component as current

They collapse into one section, Driving a Configuration Variable from a Flow.

Corrected against the live flow schemas

  • Conditional flows are keboola.flow; keboola.orchestrator is legacy. The API section now splits accordingly: conditional (variable tasks, merge-by-name, variableOverrides tri-state) vs legacy (the inherited task payload, labelled and cautioned).
  • "value": 3600"3600" — the schema declares variable values as strings, so unquoted numbers fail validation.
  • "If you omit thirdVar, it will be replaced by an empty string" is wrong; configuration-variables-resolver throws Missing values for placeholders: ….
  • Tutorial blocker: the run-job body inlined '$VARIABLES_VALUES' unquoted, so bash word-split it — curl received JSON truncated at "values": plus 12 stray argv words. It only ever worked in zsh. Now a heredoc on stdin, verified byte-identical and json.load-valid in both shells.
  • actionParameters was dropped: it belongs to the pre-keboola.orchestrator component, not to the one the heading now names.

Shared Code moved out

It substitutes code, not values, and only for transformations → /transformations/shared-code/ (UI + the API section lifted off the Variables API page), 17 images with it.

Verification

Left for the owner

  • VERIFY: whether a legacy keboola.orchestrator task honours variableValuesId / variableValuesData, and whether run-level values override per-task ones. Inherited from developers.keboola.com, absent from every public orchestrator schema — now carries an inline caution instead of reading as current behaviour.
  • VERIFY: that Task Parameters on a conditional-flow task accept variableValuesData and the runner forwards them; the keboola.flow schema declares neither.
  • The Configuration Variables section has no numbered click path and its screenshot is 2021-era — screenshots are PRDCT-594: reshoot the flow screenshots that showed retired UI #1078's scope, deliberately not touched here.
  • Tutorial polish not done: the token step doesn't name the master token, and "Check the job result" has no text success criteria (both screenshot-only today).

…-api-recut

# Conflicts:
#	public/transformations/variables/orchestration-parameters.png
#	public/transformations/variables/task-parameters.png
#	src/content/docs/flows/index.md
#	src/content/docs/transformations/variables/index.md
#	src/content/docs/transformations/variables/orchestration-parameters.png
#	src/content/docs/transformations/variables/task-parameters.png
… name

Jordan ruled Components on the 08-05 sync; Matyáš confirmed on 08-07
("Call it Variables and I don't mind wherever it is. Components as a page is
fine."), which also settles the second question — there is no separate
"Variables API" product, the calls are configuration/Storage API plus Vault.

- /flows/variables/** -> /components/variables/** (3 pages, 6 assets)
- "Variables API" -> "Variables Through the API", parallel to the UI page
- nav: the group moves from Conditional Flows to Components, next to Running
  Jobs in Parallel rather than buried under the connector catalogues
- 43 references repointed; PageTitle emoji key follows the slug

No new redirects: /flows/variables/ only ever existed on this branch, so the
five live URLs (/transformations/variables/*, /integrate/variables/*) still
point at the canonical page — verified in dist.
@Iamfle4ka Iamfle4ka changed the title PRDCT-554: Variables under Flows — one home, four override treatments collapsed PRDCT-554: Variables under Components — one home, four override treatments collapsed Aug 7, 2026
@Iamfle4ka

Copy link
Copy Markdown
Collaborator Author

Moved to Components (08-08)

Placement is settled, so the move is folded in here rather than left to a follow-up PR.

  • Jordan, 08-05 sync: Components is the correct home — "flows are technically a component, as are transformations".
  • Matyáš, 08-07 in #ask-component-factory: "Call it Variables and I don't mind wherever it is. Components as a page is fine." That also answers the second question I asked him — there is no separate "Variables API" product; the calls are configuration/Storage API plus Vault.

What changed

before after
pages /flows/variables/{,api/,api/tutorial/} /components/variables/{,api/,api/tutorial/}
child page title Variables API Variables Through the API
nav under Conditional Flows under Components, next to Running Jobs in Parallel

No new redirects were needed. /flows/variables/ only ever existed on this branch, so folding the move in now costs nothing; landing first and moving later would have published a URL we would retire days later. The five live URLs still resolve — verified in dist:

/transformations/variables/              OK
/transformations/variables/api/          OK
/transformations/variables/api/tutorial/ OK
/integrate/variables/                    OK
/integrate/variables/tutorial/           OK

43 references repointed across 12 files; the PageTitle.astro emoji key follows the slug. Build clean at 311 pages, audit-phase2 reports 0 missing images.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dev-docs-migration developers.keboola.com → help.keboola.com migration

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant