token create: persistent upstream 500 on POST /v2/storage/tokens surfaces as a blind retry + generic error, no actionable guidance

[vojtechnovotny-heu]

Summary

kbagent token create against a project on the europe-west3.gcp (EU GCP) stack fails every time with a raw 500 from the Keboola Storage API. Investigation shows kbagent itself is behaving correctly (correct request, retries per its documented policy, faithfully surfaces the upstream error) — this looks like an upstream Storage API defect, not a kbagent bug. Filing here anyway because (a) kbagent is the only way I have to observe/reproduce it, and (b) there's a concrete CLI-side hardening gap exposed by it (see "Proposal").

Environment

• kbagent v0.84.2 (keboola-cli PyPI package)
• Stack: connection.europe-west3.gcp.keboola.com

Repro

kbagent -v token create -p <project> -d "<description>" --expires-in 60

Response: 500 {"error": "Application error."} from POST /v2/storage/tokens, retried 3x by kbagent's existing backoff policy (500 is in RETRYABLE_STATUS_CODES), all 3 attempts fail identically.

Ruled out as the cause, so this doesn't look like a request-shape bug on the client side:

• Reproduces on 2 different projects on the same stack, with 2 different acting tokens.
• Reproduces with no scope, and with --bucket-read on a real bucket.
• Reproduces with --expires-in 60, --expires-in 3600, and omitted entirely (never expires).
• Acting token confirmed to have canManageTokens — rules out the missing-permission theory (which per http_base.py's own _raise_api_error should surface as a clean 403 → ErrorCode.ACCESS_DENIED, not a 500).
• No incident reported for the europe-west3 stack on status.keboola.com / keboolastatus.com at the time of testing.

Proposal

Not asking for a fix to the upstream API from this repo — that's a Keboola Connection backend issue I'll raise with Keboola support separately. But two things are actionable in kbagent itself:

1. Non-idempotent retry risk: POST /v2/storage/tokens is a create/mint call, not idempotent, yet it's retried on 500/502/503/504 like any other request. If the API ever creates the token server-side and then fails to respond cleanly (unlike this case, where all 3 attempts failed identically with no sign of partial success), a blind retry could silently mint a duplicate live credential the caller never sees. Worth excluding mutating/non-idempotent endpoints from the generic retry policy, or gating the retry on an idempotency key.
2. Generic 500 message gives no next step: today this surfaces as API error 500 ... Application error. with no hint that a persistent 500 (survives all retries, stable across projects/tokens/payloads) is very likely an upstream incident rather than something the caller can fix. A short, specific hint in the ErrorCode.API_ERROR 500 path (e.g. "if this persists across retries/projects, it's likely an upstream Keboola incident — contact Keboola support") would save the next person the same investigation.

Open questions

• Is there a better internal channel (Keboola support / a status page not covered by status.keboola.com or keboolastatus.com) that kbagent maintainers already use to escalate confirmed upstream 500s like this one? If so, pointing to it from the error message would be more useful than a generic hint.

[vojtechnovotny-heu]

Separate from the 500 above — would you consider adding a `kbagent token list` (e.g. `GET /v2/storage/tokens`)? Right now `token` only has `create`/`delete`/`refresh`, so there's no way to see what tokens already exist (or get an id to pass to `delete`/`refresh`) without going through the GUI or a raw API call. Would round out the token subcommand group and fit the "no need to switch to the GUI" goal.

[padak]

Thanks — this was a good report, and the investigation you did up front made it much faster to act on. Both proposals are now merged in #616, along with the token list from your comment. Some findings that change the picture on your first point.

Your "non-idempotent retry risk" is not hypothetical — that branch is reachable.

You wrote it conditionally ("if the API ever creates the token server-side and then fails to respond cleanly"). Reading Storage_Service_Tokens::createToken, that is exactly the shape of the code:

$newToken = $this->_accessTokensTable->createRow($tokenData);
$newToken->save();                                    // token already persisted
$newToken->refreshToken(...)->fillAdminOwner()->save();

if ($bucketPermissions !== null && !$canManageBuckets) {   // can throw
    $bucketIdsMap = $this->getAllProjectBucketIdsMap(...);
    $container = $bucketPermissions->evaluate(...);
    ...
}

The token row is saved and its secret generated before the block that can throw, and — unlike updateToken immediately below it, which wraps the equivalent work in a transaction — createToken is not transactional. An exception raised in that block leaves a live token behind that the caller never sees. So on a bad day, your three attempts would have left three of them.

You had no way to check whether that happened, because there was no way to list tokens. That is why the two asks in this issue ship together rather than separately.

On the 500 itself: the endpoint was not down at the time.

Datadog on that stack (env:com-keboola-gcp-europe-west3) records POST https://connection.europe-west3.gcp.keboola.com/v2/storage/tokens "HTTP/1.1 200 OK" at 2026-08-18 08:58:43 UTC — 14 minutes after you filed this. So it is not a stack-wide outage, which matches your observation that status pages were clean.

The difference in that successful call is suggestive: it sets canManageBuckets: true, which is precisely the flag that makes createToken skip the getAllProjectBucketIdsMap / BucketPermissions::evaluate block. kbagent never sets it, so it always enters that block. And in the storage-token path (TokenCreateProcessor::process) a Storage_Service_TokensException thrown there is not caught — unlike the manage-token path (ProjectTokensService), which converts it into a validation error. An uncaught exception surfaces as exactly the generic 500 {"error": "Application error."} you saw.

I could not locate your specific failing request in the logs, so treat that as a hypothesis from reading the source rather than a confirmed root cause. It should narrow things down when you raise it with support, and it argues for including the exceptionId (see below) in that report.

What changed in kbagent (#616)

1. 5xx is retried only on idempotent methods (GET/HEAD/OPTIONS/PUT/DELETE). A POST/PATCH answered with a 5xx now fails on the first attempt and reports retryable: false. 429 is still retried on every method — the server is stating it did not process the request. Transport failures are split by what they prove: a refused connection or a connect/pool timeout never delivered the request and is still retried, while a read/write timeout on a POST is not (that path was the same hazard as the 500 one, just quieter). This applies to all 29 POST call sites, not just the token mint — config oauth-url mints a token through the same endpoint.

2. The 500 message now carries the exceptionId. The old parser read Keboola's generic error: "Application error." and silently discarded the exceptionId beside it — which, to your open question about escalation, is the one field Keboola support can trace the incident by. It is now included (bounded and character-filtered, since it is untrusted input being rendered into a terminal), alongside one of two hints: not retried because the method is not idempotent — the operation may already have taken effect, verify before repeating, or the same 5xx survived every attempt — likely an upstream incident, escalate with the id.

   Your repro gets the first of those, not the second: a token mint is a POST, so it is no longer retried at all and the "already taken effect" warning is the one that fires. That is the message you would have wanted on day one — it points straight at the check in the next section rather than at a support ticket.

3. kbagent token list --project P (GET /v2/storage/tokens) — id, description, created, expires with an expired marker, master flag, and the creating token. Secret values are stripped from every row, --json included: on a project with the force-decrypted-token feature the API embeds live secrets in the listing, and echoing those would break the "revealed once, at mint" rule for every token in the project at once. Also mirrored on kbagent serve and on the importable SDK as Client.list_tokens().

One thing worth doing on your side once this ships

Given the first point, it is worth running kbagent token list --project P against those two projects and looking for tokens matching the descriptions you passed during testing. If your repro did leave orphans behind, they are live credentials sitting in the project — kbagent token delete --project P --token-id ID revokes them. Worth doing even if the count looks right: a token minted by a failed attempt carries the same description as the one you meant to create, so duplicates show up as repeated rows rather than as anything obviously wrong.

Merged; it ships in the next release and I will note the version here when it goes out.

[vojtechnovotny-heu]

Verification on the orphan-token question: checked kbagent token list across the projects that had been used with these scripts — none show more than intended tokens, so the retry-orphan scenario didn't materialize in practice.

Also re-reproduced the original 500 today on v0.86.0, this time with --json to capture the trace id:

$ kbagent --json token create -p <project> -d <description> --expires-in 7884000
 {"status":"error","error":{"code":"API_ERROR","message":"API error 500 ... [exceptionId: com-keboola-gcp-europe-west3-connection-c77f3110002e88f5f91e170f9dd9d53e] POST is not idempotent, so this request was not retried -- the operation may already have taken effect server-side; verify the resource state before trying again."}}

token list right after confirms no token was created — still a clean failure, not a silent partial success.

Thanks for your help Petr!

[padak]

Ahoj Vojto!
Sorry for the delay — I was in transit and am now surviving jet lag :) Thanks for the comment — the exception ID really helps, and I can check it in our Datadog!

[padak]

Root cause found — and your re-repro on 0.86.0 is what cracked it. The exceptionId you captured resolves (in Keboola's internal logs) to:

LogicException: "Normal token cannot have manage tokens"
  at connection/src/Storage/Tokens/Voters/CreateTokenVoter.php:83
  via TokenCreateAction (POST /v2/storage/tokens)

What is actually happening

Keboola distinguishes admin (master) tokens — bound to a user account — from normal tokens (created for scripts/services, bound to nobody). The Storage API's create-token authorization voter treats "a normal token that carries canManageTokens" as an impossible state: it doesn't deny it, it throws a LogicException, which surfaces as the generic 500 Application error. you saw — not a 403.

The trap: that state is manufacturable. The Manage API create-token path applies no such check — and that is exactly the path kbagent org setup / kbagent project refresh use to mint project tokens, with canManageTokens: true hardcoded. So every project onboarded that way holds a token the Storage API considers impossible.

The assert lives only in the create-token action. Jobs, configs, storage, sync, even token list never touch it — which is why the token works everywhere else and why this looked like a server-side incident rather than a token problem. Every observation in your report is consistent with it:

• reproduces regardless of scope/expiry/bucket flags → the voter throws before the request body matters
• reproduces across 2 projects / 2 tokens → both were minted by the same kbagent path
• "canManageTokens confirmed" → that flag is the trigger, on a non-admin token
• no orphan tokens (your token list check) → the voter throws before anything is persisted; my earlier partial-persistence hypothesis was wrong for this failure — nothing is ever created

Verified by A/B on an unrelated project (different org, same stack)

Acting token	token create
master token (admin)	succeeds (minted + revoked, id 7434917)
token minted by kbagent project refresh (non-admin, canManageTokens=true, id 7434918)	500, exceptionId ...c43dc0b0b38659003b6e652f1528a792 → same LogicException, same line

Same command, same project, ~2 minutes apart. The only variable is the kind of acting token.

Workaround (now)

Point the project alias at a master token for token-minting work:

kbagent project edit --project <alias> --token <MASTER_TOKEN>

(Master token = the one from your own user account in the Keboola UI, isMasterToken: true in kbagent token list.) Your current org-setup token remains fine for everything else — only token create and config oauth-url need the master one. config oauth-url already has a pre-flight guard for exactly this (clean MISSING_MASTER_TOKEN error); token create will get the same guard so the next person sees an actionable message instead of a 500.

Follow-ups on our side

1. kbagent: pre-flight master-token guard on token create + fix the docs wording ("needs canManageTokens" is necessary but not sufficient; the real requirement is a master/admin token).
2. The underlying questions (why the Manage API can manufacture a state the Storage API rejects, whether kbagent should stop setting canManageTokens, and 500-vs-403) go to Keboola's token expert — call-prep materials in the next comment.

Thanks for the disciplined verification along the way — the --json capture with the exceptionId is what made this traceable.

[padak]

Call-prep materials for @zajca

These are prepared materials for a call with @zajca (Keboola token expert) — @padak will walk through them with him. Everything below is verified against production europe-west3.gcp on 2026-08-21 unless marked as an open question.

The inconsistency in one sentence

The Manage API create-token endpoint accepts canManageTokens: true for a token with no admin binding, while the Storage API's CreateTokenVoter treats exactly that combination as an impossible state and throws LogicException('Normal token cannot have manage tokens') — surfaced to the caller as a generic 500 Application error., only on POST /v2/storage/tokens.

// connection/src/Storage/Tokens/Voters/CreateTokenVoter.php:83
if (!$authorizedTokenIsAdminToken && $tokenCanManageTokens) {
    throw new LogicException('Normal token cannot have manage tokens');
}

How kbagent manufactures the state

kbagent org setup / kbagent project refresh mint project tokens via Manage API (POST /manage/projects/{id}/tokens) with these hardcoded defaults (manage_client.py):

canManageBuckets:        true
canReadAllFileUploads:   true
canReadAllProjectEvents: true
canManageDevBranches:    true
canManageTokens:         true      <-- the trigger

Description pattern: "{prefix} [{owner}]", e.g. kbagent-cli [petr@keboola.com].

Exact reproduction (any project, ~3 minutes)

# 1. mint the token via the suspect path (prompts for a manage token)
kbagent project refresh --project <alias> --force

# 2. confirm the state: acting token has isMasterToken=false, no admin block, canManageTokens=true
kbagent --json token list --project <alias>

# 3. the decisive call
kbagent --json token create --project <alias> -d "repro" --expires-in 300
#    -> 500 {"error":"Application error.","exceptionId":"..."}
#    -> token list confirms nothing was created (voter throws pre-persistence)

# control: repeat step 3 with a master token in the alias -> succeeds

Live evidence on stack europe-west3.gcp:

• our A/B (project 3028, org 425): master token 6610637 → create OK; refresh-minted token 7434918 → 500, exceptionId com-keboola-gcp-europe-west3-connection-c43dc0b0b38659003b6e652f1528a792 (2026-08-21 ~21:42 UTC), Datadog trace ends at CreateTokenVoter.php:83 via TokenCreateAction.php:82
• reporter's independent case: exceptionId com-keboola-gcp-europe-west3-connection-c77f3110002e88f5f91e170f9dd9d53e (2026-08-20 ~08:33 UTC) — same exception, same line

What such a token can and cannot do (verified)

Operation	Result
jobs, configs, storage, sync, sharing (daily kbagent traffic)	works
GET /v2/storage/tokens (token list)	works
token delete / refresh of other tokens	works (used in production for months)
POST /v2/storage/tokens (token create; also behind config oauth-url)	500

So the "impossible" state is enforced at exactly one action, which is why it survived unnoticed since the org-setup feature shipped.

Questions for the call

1. What is canManageTokens supposed to mean on a non-admin token? Empirically it grants list/delete/refresh and forbids create (by crash). Is the Manage API wrong to accept it, or is the voter wrong to reject it?
2. What would break if kbagent stopped setting it? Our own docstring claims it is "Required for Scheduler (Orchestrator) to create run tokens" — unverified. Does the Scheduler really create run tokens as this token? And would token list/delete/refresh stop working without it?
3. Events & dev branches: @padak's working assumption is that CMT is needed for reading events and working with dev branches — but those appear to have dedicated flags (canReadAllProjectEvents, canManageDevBranches). Can you confirm the flags are independent, i.e. dropping CMT alone would not affect them?
4. 500 vs 403: even if the state stays "impossible", should the voter answer with a clean 403/validation error instead of LogicException → 500? The 500 is what cost the reporter (and us) the investigation.
5. Existing fleet: every project onboarded via kbagent org setup holds such a token today. If the server-side rules change, is any migration/backfill needed, or do existing tokens keep working as-is?

kbagent-side plan (independent of the call's outcome)

• pre-flight guard on token create mirroring the existing config oauth-url MISSING_MASTER_TOKEN guard (clean error instead of 500)
• docs wording fix: "acting token must carry canManageTokens" → "requires a master/admin token" across CLAUDE.md / commands-reference / agent prompt / docstrings

[padak]

Status update: the diagnostics/retry-safety work from this thread is fully shipped (0.86.0) — token list (id/description/expiry/master-flag), 5xx retried only on RFC 9110-idempotent methods so a token-create POST fails clean on the first try, and the exceptionId surfaced in the error message. That's what let us trace the root cause to CreateTokenVoter.php:83 (LogicException: Normal token cannot have manage tokens on a non-admin token with canManageTokens=true, exactly the shape org setup / project refresh mint).

The one item from that root-cause comment still outstanding: token create (and token refresh) don't yet have the pre-flight master-token guard that config oauth-url already has (MISSING_MASTER_TOKEN, exit 3, checked via get_project_info().isMasterToken before the write). Right now a non-master token still gets the raw 500 from the API instead of a clean local error. The docstrings in token_service.py / client/tokens.py also still say "the acting token must carry canManageTokens," which we now know is misleading — the real requirement is a master/admin token. Leaving this open until that guard + docs fix land; the platform-side question ("should kbagent stop setting canManageTokens on non-admin mints") may also change what the guard should check.