Skip to content

[fix](cloud) Reject unregistered cloud_unique_id and mismatched instance_id in meta service - #69732

Open
deardeng wants to merge 4 commits into
apache:masterfrom
deardeng:safe-unique-id
Open

deardeng wants to merge 4 commits into
apache:masterfrom
deardeng:safe-unique-id

Conversation

@deardeng

@deardeng deardeng commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

This PR strengthens instance identity validation in Meta Service RPCs.
Existing management APIs remain compatible.

Check List (For Author)

  • Test: Unit test coverage in meta_service_test, resource_test, and RecycleVersionedKeysTest. The tests cover valid requests and rejection of invalid instance identities.
  • Behavior changed: Yes. See the release note above.
  • Does this need documentation? No.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@deardeng

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Codex automated review failed and did not complete.

Error: workspace routing discovery unauthorized (401)
Workflow run: https://github.com/apache/doris/actions/runs/40648808823

Please inspect the workflow logs and rerun the review after the underlying issue is resolved.

@deardeng

Copy link
Copy Markdown
Contributor Author

/review

@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files and their FE/BE callers, resource-manager cache/registration paths, proxy retries, HTTP dispatch, and test-fixture consumers at 5eddd16. One P2 finding: a transient membership KV failure becomes terminal INVALID_ARGUMENT for a legitimate registered node.

Critical checkpoint conclusions:

  • Goal and proof: exact membership and populated singular identity-pair checks cover the added GetCluster scenarios; KV-failure handling remains incorrect as reported inline.
  • Scope and reuse: focused shared resolver/proxy changes and necessary fixture adjustments; no unrelated production change.
  • Concurrency: tenant RPC bthreads race asynchronous topology refresh; existing shared/exclusive ResourceManager mutex protects the cache. New scans use local protobufs; no added lock-order issue.
  • Lifecycle and initialization: early returns complete the closure and normalize status; the per-request-type local static is thread-safe. No new ownership or global initialization dependency.
  • Configuration: no new knob. Existing enable_check_instance_id defaults true; its false mode remains an explicit validation opt-out.
  • Compatibility: no symbol/storage-format change requiring migration; actual FE/BE single-identity producers and successor routing were traced without another new regression.
  • Thrift/protobuf IDs: no definition changes.
  • Parallel paths: cached/degraded resolution, all scalar instance_id request shapes, plural status requests, bootstrap, and token-authorized HTTP were checked. Existing instance-only/plural identity policies are unchanged limitations, not new findings or proof of complete tenant authentication.
  • Conditional checks: missing identity compatibility matches current FE bootstrap; unsuccessful KV reads must retain their error classification as the finding explains.
  • Test coverage: inspected positive registered-ID and negative forged/mismatched-ID cases plus all changed fixture consumers. Storage-vault fixtures exercise successful cache-miss membership; transient membership KV failures lack coverage.
  • Test results: no expected-result files changed; builds and tests were not run under this static-only contract.
  • Observability: rejection logs identify the request/identities; the reported failure currently misleadingly logs an unregistered node. No other material logging gap found.
  • Transactions and persistence: production changes read metadata; existing commit-before-refresh behavior and retry contracts were traced. No new persisted state or failover protocol.
  • Writes and crash safety: the finding can abort prepare_rowset before mutation; no additional partial-write, atomicity, or crash-lifetime issue substantiated. Visible versions and delete-bitmap rules are unchanged.
  • FE/BE variable passing: no new transmitted variables or sending paths.
  • Performance and memory: existing cache-miss KV read gains a linear scan of its local protobuf; no material additional allocation or hot-path issue substantiated.
  • Other concerns: all candidate/risk items were accepted, merged, or dismissed with evidence; no additional user focus was supplied. No existing inline findings required carrying forward.

Verification limits: static inspection only, without builds, tests, benchmarks, or runtime validation. This review assesses this diff and does not certify the pre-existing Meta Service identity design.

Review converged after 3 round(s).

Comment thread cloud/src/meta-service/meta_service.cpp Resolved
@deardeng

Copy link
Copy Markdown
Contributor Author

/review

@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed all five changed files at 2d895c7e69326d758ac1b26a4e6f229f02f26ddf. No new actionable findings. Both the complete-review and separate risk-focused reviewer returned NO_NEW_VALUABLE_FINDINGS; all initial risks were verified and dismissed or deduplicated.

The existing P2 about retryable KV errors remains applicable: a legitimate uncached node can receive INVALID_ARGUMENT after a transient instance read failure. This head explicitly rejects/logs failed reads, and the author documents that behavior as intentional for this fix. The concern is already reported and is not duplicated. No existing P0/P1 finding requires carrying forward.

Critical checkpoint conclusions:

  • Goal and proof: the resolver checks persisted node membership on degraded misses, and the proxy rejects mismatched nonempty instance_id/cloud_unique_id pairs. New real-ResourceManager tests cover valid IDs, forged suffixes, unresolvable IDs, cross-instance pairs and KV-error rejection. These tests were inspected, not executed.
  • Scope and clarity: changes are confined to two production files and three relevant test/fixture files; shared preflight avoids duplicated handler checks.
  • Concurrency and locks: get_node copies under a shared cache mutex; refresh uses its exclusive lock. Added instance reads and membership scans run after cache-lock release on request-local state. No added lock nesting or deadlock path was identified.
  • Lifecycle and initialization: closure completion and response-code normalization are installed before early rejection. The descriptor-presence bool is a function-local static; no new cross-translation-unit initialization dependency or ownership cycle was identified.
  • Configuration: no configuration is added. Existing enable_check_instance_id defaults true; its disabled behavior remains unchanged.
  • Compatibility: no RPC signature, wire or storage format changes. Instance-only management requests remain supported. FE first-node registration uses instance_id alone, so there is no membership-before-registration cycle.
  • Thrift/protobuf IDs: no definitions changed. Every reflected field is an optional singular string in the current schema; no field-ID migration is involved.
  • Parallel paths: checked GetInstance, AlterCluster, GetCluster, ListSnapshot and CompactSnapshot, their FE/BE callers, and the token-authorized HTTP implementation route. CloneInstance uses distinct source/target fields and is unaffected. Plural selectors remain unchanged.
  • Conditional checks and errors: failure to read membership terminates validation; empty-field compatibility is documented. INVALID_ARGUMENT retry behavior is the existing P2 above.
  • Test coverage: real-resource proxy/resolver cases exercise the changed checks; recycler and storage-vault fixtures now persist matching nodes. Per-handler integration and runtime concurrency behavior were not executed under the static-only contract.
  • Expected results: updated C++ expectations match the intended rejection behavior; no golden result files changed.
  • Observability: new rejection logs include IDs, request type or KV failure context. No new metric cardinality or hot-path INFO logging was introduced; no additional actionable observability defect was found.
  • Transactions, persistence and failover: production changes add validation reads only. Commit path selection, durability/replay, versionstamp allocation, partition versions and delete-bitmap consistency are unchanged.
  • Data writes and crashes: rejected calls stop before handler side effects. Registration still commits before local refresh; fixture cluster removal commits before DROP. No new partial-write or crash-cleanup path was identified.
  • FE/BE variable passing: no new transmitted variable or sender-path change is required.
  • Performance: descriptor presence is cached; fallback membership is a linear scan over the instance already read. Both-field requests add a resolver lookup, with no substantiated material regression from static inspection.
  • Other invariants: no new memory-ownership, committed-data visibility, rowset-version or MoW bitmap issue was identified.

No additional user focus was supplied. All changed-file coverage and candidate/risk dispositions are complete. Review converged in one round. Limits: static inspection only; no builds, tests or runtime validation were performed. SnapshotManager list/clone/compact implementations in this checkout return Not implemented, so no conclusions extend to external implementations.

Review converged after 1 round(s).

@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

1 similar comment
@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

deardeng added a commit to deardeng/incubator-doris that referenced this pull request Nov 17, 2026
Related PR: apache#69732

Problem Summary:

Rpcs naming an instance by `instance_id` alone, e.g. `alter_cluster`, `alter_instance`,
`update_ak_sk`, `alter_ram_user`, `list_snapshot`, `compact_snapshot` and `get_cluster_status`
by `instance_ids`, are accepted by meta service without any proof that the caller belongs to
that instance. They can not simply be rejected, because FE sends `alter_cluster` and
`alter_instance` with `instance_id` only, and `AlterInstanceRequest` has no
`cloud_unique_id` field.

Fix:
1. New meta service config `enable_strict_instance_auth` (default false, not mutable). When
   enabled, `MetaServiceProxy::call_impl` rejects with INVALID_ARGUMENT an rpc carrying
   `instance_id` without `cloud_unique_id`, and `get_cluster_status` with `instance_ids`.
   `alter_instance` REFRESH, which meta service sends to other meta services by instance_id,
   is still accepted. The http api authorized by token calls the impl directly and is not
   affected.
2. `AlterInstanceRequest` gets `optional string cloud_unique_id = 6`.
3. FE `MetaServiceClient` fills its `cloud_unique_id` into `alter_cluster` and
   `alter_instance` requests that do not carry one, as it already does for other rpcs. The
   cloud_unique_id derived from cluster_id (`1:<id>:fe`) is not filled, since anyone can build
   it and it is not registered before the first FE adds itself.

This only requires `cloud_unique_id` to be present. Whether `instance_id` belongs to the
`cloud_unique_id` is checked by apache#69732, and the two are meant to be used together.

Rollout: upgrade meta service, then all FE, then enable the config. It is not meant for
cluster_id mode, where FE creates and joins its instance by instance_id.

Add meta service config `enable_strict_instance_auth` to reject rpcs that name an instance
by instance_id without a cloud_unique_id. FE now sends its cloud_unique_id in alter_cluster
and alter_instance requests.

- Test: Unit Test
    - meta_service_test StrictInstanceAuthTest covers the config off and on
    - FE MetaServiceClientTest covers filling, keeping a given and skipping a derived
      cloud_unique_id
- Behavior changed: Yes. FE sends cloud_unique_id in alter_cluster and alter_instance; with
  the new config enabled, rpcs naming an instance by instance_id only are rejected.
- Does this need documentation: Yes, the new meta service config
@deardeng

Copy link
Copy Markdown
Contributor Author

run buildall

hello-stephen pushed a commit that referenced this pull request Nov 18, 2026
Related PR: #69732

### What problem does this PR solve?

This PR improves instance-level validation for Meta Service RPCs and
enables it by default.

It updates FE request handling and combines first-FE registration into a
single RPC. Existing `cluster_id` deployments remain supported.
Temporary compatibility is retained for virtual compute group updates
during FE upgrades.

### Release note

`enable_strict_instance_auth` now defaults to `true`.

FE self-registration in `cluster_id` mode requires no additional
configuration. For mixed-version deployments,
`enable_strict_instance_auth = false` remains available as a temporary
compatibility setting.
deardeng added a commit to deardeng/incubator-doris that referenced this pull request Nov 19, 2026
Related PR: apache#69732

Problem Summary: Meta service and recycler write secrets to their logs, and
some into error messages, in plaintext:

- http admin token: `MetaServiceImpl::http` formats the whole http request
  (every query string and header) and logs it at INFO for every request,
  successful or not, so the `token` in the query string, which can act on
  every instance, reaches the log. When the token check fails, meta service
  and recycler also echo the supplied token in the 403 body and the WARNING
  log.
- cloud_unique_id: since apache#69732 a cloud_unique_id
  `${version}:${instance_id}:${node_secret}` authenticates a node, but
  `begin_rpc` / `finish_rpc` log the request and response debug strings
  (e.g. every `get_cluster` logs the cloud_unique_id of the caller and of
  every node in the returned cluster), the http request log contains it,
  about 60 log lines and error messages print it, and about 40 dump an
  instance, cluster or node PB that contains it.
- ak/sk: when `alter_obj_store_info`, `alter_storage_vault`,
  `alter_ram_user` or `update_ak_sk` rejects a request, the error message
  embeds the request as JSON, ak/sk included; `finish_rpc` logs it and the
  caller receives it. `begin_rpc` hides sk only for CreateInstance,
  CreateStage and AlterObjStoreInfo, so `update_ak_sk` and `alter_ram_user`
  log the sk.

Fix:
1. Mask the `token` query value as `***` in the logged http request, and
   drop the echoed token from the 403 message (keep `token=(not given)`
   when no token is passed).
2. Add `hide_cloud_unique_id()`, which keeps `${version}:${instance_id}:`
   for debugging and replaces the node secret with `***`, and
   `hide_cloud_unique_ids()`, which applies it to every cloud_unique_id(s)
   value in a protobuf debug string, a JSON string or a
   `cloud_unique_id=xxx` message. Apply the latter once to every debug
   string in `begin_rpc` / `finish_rpc` and to the logged http request,
   and apply them at the other log lines and error messages.
3. Add `request_to_json_for_log()`, which applies `encryt_sk()`,
   `hide_ak()` and `hide_cloud_unique_ids()` to the request JSON, and use
   it wherever meta_service_resource.cpp puts a request in an error message
   or a log. Add UpdateAkSkRequest and AlterRamUserRequest to the requests
   whose sk `begin_rpc` hides.

`hide_cloud_unique_ids()` only scans for "cloud_unique_id" and copies the
string once when there is a match. Compared with the `ShortDebugString()`
it wraps, on a Release build it adds 0.65us to a 91-byte request, 0.65ms
(7%) to a 1.7MB request with one cloud_unique_id, and 0.4ms (29%) to a
`get_cluster` response with 2000 nodes.

Meta service logs and error messages show cloud_unique_id as
`1:<instance_id>:***`, the sk as an md5 and the ak partially hidden, and no
longer contain the http admin token. The 403 body of meta service and
recycler http APIs is `incorrect token` instead of
`incorrect token, token=xxx`.

- Test: Unit Test
    - `MetaServiceHttpTest.TokenNotLogged` captures glog output of one
      authorized and one rejected http request and checks that neither
      token value appears.
    - `MetaServiceHttpTest.HideCloudUniqueIds` covers the masking formats.
    - `MetaServiceHttpTest.CloudUniqueIdNotLogged` captures glog output of
      an http `get_cluster` call and checks that neither the caller's nor
      the cluster node's secret is logged (leaks via the http body,
      `begin_rpc` and `finish_rpc`).
    - `MetaServiceHttpTest.AkSkNotLogged` captures glog output of rejected
      `alter_obj_store_info`, `alter_storage_vault`, `alter_ram_user` and
      `update_ak_sk` calls and checks that neither ak nor sk is logged.
    - Each of these fails without the change.
- Behavior changed: Yes. The 403 body no longer echoes the supplied token,
  and error messages that contained a cloud_unique_id or ak/sk now contain
  the masked form.
- Does this need documentation: No
deardeng added a commit that referenced this pull request Nov 19, 2026
#70006)

Related PR: #69732

Problem Summary: Meta service and recycler write secrets to their logs,
and some into error messages, in plaintext:

- http admin token: `MetaServiceImpl::http` formats the whole http
request (every query string and header) and logs it at INFO for every
request, successful or not, so the `token` in the query string, which
can act on every instance, reaches the log. When the token check fails,
meta service and recycler also echo the supplied token in the 403 body
and the WARNING log.
- cloud_unique_id: since #69732 a cloud_unique_id
`${version}:${instance_id}:${node_secret}` authenticates a node, but
`begin_rpc` / `finish_rpc` log the request and response debug strings
(e.g. every `get_cluster` logs the cloud_unique_id of the caller and of
every node in the returned cluster), the http request log contains it,
about 60 log lines and error messages print it, and about 40 dump an
instance, cluster or node PB that contains it.
- ak/sk: when `alter_obj_store_info`, `alter_storage_vault`,
`alter_ram_user` or `update_ak_sk` rejects a request, the error message
embeds the request as JSON, ak/sk included; `finish_rpc` logs it and the
caller receives it. `begin_rpc` hides sk only for CreateInstance,
CreateStage and AlterObjStoreInfo, so `update_ak_sk` and
`alter_ram_user` log the sk.

Fix:
1. Mask the `token` query value as `***` in the logged http request, and
drop the echoed token from the 403 message (keep `token=(not given)`
when no token is passed).
2. Add `hide_cloud_unique_id()`, which keeps
`${version}:${instance_id}:` for debugging and replaces the node secret
with `***`, and `hide_cloud_unique_ids()`, which applies it to every
cloud_unique_id(s) value in a protobuf debug string, a JSON string or a
`cloud_unique_id=xxx` message. Apply the latter once to every debug
string in `begin_rpc` / `finish_rpc` and to the logged http request, and
apply them at the other log lines and error messages.
3. Add `request_to_json_for_log()`, which applies `encryt_sk()`,
`hide_ak()` and `hide_cloud_unique_ids()` to the request JSON, and use
it wherever meta_service_resource.cpp puts a request in an error message
or a log. Add UpdateAkSkRequest and AlterRamUserRequest to the requests
whose sk `begin_rpc` hides.

`hide_cloud_unique_ids()` only scans for "cloud_unique_id" and copies
the string once when there is a match. Compared with the
`ShortDebugString()` it wraps, on a Release build it adds 0.65us to a
91-byte request, 0.65ms (7%) to a 1.7MB request with one
cloud_unique_id, and 0.4ms (29%) to a `get_cluster` response with 2000
nodes.

Meta service logs and error messages show cloud_unique_id as
`1:<instance_id>:***`, the sk as an md5 and the ak partially hidden, and
no longer contain the http admin token. The 403 body of meta service and
recycler http APIs is `incorrect token` instead of `incorrect token,
token=xxx`.
…nce_id in meta service

### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Meta service finds the instance of a request by its cloud_unique_id. When the node index
misses, `get_instance_id` parses the degraded format `1:<instance_id>:<any>` and, with
`enable_check_instance_id`, only checks that the instance exists and is not deleted. So a
caller knowing an instance_id can build a cloud_unique_id with any third segment and is
treated as a node of that instance.

Besides, `get_cluster` falls back to the request supplied `instance_id` when the
cloud_unique_id can not be resolved, and rpcs carrying both fields use `instance_id` without
checking that it belongs to the cloud_unique_id, so a registered cloud_unique_id of one
instance plus the instance_id of another reaches the other instance.

Fix:
1. A degraded format cloud_unique_id is only accepted when it is a registered node of the
   parsed instance.
2. `MetaServiceProxy::call_impl` rejects an rpc whose `instance_id` is not the instance its
   `cloud_unique_id` belongs to with INVALID_ARGUMENT. The http api, authorized by token,
   calls the impl directly and is not affected. Rpcs carrying only `instance_id` are still
   accepted, since FE sends `alter_cluster` and `create_instance` that way.

In unit test, before the fix a forged third segment, an unresolvable cloud_unique_id with an
instance_id, and a cloud_unique_id of instance A with the instance_id of instance B all get
OK with the clusters of the target instance; after the fix all get INVALID_ARGUMENT, and a
registered cloud_unique_id still gets OK.

### Release note

Meta service rejects a cloud_unique_id that is not a registered node of its instance, and an
rpc whose instance_id does not belong to its cloud_unique_id.

### Check List (For Author)

- Test: Unit Test
    - meta_service_test, resource_test and RecycleVersionedKeysTest. Fixtures using an
      unregistered cloud_unique_id now register the node.
- Behavior changed: Yes. Such requests get INVALID_ARGUMENT. When the degraded format
  fallback fails to read the instance from kv, the request is now rejected instead of
  accepted.
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

`MetaServiceProxy::check_request_instance_id` passed the `instance_id` field descriptor to
reflection without checking it, relying on `call_impl` only calling it for requests with that
field. Make the precondition explicit with `CHECK`, and state in the comment that requests
carrying only one of `instance_id` and `cloud_unique_id` are accepted without cross check.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - meta_service_test
- Behavior changed: No
- Does this need documentation: No
### What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

When a cloud_unique_id misses the node cache, `get_instance_id` reads the instance from kv to
check that the cloud_unique_id is a registered node. If the read fails with an error other than
`TXN_KEY_NOT_FOUND`, e.g. `TXN_TOO_OLD`, the instance is empty and the request was rejected with a
log saying the node is not registered, which is misleading.

Handle a failed read explicitly: keep rejecting it, since the node membership can not be
checked, and log the kv error instead.

### Release note

None

### Check List (For Author)

- Test: Unit Test
    - meta_service_test GetInstanceIdKvErrorTest: a registered node taking the degraded path
      is accepted, and rejected when the instance read fails
- Behavior changed: No
- Does this need documentation: No
### Background

MetaService resolves cloud_unique_id to an instance and rejects unregistered nodes or mismatched instance IDs. Its validation failures are logged at WARNING.

### What problem does this PR solve?

Issue Number: None
Related PR: apache#70006 (preserve its credential logging protections)
PR Type: Bugfix

Problem Summary:

Three instance-validation warnings added after the logging cleanup printed the full cloud_unique_id, including the node secret. A registered node paired with another instance ID was rejected correctly but its credential reached the server log. Log the instance ID instead and cover all three failure paths with the existing log sink.

### Details

- **Frequency:** Always when a mismatched instance ID, an unregistered degraded node ID, or a failed degraded instance lookup reaches these warnings.
- **Root cause:** The warnings streamed the raw cloud_unique_id directly into the log.
- **Why existing tests missed it:** GetClusterForgedCloudUniqueIdTest and GetInstanceIdKvErrorTest checked status codes rather than logs. Existing HTTP credential-log tests did not exercise these warnings. SQL regression cases do not capture these server-side warning messages.
- **Fix:** Remove cloud_unique_id values from the three warnings while retaining the instance ID, request type where available, and KV error context.
- **Other paths and branches:** Both get_instance_id failure branches and the shared MetaServiceProxy instance check are covered. Existing AK/SK and HTTP token masking remain intact. The introducing commits are local 55abe56 and 4339092; no published introducing PR is known. Release-branch containment was not checked.
- **Behavior change:** Validation decisions and response codes are unchanged; warning logs omit node credentials.

### Testing

- ASAN Cloud UT: the new InstanceValidationDoesNotLogCloudUniqueId test failed before the production fix because both secret markers appeared in captured server logs.
- With the fix, all 39 MetaServiceHttpTest tests passed, including the new case, existing AK/SK masking, HTTP token masking and credential-update logging tests.
- Five related instance-lookup, forged-ID, strict-authentication and bootstrap tests passed.
- clang-format 16.0.6 and git diff --check passed for the final candidate.
- clang-tidy passed for both changed files with CPLUS_INCLUDE_PATH set to the installed Clang 20 resource headers. The default invocation cannot find stddef.h.

### Release note

MetaService instance-validation warnings no longer include cloud_unique_id node credentials.

### Check List (For Author)

- Behavior changed: Yes (see Details)
- Does this need documentation: No
- Pre-submission status: Ready. Targeted checks passed; full regression suite not run.

### AI Disclosure

- Tool: Codex
- Generated: log changes, regression test and description.
- Human review: Pending author review.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants