Skip to content

HDDS-15533. DNS refresh on heartbeat failure for DN to SCM (on HDDS-15682)#11

Closed
kerneltime wants to merge 2 commits into
szetszwo:HDDS-15682from
kerneltime:HDDS-15533-on-15682
Closed

HDDS-15533. DNS refresh on heartbeat failure for DN to SCM (on HDDS-15682)#11
kerneltime wants to merge 2 commits into
szetszwo:HDDS-15682from
kerneltime:HDDS-15533-on-15682

Conversation

@kerneltime

Copy link
Copy Markdown

What changes were proposed in this pull request?

Daisy-chained on apache#10612 (HDDS-15682), which keys SCM-node identity on the configured host:port instead of the resolved InetSocketAddress. Two commits:

  1. Address review nits on HDDS-15682. Use the original configured host and port to identify SCM nodes apache/ozone#10612 — null-safe SCMNodeInfo getters (the String getters returned null gracefully before HDDS-15682. Use the original configured host and port to identify SCM nodes apache/ozone#10612; the HostAndPort versions NPE on the nullable block/client/security/datanode addresses), plus a contract note on the InetSocketAddress-derived HostAndPort constructor.

  2. HDDS-15533 DN→SCM DNS refresh — ports the feature from HDDS-15533. DNS refresh on heartbeat failure for DN to SCM apache/ozone#10488 onto the new identity model. Because the map key is now the stable host:port, it is far smaller than HDDS-15533. DNS refresh on heartbeat failure for DN to SCM apache/ozone#10488: it drops the endpoint re-key dance, the resolved-address collision check, and the StateContext queue migration.

    • HostAndPort.refresh() re-resolves and swaps the cached address (DNS lookup outside the monitor; only the swap synchronized) — completing the mutable-address hook HDDS-15682. Use the original configured host and port to identify SCM nodes apache/ozone#10612 left, and justifying the synchronized getter.
    • SCMConnectionManager.refreshSCMServer() rebuilds the endpoint value under the same key and closes the stale proxy (buildScmEndpoint() factored out of addSCMServer).
    • HeartbeatEndpointTask triggers it on a connection-class failure, gated by ozone.client.failover.resolve-needed and the new hdds.heartbeat.address.refresh.threshold knob.

Verified that apache#10612 alone does not fix the stale-IP bug: HostAndPort resolves once in its constructor and getAddress() returns the cached value, and the RPC proxy freezes it — so the failure-triggered refresh is still required.

Relationship to the existing PRs

What is the link to the Apache JIRA?

https://issues.apache.org/jira/browse/HDDS-15533

How was this patch tested?

New TestHeartbeatEndpointTaskDnsRefresh covers the trigger gating: a connection-class failure at the threshold triggers a refresh; flag-off, application-level errors, and below-threshold do not. mvn checkstyle:check passes on hadoop-hdds/common and hadoop-hdds/container-service. Full module build/test was not run locally; relying on CI.

Generated-by: Claude Code (AI-assisted)

…ent HostAndPort host-spelling

From the AI-assisted review of apache#10612:
- SCMNodeInfo block/client/security/datanode getters NPE on null fields; the prior String getters
  returned null gracefully, so restore that contract.
- Document the host-spelling contract on the InetSocketAddress-derived HostAndPort constructor so
  equal-by-host instances do not miss each other in the endpoint maps.

Generated-by: Claude Code (AI-assisted)
…ort identity model)

Ports the DN->SCM DNS-refresh feature onto HDDS-15682's host:port identity. Because SCM nodes are
now keyed by the stable host:port, this drops the endpoint re-key dance, the resolved-address
collision check, and the StateContext queue migration that the InetSocketAddress-keyed version
(apache#10488) needed.

- HostAndPort.refresh(): re-resolve the host:port and swap the cached address if the IP changed
  (DNS lookup outside the monitor; only the swap synchronized). Completes the mutable-address hook.
- SCMConnectionManager.refreshSCMServer(): on a changed IP, rebuild the endpoint value under the
  same key and close the stale proxy. buildScmEndpoint() factored out of addSCMServer.
- HeartbeatEndpointTask: on a connection-class heartbeat failure, gated by the resolve-needed flag
  and the new hdds.heartbeat.address.refresh.threshold knob, ask the connection manager to refresh.

Generated-by: Claude Code (AI-assisted)
@github-actions

Copy link
Copy Markdown

This PR has been marked as stale due to 21 days of inactivity. Please comment or remove the stale label to keep it open. Otherwise, it will be automatically closed in 7 days.

@kerneltime

Copy link
Copy Markdown
Author

Superseded by apache#10844, rebuilt on master now that apache#10612 (HDDS-15682) has merged. In the merged apache#10612 the SCMNodeInfo/HostAndPort nits from commit 1 were addressed directly (and HostAndPort.address was made final with a // TODO: HDDS-15533), so commit 1 is obsolete and the feature has been reworked to complete that TODO. Closing.

Generated-by: Claude Code (Opus 4.8)

@kerneltime kerneltime closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant