Skip to content

HDDS-17302. OM client retry wait time grows without bound with a single OM - #11682

Draft
smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-17302
Draft

smengcl wants to merge 2 commits into
apache:masterfrom
smengcl:HDDS-17302

Conversation

@smengcl

@smengcl smengcl commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

With a single OM, the OM client increases the wait time between retries linearly and without a cap when the OM is not reachable, because round robin failover selects the same OM again and getWaitTime treats that as a retry on the same OM requested by the OM. A client started before OM is ready (e.g. right after starting a Docker Compose cluster) can keep sleeping long after OM is up.

This change checks whether all OMs were attempted in the current round first, so a single OM gets the same fixed wait as a full round over multiple OMs. Retries on the same OM requested by the OM (e.g. OMLeaderNotReadyException) still increase the wait time. Behavior with multiple OMs is unchanged.

As a result, a client talking to a single OM that stays down gives up much sooner: with the defaults (ozone.client.failover.max.attempts=500, ozone.client.wait.between.retries.millis=2000), the total wait drops from about 70 hours to about 17 minutes, in line with a full round over multiple OMs.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17302

How was this patch tested?

Added TestOMFailoverProxyProvider#testWaitTimeWithSingleNode, which fails without the fix, including when the reset of the same-OM retry count is removed. Existing failover proxy provider and transport unit tests pass.

🤖 Generated with Claude Code

…le OM

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings October 22, 2026 07:56
@smengcl smengcl added the AI-gen label Oct 22, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants