Repository navigation
HDDS-16798. Stale or Dead Datanodes whose software is not upgraded may still communicate with a finalized SCM - #11692
Merged
Merged
Conversation
Contributor
There was a problem hiding this comment.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Copilot review overview
4 open issues
NodeNotFoundExceptionwithout any logging can make operational troubleshooting… · Newheartbeat.hasDatanodeVersion(), add a unit test… · NewdnIDis computed but not used foraddDatanodeCommand, which repeatsdatanodeDetails.getID().… · New
What changed in this PR
Ensures SCM finalization fences stale/dead datanodes running incompatible software by forcing re-registration on heartbeat, and adds unit tests to validate the behavior.
Changes:
- Add heartbeat-time version gating for unhealthy (STALE/DEAD) datanodes and issue
ReregisterCommandwhen incompatible. - Expose
shouldFenceDatanodeviaNodeManagerAPI and update implementations to be callable from the dispatcher. - Add unit tests for stale/dead nodes with incompatible versions.
| File | Description |
|---|---|
| hadoop-ozone/recon/src/main/java/org/apache/hadoop/ozone/recon/scm/ReconNodeManager.java | Makes shouldFenceDatanode publicly callable to align with the new NodeManager API. |
| hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMDatanodeHeartbeatDispatcher.java | Adds unhealthy-node version gate and triggers re-registration when incompatible. |
| hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/NodeManager.java | Introduces shouldFenceDatanode as a default interface method for version-based fencing. |
| hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java | Updates shouldFenceDatanode visibility to satisfy/override the new interface method. |
| hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/server/TestSCMDatanodeHeartbeatDispatcher.java | Adds tests to ensure stale/dead incompatible nodes get a re-register command and heartbeat is not processed. |
Review effort: Lite
added 2 commits
October 22, 2026 11:59
…y still communicate with a finalized SCM
errose28
added
the
zdu
errose28
reviewed
Oct 22, 2026
added 7 commits
October 23, 2026 12:24
…nodeIfNeeded for fenced nodes
Contributor
Author
|
@errose28 Thanks for the comments. I have hopefully address them all in the latest push. |
errose28
approved these changes
Oct 29, 2026
errose28
left a comment
errose28
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Just one minor test comment that would be nice to fix before merge.
| } | ||
|
|
||
| @Test | ||
| public void testHealthyNodeWithVersionReportIsNeverFenced() throws Exception { |
Contributor
There was a problem hiding this comment.
This test trivially passes regardless of the node status because MockNodeManager#shouldFenceDatanode always returns false. It needs to be mocked to return true to ensure that the status is actually what caused the node to be let through.
Contributor
|
Thanks @sodonnel for the patch, @errose28 for the review.
Merged without this fixup because test-only changes are much cheaper for CI than doing another round here. Let's do it in follow-up or can include it in any related fix in the near future. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Only all HEALTHY nodes need to be upgraded for SCM start finalization, but once SCM finalizes, it should not allow communication with nodes in the older software. However this is checked at the time of registration and STALE and DEAD nodes can heartbeat back to SCM without reregistering.
This is fixed by intercepting the heartbeat for stale and dead nodes and telling them to re-register. Then they are fenced using the same logic as newly started nodes.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16798
How was this patch tested?
New unit tests added