HDDS-12991. Support enabling Ratis DataStream on an existing cluster by yandrey321 · Pull Request #10746 · apache/ozone

yandrey321 · 2026-07-13T23:13:04Z

What changes were proposed in this pull request?

Root cause

A datanode advertises its RATIS_DATASTREAM port in DatanodeDetails only when streaming is enabled at start-up, and a pipeline persists a snapshot of its members' DatanodeDetails. On a cluster that predates streaming:

When a datanode re-registers after streaming is enabled, SCMNodeManager.register() refreshed the stored node record only on an IP/hostname or version change, so the newly-exposed RATIS_DATASTREAM port was dropped (HDDS-15799).
DatanodeDetails.getPort(RATIS_DATASTREAM) silently falls back to the RATIS/gRPC port, so a streaming client could target the wrong port (HDDS-8687).
A pipeline created before streaming keeps a portless member snapshot. Even after its datanodes restart with streaming enabled, that pipeline cannot serve a stream: its Ratis group's persisted configuration was built with a datastream address that resolves to the gRPC port, and the group configuration is not rebuilt on restart. Such a pipeline has to be recreated to become streaming-capable.

Changes

SCM refreshes a datanode's ports on re-registration — SCMNodeManager. When a registered datanode re-registers with a changed port set (e.g. now exposing RATIS_DATASTREAM), SCM updates its stored node record, comparing ports as a name→value set. This reuses the existing NodeStateManager.updateNode path; node state is rebuilt from registrations, so there is no schema or upgrade impact.
The streaming client degrades gracefully instead of failing — hadoop-hdds/client, hadoop-ozone/client, hadoop-ozone/common, ozonefs-common. A new StreamNotSupportedException is thrown by BlockDataStreamOutput.setupStream when a pipeline node lacks the RATIS_DATASTREAM port (using the explicit hasPort check, without changing getPort fallback semantics). KeyDataStreamOutput propagates it recognizably, and SelectorOutputStream recovers on the first flush by closing the streaming stream and replaying the buffered data to a non-streaming fallback stream, wired in BasicOzoneFileSystem/BasicRootedOzoneFileSystem. Writes to a not-yet-streaming pipeline therefore keep succeeding.
SCM replaces pipelines that cannot stream — PipelineManager. A new closeNonStreamablePipelines(), run from the background pipeline scrubber, closes and deletes OPEN pipelines whose registered datanodes now expose a port the pipeline's stored snapshot lacks. BackgroundPipelineCreator then builds fresh, streaming-capable pipelines from the now-capable nodes. Closing (rather than editing the snapshot) is required because a pre-streaming Ratis group cannot be made streamable in place. It reuses the existing replicated closePipeline/deletePipeline operations, so no new HA/Ratis operation is introduced.

Net effect: after streaming is enabled and the datanodes restart, writes keep succeeding via the fallback while SCM replaces the legacy pipelines; once the new pipelines exist, streaming writes flow over them — no metadata reset required.

Co-authored with Cloude Code

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12991
https://issues.apache.org/jira/browse/HDDS-15799
https://issues.apache.org/jira/browse/HDDS-8687

How was this patch tested?

Unit: TestSCMNodeManager (port refresh on re-registration; no refresh when unchanged); TestSelectorOutputStream (fallback on first flush for write(int) and write(byte[]), buffered-data replay, failing-close handling, no-fallback propagation, hasCapability); TestPipelineManagerImpl (closeNonStreamablePipelines closes a legacy pipeline while leaving streaming-capable, unregistered-node, and non-open pipelines untouched; the scrub-and-close background task). New non-test code has 100% line and branch coverage.
Integration: TestOzoneFileSystemWithStreamingDisabledDatanode (a streaming write falls back and round-trips over both o3fs and ofs); new TestOzoneFileSystemDataStreamEnablement: (a) after a rolling restart enables datastream, SCM refreshes the node ports and writes keep succeeding via fallback; (b) a pipeline that predates datastream is closed and a fresh one is created, over which a streaming write then succeeds end-to-end. Both integration tests use bounded @Timeouts.

… in case if pipeline does not have streaming port info failback to non-streaming write

peterxcli · 2026-07-14T15:31:27Z

might relate to https://github.com/apache/ozone/pull/10743/changes#r3573417136

yandrey321 · 2026-07-15T19:50:32Z

might relate to https://github.com/apache/ozone/pull/10743/changes#r3573417136

this one would be the separate work item.

jojochuang · 2026-07-20T16:44:53Z

@ss77892

szetszwo · 2026-07-20T17:15:58Z

@yandrey321 , thanks for working on this!

This AI generated PR has the usual problems of AI generated PRs:

the size is too big; see HDDS-15496. SCM async container ID export CLI #10673 (comment)
it has many unnecessary changes;
the comments are too long.

Could you clean them up.

Copilot

Pull request overview

This PR enables turning on Ratis DataStream in an existing Ozone cluster without requiring metadata resets by (1) ensuring SCM refreshes datanode port changes on re-registration, (2) making the client detect non-streamable pipelines and fall back to non-streaming writes, and (3) having SCM proactively replace legacy pipelines whose persisted member snapshots cannot support streaming.

Changes:

Refresh SCM’s stored datanode port set on re-registration when ports (eg RATIS_DATASTREAM) change.
Introduce StreamNotSupportedException and a client-side streaming→non-streaming fallback path (SelectorOutputStream + OzoneFS wiring).
Close/delete legacy OPEN pipelines that can’t be made streamable in place, plus add unit/integration coverage for end-to-end enablement.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicRootedOzoneFileSystem.java	Wires a non-streaming fallback selector into FS create output stream.
hadoop-ozone/ozonefs-common/src/main/java/org/apache/hadoop/fs/ozone/BasicOzoneFileSystem.java	Same fallback wiring for o3fs scheme.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemWithStreamingDisabledDatanode.java	Updates integration test to validate fallback + correct round-trip instead of fail-fast.
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/fs/ozone/TestOzoneFileSystemDataStreamEnablement.java	New end-to-end tests for port refresh + legacy pipeline replacement + eventual streaming success.
hadoop-ozone/common/src/test/java/org/apache/hadoop/ozone/client/io/TestSelectorOutputStream.java	Adds unit tests for StreamNotSupportedException fallback behavior and capability delegation.
hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/client/io/SelectorOutputStream.java	Adds fallback selector support and first-write recovery on StreamNotSupportedException.
hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/io/KeyDataStreamOutput.java	Propagates StreamNotSupportedException cleanly to enable callers to fall back.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/TestPipelineManagerImpl.java	Unit tests for closing non-streamable pipelines and scrubber wiring.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/pipeline/MockPipelineManager.java	Implements new PipelineManager API for tests.
hadoop-hdds/server-scm/src/test/java/org/apache/hadoop/hdds/scm/node/TestSCMNodeManager.java	Unit tests validating SCM port refresh on re-registration.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManagerImpl.java	Adds background scrubber hook + logic to close/delete pipelines that can’t stream.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/pipeline/PipelineManager.java	Adds `closeNonStreamablePipelines()` to the PipelineManager interface.
hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/node/SCMNodeManager.java	Adds port-set change detection and triggers nodeStateManager update when ports change.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/StreamNotSupportedException.java	New exception type to distinguish “pipeline cannot stream” from generic IO failures.
hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java	Throws StreamNotSupportedException when a pipeline lacks RATIS_DATASTREAM ports.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+        LOG.warn("Streaming output is not supported for this write; "
+            + "falling back to the non-streaming output stream.", e);
+        try {
+          out.close();
+        } catch (IOException suppressed) {
+          e.addSuppressed(suppressed);
+        }


+  private static boolean exposesNewPorts(DatanodeDetails stored,
+      DatanodeDetails current) {
+    final Set<DatanodeDetails.Port.Name> storedNames = stored.getPorts().stream()
+        .map(DatanodeDetails.Port::getName).collect(Collectors.toSet());
+    return current.getPorts().stream()
+        .map(DatanodeDetails.Port::getName)
+        .anyMatch(name -> !storedNames.contains(name));
+  }


amaliujia

Reading the code and I had the same feeling above: it might be much better if this PR can be split. Smaller PRs can be used to divide and conquer the problems.

I personal found it is hard for verify if the testing is complete or sufficient for a specific fix because there are many test code involved.

amaliujia · 2026-07-21T08:44:22Z

+   */
+  private static boolean portsChanged(DatanodeDetails oldNode,
+      DatanodeDetails newNode) {
+    return !portValues(oldNode).equals(portValues(newNode));


Curious that if we know that we are fixing for DatanodeDetails.Port.Name.RATIS_DATASTREAM missing issue, can we only look for the case in which the newNode has the data stream port while the old node does not, thus we need to do an update?

It does not look wrong that we compare entire port set, but the side effect is I do not know if there is any port that do not need to be updated.

yandrey321 · 2026-07-21T16:12:46Z

@szetszwo @jojochuang @amaliujia Please review the first subset of these changes: #10828

I'll add another PR for the server side part of this PR.

yandrey321 added 3 commits July 13, 2026 14:17

Graceful hanling of enabling ratis streaming with existing pipelines,…

da86466

… in case if pipeline does not have streaming port info failback to non-streaming write

Close non streamable piplines after enabling data streaming

eadd31f

fixed issues found by findbugs

1040d5e

yandrey321 mentioned this pull request Jul 20, 2026

HDDS-15887. SCM should update datanode ports on re-registration or heartbeat when only ports change #10784

Closed

jojochuang requested review from jojochuang and szetszwo July 20, 2026 16:28

jojochuang requested a review from Copilot July 20, 2026 21:16

Copilot started reviewing on behalf of jojochuang July 20, 2026 21:17 View session

Copilot AI reviewed Jul 20, 2026

View reviewed changes

amaliujia self-requested a review July 21, 2026 08:36

amaliujia reviewed Jul 21, 2026

View reviewed changes

peterxcli self-requested a review July 21, 2026 09:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-12991. Support enabling Ratis DataStream on an existing cluster#10746

HDDS-12991. Support enabling Ratis DataStream on an existing cluster#10746
yandrey321 wants to merge 3 commits into
apache:masterfrom
yandrey321:HDDS-12991

yandrey321 commented Jul 13, 2026

Uh oh!

peterxcli commented Jul 14, 2026 •

edited

Loading

Uh oh!

yandrey321 commented Jul 15, 2026

Uh oh!

jojochuang commented Jul 20, 2026

Uh oh!

szetszwo commented Jul 20, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

amaliujia left a comment

Uh oh!

amaliujia Jul 21, 2026 •

edited

Loading

Uh oh!

yandrey321 commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Uh oh!

Conversation

yandrey321 commented Jul 13, 2026

What changes were proposed in this pull request?

Root cause

Changes

What is the link to the Apache JIRA

How was this patch tested?

Uh oh!

peterxcli commented Jul 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yandrey321 commented Jul 15, 2026

Uh oh!

jojochuang commented Jul 20, 2026

Uh oh!

szetszwo commented Jul 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

amaliujia left a comment

Choose a reason for hiding this comment

Uh oh!

amaliujia Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yandrey321 commented Jul 21, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

peterxcli commented Jul 14, 2026 •

edited

Loading

amaliujia Jul 21, 2026 •

edited

Loading