HDDS-12991 failback to non-streaming write when pipeline does not have streaming port info by yandrey321 · Pull Request #10828 · apache/ozone

yandrey321 · 2026-07-21T16:11:01Z

What changes were proposed in this pull request?

HDDS-12991 Gracefully enable Ratis DataStream on a running cluster with pre-existing pipelines

When Ratis DataStream is turned on for an already-running cluster, datanodes and pipelines created before the feature was enabled do not expose the RATIS_DATASTREAM port. A streaming-enabled client attempting to write to such a pipeline previously failed. This change makes that transition graceful — a streaming client transparently falls back to the non-streaming write path when the target pipeline can't stream:

BlockDataStreamOutput now throws a typed StreamNotSupportedException (new) instead of a generic IOException when a pipeline node lacks the RATIS_DATASTREAM port.
KeyDataStreamOutput surfaces this exception as-is (after closing the stream) so callers can distinguish "pipeline can't stream" from other I/O errors.
SelectorOutputStream gains an optional fallback selector: on the first flush, if the selected streaming output throws StreamNotSupportedException, it closes that output and replays the buffered prefix + payload to a non-streaming fallback stream. Once the first flush succeeds, the buffer is released and subsequent writes go straight to the selected stream (no behavior change when streaming works or when no fallback is configured).
-BasicOzoneFileSystem / BasicRootedOzoneFileSystem wire the non-streaming path (createFSOutputStream(adapter.createFile(...))) as that fallback.

Co-authored with Cloude Code

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-12991

How was this patch tested?

Unit tests

TestSelectorOutputStream (12 tests) — covers the fallback-on-first-flush path, replay of buffered data, failing-close during fallback, and the no-fallback-configured (propagate) path.
TestSCMNodeManager — testRegisterRefreshesPortsOnPortChange (ports changed → node record updated) and testRegisterKeepsPortsWhenUnchanged (ports unchanged → no update).

Integration test

TestOzoneFileSystemWithStreamingDisabledDatanode — brings up a cluster with RATIS_DATASTREAM disabled on the datanodes but streaming enabled on the client (auto-threshold 1B), then asserts a write falls back to non-streaming and round-trips correctly over both o3fs and ofs.

…, in case if pipeline does not have streaming port info failback to non-streaming write

szetszwo · 2026-07-21T21:22:53Z

This change makes that transition graceful — a streaming client transparently falls back to the non-streaming write path when the target pipeline can't stream ...

Let's just close all the non-stream pipelines and don't fall back. It makes the code really complicated.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12991
https://issues.apache.org/jira/browse/HDDS-15799
https://issues.apache.org/jira/browse/HDDS-8687

This section is supposed to put only one JIRA -- the JIRA corresponding to this PR.

yandrey321 · 2026-07-21T22:52:29Z

This change makes that transition graceful — a streaming client transparently falls back to the non-streaming write path when the target pipeline can't stream ...

Let's just close all the non-stream pipelines and don't fall back. It makes the code really complicated.

This solution creates a race condition, it fails the integration test in mini Ozone cluster. Changes in this PR allows graceful fail back to non-stream writes.

What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-12991
https://issues.apache.org/jira/browse/HDDS-15799
https://issues.apache.org/jira/browse/HDDS-8687

This section is supposed to put only one JIRA -- the JIRA corresponding to this PR.

Fixed.

jojochuang · 2026-07-21T22:59:53Z

        }
        len -= writtenLength;
        off += writtenLength;
+      } catch (StreamNotSupportedException e) {


Looks like StreamNotSupportedException would be caught by handleException() instead, it would be caught here after several retry attempts. That would be fine functionally but delays connection in this case.

good catch! Updated the fix to address the comment.

szetszwo · 2026-07-22T00:28:56Z

This solution creates a race condition, ...

What is the race condition?

... Changes in this PR allows graceful fail back to non-stream writes.

The change in this PR is very complicated. It does not worth to implement fallback for just changing the conf from disable streaming to enable streaming.

yandrey321 · 2026-07-22T16:49:45Z

This solution creates a race condition, ...

What is the race condition?

... Changes in this PR allows graceful fail back to non-stream writes.

The change in this PR is very complicated. It does not worth to implement fallback for just changing the conf from disable streaming to enable streaming.

here is the part for re-creating pipelines when streaming is enabled: #10842

without client side fix (changes in the current PR) the part 2 fails with the following error on writing files after enabling the streaming:

mvn -pl hadoop-ozone/integration-test test \
  -Dtest='TestOzoneFileSystemDataStreamEnablement#testBatchWritesAcrossStreamingEnablement' \
  -DskipShade -DskipRecon \
  -Ddevelocity.cache.local.enabled=false -Ddevelocity.cache.remote.enabled=false -Dscan=false

Tests run: 1, Failures: 0, Errors: 1  <<< FAILURE!

java.io.IOException: RATIS_DATASTREAM port is missing for datanode <dn>
  in pipeline <id>; datastream is disabled for this pipeline
    at org.apache.hadoop.ozone.client.io.KeyDataStreamOutput.handleWrite(KeyDataStreamOutput.java:212)
    ...
Caused by: java.io.IOException: RATIS_DATASTREAM port is missing ...
    at org.apache.hadoop.hdds.scm.storage.BlockDataStreamOutput.setupStream(BlockDataStreamOutput.java:200)
    at org.apache.hadoop.hdds.scm.storage.BlockDataStreamOutput.<init>(BlockDataStreamOutput.java:173)

chungen0126

+1 LGTM

chungen0126 · 2026-07-23T03:05:35Z

This solution creates a race condition, ...

What is the race condition?

... Changes in this PR allows graceful fail back to non-stream writes.

The change in this PR is very complicated. It does not worth to implement fallback for just changing the conf from disable streaming to enable streaming.

Thanks @yandrey321 for working on this.

I believe this fallback is necessary for now. Here is my understanding of the problem:

Imagine a scenario where the cluster inevitably contains a mix of DataNodes that support Ratis streaming and those that do not. This ultimately leads to two issues:

When creating a pipeline, SCM pipeline manager treats DataNodes regardless of whether they support Ratis streaming as identical, meaning we always run the risk of creating a pipeline that does not support Ratis streaming.
When selecting a pipeline, there is no awareness of whether the DataNodes within that pipeline support Ratis streaming. As a result, we might still end up selecting a non-streaming pipeline and failing during writes.
Even if we have enough DataNodes supporting Ratis streaming, the above two reasons will ultimately force us to fall back to non-streaming writes.

My perspective:

I view this as a short cup to keep writes functioning properly, while we still need to close and update the pipeline states for those whose DataNode port information is outdated.

However, I think we should create a ticket to ensure that either pipeline creation or pipeline selection is aware of the write mode. This way, the client can accurately target pipelines that support Ratis streaming.

@yandrey321 , @szetszwo , what are your thoughts on this?

Graceful handling of enabling ratis streaming with existing pipelines…

120cc34

…, in case if pipeline does not have streaming port info failback to non-streaming write

yandrey321 mentioned this pull request Jul 21, 2026

HDDS-12991. Support enabling Ratis DataStream on an existing cluster #10746

Open

jojochuang reviewed Jul 21, 2026

View reviewed changes

Avoid retry when handling StreamNotSupportedException exception

39b77d1

yandrey321 mentioned this pull request Jul 22, 2026

HDDS-12991 recreate pipelines after enabling ratis write streaming #10842

Open

chungen0126 approved these changes Jul 23, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

HDDS-12991 failback to non-streaming write when pipeline does not have streaming port info #10828

HDDS-12991 failback to non-streaming write when pipeline does not have streaming port info #10828
yandrey321 wants to merge 2 commits into
apache:masterfrom
yandrey321:HDDS-12991-part-1

yandrey321 commented Jul 21, 2026 •

edited

Loading

Uh oh!

szetszwo commented Jul 21, 2026

Uh oh!

yandrey321 commented Jul 21, 2026

Uh oh!

jojochuang Jul 21, 2026

Uh oh!

yandrey321 Jul 21, 2026

Uh oh!

szetszwo commented Jul 22, 2026

Uh oh!

yandrey321 commented Jul 22, 2026

Uh oh!

chungen0126 left a comment

Uh oh!

chungen0126 commented Jul 23, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

yandrey321 commented Jul 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

What is the link to the Apache JIRA

How was this patch tested?

Unit tests

Integration test

Uh oh!

szetszwo commented Jul 21, 2026

Uh oh!

yandrey321 commented Jul 21, 2026

Uh oh!

jojochuang Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

yandrey321 Jul 21, 2026

Choose a reason for hiding this comment

Uh oh!

szetszwo commented Jul 22, 2026

Uh oh!

yandrey321 commented Jul 22, 2026

Uh oh!

chungen0126 left a comment

Choose a reason for hiding this comment

Uh oh!

chungen0126 commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yandrey321 commented Jul 21, 2026 •

edited

Loading

chungen0126 commented Jul 23, 2026 •

edited

Loading