Skip to content

HDDS-17261. Count disk I/O errors from the datanode data path per volume - #11668

Draft
smengcl wants to merge 1 commit into
apache:masterfrom
smengcl:HDDS-17261
Draft

smengcl wants to merge 1 commit into
apache:masterfrom
smengcl:HDDS-17261

Conversation

@smengcl

@smengcl smengcl commented Oct 21, 2026

Copy link
Copy Markdown
Contributor

Generated-by: Claude Code (Opus 5.5)

What changes were proposed in this pull request?

When a read or write of container data or metadata hits an I/O error, the datanode only schedules an on-demand volume check. That check writes a small file, syncs it and reads it back, so it lands on new sectors and the read is likely served from page cache. It keeps passing on a disk that fails reads of existing data, and such a disk only shows up as containers being marked UNHEALTHY one at a time, with nothing tying the errors to the same volume.

HDDS-11943 proposed failing the volume after enough of these errors and was closed in favour of the degraded volume design in #8405. Its "IO Error Count" check needs these errors counted per volume first. This PR counts and reports them, without changing the volume state:

  • Add DiskCheckUtil#isDiskIOError. It matches EIO, EROFS, ENXIO/ENODEV, EUCLEAN and EREMOTEIO messages, and SyncFailedException, anywhere in the cause chain. Missing files, ENOSPC, EMFILE and permission errors are not counted.
  • Add StorageVolumeUtil#onFailure(StorageVolume, Throwable) and pass the exception from the existing data path call sites: ChunkUtils read/write, FilePerBlockStrategy, StreamDataChannelBase, KeyValueContainer, BlockUtils and HddsVolumeUtil.
  • Record disk I/O errors found by the container scanners as well.
  • Add a NumDataPathIOErrors counter to VolumeInfoMetrics, plus a WARN log naming the volume, logged at most once per hdds.datanode.disk.check.min.gap.

A later change can feed this count into the degraded sliding window from the design.

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-17261

How was this patch tested?

New unit tests for the error classification (TestDiskCheckUtil), the per volume count through StorageVolumeUtil#onFailure (TestHddsVolume), and recording from the container data scanner (TestBackgroundContainerDataScanner).

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@smengcl smengcl added the AI-gen label Oct 21, 2026
Copilot AI lite review requested due to automatic review settings October 21, 2026 18:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants