Repository navigation
Conversation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Generated-by: Claude Code (Opus 5.5)
What changes were proposed in this pull request?
When a read or write of container data or metadata hits an I/O error, the datanode only schedules an on-demand volume check. That check writes a small file, syncs it and reads it back, so it lands on new sectors and the read is likely served from page cache. It keeps passing on a disk that fails reads of existing data, and such a disk only shows up as containers being marked UNHEALTHY one at a time, with nothing tying the errors to the same volume.
HDDS-11943 proposed failing the volume after enough of these errors and was closed in favour of the degraded volume design in #8405. Its "IO Error Count" check needs these errors counted per volume first. This PR counts and reports them, without changing the volume state:
DiskCheckUtil#isDiskIOError. It matches EIO, EROFS, ENXIO/ENODEV, EUCLEAN and EREMOTEIO messages, andSyncFailedException, anywhere in the cause chain. Missing files, ENOSPC, EMFILE and permission errors are not counted.StorageVolumeUtil#onFailure(StorageVolume, Throwable)and pass the exception from the existing data path call sites:ChunkUtilsread/write,FilePerBlockStrategy,StreamDataChannelBase,KeyValueContainer,BlockUtilsandHddsVolumeUtil.NumDataPathIOErrorscounter toVolumeInfoMetrics, plus a WARN log naming the volume, logged at most once perhdds.datanode.disk.check.min.gap.A later change can feed this count into the degraded sliding window from the design.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-17261
How was this patch tested?
New unit tests for the error classification (
TestDiskCheckUtil), the per volume count throughStorageVolumeUtil#onFailure(TestHddsVolume), and recording from the container data scanner (TestBackgroundContainerDataScanner).🤖 Generated with Claude Code