[cheeeee]

Follow-up to #23445, under the same ticket (resolved, fix version
4.5.0).

Since #23445, a failed send of fetched records is passed to a handler
that re-reads the region on the scheduler, and the log directory is
offlined only if the re-read fails too. The re-read is needed because
over PLAINTEXT sendfile reads the file and writes the socket in one
call, so a disk error and a socket error are the same bare
IOException. SslTransportLayer.transferFrom does not have that
problem: it reads the file with FileChannel.read, then encrypts and
writes, so it knows which of the two failed. As discussed on #23445,
this passes that on instead of finding it out again:

• SslTransportLayer.transferFrom throws a new FileReadException, an
  IOException with the position of the read, when FileChannel.read
  or FileChannel.size fails. A ClosedChannelException, from a
  deletion or a shutdown, is rethrown as is.
• TransferableChannel.reportsFileReadFailures() says whether a
  transport does that: true for TLS, false, the default, for
  PLAINTEXT.
• FileRecords.writeTo passes a FileReadException to the handler,
  and any other failure only if the transport cannot tell the two
  apart. A failure of its own channel.size() call before the transfer
  becomes a FileReadException as well.
• LocalLog reports a FileReadException as a failure of the log
  directory without re-reading the region, on the scheduler and at most
  one report at a time per log directory, as for the re-read. Reports
  are tracked apart from re-reads, so that a re-read running in the
  same directory does not drop a known failure. Other failures are
  re-read as before.

Consequences worth stating:

• On TLS listeners a disconnect no longer re-reads the region that was
  being sent. In the test notes, consumers killed mid-fetch over TLS
  for 5 minutes made the scheduler re-read 151 MB without this change
  and nothing with it.
• On TLS listeners a read error now offlines the log directory even if
  the region could be read again by the time of a re-read, e.g. after a
  transient media error. Before, such an error was attributed to the
  socket. This is the criterion LocalLog applies to any other
  IOException.
• A failed FileChannel.size() call on the segment, over either
  transport, is now reported as a read failure without a re-read.
• PLAINTEXT listeners still re-read: sendfile cannot tell the two
  apart.
• KRaft snapshots, sent through UnalignedFileRecords, are still not
  covered.

Generated-by: Claude (Anthropic)

[cheeeee]

Test notes for 73fc46a, on trunk d18f93f.

Unit tests:

• SslTransportLayerTest, FileRecordsTest and LocalLogTest pass,
  as do :clients:test (13757 tests), :storage:test (1355 tests),
  checkstyle and spotless. 2 of 5 runs of :storage:test failed in
  TransactionsWithTieredStoreTest (a timeout, and the test JVM
  exiting), which passes alone 3 times out of 3 here and on trunk. On
  trunk 2 of 3 runs failed in tiered storage integration tests too
  (TransactionsTest, RemoteTopicCrudTest); see KAFKA-15772.
• Each new or changed test fails when the part of the change it covers
  is undone. 15 such mutations, e.g. transferFrom wrapping a closed
  file or a failed socket write, reportsFileReadFailures() returning
  false for TLS, a report sharing the re-read's per-directory slot, or
  a failed task not releasing it: each makes the test named for it
  fail.

Cluster: 3 brokers and a controller (KRaft) in Docker on a Linux 6.8
VM. Broker 1 has two log directories, one of them on a dm-dust
device. Partition t1-0 has replicas 1, 2 and 3, broker 1 leads it, and
its replica on broker 1 is in the dm-dust directory. 20000 records of
512 bytes are produced, then a console consumer reads the partition
from the beginning with an 8 MiB fetch size, over PLAINTEXT or TLS.
strace -yy on broker 1 shows which thread reads which file at which
position.

1. Persistent media error: dm-dust makes reads of the tail of the
   segment fail on the device. In every run the first read to fail is
   at position 2867200.
   • TLS, trunk: the network thread fails to read position 2867200.
     The scheduler re-reads the region from position 0, 2867200 bytes,
     and fails at 2867200. The directory goes offline.
   • TLS, this change: the same failed read on the network thread, and
     no read of the segment on the scheduler. The directory goes
     offline, with "Failed to read .../00000000000000000000.log at
     position 2867200 while writing 8388608 bytes from position 0 to a
     network channel".
   • PLAINTEXT, both: sendfile fails, the re-read on the scheduler
     fails at the same position, and the directory goes offline.
   • In all of them the leader moves to broker 2 and the consumer reads
     all 20000 records.
2. Error gone on the next read: strace makes the first read of the
   segment on each network thread of the listener fail with EIO, and
   the following reads succeed.
   • TLS, trunk: 3 failed reads, one on each network thread as the
     consumer retried, each followed by a successful re-read of 8 MiB
     on the scheduler. The directory stays online.
   • TLS, this change: the first failed read offlines the directory,
     without a re-read. The consumer reads all 20000 records from
     broker 2.
   • PLAINTEXT, both: as TLS on trunk, the directory stays online.
3. Disconnects, with a healthy disk: 3 TLS consumers read 4 partitions
   led by broker 1, about 200 MB, and are killed after 2 to 8 seconds
   and restarted, for 5 minutes, about 180 times in all.
   • trunk: 4629 reads of segment files on the scheduler, 151 MB.
   • this change: none.
   • No directory went offline in either.

With this change the failed send is still logged as a disconnect, at
DEBUG, now with the FileReadException, and the ERROR comes from
LogDirFailureChannel. The timings depend on the host, a VM under
load, so I have left them out.

[cheeeee]

Test notes, continued: the same on macOS 26.7 (Apple Silicon, JDK 17),
with the brokers as local processes, on 73fc46a and on trunk
d18f93f.

• Persistent media error: broker 1's log directory is on a disk image,
  and a 1 MiB band of the image in the middle of the segment is made
  unreadable, so the kernel fails reads of it with EIO. macOS ejects
  the image right after, so every later read fails too. Over TLS,
  trunk re-reads the region on the scheduler, which fails; this change
  offlines the directory without a re-read. Over PLAINTEXT both
  re-read, and the re-read fails. The directory goes offline in all
  four runs, and the consumer reads all 20000 records from broker 2.
• Error gone on the next read, injected by an interposer on pread
  and sendfile (DYLD_INSERT_LIBRARIES, the macOS stand-in for
  strace -e inject): as on Linux, trunk re-reads 8 MiB after each
  failure and keeps the directory online; over TLS this change
  offlines it; over PLAINTEXT both keep it online.
• Disconnects, 3 TLS consumers killed mid-fetch for 5 minutes, about
  180 times: trunk 19497 reads of segment files on the scheduler,
  636 MB; this change none.

On macOS a read fails before the unreadable bytes: with the band at
positions 3129344 to 4177919, pread failed at 2129920, and
sendfile at 2736128 after reporting progress up to that page. So
FileReadException.position() is where the failed read started,
which is not necessarily where the medium failed.

[cheeeee]

Test notes, continued: OpenBSD 8.0 (arm64, QEMU on macOS) and Windows
10 LTSC (x86-64, QEMU/KVM), both with JDK 17, on 73fc46a and on
trunk d18f93f. Each VM runs one broker and controller, with t1-0 on
a data disk behind QEMU's blkdebug, which fails reads of the disk
sector holding position 4 MiB of the segment with EIO: every read, or
only the first one. A consumer reads the partition from the beginning.

• TLS, every read fails: on both, the network thread's read fails at
  position 4194304. Trunk re-reads the region on the scheduler (3.0 MB
  on OpenBSD, 3.6 MB on Windows), the re-read fails too, and the
  directory goes offline; this change offlines it without a re-read.
• TLS, only the first read fails: on OpenBSD trunk re-reads the region
  successfully and keeps the directory online, while this change
  offlines it. On Windows the disk driver retries the failed read
  (System event 153) and the retry succeeds, so the JVM never sees an
  error, and both keep the directory online.
• PLAINTEXT, every read fails: the JDK writes the file to the socket
  from a memory mapping on both, and the write fails with EFAULT ("Bad
  address") on OpenBSD and WSAEINVAL ("An invalid argument was
  supplied") on Windows rather than with an I/O error. Trunk and this
  change both re-read the region, the re-read fails, and the directory
  goes offline.

No InternalError was thrown in any run.

[cheeeee]

Windows is on a best-effort basis today. Too different memory management/disk read error handling by the NT kernel and surrounding libraries.

I will probably take a deeper look after this PR. Maybe, we can bring some better support for Windows.