Skip to content

fix(tcp): retry messages after write failures - #905

Open
dlanov wants to merge 1 commit into
openconfig:mainfrom
dlanov:fix-904-retry-tcp-message
Open

fix(tcp): retry messages after write failures#905
dlanov wants to merge 1 commit into
openconfig:mainfrom
dlanov:fix-904-retry-tcp-message

Conversation

@dlanov

@dlanov dlanov commented Aug 3, 2026

Copy link
Copy Markdown

Summary

The TCP output worker currently removes a telemetry message from its buffer before writing it to the active connection. If that write fails, the worker reconnects but does not retain the dequeued message, causing the first message at the connection failure boundary to be dropped.

This change:

  • keeps the current framed payload pending until it is written successfully;
  • reconnects and retries the pending payload before dequeuing another message;
  • preserves per-worker message ordering across reconnects;
  • handles short writes by continuing until the complete payload is written;
  • makes TCP dialing and retry delays responsive to context cancellation; and
  • closes failed connections before reconnecting.

Delivery across an ambiguous TCP write failure is at-least-once. A complete payload may be retried if part of it was accepted by the failed connection, but the message is no longer silently lost.

Testing

  • go test ./pkg/outputs/tcp_output -run 'TestTCP_(WritePayloadHandlesShortWrites|RetriesMessageAfterWriteFailure)' -count=50
  • go test -race ./pkg/outputs/tcp_output -run 'TestTCP_(WritePayloadHandlesShortWrites|RetriesMessageAfterWriteFailure)' -count=10
  • CGO_ENABLED=0 ./tests/run_tests.sh
  • go vet ./...
  • staticcheck ./pkg/outputs/tcp_output

Fixes #904

Signed-off-by: Dennis Lanov <dennis.lanov@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The first telemetry message is dropped when the output type is TCP

1 participant