Skip to content
Open
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions changelog.d/19418_gcp_pubsub_idle_reconnect.fix.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
The `gcp_pubsub` source no longer floods the logs with misleading errors on idle, low-volume, or bursty subscriptions. GCP routinely ends a streaming pull with a transient error (such as an `Unavailable` status when there are simply no messages to deliver), which Vector retries automatically. These transient failures are now logged at `debug`, and a `failed_fetching_events` component error is only emitted once several failures occur in a row without any successful fetch in between. The threshold is configurable via the new `max_retry_errors` option (default `5`); a successful fetch resets the counter.

The source is also more resilient to connections that silently break and stop delivering messages until Vector is restarted:

- HTTP/2 keepalive pings are sent on the gRPC connection so a dead-but-open connection is detected and torn down instead of hanging. The ping interval and timeout are configurable via the new `keepalive` option.
- The stream keepalive uses a non-blocking send, so a backed-up request channel can no longer deadlock the task.
- A new `idle_timeout_secs` option (defaulting to `900`) restarts the stream if no activity is seen on an active connection within that window.
Comment thread
SamyDjemai marked this conversation as resolved.
9 changes: 9 additions & 0 deletions lib/vector-config/src/external/serde_with.rs
Original file line number Diff line number Diff line change
Expand Up @@ -143,3 +143,12 @@ impl Configurable for Option<serde_with::DurationMilliSeconds<u64, serde_with::f
generate_optional_schema(&u64::as_configurable_ref(), generator)
}
}

impl Configurable for Option<serde_with::DurationSecondsWithFrac<f64, serde_with::formats::Strict>> {
fn generate_schema(generator: &RefCell<SchemaGenerator>) -> Result<SchemaObject, GenerateError>
where
Self: Sized,
{
generate_optional_schema(&f64::as_configurable_ref(), generator)
}
}
Loading
Loading