RRDCached backend: good enough (fixes, real tests, documented differences)

[fungiboletus]

Why

RRDCached is a side-track backend, but "keep Prometheus' data in RRDtool" is a story worth keeping true. Probing a real rrdcached found that it was not: see done/rrdcached-good-enough.md for the audit table.

What was wrong, and is fixed

• Samples more than 20 s apart were lost (data source heartbeat of 20 s): five samples a minute apart returned one point. Now an hour, ?heartbeat= sets it.
• A restart erased the history: CREATE replaces an existing file and SensApp only remembered created files in memory (same for a second instance or two concurrent requests). Files are created with no_overwrite (rrdcached-client 0.4, -O on each CREATE): the daemon refuses to replace one that exists, whatever options it was started with, and "File exists" is accepted.
• A retried or out-of-order write failed with a 500 (Prometheus then retried forever): batches are sorted, one sample per second, and the daemon's refusal of times that are not after the last update is not a failed write.
• A request cancelled by the HTTP timeout left its answer on the shared connection for the next request: the connection is dropped instead; 15 s operation timeout; an unreachable daemon is a 503.
• FLUSHALL on every write, read and health check is gone (health check is a PING).
• Reads: first row of a window, end alone, backwards windows, errors reported as errors (not "not found"), limit, empty vs missing series, and the freshest rows of long windows (a coarse archive does not have them yet).
• Listing: sorted so the bookmark pagination works, only files named like a series (others got a random UUID per call), no fake fallback when the daemon fails.
• Selectors read each series once instead of twice.
• rrdcached+unix:// was never routed by the storage factory ("Unsupported storage type").

Documentation

docs/RRDCACHED.md is rewritten: what is different from the other backends (a table), what it means for Prometheus (remote write works, remote read only by UUID), presets and file sizes, how time, gaps and consolidation behave, how reads pick an archive, operating the daemon (-O), performance. BACKENDS.md and CONFIGURATION.md follow.

Tests

• 54 unit tests with a scripted daemon (cancellation, concurrent creation, stale refusals, listing, selectors, fresh-tail search...).
• 26 integration tests against a real daemon (the 12 old ones mostly printed): 20 of the first 24 fail on the previous code. Includes Prometheus remote write sent twice then read through the HTTP API.
• Passing against rrdcached 1.11 (the CI image, built from the upstream release) and 1.7.2 (Debian's package); Unix socket checked by hand with 1.11.

Performance (debug build, daemon in Docker, 2 runs each)

	before	after
Bulk load	190k samples/s	188-191k
Small write (50 series)	1.95-2.09 ms	1.28-1.35 ms
Scrape of 1000 series	8.0-9.3 ms	7.7-8.0 ms
First write of 1000 series	0.72-0.82 s	0.72-1.0 s
Reads	same	same
list_series	74-96 ms	35-49 ms

Notes

• Files created by older versions keep their 20 s heartbeat; rrdtool tune <file> --heartbeat sensapp:3600 fixes them (in the docs).
• Left out on purpose (ideas/rrdcached-follow-ups.md): a metadata sidecar, a connection pool, and using no_overwrite of rrdcached-client once released (a local, unpushed branch create-no-overwrite of that crate adds -O to CREATE, which would remove the need for the daemon's -O).

🤖 Generated with Claude Code

Update

• Uses rrdcached-client 0.4.0 (no_overwrite). No existence check before a creation, no lock, and the daemon does not need -O: the 26 integration tests pass on a daemon started without it.
• The test image is a plain copy of the one of rrdcached-client: latest upstream rrdcached (1.11.0) built from source, non-root. CI builds it with buildx and a layer cache; the compose service and its health check follow.

🤖 Generated with Claude Code