Skip to content

lib/logstorage: skip unsafe last-N optimization when pipes may overwrite _time - #1549

Open
immanuwell wants to merge 1 commit into
VictoriaMetrics:masterfrom
immanuwell:fix/lastn-unsafe-time-overwrite
Open

lib/logstorage: skip unsafe last-N optimization when pipes may overwrite _time#1549
immanuwell wants to merge 1 commit into
VictoriaMetrics:masterfrom
immanuwell:fix/lastn-unsafe-time-overwrite

Conversation

@immanuwell

Copy link
Copy Markdown
Contributor

This fixes a kinda nasty last-N edge case.

If unpack_json, unpack_logfmt, extract, or extract_regexp can overwrite _time, a wide enough time range may hit the last-N fast path and fail with missing _time field in the query results. A narrow range works, so it is pretty easy to miss.

Fix is simple: skip that optimization for those unsafe cases. Safe cases with result_prefix still keep the fast path.

Repro

  1. Insert a row like {"_msg":"{\"_time\":\"not-a-timestamp\",\"msg\":\"bad\"}","_time":"2025-01-01T01:00:00Z"}
  2. Run * | unpack_json | keep _msg, _time | sort by (_time desc) limit 1 on 2025-01-01T01:00:00Z..2025-01-01T01:00:00.000000001Z
  3. Run the same query on 2025-01-01T01:00:00Z..2025-01-01T01:00:03Z
  4. Before this fix, step 3 returns 400. After this fix, both ranges work.

Tests:

  • go test ./lib/logstorage
  • go test ./apptest/tests -run "TestVlsingleLastnOptimization$|TestVlsingleLastnOptimizationWithUnpackedInvalidTime$" -count=1

Related to #1360.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All reported issues were addressed across 6 files

Reply with feedback, questions, or to request a fix.

Re-trigger cubic

Comment thread lib/logstorage/pipe_unpack_logfmt.go Outdated
@immanuwell
immanuwell force-pushed the fix/lastn-unsafe-time-overwrite branch from 4db0b3d to ee2f3bc Compare July 12, 2026 07:34
@immanuwell
immanuwell force-pushed the fix/lastn-unsafe-time-overwrite branch from ee2f3bc to 669648e Compare July 12, 2026 07:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants