test_slot_migration_oom #8430

Activity

1. dragonclaw-dragonflydb

   added

   failing-test

   on Sep 30, 2026
2. Shikha-code36 commented on Oct 3, 2026

   

   Shikha-code36

   on Oct 3, 2026

   Contributor

   More actions

   I reproduced this on a Debug build. It's a stress-only failure, about 1 in 30 runs with CPU load on the machine.

   The abort is DCHECK_EQ(it.GetVersion(), snapshot_version_) in SerializerBase::SerializeBucket (src/server/serializer_base.cc:291 at 101a973). The log shows 39 vs. 0.

   Root cause as I read it:

   • ProcessBucket sets the bucket version to snapshot_version_ and then calls SerializeBucket.
   • SerializeBucket loops over the entries in that bucket. A large value can preempt the fiber inside SerializeEntry.
   • If the migration is cancelled during that preemption, DbSlice::UnregisterOnChange sets consumer->snapshot_version_ = 0.
   • When the loop resumes, the next entry's DCHECK compares the bucket version (39) with 0 and aborts.

   The DCHECK is compiled out in Release, so the effect there is serializing the rest of the bucket after cancel. In Debug it aborts, which is why CI sees it.

   I plan to add a guard in the SerializeBucket loop that stops once the context is cancelled, before the DCHECK, plus a test that cancels during a large-value bucket. Before I start, is anyone already working on this? If not, I'd like to claim it.
3. abhijat commented on Oct 6, 2026

   

   abhijat

   on Oct 6, 2026

   Member

   More actions

   @Shikha-code36 please feel free to send a PR

   Iit might make more sense to add your check or guard after SerializeEntry returns though, at the end of the loop. If there is a yield it will only happen in SerializeEntry so a check after it signals intent better and avoids an extra check the first time.
4. Shikha-code36 commented on Oct 7, 2026

   

   Shikha-code36

   on Oct 7, 2026

   Contributor

   More actions

   @Shikha-code36 please feel free to send a PR

   Iit might make more sense to add your check or guard after SerializeEntry returns though, at the end of the loop. If there is a yield it will only happen in SerializeEntry so a check after it signals intent better and avoids an extra check the first time.

   I stress-tested this on a Debug build (CPU-saturated machine, 30 runs in a loop) and reproduced the signal 6 abort on commit 101a9736: DCHECK_EQ(it.GetVersion(), snapshot_version_) in SerializerBase::SerializeBucket (serializer_base.cc:291), firing as 39 vs. 0.

   Root cause: a large value can preempt the traversal fiber inside SerializeEntry. If cancellation lands during that preemption, UnregisterOnChange used to reset snapshot_version_ to 0 while the bucket loop was still mid-flight, so the next entry's DCHECK failed.

   I re-ran the same 30-iteration stress loop on current main and it passed cleanly. #8473 ("SlotMigrationStreamer::Cancel() waits for Run finish", merged 2026-10-05) makes Cancel() wait on traversal_latch_ before unregistering the listener, which closes exactly this race.

   Looks like this can be closed as fixed by #8473.

   👍React with 👍1Reacted by Borys
5. BorysTheDev

   closed this as completedon Oct 7, 2026