Master branch code since #9846, when used for a full index rewrite, builds a complete, sorted index key list as a python list. That can eat a lot of memory for big repositories.
The whole point is to partition the index into smaller batches in a reproducible way that only depends on index contents.
Idea:
Index keys are uniformly distributed (cryptographic hash digests), so if we want 2^N batches rather than a single big batch, we could also partition by N bits of key prefix.
After selecting the keys for the current batch, we can sort this smaller list of keys.
N can be determined by looking at the amount of index entries that shall be written, targetting reasonable batch sizes of max. 32MB.
Master branch code since #9846, when used for a full index rewrite, builds a complete, sorted index key list as a python list. That can eat a lot of memory for big repositories.
The whole point is to partition the index into smaller batches in a reproducible way that only depends on index contents.
Idea:
Index keys are uniformly distributed (cryptographic hash digests), so if we want 2^N batches rather than a single big batch, we could also partition by N bits of key prefix.
After selecting the keys for the current batch, we can sort this smaller list of keys.
N can be determined by looking at the amount of index entries that shall be written, targetting reasonable batch sizes of max. 32MB.