FatFs bridges: build a cluster link map on open

[rtissera]

Without a cluster link map, every backward f_lseek walks the FAT chain again from the file's first cluster — and every COMPRESSION_SELF hunk is a backward seek, to an earlier hunk the image references instead of storing twice. The cost grows with how far into the file the reader is, so a short test near the start never shows it.

On an ESP32-P4 a 29%-self-referenced image did not finish in 47 minutes until the benchmark built a map (070341a). The BL616 bridge that an integration copies never got that fix, and TangCore's FatFs is built with FF_USE_FASTSEEK 1 — the capability is there, nothing was using it.

What changes

chd_fatfs_open() builds the map right after f_open, before chd_open, so the header and map reads benefit too. The table needs two words per fragment plus one: it starts at 16 words, grows to exactly what FatFs reports it needs, and past CHD_FATFS_CLMT_MAX runs without a map rather than failing — slower backward seeks, still correct. Compiled out when FF_USE_FASTSEEK is 0.

The close callback frees the table. Since #195, libchdr calls it on a failed open too, so the bridge's error path has nothing left to close.

The RP2350 benchmark builds the same map behind BENCH_LINKMAP, so it can be measured with and without.

Measured

On an ESP32-C6: FatFs over SPI, no SDMMC host, a single RISC-V core, which is the same storage shape as the FatFs firmware targets. The benchmark compiled this PR's chd_fatfs.c as is, with ESP-IDF's FF_USE_FASTSEEK on and off, everything else identical. Every hunk was CRC-checked and read back correctly in both runs.

read	map	seek	total
first 6,200 hunks of a 259 MB CD image	built	58 ms	8,353 ms
	off	191 ms	8,546 ms
all of a 27 MB image, one fragment	built	180 ms	52,534 ms
	off	526 ms	52,856 ms
same image copied into 52 fragments	grown to 106 words	180 ms	52,359 ms
	off	559 ms	52,764 ms
same image copied into 822 fragments	none, past CHD_FATFS_CLMT_MAX	857 ms	53,059 ms
	off	790 ms	52,958 ms

The map makes seeks about 3x cheaper, and the 52-fragment copy seeks as fast as the contiguous file. End to end that is 1-2% on these reads: storage is a quarter to two fifths of the time, and seeks are a small part of storage. The 822-fragment copy needs more than 1,024 words, so the bridge runs without a map and behaves like the build without one.

The seeks here are not self-references. The benchmark builds with LOWRAM_TARGET, which reads the v5 map lazily, a window at a time, and the large image's map sits at 99.9% of the file. Every window fetch is a seek to the end and the next hunk read a seek back to the start, a full chain walk each way. So the map pays on any image under LOWRAM_TARGET, self-referenced or not.

The fragmented copies were written on the card itself, one cluster at a time, with a cluster of a filler file after every 16th (or every) cluster, and the filler deleted afterwards.

On an RP2350 (a second FatFs stack, carlk3's no-OS-FatFS-SD-SDIO-SPI-RPi-Pico, SD over SPI at 25MHz), the same map-on/off switch against three real PC Engine CD images already on the card, freshly written so each is one fragment near the start:

image	hunks	map	no map
Insanity	11,825	121,699 ms	121,611 ms
Pyramid Plunder	125	1,319.2 ms	1,319.6 ms
Hawaiian Island Girls	147	2,747.3 ms	2,747.9 ms

No measurable difference either way — within 0.1% noise. These files never trigger the pathology this PR fixes: single fragment, map near the front of the file, nothing forces a chain walk. The C6 numbers above show the same map matters once the map is far into the file (LOWRAM_TARGET) or the file is fragmented; on a corpus without either condition, building it is free but buys nothing.

Checked

• Compiles cleanly against TangCore's actual FatFs (bouffalo_sdk/components/fs/fatfs, R0.15) with FF_USE_FASTSEEK at 1 and at 0.
• RP2350 benchmark builds both ways; the switch is real — +128 bytes of text and three extra f_lseek call sites with it on. Measured on-hardware above.
• Table growth (FR_NOT_ENOUGH_CORE, then a resize) and the fall-back past CHD_FATFS_CLMT_MAX, on real fragmented files as above.

Not covered

• Not run on a BL616 itself; the ESP32-C6 above has the same FatFs-over-SPI shape.

🤖 Generated with Claude Code

[rtissera]

Tested this on real BL616 hardware (Sipeed Tang Console 60K, TangCore firmware) and it
breaks chd_fatfs_open() on one image while another is fine, so flagging before merge.

Symptom

image	size	result
Dungeon Explorer II (USA).chd	371 MB	chd_fatfs_open → err=14 CHDERR_DECOMPRESSION_ERROR
Prince of Persia (USA).chd	244 MB	err=0, opens and reads fine

DE2 opened cleanly on every build before this change.

Bisect

libchdr src/+include/ held constant at master 8e7b8bd. The only variable was
contrib/tangcore-bl616/chd/chd_fatfs.c:

• from 9fcd4bc (this PR) → DE2 err=14
• from master (no CLMT) → DE2 err=0

So the cluster link map is the cause.

Why it looks like wrong seeks

A decompression error at open means the huffman-compressed v5 hunk map decoded from the
wrong bytes — i.e. f_lseek landed somewhere unintended once the CLMT was installed. With
LOWRAM_TARGET=1 the map is read in windows at open (42 seeks / ~156 KB on this image),
so a bad map shows up immediately rather than later during chd_read.

Environment, since it is probably fragmentation-dependent

• Card is exFAT, 128 KB clusters, 94% full
• FF_USE_FASTSEEK 1, FF_FS_EXFAT 1, FF_MAX_SS 512, FF_FS_TINY 0
• DE2 371 MB vs PoP 244 MB — consistent with the larger file being more fragmented
• filefrag does not work on exFAT here, so I could not measure extent counts directly

What I could not determine

I read the builder and FatFs's side carefully and could not fault the logic:

• the size negotiation looks right — required items are 2 + 2*nfrag, clmt[0] is set to
  the table size in items, and realloc(need) matches what FatFs reports
• both fallbacks look right — clmt[0] > CHD_FATFS_CLMT_MAX, and a failed realloc, both
  leave fr != FR_OK so cltbl is cleared and the buffer freed
• FatFs's get_fat() exFAT paths terminate correctly for a contiguous file
  (stat == 2 synthesises 0x7FFFFFFF at the last cluster) and read the FAT for a
  fragmented one, and clmt_clust() is pure table arithmetic

So this is an empirical result, not a diagnosis. Two things I would look at first:

1. Whether CREATE_LINKMAP is sound on exFAT specifically — a contiguous exFAT file
   has NoFatChain set and no FAT chain at all, and the builder walks it via get_fat().
2. Whether exceeding CHD_FATFS_CLMT_MAX (1024 words ≈ 511 fragments) on a heavily
   fragmented 371 MB file can leave a partially-populated table installed rather than
   falling back cleanly.

Happy to run more instrumented builds on this hardware if it would help — I can log
f_lseek's FRESULT, the negotiated clmt[0], and the resulting fragment count from
inside chd_fatfs_map_clusters().