Compact data tables, and grapheme forward fast path

[cometkim]

Hello. It may be a bit sudden, but I'd like to share an experiment.

TL;DR: Ports the techniques from unicode-segmenter (a JS implementation originally derived from this crate, since heavily reworked) back to Rust. It results in 4-5x faster forward-only grapheme segmentation.

First, I want to thank the maintainers of this library.

I'm the maintainer of the JS library unicode-segmenter, which was initially based on this library which I manually ported from. There is an article that shares the story.

I optimized it to match the language characteristics and its own priority. Because JS clients care a lot about the code size.

On the initial port, I focused on data packing to improve the compression ratio. Simple caching or code generation is not allowed in JS because it inflates code size. I explored the grapheme area further to find inlining opportunities, and aside from that, it was a repetition of minor improvements.

At that time, even asking AI didn't make things better, but starting with Opus 4.8 and GLM 5.2, they began to discover real ideas. They discovered an effective way to encode not only data but also state, leading to significant improvements in both code size and performance.

Afterward, I wondered if this could be applied to the Rust side as well, and this PR is the result.

Changes

• Introduce the grapheme fast path (GraphemeFwd), UAX#29's pairwise rules are evaluated at compile time into a 256-byte mask table, and the context-sensitive rules (GB9c, GB11, GB12/13) become bits in a packed u8 carried forward, so a boundary is state & PAIR_MASK[before << 4 | after] == 0.
• grapheme tables packed as sorted u32 (start << bits | category), and tiered around known computable ranges.
• word/sentence/emoji tables also re-packed in the same way.
• Drop InCB_Extend_table and a binary search over 377 ranges for every codepoint. Now derived a simple inline function is_incb_extend.
• Flat category tables, nibble-packed. The category is now a direct index instead of a binary search.
• Inlined computable ranges in grapheme_category_raw. Only a 279-range residual reaches the binary search.

Result

Speed (benchmarked on Apple M4 Pro)

chars/grapheme/arabic   time:   [65.336 µs 65.774 µs 66.254 µs]
                        change: [-72.562% -72.368% -72.179%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 10 outliers among 100 measurements (10.00%)
  3 (3.00%) high mild
  7 (7.00%) high severe
chars/grapheme/english  time:   [78.618 µs 78.998 µs 79.450 µs]
                        change: [-77.827% -77.573% -77.333%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 18 outliers among 100 measurements (18.00%)
  2 (2.00%) high mild
  16 (16.00%) high severe
chars/grapheme/hindi    time:   [47.859 µs 48.094 µs 48.361 µs]
                        change: [-84.821% -84.706% -84.584%] (p = 0.00 < 0.05)
                        Performance has improved.
chars/grapheme/japanese time:   [48.550 µs 49.102 µs 49.633 µs]
                        change: [-80.129% -79.971% -79.822%] (p = 0.00 < 0.05)
                        Performance has improved.
chars/grapheme/korean   time:   [60.946 µs 61.383 µs 62.202 µs]
                        change: [-80.966% -80.870% -80.724%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 7 outliers among 100 measurements (7.00%)
  5 (5.00%) high mild
  2 (2.00%) high severe
chars/grapheme/mandarin time:   [41.361 µs 41.467 µs 41.633 µs]
                        change: [-80.327% -80.270% -80.206%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 11 outliers among 100 measurements (11.00%)
  2 (2.00%) high mild
  9 (9.00%) high severe
chars/grapheme/russian  time:   [66.964 µs 67.349 µs 67.781 µs]
                        change: [-73.146% -72.983% -72.831%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 8 outliers among 100 measurements (8.00%)
  3 (3.00%) high mild
  5 (5.00%) high severe
chars/grapheme/source_code
                        time:   [78.823 µs 79.148 µs 79.539 µs]
                        change: [-77.355% -77.192% -77.047%] (p = 0.00 < 0.05)
                        Performance has improved.

(No change detected in other benches)

And __TEXT,__const section in the rlib: 74,723 -> 39,507 bytes reduction

Regardless of the outcome, I fully understand that as a maintainer, you may not happy with questionable and massive changes.

I'm leaving this as a draft, but please feel free to let me know if you intend to merge it or if there is anything I can help with.

[Manishearth]

Hmm. I don't really feel comfortable reviewing such a large and monolithic change, even if the tests still pass. Small piecewise PRs might be fine.

[cometkim]

I totally understand.

The range inlining can be easily extracted. They are simple, clear improvements for reducing the data table and binary search range.

However, GraphemeFwd is better for runtime speed, but I think it is difficult to make it smaller because it is heavily coupled with the data compression method, which is the biggest change that's not split.

[cometkim]

I will break down the improvement steps into a few PRs. I expect that changes to codegen(unicode.py) will remain large.

[Manishearth]

I expect that changes to codegen(unicode.py) will remain large.

In that case I may not have the capacity to review this, sorry. FIne to still make the PR but it's unlikely I'll get around to it.

[cometkim]

One is #178, let me know if that's an acceptable size then.