Repository navigation
Replace the zstd C library with a port to safe Rust - #47079
Open
dylan-conway wants to merge 224 commits into
Open
dylan-conway wants to merge 224 commits into
dylan-conway wants to merge 224 commits into
Conversation
The build no longer fetches, patches, compiles or links facebook/zstd. Nothing replaces it yet, so the build fails: bun_icu_decompress.cpp and ProcessBindingConstants.cpp include zstd.h, BunProcess.cpp reads BUN_VERSION_ZSTD_HASH, and bun_zstd declares zstd's symbols.
bun_zstd no longer declares zstd's C symbols. It now holds a port of the decoder of zstd 1.5.7 with no unsafe code: frames, skippable frames, dictionaries, prefixes, one call and streaming. For every input it is meant to give the output, the error code and the stream positions that the C decoder gives. bun_pod gets try_zeroed_vec, for buffers whose size the input dictates. The compressor is still missing and nothing is moved to the new API yet, so the build still fails.
Parameters, frames, dictionaries, block splitting, streaming and the multithreaded mode, on top of the match finders and the entropy coders. The users of the C API move to the new one. src/zstd/wip holds the differential test harness and scratch copies, to be removed.
…g calls - VecOutput: output at the end of a Vec that zeroes about as much as gets written. StreamingDecoder and the CompressionStream coder zeroed the whole spare capacity before every call, which is quadratic in the number of frames. - node:zlib: with workers a call returns after any progress, so call again until the input or the output is used up. Streams with nbWorkers lost data. - The size that node:zlib reports to the GC comes from the types. - A job's input buffer grows with the input instead of being reserved whole. - Remove CCtx::reset, DCtx::reset, parameter updates in the middle of a frame and unused exports; share duplicated constants and helpers. - Tests: a trained dictionary, a stream with workers, one shared sample.
Collaborator
|
Updated 4:44 PM PT - Nov 7th, 2026
✅ @claude, your commit 46048ba7ac67a33e1ffd1c8b74e16e2b6739dacb passed in 🧪 To try this PR locally: bunx bun-pr 47079That installs a local version of the PR into your bun-47079 --bun |
It needs the C sources. It stays in the history of this branch.
The loop that decodes and executes sequences took 50% more instructions than the C one. Now it works on local copies of its state, which stay in registers (what is behind a reference is stored and loaded again around every write to the output), and the fast path is written with slices that shrink instead of positions, so that the compiler sees that most bounds hold. A match that starts before the prefix takes the slow path. 15 MB of source text at level 3: 977 MB/s before, 1219 MB/s now, C 1217 MB/s.
- fast and dfast without a dictionary work on 32-bit positions in the part of the input that they can read (Near), where a bounds check is one comparison, read each position once, and index tables through a mask. An index that is too low wraps around and fails the same check as one that is out of bounds. - SeqSink: the sequence store as a local variable while a hot loop adds to it. - The row match finder compares 8 tags at a time instead of one, and reads the bytes that a longer match has to have once per length instead of once per candidate. Level 12 goes from 1.43x to 1.16x the time of the C library. - The bit writer clamps its position where it writes, which removes the check. - The parts of generic functions that do not depend on the parameter are functions of their own (NoDictBlock, TwoSegments, SavedRep, Unsorted, ChainSearch), tuples of two of the same type are a struct or a range, and the size of the loaded header is in the stage that it belongs to.
- fast, dfast: what does not depend on the minimum match length moves out of the functions that are compiled once for each (first position, least step, repeat offsets with an older segment, width of a tagged hash). - btlazy2: the slot of the position to sort and its first candidate are worked out with the rest of what is known about it. - Workers use mpmc channels, whose receivers are Sync, so that a context is Send + Sync and an unsafe impl around one has nothing to vouch for. - Removes what nothing reaches: indexing a dictionary for long distance matching (only for a prefix from the user) and the full load method when filling the table of a context (the C code asserts that it is not used).
…its own input The buffer was sized, and zeroed, for a whole section (jobSize / ldmMinMatch entries of 12 bytes) for every job, so a flush of 1 KB cost as much as that: 20x the time of the C library at level 22, and gigabytes with a large jobSize. The C library allocates as much but only touches what it uses. The limit that the search stops at is still that of a whole section, as it decides whether the input gets indexed at all.
The optimal parser prices every length of every match it finds. The price is the sum of a part for the offset, which is now worked out once for a match, and a part for the length, which is now worked out when the code of the length changes. The positions that a match reaches are walked as a slice. With targetLength at its maximum and repetitive input, where this loop is nearly all there is, level 19 took twice the time of the C library and now takes two thirds of it. Other input is within 2% of what it was.
dylan-conway
force-pushed
the
claude/zstd-rust-replacement-4qwmbq
branch
from
October 27, 2026 17:06
802a3d5 to
078890f
Compare
…before A job is only told its own size, and the model of the C workspace took a job that was a little larger than any before it for a reason to make a new workspace, which clears the tables: 640 MB at level 22, for each flush of a stream whose flushes grow. The C library has the same rule but hardly gets there, as its small jobs all go to the context that had the first job of the frame. A worker's context now counts with blocks of any size from the start.
… 8 at a time Where two words differ takes no branch, so most matches are measured without one that is hard to predict: 15 to 20% fewer branch misses from level 1 to 9.
…d as they are made, and a sequence is written at once
…search, and takes no branch on an index that is too low
…used again are not copied
… of the machine `zstd.enabled` is true for a prebuilt WebKit whose tarball has the objects of `jsc` and `testaot`, because `testaot` then compiles zstd's decoder. The case that expects false looked at the shared build cache, so it failed on a machine that has fetched such a tarball. It has a cache of its own now, which is empty, and the test that puts these objects in a cache expects true.
Member
Author
|
CI on 285d751 (#136300, still running): 18 test files fail so far, 17 under Most are about mocks under Before the merge, the only failure on this branch was Generated by Claude Code |
… repeat offset leads into the dictionary With a dictionary that is searched in place, `skip_matchless_in_place` returned at once when one of the three repeat offsets led before the input. After a match in the dictionary that is so until three other offsets have come, or until the position is past the offset: 86% of the calls, in runs with random parameters. Now there is `skip_matchless_far_reps`, the same loop with such an offset looked up in the dictionary. It runs up to the position from which all three offsets lead into the input, where the loop that was there takes over. What it looks at is `far_rep`, the part of `rep_len` that finds where the offset leads and compares the first 3 or 4 bytes. It is inlined in the loop: with a call of `rep_len`, which also counts, noise with pieces of the dictionary takes 19 to 28% longer at level 13. Before the loop, `skip_matchless_in_place` looks at the last offset alone. In a copy of the dictionary in which single bytes differ, the position after such a byte goes on at that offset, and all that the loop does there is done again by `get_all_matches`: without this look, such copies take 3 to 8% longer than before at level 13. Time relative to before, means of 16 orders of the functions in the binary for each, a dictionary of 64 KB. At level 13, inputs of 1, 4, 16 and 32 KB: - skewed bytes 0.96, 0.91, 0.84, 0.57; the last was at 1.45 of C and is at 0.83 - base64 0.97, 0.93, 0.77, 0.72; the last was at 1.12 of C and is at 0.81 - noise 0.98, 0.96, 0.92, 0.86 These three kinds, wherever the dictionary is searched in place (54 rows, levels 13 to 19, a dictionary of 30 KB too): 0.57 to 1.00 of before. They were at 0.78 to 1.45 of C and are at 0.75 to 0.88. The review of a first form of this change measured what the 11 kinds of data so far do not have, so there are 11 more: - Noise of 6 or 8 bits with 16 bytes of the dictionary every 40 or 200 bytes (48 rows): 0.53 to 0.79 of before. They were at 1.25 to 1.68 of C, the worst rows there were, and are at 0.88 to 1.00. - Pieces of the dictionary with one byte altered, at 2 or 4 distances in turn, among noise of 5 bits (24 rows): 0.91 to 0.99. 1.14 to 1.29 of C before, 1.07 to 1.21 now. - The dictionary itself with one byte in 5 altered (12 rows): 0.92 to 0.99. - The same with one byte in 8, 16, 32 or 100 (48 rows): 0.99 to 1.017, mean 1.004. 15 of these rows take 0.5 to 1.7% longer, at 3 to 7 standard errors. That is the look at the last offset, which is for nothing there. The 8 other kinds of the first 11, by level: 1.0001, 1.0012, 1.0009, 1.0007, 1.0000. Four of their 208 rows are more than 3 standard errors slower: hex digits of 16 and 32 KB at level 15 (0.7 and 0.4%), 16 KB of records of 1 KB at level 13 and 1 KB of JSON at level 15 (0.3%). `rep_len` is other instructions than before, for all who call it. A dictionary whose tables are copied (66 rows of 64 and 256 KB, and the 44 of the new kinds): means of 0.999 to 1.001, no row of the 66 beyond 3 standard errors, two of the 44 (one byte in 8 altered, 32 KB, levels 16 and 19: 1.2%, where a second run over 8 of the orders has 0.993). No dictionary (75 rows of 16 KB): 1.001 to 1.002, none beyond 3. Nothing else changes: `skip_matchless_in_place` and `rep_len` are other instructions, `skip_matchless_far_reps` is new, and the other 383 functions of the crate are the same as before, `get_all_matches`, `skip_matchless`, `search_dms` and the parsers among them. It is 4.3 KB more code. The output does not change. What else was tried, 8 orders each: - `far_rep` for the candidates of `get_all_matches`, which calls `rep_len` for every offset that leads before the input: the function comes out different, and hex digits take 14 to 19% longer, text 5%. - The offsets before the root of the dictionary's tree in the new loop: copies with altered bytes 0 to 2% faster, noise 2 to 4% slower. - No loop after a single literal, and no look at the last offset: free where one byte is altered, but two in a row are common with one in 8 (2.4% at level 15), and noise with pieces every 40 bytes takes 3% longer. Both rules: the loop that was there comes out different, and noise of 1 KB gains nothing. - The look at the last offset in `skip_matchless_far_reps` (1 to 3% on the copies) or in a function of its own (the old loop 2% slower on noise). - `rep_len` as it was and a second function for the loop: no faster. - `far_rep` with a closure for what to do with the place: the closure is not inlined, and most rows with a dictionary take 1 to 5% longer. - One condition, `in_place != self.dms.is_some()`, for two at the start of the loop: noise and base64 of 1 KB 3 to 4.5% slower. - Whether the trees are up to date before the look at the last offset, which is the cheaper question: the old loop comes out different, noise, base64 and skewed bytes 1 to 3% slower, nothing faster. - The new loop for an older segment of the window as well, which is where a dictionary is whose tables are copied: noise with pieces of it, 16 KB at level 16, goes from 1.66 of C to 1.00. That changes `skip_matchless`, which all data without a dictionary goes through, so it is for a change of its own. The test in `zstd.test.ts` compares with what Bun 1.4.4 writes, as all of them there, so it passes before this change too. Of 24 faults put into the new code, runs against the C library with random parameters see 16, and each of these changes one of the 8 digests of the test. Five of the others only cost time (the new loop not entered, left one position early, or entered without the look at the last offset, and the like). Three are bounds that no offset gets to, which were in `rep_len` before. The generator of the test is one that 4 other places of that `describe` had a copy of. They share it now.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
… bool `skip_matchless_generic(.., true, false)` does not say which is which, and mordant's `bare_bool_args` refuses it. `far_reps` is `Reps::Near` or `Reps::Far` now. All 386 functions of the crate are the same instructions as before.
… of the window A dictionary whose tables are copied (inputs of more than 32 KB at levels 13 to 15, of more than 8 KB from level 16 on) is an older segment of the window, as is what a stream has before the place where its buffer starts again, and what a job of a worker has of the job before. After a match there, `skip_matchless` returned at once, as `skip_matchless_in_place` did until 666b766. On noise with pieces of the dictionary these levels took 1.25 to 1.71 times the time of C. Now there is `skip_matchless_older`: the look at the last offset, the loop of 666b766 with the offsets looked up in the older segment (`skip_matchless_far_reps_older`), and then the loop for offsets that are near. What the two functions for before the current segment share is `skip_matchless_far_then_near`. `skip_matchless_far_reps` is `skip_matchless_far_reps_in_place` now. Where a match can be 3 bytes, the loop also left at every candidate from the table for 3 bytes that is in the older segment, without a look at it. In a stream that is longer than its window, that is how nearly all calls ended: of 545,000 on 3 MB of noise at level 19, 537,000, and 4,200 of those candidates were a match. `older_starts_alike` compares them now. That is in the two loops for an older segment, of which the one for near offsets is `skip_matchless_near_reps_older`. With a longer minimum that loop would be `skip_matchless` again, so it is `skip_matchless` that is called. The parsers that call all this are copies again, `compress_block_opt0_older` and `compress_block_opt2_older`, which `compress_block_opt` takes for `DictMode::ExtDict`. What there is before the current segment is an enum now, `Before`, where the parsers and the loop had a bool. Time relative to before, means of 16 orders of the functions in the binary for each. A dictionary of 64 KB whose tables are copied: - skewed bytes: 64 KB at level 13 0.66 (1.33 of C, now 0.88), 256 KB 0.79; 16 to 64 KB at levels 16 to 19 0.81 to 0.89 (1.01 to 1.08 of C, now 0.88 to 0.90) - noise: 64 and 256 KB at level 13 0.78 and 0.82; levels 16 to 19 0.96 to 1.002 - base64: 64 KB at level 13 0.79 (1.09 of C, now 0.86); levels 16 to 19 0.94 to 1.008 - noise with 16 bytes of the dictionary every 40 or 200 bytes (12 rows): 0.60 to 0.86. They were at 1.25 to 1.71 of C and are at 1.00 to 1.08. - pieces of the dictionary with a byte altered among noise (8 rows): 0.93 to 1.006; the dictionary with one byte in 5 altered: 0.93 to 0.98; with one in 8 to 100 (16 rows): 0.99 to 1.002 - the 8 other kinds, by level (13, 16, 17, 19): 0.9986, 1.0001, 1.0004, 1.0004 Of these 139 rows, 69 were more than 3% behind C and 55 are. Their mean relative to C goes from 1.040 to 0.983. Five take longer by more than 3 standard errors: hex digits of 32 KB at level 17 and of 16 KB at level 19 (0.6 and 0.7%), 64 KB of the pieces with an altered byte at level 16 (0.6%), 64 KB of records of 100 bytes at level 13 (0.5%), 16 KB of text at level 16 (0.2%). Streams were not timed so far: 1 MB in pieces of 64 KB, no size given, windows of 16, 64 and 256 KB, levels 16 to 19, 17 kinds of data (204 rows). By window: 0.92, 0.95, 0.985 of before. By level: 0.94, 0.95, 0.96, 0.96. By kind, mean and best row: skewed bytes 0.84 and 0.66, base64 0.86 and 0.69, noise 0.90 and 0.74, noise of 5 bits 0.94 and 0.65, noise with pieces from up to 12,000 bytes back 0.88 to 0.94 and 0.76, the same with a byte altered 0.98 and 0.91, hex digits 0.99 and 0.91, the 7 others 0.999 to 1.001. The mean relative to C goes from 0.84 to 0.80. These rows are less certain than the others. At level 13, which without a size is not an optimal parser and runs the same instructions as before, the 51 rows are at 0.97 to 1.04, one of them 3.5 standard errors slower (in the run for another form of this change, one by 7% at 4.6). At levels 16 to 19 four rows are more than 3 slower, by 0.6 to 3.6%. Each was run three more times with more rounds: 0.992 to 1.004. Jobs of workers were not timed. Without a dictionary nothing of the optimal parser changes: `skip_matchless`, `get_all_matches`, `compress_block_opt0` and `compress_block_opt2` are the same instructions, as are the functions for a dictionary that is searched in place. `compress_block`, which all levels go through, has 21 instructions more, for the choice of a parser. 175 rows without a dictionary (25 kinds, 1 KB at levels 1, 5, 9, 13 and 16, 16 KB at levels 13 and 16): means of 0.9991 to 1.0000 by level, no row slower by more than 3 standard errors. It is 22 KB more code. The output does not change. What else was tried, 8 or 16 orders each: - No copies of the parsers, `skip_matchless` itself with the new part in front: its loop comes out different, and without a dictionary noise, base64 and skewed bytes take 2 to 6% longer, all rows 1%. - The candidates of 3 bytes as before, and `skip_matchless` for all near offsets: what the review of this change found. In streams at level 18, noise and the like take 1.0 to 1.8% longer than before, at 3 to 5 standard errors, since every one of those calls goes through one function more. - A loop of its own for near offsets whatever the minimum, inlined in `skip_matchless_older`: noise in a stream takes 3.5 to 8% longer at level 16. Out of line: 64 KB of records of 1 KB 1.4% at level 13. - That loop for a minimum of 3 only, but inlined: 256 KB of noise with a copied dictionary 1.5% longer, at 12 standard errors. - `older_starts_alike` not inlined: the same rows 0.9 to 1.2%, and streams gain less at levels 18 and 19. - `index_overlap_check` for the same condition in it: the loop comes out with 697 instructions for 663. Not timed. - `skip_matchless_older` inlined in the two parsers: nothing faster, HTML, JSON and records 0.3 to 0.6% slower. - The parsers for a dictionary that is searched in place for an older segment too, which is one line and no more code (the review, one order): most rows up to 6% slower than this, two 11 to 12%, and noise 2 to 7% slower than before. The test in `zstd.test.ts` has one more input, 33,000 bytes, for which both of its levels copy the tables of the dictionary, and there is a second test: 63,000 bytes without a dictionary in windows of 8 and 16 KB. They compare with what Bun 1.4.4 writes, as all of them there, so they pass before this change too. Of 19 faults put into the new code, runs against the C library with random parameters see 11, and each of these changes a digest of the new inputs. Three of them, in `older_starts_alike`, are seen only by runs that are aimed at this: optimal parsers, windows of 1 to 256 KB, pieces from far back. The tree mostly finds the same match of 3 bytes. The other eight cannot change the output: six choose another of the loops or parsers, leave the loop where it could go on, or hand on a dictionary where there is none, and two are at the last 3 bytes of the older segment, which never get into the table for 3 bytes.
…es at a time At levels 13 to 19, HTML and records were what took longest relative to C: with a dictionary whose tables are copied 1.05, 1.07 and 1.29 times its time in the mean (HTML, records of 100 bytes and of 1 KB), up to 1.35. After a long match, all its positions are added to the trees, and that is where the time goes: 96% of it for 64 KB of records of 100 bytes at level 13 with such a dictionary, in `update_tree_in_part` and what it calls. A step down a tree takes as many instructions there as in `ZSTD_insertBt1`, 86. The comparison of the two positions takes 59 where C takes 30, and a step 31 cycles where C takes 23. The comparison took 16 bytes at once, as a vector. Now it takes them as two words of 8, each with a branch (`Common::within_words`): - in `insert_bt1`, which is what `update_tree_in_part` calls for each position. Where matches are looked for, `tree_matches`, it stays as it was. - in `TreeInput::compare_older`, for all who get to a match in the older segment. It also has two conditions where it had five: `TreeInput::new` cuts the buffer of the older segment where the segment ends. The field is private now, so that there is no other way to make one. What `Near::compare_chunks` does with the first 16 bytes is an enum now, `First`, where it was a bool, and `bt_walk` takes it where it had a bool for `Near::compare_head`, which goes with `First::Alike`. Time relative to before, means of 16 orders of the functions in the binary for each, 1,206 rows: 283 were more than 3% behind C, 218 are. 444 take less time by more than 3 standard errors, 289 by more than 1.5%. By kind, all rows of levels 13 to 19 (mean, best row, how many are more than 3% behind C): - HTML (61 rows) 0.92, 0.79, 36 before and 11 now - records of 1 KB (61) 0.93, 0.83, 43 and 31 - records of 100 bytes (61) 0.95, 0.82, 35 and 25 - JSON (61) 0.98, 0.95, 5 and none By what there is before the input (rows, how many behind C, mean relative to C): - a dictionary of 64 KB whose tables are copied, 11 kinds (99): 30 and 21, 0.970 and 0.926. HTML 0.89 of before (1.05 of C, now 0.93), records of 1 KB 0.87 (1.29, now 1.12), of 100 bytes 0.91 (1.07, now 0.97). - the same, pieces of the dictionary and copies with bytes altered (40): 24 and 22, 1.015 and 1.006 - a dictionary of 64 or 30 KB that is searched in place, 11 kinds (286): 51 and 37, 0.89 and 0.87; the pieces and copies (176): 75 and 72, 0.976 and 0.972 - nothing, 1 and 16 KB (175): 24 and 22, 0.928 and 0.922; 128 and 512 KB (175, not timed so far): 35 and 26, 0.914 and 0.902 - a stream that is longer than its window (255): 44 and 18, 0.820 and 0.801 Six rows take longer by more than 3 standard errors. Each was run three more times with more rounds. 128 KB at level 13 of words drawn from 60 at random: 0.9%, then 0.9 to 1.2% (it is at 0.91 of C). 128 KB of text at level 13 0.4%, then 0.2 to 0.4%. 64 KB of pieces of a copied dictionary with a byte altered at level 13 0.5%, then 0.3 to 0.5%. 4 KB of text at level 16 with a dictionary of 30 KB 0.3%, then 0.1 to 0.3%. Two of streams, 2.1 and 1.6%, then 0.986 to 1.018. `get_all_matches`, the `skip_matchless` functions and the six parsers are the same instructions. Other ones: `compare_rest`, which has `compare_older`, the three `update_tree_in_part`, `Trees::new`, and the six `compress_block_lazy_part` of the binary tree finder of levels 13 to 15, which have `TreeInput::new`. 512 KB at levels 13 and 15, where that finder is used: 0.998 in the mean, 0.967 to 1.017, none beyond 3 standard errors; streams at level 13 0.989. It is 223 bytes more code. The output does not change. What else was tried, 8 or 16 orders each: - The two words where matches are looked for as well: HTML and records as here, but 35 of 314 rows take 0.3 to 1.8% longer, at up to 18 standard errors: text, logs, JSON, floats, and the copies of a dictionary with bytes altered. There the first difference is often in the second word. - The same and `compare_older` as it was: with a copied dictionary, 41 of 139 rows take 0.5 to 1.6% longer, all but two of them from level 16 on. - The two words only after 16 bytes in common, else the vector: half the gain, and noise, base64 and skewed bytes take 1 to 5% longer, in streams up to 13%. - One branch for whether the 16 bytes differ, and the word chosen with `select_unpredictable`: HTML takes 2 to 3% longer than before. - The same with 128 bits (`u128`): no gain (one order). - `le64_at` for the words: all four are read before the first is compared, and `compare_rest` has 104 lines of assembly for 84. Not timed. - A loop over 16 bytes at a time in `compare_rest`: it has registers to save then, and records take 2.5% longer (one order). Of 12 faults put into the new code, runs against the C library with random parameters see 9. The unit tests see these 9, and a tenth that only costs time. 8 of the 9 change a digest of the tests in `zstd.test.ts` for literals while a repeat offset leads before the current segment. The ninth is the buffer of the older segment not being cut. The last two only choose the other comparison. `the_head_of_a_position_is_compared_as_all_of_it` and `a_match_that_leaves_the_older_segment_goes_on_in_the_current_one` go over the three values of `First`. All times are from x86-64.
… an older segment At levels 13 to 19, what took longest relative to C was an input with something before its current segment: a dictionary that is searched in place, or an older segment, which is what a dictionary whose tables are copied is, or the part of a stream before the place where its buffer starts again. Two things cost there. A repeat offset that leads before the current segment was a candidate without a look at what is there, so `get_all_matches` called `rep_len` for it, up to three times a position. `far_rep`, which the loops over literals use, tells whether it starts like the position. A step down a tree that gets to the older segment first failed in `Near::compare_chunks` and then called `compare_rest`. For 32 KB of records of 100 bytes at level 16 with a copied dictionary of 64 KB, 86% of the time is `get_all_matches` and the `compare_rest` that its walk calls, which alone is 39%. `get_all_matches` did both for all modes, and a change to it moves the times of all of them. Now there is a copy for each `Before`, as there is of the loops over literals and of the parsers: `get_all_matches_in_place` and `get_all_matches_older`. `get_all_matches` is the same instructions as before. - Both copies ask `starts_like_rep`, which has `far_rep`, for the three repeat offsets. - `bt_walk` takes an `Older`. `Called` is what it did. `TriedLast` is the same without the call. `TriedFirst` looks at whether the position is in the older segment before anything else, which takes a cycle for one that is not. - `get_all_matches_older` looks for matches with `TriedFirst`, and adds the positions that a match skipped with `TriedLast`, but for a long run in a current segment of more than 64 KB. That is one more `update_tree_in_part`. - All else stays `Called`: the loops over literals, and the binary tree finder of levels 13 to 15, which has not been tried with the others. Time relative to before, means of 16 orders of the functions in the binary for each, 1,031 rows: 197 were more than 3% behind C, 116 are. 499 take less time by more than 3 standard errors, 185 by more than 5%, 64 by more than 10%. By what there is before the input (rows, how many behind C, mean relative to C): - a dictionary of 64 KB whose tables are copied, 11 kinds, 16 to 256 KB (99): 24 and 7, 0.928 and 0.880. Records of 100 bytes 0.81 of before (0.97 of C, now 0.80), of 1 KB 0.88 (1.14, now 1.01), HTML 0.94 (0.93, now 0.87), JSON 0.94, hex digits 0.96, text 0.97. - the same, pieces of the dictionary and copies with bytes altered (40): 23 and 15, 1.006 and 0.933 - a dictionary of 64 or 30 KB, 1 to 32 KB, 11 kinds (286): where it is searched in place (198) 17 and 13, 0.862 and 0.851; where it is copied (88) 22 and 2, 0.906 and 0.842 - the same, the pieces and copies (176): in place (132) 50 and 35, 0.982 and 0.940; copied (44) 23 and 12, 0.981 and 0.901 - a stream that is longer than its window (255): 16 and 10, 0.809 and 0.800 - nothing, 1 and 16 KB (175), the same instructions: 22 and 22, 0.922 and 0.923 Three rows take longer by more than 3 standard errors. Each was run three more times with more rounds. 64 KB with a period of 60 at level 16 with a copied dictionary 0.2%, then 0.2 to 0.4% (it is at 0.88 of C). Two of streams, 3.7 and 1.4%, then 0.976 to 1.003. 54 more rows after a review, which has 2% more instructions where there is a copied dictionary but the trees lead to the current segment: six kinds of other data, 32 to 256 KB, levels 13, 16 and 19. Records of 100, 300 and 1,000 bytes of which each is the one before with bytes altered: 0.993 to 1.006, one row beyond 3 standard errors, 0.4%. A period of 60: 0.998 to 1.005, two rows, 0.4 and 0.5%. JSON 0.92 to 1.00, logs 0.99 to 1.00. All of these rows but three are ahead of C. The same instructions: `get_all_matches`, `compare_rest`, the loops over literals, the two parsers without a mode, the binary tree finder, the three `update_tree_in_part` there were. The four parsers with a mode call other functions. It is 11,958 bytes more code. The output does not change. What else was tried, 8 or 16 orders each unless it says one: - `TriedFirst` for all walks where there is an older segment: more for records (a copied dictionary: 3 rows behind C, 0.873 in the mean; 256 KB of records of 1 KB 0.85 where this has 1.00), but 24 rows take 0.5 to 2.5% longer, one 3.5%: a period of 60 at all sizes, 256 KB of noise, HTML, JSON and logs, streams of records with a window of 256 KB. - `TriedLast` for all of them (one order): two thirds of that gain, and 256 KB of noise takes 4% longer. - With `TriedFirst`, no call either for what `compare_chunks` leaves of a match in the current segment: a period of 60 takes 0.3 to 1.6% longer from level 16 on, 7 rows. - Only the copies, with the call: records of 100 bytes with a copied dictionary take 1.1 to 3.8% longer from level 16 on, 15 rows. The walk of the copy came out with 3% more instructions. - The dictionary handed to `starts_like_rep` by the caller: hex digits take 0.6 to 2.6% longer with a dictionary in place. - The far case in the closure that `rep_offsets.map` takes: the closure is not inlined any more, and all data takes 10 to 18% longer (one order). - `Older` as a const parameter of `bt_walk`, so that `tree_matches` has two calls: other instructions for `get_all_matches` and the loops in place. Not timed. - A function for the comparison without the call, which returns a `Result`: 1 to 2% more instructions (one order). - Two loops, since the indices only go down on the way: one for the current segment, then one for the older. 2% more instructions where few walks get there, and no less time (one order). The second loop out of line: what the closure captures is in memory then, 3 to 10% more instructions (one order). - `cold_path` before the code for the older segment: nothing, or slower (one order). Of 15 faults put into the new code, runs against the C library with random parameters see 9. The other six cannot change the output: five only choose another way to the same comparison, one is about an offset that `far_rep` refuses. All 9 change a digest of the tests in `zstd.test.ts` for literals while a repeat offset leads before the current segment. `a_way_down_a_tree_is_the_same_whatever_is_done_with_the_older_segment` is new: it fills a tree over two segments with each `Older` and each `First`, and compares each step with the bytes. It sees the two faults in `bt_walk`. It is not run under Miri, where it takes minutes. `a_match_that_leaves_the_older_segment_goes_on_in_the_current_one` does the comparison without the call too. All times are from x86-64.
…the one before At levels 13 to 19, records were still behind C where nothing comes before the input: 16 KB of records of 100 bytes took 1.13 and 1.15 times its time at levels 13 and 16, with fewer instructions (8.57 G for 9.03 G) and fewer of them a cycle (4.4 for 5.25). 83% of the time is `update_tree_in_part`, which adds the positions that a match skipped. Five times as many cycles as in C go by with a scheduler of the integer units full (0.27 G for 0.05 G of `int_sch0_token_stall`), and 2.7 times as many places to dispatch to stay empty for want of room behind. `insert_bt1` returns how many positions to move forward, which comes from the longest match that the walk met and from how far a match reached. That is a chain from one position to the next: the last comparison, the length, two conditional moves, the index, the 8 bytes there, the hash, the cell of the table, the first comparison. C has the same chain. The answer is 1 but for matches that overlap the position or are longer than 384 bytes. Now `update_tree_in_part` has a branch for 1, so the processor goes on with the next position while the walk of this one ends. The compiler folds such a branch into the addition that it stands for, so the other arm goes through `core::hint::black_box`. Also, `update_tree` hands `OLDER` to the instance without hints too, which is for long runs in a current segment of more than 64 KB where ways down are short. So far only short runs had it. That is one more `update_tree_in_part`. The one with hints at the bytes alone stays as it is: without the call, 256 KB of records of 1 KB at level 16 take 15% less time, but HTML 2.5% and a period of 60 1.3% more (one order). Time relative to before, means of 16 orders of the functions in the binary for each, 1,206 rows: 144 were more than 3% behind C, 97 are. 332 take less time by more than 3 standard errors, 68 by more than 5%, 18 by more than 10%. By kind, all rows of levels 13 to 19 but those with pieces of a dictionary (mean, best row, how many are more than 3% behind C): - records of 100 bytes (63 rows) 0.955, 0.80, 21 before and 4 now - records of 1 KB (63) 0.960, 0.85, 19 and 3 - words drawn from 2,000 at random (63) 0.976, 0.93; HTML (63) 0.983, 0.92, 3 and none - CSV (13) 0.983, 0.95; a period of 60 (63) 0.985, 0.94 By what there is before the input (rows, how many behind C, mean relative to C): - nothing, 1 and 16 KB (175): 22 and 15, 0.921 and 0.912; 128 and 512 KB (175): 21 and 17, 0.894 and 0.889 - a dictionary of 64 or 30 KB, 1 to 32 KB, 11 kinds (286): where it is searched in place (198) 13 and none, 0.850 and 0.826; where it is copied (88) 2 and 1, 0.837 and 0.834 - the same, pieces of the dictionary and copies with bytes altered (176): 48 and 44, 0.930 and 0.922 - a dictionary of 64 KB whose tables are copied, 16 to 256 KB (99): 8 and 3, 0.880 and 0.869. 256 KB of records of 100 bytes at level 13 0.85 of before (1.22 of C, now 1.04), of 1 KB 0.90 at level 13 (1.12, now 1.00) and 0.85 at level 19 (1.12, now 0.95). - the same, the pieces and copies (40): 15 and 9, 0.931 and 0.920 - a stream that is longer than its window (255): 15 and 8, 0.805 and 0.798 Four rows take longer by more than 3 standard errors, by 0.7 to 1.4%. Each was run three more times with more rounds: 0.995 to 1.005, none beyond 3 standard errors. Other instructions: the four `update_tree_in_part` there were, and nothing else. It is 1,954 bytes more code. The output does not change. What else was tried: - `cold_path` where `insert_bt1` notes how far a match reaches and where it asks for more than 384 bytes, so that both stay branches and the answer is a constant on the usual way (16 orders): as much for records, but the loop of the walk comes out with 5% more instructions. With a copied dictionary from level 16 on HTML takes 2.2 to 4.7% longer and records of 1 KB 3.1 to 3.7%; 50 rows in all. - The other arm as a function that is cold and not inlined (one order): 3% more for records, but 3% more instructions where walks are long, and HTML with a copied dictionary takes 2% longer at level 16. - The branch with `cold_path` alone: folded, the same instructions as before. - `Descent::step` reads the one child that it takes, 4 bytes, where `bt_children` reads both as 8: a third fewer loads that wait for a store of 4 bytes, 3% fewer instructions, 0.5 to 1.8% less time for records and HTML, 1% more for noise and JSON (one order). Left for later: it changes all walks. - Hints at the older segment for the children that are there: 5 to 13% more instructions, slower in all rows (one order). - `rep_len` inlined in the two copies of `get_all_matches`: 2 to 3% more instructions and time (one order). Of 6 faults put into the new code, runs against the C library with random parameters see 5. The sixth only chooses the call for the older segment. The test "at level %i" in `zstd.test.ts` sees the 5: with its data, in one call as there, all change what level 16 makes, and 4 what levels 17 to 19 make. `black_box` is documented as best effort. If a compiler folds the branch one day, the output stays the same and no test fails: only the time is lost. The branch is there in the assembly for x86-64 and for aarch64. All times are from x86-64.
…, and ahead where the history is long A survey of the port against C without a dictionary, 25 kinds of data from 1 to 512 KB: of 1,200 rows of compression at levels 1 to 12, 116 are more than 3% behind, of 682 of decompression into a buffer that is there 139, of 700 into a new one 201. So this is about the decoder. What took longest relative to C was HTML, 1.09 times its time in the mean and up to 1.24, and records. Where the table of match lengths says that matches are long, and for blocks of up to 32 sequences, `Executor::run` takes one sequence at a time: it decodes it and executes it. Two things were wrong with that loop. It worked on `seq_state`, whose address `decode_batch` takes further down in the function. So all of it was in memory: the bits, how many are consumed, the three states, the last offsets. For 128 KB of HTML of level 3, 17% of the samples are on one reload of the bits, and there are 0.53 G loads that get their value from a store that is not done, for 0.42 G in C. Now the loop of `run::<false>` has a copy that nothing takes the address of, as it has of the executor (`fork`): 0.28 G such loads, 8% fewer cycles. With `LONG`, which is for a history of more than 16 MB with far offsets, the same loop ran, and it asks for no match ahead of time. That had not been timed. 48 MB of which the second half is pieces of the first taken at random, with a window of 64 MB, took 1.18 to 1.64 times the time of C. Now `run_long_matches_ahead` does what `ZSTD_decompressSequencesLong_body` does: a sequence is executed once 8 more are decoded and their matches asked for, up to the end of the block. `LONG` is also for the first block after a dictionary that counts as cold, which is each one in a new context. Such a dictionary has just been copied, so it is in a cache. There the loop stays as it was, without the copy either: 128 KB of HTML, of records of 100 bytes and of copies of the dictionary with bytes altered take 5 to 7% longer with the copy, and 14 to 16% with 8 ahead (one order). So the new function is for a history of more than 16 MB alone. Time relative to before, means of 16 orders of the functions in the binary for each. Rows, how many are more than 3% behind C, mean relative to C, how many take less time by more than 3 standard errors: - into a buffer that is there (682): 123 and 107, 0.875 and 0.869, 107, of which 33 by more than 5%. HTML 0.950 of before (1.081 of C, now 1.027), records of 100 bytes 0.971 (1.012, now 0.982), a period of 60 0.972, words drawn from 200 at random 0.983, records of 1 KB 0.985. - into a new buffer (700): 191 and 190, 0.908 and 0.903, 105. HTML 0.953 (1.137, now 1.083), a period of 60 0.973, records of 100 bytes 0.977. - frames of 48 MB with a window of 64 MB (19, not timed so far): 14 and 14, 1.206 and 1.048, 14. At level 3 with long distance matching and at level 9, pieces from far back of 100 bytes 0.794 and 0.644 (1.39 and 1.64 of C, now 1.11 and 1.06); of 300 bytes 0.794 and 0.724 (1.35 and 1.52, now 1.07 and 1.10); of 1 KB 0.926 and 0.893 (1.18, now 1.09 and 1.05); of 70 to 330 bytes 0.781 and 0.718; of 8 to 2,000 bytes 0.945 and 0.926; of 20 to 90 bytes 0.971 and 0.868; of 40 bytes at level 9 0.680 (1.57, now 1.07). Of 20 bytes, which go in batches, 0.998. - a new context, a dictionary of 64 KB is loaded, a frame of 1 to 128 KB is decompressed, 26 kinds, levels 1, 3, 9 and 19 (356, not timed so far): 80 and 79, 0.928 and 0.929, 2. Eleven rows take longer by more than 3 standard errors. All but the first and the last were run three more times with more rounds. - 17 MB of source code three times over, with a byte altered every 200 bytes in the mean, at level 3 with long distance matching: 3.7% (0.897 of C, now 0.929). Its matches are far but one after the other, which the processor sees by itself. There the loop without anything ahead took a quarter less time than C's, and this one takes what C's takes. Two thirds of it is the delay, not the hints (one order). Nothing cheap tells such a block from one with pieces taken at random: long offsets have the same share of the table, 125 of 256. - 512 KB with a period of 1 KB, the frame of levels 9 and 13, into either buffer, four rows: 1.9 to 2.2%, then 1.3 to 2.6% (0.91 to 0.94 of C, and 1.01 to 1.02). A block is two sequences, and takes 181 more instructions: what is in registers is saved around the calls that copy. - 512 KB with a period of 100 KB of level 3 into a new buffer 1.2%, then 0.6 to 1.2%. 1 KB of skewed bytes of level 9 into a new buffer 2.7%, then 1.7 to 2.4%, within 3 standard errors (0.98 of C). - 16 KB of a binary of level 5 0.7%, then 0.998 to 1.001. With a dictionary, 128 KB of HTML of level 1 0.6% and 32 KB of records of 500 bytes of level 9 0.8%, then 0.0 to 0.4%, within 3 standard errors. - 16 KB of integers of level 19 into a new buffer 0.1%. Other instructions: `run::<false>` and `run::<true>`, which has the test for the new function. It is 5,736 bytes more code. The output and the errors do not change: the sequences are decoded in the same order, and the same ones are executed before the end of the stream is looked at. What else was tried, one order unless it says more: - The new loop in `run` itself (16 orders): as much for far matches, or 2% more. But `run::<false>` comes out with other instructions, which gain no more, and 512 KB with a period of 100 KB of level 3 take 2.5% longer. - The call in `decompress_block`: that has 992 lines of assembly for 921. - The new function for all blocks with a long history, not only where the table says that matches are long: pieces of 20 bytes take 9% longer, of 20 to 90 bytes 3%. So blocks with the predefined table, which has few long lengths, stay in batches whatever their matches are. - With a long history, batches for long matches too: 1.14 to 1.37 of C. Batches of 16, 32 or 64 for 128: 1.15 to 1.31. - The 8 that wait are executed before the last 8 are decoded: 1.04 to 1.18 of C. Blocks of such data have 200 to 1,600 sequences. - 16 ahead: nothing. - Hints at all of a match, up to 512 or 1,024 bytes: pieces of 1 KB 0.92 to 0.97 of C, but those of 300 bytes take 2% longer, and so does the source code. - No hint for a match with the offset of the one before: nothing. - The loop as it was for blocks of up to 32 sequences: a third of what HTML gains is lost, most of what records of 100 bytes gain, and the period of 1 KB keeps two thirds of what it loses. - `History::prefetch` on top of what takes a `Sequence`, so that nothing is converted to and fro: `decode_batch` comes out with other instructions. Not timed. - Batches for all blocks, with a short history: HTML as here or 2 to 4% less, records of 100 bytes 13% more than before. - Matches of 17 to 64 bytes without a call of `memcpy`: HTML 1 to 2.5% less, records of 1 KB 2% more. Left for later. - For a new buffer, where 15 to 19% of the time of records of 1 KB is `memset`: blocks with long matches grow the buffer as they go, where now all that a block may write is zeroed first: records of 1 KB take 14 to 22% longer, HTML 3%. Of 13 faults put into the new code, 9 are seen. The state that is not written back after the loop of `run::<false>`: runs against the C library with random input see it at once. Eight in the new function: 7 that wait for 8, 7 decoded ahead, no rotation, rotation the other way, the newest executed for the oldest, the last sequence not told from the others, the offsets of the start returned, the position not written back. Random input does not get there, as it takes 16 MB. The test "long matches from more than 16 MB back" in `zstd.test.ts` is new: its frame, whose blocks have 592 to 857 sequences, sees all eight, in one call and as a stream, and so do frames that were timed. Not seen: the end of the stream is not looked at, which takes a damaged frame; the last sequence is not told from the others among the first 8, which takes a block of up to 8 sequences; literals that would be in the output; a history of any length, which only costs time. 642 results of the debug build are those of Bun 1.4.4: the 19 frames, each also damaged at three places and cut short, and frames with a dictionary, with it, without it and damaged. All times are from x86-64.
Member
Author
|
Build #136583 ( It fails 1 test of 51,008: The rest of the build is as before: the 40 files that Generated by Claude Code |
…onary is indexed faster at levels 5 to 10 When a frame with workers is given up, they are told to stop, and the next frame waits until they have. A worker looked at that flag before each block, and every 4,096 positions while it adds the prefix of its job to the binary trees. It did not look while it adds it to rows. 3 workers, level 12, jobs of 1 MB, long distance matching, which makes the prefix up to 4 MB: with 4.6 MB of noise fed, from the reset of the session to the end of a next frame of 5 bytes it took 254 to 324 ms, where the C library, which waits for all jobs to end, takes 200 to 270 ms. `row_update` now takes 65,536 positions at a time and looks at the flag before each part, as `update_tree` does: 19 to 35 ms. Level 8 with the longest overlap and 8 MB: 12 to 19 ms, now 0.7 to 2.6 ms. Not changed: - Hash chains. Rows take over from a window of 32 KB on, so there the prefix of a job is 16 KB at most. - `fast` and `dfast`, which index every third position. With long distance matching, or a window of 128 MB and the longest overlap, and the tables of the level: 0.2 to 4.6 ms. With a hash table of 2^27 entries as well, a job indexes 128 MB first, which a reviewer timed at 1.6 s. To index that in parts takes more care than a loop, as what is done near the end of a part depends on where it ends. To see what that costs, a dictionary is loaded into a context and 1 KB compressed, which had not been timed: 11 dictionaries of 8 KB to 1 MB, levels 5 to 12, 88 rows, 8 orders of the functions in the binary for each. The port was behind C: 1.08 in the mean, 69 rows by more than 3%. Two loops were the reason. - Hash chains (a dictionary of 8 KB at levels 5 to 8: 1.43 to 1.55 of C). `insert_and_find_first_index` chose the hash by the least length of a match at each position, and checked three indices. It now calls `Chains::insert_all`, which the search has for the same thing. `hash_ptr`, `hash_ptr_salted` and `hash5` to `hash8` have no caller left and are gone; `hash3` and `hash4`, without a salt now, are in their one caller each. - Rows. `row_update` checked at each position that 8 bytes can be read. It now takes the bytes of a part once and goes through windows of 8. It is also a function of its own: within `load_dictionary_content`, its loops came out 5 to 10% slower or faster with what else is in there. Time relative to before, and relative to C before and after: - all 88 rows: 0.856; 1.080, now 0.925. 69 rows more than 3% behind C, now 7. 67 take less time by more than 3 standard errors, none takes more. - by level: 5 to 8 0.80 to 0.81 (1.12 to 1.14, now 0.89 to 0.92); 9 and 10 0.85 (1.08, now 0.91 and 0.92); 11 and 12, where most of these are binary trees, 0.98. - by dictionary: 110 KB of JSON 0.79, 64 KB of HTML 0.81, of JSON 0.82, of text 0.85, of noise 0.91; 200 KB of HTML 0.82; 8 KB of HTML 0.84; 400 KB of text 0.86, of noise 0.94; 30 KB of text 0.88; 1 MB of a binary 0.93. - still more than 3% behind: 64 to 200 KB of HTML and JSON at levels 11 and 12 (1.04 to 1.12, binary trees), 400 KB of noise at level 12 (1.06), 8 KB of HTML at level 5 (1.06). These times are from an Intel Xeon of the Skylake family alone. On AMD Zen 4, where all earlier times are from, this has not been run. Other instructions: `load_dictionary_content`, and `row_update`, which is new as a function. It is 172 bytes less code. The output does not change: both loops add the same positions in the same order. What else was tried: - The loop over parts in `load_dictionary_content`, for rows and hash chains (8 orders): the wait is as short. 57 rows take 4 to 10% less time, only because the loops of `row_update` come out otherwise, and the 8 KB dictionary at levels 5 to 8 takes 13 to 14% longer. - That with `Chains::insert_all` and `row_update` as a function of its own, reading 8 bytes at a time as before (8 orders): 0.930 of before, 0.999 of C. Tests: - The unit test of a worker that stops now also runs with rows. Without the look at the flag it fails. Not tested: that the flag is looked at again between parts, which takes a race. - "after a dictionary that is indexed in several parts" in `zstd.test.ts` is new: 200,000 bytes of 3 bits each, and pieces from all over it, from 9 positions before to 2 after where each part ends, and from its end; hash chains with a least length of 3 to 7, and rows of levels 5, 8 and 10, with values from Bun 1.4.4. Like the rest of that block it passes before the change too. - Of 12 faults put into the two loops, runs against the C library with random input see all 12, the new test 9: only the first part is indexed; the first or the last position of a part is left out; one or 7 windows too few; the bytes of the next position; indices one too high; a hash of 4 bytes always, or of 6 at most. It does not see: where parts meet is indexed twice; one position more or less at the end of the hash chains. - 342 results of the debug build are those of Bun 1.4.4: dictionaries of 65,543 to 400,000 bytes at levels 5 to 12, with hash chains too, and 4 MB with 3 workers, with and without long distance matching and overlap.
…old instructions The change of #47613, as it is there at eb43446, so that it is nothing once main has it. On macOS arm64 the test fails for some layouts of the executable, and for that of 1e78952 (Buildkite #136744, all four attempts): `placesThatCall` finds a place that calls `execve` in an executable without what starts programs. Its bytes are 0f0f0f0f0f0f0f0f0f10101010..., a table in `__TEXT,__const`: `readMachO` gave whole segments to the scan, which reads every word of an executable segment as an instruction. Now it lists sections, and only one that says that it holds instructions is read as such. The two tests "a Mach-O file" pass here and fail with the helper as it was. The rest of the file is skipped in a build with AddressSanitizer, so that it does what is expected of it on macOS is left to CI.
Member
Author
|
Build #136744 (
A
Generated by Claude Code |
…acement-4qwmbq main has the rewritten ahead-of-time compiler (#47942). Its src/sema/program/retire.rs and src/aot/report/source_texts.rs no longer compress anything, so both are main's, and `bun_zstd::Contexts`, which only retire.rs used, goes here as it goes from the binding on main, with its unit test. In a build of the crate as a release build makes it, `compress_append` is the one function of 397 whose instructions change (175 to 172). test/aot/sanitizers/thread-sanitizer.ts is deleted, as on main: this branch had changed one line of its help.
Member
Author
|
Buildkite #138405 for Generated by Claude Code |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Replaces the vendored zstd C library (v1.5.7 at
f8745da6, plus Bun's two patches) with a port of it to safe Rust, in thebun_zstdcrate (src/zstd). Nothing of the C library is built for Bun or linked into it any more. (One test program in one developer configuration still takes its decoder: see Build.)The goal of the port is that nobody can tell: the same compressed bytes for the same input and parameters, the same decompressed bytes, the same error codes, and the same input/output positions after every streaming call.
The crate
#[forbid(unsafe_code)], andindexing_slicing,unwrap_used,expect_used,panicandunreachableare denied: a frame from the network cannot make it panic. Allocations whose size depends on the input or on parameters are fallible and reportmemory_allocation. The few small ones of fixed size (a new context, a worker's state) abort on failure, like the rest of Bun.strum,thiserrorandbun_pod.targetCBlockSize, magicless frames, one-shot and streaming, andnbWorkers/jobSize/overlapLog.extDict, a dictionary), the port gives each buffer a range of a virtual address space and does the same arithmetic on those addresses.Not ported, because nothing in Bun can reach it:
ZDICT_*(training), legacy formats, static and stable-buffer contexts,refPrefixon the compressing side,ZSTD_compress2,ZSTD_CCtx_reset, changing parameters in the middle of a frame, the sequence producer API, indexing a prefix for long distance matching, rsyncable mode, and the experimental parameters (ids 500 and 1000+).Callers:
Bun.zstdCompress*/Bun.zstdDecompress*,node:zlib(NativeZstd.rs),CompressionStream/DecompressionStream,fetchandBun.servebodies, compressed embedded assets, standalone executables, and the ICU data decompressor, which now calls a small Rust shim (src/runtime/icu_decompress.rs).ProcessBindingConstants.cppno longer haszstd.hto take theZSTD_*constants from, so it asksNativeZstd.rs, which takes them from the crate.Build:
bun_sema_standalonehad a build script that compiled the C decoder fromvendor/zstdinto its test binaries. It is gone, with theccbuild dependency and theclone-zstdtarget of thecargo teststep (scripts/rust-test.ts), which now testsbun_zstdtoo. What stays:jscandtestaot, WebKit's own programs beside Bun, read the compressed data of a prebuilt WebKit's ICU with the C decoder, which since #47316 they compile for themselves from zstd's sources (JSCProgram.sourcesFrom).scripts/build/deps/zstd.tsis kept to fetch these sources where one of them asks; it builds nothing, and it is not among Bun's dependencies but in a list of its own,programDeps, so nothing of it reaches Bun's compiles or link. TheZstdCompressorfeature of--trimlisted the C entry points; it now names the crate's compressor by path (cxx-function bun_zstd ^<?bun_zstd::compress), since the macros that mark Rust entry points needunsafe.What changes for users
process.versions.zstdis"1.5.7"instead of a commit hash. There is no commit to name.node:zlibonly lets the documented parameters through, but_handle.init()called by hand with a longer array could set experimental ones. Those are now refused (ERR_ZSTD_PARAM_SET_FAILED).zlib.createZstdCompress()stream withZSTD_c_nbWorkers >= 1lost its data as soon as the input was enough for one job (Bun 1.4.4 ends such a stream with 0 bytes, without an error). With workers,ZSTD_compressStream2(ZSTD_e_continue)returns after any progress, andzlib.tstakes room left in the output to mean that the input was consumed.NativeZstd::do_worknow calls it until one of the two is used up. It is in this PR because it is what makes the multithreaded code reachable from a stream at all.zlib.createZstdDecompress({ dictionary })andzlib.zstdDecompressSync(..., { dictionary })with a dictionary that cannot be loaded gave empty output without an error. They now throwERR_ZLIB_INITIALIZATION_FAILED, as Node does.new DecompressionStream("zstd")failed after 16 empty chunks in the middle of a frame.1e789521e, 65,536 in rows (after a frame that is given up at level 12, the next one waited 254 to 324 ms, as in C, now 19 to 35).fastanddfastindex theirs first: up to 5 ms, or 1.6 s with a table of 2^27 entries. 30 streams at level 19 with 4 workers that are dropped in the middle of a frame leave at most 8 threads and 380 MB behind, for 0.04 s.zlib-estimated-size-gc.test.tshad the old size in it.bun build --compile --aotof a hello world maps on Linux x64 (test/bundler/compile-aot-size.test.ts), at46048ba7a(Buildkite #138405) againstmainat the commit it has merged (#138398): 67,934,305 bytes instead of 67,672,289 (+262 KB, limit 68,440,000), and with--trim25,397,345 instead of 25,331,937 (+65 KB, limit 25,880,000). The loops that find matches are compiled once for each size of table that a level comes with, which is what makes them as fast as they are, and the optimal parser once for each thing that can come before the input.c7888487and9b6d3303took out the 156 KB of instances that bought nothing.Speed
All times in this text but those of this paragraph are from one kind of processor, AMD Zen 4. On an Intel Xeon of the Skylake family the port compares less well. The same binaries of
fc13be22b, 25 kinds of data from 1 to 512 KB, relative to C: to compress at levels 1 to 13, 16 and 19, 0.96 in the mean, and 437 of 1,500 rows are more than 3% behind (Zen 4, levels 1 to 12: 0.91, 116 of 1,200). To decompress into a buffer that is there 0.97, 369 of 700 behind (Zen 4: 0.87, 107 of 682); into a new one 1.01, 457 of 700.fc13be22bitself is mixed there: records of 100 bytes take 2 to 5% longer, 16 rows with a dictionary 1 to 4%. Bun's x64 build is for Nehalem, so the port has no BMI2, which C picks at run time. Built with it (4 orders): to compress 0.92, 244 behind (skewed bytes, level 1: 1.40, then 1.05); to decompress 0.94, 311, and 0.97, 386. Open.Latest measurement,
9b6d3303againstmain. The survey that the next paragraph describes (266 rows) was run again with a better baseline:mainate64b59d9, which has the C library, built on the same machine with the same profile as this branch, so that nothing differs but zstd (Bun 1.4.4, the baseline of the older numbers below, also has another WebKit). Two passes of each binary in turn, the better time of each row counts. Time of this branch, as a share of that ofmain, geometric mean: 0.90 to compress and 0.95 to decompress. 54 rows to compress and 73 to decompress are more than 3% slower, 144 and 139 more than 3% faster. By size, 1 KB to 8 MB: 0.92, 0.94, 0.93, 0.87, 0.84 to compress and 0.96, 0.98, 0.92, 0.94, 0.95 to decompress. By level, 1, 3, 9, 19: 0.91, 0.91, 0.88, 0.90 and 0.96, 0.96, 0.95, 0.94. What is said below about the rows that are slower still holds.Correction to four rows of this table. To decompress zeros, noise or noise with a period is little more than a copy, and in Bun such a call takes one of two times from one pass to the next, with either binary. Two passes were too few for such rows, in either direction. These four kinds were run six more times with each binary, in turn, and their rows above are now the best of all eight passes. With them the means of all 266 rows are 0.90 to compress and 0.94 to decompress, 54 and 75 rows are more than 3% slower, 142 and 141 more than 3% faster.
What this survey did not see, fixed in
41fad94a3. It has four levels and no data made of records. With records that differ in one field the port took 1.3 to 1.9 times the time of C on 16 KB at levels 6 to 8, and up to 2.2 times on 1 MB at level 12. A search beside a match that the parser already has first looks at 4 bytes of a candidate, where a match that is good enough would end, which is a shortcut of the port. Here that is past the field, where all records are the same again, so every candidate passed and was compared in full, up to 64 for each search. The first one that turns out too short after 16 bytes or more now moves the 4 bytes to where it ended, which is where the C code has them. In the harness, time relative to C before and after: 100 bytes and a counter, 16 KB at level 8: 1.90, 0.87, 1 MB at level 12: 1.86, 1.04. It has a price: text, JSON, logs and binaries of 16 KB at levels 5 to 8 take 1.0 to 1.7% longer than before (0.70 to 0.95 of C).All levels on 25 kinds of data, and what that found in the decoder, fixed in
605aa6e57. At41fad94a3, in the same harness (not in Bun): 25 kinds of data, four sizes from 1,000 to 1,000,000 bytes, levels 1 to 19, 1,900 rows. The output is that of C in every row. The geometric mean of the time relative to C is 0.900 to compress and 0.915 to decompress; 335 rows to compress and 523 to decompress are more than 3% slower than C. Slowest to decompress were records, at 1.9 to 2.45 at some levels: a bug. A block with few literals has them in a single Huffman stream. A valid stream of 129 to 183 symbols of about a bit each suited neither part of the quick decoder for that and went to the one that follows the C code step by step, which fills a table of its own each time (8 us a block). Records of 1 KB, 1 MB at level 17: 2.45 of C before, 1.22 after. The new loop was compared withHUF_decompress1X_usingDTableof the C library on 196 million streams, most of them damaged, without a difference. What is left, 1.1 to 1.3 to decode data that is nearly all long matches (records, HTML, short periods), is open; a good part of it is that safe code clears the room for a block before it writes there.128 KB at level 10, fixed in
07e8c4e6a. Records took 1.2 to 1.4 times the time of C there, from something the port has and C has not. Rows of 64 entries get a second tag per entry if the input is at least as large as the hash table, which is the case for exactly 131,072 bytes at level 10, from 4 MiB at level 11 and from 8 MiB at level 12. It costs time for every position, and whether it pays was asked at the end of a block, so an input of one block paid all the way. The parser now stops every 8 KB while there are second tags and goes on from where it was, with them or without. In the harness, relative to C, before and after: records of 200 bytes 1.44, 1.08; of 100 bytes 1.33, 1.03. Where the second tags pay, the stops cost up to 2%. Open: on 5 to 17 MB at levels 11 and 12, never giving up the second tags is 8 to 25% faster on source text and mixed data, and 12 to 30% slower on HTML, numbers and records.128 KB at levels 11 and 12, and the room for the literals, in
9778f063candb06f15db1. 21 and 19 of the 25 kinds were more than 3% slower than C there.btlazy2now gives hints at what the trees are about to read only after the first block, and the loop that adds the positions that a match skipped reads 4 bytes where 4 count. Every new context cleared 131 KB for literals: now 4 KB ahead of them, but for levels 3 and 4, whose loops are 3 to 6% slower that way; open. Relative to C, 25 kinds of 128 KB: level 11 1.087, now 1.031; level 12 1.081, 1.036; level 1 0.965, 0.924.The ways down the trees, in
dd2ac327cand8f8c6f481. A step down a binary tree compared 16 bytes, looked for the first that differ and read these two again. The 16 bytes as two big-endian numbers tell the way without that wait. And the optimal parser gives fewer hints while it adds a run of 32 or more skipped positions. Relative tob06f15db1, 16 KB to 1 MB at levels 11 to 22: 0.962 to 0.986. Relative to C, 128 KB: levels 11 and 12 1.000, levels 13 to 15 0.956 to 0.998. Of 548 rows none takes more than 2% longer.Matches in the older segment of the window, in
d0d121dc9. With a dictionary whose tables are copied, or a stream longer than its window, the trees lead to the older segment, and each position there was compared out of line: 58% of the time for records of 1 KB at level 16 with a dictionary of 64 KB. That function now starts with 16 bytes at once. 352 rows: means of 0.96 to 0.99, my records 0.75 to 0.87 at levels 13 to 19, none slower when measured again. Relative to C, 64 KB and 128 KB: 1.038 before, 0.996 now, records still at 0.89 to 1.38.Rows with a dictionary that is searched in place, in
8aec2cdde. With a dictionary and inputs of up to 32 KB, levels 6 to 9 took 1.06 to 1.47 times the time of C on data with few matches. A position whose row has no candidate was searched all the same, for the sake of the dictionary's row: that row is now looked at first, and searched alone if it has one. And in these functions, the largest, the loop that inserts positions read its operands from the stack on every pass: it now has copies of them. 396 rows (11 kinds, levels 5 to 10): 0.83 to 0.94 of the time before by level, no row slower; of C, 0.65 to 0.90, from 0.70 to 1.05. 58 rows are behind C, by 5% at most (hex digits, skewed bytes), where 185 were, by up to 47%.The optimal parser with such a dictionary, in
306afc849, and how time is measured from here on. The parser's loop for positions that follow literals and have no match returned at once if there was a dictionary to search in place, and levels 13 to 19 took 1.15 to 1.36 times the time of C on data with few matches. It now searches the dictionary's tree too. Noise, base64 and skewed bytes: 0.64 to 0.84 of the time before, 0.78 to 1.03 of C. Of 286 rows 111 were more than 3% behind C, 61 are. Two builds of one source that differ only in the order of the functions (--shuffle-sections) differ by 0.5 to 2% in most rows, 5 to 10% in some and up to 35% at level 1, so these are means of 16 orders on each side, judged by their standard errors. That is why the parsers for such a dictionary are copies (19.7 KB): one parser that chooses makes 65 of 150 rows without a dictionary 0.4 to 1.4% slower.The same loop while a repeat offset leads into the dictionary, in
666b76602. After a match in the dictionary the loop returned at once, in 86% of the calls. Now such an offset is looked up there. 32 KB at level 13: skewed bytes 0.57 of before (1.45 of C, now 0.83), base64 0.72 (1.12, now 0.81). 11 more kinds of data, after a review: noise with pieces of the dictionary is at 0.53 to 0.79 of before (1.25 to 1.68 of C, now 0.88 to 1.00). Slower, of 603 rows, at more than 3 standard errors: 17 of copies of the dictionary with one byte in 8 to 32 altered, by 0.5 to 1.7%, and 4 others by 0.3 to 0.7%.And into an older segment of the window, in
55c964301: a dictionary whose tables are copied, or what a stream has before the place where its buffer starts again. Noise with pieces of a copied dictionary: 0.60 to 0.86 of before (1.25 to 1.71 of C, now 1.00 to 1.08). Streams longer than their window (204 rows, levels 16 to 19): 0.95 of before. 5 of 139 rows with a dictionary are 0.2 to 0.7% slower. 22 KB more code.Positions that are added to the binary trees are compared 8 bytes at a time, in
de7ed9c89. HTML and records were furthest behind C. With a copied dictionary: HTML 0.89 of before (1.05 of C, now 0.93), records of 1 KB 0.87 (1.29, now 1.12). Of 1,206 rows, 283 were more than 3% behind C and 218 are; 4 are 0.1 to 1.2% slower. 223 bytes more code.Where something comes before the current segment, matches are looked for in copies of
get_all_matches, inf9f4dec21. A repeat offset that leads there is looked at before it is counted, and a step down a tree into an older segment makes no call. With a copied dictionary: records of 100 bytes 0.81 of before (0.97 of C, now 0.80), of 1 KB 0.88 (1.14, now 1.01). Of 1,031 rows, 197 were more than 3% behind C and 116 are; one is 0.2 to 0.4% slower, and 3 of 54 more rows 0.4 to 0.5%. 12 KB more code.A position that is added to the binary trees does not wait for the one before, in
209068dfd. How far to move on came from what the walk had met, a chain from each position to the next that C has too; now 1, which it nearly always is, is a branch. 16 KB of records of 100 bytes at level 16: 0.82 of before (1.14 of C, now 0.93). Of 1,206 rows, 144 were more than 3% behind C and 97 are; none is slower. 2 KB more code.The decoder where it takes one sequence at a time, in
fc13be22b. That loop, for long matches and small blocks, had its state in memory, because a call further down takes its address: now it has a copy, in registers. HTML 0.95 of before (1.08 of C, now 1.03), records of 100 bytes 0.97; of 682 rows into a buffer that is there, 123 were more than 3% behind C and 107 are. With more than 16 MB of history it asked for no match ahead of time: now for 8, as C does. 19 frames of 48 MB, not timed before: 1.21 of C in the mean, now 1.05; pieces of 40 to 330 bytes from far back 0.64 to 0.79 of before. Slower: source code three times over with long distance matching 3.7% (0.93 of C), 512 KB with a period of 1 KB 2%, two more rows 1 to 2%. A dictionary in a new context, 356 rows, not timed before: 0.93 of C, unchanged. 5.7 KB more code.Dictionaries at levels 5 to 10, in
1e789521e. To load one of 8 to 512 KB and compress 1 KB had not been timed: 1.12 of C over levels 5 to 12, 68 of 88 rows more than 3% behind (Intel Xeon: 1.08, 69). The loops that index it in hash chains and in rows checked every position. Now 0.89 of C and no row (0.93, 7); levels 5 to 8 take 0.74 of the time (0.80), no row longer.The tables further down are for two files, source text and an executable, and they do not hold for every kind of data. A wider survey (14 kinds of generated data, 1 KB to 8 MB, levels 1, 3, 9 and 19,
Bun.zstdCompressSyncandBun.zstdDecompressSyncagainst Bun 1.4.4, best of 24 rounds, 266 rows) showed that atefbf33ddthe port was slower than C on average: the geometric mean of all rows was 1.065 to compress and 1.077 to decompress, the worst rows 6.1 and 2.4. As ofd106dc84it is 0.93 and 0.96, the worst rows 1.9 and 1.6, and 85 rows to compress and 87 to decompress are still more than 3% slower. Rows that run at the speed of memory depend on what the process has allocated before: measured alone in a new process, five of the worst rows to decompress (1.30 to 1.56 in the survey) come to 1.00, 1.07, 1.08, 1.14 and 1.24, and the two worst to compress after the 1 KB of zeros (1.39 and 1.44) to 1.08 and 1.11. So the port is not yet faster than C everywhere, and this is what is being worked on. Time relative to C by kind of data (geometric mean over sizes and levels, and the worst row):efbf33ddd106dc84efbf33ddd106dc84What is known about the rows that are slower: level 19 on 1 KB of zeros (1.9: tables that are cleared) and on 1 to 16 KB of integers or of mostly zeros (1.15 to 1.2); 8 MB of hex digits at level 9 (1.15, also when measured alone); literals with long Huffman codes, to encode (skewed bytes, base64 and 4 letters at levels 1 to 9: 1.07 to 1.14) and to decode (1.09 to 1.15: C has assembly and BMI2 for that); inputs of 1 KB at level 1 (1.1 to 1.25, of 1.5 us); integers to decode at levels 1 to 9 (1.1 to 1.25); mostly zeros of 1 MB and more to decode (1.1). In the harness, where both sides get the same buffers, 1 MB of noise, of zeros and of either period decode in 1.00, 0.78, 0.85 and 1.00 of the time of C.
What
d106dc84and the six commits before it changed. To compress: long matches are counted 64 bytes at a time, and at levels 13 and up the positions of a stretch without any match are gone over in one loop (128 KB of noise at level 19: 0.85 of the time of C, from 1.25). To decompress: what is appended to a large new buffer is copied 2 KB at a time, because glibc'srep movsbis slow to write to pages for the first time; and short matches from less than 16 bytes back have an instance of the loop for common sequences of their own, whose test costs other data 0.5 to 2.5%.e48dc2dc: that loop lost sight of the dictionary in every frame but the first of a call.39b1e9fa: blocks of more than 64 KB of literals, some of which C keeps in the output buffer, were decoded one sequence at a time; now the loop stops short of where those literals would be.What
0cfb03b5and the two commits before it changed: a block asks for room in the output when its sequences are known and not before, a block that is stored is appended, and a block without a sequence is neither copied to the store of literals nor, if it is seen to stay as it is, written as raw literals first. Noise of 128 KB and more now compresses in 0.6 to 0.96 of the time of C at levels 1 to 9 but for one row (1.08). Blocks that may be split (levels 16 and up) or have a target size are left as they were, on purpose: they spend their time elsewhere.What
a87cc412and the two commits before it changed: literals whose codes are long are decoded one to a lookup, from a table chosen by how many symbols a lookup would yield; at levels 16 and up a position without a match takes less, and the tables of the parser are as long as a block, not always 4099 positions. Rows measured again with the binaries in turn showed no row slower than before.What
5a031e0afixed: glibc 2.39 copies more than 2 KB at 3.5 GB/s instead of 57 to 97 on Zen 4 if the destination is 1 to 15 bytes further into its page than the source (BZ 30994), which the port always hit with literals that are stored, since mimalloc aligns its buffers to pages (the C library hits it too, with stored blocks of an input that is aligned to a page: 49 us for 128 KB of noise, the port 87 before and 16 now); the output of one call was zeroed before it was written; and the counts of a table were read with 13 cycles of latency each.The aim is to be faster than the C library everywhere. As of
efbf33dd, in Bun itself, for these two files, for inputs of 3 MB and more it compresses faster at every level measured, by 3 to 22%, and decompresses faster, by 1 to 11%, except what level 1 made of 15 and 20 MB, where the two are even (1.00). For smaller inputs it depends on how one measures. With a different input for every call, which is what a program does, it is faster in 31 of the 32 cells measured (700 bytes to 100 KB, levels 1 to 9, both directions), by 2 to 15%. With the same call repeated thousands of times it is faster from level 6 on, and 1 to 4% slower for 4 KB and 100 KB at levels 1 and 3 to compress and for 4 KB to decompress. The C library has hand-written assembly for Huffman decoding on x64 and picks BMI2 kernels at run time; the port has neither.The numbers that were here before
0a40fdcfwere too good. They came from a harness in which rustc generated the crate's code. A release build of Bun leaves that to ThinLTO in the link, and the same source comes out differently there. One difference was large: the comparison of the 64 tags of a row (levels 11 and 12) was no longer a comparison of vectors (281 instructions instead of 31), so that in the real binary level 12 took 1.10 (text) and 1.20 (executable) times as long as C, where the harness said 0.81 and 0.93. That is fixed, no other function has lost its vector code, and the harness now builds the crate as a release build does (bitcode from rustc, ThinLTO at level 2 in lld, the standard library from source). The first tables below are of Bun itself.Bun.zstdCompressSyncandBun.zstdDecompressSyncin Bun 1.4.4 (C) and in a release build of this branch, so with the allocation of the output. One thread, x64 (an AMD EPYC), built fornehalemas Bun is, MB/s; the last column of each group is the time of the port relative to C (below 1 is faster). Bun 1.4.4 is the official build and the other is a local one (same flags for the crate; the C++ around it is not under LTO). At901a9d9athe same local build ofmainas of the merge base, with the C library, was within 3% of Bun 1.4.4 in every row of the table of smaller inputs, mostly within 1%, so the comparison is fair. These are the best of 4 to 48 calls (the more the faster the level) and of 32 for decompression. The libraries alone, further down, are as ofe5c4c3cf. That many are needed: with buffers of 15 to 20 MB a call takes up to half as long again, in either binary, when the allocator has to get pages from the kernel, and with fewer calls it is luck which of the two is seen at its best. At levels 1 and 3 the medians give the same ratios to within 0.02.15 MB of TypeScript and JavaScript sources:
20 MB of an executable:
Smaller inputs (text), time relative to Bun 1.4.4. A different input for every call, 150 to 3,800 of them in turn, taken from all over the 15 MB:
The same call again and again:
Safe code can only write to memory that has a value, and the room that a frame can take is as large as its input. Up to
e5c4c3cfall of it was made zeros first, which at level 1 on the executable cost as much in the kernel as the port was ahead in user mode (20 against 15 million cycles per call, of some 160 million). Now the room is the capacity of aVecwhose length grows, with zeros, ahead of each block (15 million).One context per thread for
Bun.zstdCompressSync, kept from call to call, would save 0 to 7% for inputs of up to 100 KB (7% at 700 bytes, 4% at 4 KB, 1% at 100 KB, at levels 1 and 3) (measured in the harness, with the same bytes out). That is not worth memory that a thread never gives back, so each call makes its own, as with the C library.The libraries alone, in the harness, into buffers that are there already. Correction: the C library here is built from the same pin, but with
-O2, where Bun's release build has-O3, which I had wrong. With Bun's flags it compresses JSON 5 to 6% faster at levels 1, 3 and 19 and data without matches 7 to 13% faster at level 19, is as fast on literals alone, and decompresses 4% slower, so the compression ratios in these tables flatter the port by as much. The tables and the survey above, which are against Bun 1.4.4 itself, are not affected. The harness now has both builds.15 MB of TypeScript and JavaScript sources:
20 MB of an executable:
Differences of less than 2% are not to be trusted either way: moving the hot functions of the decoder by 32 bytes changes as much. The list that follows was measured before the harness was built like a release build, and has not been measured again.
Where C is still ahead: inputs of up to 16 KB at level 11, where the optimal parser starts (0 to 14%, half of it from clearing its tables), and of 32 to 128 KB at levels 11 and 13 (up to 7%); random data at levels 13 and 15 (about 30%) and 19 to 22 (5%); very repetitive data at levels 13 to 16 (6 to 8%); a dictionary that is copied, on binary data at level 11 (4%); small JSON at level 9 (1 to 3%); the match finder of levels 3 and 4 on log files and JSON (4 to 7%, even for the whole compression); data that is mostly literals, where the Huffman encoder is 15% behind (5 to 7% overall); some synthetic data at levels 9 to 19 (8 to 16%); in the decoder, frames of 30 bytes with a dictionary (4 to 6%), very repetitive data with matches of a few hundred bytes (3 to 21%), random far matches (10 to 15%), a stream of frames with a window of 128 MB on one decoder (10 to 16%), and more than 16 MB of mostly zeros decompressed in one call (up to 30%, since the output is cleared before it is written; even on real data). These numbers hold for the pinned toolchain: with an older LLVM (21), which does not vectorize the comparison of 16 bytes in the binary trees, levels 13 to 19 take up to 3x the time of C on data with very long matches (with the pinned one, built like a release build: 0.71 to 0.98 on the same files).
Where the port no longer follows the C code line by line, to be faster with the same result:
Vecthat grows: blocks that are stored or one byte, and literals and matches of 256 bytes or more that do not fit in what is there, are appended and never zeroed; sequences are decoded 128 at a time into an array and then executed; a small output is decoded where there is room to copy in chunks up to its end, and then copied out; a Huffman lookup yields up to three literals (one, in a simpler loop, where codes of up to 11 bits are so long that few lookups would yield more), and a stream that does not end exactly where it should is decoded again by a second decoder that does what C does, step by step, so that damaged input gives the same bytes and errors; matches in a dictionary or in the wrapped part of the window are handled by the common path.Vec: a block asks for room, which has to be zeroed, once its literals and sequences are known, which bound what they take; a block that is stored is appended. A block without a sequence reads its literals where they are.usizepositions (a bounds check that the compiler can remove, or that is one comparison), tables indexed through a mask, match lengths counted 16 bytes at a time.Buffers whose size a frame header or a parameter dictates are no longer cleared when they are allocated. They grow as they are written: with mimalloc, clearing a recycled window-sized buffer for a stream of a few bytes was by far the largest cost of such a stream.
bun_zstdis built withopt-level = 3in debug builds too, as the C library was; atopt-level = 0tests time out.bun_podgetstry_zeroed_vec, a falliblecallocfor aVecofZeroableelements, which is how the crate allocates its tables withoutunsafeof its own.How did you verify your code works?
Against the C library. A harness (not in this PR, it needs the C sources) links the C library, built from the same pin with the same patches and flags, and the crate into one program and compares them on random inputs: every return value, every position, every output byte.
No difference is known other than those listed below. Each push is checked with tens of thousands of iterations of this harness (the push of the rewritten decoder: 48,000 decoder iterations, each of which decodes one set of frames about 20 ways, after 220,000 while it was written; 12,000 compressor iterations, 4,500 with dictionaries, 250 with workers).
Fuzzing, which goes on while this PR is open, in lockstep with the C library:
lib.rs, frames built from sequences that the fuzzer chooses, and the entropy coders called directly: about 50 million executions. The one for the compressor has a slow, large corpus, and from306afc849to1e789521edid little more than replay it (2,200 new inputs a round); split up on 2 November.decodecorpus. The decoder's corpora were also run against a C build without assembly or BMI2, which gives the same as the one with.from_raw_parts:frame_extentnever asks for more bytes than the frame has.Line coverage of all of it, measured with a counter in every basic block: 6,398 of the 6,854 lines with code (93.3%; an earlier, coarser measure said 95%). Half of the rest are headers and braces of functions that are always inlined. 196 of the lines were then probed one by one: 67 were reached after all (equal
return Err(..)blocks are merged), 129 never. What of that can be reached has a tool of its own now, each compared with C: indices near their limit (about 150,000 times that they are brought down, every strategy, with long distance matching, a dictionary in reach, an older segment), the encoder's entropy functions called one by one, sub-blocks, a block that the splitter cuts with a raw part in the middle, dictionaries of 16 MB and more and dictionaries that are wrong in one place, and inputs, sizes told and skippable frames around 2^32 bytes. The tools were checked with 38 faults planted in the port: 33 were seen; the other 5 change nothing, because a check right after does the same, in C as well. What is left is behind a check that a caller has made before, as in C.It has found no panic and no hang, and one difference in output, fixed in
dd89d7876:ZSTDMT_getCCtxtakes the context that was given back last, and a job gives its context back before it reports its end, so frames of one job, a stream that is flushed after each write, and anything with a single worker go through one and the same context. What that context has been through shows in where its indices stand, which decides where they are brought down, which lets go of an attached dictionary. It takes a context that lives for 3.4 GB, then a frame with a dictionary that is attached and still in reach 16 MB into the frame (a window of 32 MB or more, a first job of more than 16 MB). One case: one worker, 200 MB, a largerhashLog, 3,264 MB, then 60 MB with a dictionary: 41,784,989 bytes in C, 43,041,965 here. The contexts are now in a pool as in C, and keep the books of the C workspace like any other context. Checked against the unmodified C library: those cases and the lengths around them with 1, 2 and 4 workers; about 1,100 iterations at random on copies of both libraries whose indices start high (parameters and the number of workers changing between frames, many small jobs, workspaces that C replaces as too small and as too large for too long, the latter counted: 60 of 60), in 300 of which the parameters are held so that indices are brought down in the last frame, with the dictionary in reach, about 4 times in 5. 52 runs of 3.8 GB through one context of the libraries as they are (every strategy, 16 with workers) showed no other difference.It has found performance bugs, which are fixed here but for what the last item calls open:
targetLengthat its maximum on repetitive input.dd89d7876what prevents that is what prevents it in C: small jobs all go to the context that had the first job of the frame. With a dictionary that job has the small tables of the dictionary, so a flush that is larger than all before it does clear the tables, in both. 300 flushes of 100 to 30,000 bytes with a dictionary of 100 KB and one worker take 8 to 11 ms at level 3 (C: 12 ms), 290 ms at level 19 (20 ms before that commit; C: 15.7 s) and 110 ms at level 22 (87 ms; C: 148 s).41fad94a3). Most of what they showed at first was its compiler, an older one, in which the comparisons of 16 bytes at a time are not turned into vector instructions. With the pinned toolchain three kinds are left above 1.25 times: binary trees atbtlazy2on data with a short period (1.3), the optimal parser with an attached dictionary and very small tables (1.4 to 1.6), and long distance matching with aldmMinMatchof 12 to 15 and aldmHashRateLogof 1 (1.3 to 1.6). None is at the parameters of a level. They are open.And one bug that only shows when an allocation fails: the block splitter took that for an estimate that failed, which means no split, so the frame was valid and success was reported, but the bytes were not those of C. It is an error now.
It has also found two crashes in the C library (so in Bun 1.4.4, if anything there gets to them; I have not tried). When a frame with workers is given up while a job waits for a free worker,
jobReadystays set, and the next frame on that context posts a job description that has been zeroed, which a worker dereferences; I have checked this one against the source. The other, which I have not checked myself: after a frame is given up, raisingnbWorkersfor the next one resizes the pool before the old jobs are waited for, and they use memory that has been freed. The port has no such state.One more thing in 1.5.7 that the port does as well, found on the way: there are inputs, made on purpose, that a stream at level 22 cannot compress. One block of them is split into so many parts, all of which are stored, that they do not fit the stream's own output buffer (
ZSTD_compressBoundof a block), andZSTD_compressStream2fails withdstSize_tooSmall;ZSTD_compresswith the bound for the whole input has the room. Both do the same here as in C. Only levels 16 and up split blocks in that way.What is not the same as 1.5.7, on purpose. The rule: where 1.5.7 reports success for something that breaks the format or the API's contract, and what it gives is not defined, the port reports the error.
mallocreturns. The port gives what C gives when that memory is zero.corruption_detectedfrom that point on. Decoding in one call, where all of the output is still there, is as in C.corruption_detectedin both.srcSize_wrong, as both do without workers.ldmHashRateLogabove the window log, 1.5.7 makes a table of 8 GiB because a subtraction wraps around. The port does what upstream does sincecef5a5611(March 2026) and makes the smallest table, which can give other output than 1.5.7 for those parameters.ZSTD_decompressStreamhas returned an error,zstd.hcalls another call undefined behavior, and 1.5.7 goes on with sizes that no longer belong together. The port returns the same error again until the stream is reset. Only the two errors that tell of a lack of progress leave the stream as it was, in both.zstd.hsays that what starts a new decompression job,ZSTD_decompressDCtxamong them, resets the context. 1.5.7 does not reset the stream: one that was under way goes on afterwards with what the other job left. The port does what the header says.ZSTD_compressStream2has failed in the middle of a frame, which the header calls undefined as well, 1.5.7 goes on with the frame if it has no workers (later calls can succeed) and starts a new one if it has. The port returns the same error from then on and lets its workers go. Their contexts go with them, where C keeps its own, so after a reset of the session the indices of the next frames start again: that shows in the way described under Fuzzing, 3.4 GB later (seen in 3 of 4 seeds). The same holds for a frame that is given up with jobs under way, which the port stops at their next block; 1.5.7 does not get over that (see the crashes above), so there is nothing to compare with. Nothing in Bun uses a context again after an error, or mixes streams and whole frames on one.One more thing can show, with frames that no encoder makes. A raw or RLE block larger than the frame's largest block size is accepted by 1.5.7 when a frame is decoded in one pass and refused when it is streamed, and a streaming decoder takes the one pass if the frame says its size, all of it is there and there is room for all of its output. The port does all of that. But the decoder of
fetchresponse bodies no longer offers the decoder all the spare capacity of its buffer at once, so such a frame of more than 4 KB that used to get through, depending on that capacity, is now an error. The other way round, a frame without a checksum that ends with an empty last block before it has made as many bytes as its header says is refused in one pass and accepted when it is streamed, by 1.5.7 and by the port alike. Such a frame that says more than the room it is offered (4 KB to 256 KB), and that used to be refused if the spare capacity happened to be enough, now gets through. Two more answers of 1.5.7 depend on that room in the same way, and the port gives them call by call. The limit on the window (128 MB) is only checked when a frame is streamed, so a frame with a larger window that says a small size, which is legal but which no encoder I know of writes, used to get through if the spare capacity was enough for all of it, and is now refused if it says more than the room it is offered. And a frame that makes more than its header says is refused at the block that does not fit or only at its end, depending on the buffer that an earlier frame left in the decoder: if such a frame is also cut off, the error offetchcan now be that the body ended early instead of that it is corrupt.The differential harness now compares with two builds of the C library: the pin, and the pin with these changes made to it.
zstd's own history. All 4,141 commits that touched the library or its tests, from 2015 to upstream's
devof October 2026, were gone through for bug fixes, each of which was checked against the Rust code as it is now and, where a reproducer can be made, run on both sides. The four CVEs: CVE-2019-11922 (one-pass compression to a buffer below the bound) is not there: the check that fixed it is, and compressing and decompressing to every capacity from 0 to the bound gives what C gives (1.5 million calls); CVE-2021-24031, CVE-2021-24032 and CVE-2022-4899 are in the command line tool, which is not ported. The last four differences above come from this, and so do the two smaller bug fixes for users. For index overflow correction, both sides were also built with the limit at 48 MB instead of 3.5 GB, so that it happens every few frames: about 200 runs over every strategy, dictionaries, long distance matching and workers gave the same bytes. Parts of the crate were rewritten for speed after the commits that concern them had been checked, so those findings were checked again against the code as it is now: of 386 findings, 335 concerned code that had changed; 330 were read again and 236 run again. That found three bugs that the rewrites had brought in, all fixed: after 3.4 GB through one context and a change of parameters, an attached dictionary in a window of 32 MB or more was let go of at another place than in C (valid output, other bytes); a context whose tables could not be allocated went on with stale indices in the tables it had (a frame that does not decode, after 3.4 GB through that context); and a stream with workers could hang after a thread had failed to start. The 28 findings about the binary trees and the optimal parser were checked last, on that code as it is now: none is a bug. For these, 31 copies of the crate were made with one fix taken out of each, to see whether the runs that count as evidence notice: seven went unnoticed, so six reproducers were written that do notice them. Four of those are now tests ("the optimal parser"); the other two need a context that changes its dictionary, which no API of Bun does, or the build with the low index limit. Limits of this review: nothing above 4 GB in one call was run with the binary trees at the real index limit, and about 340 of the oldest commits were dismissed by their subject line alone.In this repo.
New in
test/js/bun/util/zstd.test.ts, "the output is that of zstd 1.5.7": size and hash of the output at every level, with 14 sets of parameters, with a raw and with a trained dictionary at 8 levels and 2 sizes, and in a stream with flushes. The expected values were recorded with Bun 1.4.4, which has the C library at the same pin, so these tests pass there too; that is the point of them."in a stream with workers" fails with Bun 1.4.4 and passes here.
"in a stream with workers, where a flush does not cost as much as a whole job" and "where a larger job than those before does not clear the tables" are for the performance bugs above: they timed out before the fixes. Bun 1.4.4 passes them.
"bugs of zstd 1.5.7": one test for each of the last four differences above. The size of the table of long distance matches does not show from JavaScript, so that has a unit test in the crate as well, which
cargo miri testnow runs."bugs that zstd has had": 87 tests from the review of the history (frames that were once decoded wrongly or crashed, inputs that were once compressed wrongly, parameter bounds, dictionaries, workers). The expected values are those of the C library (Bun 1.4.4, and Node 24 where Bun 1.4.4 has a bug of its own), so 81 of them pass on Bun 1.4.4; they are there because the port does not follow the C code line by line.
"few literals whose codes are long": three tests for the Huffman decoder's loop of one literal to a lookup. While that was being written it read one bit too many of a stream whose last byte is 1 and whose first seven codes have 8 bits, and gave a wrong byte without an error; the differential harness found that in 1 of some 20,000 frames, the first of these tests finds it in its third. Bun 1.4.4 passes them.
"literals that take many bits": three kinds of bytes at six sizes, decoded both ways, with how the literals of each block are written pinned, so that blocks that go on with the code of the block before are known to be among them. A unit test in the crate drives the tables directly: codes of 8 to 12 bits, blocks of 200 to 16384 literals in one and four streams that go on with one code, every change from one kind of table to the other, damaged streams compared with the exact decoder.
"where a block is one match, which is too short for the optimal parser to take at once" reaches the last position of the parser's tables: with one position less it fails. A unit test compresses a long, a short and the long input again on one context. Both, like most tests here, hold values from Bun 1.4.4 and pass there.
"where blocks are nothing but literals", in one call and streamed, at levels -5 to 16. A unit test has literals that start and end like noise with something compressible between, which the C library stores, at the capacities around the size of the frame: a review found a first version to compress them, or to succeed where C fails, when there were fewer than 7 bytes to spare.
"where blocks that are stored are between blocks that are compressed", with values from Bun 1.4.4, and a unit test in the crate (not under Miri, where it takes over 10 minutes) that compares room that grows with room that is there, at five levels and five capacities. The first version gave a block that is made in parts (levels 16 and up) room for 197 of them. A review made two inputs of 35 MB for level 22 that need more, on which it failed or gave other bytes than C; 100,000 frames of the differential harness had not come near that. Such blocks now get all the room at once, as before.
"short matches from less than 16 bytes back": noise with copies planted 1 to 15 bytes back in every, every second or every eighth sequence, decoded in one call, as a stream, in a window of 1 KB, and after a dictionary that the first two matches reach into. With the new loop made to skip a match that reaches before the output, it fails. "where matches are far between, for the optimal parsers" has values from Bun 1.4.4 for the loop that goes over what has no match.
Unit tests in the crate, which
cargo miri testruns: three for the contexts of workers (the indices of a context start again where C makes a new workspace: too small, too large at the 129th reset and not at the 128th, and never if it is as large as it takes; the pool gives out the context that was given back last and keeps no more than there are workers; with 4 workers, jobs that come one after the other have the same context, which stays when there are fewer workers and goes when there are more than ever), each of which fails with one of six faults planted, among them a context that is given back after the job has reported its end; every offset and length of the copy for short close matches; matches counted up to where they end, around the 64 bytes that are compared at a time; long copies in parts; three for streams after an error, each clause of which was taken out once to see the test fail; and two for matches in a dictionary from frames that are not the first of a call, one of which calls the loop for common sequences itself and fails with the rule that was there before. "frames one after the other whose matches are in a dictionary" does the same from JavaScript. One more has blocks made by hand with more than 64 KB of literals, in one call and in a stream whose buffer has wrapped: valid ones, blocks that make too much, and blocks whose 125th sequence the C library refuses because its output catches up with its literals; the results were checked against the C library, and each of the four conditions of the change was taken out once to see the test fail. "blocks of more than 64 KB of literals" has four cases from JavaScript (matches all over, only at the start, in the older part of a stream's buffer, in a dictionary)."where records differ in a few fields": 16 cases with values from Bun 1.4.4, for the hash chains and the rows, with second tags, with a small window and with a dictionary; which of them each case reaches was checked with counters in a copy of the crate. Like the rest of that block it holds the output to that of C, so it passes before the change as well: it is there for wrong variants of it (taking every candidate that is too short, not only the first, changes 6 of the values). That the change is there at all is what a unit test in the crate is for, which fails with each of four faults planted. Nothing in the repository measures the time.
"where the binary trees lead to what is before the current segment of the window": 9 cases with values from Bun 1.4.4, after a dictionary and in a buffer that is written to from its start again. All change with a fault in the new code. Two unit tests see the 15 faults put into it (one only since
615100a88)."long matches from more than 16 MB back", in one go and as a stream: its frame sees 8 of the 12 faults put into the new function. Bun 1.4.4 passes it.
"where the rows of a dictionary have candidates and those of the input have none": 10 cases with values from Bun 1.4.4, rows of 16, 32 and 64 entries. It fails with each of 8 faults put into the new code. A unit test holds the two loops that insert positions to the same tables, and sees 4 of 4 faults.
"after a dictionary that is indexed in several parts": 8 cases with values from Bun 1.4.4. Of 12 faults it sees 9, the fuzzers 12. That a worker looks at the flag again between parts is not tested.
"where there are literals before a match in the trees of a dictionary that is searched in place": 15 cases with values from Bun 1.4.4. Of 22 faults put into the new code the fuzzers see 20, and 19 of these change a digest.
"where the kind of data changes within a block, for rows of 64 entries": 12 cases with values from Bun 1.4.4, where the parser hands over with and without second tags, with literals pending and while it skips. Like the rest of that block it passes before the change too; it is there for the hand-over (it is the only test that fails if the start of the pending literals is lost, and one of three if the tags are not moved). That the change is there is what two unit tests in the crate are for: one has blocks of 128 KB and looks at the tags afterwards (it fails with the rule as it was; not under Miri, where it takes over 6 minutes), the other has the rule alone.
A unit test in the crate for single Huffman streams whose most frequent symbol takes one bit: every length from 1 to 1,023, two codes (longest code 3 and 11 bits), three forms of the table. Each valid stream has to be decoded by the quick decoder, and streams with one bit flipped have to give what the careful decoder gives, for both kinds of table that C has. It fails with the old limit and without the new clamp; a smaller limit for what is left does not compile. Under Miri it has two lengths and no damaged streams (11 s).
Unit tests in the crate for
9778f063candb06f15db1, under Miri too: many positions added to the trees at once against one by one, around the end of the table, for hashes of 4 and of 5 bytes; the hash from 4 bytes against the one from 8; the literals and the length of their buffer over two blocks, up to a whole block, after a failure, and where it does not grow. 16 faults put into the code are each seen by one of them. Fordd2ac327c: the comparisons, for a first difference after each of 0 to 17 bytes, bytes on both sides of 0x80, and what follows the difference in the other order; 15 faults put in by hand are all seen."frames that no encoder makes decode as with zstd 1.5.7": 116 frames from upstream's
decodecorpus, each decoded 20 ways (one-shot, streams cut in several ways, truncated, damaged), with the results recorded with Bun 1.4.4.Run on a debug (ASAN) build on Linux x64:
test/js/bun/util/zstd.test.ts(258),test/js/node/zlib/zlib.test.js(473),zlib-handle-bounds-check(41),zlib-estimated-size-gc(6),zlib-reset-race(5),zlib-onerror-reentrancy(3),leak(8)test/js/web/streams/compression.test.ts(74),test/js/web/fetch/fetch-compress.test.ts(32),test/js/web/intl/intl.test.ts(21, for the ICU data)test/regression/issue/18413-*,20053,23314/*,test/js/node/errors/error-code-messages.test.tstest/js/node/test/parallel/test-zlib-zstd*.jsand the othertest-zlib*.jsthat have a zstd case--trim, on a release build with the trimming link (local WebKit and the system's ICU, so without ICU's data package): all ofcompile-trim-runtime-by-use(243),compile-trim-runtime-by-use-features(44) andcompile-trim-remove.test.ts(18) pass. The compressor is 643 KB of an executable that uses it, an executable that only decompresses does not have it, and one that has it forced out says--trim removed zstd compression from this executable.--compile --sourcemapexecutable remaps its stack traces.Left to CI: the rest of the test suite and every platform other than Linux x64. Not run locally either:
test/js/node/process/process.test.js, which needs a package that I could not install.Generated by Claude Code