Skip to content

js_parser: fold [n], .charCodeAt, .length, < and unary + of an 8-bit string per UTF-16 code unit - #46915

Draft
robobun wants to merge 13 commits into
mainfrom
robobun/d5fcea44/estring-code-unit-readers
Draft

robobun wants to merge 13 commits into
mainfrom
robobun/d5fcea44/estring-code-unit-readers

Conversation

@robobun

@robobun robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

  • A non-ASCII --define string folds per UTF-8 byte: with U="é😀", U[0] is "\0" and `${U}`.length is 6. Debug builds hit assertion failed: strings::is_all_ascii(&literal).
  • 1.3.14 is right for --define and macro JSON. 1.4.x is wrong: Rewrite the JSON parser around a SIMD structural index #33032 stores these strings as 8-bit. Env values and paths are wrong on both.
  • Cause: five folds read one byte of an 8-bit string as one character (src/js_parser/visit/visit_expr.rs:1149).

Fix

  • EString readers decode WTF-8 and walk rope segments. The [n], .charCodeAt, .length, < and unary + folds call them.
  • A read of bytes that are not UTF-8 does not fold (Notes).
  • Verified: 25 of 26 new tests (test/bundler/bundler_minify.test.ts, 6 more files) fail on main. 10,000 rows match node (main: 8,865 differ). Self-reviewed: 94 concerns, 86 addressed (Notes).

Background

Downsides

Notes

Questions for a maintainer.

  1. Rewrite the JSON parser around a SIMD structural index #33032 says JSON parser strings are "always UTF-8/WTF-8, never UTF-16", and it is why --define and macro JSON strings are 8-bit since 1.4.0. This PR takes that as the contract for every 8-bit E.String that the visitor folds, and writes it as the doc of EString.data. Store non-ASCII define, env, macro and inlined path strings as UTF-16 so they fold like literals #42019 is the other design: UTF-16 at each producer. If UTF-16 at the producers is wanted, say so: Store non-ASCII define, env, macro and inlined path strings as UTF-16 so they fold like literals #42019 is then the PR, and this one closes. Make E.String always ascii or utf-8, never utf-16 #15163 (closed) was one storage for all strings.
  2. Where does the text of bytes that are not UTF-8 come from (a Latin-1 .env value, a path)? This PR gives no rule: the readers do not fold such a string. A built rule waits on the branch robobun/d5fcea44/estring-not-utf8-text (see "Bytes that are not UTF-8"). It reads them as TextDecoder does. The lexer and the printer of Stop printing NUL for non-UTF-8 bytes: one WTF-8 step for the string printers and line counters #46269 read a lone byte as Latin-1 today, and lexer: decode ill-formed UTF-8 in JS source as U+FFFD #38262 changes the lexer. The review of Preserve non-ASCII in regex .source and tagged template .raw #33866 asks that a fold and a literal of the same bytes agree, so that rule has to follow the lexer.

How it was found. No issue and no user report. A fuzz run over --define strings found (X + "x")[1]. The other rows come from a read of each fold that indexes E.String.data.

Releases. Two rows, each run on the release binary. The define row is bun run --define 'U="é😀"' with [U[0], U[1], `${U}`.length, U.length, (U + "")[0]]. The env row is PUB_L=é bun build --env 'PUB_*' --minify-syntax with [process.env.PUB_L?.[0], `${process.env.PUB_L}`.length].

build define row env row
1.3.13, 1.3.14 ["é","\ud83d",3,3,"é"] ["\0",2]
1.4.0 to 1.4.4, main 179664e ["\0","©",6,3,"\0"] ["\0",2]
this PR ["é","\ud83d",3,3,"é"] ["é",1]

#33032 is in 1.4.0. It stores --define and macro JSON strings as 8-bit. Env values and inlined paths were 8-bit before it.

Where it reaches (all wrong on main 179664e, all equal to node here).

  • No gate: .charCodeAt(n) and unary + and -. They fold in a plain bun build --define too.
  • minify_syntax or inlining: <, <=, >, >= and unary ~. The same folds always run in a TypeScript enum initializer, in import() and in the arguments of require, require.resolve and a macro call.
  • minify_syntax only: [n], .length and the {k: v}.k key compare.
  • The runtime transpiler sets both flags for every file (src/bundler/options.rs:2069), so bun run --define, a bunfig [define] and a macro result are wrong under bun run and bun test with no flag. --coverage and --inspect drop minify_syntax only.
  • Producers of an 8-bit string that is not ASCII: --define and Bun.build({define}), bunfig [define], --env=inline and --env=PUB_* from the process or a .env file, the JSON and the data: URL of a macro Response or Blob, inlined import.meta.dir|file|path and module.filename|id|path, a regexp source coerced by +, exports.replace of Bun.Transpiler. A macro text body is JSON-escaped ASCII (macros: inline a text Response or Blob as its text, not as its JSON-escaped form #46680 owns it).

Folds that are new. Four reads are right on main because main leaves them for run time. They fold now, so that a build does not depend on whether a value is ASCII: .length of a flat 8-bit string that is not ASCII, .charCodeAt of a code unit of 0x80 or more, < of an 8-bit string against a UTF-16 literal, and [n] or .charCodeAt past the first segment of a join. In the seeded differential they are about a fifth of the cells that change.

Tests. 26 new tests in 7 files. 25 fail on a release build without this change (main 179664e for 24 of them, the 1.4.3 canary for the newest one). minify/DefineNonAsciiRequireJoin pins a join that main already gets right.

  • test/bundler/bundler_minify.test.ts (15): CLI and API folds, 29 dead branches that each guard require("./missing-N.js"), order pins where code-unit order and byte order differ, inlined enum routes, a fold past a rope seam, the bounds of a folded index, .length where it is assigned to or deleted, an index under delete, inlined paths, the folds with no minify flag (CLI and API), and a seeded differential against the engine (BUN_DEFINE_FOLD_SEED) with a value oracle, a count of the folds that are left, and 8 long texts.
  • test/bundler/bundler_env.test.ts (3): --env=inline and --env=PUB_* with a UTF-8 value, and a .env file with a UTF-8 value and three values that are not UTF-8. A read of those gives what an unfolded read of the same value gives.
  • test/bundler/transpiler/transpiler.test.js (1), runtime-transpiler.test.ts (2, one with bunfig [define]), macro-test.test.ts (1), react-compiler.test.ts (1: .length of an env value, UTF-8 and not), test/bake/dev/import-meta-inline.test.ts (3: the server graph, the client graph and bun build --app).
  • Rust unit tests: four in src/ast/e.rs run in the bun_ast Miri job. The ones in src/bun_core run in no CI job. They pass when run by hand with simdutf linked in, and the JS tests above pin the same cases.
  • Existing expectations rewritten: 0. Also run on the release build: bundler_string, bundler_merged_enum, template-literal, transpiler-cache, esbuild default, ts and dce, bundler_edgecase, bundler_loader and internal-sourcemap.

Repro rows.

  • bun build --define 'X="é"' --minify-syntax with [(X + "x")[1], (X + "x").length, (X + "x")[0]]: was ["©",3,"\0"], now ["x",2,"é"]. Same for --env=PUBLIC_*, a .env file, €, é😀 and --minify.
  • 17 dead-branch and macro-argument builds at depth 1 (if (X.length === 1) … else require("./missing.js"), if (+X), if (X < "z"), macro(X[0]), …): 17 exit 0 with node's value here. Main: 11, two of them with a wrong value. Store non-ASCII define, env, macro and inlined path strings as UTF-16 so they fold like literals #42019: 9. 1.3.13 and 1.3.14: 7.
  • 18 chained builds ((X[0] + "x").length === 2, X[0] > "z", macro(+X[0]), …): 18 of 18 here and on main. Main passes them on a wrong one-byte X[0]. Eight twins that see the value (X[0] + "x" === "éx", U[1] + U[2] === "😀", …): 8 of 8 here, 0 of 8 on main.
  • 43 expressions over --define strings (T="aéb", H="😀", L="\uffee", N="\u00a01", A="abc"), under bun run --define and under bun build --minify-syntax --define, each against the same expression in a new Function: 0 differ on a debug build of this head, 16 on 1.4.2. 8 statements with a write or delete target ("abc".length = 1, delete "abc"[0], delete [x][0], …), run as they are and after a build: 0 differ here, 5 on 1.4.2.
  • 18 statements if (test) require("./missing.js") with T="é" and U="ü" from --define, each built with and without --minify-syntax: every build that works on 1.4.2 works here. +T, -T, ~T + 1, U < T, T > U and T >= U are among them. Three more build here with --minify-syntax and fail on 1.4.2: L < H, ("x" + T).length === 3 and T.length === 2.

Measurements. Release builds on linux x64 of main 179664e and of this diff on that commit (the branch before its merge of main). The measured build differs from the branch in these things: one doc comment, one iterator that was pub and is private to its module now, the later merge of main 3f3a9ff, and one more test of a flag at the index fold (the delete guard). No release build was made after them. Instructions are instructions:u from perf_event_open, the minimum of 5 runs of bun build --no-bundle --minify-syntax over 100,000 equal statements, net of a control file with no fold.

  • Size: release .text +3,328 B (size -A).
  • Frame: visit_expr_in_out has sub $0x238,%rsp and 16 calls to itself, as on main. The longest x["b"]["b"]… chain that builds has 13,177 links and x.b.b… has 13,174 (main 13,175 and 13,177, and a run differs from the next by a few links). The first version had 6,588. Only linux x64 is measured.
  • ASCII folds, instructions per fold against main: "abcdefghij"[5] +91, .charCodeAt(5) +49, .length +36, "abcdefghij" < "abcdefghik" +150, ("abcde" + "fghij").length +105, unary + +37, === +2, o["abcdefghij"] to o.abcdefghij +5.
  • A 4,000-byte ASCII define: X[4000] +600 (the bytes before the index are tested for ASCII), X.length -19,896, X < Y where the two differ in the last byte +1,142.
  • A define that is not ASCII: "héllo wörld"[7] +511. Index 1,000 of 1,642 characters of é€: +103,346. The walk costs 100 instructions for each 2-byte code point before the index, 115 for each 3-byte one and 144 for each 4-byte one. .length of that define: -279,274 (one simdutf count, where main prints the literal). X < Y of two such defines that differ at the end: +337,492.
  • flatten_rope tests the first byte of each segment for half of a pair: +7.1 instructions per seam (50,000 joins of 18 segments against 50,000 joins of 2).
  • Allocations (hardware breakpoints on the 16 mimalloc entry points) for 100,000 folds of each kind, and syscalls per build: the same as on main.
  • typescript.js (9.1 MB, no non-ASCII constant): 975,756,793 instructions here, 976,075,182 on main, and the same output. The readers are entered 17 times there (.length of 17 literals) and 0 times in bun hello.js and bun build hello.js (hardware breakpoints).
  • order_utf16 is the code of main. One call, single-stepped in gdb, has the same instruction count as on main for 9 pairs of keys.
  • N folds into one N-unit define with Bun.build and minify.syntax, millions of instructions. An index fold is one scan of the text before the index, where main reads one byte. So N folds are quadratic. Constant time needs UTF-16 storage (question 1) or an index that the string keeps. Main is quadratic too where it does not fold a read of a non-ASCII define (.charCodeAt, .length, < against a literal): it prints the define at each read. bench/snippets/transpiler-define-folds.mjs has the same cases for a wall-clock run.
fold text main, N = 16,384 / 32,768 here, N = 16,384 / 32,768
X[n] ASCII 180 / 325 189 / 402
X[n] "é", then ASCII 173 / 328 4,075 / 15,917
X[n] Latin 173 / 333 12,942 / 51,349
X[n] CJK 184 / 340 15,081 / 59,953
X[n] astral 177 / 341 9,650 / 38,214
X[N - 1] Latin 173 / 327 25,682 / 102,349
X.charCodeAt(n) ASCII 190 / 354 206 / 422
X.charCodeAt(n) Latin 43,602 / 173,513 12,950 / 51,380
X.length Latin 43,513 / 173,279 517 / 1,718
X < Y ASCII 214 / 460 273 / 735
X < Y Latin 239 / 606 51,183 / 204,338
X < Y CJK 279 / 724 59,773 / 238,752
X < K Latin 67,254 / 268,008 26,485 / 105,561
  • A version that counts blocks of UTF-8 with simdutf and does not decode them is on the branch robobun/d5fcea44/estring-readers-block-counting. Its Latin rows are 523 / 1,445 for X[n] and 604 / 2,054 for X < Y, and its .text is +5,120 B. It is not in this PR (Self-review).
  • Transpiler cache 51 to 52 (43 to 44 when it was measured): 20 of 20 cached modules are transpiled again on the first run after the upgrade, 0 on the second. Stop printing NUL for non-UTF-8 bytes: one WTF-8 step for the string printers and line counters #46269 and js_parser: fix string folds that read non-ASCII 8-bit text by byte or rewrite a write or delete target #47655 also take 52. The second to merge takes the next number.
  • Node differential: 10,000 generated cases of 17 expressions each (X[i], X.charCodeAt(i), X.length, (X + Y).length, (X + Y)[j], <, <=, >, >=, +X, -X, ~Y, X + Y === Z, template forms). The values are --define strings made of 1- to 4-byte sequences, lone surrogates, pairs split over +, NBSP, BOM and U+2028 around digits, and astral characters against U+E000..U+FFFF. The --minify-syntax bundle under node against node on real constants: 0 of 10,000 rows differ. Main: 8,865.
  • size_of::<EString>() is 40 bytes on main and here. A const assert pins it.
  • New pub items: wtf8_code_unit_at and its result type, wtf8_checked_len_utf16, wtf8_pair_across, checked_order_utf16 and checked_order_wtf8_utf16 in bun_core, and js_order, code_unit_at and char_at on EString. EString::order is private now. Data::e_number is removed (no caller is left).

Bytes that are not UTF-8. An env value or a path can hold them. The readers do not fold such a string, at and after the first such byte. The read then sees the literal that the printer writes.

Other results of the same readers.

  • ({"é": 1}).é folds to 1 on main, because the key compare widened bytes. It is left for run time now, where it is undefined. The fold compares 8-bit keys only, so ({"é": 1}).é stays unfolded as on main (js_parser: use EString::eql_bytes in the {k: v}.k fold, keep this for template tags #32297 folds it, and needs the template-tag guard of js_parser: keep (0, x) where a fold leaves a property access as a callee or a template tag #46818 for that).
  • Two defines that are the halves of a surrogate pair: (H + L).length is 2 and H + L === "😀" is true. flatten_rope joins the halves at the seam.
  • An ASCII rope folds [n] and .charCodeAt(n) past its first segment.
  • The index fold keeps the operand of delete, as the .length fold does: delete "abc"[0] and delete [x][0] stay as written. Main prints delete "a" and delete x.
  • +N with the define N="\u00A01" is 1: Number trims NBSP, BOM and U+2028.
  • A define compared with a non-ASCII literal (X < "ü") folds: js_order compares 8-bit with UTF-16 by code unit. An ASCII literal against a non-ASCII literal ("a" < "é") folds too, as in esbuild.
  • React Compiler: .length of a non-ASCII env value is its number of UTF-16 code units. For a value that is not UTF-8 it is left for run time.
  • The data: URL of a macro Response with a non-ASCII content type: .length was 52, is 51.

Reader audit of src/ast, src/js_parser and src/react_compiler.

  • Through the readers now: the five folds, the .length and key compare in fold.rs (its two local copies are deleted), and JsString::len_utf16 of the React Compiler.
  • Byte-safe and unchanged: scan_side_effects.rs:890 (emptiness), js_parser_jsc/expr_jsc.rs:272 (a capacity), expr.rs:2001 (a clone), import paths and directives in parse/mod.rs, hashing in function_identities.rs, the React Compiler lowering (text only).
  • Still byte readers, on purpose: the React Compiler join push_units, which react-compiler: decode a UTF-8 string when constant propagation joins it to a UTF-16 string #45172 owns with its bun lint test (under --react-compiler, a join of the env value Café and a non-ASCII literal prints Café on main and here), and js_to_number of the React Compiler, which another session has.

Open PRs on the same code.

Left as they are.

Self-review. Three rounds, 94 concerns, 86 addressed.

  • Round 1, on the first version: 33. Three are follow-ups and not built here: a lint on byte reads of EString.data, the cache version scheme, and js_to_number of the React Compiler (another session has it).
  • Round 2, on the same version: 19, all addressed. fold_constant_index takes its target by reference, so the longest a["b"]["b"]… chain is main's again. order_utf16 is the code of main again. The cost lines are measured for text that is not ASCII, and the bench is in the repo. The index bounds and the folds under bun run have rows. The React Compiler join stays with react-compiler: decode a UTF-8 string when constant propagation joins it to a UTF-16 string #45172. Round 2 also proposed block counting with simdutf for long text. It was built and is not in this PR: round 3 asked for a plain decode, and REVIEW.md asks for the simplest mechanism.
  • Round 3, on a stacked head that also gave bytes that are not UTF-8 a text rule: 42, and the verdict to split. This PR is the valid-text part on main, and the rule for other bytes waits on its branch for question 2. Also taken: the plain decode, .length is not folded where it is assigned to or deleted, the seeded differential has long texts so that CI runs the SIMD paths, this text starts with the regression and names the new folds, the note on init_re_encode_utf8 stays, and the tests are out of the heavy bundler_edgecase file. Five are not taken. Two ask for the React Compiler join and for one shared decoder for it and for macro arguments (react-compiler: decode a UTF-8 string when constant propagation joins it to a UTF-16 string #45172 owns the join). Two ask for folds on UTF-16 strings (they stay as on main). One asks for a test under --inspect or --coverage (the bundler rows with no minify flag run the same folds). The verdict also asked for a tracker issue (none is filed) and for flatten_rope as on main (the join of two surrogate halves is what makes H + L === "😀" true, as on 1.3.14).
  • After the rounds: main 3f3a9ff is merged in (cache version 52), and the index fold keeps the operand of delete. That check is one condition beside the write check on the same line, so it had no round of its own.

…F-16 code unit for 8-bit WTF-8 and ropes

The constant folds of "lit"[n], .charCodeAt(n), .length, <, <=, >, >= and
unary +, - and ~ read an 8-bit E.String one byte per character. A string
that does not come from the lexer (a --define or inlined env value, a macro
result, an inlined path, a regexp source) is 8-bit WTF-8 and can be
non-ASCII, so those folds gave a wrong constant.

The folds now go through readers on EString that decode WTF-8, walk rope
segments, and decline on bytes that are not WTF-8. The {k: v}.name fold
compares through EString::eql_bytes. flatten_rope joins the two halves of a
surrogate pair that meet at a seam. The React Compiler decodes an 8-bit
string when it joins it to a UTF-16 one.
@robobun

robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Reproduced on main 179664ea89 (release build) and on the releases 1.4.0 to 1.4.4. 1.3.14 prints the right values for the three --define rows and the same wrong value for the env row.

$ bun run --define 'U="é😀"' m.ts     # [U[0], U[1], `${U}`.length, U.length, (U + "")[0]], then the `case "é":` taken for U[0]
["\^@","©",6,3,"\^@","default"]
$ bun build --define 'X="é"' --minify-syntax len.mjs   # [(X + "x")[1], (X + "x").length, (X + "x")[0]]
["©",3,"\^@"]
$ PUB_L=é bun build --env 'PUB_*' --minify-syntax e.ts   # process.env.PUB_L?.[0], `${process.env.PUB_L}`.length
["\^@",2]
$ bun build --define 'X="é"' --minify-syntax dead.mjs   # if (X.length === 1) console.log("one"); else require("./missing.js");
error: Could not resolve: "./missing.js"

With this branch (6159607fa6, merged with main 3f3a9ff87c, debug build) the same commands print what node prints for the source: ["é","\ud83d",3,3,"é","case é"], ["x",2,"é"], ["é",1], and the dead branch is removed. The same holds for --env with a .env file, for --minify, for X="€" and X="é😀", for a bunfig [define] under bun run, for the JSON of a macro Response, for inlined import.meta.file / module.filename under a non-ASCII path, for (/é/ + "x")[1], and for folds chained on X[0] such as X[0] + "x" === "éx".

PR: #46915

CI on 6159607fa6 (build 136530, finished): 124 jobs pass. The 64 failed jobs fail on 28 test files that fail on main too (build 136408): 27 under test/aot, and test/bundler/compile-aot-size.test.ts. The other entries are flaky tests that pass on retry. The 7 test files of this PR pass on every lane, macOS x64 too.

@robobun

robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author
Updated 1:39 PM PT - Nov 1st, 2026

❌ @robobun, your commit 6159607 has 40 failures in Build #136530 (All Failures):


🧪   To try this PR locally:

bunx bun-pr 46915

That installs a local version of the PR into your bun-46915 executable, so you can run:

bun-46915 --bun

@robobun

robobun commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator Author

Bytes that are not UTF-8 (a Latin-1 .env value, a path) are outside this PR. The readers do not fold a string that holds them, and the read then sees the literal that the printer writes. #46269 corrects that printer. Land #46269 first or in the same release.

A fold rule for these bytes is built on the branch robobun/d5fcea44/estring-not-utf8-text. It has no PR and waits for question 2 in the next comment.

The two PRs conflict in two places, and both are mechanical: the pub use wtf8_text::{..} list in src/bun_core/string/immutable.rs and the cache version line in src/jsc/RuntimeTranspilerCache.rs (both take 52).

…ing, and keep the fold out of the visitor's frame

- EString::char_at returns 8-bit WTF-8 for every code unit, so a fold of the
  result goes through the same readers. Half of a pair that is one four-byte
  sequence is three bytes in the arena.
- EString::js_order compares an 8-bit with a UTF-16 string by code unit.
  checked_order_utf16 also decodes what both texts start with when it is not
  ASCII.
- The "lit"[n] fold is a function of its own, and resolve_rope_if_needed is a
  test and one call again: both were in the frame of visit_expr_in_out.
- The {k: v}.name fold matches an 8-bit key only. A UTF-16 key matched by
  widening the bytes of the name.
- The React Compiler does not fold .length of bytes that are not WTF-8.
- Unit tests for the readers that Miri can run.
…r_in_out

`"lit"[n]` and `[a, b][n]` fold in `fold_constant_index`. With only the
string fold out of line, the frame of the recursive `visit_expr_in_out`
was 584 B against 568 B on main: the stack objects were the same, and
LLVM packed them 16 B worse. It is 568 B again. The function is 1,057 B
shorter than on main, and it spills one field address on entry where it
spilled six.

`Data::e_number` had one caller, in that fold. `as_e_number` is the same
function, so `e_number` is removed.
`bun build --app` of a module under a non-ASCII path is one test in
import-meta-inline.test.ts. production.test.ts is as on main.
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/e.rs Resolved
Comment thread src/ast/expr.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/bun_core/string/immutable.rs Resolved
Comment thread src/js_parser/fold.rs Resolved
Comment thread src/js_parser/fold.rs Resolved
Comment thread src/js_parser/visit/visit_expr.rs Resolved
Comment thread src/js_parser/visit/visit_expr.rs Resolved
Comment thread src/jsc/RuntimeTranspilerCache.rs Resolved
@robobun

robobun commented Oct 26, 2026 •

Copy link
Copy Markdown
Collaborator Author

Two questions need a maintainer. The first decides between this PR and #42019. The second decides what comes after this PR.

1. Which contract holds for a string that does not come from the lexer?

This PR makes the folds read an 8-bit E.String as WTF-8. That follows #33032, which says that JSON parser strings are "always UTF-8/WTF-8, never UTF-16". It is why --define and macro JSON strings are 8-bit since 1.4.0.

The other design stores UTF-16 at each producer, as #11994 and #14168 did for one producer each. #42019 does that for all producers, and #46680 and #46688 do it for a macro text body. For a --define it gives what 1.3.14 gave. UTF-16 strings fold less: +x, < and [n] do not fold on them, so 7 of 11 dead-branch and macro-argument builds that pass on 1.4.x fail with #42019.

#42019 is a draft until this is decided.

#47655 (opened on 1 Nov) is a third PR for the same fault, and it needs no answer to this question. At its head 84ac132d11 the < family and the unary folds read 8-bit UTF-8 text by UTF-16 code unit and with the white-space trim of Number(), as this PR does. It leaves .length of a joined string, "s"[n] and .charCodeAt(n) for run time where the text is not ASCII up to the read. This PR folds those per UTF-16 code unit, and that is the difference that is left. Either design can replace the checks of #47655 later.

2. Which text do bytes that are not UTF-8 have in a fold?

An env value or a path can hold such bytes, for example a .env file in Latin-1. This PR gives them no rule. The readers do not fold such a string, and the read sees the literal that the printer writes. #46269 corrects that printer.

A rule has to agree with three other readers:

The review of #33866 asks that a fold and a string literal of the same bytes agree. So a fold rule follows the lexer, and #38262 decides what that is.

The branch robobun/d5fcea44/estring-not-utf8-text has a built version and no PR. It has #46269 merged in, and its readers use the TextDecoder rule. Its numbers are in the Notes of the description. Is that rule wanted after #38262, or do these strings stay unfolded?

…sign target, more tests

- `fold_constant_index` takes its target by reference, so the longest
  `a["b"]["b"]...` chain is main's again.
- `order_utf16` is the code of main again. `checked_order_utf16` compares
  two ASCII texts as bytes and decodes other texts from the start.
- `.length` of a string is not folded where it is assigned to, updated or
  deleted.
- `push_units` of the React Compiler is the code of main again: #45172
  owns that join.
- The checked iterator is private to the string module, and the note on
  `init_re_encode_utf8` stays.
- Tests: the bounds of a folded index, the order and key folds under
  `bun run`, long texts in the seeded differential, and the folds with no
  minify flag beside the other define folds.
- bench/snippets/transpiler-define-folds.mjs times the folds by kind of
  text and length.
…ing-code-unit-readers

# Conflicts:
#	src/jsc/RuntimeTranspilerCache.rs
`delete "abc"[0]` was printed as `delete "a"`, and `delete [x][0]` as `delete x`.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants