[robobun]

Fixes #8207
Fixes #8745
Fixes #13853
Fixes #15492
Fixes #15929
Fixes #16763
Fixes #18115
Fixes #33930

Last of a stack (#38708, #38714). The diff is against #38714.

Problem

• String.raw`日本語` returns "\u65E5\u672C\u8A9E" and /é/.source returns "\u00E9". Node returns the text as written.
• print_raw_template_literal and print_reg_exp_literal (src/js_printer/lib.rs) escape non-ASCII text for the Bun target, and a program can read both texts.

Fix

• The printer writes regex text, raw template text and preserved comments as written. Other text stays escaped.
• The runtime's BufferWriter holds Latin-1. It is widened in place to UTF-16 once, at the end of a print that had a character above U+00FF (the design of Preserve non-ASCII in tagged template literals and regex source #28214). Each consumer takes the buffer in its final width: the loaders, the --hot intern map, the transpiler cache, NODE_COMPILE_CACHE, the source map builder.
• Verified: test/bundler/transpiler/template-literal.test.ts (67 tests), test/cli/inspect/inspect-inline-sourcemap.test.ts, and the suites in Notes.
• Self-reviewed: 9 concerns raised, 7 addressed. The other two are in Notes.

Background

• For the Bun target the printer escapes identifiers and string literals, so JSC can take a module as an 8-bit string.
• A JSC string is 8-bit (Latin-1) or 16-bit (UTF-16), never UTF-8.
• Considered: UTF-8 output that each consumer decodes (an earlier revision). That scans each module in each consumer.

Downsides

• bun build --target=bun output with a non-ASCII regex or tagged template now has raw UTF-8. A bun before Decode module text that comes from disk or bundler output as UTF-8 #38714 reads it as Latin-1.
• ASCII modules: one more branch per print. Binary: +20,480 bytes on linux-x64 against Decode module text that comes from disk or bundler output as UTF-8 #38714 (CI binary-size, builds 130457 and 130282).
• The transpiler cache version moves to 44: each entry is built again once.

Notes

Rebase of 2026-10-23 (what changed against bf165d1)

• The base is Decode module text that comes from disk or bundler output as UTF-8 #38714's head b518e9a, on main 5c6d4e2. Main moved the transpiler cache version to 43, so the version here is 44.
• CI build 130457 of bf165d1 failed on debian 13 x64-asan. In react-compiler.test.ts, does not overflow the stack on a component that is nested deeply > JSX printed Maximum call stack size exceeded while generating code for this file. Decode module text that comes from disk or bundler output as UTF-8 #38714 passed that lane on the same main (build 130282).
• The cause was in this PR. BufferWriter::write_all had a debug_assert! with a formatted message. The asan lane builds with assertions on. The assert was inlined at each print site, and ASAN gives each temporary of the message its own stack slot. So each frame of the printer grew.
• The check is now in assert_accepts_printed_bytes, which is not inlined. A UTF-8 writer does not call it. A build without assertions has the write_byte and write_all of main, as before.
• Measured on --profile=release-asan builds. The frames are those of the printer for --target=browser with no source map. The depth is the deepest nesting of <a> that bun build --react-compiler prints. The test uses 600.

	#38714 (b518e9a)	bf165d1 on the same main	this head
print_expr frame	1504 bytes	2400 bytes	1504 bytes
print_property frame	384 bytes	2304 bytes	384 bytes
print_stmt frame	448 bytes	3264 bytes	480 bytes
deepest JSX	1009	494	1009

• Five more shapes have the same limit on Decode module text that comes from disk or bundler output as UTF-8 #38714 and on this head: calls 1151, conditional expressions 1435, object literals 969, arrow functions 817, if statements 875.
• On the release-asan build of this head: react-compiler.test.ts (121 pass, 1 skip, 1 todo) and template-literal.test.ts (67 pass).
• bundler_bytecode_portable.test.ts: the same seven fingerprints move. They are computed again because the WebKit upgrade (Upgrade WebKit to 33ce3bf11a71 #46279) changed the .jsc bytes on main. The .js hashes of the seven did not change.
• Pass on a debug (ASAN) build of this head: template-literal (67), inspect-inline-sourcemap (5), bundler_bun (26), transpiler.test.js (258), repl (131), coverage (44), transpiler-cache (24), 14976 (7), runtime-transpiler (18), inspect-error (46), bunshell (466), isolation (50), source-positions (325), source-too-large (3), hot (28), filesystem_router (50), production (23), bundler_compile (88), bundler_bytecode_portable (22), react-compiler (121), test/js/bun/sourcemap/ (367), the 14 node test-compile-cache-* tests. No test failed.
• template-literal.test.ts with the released 1.4.3: 58 of 67 fail.
• Binary size, from CI binary-size for the heads of 2026-10-21 (bf165d1 in build 130457, Decode module text that comes from disk or bundler output as UTF-8 #38714's 3ea0607 in build 130282): linux-x64 +20,480 bytes, linux-x64-musl +20,480, linux-aarch64 0, darwin-x64 +16,400, darwin-aarch64 +16, windows-x64 +22,016, windows-aarch64 +10,752. Those builds have no assertions, so the change to the assert does not move them.

Rebase of 2026-10-21 (what changed against 0b4de69)

• The base is Decode module text that comes from disk or bundler output as UTF-8 #38714's head 3ea0607, on main 9634520. Build the coverage line table in UTF-16 code units #38710 is closed (Source positions: coverage, stack frames, call sites and source maps agree with the source #45677 fixed it on main), so the stack is Build the source excerpt for errors from 16-bit module sources #38708, Decode module text that comes from disk or bundler output as UTF-8 #38714 and this PR.
• Main rewrote VirtualMachine::ref_counted_string (Hardening: fix crashes across the runtime, bundler, dev server, shell and CLI #46021): it returns Result<_, StringTooLong>, checks the length first, and sets on_before_deinit after create_external. The intern for two widths follows that shape in the same function, which now takes the encoding. The length check counts code units. Main's test for it in hot.test.ts has two more rows: a UTF-16 module over the limit, and one that is under the limit in code units and over it in bytes.
• Main removed the --hot arm of AsyncModule.rs (Hardening: fix crashes across the runtime, bundler, dev server, shell and CLI #46021). That file now only copies the buffer in its width.
• The transpiler cache version is 39 (main is at 38).
• The REPL decode that Build the source excerpt for errors from 16-bit module sources #38708 had is in repl: evaluate input the REPL transform does not take through each caller's result path (-e/-p exit code, .copy, _) #45267 now. This PR does not need it: a UTF-8 writer's output is valid UTF-8 for every input, and the .load test here passes without repl: evaluate input the REPL transform does not take through each caller's result path (-e/-p exit code, .copy, _) #45267.
• The coverage test no longer shares helpers with Build the coverage line table in UTF-16 code units #38710. It is one test with three variants.
• bundler_bytecode_portable.test.ts: the same seven fingerprints move as before (features, records and all, plain and minified, and libraries), computed again for main's .jsc format. In the 16.7 MB libraries.js bundle the only difference between main's printer and this one is two lines: \u{1f4bf} in tagged template text is 💿 again.
• The comments in the diff are shorter. The review bot flagged each one that had more than one line.

Lone surrogates in a UTF-8 writer (new)

• A WTF-8 encoded lone surrogate in verbatim text has no UTF-8 form. The revision before kept its three bytes. A UTF-8 decode turns those into U+FFFD, so in a bundle the regex no longer matched the string literal next to it (the printer writes that literal as \uD800).
• A UTF-8 writer now writes \uD800, which is what main prints for the Bun target and what a string literal gets. After a backslash (an identity escape of the surrogate) that backslash starts the escape. The regex matches as before and the cooked text of a template is unchanged. .source and .raw show the escape, as on main.
• The Latin-1 writer of the runtime keeps the surrogate as one code unit.

Self-review of this revision

• Fixed: the surrogate after a backslash (above).
• Fixed: text with U+2028 or U+2029 is counted for the source map from the source. That count used the source map builder's decode, and the writer stores the lexer's decode. They differ for ill-formed bytes, so the columns after such text were wrong. The count uses the lexer's decode now (update_generated_verbatim), with a test.
• Fixed: the builder's look-ahead for CR LF tested the byte two after the CR (a defect on main: "CR, one byte, LF" loses a generated line). With one byte per code unit a non-ASCII character is one byte too, so this PR made more inputs reach it. It tests the next byte now. Two tests (a lone CR in a preserved comment, ASCII and wide).
• Fixed: String::clone_utf16_wide had a debug assertion that a transpiler cache file edited by hand could fail.
• Fixed: the inspector test for a non-ASCII directory now asserts that there is no sourceURL. Before, it also passed without the change.
• Fixed: three comments that the rebase left stale.
• Fixed: the two new rows of the hot.test.ts string limit test are labelled as guards.
• Not changed: the seven stack position tests of template-literal.test.ts also pass without the change. Their comment says that they pin line accounting.
• Not changed: the title of the existing test bun build --target=bun outputs only ascii in 14976.test.ts. Its fixture has no regex or template, and the assertion holds.

Tests on a debug (ASAN) build of 17bb0534 (the head changes only comments after it)

• Pass: template-literal (67), inspect-inline-sourcemap (5), bundler_bun (16), transpiler.test.js (255), repl (131), coverage (37), transpiler-cache (24), 14976 (7), runtime-transpiler (18), inspect-error (45), bunshell (466), isolation (47), main's source-positions (313), source-too-large (3), the two non-ASCII tests of production.test.ts, the 37 bytecode and non-ASCII tests of bundler_compile, the fingerprint snapshot of bundler_bytecode_portable, the 14 node test-compile-cache-* tests.
• hot.test.ts: 22 of 23. does not take a promise of the program's, rejected and handled, for that of the entry point prints collected twice on this machine. The same failure is in CI build 130282 of Decode module text that comes from disk or bundler output as UTF-8 #38714 (passed on retry) and shows with a release build of 2026-09-18.
• filesystem_router.test.ts: 49 of 50. reload() while Bun.build() resolves the same directory reached its 60 s limit on this build. The fixture alone prints the expected line after 71 s.

Earlier history

Rebase of 2026-10-02 (what changed against the revision alii reviewed)

• Main made the printer's writer infallible (js_printer: make the writer's operations infallible #42772) and derives the output position from the buffer length (js_printer: derive the output position from the buffer length #42767). The widening writer follows: no Result, no written counter. The buffer has one byte per code unit until the print is over, so the positions that the printer compares are code units throughout.
• Main's bytecode generators take a BunString (compile: resolve module record names through the bytecode string table #40677). The BytecodeSourceEncoding enum and its C++ mirror are gone. NODE_COMPILE_CACHE builds the string from the printer's bytes and their width (bun_js_printer::clone_as_string) and gives it to the generator. The bundler's side is in Decode module text that comes from disk or bundler output as UTF-8 #38714.
• --compile stores each module in the width JSC loads it in on main (bun build --compile: embed text imports as pre-encoded strings #40177, Never hand one StringImpl to two threads' atom tables (object URLs, BroadcastChannel, --compile workers) #40436, compile: resolve module record names through the bytecode string table #40677), which was --compile loads any non-ASCII module as a 16-bit heap copy instead of the mmapped section #38771. Nothing here touches StandaloneModuleGraph any more.
• The transpiler cache version is 34 (main is at 33).
• New in dc59b52: a UTF-8 writer (the bundler, Bun.Transpiler) writes verbatim text that is not valid UTF-8 as the UTF-8 form of the lexer's decode, and no longer as the raw bytes. Before, a source saved as Latin-1 gave a bundle whose regex did not match a string written with the same bytes. The cost is one is_valid_utf8 scan of each regex, raw template and preserved comment text in those writers.
• Costs for a module with no such text: write_byte, write_all, reserve_next and advance_by are the code that main has. Added per print: one is_empty check in resolve_encoding. Added per module in the --hot intern map: 8 bytes (RefString holds a two-variant buffer). Added per regex, raw template and preserved comment in a UTF-8 writer: one is_valid_utf8 scan.
• Binary size, from CI's binary-size job: the build of 0b4de69 (123298) is 77.12 MB on linux-x64, 60.58 MB on darwin-aarch64 and 83.79 MB on windows-x64. The build of Decode module text that comes from disk or bundler output as UTF-8 #38714 (122781) is 77.10, 60.56 and 83.77 MB. That is about +20 KB on each.
• The revision before (d523277, build 122861) was 77.38, 60.88 and 84.06 MB: about +290 KB. It turned the buffer into UTF-16 at the first character above U+00FF, so each write began with a check of the encoding, and main's writer is a plain inlined append (js_printer: make the writer's operations infallible #42772): the check and its UTF-16 arm sat at each of the 427 print sites of each printer instantiation. Now the buffer stays 8-bit for the whole print. A code unit above U+00FF is one placeholder byte with the unit in a side list, and resolve_encoding widens the buffer in place once, after the source map builder and before the first consumer. U+2028 and U+2029 do not show in the 8-bit buffer, so the builder counts text that has one from its source (update_generated_verbatim). Its UTF-16 path is gone.
• bundler_bytecode_portable.test.ts: seven fingerprints move (features, records and all, plain and minified, and libraries). Their bundles have tagged template raw text with non-ASCII characters, which is printed as written now. The other 19 entries are the same on a debug build.
• Re-ran on a debug (ASAN) build of this head: template-literal (63), inspect-inline-sourcemap (5), bundler_bun (16), transpiler.test.js (238), bunshell (438), 14976 (7), runtime-transpiler (18), repl (130), coverage (29), transpiler-cache (20), the two non-ASCII tests of production.test.ts, the 32 bytecode tests and 6 non-ASCII tests of bundler_compile, bundler_bytecode_portable (22), hot.test.ts (16), isolation.test.ts (41), inspect-error (41), the 14 node test-compile-cache-* tests.

The account of the previous revision

Problem

• String.raw`日本語` returns "\u65E5\u672C\u8A9E" (18 characters) and /é/.source returns "\u00E9"; Node returns "日本語" and "é". Same for any tag function's .raw, String(regex), and the bun build --target=bun / --compile / --bytecode output of such code. new RegExp(..) is unaffected, so the same pattern reflects differently depending on how it was written.
• Cause: print_raw_template_literal and print_reg_exp_literal in src/js_printer/lib.rs rewrite every non-ASCII code point (and, for regexes, every control byte, Transpiler rewrites raw NUL bytes (0x00) in regex literals: match failure before 1.3.12, RegExp.prototype.source still corrupted in 1.3.14 #33930) to an escape when printing for the Bun target. Escaping is invisible for identifiers, string literals and cooked template text, but these two pieces of text are exposed verbatim to the program (ES2026 22.2.6.13.1 only allows / and line terminators to be escaped in .source; esbuild's --charset=ascii exempts both constructs for the same reason).
• The escaping was load-bearing: every consumer of runtime printer output wraps the bytes as an 8-bit string without looking at them, which is only correct while they are ASCII. Preserved /*! */ comments already broke that assumption (a function containing /*! © */ reports © from Function.prototype.toString() on every path that goes through the runtime printer today), so simply printing more text verbatim needs the consumers to know what width they are holding.

Fix

• Printer: regex text, raw template text and preserved comments go through one print_verbatim and are never escaped. Everything else is still escaped, so output for code without such text is byte-for-byte what it was (test/regression/issue/14976 "outputs only ascii" still holds). Writers for files and for Bun.Transpiler are UTF-8 and copy the text through.
• This is the design from Preserve non-ASCII in tagged template literals and regex source #28214: when printing for the runtime, BufferWriter starts out as Latin-1 (init_latin1, one character per byte) and the first verbatim character above U+00FF rewrites the buffer in place as UTF-16 code units, after which printing continues in UTF-16. All printer offsets (prev_reg_exp_end, stmt_start, ...) count characters and the widening maps 1:1, so they stay valid; reserve/advance regions are converted on commit. A module whose non-ASCII text fits in Latin-1 (/café/) stays 8-bit; a module with 中 costs one O(n) widening and then two bytes per character, which is what JSC needs for it anyway. The ASCII path is one predictable branch per write.
• The source map builder is told the writer's mode: one column per byte while Latin-1, one per code unit after widening (update_generated_utf16_slow); print_verbatim flushes the pending bytes before widening and doubles the builder's byte cursor afterwards, so positions are right whether the widening happens before, after or in the middle of the line being mapped.
• Every consumer takes the buffer in its final width; nothing scans or transcodes:
  • the three loaders (jsc_hooks, RuntimeTranspilerStore, AsyncModule) copy it with clone_written_as_string (clone_latin1, or the new clone_utf16_wide);
  • the --hot / --watch intern map (VirtualMachine::ref_counted_resolved_source, RefString) owns either a Box<[u8]> or a Box<[u16]>, hands JSC an external string of the matching width (String::create_external_utf16 is new), and seeds the content hash with the width, since the map is keyed by hash alone;
  • the runtime transpiler cache's put passes the printer's bytes and the matching on-disk tag (LATIN1 or UTF16; those arms were dead until now) straight through to the writer, and load reads the entry back into a string of that width; the compile cache hook on the cache-hit path takes the width from the entry's header. The UTF8 tag is no longer written or accepted, Entry.output_code is a plain string, and EXPECTED_VERSION 25 -> 26 discards entries holding escaped output;
  • NODE_COMPILE_CACHE hashes the bytes of either width (a cache hit hands over the same bytes the print path did) and passes the width to the bytecode generators as BytecodeSourceEncoding (a repr(u8) enum mirrored in C++), and ZigSourceProvider.cpp decodes Utf8 (bundler output on disk), Latin1 or Utf16 exactly as the corresponding loader does, so the bytecode's SourceCodeKey matches at load time. This is also what makes --bytecode / --compile --bytecode work for non-ASCII bundles, which --compile already loaded as UTF-8;
  • the debugger's inline sourceMappingURL trailer (on_source_map_chunk) and the debug source dump write through the width-aware buffer. The trailer no longer appends a //# sourceURL= comment: the script's URL comes from the module loader anyway (Bun's debug adapter only reads url), on main the comment was mojibake for non-ASCII paths, and it was the only non-source text that could have widened a module;
  • bake-codegen.ts asserts the server HMR runtime, which DevServer still embeds as a Latin-1 string, stays ASCII now that its regexes are no longer escaped.
• Ill-formed UTF-8 in verbatim text is decoded by the same CodepointIterator the lexer builds string literal values with, so under bun x.js a regex or raw template containing such bytes evaluates to the same string as a literal written with the same bytes; lexer: decode ill-formed UTF-8 in JS source as U+FFFD #38262 changes that shared decoder to U+FFFD replacement and this path follows it automatically. A file written by bun build keeps the bytes and is decoded with replacement by whoever loads it, which differs from the in-process result only for WTF-8 encoded lone surrogates (the lexer keeps those for Bun.Transpiler string input); that is noted on write_verbatim_utf8.
• StandaloneModuleGraph still loads any non-ASCII module through clone_utf8, i.e. as a 16-bit heap copy; this change makes that case more common, and --compile loads any non-ASCII module as a 16-bit heap copy instead of the mmapped section #38771 is about storing such modules in their final width.
• Compatibility: bun build --target=bun output (without --compile) that contains a non-ASCII regex or tagged template now carries the raw UTF-8, and Bun versions before Decode module text that comes from disk or bundler output as UTF-8 #38714 read // @bun files as Latin-1, so such a bundle run on an older Bun gets mojibake in those two places (a /café/ built here does not match "café" on 1.3.x, where today's escaped output does). Output for code without those constructs is unchanged, and --compile and the runtime transpiler are unaffected because the runtime that reads the output is the one that wrote it. Worth a line in the release notes.
• Verification (fails on Decode module text that comes from disk or bundler output as UTF-8 #38714, passes here, unless noted):
  • test/bundler/transpiler/template-literal.test.ts: .source / .raw values for Latin-1, BMP, BOM, astral and control characters, the Puppeteer incompatibility using ::-p-xpath() selector #13853 and String.raw Iterator encoding error #18115 repros, bun build --target=bun; a matrix of four modules (regex/template text and preserved-comment text, each in an 8-bit variant café-©-ÿ and a widening variant café-中-🐰) through entry point, --hot, // @bun entry point, import, import of a // @bun module and require (the // @bun rows pass since Decode module text that comes from disk or bundler output as UTF-8 #38714 and pin it; the other 16 fail before this change); transpiler cache write + read back for both widths through the sync and async paths, asserting the entry's width tag (LATIN1 for café-©-ÿ, UTF16 for café-中-🐰) and the text stored under it, and that the second run serves the entry untouched (inode, mtime and size compared); one process that requires a widening module and then an 8-bit one, whose two entries must be tagged UTF16 and LATIN1 (the printer is reused, so this pins reset() restoring Latin-1); NODE_COMPILE_CACHE hit counts for both widths equal to an ASCII twin's, with the module transpiled again, served from the transpiler cache, and with the transpiler cache warmed before the compile cache is enabled (so the bytecode is generated from the cache-served bytes and the header's width, and the final run's hit is the SourceCodeKey match), plus the preserved-comment module; five ill-formed byte sequences (stray continuation byte, truncated sequence, overlong encoding, encoded surrogate, 0xFF) for which .source and .raw must equal a string literal built from the same bytes; and stack positions of new Error sites before and after the widening text (line accounting only; they pass before the change too). 43 of the file's 58 tests fail on the Decode module text that comes from disk or bundler output as UTF-8 #38714 build.
  • test/cli/inspect/inspect-inline-sourcemap.test.ts: decodes the runtime source map the inspector reports for a line that the transpiler prints unchanged, const x = [/.../u, new Error("b"), String.raw..., f()];, in an ASCII, an 8-bit and a widening variant with the same number of code units, and requires every mapping on the line to have equal generated and original columns, including the ones at new Error, String.raw and f(). That pins the builder's byte mode, its code unit mode and the cursor handoff at the point of widening; the 8-bit and widening variants fail on Decode module text that comes from disk or bundler output as UTF-8 #38714 (the escaped text shifts the columns). A further case runs a widening module from a non-ASCII directory and checks scriptParsed.url and the decoded map.
  • test/bundler/bundler_bun.test.ts: the .jsc sidecar accepted for a non-ASCII CJS bundle; a non-ASCII --target=bun bundle's error preview (regex shown as written next to a still-escaped string literal) and frame positions without source maps, and with --sourcemap=inline, where the sourceMappingURL comment is located in the now 16-bit // @bun source and positions remap to entry.ts. test/bundler/bundler_compile.test.ts: --compile --bytecode hits for ESM and CJS.
  • test/cli/test/coverage.test.ts: lcov records for 8-bit and widened regex/template text equal the ASCII twin's (the preserved-comment case is in Build the coverage line table in UTF-16 code units #38710). test/bake/dev/production.test.ts: a prerendered page renders both values as written (the preserved-comment case is in Decode module text that comes from disk or bundler output as UTF-8 #38714). test/js/bun/repl/repl.test.ts: -e with non-ASCII .source / .raw, and .load of a file with an ill-formed byte inside a regex evaluating to U+FFFD (the preserved-comment case is in Build the source excerpt for errors from 16-bit module sources #38708). transpiler.test.js (Bun.Transpiler keeps both constructs verbatim while still escaping string literals), bunshell.test.ts ([Bun Shell]: ShellError with path string with NON-ASCII string in Bun shell #15929) and the previously skipped case in 14976.
  • Also run on this build: runtime-transpiler, transpiler-cache, node-module-module (compile cache), test/js/bun/sourcemap/, hot.test.ts, the rest of bundler_bun, bundler_compile's bytecode and source map cases, repl, coverage, bunshell, the rest of test/cli/inspect/ (debugger-buntranspiledmodule matches scripts on url; inspect.test.ts's connection tests cannot bind in this environment with or without the change), clippy and fmt on the touched crates.

Background

• Bun-target printing: the printer has an ASCII-only mode used whenever Bun itself will execute the output (runtime transpiler, bun build --target=bun, --compile). Identifiers and string literals are printed as escapes so the common module has no bytes above 0x7F and can be handed to JSC as an 8-bit string without a copy of any kind.
• JSC strings are either 8-bit (one Latin-1 character per byte) or 16-bit (UTF-16 code units); there is no UTF-8 string, so UTF-8 input has to be scanned and, if not ASCII, transcoded. The widening here produces the 16-bit form directly, so the loaders only ever memcpy.
• External strings / intern map: under --hot and --watch the runtime keeps each module's transpiled text in a refcounted map so reloads of unchanged text share one buffer; JSC is given an external string pointing at that buffer, and the map is keyed by a hash of the content alone.
• Runtime transpiler cache: on-disk cache of transpiler output keyed by source hash. Each entry records the width of the stored output so load can read it directly into a string of that width; EXPECTED_VERSION invalidates every entry when the format or the output changes.
• SourceCodeKey: JSC bytecode caches (--bytecode, NODE_COMPILE_CACHE) are validated at load time by comparing a key derived from the source string against the one stored with the bytecode; a mismatch silently falls back to parsing, which is why the generator has to build the same string (same width, same content) the loader will.
• Source maps count generated columns in UTF-16 code units, and the builder derives them by walking the printer's buffer, so it has to know what the bytes in that buffer are.

Related

• Preserve non-ASCII in tagged template literals and regex source #28214 is where this design was proposed; fix: preserve unicode in tagged template literals and regex source #27274 (a rebase of Fix tagged template literal with unicode #15047), fix: preserve non-ASCII chars in String.raw and RegExp.source - bun didn't print out a bun: ╭─╮ #30558, Preserve non-ASCII bytes in String.raw and RegExp.source #30566 and fix(transpiler): preserve non-ASCII in String.raw and RegExp.source #31394 were earlier attempts that fixed the printer and switched the loaders to UTF-8 decoding. pfgithub and kernoeb are credited as co-authors on the printer commit; the transpiler cache round-trip test is taken from fix(transpiler): preserve non-ASCII in String.raw and RegExp.source #31394.
• transpiler: preserve non-ASCII source text at runtime #33868 was an alternative that turned ASCII escaping off for the whole runtime path; closed in favour of this scope. Decode module sources with the // @bun pragma as UTF-8 #37162 is folded into Decode module text that comes from disk or bundler output as UTF-8 #38714.
• Earlier revisions of this PR decoded UTF-8 printer output in every consumer instead, and carried parts 1 to 3; they were split out at review.

---

no test proof · iteration 19 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/bun/shell/bunshell.test.ts, test/js/bun/repl/repl.test.ts, test/bundler/bundler_compile.test.ts, test/bundler/bundler_bytecode_portable.test.ts, test/bake/dev/production.test.ts

[coderabbitai]

Walkthrough

The printer now preserves non-ASCII raw template and regex contents. Runtime source, cache, bytecode, and HMR paths use UTF-8 handling. The cache format version increases from 24 to 25. Tests cover runtime values, builds, bytecode cache hits, shell output, and a regression case.

Changes

Non-ASCII preservation in printer, transpiler, and cache

Layer / File(s)	Summary
Printer emits non-ASCII literals directly src/js_printer/lib.rs	Raw template and regex printers now emit stored bytes without unicode-escape conversion.
Source conversion preserves UTF-8 src/bundler/OutputFile.rs, src/jsc/AsyncModule.rs, src/jsc/RuntimeTranspilerStore.rs, src/runtime/jsc_hooks.rs, src/jsc/bindings/ZigSourceProvider.cpp, src/runtime/bake/DevServer.rs	Bundler, runtime transpiler, module loading, bytecode, and HMR paths use UTF-8 source handling while retaining ASCII-specific paths.
Transpiler cache stores UTF-8 output src/jsc/RuntimeTranspilerCache.rs	The cache format version increases from 24 to 25. Cache storage now selects ASCII or UTF-8 representations.
Non-ASCII output validation test/bundler/bundler_compile.test.ts, test/bundler/transpiler/template-literal.test.ts, test/js/bun/shell/bunshell.test.ts, test/regression/issue/14976/14976.test.ts	Tests validate regex and raw template preservation, build output, bytecode cache hits, shell output, and the enabled regression case.

Possibly related PRs

• oven-sh/bun#35971: Both PRs modify shared transpilation and ResolvedSource construction paths.
• oven-sh/bun#35978: Both PRs modify OutputFile::to_bun_string_ref.

Suggested reviewers: jarred-sumner

Pre-merge checks: 4

Check name	Status	Explanation
Linked Issues check		The changes address the coding requirements for preserving non-ASCII raw templates and regex source, including BOM, emoji, and raw NUL cases [#8745, #18115, #15492, #16763, #33930].
Out of Scope Changes check		The implementation and tests remain focused on non-ASCII regex and raw-template preservation and the required UTF-8 propagation paths.
Title check		The title clearly and concisely describes the primary change: preserving non-ASCII text in regex .source and tagged template .raw.
Description check		The description clearly explains the problem, implementation, compatibility impact, and verification results. It does not use the template headings exactly, but it provides equivalent content for both…

Full details: Description check

Explanation

The description clearly explains the problem, implementation, compatibility impact, and verification results. It does not use the template headings exactly, but it provides equivalent content for both required sections and includes extensive test evidence.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[robobun]

^{Updated 2:47 AM PT - Oct 23rd, 2026}

✅ @robobun, your commit 297c680b4a218991a021df6d89b5edb24cd4dd9a passed in Build #131182! 🎉

---

🧪   To try this PR locally:

bunx bun-pr 33866

That installs a local version of the PR into your bun-33866 executable, so you can run:

bun-33866 --bun

[github-actions]

Found 3 issues this PR may fix:

1. bunx --bun vite change emoji to \uxxxx #25169 - Transpiler converts emoji to \uXXXX escapes when running with --bun flag (same root cause: print_reg_exp_literal/print_raw_template_literal escaping non-ASCII)
2. A build option to disable converting non-Latin characters to Unicode codes #10747 - Bundler converts non-Latin characters (e.g. Cyrillic) to Unicode escapes, bloating bundle size (the removed ASCII_ONLY escaping logic directly addresses the default --target=bun behavior)
3. Unicode double-encoding when running --target=node bundle under Bun runtime #25767 - Unicode double-encoding when running --target=node bundle under Bun runtime (the Latin-1 → UTF-8 module loading fix in this PR resolves the mojibake)

If this is helpful, copy the block below into the PR description to auto-close these issues on merge.

Fixes #25169
Fixes #10747
Fixes #25767

🤖 Generated with Claude Code

[robobun]

Checked the three suggested issues against this branch:

• A build option to disable converting non-Latin characters to Unicode codes #10747 is a feature request about string-literal escaping under --target=bun. This PR only stops rewriting regex literals and raw (tagged) template contents; string literals still escape, so that issue is unchanged.
• Unicode double-encoding when running --target=node bundle under Bun runtime #25767's repro already prints correctly on main; the hashbang variant is the subject of bundler: gate @bun pragma and @bun-cjs wrapper on build target, not entry hashbang #33859.
• bunx --bun vite change emoji to \uxxxx #25169 depends on how unplugin-vue-router builds the header string and I couldn't reproduce it locally, so not claiming it.

Leaving the closes list as-is.

[github-actions]

This PR may be a duplicate of:

1. Preserve non-ASCII source text in TemplateStringsArray.raw and RegExp.source #33867 - Nearly identical fix: same print_raw_template_literal/print_reg_exp_literal changes in src/js_printer/lib.rs, same clone_latin1 → clone_utf8 sites, same EXPECTED_VERSION bump, overlapping issue references
2. fix(transpiler): preserve non-ASCII in String.raw and RegExp.source #31394 - Earlier attempt at the same fix: modifies the same printer functions and clone_latin1 → clone_utf8 conversion sites with the same version bump
3. Fix tagged template literal with unicode #15047 - Older PR fixing the same bug class (tagged template unicode / regex source non-ASCII escaping), references the same issues (raw tagged template literals show escapes for non ascii text #8745, Bun prints Unicode \uFEFF differently from Node,Deno #15492, Transpiler not respecting String.raw that containing emoji #16763, String.raw Iterator encoding error #18115, satori-html / ultrahtml packages not working in Bun #8207)

🤖 Generated with Claude Code

[robobun]

On the duplicate-PR report:

• Fix tagged template literal with unicode #15047 is the original fix from before the Rust rewrite; the .zig files it touches no longer exist.
• fix(transpiler): preserve non-ASCII in String.raw and RegExp.source #31394 is the same core fix applied to the Rust printer and the five clone_latin1 runtime-transpiler sites.
• Preserve non-ASCII source text in TemplateStringsArray.raw and RegExp.source #33867 was opened in parallel with this one and covers the same set of sites as fix(transpiler): preserve non-ASCII in String.raw and RegExp.source #31394.

This PR additionally gates the zero-copy Latin-1 external-string paths on is_all_ascii (ref_counted_resolved_source callers in jsc_hooks/AsyncModule, OutputFile::to_bun_string{,_ref}), handles the vtable put in the transpiler cache so on-disk entries are tagged UTF8 rather than LATIN1 when non-ASCII, switches the Bake dev-server HMR patch path, and updates the two existing tests that encoded the old behaviour (bunshell.test.ts "escape unicode" and the skipped 14976 case).

Added Co-authored-by for the authors of #15047 and #31394.

[robobun]

Same root-cause fix landed here 23s before #33867 and with broader coverage (OutputFile, DevServer, bunshell.test.ts). Closing #33867 in favour of this one.

One thing #33867 had that may be worth lifting: test/regression/issue/18115.test.ts on farm/392d212f/fix-raw-template-non-ascii runs the same fixture through the runtime transpiler cache round-trip (write then read back from disk), --watch, and the // @bun already-bundled path in addition to the plain sync load, so each of the clone_latin1 → clone_utf8 handoff sites is exercised independently.

[claude]

Thanks for the quick fix on the bytecode SourceCodeKey path — that and the new compile/BytecodeNonAsciiRegexAndRawTemplate tests look right. I didn't find further issues, but this changes a load-bearing invariant (--target=bun output is no longer guaranteed ASCII) across ~8 encoding-sensitive consumers in the module loader, transpiler cache, and C++ bytecode path, so it's worth a human pass to confirm no other Latin-1-assuming sites were missed.

Extended reasoning...

Overview

This PR stops the JS printer from rewriting non-ASCII code points to \uXXXX inside regex literal patterns and tagged-template raw contents (both are runtime-observable via .source / .raw), and updates every downstream consumer that previously relied on --target=bun printer output being pure ASCII/Latin-1. Touches: js_printer/lib.rs (the two escape loops removed), RuntimeTranspilerStore.rs / jsc_hooks.rs / AsyncModule.rs / DevServer.rs (clone_latin1 → clone_utf8, plus is_all_ascii gates on the zero-copy ref_counted_resolved_source path), OutputFile.rs (to_bun_string{,_ref} gated on is_all_ascii), RuntimeTranspilerCache.rs (vtable put tags UTF8 for non-ASCII, cache version bumped 22→23), and ZigSourceProvider.cpp (bytecode build entry points now decode via Bun::toString so the SourceCodeKey matches runtime). Four test files updated/extended.

Security risks

None identified. No auth/crypto/permissions surface; the change is purely about string encoding fidelity between the printer and its consumers.

Level of scrutiny

High. This is not a mechanical change — it removes a global invariant ("printer output for target=bun is pure ASCII") that at least eight separate consumers across Rust and C++ were silently depending on, and replaces it with per-site is_all_ascii gates or unconditional UTF-8 decoding. Getting one site wrong yields mojibake or (as my earlier comment caught) silent bytecode-cache misses. The fix touches hot module-load paths, the on-disk transpiler cache format, zero-copy external-string construction, and an FFI boundary. Each individual edit is small, but the correctness argument is "I found and updated every consumer," which is exactly the class of change that benefits from a second pair of eyes.

Other factors

• My previous review flagged the --bytecode SourceCodeKey mismatch; the author confirmed it with BUN_JSC_verboseDiskCache=1, fixed it in 60c729b via Bun::toString, and added regression tests for both ESM and CJS. That thread is resolved.
• Test coverage is thorough for the observable behaviour (.source/.raw across Latin-1/BMP/BOM/astral, round-trip through bun build --target=bun, bytecode cache-hit assertion), and two pre-existing tests that encoded the old buggy behaviour were corrected.
• The PR description explicitly notes a coordination point with #33447 on the EXPECTED_VERSION bump, and there are three prior/parallel PRs on the same fix — a maintainer should confirm this is the one to land.
• Minor perf consideration: new is_all_ascii scans on the watcher / OutputFile paths are cheap (SIMD) and only add a clone_utf8 copy in the rare non-ASCII case, but worth a glance from someone who owns those hot paths.

[claude]

Both earlier findings are addressed and I didn't spot anything new, but this removes the "target=bun output is pure ASCII" invariant and touches every module-loading consumer of it — worth a human pass to confirm the Latin-1-consumer audit is complete.

Extended reasoning...

Overview

Removes non-ASCII → \uXXXX escaping from print_reg_exp_literal and print_raw_template_literal in the JS printer, then updates every downstream consumer that assumed --target=bun output was pure ASCII/Latin-1: RuntimeTranspilerStore, jsc_hooks, AsyncModule, RuntimeTranspilerCache (both inherent and vtable put), OutputFile::to_bun_string{,_ref}, DevServer HMR patch, and the C++ bytecode-generation entry points in ZigSourceProvider.cpp. Bumps EXPECTED_VERSION to 23. 12 files changed with new tests in template-literal.test.ts and bundler_compile.test.ts, plus un-skipping/fixing two existing tests.

Security risks

None identified. No auth/crypto/permissions surface. The change affects string encoding of transpiler output; malformed UTF-8 would go through clone_utf8 → WTF's UTF-8 decoder, which is well-exercised.

Level of scrutiny

High. This removes a load-bearing invariant (all IS_BUN_PLATFORM printer output is ASCII) that multiple hot-path consumers relied on for zero-copy Latin-1 external strings. The PR itself found six such sites; my first review found a seventh (bytecode SourceCodeKey); the sibling PR #33867 found a subset. That iteration history is exactly why a human should confirm the audit — a missed site produces silent mojibake or a silent bytecode-cache miss, not a crash or test failure.

Other factors

• Both of my prior findings were fixed (60c729b, e607b51) and the latest bug-hunt run is clean.
• Test coverage is thorough: Latin-1/BMP/BOM/astral for both .source and .raw, bun build --target=bun round-trip, and --compile --bytecode cache-hit assertions for both ESM and CJS.
• The is_all_ascii gates preserve the zero-copy fast path for the overwhelmingly common all-ASCII case; only genuinely non-ASCII output pays the clone_utf8 transcode.
• The PR description notes a version-bump conflict with #33447 that whichever lands second must resolve.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@test/bundler/bundler_compile.test.ts`:
- Around line 141-167: The stderr assertion in the `itBundled` test for
`compile/BytecodeNonAsciiRegexAndRawTemplate` is checking the wrong verbose
bytecode message. Update the `validate({ stderr })` expectation to match the
actual `BUN_JSC_verboseDiskCache` output for each `format` case
(`generateModule` for `esm`, `generateCJS` for `cjs`) using the test’s `format`
branch, or remove the assertion if you do not want to pin this log string.

In `@test/bundler/transpiler/template-literal.test.ts`:
- Around line 124-127: The negative assertions in template-literal.test.ts are
using uppercase Unicode escape strings, which can miss the actual emitted output
from the printer. Update the checks around the out.js assertions in the template
literal test to use lowercase escape forms, or make the matching
case-insensitive, so the expectations correctly detect escaped output from the
transpiler.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 360ede32-ff9b-4eba-9a59-ea2a38e390d9

📥 Commits

Reviewing files that changed from the base of the PR and between fc865b3 and e607b51.

📒 Files selected for processing (12)

• src/bundler/OutputFile.rs
• src/js_printer/lib.rs
• src/jsc/AsyncModule.rs
• src/jsc/RuntimeTranspilerCache.rs
• src/jsc/RuntimeTranspilerStore.rs
• src/jsc/bindings/ZigSourceProvider.cpp
• src/runtime/bake/DevServer.rs
• src/runtime/jsc_hooks.rs
• test/bundler/bundler_compile.test.ts
• test/bundler/transpiler/template-literal.test.ts
• test/js/bun/shell/bunshell.test.ts
• test/regression/issue/14976/14976.test.ts

[robobun]

CI status: the changes here are green. Builds #71127 and #71149 both failed only on unrelated flakes:

• #71127: test/regression/issue/26030.test.ts on alpine-aarch64 (MySQL docker container health check timed out), plus Windows node-http/postgres/update_interactive_install ECONNREFUSED retries.
• #71149: test/js/bun/http/proxy-stress-concurrent.test.ts on darwin-x64 (1 of 1200 proxy redirects failed), plus Windows dev-and-prod HMR timing and bun-install EBADF retries.

None of the new or modified tests (template-literal.test.ts, bundler_compile.test.ts bytecode cases, bunshell.test.ts, 14976.test.ts) appear in any failure annotation; the linux-x64-asan test lane passed on both builds. Ready for review.

[robobun]

Issue #33930 reports the control-character variant of this bug: raw NUL bytes (0x00) inside a regex literal are rewritten into the literal 6-character text \u0000, corrupting RegExp.prototype.source (length 23 instead of 13 in the report's repro). Printing the pattern text verbatim here fixes that case as well, so I added it to the Fixes list.

For reference, a minimal subset of this change covering only ASCII control characters (one range bound in print_reg_exp_literal plus the matching print_raw_template_literal bound, no module-loading encoding changes needed since 0x00-0x7F is Latin-1 safe) with tests for the raw-NUL repro is on the farm/94e20982/regex-literal-control-chars branch, in case the full non-ASCII change gets split up.

[robobun]

Rebased onto main at 5f7e62d (633 commits). Conflicts resolved: OutputFile::to_bun_string and RuntimeTranspilerCache::put (inherent method) were removed by dead-code cleanup on main, so those hunks are dropped; EXPECTED_VERSION is now 25. Re-audited for new Latin-1 consumers added since the original branch: none found (new REPL/DevServer/pm-view sites already go through clone_utf8 or raw bytes).

All 13 template-literal.test.ts tests pass locally (11 fail on the current release). 14976.test.ts, bundler_compile.test.ts bytecode cases, bunshell.test.ts, transpiler.test.js, bundler_string.test.ts, bundler_bun.test.ts, transpiler-cache.test.ts and bun-pragma.test.ts all pass.

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[brainkim]

@claude I believe in you you can do it.

[claude]

Code review found no issues

No high-confidence issues detected in this change.

[brainkim]

@alii @claude @robobun It seems like this PR is superseded by #44833, namely by this commit, which down-levels template tags when targeting Bun. A much simpler solution for this impasse?