Repository navigation
node:module: select the Module._extensions handler of a require() with one rule, and pass it the file name - #47527
Conversation
…h one rule transpile_file read the table of Module._extensions handlers in two places. A counter of built-in extensions that have a handler chose one. The first read matched the module key with its ?query. The second read ran only after the entry point was loaded, so a compiled executable and the REPL never reached it. registered_extension_for_require is now the one selection. It matches the path of the key and reads no flag. A ?raw query or a type attribute that names a loader wins over a handler. The second read and the counter are gone. The built-in handlers read the extension from the path of the key.
|
Status of #47527: open, stacked on #47216. It needs a decision by a maintainer on the six rows at the top of the notes in the PR body. Reproduction on main (f7607a3), plain // x.foo
module.exports = "plain x.foo";
// main.cjs
const Module = require("node:module");
const original = Module._extensions[".js"];
Module._extensions[".js"] = (m, f) => original(m, f);
Module._extensions[".foo"] = (m, f) => { m.exports = "the .foo handler got " + require("node:path").basename(f); };
console.log("with query:", require("./x.foo?query"));Main prints Without the CI of 0435891 (build 136201, finished):
|
… with a query A handler that a require() selects for a key with a ?query got the key as its filename, and a hook that reads the file threw ENOENT. The selection now returns the file name with the entry, and the handler gets that name. module.filename and the require.cache key keep the query. __filename of a module that a handler compiles is the name that the handler gives to module._compile. A wrapper hands that file name back to the built-in handler. While require() calls a handler, builtinLoader loads the key of that call when it is given the file name of the call, so each query stays its own module. A loader that --loader or bunfig [loader] names for an extension with no default loader wins over a handler. A file that a standalone executable holds is not handed to a handler, with a query or without one. Fix clippy::or_fun_call in NodeModuleModule__builtinHandlerLoader.
|
a18c042 and 21562f4 answer the reviews of 0424ed5 and a18c042.
The notes of the PR body start with a table of six rows that need a decision by a maintainer. |
There was a problem hiding this comment.
Nothing blocking. The comments below are optional suggestions. There is no need to push a fix for them before merging.
Beyond the inline nits, I also checked the new currentExtensionHandlerCall thread-local in src/jsc/bindings/ModuleLoader.cpp — its key/filename references point at locals of fetchCommonJSModuleNonBuiltin that outlive the handler call, and SetForScope nests correctly under a re-entrant require() from inside a handler — and the repurposed ResolvedSource.source_url on the CommonJSCustomExtension tag, whose only consumer is the new read at src/jsc/bindings/ModuleLoader.cpp:930. The second push (handler now receives the query-less file name; a configured non-default loader now wins over a handler) covers the two actionable points from my earlier review.
Extended reasoning...
Reworks how a CommonJS require() selects a Module._extensions handler across Rust (jsc_hooks.rs, NodeModuleModule.rs, VirtualMachine.rs) and the C++ module loader, with a new thread-local handler-call record and a repurposed source_url field; no security-sensitive surface. The hunt was cut off at its finding cap and two nits are posted inline, so this is a ruled-out note rather than an approval.
|
0435891 answers the two optional findings of the last review.
|
Stacked on #47216.
Problem
bun repl, aModule._extensionshandler on a made-up extension (.hbs) does not run. A handler on a built-in extension makes it run.?queryit is the opposite. When it runs,fs.readFileSync(filename)throwsENOENT.transpile_file(src/runtime/jsc_hooks.rs) reads the handler table in two places, chosen by the counterhas_mutated_built_in_extensions.Fix
registered_extension_for_require(src/jsc/NodeModuleModule.rs) is the one selection, from the path of the key. A loader from the importer (?raw,type) or from the configuration wins.module.filenamekeeps the query. The built-in handler loads the key when a wrapper hands the name back.test/js/node/module/require-extensions.test.ts(25 of 54 fail on the base),test/aot/strict-flag-refuses-as-without-aot.test.ts.Background
Module._extensionsmaps an extension to a handler(module, filename). Bun keeps the stored functions in a table.?queryis part of a module key.process.dlopenalready gets the path.ENOENT, andtsx.require()takes 1,793 ms, not 32 ms.Downsides
filenameis notmodule.filenamefor a key with a query, and.jshooks now run for it. tsx withtsx/cjsand a scope: 104 ms, not 32 ms.what?no longer reaches a.jsread hook. Main and Node load it.require()pays +324 of 67,384 instructions (implementrequire.extensions#18478's fast path).Notes
Decisions for a maintainer. No maintainer has ruled on these rows. Each is pinned by a test, so a different ruling changes one expectation.
.foohandler,require("./x.foo?v=1"): thefilenameargumentx.foo?v=1, equal tomodule.filename, while no built-in extension has a handler. No call otherwisemodule.filenameon both of its routesx.foo.module.filenameand therequire.cachekey arex.foo?v=1__filenamein a CommonJS module with a query that a hook compiles withmodule._compile(code, filename)_compiledelete require.cache[__filename]does not remove the module.jshook that reads the file, path<dir>/what?/x.jsENOENT reading "<dir>/what". #41929 pins the load.foohandler and--loader .foo:textor bunfig[loader]require("./p.js", { type: "js" })under a.jshookbun build --compile --aotexecutable,.foohandler, file on diskCannot load, or the handler once a built-in extension has onemodule._compilein it is code from a stringThe stack. #47216 makes the built-in
.jshandler pick the loader from the file name. Without it this change turns a load into the error of #47175 for a.jshandler that is assigned or passed through under a TypeScript key with two dots, and for a?queryon a TypeScript or JSX file under a.jshook. On main those requires never reach the handler. This PR deletes the counter bookkeeping that #47216 extends (shadows_default_loader), and makesNodeModuleModule__builtinHandlerLoaderread the extension from the path of the key.The report. Two files in one directory, plain
bun run.Main prints
with query: plain x.foo. Without the two.jslines it printswith query: the .foo handler got x.foo?query. This branch printswith query: the .foo handler got x.fooin both cases.What the two reads did. The counter came with #18478 (relanded as #18686) as a cost tier: "mutating a built in extension will fire this slow path in more situations". The first read ran when a key of the default loader table (
.js,.css,.json, 23 keys) had a handler. It ran before the loader was known, so a match also won over?rawand atypeattribute. The second read ran only when the first did not, the path had no loader, andhas_loadedwas set.has_loadedis set when the source of the entry point is produced (#33371). A compiled executable and the REPL do not set it. The counter stayed up when the original handler was assigned back. Nobody chose the operand of each read, their order against?rawandtype, or thehas_loadedgate.The first read took the extension from the text of the query. On main, with a
.jshandler,require("./p?x.js")(a filep) andrequire("./y.bar?v=1.js")call the.jshandler, andrequire("./p.js?q")does not. The payload of adata:URL selected a handler in both reads.Which line is the fix. The selection in
transpile_filehands the module key toregistered_extension_for_require, which cuts the query (path_of_module_key), where main handed it to the matcher as it was. The gate reads the table length, not the counter, andLoaderResult::importer_chose_loader, whichget_loader_and_virtual_sourcesets where it applies?rawandtype. The rest is the deletion of the second read and of the counter.The file name that a handler gets.
registered_extension_for_requirereturns the file name with the entry: the key cut by the one Rust rule.transpile_filehands it to C++ insource_urlwhen it is not the key. C++ has no cut of its own. Node calls a handler withmodule.filename, a path (loader.js#L1596).internalRequire(src/js/builtins/CommonJS.ts) already handsprocess.dlopenthe path of a.nodeid (#33148).A wrapper hands that file name back to the built-in handler. While
require()calls a handler,evaluateCommonJSCustomExtensionrecords the key and the file name of the call (currentExtensionHandlerCall, per thread, on the stack).builtinLoaderloads the key when it is given that file name. It does not readmodule.filename, which script can assign. Without this step, threerequire()calls of one ES module (?v=1,?v=2, no query) under a pass-through.jswrapper return one instance, andimport.meta.urlloses the query.Behaviour changes. The tests in
describe("the handler that a require() selects")pin each of them, except where a line says otherwise.bun build --compileexecutable and inbun repl. The path is one that the build does not resolve. A literalrequire("./t.hbs")is answered at build time, on main and here. With--disallow-code-generation-from-strings=strictthe handler runs too, andmodule._compileof text from the disk is still refused.docs/runtime/code-generation-from-strings.mdxgets that row.--aotexecutable, where main refuses the load withCannot load ... it isn't in this executable. A handler that callsmodule._compilegetsEvalErrorunless the build saw an evaluator. The AOT rule holds: the route "own extension handler compiles" ofcode-the-compiler-never-saw-cases.tspasses before and after.strict-flag-refuses-as-without-aot.test.tspins the two roads.?query, also on a built-in key. It gets the file name. Such a key gets what the key without the query gets: a.jshook that compiles the text throwsSyntaxErrorfor an ES module or a TypeScript file (node:module: make module._compile() accept ES module text and a hashbang line #47519 changes that), atsconfig.jsonwith a comment throws under a.jsonwrapper, andmodule.loadedstaysfalsewhen a handler answers by itself.require(id, { type })gets the attribute's loader also while a built-in key has a handler. Atypethat names no loader is not a choice.--loaderor bunfig[loader]names for an extension with no default loader wins over a handler in every state. With bunfig".hbs" = "text"and handlebars loaded,require("./t.hbs")stays the text. For an extension with a default loader the handler wins, as on main:--loader .js:jsxdoes not switch off a.jshook..sh,.base64,.fileor a--loaderextension runs in every state. So does a handler under a key with two dots (.lit.md, the shape of coffeescript's.coffee.md), which node:module: let the .js extension handler pick the loader from the file name #47216 reaches through the counter.data:URL never select a handler. The query starts at the first?of the key.--compileexecutable holds is not handed to a handler, with a query or without one. That is main's result for both forms.bun test --isolatea handler that an earlier test file left is still in the table. It now also answers keys with a query. A file that awaits an async export of./a.js?v=2after another file left a.jswrapper times out here and passes on main. The same file without the query times out on main too. bun test --isolate: reset Module._extensions per file and ask it before the source cache answers a require() #47477 resets the table for each file. Not covered by a test here.--watchor--hot, on main and here. Bun watches a file when its own transpiler opens it. That now holds for the keys that newly select a handler.node --watchrestarts for such a file. Watch the file that a runtime plugin's onLoad serves, before the callback runs #47566 adds a watch by path for the file of a runtime plugin'sonLoad. The same call before the handler runs, with the file name that this PR hands over, closes it for handlers. That change needs both PRs, so it is not in this one. Not covered by a test.Node. Node selects in one place,
findLongestRegisteredExtensioninModule.prototype.load, from the file name. For every input that Node defines (16 rows in 4 states of the other handlers, one fresh process each), the cells equal to Node v26.3.0 are 56 of 64 on main, 61 on the base and 64 here.require-extensions.test.tsruns 7 processes of such requests undernodetoo, wherenodeis installed.A name with a
?reaches Node's selection in two ways. When aModule._resolveFilenameoverride returnsx.foo?ns=1, Node walks the whole name, finds no key and calls the.jshandler withx.foo?ns=1. When aregisterHooksresolve hook returns a URL with a query, Node drops the query (loader.js#L1171): the.foohandler getsx.foo, andmodule.filenameand the cache key have no query. Bun selects from the path and hands the handler the path, as the second way does.module.filenamekeeps the key, which is Bun's. No Node route has both.Node hands a file with no registered extension to its
.jshandler. Bun does not, as on main: under a wrapped.js, Node calls the wrapper for./p,./y.barand./x.cjs, and Bun calls it for none of them. ts-node and tsx use that step on Node. #46828 addsModule.prototype.loadwith Node's rule.Real hooks and a
?query. Release builds of main, the base, the first form of this PR (the handler gets the key) and this PR. The hook on a built-in extension now also runs for a key with a query. That is the cause of every row where this PR differs from main.require()under.js,./p.js?v=2.js, a module that doesdelete require.cache[__filename], required twice as./self.js?v=1./t.hbs?v=2ENOENTENOENTENOENT".hbs" = "text",./t.hbs./plain.coffee?v=2ENOENTENOENT./logo.svg?url2 errors building2 errors building{}{}.jshook that reads the file and callsmodule._compile,./q.js?v=2(CommonJS)ENOENT./e.js?v=1(an ES module) or./t.ts?v=1ENOENTSyntaxError, as without the query on every buildSyntaxErrorfor TypeScripttsx.require(), 300 modules, median of 5require("tsx/cjs")also registered, warm tsx cacheregister({ namespace })scopes, one entry[1,2,3][1,2,3][1,1,1][1,2,3][1,2,3]require("tsx/cjs")also registered[1,2,3][1,2,3][1,1,1][1,1,1][1,2,3]tsx appends
?namespace=in aModule._resolveFilenameoverride. Its handler reads the namespace fromfilenameand hands every other file to the handler that it replaced. Here the handler of a scope finds no namespace and declines. The handler that it replaced is Bun's when only scopes are registered. It is tsx's global handler whentsx/cjsis registered too, and that handler transforms the file. With an empty tsx cache the first run of the 104 ms row takes 1 to 2 s. The test "a key that Module._resolveFilename gives a query" has both states.Probes. One fresh process for each cell, release builds of the base and of this branch.
.csshandler, a.jswrapper, a.jswrapper that was restored,.jsassigned to itself). Rows whose result differs between "none" and "an unrelated.csshandler": 6 of 18 on main, 5 of 18 on the base, 0 of 18 here. 31 of 90 cells change against the base:x.foo?queryand./x?q(10),x.foowithtype: "text"(4),p.jswithtype: "text"(3),p.js?qunder the wrapper (1), three query texts that end in.js(3), adata:URL (5), a.shhandler alone (1), a.foohandler with--loader .foo:textwhile a built-in key has a handler (4)..jshandler: 9 keys (.spec.js,.min.js,.es.jsx,.spec.ts,.d.ts,.spec.tsx,.sh,.base64,.file) times 3 shapes (assigned, passed through, pirates). 9 cells change, the three loader names: main and the base give the path of the file, this branch and Node v26.3.0 evaluate it. Cells equal to Node go from 13 to 22. No cell leaves Node's result.Numbers. Release builds (lto, baseline) of the base dbff1d5 and of this branch, same build directory. Instruction counts are gdb single-step counts of one call with the JIT off. A row for one function counts the instructions of that function alone. Whole-call counts include the dynamic linker, whose share moves with the layout of the binary. valgrind, perf, strace and bloaty are not in the build container.
require()that loads a fileimporttranspile_fileandget_loader_and_virtual_sourcealone, no handler:import/require()Bun__transpileFile, no handler:require()of a.jsfile / of a.barfileBun__transpileFile, no handler:importBun__transpileFile,require()of a.jsfile, only.hbsand.handlebarsregisteredBun__transpileFile,require()of a.jsonfile under a pass-through.jswrapperBun__transpileFile, the call that returns the handler,.jswrapper /.hbshandlerrequire()that a handler on a made-up key answers: no query / with a querybuiltinLoaderandNodeModuleModule__builtinHandlerLoaderalone, called by a.jswrapperBun__transpileFile, 6 fixtures.textsize_of::<VirtualMachine>()An earlier form placed the selection before
get_loader_and_virtual_source, so that a handler hit skips it. It added 13 instructions to everyimport. A check of the file name against the keys before the hash walk saved about 150 instructions per miss with two keys. Its cost grows with the number of keys, and it skipped a handler for a key with a trailing separator. Neither is in the PR.Tests.
describe("the handler that a require() selects")inrequire-extensions.test.ts:.fooalone, a wrapped.js, a restored.js,.css, handlers on 5 more keys, and three--loaderstates. The requests have keys with two?, a separator in the query, and names that are not ASCII.nodewhere it is installed: a handler alone on.sh,.base64and.file, the.jshandler assigned and called under one of them, keys with two dots, names that begin with a dot..fooalone and on.fooand.js, with an ES module under the second.module._compile.Module._resolveFilenamegives a query (the shape of tsx, two states),bun repl, and abun build --compileexecutable with a file on disk and an embedded file, run plain and with=strict.Removing one clause at a time fails tests: the
data:clause (4), the query cut (21), the importer gate (7), "anytypeis a choice" (7), "atypethat names no loader hides?raw" (6), the arm for a built-in handler under another key (3), the query cut in the built-in handler (4), the file name for the handler (16), the configured loader (2), "only an extension with no default loader" (35), the embedded file (1), the record in the built-in handler (2), the name test of the record (1).Suites run on the debug (ASAN) build of this branch:
require-extensions.test.ts(54 pass),strict-flag-refuses-as-without-aot.test.ts(15 pass),require-extensions-override.test.ts,22929-module-extensions-asi.test.ts,node-module-module.test.js,module-resolve-filename-paths.test.js,module-sourcemap.test.js,import-query.test.ts,import-meta.test.js,plugins.test.ts,run-cjs.test.ts,transpiler-cache.test.ts,module-graph.test.ts,test/cli/test/isolation.test.ts,isolation-one-module-fetched-twice.test.ts,test/js/bun/repl/repl.test.ts, and the 48test-require-*andtest-module-*files oftest/js/node/test/parallel. On the release builds of the base and of this branch:module-graph-isolation.test.ts(413 pass on both),disallow-code-generation-from-strings.test.ts(34 pass on both) and the 5 routes about extensions oftest/aot/parts.test.ts(pass on both).cargo clippypasses forbun_jscandbun_runtime.cargo checkpasses forx86_64-pc-windows-msvcandaarch64-apple-darwin.Other open PRs.
bun test --isolateand adds a second reader that mirrors both old reads. After this PR that reader is one call ofregistered_extension_for_require.?rawpicks the loader.importer_chose_loaderis set on those lines, so the two conflict there and nowhere else.module.registerHooks) calls its load hook between the two reads that this PR makes one.what?through a.jshook that reads the file. That test expects the load that this PR turns intoENOENT. Decide in_resolvewhether a?is part of a path or starts a query #36627, Preserve Windows namespace module keys and UNC resolution errors #45053 and resolver: keep ?query and #fragment on file:// URL import specifiers #35601 change where the path of a key ends.path_of_module_keyis the one place here that must follow them.module._compilegets or uses for a key with a query. node:module: add Module.prototype.load and Module.prototype.isPreloading #46828 adds a second selection (Module.prototype.load) with Node's.jsstep.process.envandrequire.extensionssetters on every assignment #45282, require.extensions: assignments from a repeated statement no longer bypass the loader table #38854 and require.extensions: do not register a handler the object refused to store #38991 make the table follow every store. Today a store that JSC answers from an inline cache, and an accessor, do not reach it. A handler that was removed by such a store still runs, on main and here, and here also for keys with a query.Not changed.
require()that reaches the loader. A.nodeid (module loader: don't abort on .node specifiers with a query string #33148), a file embedded in an executable, a key that the ES module registry holds and the--isolatesource cache answer before it. The entry point, a-rpreload andimportof a CommonJS file do not ask the table.has_loadedis still not set in an executable, the REPL, an embedded Worker and a macro. Thereimport()of a file with an unknown extension still runs it as code, andrequire()of one still parses it as TSX.Bun.pluginonLoadstill gets the key with its?query(bun test: Vitest compatibility #44833 and Runtime plugins: callonLoadfor a dotted or colon query, an extension that starts with a digit, and drives A: and Z: #45009 edit it).?in the resolver, the loader and here. A Windows path with the\\?\prefix is cut to\\in all three.?raw.Self-review. The second review ran on the second commit. It kept the selection and asked for: one rule for the path of a key (done: Rust hands the file name to C++), a key for the built-in handler that script cannot write (done: the record of the call), the configured loader (done), an embedded file (done), rows that run under Node, rows for two
?, a separator, names that are not ASCII, a hook that names another file,__filename, the shape of tsx, the closed--aotexecutable, and a body that states each row above. Not done: a ruling by a maintainer on the table at the top, a tracking issue for the rows that are not changed, the--isolatecase (#47477 has the reset), and cases in the three-ways harness oftest/aot(main replaced that harness in #47129 after this branch forked).no test proof · iteration 0 · platform-specific test(s) that do not run on this machine, deferring to CI, which covers all platforms: test/js/node/module/require-extensions.test.ts, test/aot/strict-flag-refuses-as-without-aot.test.ts