Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions docs/runtime/module-resolution.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,8 @@ NODE_PATH=./packages:./lib bun run src/index.js # Unix/macOS
NODE_PATH="./packages;./lib" bun run src/index.js # Windows
```

Bun reads the `exports` field of a package that it finds through `NODE_PATH`.

### Custom conditions

The `--conditions` flag specifies the conditions to use when resolving packages from `package.json` `"exports"`.
Expand Down
105 changes: 105 additions & 0 deletions src/resolver/resolver.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3064,6 +3064,22 @@ impl<'a> Resolver<'a> {
if !node_path.is_empty() {
let delim = if cfg!(windows) { b';' } else { b':' };
for path in node_path.split(|&b| b == delim).filter(|s| !s.is_empty()) {
// A subpath that did not fit the buffer is empty.
if let Some(esm) = esm_.as_ref()
&& !esm.subpath.is_empty()
&& let Some(status) = self.load_node_path_package_exports(
path,
import_path,
esm.subpath,
kind,
out,
)
{
if let Some(d) = self.debug_logs.as_mut() {
d.decrease_indent();
}
return status;
}
let Some(abs_path) = self
.fs_ref()
.abs_buf_checked(&[path, import_path], bufs!(node_modules_check))
Expand Down Expand Up @@ -3845,6 +3861,95 @@ impl<'a> Resolver<'a> {
unreachable!("TODO: implement enqueueDependencyToResolve for non-root packages")
}

/// The file that "exports" of a package in `node_path`, a `NODE_PATH` entry, has for `subpath`, which is `import_path` without the name of the package. `None` without one: the caller goes on to `load_as_file_or_directory`.
#[cold]
#[inline(never)]
fn load_node_path_package_exports(
&mut self,
node_path: &[u8],
import_path: &[u8],
subpath: &[u8],
kind: ast::ImportKind,
out: &mut MatchResult,
) -> Option<MatchStatus> {
use crate::package_json::EntryData;

// As in Node, the folder has the whole text in front of the subpath as its name: `npkg@1.2.3`, not `npkg`.
let name = import_path.get(..(import_path.len() + 1).checked_sub(subpath.len())?)?;
let abs_package_path = self
.fs_ref()
.abs_buf_checked(&[node_path, name], bufs!(esm_absolute_package_path))?;
// `dir_info_cached` opens a file of this name on every call, because it does not remember ENOTDIR.
let is_listed_file = bun_paths::dirname(abs_package_path)
.and_then(|parent| self.dir_info_cached(parent).ok().flatten())
.and_then(|parent| {
parent.get_entry(self.generation, bun_paths::basename(abs_package_path))
})
.is_some_and(|found| {
found.entry().kind(self.rfs_ref(), self.store_fd) != Fs::file_system::EntryKind::Dir
});
if is_listed_file {
return None;
}
let package = self.dir_info_cached(abs_package_path).ok().flatten()?;
let package_json = package.package_json()?;
// Node loads "main" when "exports" is `false` or a number.
Comment on lines +3881 to +3896

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 (optional) Projects whose NODE_PATH contains an entry under a regular file (the pnpm shape resolver.rs:4833 was written for) now pay two more failed open(2) calls on every resolution that reaches NODE_PATH, on top of the one the base paid. load_node_path_package_exports at src/resolver/resolver.rs:3881-3896 calls dir_info_cached on the entry and on <entry>/<name>, and dir_info_cached_miss returns Ok(None) on ENOTDIR at resolver.rs:4836-4840 without recording the miss, so the helper's probes repeat on every call. Fix: remember ENOTDIR for the entry (cache the not-found like ENOENT at :4867), or probe dir_info_cached(node_path) once per entry and skip both the helper and the plain lookup when it is None.

Why this was flagged

Trigger: NODE_PATH holds an entry whose ancestor is a file (the pnpm-generated list named in the comment at src/resolver/resolver.rs:4829-4835) and the project resolves bare specifiers that miss the node_modules walk. The new helper at resolver.rs:3881-3896 does dir_info_cached(dirname(abs_package_path)) and dir_info_cached(abs_package_path) before the old lookup at :3095. Each walk reaches the file ancestor, open_dir fails with ENOTDIR, and resolver.rs:4836-4840 returns Ok(None) before the mark_not_found at :4867, so nothing is cached and the next resolution opens again. The base did the same two uncached opens once in load_as_file_or_directory; the change doubles them to four per NODE_PATH entry per resolution, in both the runtime and the bundler, for every import that misses node_modules. Remedy: cache the ENOTDIR miss, or check the entry once per loop iteration and skip both lookups.

Verification: Triggered when NODE_PATH holds an entry whose ancestor is a regular file and a bare specifier misses the node_modules walk. In dir_info_cached_miss, an open that fails with ENOTDIR/EISDIR does return Ok(None) at src/resolver/resolver.rs:4836-4840 before the mark_not_found calls at :4867-4868, so the next call re-opens the same path. So the PR adds 2 uncached failed opens per such entry per miss.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not changed in this PR. The count is right: for an entry that is a regular file, or lies below one, one warm call makes 3 failed openat where main makes 1. The PR body has it under Measurements. It applies to a bare specifier that no node_modules has.

I checked both proposals.

  • Remember ENOTDIR in dir_info_cached_miss. This is the root: the arm returns before the two mark_not_found calls. The change also removes the one open that main makes, and it makes the is_listed_file guard of the helper redundant. But it changes the negative cache for every caller of the resolver, so it is not a change for this PR.
  • Skip the helper and the plain lookup when dir_info_cached(entry) is None. None is also the answer for a folder that cannot be listed. With chmod 311 on the entry and a package below it, as a user that is not root: main loads legacy.js by "main", and this PR loads m.js by "exports". With the skip, the package is not found.

The helper cannot tell ENOTDIR from EACCES on the parent, because dir_info_cached answers Ok(None) for both.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on the first proposal (remember ENOTDIR in dir_info_cached_miss). I built it and measured it. I do not send it as a PR.

What it gives. With NODE_PATH set to a file, 1000 failed lookups of 3 names make 4 failed opens (main 34cbc70: 2001).

What it costs. A path that was a file stays "not found" after it becomes a directory, until bust_dir_cache drops that exact path. Measured against main:

  • import "late" builds while node_modules/late is a file. After late becomes a package folder, each later Bun.build() in the process fails with Could not resolve: "late". Main builds.
  • require.resolve("./was/dir") fails once while was is a file. After was becomes a directory, main resolves it. A path that was missing at first also resolves. The change gives MODULE_NOT_FOUND.
  • import "./sub/x" while sub is a file, then Bun.build() again after sub becomes a directory: 0 of 4 shapes build (a file or a directory import, target bun or browser). Main builds 3 of 4.
  • Modules below such a path lose the node_modules above them. With auto-install on, require loaded latest from the registry. On main the installed version loads.

I found no issue that reports the repeated open. How long a cached "not found" may live is the subject of #40279 and #46805, and ENOTDIR belongs in that decision.

An option for this PR that adds no cache entry. RealFS::read_directory already remembers ENOTDIR for the entry (src/resolver/lib.rs:1046) and tells it from EACCES. I applied this head on main 34cbc70 and made the helper ask that listing in place of dir_info_cached(parent):

  • The same 1000 lookups make 2001 failed opens, as on main (this head: 6001).
  • The chmod 311 entry still resolves m.js, as a user that is not root.
  • test/js/bun/resolve/resolve.test.ts of this PR passes (111 pass). I did not run the bundler, AOT and Rust tests.
  • One order changes. An entry that was a file at the first lookup, and then becomes a directory, keeps its cached ENOTDIR. Packages below it resolve by "main", as on main. This head reads "exports" there.
Patch for the helper
--- a/src/resolver/resolver.rs
+++ b/src/resolver/resolver.rs
@@ -3880,15 +3880,30 @@
             .fs_ref()
             .abs_buf_checked(&[node_path, name], bufs!(esm_absolute_package_path))?;
         // `dir_info_cached` opens a file of this name on every call, because it does not remember ENOTDIR.
-        let is_listed_file = bun_paths::dirname(abs_package_path)
-            .and_then(|parent| self.dir_info_cached(parent).ok().flatten())
-            .and_then(|parent| {
-                parent.get_entry(self.generation, bun_paths::basename(abs_package_path))
-            })
-            .is_some_and(|found| {
-                found.entry().kind(self.rfs_ref(), self.store_fd) != Fs::file_system::EntryKind::Dir
-            });
-        if is_listed_file {
+        // SAFETY: `rfs_ptr()` points at the process-global RealFS singleton.
+        let listing: bun_ptr::BackRef<Fs::file_system::real_fs::EntriesOption> =
+            match unsafe { &mut *self.rfs_ptr() }.read_directory(
+                strings::without_trailing_slash_windows_path(Dirname::dirname(abs_package_path)),
+                None,
+                self.generation,
+                self.store_fd,
+            ) {
+                Ok(listing) => bun_ptr::BackRef::new(&*listing),
+                Err(_) => return None,
+            };
+        let is_file_or_below_one = match listing.get() {
+            Fs::file_system::real_fs::EntriesOption::Err(err) => {
+                err.original_err == crate::Error::Sys(bun_errno::SystemErrno::ENOTDIR)
+            }
+            entries => entries
+                .lookup(bun_paths::basename(abs_package_path))
+                .0
+                .is_some_and(|found| {
+                    found.entry().kind(self.rfs_ref(), self.store_fd)
+                        != Fs::file_system::EntryKind::Dir
+                }),
+        };
+        if is_file_or_below_one {
             return None;
         }
         let package = self.dir_info_cached(abs_package_path).ok().flatten()?;
Two test rows: an entry that is a file, or lies below one, is skipped (they pass on main, on this head, and with the patch)
  // pnpm 5 wrote `<bin file>/node_modules` as the first entry.
  it.concurrent.each([
    ["at runtime", ["--no-install", "index.js"], "NODE_PATH works\n"],
    ["in a bundle", ["build", "--target=bun", "index.js"], expect.stringContaining("NODE_PATH works")],
  ])("should skip NODE_PATH entries that are a file or lie below one %s", async (_, args, expected) => {
    const { nodePath, testDir, delimiter } = prepareTest();
    const file = joinP(testDir, "index.js");

    await using proc = Bun.spawn({
      cmd: [bunExe(), ...args],
      env: { ...bunEnv, NODE_PATH: [file, joinP(file, "node_modules"), nodePath].join(delimiter) },
      cwd: testDir,
      stdout: "pipe",
      stderr: "pipe",
    });
    const [stdout, stderr, exitCode] = await Promise.all([proc.stdout.text(), proc.stderr.text(), proc.exited]);

    expect({ stdout, stderr }).toEqual({ stdout: expected, stderr: "" });
    expect(exitCode).toBe(0);
  });

let exports_map = package_json.exports.as_ref().filter(|exports| {
matches!(
exports.root.data,
EntryData::String(_) | EntryData::Array(_) | EntryData::Map(_)
)
})?;
if let Some(debug) = self.debug_logs.as_mut() {
debug.add_note_fmt(format_args!(
"Reading \"exports\" of the package in the NODE_PATH directory \"{}\"",
bstr::BStr::new(abs_package_path)
));
}

let prev_extension_order = self.extension_order;
// A `NODE_PATH` entry need not be in node_modules.
self.extension_order = self
.opts
.extension_order
.kind(kind, package.is_inside_node_modules());
// The node_modules lookup also reads `pkg/key.js` as the key "./key".
let without_js = (bun_paths::extension(subpath) == b".js" && subpath.len() > 3)
.then(|| &subpath[..subpath.len() - 3]);
let mut is_found = false;
for key in core::iter::once(subpath).chain(without_js) {
let esm_resolution = ESModule {
conditions: match kind {
ast::ImportKind::Require | ast::ImportKind::RequireResolve => {
&self.opts.conditions.require
}
ast::ImportKind::At | ast::ImportKind::AtConditional => {
&self.opts.conditions.style
}
_ => &self.opts.conditions.import,
},
debug_logs: self.debug_logs.as_mut(),
stack_check: StackCheck::init(),
}
.resolve(b"/", key, &exports_map.root);
is_found = self
.handle_esm_resolution(
esm_resolution,
abs_package_path,
kind,
package_json,
subpath,
out,
)
.is_success();
if is_found {
break;
}
}
self.extension_order = prev_extension_order;
is_found.then_some(MatchStatus::Success)
}

fn handle_esm_resolution(
&mut self,
mut esm_resolution: crate::package_json::Resolution,
Expand Down
30 changes: 30 additions & 0 deletions src/sema/resolve/code/computed.rs
Original file line number Diff line number Diff line change
Expand Up @@ -222,6 +222,13 @@ impl<'a, 'h> CandidateSearch<'a, 'h> {
}
for node_path in &resolver.code_options().node_paths {
self.walk_below(node_path, b"", SpecifierForm::Path)?;
// The folder has the written name: in a `NODE_PATH` entry a version is a part of it.
let manifest = Some(join_unrooted(node_path, name))
.filter(|package_directory| resolver.is_dir(package_directory))
.and_then(|package_directory| resolver.code_manifest(&package_directory));
if let Some(exports) = manifest.and_then(|manifest| manifest.exports.as_ref()) {
self.try_exact_exported_keys(exports, name)?;
}
}
Ok(())
}
Expand Down Expand Up @@ -262,6 +269,29 @@ impl<'a, 'h> CandidateSearch<'a, 'h> {
}
for node_path in &resolver.code_options().node_paths {
self.walk_below(node_path, b"", SpecifierForm::Path)?;
// After the walk: it ends the search in a folder where more packages start with the prefix than the limit allows, and then no manifest is read.
for name in self.list_installed_names(node_path) {
let manifest = resolver.code_manifest(&join_unrooted(node_path, &name));
if let Some(exports) = manifest.and_then(|manifest| manifest.exports.as_ref()) {
self.try_exact_exported_keys(exports, &name)?;
}
}
}
Ok(())
}

/// Tries the keys of "exports" of a package in a `NODE_PATH` entry that name one subpath each. A key with a star or a key of a folder lists no folder here: of its files, only those that the walk lists are found.
fn try_exact_exported_keys(
&mut self,
exports: &ExportsEntry<'h>,
name: &[u8],
) -> Result<(), ComputedUnknown> {
let ExportsEntry::Map(map) = *exports else {
return Ok(());
};
let subpath_keys = map.entries.iter().filter(|it| it.0.starts_with(b"./"));
for &(key, _) in subpath_keys.filter(|it| !strings::contains_char(it.0, b'*')) {
self.try_exported_specifier(&[name, &key[1..]].concat())?;
}
Ok(())
}
Expand Down
104 changes: 104 additions & 0 deletions src/sema/resolve/code/computed/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -500,3 +500,107 @@ fn packages_in_node_path_are_listed() {
""
);
}

fn node_path_candidates(host: &MemoryHost, limit: u32, prefix: &str, suffix: &str) -> String {
let options = CodeOptions {
node_paths: vec![b"/global".to_vec()],
max_computed_candidates: limit,
..CodeOptions::bun_run(b"/p")
};
let pattern = ComputedPattern {
importer: "/p/main.ts",
prefix,
suffix,
kind: Import,
};
resolve_computed_with(host, &options, pattern)
}

#[test]
#[cfg_attr(miri, ignore)]
fn exact_exported_subpaths_of_packages_in_node_path_are_listed() {
let host = MemoryHost::new(&[
("/p/main.ts", ""),
(
"/global/pkg/package.json",
r#"{ "exports": { ".": "./lib/root.js", "./alias": "./lib/target.js" } }"#,
),
("/global/pkg/lib/root.js", ""),
("/global/pkg/lib/target.js", ""),
("/global/pkg/plain.js", ""),
])
.node_path("/global");
let candidates = |prefix, suffix| node_path_candidates(&host, 1024, prefix, suffix);
assert_eq!(candidates("pkg/al", ""), "/global/pkg/lib/target.js");
assert_eq!(candidates("pk", "/alias"), "/global/pkg/lib/target.js");
assert_eq!(candidates("pk", "g"), "/global/pkg/lib/root.js");
assert_eq!(candidates("pkg/pl", ""), "/global/pkg/plain.js");
}

#[test]
#[cfg_attr(miri, ignore)]
fn exports_of_package_in_node_path_list_no_folder() {
let host = MemoryHost::new(&[
("/p/main.ts", ""),
(
"/global/pkg/package.json",
r#"{ "exports": { "./*": "./*", "./icons/": "./icons/" } }"#,
),
("/global/pkg/a.js", ""),
("/global/pkg/b.js", ""),
("/global/pkg/c.js", ""),
("/global/pkg/icons/x.js", ""),
("/global/pkg/icons/y.js", ""),
])
.node_path("/global");
// The walk lists the two files of `icons`. The package has more files than the limit.
assert_eq!(
node_path_candidates(&host, 2, "pkg/icons/", ""),
"/global/pkg/icons/x.js, /global/pkg/icons/y.js"
);
}

#[test]
#[cfg_attr(miri, ignore)]
fn package_in_node_modules_and_in_node_path_counts_once() {
let host = MemoryHost::new(&[
("/p/main.ts", ""),
(
"/p/node_modules/pkg/package.json",
r#"{ "exports": { "./icons/*": "./dist/icons/*" } }"#,
),
("/p/node_modules/pkg/dist/icons/x.js", ""),
("/p/node_modules/pkg/dist/icons/y.js", ""),
])
.link("/global/pkg", "/p/node_modules/pkg")
.node_path("/global");
assert_eq!(
node_path_candidates(&host, 2, "pkg/icons/", ""),
"/p/node_modules/pkg/dist/icons/x.js, /p/node_modules/pkg/dist/icons/y.js"
);
}

#[test]
#[cfg_attr(miri, ignore)]
fn walk_over_node_path_ends_search_before_manifests_are_read() {
let manifest = r#"{ "exports": { "./alias": "./target.js" } }"#;
let host = MemoryHost::new(&[
("/p/main.ts", ""),
("/global/pka/package.json", manifest),
("/global/pka/target.js", ""),
("/global/pkb/package.json", manifest),
("/global/pkb/target.js", ""),
("/global/pkc/package.json", manifest),
("/global/pkc/target.js", ""),
])
.node_path("/global");
assert_eq!(
node_path_candidates(&host, 2, "pk", "/alias"),
"All(TooManyCandidates)"
);
assert_eq!(host.reads("/global/pka/package.json"), 0);
assert_eq!(
node_path_candidates(&host, 8, "pk", "/alias"),
"/global/pka/target.js, /global/pkb/target.js, /global/pkc/target.js"
);
}
19 changes: 12 additions & 7 deletions src/sema/resolve/code/memory_host.rs
Original file line number Diff line number Diff line change
Expand Up @@ -239,19 +239,24 @@ impl CodeHost for MemoryHost {
};
}
let in_node_paths = || {
let files = self.node_paths.iter().map(|directory| {
for directory in &self.node_paths {
let exported = super::search::find_exported_in_node_path(self, search, directory);
if exported.is_some() {
return exported;
}
let specifier = &join(directory, search.specifier);
find_file_plainly(
let found = find_file_plainly(
self,
&FileSearch {
specifier,
..*search
},
)
});
files
.into_iter()
.find(|it| matches!(it, FoundCode::File { .. }))
);
if matches!(found, FoundCode::File { .. }) {
return Some(found);
}
}
None
};
let found = self.find_through_imports(search);
let found = found.unwrap_or_else(|| find_file_plainly(self, search));
Expand Down
33 changes: 33 additions & 0 deletions src/sema/resolve/code/search.rs
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,39 @@ impl CodeOptions {
}
}

/// What `bun run` finds by the "exports" of the package that the specifier names in `node_path`, a `NODE_PATH` entry. `None` where it goes on to the file or the folder of that name.
#[cfg(test)]
pub(super) fn find_exported_in_node_path<H: Host + ?Sized>(
host: &H,
search: &FileSearch<'_>,
node_path: &[u8],
) -> Option<FoundCode> {
// The folder has the written name: in a `NODE_PATH` entry a version is a part of it.
installed_name(search.specifier)?;
let name = written_package_name(search.specifier)?;
let subpath = [b".".as_slice(), &search.specifier[name.len()..]].concat();
if subpath.len() > SUBPATH_CAPACITY {
return None;
}
let plain = PlainSearch {
host,
parse_manifest: H::parse_package_json,
orders: &BUN_RUN.extension_orders,
kind: search.kind,
prefers_module_field: search.prefers_module_field,
holds: &|condition| BUN_RUN.matches_condition(condition, search.kind),
arena: OnceCell::new(),
};
let directory = join_unrooted(node_path, name);
let manifest = plain.read_manifest(&directory)?;
let exports = plain.read_exports(&manifest)?;
if matches!(exports, ExportsEntry::Invalid) {
return None;
}
let found = plain.find_in_exports(&directory, &exports, &subpath);
matches!(found, FoundCode::File { .. }).then_some(found)
}

#[derive(Copy, Clone, PartialEq, Eq, Debug)]
enum KeyKind {
Exact,
Expand Down
Loading
Loading