Repository navigation
Conversation
Refs meta#9529 Flow's file crawl follows every directory symlink. jwalk's loop check compares the raw link target with the paths of the directories above it, so it never catches a relative link: https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229 Packages linked into each other's `node_modules` (eg `packages/a/node_modules/b -> ../../b` and back, as pnpm links workspace packages) are walked until the OS refuses to resolve the path, once for every path through the links, which in practice never ends. Files reached through a symlink are reported by their real path, so a link to a directory under the root that the crawl reads anyway adds nothing. This stops following those links, and records each one in `SymlinkMap` so that module resolution still maps paths through it. Links to anywhere else (out of the root, or outside the directory passed to `flow ls`) are still followed - a follow-up stops at cycles among them. One difference: the initial check now resolves through a link whose path is `[ignore]`d, as rechecks and saved-state init already do. Before, it only resolved the files it had visited through the link. Test plan: New `node_modules_symlink_cycles` and `symlink_into_included_dir` tests, which create their links in `test.sh`, and unit tests. The full `flow dev-tools runtests` suite passes (macOS). On react/metro installed with pnpm (react/metro#2110) with its `[ignore]` workaround removed, `main` hangs on `flow ls` and `flow check`. With this, `flow check` finishes in 2.3s with the same result as with the workaround.
|
This is no.3 of the stack described in #9530 (comment), following #9532 and #9533. |
|
This pull request has been imported. If you are a Meta employee, you can view this in D132920137. (Because this pull request was imported automatically, there will not be any future comments.) |
| /// Whether the crawl reads the directory at the real path `path` and reports each file under it | ||
| /// that it reports through a symlink to it: the directory is under the implicitly included root | ||
| /// (and `subdir`), and isn't pruned. `[include]` and libdef paths don't count, as a pattern that | ||
| /// matches a directory needn't match the files under it. | ||
| fn crawl_reaches_dir( | ||
| options: &FileOptions, | ||
| root: &Path, | ||
| subdir: Option<&Path>, | ||
| path: &Path, | ||
| can_prune: bool, | ||
| all: bool, | ||
| include_libdef: bool, | ||
| ) -> bool { | ||
| options.implicitly_include_root | ||
| && path.starts_with(root) | ||
| && subdir.is_none_or(|subdir| path.starts_with(subdir)) | ||
| && !path | ||
| .ancestors() | ||
| .take_while(|dir| dir.starts_with(root)) | ||
| .any(|dir| crawl_prunes_dir(options, can_prune, dir, all, include_libdef)) | ||
| } |
There was a problem hiding this comment.
This still feels like reimplementing things and why does it only work for options.implicitly_include_root
| ); | ||
| } | ||
|
|
||
| #[test] |
There was a problem hiding this comment.
are these tests actually useful or just repeating what the integration tests are doing
|
Apologies @SamChou19815 - I'm trying not to push slop but it's not my codebase or language so I admit I'm quite reliant on AI here! If you'd rather I'd leave it to you lmk, I'm just looking to unblock Metro and pnpm really. Working on trimming this down now. |
|
Updated for more re-use of existing crawl/prune code, cut a redundant unit test, and eliminated IO (realpath) during resolution. Down to |
…s as covering a directory
… the root, and cover each case in tests
| && entry.parent_path().starts_with(&root_buf) | ||
| && let Ok(target) = cached_canonicalize(&entry.path()) | ||
| && target.starts_with(&root_buf) |
There was a problem hiding this comment.
still unclear to me why there are so many conditions needed here
| subdir: Option<&Path>, | ||
| path: &Path, | ||
| include_libdef: bool, | ||
| subtree: bool, |
There was a problem hiding this comment.
This is "is this specific path included?" vs "is everything under this directory included?".
Part of an effort to reuse this code where we previously had a second method you correctly called out as a reimplementation. I'll see if we can refine it a bit further.
…and trim the symlink tests
|
Slimmed a bit further, removed the Feel free to adjust the implementation here, I've no opinion on it myself - as long as Flow doesn't hang on real world node_modules layouts I can go back to my work and leave you alone :) |
|
This pull request has been merged in 6a84f01. |
…n flow ls .. Follow-up to meta#9534, which stops the file crawl following a symlink to a directory it reads anyway. - A change to a TypeScript global declaration re-inits the server, which crawls again in the same process. The crawl took real paths from a process-wide canonicalisation cache that was never cleared, so a symlink that had changed since it was cached still resolved to its old target. The new target's files went missing, and paths through the symlink resolved to the old one. This was already the case for symlinks the crawl follows, and meta#9534 extended it to the decision to skip one. A re-init now clears the cache before it crawls. - `flow ls` of a directory containing the root, eg `flow ls ..`, treated a symlink under the root to anywhere in that directory as read anyway, so files outside the root that it lists only through such a symlink went missing. The target now has to be under the root as well. Test plan: New `symlink_change_before_reinit` test, which changes a skipped and a followed symlink before a re-init, and `symlinks_out_of_root` now also lists `..`. Both fail without this change. The full `flow dev-tools runtests` suite passes (macOS).
… the root's parent Follow-up to meta#9534, which stops the file crawl following a symlink to a directory it reads anyway. - A change to a TypeScript global declaration reinits the server, which crawls again in the same process. The crawl took real paths from a process-wide canonicalisation cache that was never cleared, so a symlink that had changed since it was cached still resolved to its old target. The new target's files went missing, and paths through the symlink resolved to the old one. This was already the case for symlinks the crawl follows, and meta#9534 extended it to the decision to skip one. An init from scratch now clears the cache before it crawls. - `flow ls` of a directory containing the root, eg `flow ls ..`, treated a symlink under the root to anywhere in that directory as read anyway, so files outside the root that it lists only through such a symlink went missing. The target now has to be under the root as well. Test plan: New `symlink_change_before_reinit` test, which changes a symlink's target before a reinit, and `symlinks_out_of_root` now also lists `..`. Both fail without this change, and the new test also fails with `runtests --saved-state`. The full `flow dev-tools runtests` suite passes (macOS).
Summary: Fixes #9529 Last part of the stack #9532, #9533, #9534 (split from #9530). #9534 stops the file crawl following a symlink to a directory it reads anyway, but it still follows any other symlink, eg out of the root or out of the directory passed to `flow ls`, and goes round a cycle of them until the OS refuses to resolve the path. With a few packages linked to each other, eg `flow ls packages/a` in a pnpm workspace, that never ends in practice. This stops following a symlink to a directory the crawl passed through to reach it. Every walk now ends, though a dense web of links out of the crawl is still walked once per path through it, which grows quickly with the number of links. Pull Request resolved: #9535 Test Plan: `symlinks_out_of_root` now also has a cycle out of the root, whose files `main` lists once per lap. The full `flow dev-tools runtests` suite passes (macOS). On react/metro installed with pnpm (react/metro#2110), `flow ls packages/metro-config` hangs on `main`. With this, `flow ls` of each workspace package finishes in under a second, and `flow full-check` gives the same result in the same time. Reviewed By: SamChou19815 Differential Revision: D133227488 fbshipit-source-id: 889fee75094aed805ff360dbd21939cbe8cf465b
… the root's parent Follow-up to meta#9534 and meta#9535, which stop the file crawl following a symlink to a directory it reads anyway or to one the walk is already in. - A change to a TypeScript global declaration reinits the server, which crawls again in the same process. The crawl takes real paths from a process-wide canonicalization cache that was never cleared, so a symlink changed since it was cached still resolves to its old target: the new target's files go missing, and paths through the symlink resolve to the old one. This was already the case for symlinks the crawl follows, and meta#9534 extended it to the decision to skip one. An init from scratch now clears the cache before it crawls. - `flow ls` of a directory containing the root, eg `flow ls ..`, treated a symlink under the root to anywhere in that directory as crawled directly, so files outside the root that it lists only through such a symlink went missing. The target now has to be under the root as well. A reinit from saved state still keeps the cached targets. That's unchanged here. Test plan: New `symlink_change_before_reinit` test, which changes a symlink's target before a reinit, and `symlinks_out_of_root` now also lists `..`. Both fail without this change, and the new test also fails with `runtests --saved-state`. The full `flow dev-tools runtests` suite passes (macOS).
Refs #9529
Flow's file crawl follows every directory symlink. jwalk's loop check compares the raw link target with the paths of the directories above it, so a relative target like
../../bnever matches, jwalk descends with a longer path and walks again:https://github.com/Byron/jwalk/blob/a5b1ea634fcacee06f7ef6e9411e0cfecfccb128/src/core/dir_entry.rs#L219-L229
Packages linked into each other's
node_modules(egpackages/a/node_modules/b -> ../../band back, as pnpm links workspace packages) are walked until the OS refuses to resolve the path, once for every path through the links, which in practice never ends.Files reached through a symlink under the root are reported by their real path, so following one to a directory under the root that the crawl reads anyway reports nothing new. This stops following those links: the link and its target have to be under the root, the target has to be crawled as a whole (the implicit root, or an
[include]glob that covers it), and a pruned target is still followed so the links inside it resolve. Each skipped link is recorded inSymlinkMap, which now also maps paths under a directory link, so module resolution still works through it without IO. Other links are still followed - a follow-up stops at cycles among them.One difference:
[ignore]is now matched against a skipped link's target by its real path, as it is for the target itself. So a link whose own path is ignored now resolves on the initial check, as it already did on rechecks and saved-state init, and an ignored directory under the target is no longer reached through the link.Test plan:
node_modules_symlink_duplicatesnow also links three workspace packages to each other, whichmainnever finishes crawling, and the newsymlink_cycles_include_onlydoes the same with an[include]glob and no implicit root.node_modules_with_symlinksnow also resolves through two links, through a link to a pruned directory, and through a link whose own path is ignored. Newsymlinks_out_of_roottest for links the crawl must still follow, and a unit test for resolving through a cyclic map, which a stale canonicalization can produce on re-init. Each fails or hangs without the part of this change it covers, and the fullflow dev-tools runtestssuite passes (macOS).On react/metro installed with pnpm (react/metro#2110) with its
[ignore]workaround removed,mainhangs onflow lsandflow full-check. With this,flow full-checkfinishes in 2-3s with the same result as with the workaround.