Skip to content

Fix stack overflow when filtering many entries (#13) and prepare 2.2.0 - #20

Merged
ririsoft merged 4 commits into
masterfrom
fix/issue-13-stack-overflow
Sep 28, 2026
Merged

ririsoft merged 4 commits into
masterfrom
fix/issue-13-stack-overflow

Conversation

@ririsoft

@ririsoft ririsoft commented Sep 28, 2026 •

Copy link
Copy Markdown
Owner

Fixes #13.

Problem

Filtering out many consecutive entries with WalkDir::filter overflowed the stack on threads with a small stack, such as tokio worker threads. The private walk and walk_entry functions called each other recursively through boxed futures, once for every ignored entry. As a result, stack depth grew with the number of filtered-out entries. Without a filter, each entry is returned right away, which is why the unfiltered case worked.

Changes

  • Fix stack overflow when filtering many entries: the traversal is now a single loop inside stream::unfold, with no recursion and no per-entry boxing. The public API is unchanged.
  • Do not read directories filtered with IgnoreDir: read_dir was called on them even though they were never walked. That was wasted work and could return IO errors, such as permission denied, for directories the user asked to skip.
  • Prepare release 2.2.0: bump the version, update the CHANGELOG, and move dependencies to their latest releases: futures-lite 2.6, async-fs 2.2, tempfile 3.27. No dependency has a newer major version.

Tests

  • tests/issue_13_stack_overflow.rs is a regression test. It runs WalkDir inside tokio::spawn on a runtime with a 512 KiB stack, with 4,000 ignored entries.
    • It covers both Ignore and IgnoreDir.
    • It aborts with a stack overflow on the old code in both debug and release builds. In release builds, the old code overflows between 1,000 and 2,000 entries.
    • It passes with the fix, including a manual stress run with 20,000 entries.
    • 64 KiB turned out to be too small a stack for Windows debug builds, even for the fixed code.
    • tokio is added as a dev-dependency only.
  • A new portable unit test covers read_dir errors on subdirectories. It also keeps line coverage above 90% on Windows, where the unix-only tests don't run.
  • A new unix unit test checks that an unreadable directory filtered with IgnoreDir produces no error.
  • cargo test, cargo clippy --all-targets -- -D warnings, cargo fmt --check and cargo publish --dry-run all pass.
  • cargo semver-checks against 2.1.0 reports no semver update required.

Walking recursed through boxed futures for each filtered-out entry,
overflowing small stacks such as tokio worker threads. Rewrite the
traversal as a loop and add a regression test.

Fixes #13
Reading them was wasted work and could yield IO errors for
directories the user asked to skip.
Windows CI does not run the unix-only error tests, which dropped line
coverage below 90% after the traversal rewrite. Cover the read_dir
error path portably and factor the IO error construction into a
helper.
@ririsoft
ririsoft force-pushed the fix/issue-13-stack-overflow branch from 2bb6f7c to 7aa5cee Compare September 28, 2026 13:37
@ririsoft
ririsoft merged commit 73203b4 into master Sep 28, 2026
2 checks passed
@ririsoft
ririsoft deleted the fix/issue-13-stack-overflow branch September 28, 2026 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

stack overflow

1 participant