feat(concurrency): large-stack task executor adoption and cross-version parallel preparation (proposal large-stack-executor-and-cross-version-parallelism) - #122
Open
Mx-Iris wants to merge 5 commits into
Conversation
…e-stack-executor-and-cross-version-parallelism) 0.6.1 re-ranked the hop pool's QoS per call and slowed dump/interface 3-4x; 0.6.2 partitioned the pool by class and restored the speed; 0.6.3 ships the large-stack TaskExecutor the proposal adopts. The floor skips 0.6.1. The proposal moves to In Progress: the user asked for the implementation on top of the self-contained ABI branch.
…cutor and prepare versions in parallel (proposal draft-large-stack-executor-and-cross-version-parallelism) swift-demangling's StackSafeExecutor decides per demangle / print / remangle whether to hop to its 8 MB pool by probing the calling thread's remaining stack; cooperative threads carry 512 KB, so an async print loop paid one thread round trip and a semaphore wait per printed symbol. The new MachOSymbols.LargeStackTaskExecution.run sets swift-demangling 0.6.3's 16 MB LargeStackTaskExecutor as the task executor preference around every library entry point (indexer prepare, interface builder prepare/printRoot, diffable builder prepare, evolution builder prepare/render, diff renderer, the printer's per-definition entries, the six Dumpable.dump conformers), so the whole pipeline runs inline. Nesting is a no-op, no unstructured Task exists in the library, and below macOS 15 / iOS 18 the body runs unchanged; isEnabled (seeded by MACHO_SWIFT_SECTION_LARGE_STACK_EXECUTOR=0) turns it off. Multi-version preparation is parallel: AnySwiftEvolutionInterfaceBuilder .prepare(maximumConcurrentPreparations:) (default = processor count, 1 = the old serial order), swift-section diff / evolution --jobs N, all through a windowed Collection.concurrentMap(maximumConcurrency:_:) in Utilities. Verified: 1637 tests pass; rendering A/B 78 pairs byte-identical with the executor on and off; SwiftUICore/SwiftUI dump+interface 16-23% faster, a three-version SwiftUI evolution 2.0x faster. Bumps to 0.19.0.
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
The change set touches core indexing/printing orchestration and introduces new concurrency scheduling paths, warranting final human review despite strong test/documentation coverage.
Pull request overview
Implements the large-stack-executor-and-cross-version-parallelism evolution by adopting swift-demangling’s large-stack TaskExecutor across async entry points, and introducing bounded cross-version parallel preparation for diff/evolution (plus CLI --jobs) while keeping rendered output byte-identical.
Changes:
- Add
MachOSymbols.LargeStackTaskExecution.runand wrap key async entry points so demangling/printing/remangling runs inline on 16MB executor threads when supported. - Introduce
Utilities.Collection.concurrentMap(maximumConcurrency:_:)and use it to parallelize multi-input indexing (library builder + CLI), with bounded concurrency and ordered results. - Add tests, docs, workflow filter updates, and bump versions (swift-demangling floor to 0.6.3; bundled version to 0.19.0 with changelog).
File summaries
| File | Description |
|---|---|
| Tests/SwiftSectionCommandTests/EvolutionCommandValidationTests.swift | Adds validation coverage for swift-section evolution --jobs parsing and lower-bound rejection. |
| Tests/SwiftSectionCommandTests/DiffCommandValidationTests.swift | New suite pinning swift-section diff --jobs behavior and validation. |
| Tests/SwiftInterfaceTests/SwiftEvolutionInterfaceBuilderTests.swift | Verifies parallel prepare(maximumConcurrentPreparations:) is byte-identical to serial; clamps window semantics. |
| Tests/SwiftInterfaceTests/BoundedConcurrentMapTests.swift | New unit tests pinning ordering, bounded concurrency, and first-failure semantics of concurrentMap. |
| Tests/MachOSymbolsTests/LargeStackTaskExecutionTests.swift | New tests validating executor adoption behavior (thread identity, nesting, inheritance, disable switch). |
| Sources/Utilities/BoundedConcurrentMap.swift | Introduces windowed concurrentMap(maximumConcurrency:_:) helper used by cross-version prep and CLI. |
| Sources/SwiftPrinting/SwiftDeclarationPrinter.swift | Wraps per-definition printer entry points in LargeStackTaskExecution.run (incl. shell/body split for builders). |
| Sources/SwiftInterface/SwiftInterfaceBuilder.swift | Wraps prepare() and printRoot() bodies in LargeStackTaskExecution.run. |
| Sources/SwiftInterface/SwiftEvolutionInterfaceBuilder.swift | Updates pack façade to forward prepare(maximumConcurrentPreparations:) to erased builder. |
| Sources/SwiftInterface/SwiftDiffableInterfaceRenderer.swift | Wraps diff rendering entry points in LargeStackTaskExecution.run. |
| Sources/SwiftInterface/SwiftDiffableInterfaceBuilder.swift | Wraps prepare() body in LargeStackTaskExecution.run. |
| Sources/SwiftInterface/AnySwiftEvolutionInterfaceBuilder.swift | Adds bounded parallel prepare(maximumConcurrentPreparations:) via concurrentMap, and wraps render APIs in LargeStackTaskExecution.run. |
| Sources/SwiftIndexing/SwiftDeclarationIndexer.swift | Wraps indexing prepare() body in LargeStackTaskExecution.run. |
| Sources/SwiftDump/Dumpable/Struct+Dumpable.swift | Wraps dump entry point in LargeStackTaskExecution.run and updates imports. |
| Sources/SwiftDump/Dumpable/ProtocolConformance+Dumpable.swift | Same as above for protocol conformances. |
| Sources/SwiftDump/Dumpable/Protocol+Dumpable.swift | Same as above for protocols. |
| Sources/SwiftDump/Dumpable/Enum+Dumpable.swift | Same as above for enums. |
| Sources/SwiftDump/Dumpable/Class+Dumpable.swift | Same as above for classes. |
| Sources/SwiftDump/Dumpable/AssociatedType+Dumpable.swift | Same as above for associated types. |
| Sources/SwiftDeclarationRendering/Extensions/Node+.swift | Updates comments to accurately describe StackSafeExecutor behavior and new executor-based amortization. |
| Sources/swift-section/Version.swift | Bumps bundled CLI/library version to 0.19.0. |
| Sources/swift-section/Commands/EvolutionCommand.swift | Adds --jobs, parallelizes document loading with bounded concurrency, and forwards preparation window to builder. |
| Sources/swift-section/Commands/DiffCommand.swift | Adds --jobs, parallelizes indexing/document loading with bounded concurrency, preserves historical serial order under --jobs 1. |
| Sources/MachOSymbols/LargeStackTaskExecution.swift | New adoption seam implementing the executor preference wrapper + enable/disable/supported logic. |
| Package.swift | Raises swift-demangling dependency floor from 0.6.0 to 0.6.3. |
| Documentations/README.md | Adds index entry for LargeStackTaskExecutorAdoption.md. |
| Documentations/Internal/TaskReports/2026-09-03-large-stack-executor-and-cross-version-parallelism.md | New maintainer task report documenting investigation, decisions, and verification. |
| Documentations/Internal/Reviews/2026-07-31-node-store-migration-review.md | Marks previously-open executor-related item resolved, linking to new implementation note. |
| Documentations/Internal/ProjectEvolutionLog.md | Appends evolution-log entry for this work arc and version bump. |
| Documentations/Internal/Modules/SwiftInterface.md | Updates module doc to reflect executor wrapping and parallel preparation behavior. |
| Documentations/Internal/LargeStackTaskExecutorAdoption.md | New implementation note describing executor adoption, bounded parallelism, and measured results. |
| Documentations/Glossary.md | Adds glossary entry for “large-stack executor / LargeStackTaskExecution”. |
| Documentations/Evolutions/README.md | Moves the proposal status to In Progress in the table. |
| Documentations/Evolutions/draft-large-stack-executor-and-cross-version-parallelism.md | Updates proposal status/links and records landing deviations and verification notes. |
| Changelogs/0.19.0.md | New release notes for 0.19.0 describing executor adoption + parallel preparation. |
| AGENTS.md | Updates agent guidance to document executor usage, parallel preparation, and test-environment implications. |
| .github/workflows/macOS.yml | Extends CI test filter to include new suites related to executor adoption and --jobs. |
Review details
- Files reviewed: 37/37 changed files
- Comments generated: 0
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
…ack-executor-and-cross-version-parallelism # Conflicts: # .github/workflows/macOS.yml # Documentations/Evolutions/README.md # Documentations/Evolutions/draft-large-stack-executor-and-cross-version-parallelism.md
…p, serialized event delivery, labeled diagnostics (proposal large-stack-executor-and-cross-version-parallelism) Fifteen review findings on PR #122 (Roadmaps/2026-09-04-pr122-review- findings.md): five defects, one false positive, nine design/style calls. - concurrentMap(maximumConcurrency:) submits through addTaskUnlessCancelled and fails with CancellationError once the calling task is cancelled; the first version used addTask, which a cancelled group still accepts, so a cancelled multi-version preparation indexed every remaining version to the end (reproduced independently by the reviewer). Never a partial array — the unwrap below would trap. - LargeStackTaskExecutionTests guard isEnabled as well as isSupported: under MACHO_SWIFT_SECTION_LARGE_STACK_EXECUTOR=0 (the A/B configuration) the executor-thread assertions went red for a reason the tests do not control. - That variable now accepts 0 / false / no / off (case-insensitive) as off; the literal-"0" comparison let =false measure the executor twice. - parallelPreparationMatchesSerialPreparation prepares the parallel builder FIRST, so it meets cold caches; a three-way barrier pins that a window of three admits three. - SwiftIndexEvents.Dispatcher serializes handler invocation process-wide with a recursive lock: Handler has no Sendable requirement and parallel preparation shares one host handler across N dispatchers on N tasks. - ConsoleEventHandler(label:) prefixes [label]; diff tags old / new, evolution tags each version (AnySwiftEvolutionInterfaceBuilder gained eventHandlersPerVersion:), snapshot inputs use their label or file name. - The redundant import Utilities is gone. Every behavioral fix verified red under a mutated build. Adjudicated as A25–A33: the import MachOFoundation false positive, per-entry wraps, the concurrentMap name, #isolation forwarding, .serialized vs the process switch, the executor-off window, the default lineage window (kept, user's call), the unavoidable dual availability gate, and TypeDatabase's same-shaped addTask (deferred: no injection seam for a red test).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements evolution proposal
draft-large-stack-executor-and-cross-version-parallelism(status In Progress → Implemented at landing, numbered then). Stacked on #121 (feature/self-contained-abi-layer); the base switches tonextonce that merges. Bumps to 0.19.0.Why
The library's
asyncis mostly signature-level: the real cost was on the print path. swift-demangling'sStackSafeExecutordecides per demangle / print / remangle whether to hop to its 8 MB pool by probing the calling thread's remaining stack; Swift Concurrency's cooperative threads carry 512 KB, so an async print loop paid one thread round trip + semaphore wait per printed symbol (8–21 µs, 1.14–2.28× the work). The synchronous indexing sweep already amortizes that withwithLargeStack; an async loop cannot be wrapped that way. Separately,diff/evolutionindexed their versions one after the other although each version is an independent file.What changed
0.6.3 ..< 0.7.0, skipping 0.6.1 whose QoS re-ranking slowed dump/interface 3–4×). 0.6.0 → 0.6.3 alone: timing flat, output identical.MachOSymbols.LargeStackTaskExecution.runsets swift-demangling's 16 MBLargeStackTaskExecutor(StackSafeExecutor.taskExecutor,@_spi(Internals)) as the task executor preference around every library entry point: indexerprepare(),SwiftInterfaceBuilder.prepare()/printRoot(),SwiftDiffableInterfaceBuilder.prepare(), the evolution builder'sprepare/ render entries, the diff renderer's two entries, the printer's four per-definition entries, the sixDumpable.dump(using:in:)conformers. Nesting is a no-op; unstructuredTask {}is not used anywhere in the library (checked); below macOS 15 / iOS 18 the body runs unchanged.isEnabled(seeded byMACHO_SWIFT_SECTION_LARGE_STACK_EXECUTOR=0) turns it off process-wide.AnySwiftEvolutionInterfaceBuilder.prepare(maximumConcurrentPreparations:)(default = processor count,1= the old serial order; pack façade follows),swift-section diff/evolution --jobs N, all through a newCollection.concurrentMap(maximumConcurrency:_:)inUtilities(windowed task group: source-ordered results, first failure rethrown, pending elements never started). Intra-version parallelism is out of scope (MachOKit'sMachOFilereads share oneFileHandle).Verification
LargeStackTaskExecutionTests(executor thread by stack size + name, no hop forexecute/executeAsyncinside, nested runs stay put, child tasks inherit, disabled stays on the caller's thread),BoundedConcurrentMapTests(order, window never exceeded, window 1 serial, rendezvous-proved concurrency, first-failure semantics),parallelPreparationMatchesSerialPreparation(interface, structured stream and evolution JSON byte-identical),--jobsparsing/validation for both commands. CI filter extended.swift test --skip IntegrationTests: 1637 tests / 305 suites passed.Scripts/run-rendering-ab-verification.py, baseline = Make MachOSwiftSection self-contained: descriptors expose addresses, symbol attribution moves to SwiftInspection #121's branch, candidate = this branch): 78 pairs byte-identical with the executor on, and again with it off (current-system cache, simulator runtimes iOS 15.5 / 18.5 / 18.6 / 26.5, in-process MachOImage).evolutionover three archived SwiftUI caches (15.5 / 26.5.2 / 27.0 beta 6):--interface306.7 s → 242.6 s (executor) → 151.9 s (executor + default parallelism); lineage 282.4 → 233.8 → 139.4 s.Docs (same batch)
Proposal decision log; implementation note
Documentations/Internal/LargeStackTaskExecutorAdoption.md; task report; AGENTS.md (MachOSymbols/SwiftIndexing/SwiftInterfaceentries + test-environment note);Modules/SwiftInterface.md; glossary (large-stack executor); evolution log;Changelogs/0.19.0.md; the 2026-07-31 review record's open item marked resolved; the staleexecuteWithUncheckedSendabilitycomments inNode+.swift.Review follow-ups (2026-09-04)
A parallel review session filed 15 findings (
Roadmaps/2026-09-04-pr122-review-findings.md): 5 defects, 1 false positive, 9 design/style. Landed in the follow-up commit:concurrentMapsubmits throughaddTaskUnlessCancelledand fails withCancellationErroronce the caller is cancelled (the reviewer reproduced the old behavior — every pending version still indexed to the end); the executor-thread tests also guardisEnabled, so a run underMACHO_SWIFT_SECTION_LARGE_STACK_EXECUTOR=0no longer goes red; that variable accepts0/false/no/off; the parallel-equivalence test prepares the parallel builder FIRST (cold caches); a three-way barrier pins that the window admits its full width. Per the user's decisions:SwiftIndexEvents.Dispatcherserializes handler invocation process-wide with a recursive lock (Handlerkeeps noSendablerequirement; pinned byEventDeliverySerializationTests), the lineage default window stays at the processor count, and stderr diagnostics carry an input label (ConsoleEventHandler(label:),AnySwiftEvolutionInterfaceBuilder.init(eventHandlersPerVersion:);difftagsold/new,evolutioneach version). Every behavioral fix was verified red under a mutated build. Adjudicated (ReviewAdjudications.mdA25–A33): theimport MachOFoundationfalse positive, the per-entry wrap, theconcurrentMapnaming,#isolationforwarding,.serializedvs the process switch, the executor-off window, the default window, the unavoidable dual availability gate, andTypeDatabase's same-shapedaddTask(deferred: no injection seam for a red test).