sdk: reserve an 8 MiB main-thread shadow stack for linked programs

docs/future-improvements.md

@@ -229,13 +229,18 @@ the protocol layer.
 
 ## User-space programs
 
-### Replace per-program `-Wl,-z,stack-size` workarounds with a real shadow-stack overflow guard
-`wasm-ld` reserves a default 64 KiB shadow stack (the linear-memory region the
-compiler uses for spilled locals, `alloca`, and address-taken locals). The
-shadow stack grows **downward** from `__stack_high`, and `wasm-ld` places it
+### Add a real shadow-stack overflow guard beyond the SDK's 8 MiB floor
+Upstream `wasm-ld` reserves a default 64 KiB shadow stack (the linear-memory
+region the compiler uses for spilled locals, `alloca`, and address-taken
+locals). Kandelo's SDK raises executable links to an 8 MiB floor while
+preserving larger explicit requests. That floor covers the mainstream
+workloads that exposed the 64 KiB default, but it is a capacity policy rather
+than an overflow guard.
+
+The shadow stack grows **downward** from `__stack_high`, and `wasm-ld` places it
 *immediately below* the `.data` / `.bss` segments in the same linear memory.
 There is no guard page, no stack-pointer bounds check, and no trap: a function
-that consumes more than the remaining shadow-stack budget silently writes
+that consumes more than the effective shadow-stack budget silently writes
 through `__stack_pointer` into whatever data segment happens to be just below
 it, corrupting unrelated globals.
 
@@ -245,12 +250,11 @@ shadow-stack frame underflowed by ~108 KiB into PHP's `alloc_globals` data
 segment, silently corrupting `AG(mm_heap)`. The next `_efree` call dereferenced
 the now-bogus heap pointer and trapped — surfacing as "memory access out of
 bounds" inside the optimizer, with no indication that the actual cause was
-stack overflow ~thousands of frames earlier. The PR's workaround is
-`LDFLAGS=-Wl,-z,stack-size=4194304` (4 MiB) in `packages/registry/php/build-php.sh`.
-That sidesteps the underflow for PHP's observed workload but doesn't *prevent*
-the failure mode — a deeper recursion or a larger `alloca` will silently
-corrupt data again, and every other large port we ship today (vim, nginx,
-mariadb, etc.) has the same latent bug.
+stack overflow ~thousands of frames earlier. The PHP recipe still requests
+`LDFLAGS=-Wl,-z,stack-size=4194304` (4 MiB), which the SDK raises to its 8 MiB
+floor. The larger reserve covers PHP's observed workload but doesn't *prevent*
+the failure mode: a deeper recursion or a larger `alloca` can still silently
+corrupt data, and every linked program has the same undetected-overflow risk.
 
 A real fix needs runtime detection so the failure surfaces as an obvious
 crash, not silent corruption. Possible approaches:
@@ -280,10 +284,11 @@ crash, not silent corruption. Possible approaches:
 Once a real guard is in place, the per-program `-Wl,-z,stack-size=...`
 overrides should be audited: programs that genuinely need a larger shadow
 stack (PHP optimizer, deep parser stacks) keep the explicit override and
-document why; everything else can drop the flag and rely on the default
-+ guard.
+document why; everything else can drop the package-local flag and rely on the
+SDK floor plus the guard.
 
-**Files:** `packages/registry/php/build-php.sh` (current 4 MiB workaround),
+**Files:** `sdk/src/lib/flags.ts` and `sdk/kandelo/bin/wasm32posix-cc` (current
+8 MiB floor), `packages/registry/php/build-php.sh` (current 4 MiB request),
 `libc/glue/channel_syscall.c` (likely site for a syscall-entry bounds check),
 `host/src/worker-main.ts` (instantiation-time wiring for stack bounds),
 plus any other `build-*.sh` that hits the same wall in the meantime.

docs/sdk-guide.md

@@ -127,6 +127,11 @@ wasm32posix-ar rcs libfoo.a lib_a.o lib_b.o
 wasm32posix-cc main.c -L. -lfoo -o program.wasm
 ```
 
+The compiler wrapper preserves the caller's linker-input order. Objects,
+explicit archives, `-l` libraries, and linker group controls reach Clang in
+the same relative sequence supplied by the build system, as required by
+normal static archive resolution.
+
 ### With dynamic loading (dlopen)
 
 ```bash
@@ -166,6 +171,7 @@ leave unresolved host imports in linked programs.
 -Wl,--import-memory                # Memory provided by host
 -Wl,--shared-memory                # Enable SharedArrayBuffer
 -Wl,--max-memory=1073741824        # 1GB max memory
+-Wl,-z,stack-size=8388608          # 8 MiB main-thread shadow stack (see below)
 -Wl,--global-base=1114112          # Data segment start
 -Wl,--no-stack-first               # LLVM 22+: preserve stack-after-data layout
 -Wl,--allow-undefined              # Host imports are resolved at load time
@@ -175,6 +181,95 @@ leave unresolved host imports in linked programs.
 -Wl,--export=__wasm_init_tls       # TLS initialization
 ```
 
+#### Why an 8 MiB main-thread stack
+
+wasm-ld's default shadow stack is only ~64 KiB. WebAssembly has **no stack
+guard page**, so a program that overflows the shadow stack does not fault at the
+overflow — `__stack_pointer` simply keeps decrementing past `__data_end` and
+silently overwrites whatever lives at the top of `.bss`. On this platform that
+region holds the pthread/TLS globals (`__wasm_tp_storage`, the main thread's
+`struct pthread`, and `__pthread_tsd_main`), so an overflow corrupts thread-local
+storage and later surfaces as a **spurious "memory access out of bounds"** in an
+unrelated function (e.g. `__pthread_getspecific` or `pthread_mutex_lock`) — far
+from the actual overflow. Deep call chains in ordinary libraries hit this
+easily; GTK's `gdk_pixbuf_new_from_file` → GObject type-registration → glib
+chain is one confirmed case (see `docs/kandelo-lxde-desktop-demo.md`).
+
+POSIX does **not** mandate a default stack size — it is implementation-defined,
+with `RLIMIT_STACK` governing the main thread. We reserve **8 MiB** because that
+is the de-facto default soft `RLIMIT_STACK` on Linux/glibc (and macOS), so the
+large body of C software written and tested on Linux already assumes it fits
+within those bounds. Because a WebAssembly shadow stack cannot grow at runtime,
+the reservation is fixed at link time.
+
+Scope and cost:
+- **Main thread only.** This flag sizes the process's initial shadow stack.
+  Threads created via `pthread_create` get their own stacks from musl's
+  `__default_stacksize` (128 KiB by default), independent of this flag — an
+  8 MiB main stack does **not** multiply per thread.
+- **At least ~8 MiB of initial linear memory per process.** A larger stack raises
+  `__heap_base` 1:1, so each process instance reserves the extra space up front.
+  This is per-process; a `fork()`ed child is a separate address space with the
+  same configured main stack size.
+- The engine's native Wasm call stack (operand stack / call frames) is separate
+  and host/engine-managed; it is not part of this linear-memory reservation.
+
+The SDK treats 8 MiB as a floor. It appends the larger of that floor and every
+valid user stack request after the other linker arguments, where lld gives it
+final precedence. The Node-hosted driver first asks pinned Clang for a `-###`
+job trace without injecting executable glue. Compiler-only traces, including
+`-fsyntax-only`, dependency generation, and analyzer jobs hidden in response
+files, continue without link preparation. Confirmed executable links get a
+second `-###` trace with the complete SDK link inputs, and the SDK scans the
+exact `wasm-ld` argument vector Clang emits. This leaves option classification
+and ordering in Clang: positional inputs, `-Wl,`, `-Xlinker`, and direct `-z`
+transports cannot be misclassified by a duplicate SDK option table. A failed,
+missing, or ambiguous linker trace aborts before the real link. These traces are
+driver-only invocations and do not compile; they add two Clang processes per
+executable link and one for an otherwise-unrecognized compiler-only mode.
+
+The Kandelo-native driver also uses a non-compiling Clang job trace to preserve
+compiler-only modes before entering its manual link path. For a confirmed link,
+it invokes the adjacent `wasm-ld` directly and scans the exact linker arguments
+it has already classified.
+
+Relative lld response paths follow Clang's effective `-working-directory`,
+including when that option comes from a top-level Clang response file,
+configuration file, or environment override. The SDK reads the driver-owned cwd
+slot in the pinned Clang trace and fails before linking if the trace omits or
+contradicts that invariant. Debug, coverage, and file compilation-directory
+options do not affect linker response resolution.
+
+LLVM 21 accepts the lld forms `-z stack-size=<bytes>` and
+`-zstack-size=<bytes>`. Stack sizes follow LLVM 21's radix-0 integer syntax:
+decimal, leading-zero or `0o` octal, `0x`/`0X` hexadecimal, and `0b`/`0B`
+binary.
+
+Clang expands its response files when producing the trace. The SDK expands any
+remaining lld response files, including nested `@file` references, using LLVM's
+POSIX response-file tokenization rules. Expansion uses an iterative work stack,
+not a nesting cutoff: both drivers inspect chains accepted by LLVM and reject
+missing or recursive response files before the real link. To bound hostile or
+accidental input, they also reject an expansion after 4,096 file expansions,
+1,048,576 examined tokens, or 64 MiB of decoded response text. These are
+explicit resource limits; exhausting one never falls back to the 8 MiB floor.
+This keeps quoted or escaped paths distinct from linker options and prevents
+object filenames containing `stack-size=` from changing the stack. Bare
+`stack-size=<bytes>` tokens, `-z=stack-size=<bytes>`, and options after `--` are
+not stack requests. Response-file contents are never rewritten.
+The Node-hosted SDK accepts LLVM's UTF-8, UTF-16LE-BOM, and UTF-16BE-BOM
+response encodings. The Kandelo-native Bash driver accepts UTF-8 response files
+and rejects either UTF-16 BOM before compiling or linking: Bash variables cannot
+retain UTF-16's embedded NUL bytes, and the packaged SDK does not declare a
+transcoding tool. Generate UTF-8 response files when building inside Kandelo.
+Invalid spellings stay visible to LLVM so it can reject them; a valid stack
+larger than the SDK's fixed 1 GiB executable-memory maximum fails in the driver
+instead of being silently replaced by the floor. Smaller legacy requests
+therefore still receive 8 MiB, while programs such as SpiderMonkey that
+explicitly need 16 MiB retain that larger reservation. Changing the platform
+floor or scanner requires updating both `sdk/kandelo/bin/wasm32posix-cc` and
+`sdk/src/lib/flags.ts`.
+
 ### Files linked automatically
 
 When linking an executable (not compile-only), the SDK adds: