Shared PTW and other VM improvements

CHANGELOG.md

@@ -5,6 +5,32 @@ All notable changes to Vortex are documented here. The format is based on
 follows the version pins recorded in [VERSION](VERSION) (`VORTEX_VERSION`,
 `TOOLCHAIN_REV`, `GEM5_REV`).
 
+## [Unreleased]
+
+### Added
+
+- **Shared multi-walker page-table walker with page-walk caches; banked TLBs; RTL Sv39.** The per-MMU blocking Sv32 walker is replaced by one device-level generic Sv32/Sv39 walker (`hw/rtl/mem/VX_mmu_ptw.sv`) with `VX_CFG_PTW_NUM_WALKERS` concurrent walk slots, fetching PTEs on a dedicated L3 requestor port and skipping warm upper levels through two direct-mapped page-walk caches (`VX_CFG_PTW_WALK_CACHE_SIZE`). Per-core TLBs are banked (`VX_CFG_TLB_NUM_BANKS`) so a miss parked in one bank no longer blocks hits in the others, and superpage leaves are installed at their real level. Walk faults (invalid PTE, misaligned superpage, no leaf) are detected and reported on the fill. TLB misses travel on a new `VX_ptw_bus_if`/`VX_ptw_arb` hierarchy; SATP stays a per-core CSR, with the root PPN riding along on each walk request. The SimX timing model mirrors the same topology (`sim/simx/mem/ptw.{h,cpp}`) and surfaces the TLB/walker counters — including the new `VX_CSR_MPM_PWC*` ids — under the MEM MPM class. *Why:* closes the "RTL Sv39 + superpage fills" and "centralized multi-walker PTW with walk cache" roadmap items and makes rtlsim VM real: the old walker descended into the runtime's megapage identity leaves and returned garbage, which is why rtlsim was absent from the vm catalog.
+- **RTL command-processor VM (FPGA path).** The RTL CP now matches the software CP model: `DEV_CAPS` publishes `VM_ENABLED` (bit 24), `CP_SATP_LO/HI` (0x028/0x02C) hold the page-table root, and the DMA engine translates its device-side operand once per chunk through a new `VX_cp_mmu` walker reading PTEs over the DMA's device AXI channel (`F_MEM_PHYSICAL` skips translation; faults pass the address through untranslated). *Why:* VM previously worked on simx/rtlsim/gem5 and silently no-oped on xrt/opae because the runtime never discovered VM support from the RTL regfile.
+- **`tests/regression/vm_stress`** — TLB-pressure regression (strided page touches per task + a `VX_MEM_PHYS` identity-mapped buffer); wired into `ci/testcases/vm.yaml`, which now runs simx **and rtlsim** at both XLENs plus full-tier single-bank and multi-cluster L2/L3 variants.
+
+### Fixed
+
+- **SimX cache flush walk raced a same-tick replay.** `processFlush()` gated on `TFifo::empty()`, which hides entries inside the pipe's latency window; the end-of-kernel walk could sweep a set before a store replayed from the MSHR that tick dirtied its line, and the write was lost. The guard now uses `size()`.
+- **SimX arbiter input grouping.** `TxArbiter`/`TxRxArbiter` grouped inputs by `log2ceil(inputs / outputs)` where `VX_stream_arb` uses CDIV, so a request count that is not a multiple of the output count left the trailing inputs unserved.
+- **SimX VM timing parity.** With translation on, SimX ran 16-36% fewer cycles than rtlsim on hit-heavy kernels: the model translated a TLB hit in the same tick with unlimited throughput while the RTL crosses a per-lane elastic buffer, the lane/bank crossbar, a single-ported bank CAM, and the gather crossbar on every access. The `Mmu` model now admits one lookup per bank per cycle (round-robin ports) and charges the lookup-pipeline latency; `ci/testcases/vm.yaml` gains `model_parity-sgemm` / `model_parity-diverge` (RV32, sgemm +0.2%, diverge +3.8%).
+- **SimX lost stores around a page walk.** A store parked on a TLB miss lived outside every `SimChannel`, so the processor could declare the kernel done and flush the dcache before the walk replayed it (`vm_stress` on simx/XLEN=32 lost one store). The parked packet now stays in its `ReqIn` port until replayed, which also keeps later lane packets from overtaking it.
+- **CP DMA walker (`VX_cp_mmu`).** The Sv32 root PPN slice folded the SATP MODE/ASID bits into the table address; the 2-entry translation cache was consulted in BARE mode and never invalidated (a CACHE_FLUSH DCR or a new SATP now drops it, and a walk that straddles an invalidation is not cached); superpage leaves were not alignment-checked. The walker now imports the `VX_gpu_pkg` VM geometry and PTE helpers instead of re-deriving them.
+- **Stale installs from a walk in flight across a flush** (`VX_mmu_tlb_bank`, `VX_mmu_ptw`, SimX `Mmu`/`Ptw`): a TLB fill or page-walk-cache fill resolved against the pre-flush page table could land in the freshly flushed structures; such walks are now dropped and re-issued (TLB) or kept out of the walk caches (PTW).
+- **SimX shared PTW sizing.** Clients and the flush loop were sized by `VX_CFG_NUM_CORES`, but a cluster instantiates `NUM_SOCKETS * SOCKET_SIZE` cores; non-divisible shapes left the trailing cores' walker ports unbound.
+- **VM runtime / SimX hygiene.** Identity maps installed inside the pinned slab no longer re-reserve VA space (spurious "address range overlaps" errors); TLB/PTW perf counters reset per launch like the caches; SimX enforces the RTL's power-of-two `VX_CFG_TLB_NUM_BANKS` constraint; `VX_mmu` sizes its bypass-path address by `VX_CFG_MEM_ADDR_WIDTH` (XLEN=64 AFU-sim builds failed on WIDTHTRUNC); the `cp_axil_regfile`/`cp_dma` block testbenches carry the new `satp`/flush ports.
+
+- **`VX_MEM_PHYS` after ordinary VM allocations.** The VA allocator handed out addresses from the same base the pinned identity-mapped slab occupies, so a `VX_MEM_PHYS` buffer created after any ordinary allocation collided in the page table; the slab's range is now reserved out of the VA space at VM bring-up.
+
+- **VM RTL build under `PERF_ENABLE`.** A stray `.` in the `VX_mmu_tlb` instantiation ([hw/rtl/mem/VX_mmu.sv](hw/rtl/mem/VX_mmu.sv)) broke every `-DVX_CFG_VM_ENABLE` RTL build with profiling on.
+- **VM RTL `satp` width.** `VX_mmu` / `VX_mmu_ptw` took a 32-bit `satp` and gated translation on bit 31, silently truncating the CSR and misreading the Sv39 mode field on XLEN=64; the port is now `VX_CFG_XLEN` wide and the mode check follows the Sv32 / Sv39 encodings.
+- **`VX_CFG_TLB_SIZE` honoured.** `VX_mmu_tlb` hardcoded 5 index bits, so any size other than 32 mis-indexed the CAM; the index width is now derived from the config (power of two enforced).
+- **Dead BARE-mode VM test cases.** `ci/testcases/vm.yaml` `isa-6..10` and `tests/regression/basic/start.S` referenced the retired `VX_CFG_VM_ADDR_MODE` (and `VX_CFG_PAGE_TABLE_BASE_ADDR`, `VX_CFG_MEM_PAGE_LOG2_SIZE`) names. `VX_VM_ADDR_MODE` is a resolved `VX_types` contract fixed at configure time and cannot be overridden per case via `CONFIGS`, so the BARE cases only re-ran the default mode; they are removed and `start.S` uses the live `VX_VM_*` / `VX_MEM_*` names. *Why:* a catalog entry that claims coverage it does not provide is worse than none.
+
 ## [3.0] — 2026-06-08
 
 The 3.0 release introduces a fixed-function graphics stack (rasterizer, texture units, and output mergers), tensor core structured sparsity (2:4), warpgroup-level matrix multiplication (WGMMA), global-to-local data transfer acceleration (DXA), a new hardware kernel scheduler (KMU) and Command Processor (CP) architecture, a new asynchronous runtime API (`vortex2.h`), asynchronous barriers with arrive/wait/event semantics, compressed instruction set (RVC) support, hardware atomics, an MMU/SV32 virtual memory stack, a Mesa/lavapipe Vulkan backend (`vortexpipe`), HIP via chipStar, gem5 integration, a SimX v3 TLM architecture with fixed-size handshake channels, productized Synopsys and Yosys ASIC synthesis flows, and a refreshed toolchain (LLVM 20, POCL 7.0). Build and configuration infrastructure was reworked: TOML-driven HW configuration ([VX_config.toml](VX_config.toml) + [VX_types.toml](VX_types.toml)) decoupling SimX/runtime from the RTL source tree, a `VX_CFG_` macro namespace that resolves toolchain preprocessor collisions, retirement of the global `toolchain_env.sh` to enable parallel multi-version Vortex worktrees on the same shell, consolidation of `kernel/`/`runtime/` under a shared `sw/` root, a single-source [VERSION](VERSION) file driving CI toolchain pinning, Perfetto trace export ([ci/perfetto.py](ci/perfetto.py)), and new top-level [AGENTS.md](AGENTS.md) + [CONTRIBUTING.md](CONTRIBUTING.md) for AI-agent and contributor workflows.

VX_config.toml

@@ -136,6 +136,12 @@ VX_CFG_NUM_VPU_BLOCKS = 1
 [vm]
 # VM page-table format is a HW<->SW contract, moved to VX_types.toml [vm]; TLB depth stays here.
 VX_CFG_TLB_SIZE = 32
+# dTLB lookup banks (power of two dividing TLB_SIZE); the iTLB is always single-banked.
+VX_CFG_TLB_NUM_BANKS = 4
+# Concurrent walks in the device-level shared page-table walker.
+VX_CFG_PTW_NUM_WALKERS = 8
+# Page-walk cache entries per non-leaf level (direct-mapped, power of two).
+VX_CFG_PTW_WALK_CACHE_SIZE = 64
 
 VX_CFG_VM_PINNED_REGION_SIZE = 0x10000000   # 256 MB
 

VX_types.toml

@@ -734,7 +734,9 @@ VX_CSR_MPM_LMEM_BANK_ST = 0xB09  # bank conflicts
 VX_CSR_MPM_LMEM_BANK_ST_H = 0xB89
 VX_CSR_MPM_COALESCER_MISS = 0xB0A  # coalescer misses
 VX_CSR_MPM_COALESCER_MISS_H = 0xB8A
-# VM/MMU (per-core TLB/PTW). Hardware sums icache + dcache MMU counters.
+# VM/MMU. TLB counters are per core (icache + dcache MMU summed); the PTW and
+# page-walk-cache counters belong to the device-level shared walker and read
+# the same value on every core.
 VX_CSR_MPM_TLB_READS    = 0xB0B # total TLB lookups
 VX_CSR_MPM_TLB_READS_H  = 0xB8B
 VX_CSR_MPM_TLB_HITS     = 0xB0C # TLB hits
@@ -743,10 +745,18 @@ VX_CSR_MPM_TLB_MISSES   = 0xB0D # TLB misses (triggered PTW)
 VX_CSR_MPM_TLB_MISSES_H = 0xB8D
 VX_CSR_MPM_TLB_EVICTS   = 0xB0E # TLB evictions on fill
 VX_CSR_MPM_TLB_EVICTS_H = 0xB8E
-VX_CSR_MPM_PTW_WALKS    = 0xB0F # PTW walks completed
+VX_CSR_MPM_PTW_WALKS    = 0xB0F # PTW walks started
 VX_CSR_MPM_PTW_WALKS_H  = 0xB8F
-VX_CSR_MPM_PTW_LATENCY  = 0xB10 # PTW total latency cycles
+VX_CSR_MPM_PTW_LATENCY  = 0xB10 # sum of per-walk latencies (cycles)
 VX_CSR_MPM_PTW_LATENCY_H = 0xB90
+VX_CSR_MPM_PWC1_HITS    = 0xB11 # walks that skipped the top level via the walk cache
+VX_CSR_MPM_PWC1_HITS_H  = 0xB91
+VX_CSR_MPM_PWC1_MISSES  = 0xB12
+VX_CSR_MPM_PWC1_MISSES_H = 0xB92
+VX_CSR_MPM_PWC2_HITS    = 0xB13 # Sv39 only: walks that also skipped the middle level
+VX_CSR_MPM_PWC2_HITS_H  = 0xB93
+VX_CSR_MPM_PWC2_MISSES  = 0xB14
+VX_CSR_MPM_PWC2_MISSES_H = 0xB94
 
 [csr_mpm_dxa]
 # PERF: DXA copy engine (cluster-level, same value on all cores in cluster)

ci/testcases/vm.yaml

@@ -1,4 +1,6 @@
-# Test cases for `vm`, drafted from regression.sh.in (review/collapse).
+# Test cases for `vm`.
+# VX_VM_ADDR_MODE is a resolved VX_types contract fixed at configure time, so
+# BARE cannot be selected per case via CONFIGS; only the default mode is tested.
 category: vm
 defaults:
   xlen:
@@ -11,24 +13,28 @@ tests:
   via: make-run
   drivers:
   - simx
+  - rtlsim
   dir: tests/regression/sgemm
   target: run-{driver}
 - id: isa-2
   via: make-run
   drivers:
   - simx
+  - rtlsim
   dir: tests/regression/diverge
   target: run-{driver}
 - id: isa-3
   via: make-run
   drivers:
   - simx
+  - rtlsim
   dir: tests/regression/dogfood
   target: run-{driver}
 - id: isa-4
   via: make-run
   drivers:
   - simx
+  - rtlsim
   dir: tests/regression/raycast
   target: run-{driver}
 - id: isa-5
@@ -37,38 +43,47 @@ tests:
   - simx
   dir: tests/graphics/gfx_draw3d
   target: run-{driver}
-- id: isa-6
+- id: stress
   via: make-run
   drivers:
   - simx
-  dir: tests/regression/sgemm
+  - rtlsim
+  dir: tests/regression/vm_stress
   target: run-{driver}
-  configs+: -DVX_CFG_VM_ADDR_MODE=BARE
-- id: isa-7
-  via: make-run
+- id: xrt-cp
   drivers:
-  - simx
-  dir: tests/regression/diverge
-  target: run-{driver}
-  configs+: -DVX_CFG_VM_ADDR_MODE=BARE
-- id: isa-8
+  - xrt
+  app: sgemm
+  tier: full
+- id: tlb-banks1
   via: make-run
   drivers:
-  - simx
-  dir: tests/regression/dogfood
-  target: run-{driver}
-  configs+: -DVX_CFG_VM_ADDR_MODE=BARE
-- id: isa-9
-  via: make-run
-  drivers:
-  - simx
-  dir: tests/regression/raycast
+  - rtlsim
+  dir: tests/regression/sgemm
   target: run-{driver}
-  configs+: -DVX_CFG_VM_ADDR_MODE=BARE
-- id: isa-10
-  via: make-run
+  configs+: -DVX_CFG_TLB_NUM_BANKS=1
+  tier: full
+- id: multi
   drivers:
-  - simx
-  dir: tests/graphics/gfx_draw3d
-  target: run-{driver}
-  configs+: -DVX_CFG_VM_ADDR_MODE=BARE
+  - rtlsim
+  app: sgemm
+  shape: {cores: 2, clusters: 2, l2cache: true, l3cache: true}
+  tier: full
+
+# SimX<->rtlsim cycle parity with translation on (see model_parity.yaml for
+# the contract; check gates are RV32-pinned like the rest of the parity
+# suite). The SimX MMU charges the banked-TLB lookup pipeline
+# (Mmu::TRANSLATE_LATENCY + one lookup per bank per cycle); these cases hold
+# that model to the RTL. Both are steady-state sized; sgemm's cold TLB misses
+# also cover the walk path (vm_stress itself is too small to gate — all boot
+# skew).
+- id: model_parity-sgemm
+  check: model_parity
+  via: blackbox
+  app: sgemm
+  tier: full
+- id: model_parity-diverge
+  check: model_parity
+  via: blackbox
+  app: diverge
+  tier: full

docs/designs/command_processor.md

@@ -194,11 +194,10 @@ The doorbell commits atomically on the `Q_TAIL_HI` write
 ([`:333-336`](../../hw/rtl/cp/VX_cp_axil_regfile.sv#L333)); undecoded
 addresses return DECERR.
 
-> **FPGA/sim divergence:** the RTL regfile has **no** `CP_SATP_LO/HI`
-> registers. The Emulation CP *does* (`0x028/0x02C`,
-> [`cmd_processor.cpp:75-76`](../../sim/common/cmd_processor.cpp#L75)) and
-> the runtime always writes them
-> ([`device.cpp:303-304`](../../sw/runtime/common/device.cpp#L303)). See §8.
+The RTL regfile carries `CP_SATP_LO/HI` at `0x028/0x02C` and publishes
+`VM_ENABLED` in `DEV_CAPS` bit 24, matching the Emulation CP
+(`cmd_processor.cpp`); the runtime writes SATP on both paths
+([`device.cpp`](../../sw/runtime/common/device.cpp)).
 
 ---
 
@@ -380,11 +379,11 @@ the intent is not lost.
    ([`cmd_processor.cpp:452-462`](../../sim/common/cmd_processor.cpp#L452)).
    Non-cache-line-aligned transfers can over-write on FPGA. Needs tail
    `wstrb` on the last beat (review item C-2/P-W4).
-2. **VM in RTL.** Add `CP_SATP_LO/HI` regfile decode + a hardware
-   page-table walker + TLB in `VX_cp_dma`, and route `F_MEM_PHYSICAL`, so
-   FPGA matches the simulator's MMU-aware DMA (cp_pure_v2 VM Phase 2;
-   review items P-W1/P-W3 and the SATP gap). Today VM works on
-   simx/rtlsim/gem5 and silently no-ops on FPGA.
+2. **VM in RTL — done.** `CP_SATP_LO/HI` decode, the `DEV_CAPS`
+   `VM_ENABLED` bit, `F_MEM_PHYSICAL` routing, and a per-chunk
+   `VX_cp_mmu` walker in `VX_cp_dma` bring the FPGA path up to the
+   simulator's MMU-aware DMA (remaining: a per-transfer TLB beyond the
+   walker's 2-entry translation cache).
 3. **Real `CMD_FENCE` semantics.** The engine retires FENCE as a NOP
    ([`VX_cp_engine.sv:109-112`](../../hw/rtl/cp/VX_cp_engine.sv#L109));
    it should honor `FENCE_DMA_BIT` / `FENCE_GPU_BIT` ordering (C-7).