diff --git a/CHANGELOG.md b/CHANGELOG.md index e909ada512..c832ef2a1c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,32 @@ All notable changes to Vortex are documented here. The format is based on follows the version pins recorded in [VERSION](VERSION) (`VORTEX_VERSION`, `TOOLCHAIN_REV`, `GEM5_REV`). +## [Unreleased] + +### Added + +- **Shared multi-walker page-table walker with page-walk caches; banked TLBs; RTL Sv39.** The per-MMU blocking Sv32 walker is replaced by one device-level generic Sv32/Sv39 walker (`hw/rtl/mem/VX_mmu_ptw.sv`) with `VX_CFG_PTW_NUM_WALKERS` concurrent walk slots, fetching PTEs on a dedicated L3 requestor port and skipping warm upper levels through two direct-mapped page-walk caches (`VX_CFG_PTW_WALK_CACHE_SIZE`). Per-core TLBs are banked (`VX_CFG_TLB_NUM_BANKS`) so a miss parked in one bank no longer blocks hits in the others, and superpage leaves are installed at their real level. Walk faults (invalid PTE, misaligned superpage, no leaf) are detected and reported on the fill. TLB misses travel on a new `VX_ptw_bus_if`/`VX_ptw_arb` hierarchy; SATP stays a per-core CSR, with the root PPN riding along on each walk request. The SimX timing model mirrors the same topology (`sim/simx/mem/ptw.{h,cpp}`) and surfaces the TLB/walker counters — including the new `VX_CSR_MPM_PWC*` ids — under the MEM MPM class. *Why:* closes the "RTL Sv39 + superpage fills" and "centralized multi-walker PTW with walk cache" roadmap items and makes rtlsim VM real: the old walker descended into the runtime's megapage identity leaves and returned garbage, which is why rtlsim was absent from the vm catalog. +- **RTL command-processor VM (FPGA path).** The RTL CP now matches the software CP model: `DEV_CAPS` publishes `VM_ENABLED` (bit 24), `CP_SATP_LO/HI` (0x028/0x02C) hold the page-table root, and the DMA engine translates its device-side operand once per chunk through a new `VX_cp_mmu` walker reading PTEs over the DMA's device AXI channel (`F_MEM_PHYSICAL` skips translation; faults pass the address through untranslated). *Why:* VM previously worked on simx/rtlsim/gem5 and silently no-oped on xrt/opae because the runtime never discovered VM support from the RTL regfile. +- **`tests/regression/vm_stress`** — TLB-pressure regression (strided page touches per task + a `VX_MEM_PHYS` identity-mapped buffer); wired into `ci/testcases/vm.yaml`, which now runs simx **and rtlsim** at both XLENs plus full-tier single-bank and multi-cluster L2/L3 variants. + +### Fixed + +- **SimX cache flush walk raced a same-tick replay.** `processFlush()` gated on `TFifo::empty()`, which hides entries inside the pipe's latency window; the end-of-kernel walk could sweep a set before a store replayed from the MSHR that tick dirtied its line, and the write was lost. The guard now uses `size()`. +- **SimX arbiter input grouping.** `TxArbiter`/`TxRxArbiter` grouped inputs by `log2ceil(inputs / outputs)` where `VX_stream_arb` uses CDIV, so a request count that is not a multiple of the output count left the trailing inputs unserved. +- **SimX VM timing parity.** With translation on, SimX ran 16-36% fewer cycles than rtlsim on hit-heavy kernels: the model translated a TLB hit in the same tick with unlimited throughput while the RTL crosses a per-lane elastic buffer, the lane/bank crossbar, a single-ported bank CAM, and the gather crossbar on every access. The `Mmu` model now admits one lookup per bank per cycle (round-robin ports) and charges the lookup-pipeline latency; `ci/testcases/vm.yaml` gains `model_parity-sgemm` / `model_parity-diverge` (RV32, sgemm +0.2%, diverge +3.8%). +- **SimX lost stores around a page walk.** A store parked on a TLB miss lived outside every `SimChannel`, so the processor could declare the kernel done and flush the dcache before the walk replayed it (`vm_stress` on simx/XLEN=32 lost one store). The parked packet now stays in its `ReqIn` port until replayed, which also keeps later lane packets from overtaking it. +- **CP DMA walker (`VX_cp_mmu`).** The Sv32 root PPN slice folded the SATP MODE/ASID bits into the table address; the 2-entry translation cache was consulted in BARE mode and never invalidated (a CACHE_FLUSH DCR or a new SATP now drops it, and a walk that straddles an invalidation is not cached); superpage leaves were not alignment-checked. The walker now imports the `VX_gpu_pkg` VM geometry and PTE helpers instead of re-deriving them. +- **Stale installs from a walk in flight across a flush** (`VX_mmu_tlb_bank`, `VX_mmu_ptw`, SimX `Mmu`/`Ptw`): a TLB fill or page-walk-cache fill resolved against the pre-flush page table could land in the freshly flushed structures; such walks are now dropped and re-issued (TLB) or kept out of the walk caches (PTW). +- **SimX shared PTW sizing.** Clients and the flush loop were sized by `VX_CFG_NUM_CORES`, but a cluster instantiates `NUM_SOCKETS * SOCKET_SIZE` cores; non-divisible shapes left the trailing cores' walker ports unbound. +- **VM runtime / SimX hygiene.** Identity maps installed inside the pinned slab no longer re-reserve VA space (spurious "address range overlaps" errors); TLB/PTW perf counters reset per launch like the caches; SimX enforces the RTL's power-of-two `VX_CFG_TLB_NUM_BANKS` constraint; `VX_mmu` sizes its bypass-path address by `VX_CFG_MEM_ADDR_WIDTH` (XLEN=64 AFU-sim builds failed on WIDTHTRUNC); the `cp_axil_regfile`/`cp_dma` block testbenches carry the new `satp`/flush ports. + +- **`VX_MEM_PHYS` after ordinary VM allocations.** The VA allocator handed out addresses from the same base the pinned identity-mapped slab occupies, so a `VX_MEM_PHYS` buffer created after any ordinary allocation collided in the page table; the slab's range is now reserved out of the VA space at VM bring-up. + +- **VM RTL build under `PERF_ENABLE`.** A stray `.` in the `VX_mmu_tlb` instantiation ([hw/rtl/mem/VX_mmu.sv](hw/rtl/mem/VX_mmu.sv)) broke every `-DVX_CFG_VM_ENABLE` RTL build with profiling on. +- **VM RTL `satp` width.** `VX_mmu` / `VX_mmu_ptw` took a 32-bit `satp` and gated translation on bit 31, silently truncating the CSR and misreading the Sv39 mode field on XLEN=64; the port is now `VX_CFG_XLEN` wide and the mode check follows the Sv32 / Sv39 encodings. +- **`VX_CFG_TLB_SIZE` honoured.** `VX_mmu_tlb` hardcoded 5 index bits, so any size other than 32 mis-indexed the CAM; the index width is now derived from the config (power of two enforced). +- **Dead BARE-mode VM test cases.** `ci/testcases/vm.yaml` `isa-6..10` and `tests/regression/basic/start.S` referenced the retired `VX_CFG_VM_ADDR_MODE` (and `VX_CFG_PAGE_TABLE_BASE_ADDR`, `VX_CFG_MEM_PAGE_LOG2_SIZE`) names. `VX_VM_ADDR_MODE` is a resolved `VX_types` contract fixed at configure time and cannot be overridden per case via `CONFIGS`, so the BARE cases only re-ran the default mode; they are removed and `start.S` uses the live `VX_VM_*` / `VX_MEM_*` names. *Why:* a catalog entry that claims coverage it does not provide is worse than none. + ## [3.0] — 2026-06-08 The 3.0 release introduces a fixed-function graphics stack (rasterizer, texture units, and output mergers), tensor core structured sparsity (2:4), warpgroup-level matrix multiplication (WGMMA), global-to-local data transfer acceleration (DXA), a new hardware kernel scheduler (KMU) and Command Processor (CP) architecture, a new asynchronous runtime API (`vortex2.h`), asynchronous barriers with arrive/wait/event semantics, compressed instruction set (RVC) support, hardware atomics, an MMU/SV32 virtual memory stack, a Mesa/lavapipe Vulkan backend (`vortexpipe`), HIP via chipStar, gem5 integration, a SimX v3 TLM architecture with fixed-size handshake channels, productized Synopsys and Yosys ASIC synthesis flows, and a refreshed toolchain (LLVM 20, POCL 7.0). Build and configuration infrastructure was reworked: TOML-driven HW configuration ([VX_config.toml](VX_config.toml) + [VX_types.toml](VX_types.toml)) decoupling SimX/runtime from the RTL source tree, a `VX_CFG_` macro namespace that resolves toolchain preprocessor collisions, retirement of the global `toolchain_env.sh` to enable parallel multi-version Vortex worktrees on the same shell, consolidation of `kernel/`/`runtime/` under a shared `sw/` root, a single-source [VERSION](VERSION) file driving CI toolchain pinning, Perfetto trace export ([ci/perfetto.py](ci/perfetto.py)), and new top-level [AGENTS.md](AGENTS.md) + [CONTRIBUTING.md](CONTRIBUTING.md) for AI-agent and contributor workflows. diff --git a/VX_config.toml b/VX_config.toml index b663baa627..5633dc1f0d 100644 --- a/VX_config.toml +++ b/VX_config.toml @@ -136,6 +136,12 @@ VX_CFG_NUM_VPU_BLOCKS = 1 [vm] # VM page-table format is a HW<->SW contract, moved to VX_types.toml [vm]; TLB depth stays here. VX_CFG_TLB_SIZE = 32 +# dTLB lookup banks (power of two dividing TLB_SIZE); the iTLB is always single-banked. +VX_CFG_TLB_NUM_BANKS = 4 +# Concurrent walks in the device-level shared page-table walker. +VX_CFG_PTW_NUM_WALKERS = 8 +# Page-walk cache entries per non-leaf level (direct-mapped, power of two). +VX_CFG_PTW_WALK_CACHE_SIZE = 64 VX_CFG_VM_PINNED_REGION_SIZE = 0x10000000 # 256 MB diff --git a/VX_types.toml b/VX_types.toml index bd2eb21ff7..7a6e91fa6f 100644 --- a/VX_types.toml +++ b/VX_types.toml @@ -734,7 +734,9 @@ VX_CSR_MPM_LMEM_BANK_ST = 0xB09 # bank conflicts VX_CSR_MPM_LMEM_BANK_ST_H = 0xB89 VX_CSR_MPM_COALESCER_MISS = 0xB0A # coalescer misses VX_CSR_MPM_COALESCER_MISS_H = 0xB8A -# VM/MMU (per-core TLB/PTW). Hardware sums icache + dcache MMU counters. +# VM/MMU. TLB counters are per core (icache + dcache MMU summed); the PTW and +# page-walk-cache counters belong to the device-level shared walker and read +# the same value on every core. VX_CSR_MPM_TLB_READS = 0xB0B # total TLB lookups VX_CSR_MPM_TLB_READS_H = 0xB8B VX_CSR_MPM_TLB_HITS = 0xB0C # TLB hits @@ -743,10 +745,18 @@ VX_CSR_MPM_TLB_MISSES = 0xB0D # TLB misses (triggered PTW) VX_CSR_MPM_TLB_MISSES_H = 0xB8D VX_CSR_MPM_TLB_EVICTS = 0xB0E # TLB evictions on fill VX_CSR_MPM_TLB_EVICTS_H = 0xB8E -VX_CSR_MPM_PTW_WALKS = 0xB0F # PTW walks completed +VX_CSR_MPM_PTW_WALKS = 0xB0F # PTW walks started VX_CSR_MPM_PTW_WALKS_H = 0xB8F -VX_CSR_MPM_PTW_LATENCY = 0xB10 # PTW total latency cycles +VX_CSR_MPM_PTW_LATENCY = 0xB10 # sum of per-walk latencies (cycles) VX_CSR_MPM_PTW_LATENCY_H = 0xB90 +VX_CSR_MPM_PWC1_HITS = 0xB11 # walks that skipped the top level via the walk cache +VX_CSR_MPM_PWC1_HITS_H = 0xB91 +VX_CSR_MPM_PWC1_MISSES = 0xB12 +VX_CSR_MPM_PWC1_MISSES_H = 0xB92 +VX_CSR_MPM_PWC2_HITS = 0xB13 # Sv39 only: walks that also skipped the middle level +VX_CSR_MPM_PWC2_HITS_H = 0xB93 +VX_CSR_MPM_PWC2_MISSES = 0xB14 +VX_CSR_MPM_PWC2_MISSES_H = 0xB94 [csr_mpm_dxa] # PERF: DXA copy engine (cluster-level, same value on all cores in cluster) diff --git a/ci/testcases/vm.yaml b/ci/testcases/vm.yaml index fff93847a7..519419bea4 100644 --- a/ci/testcases/vm.yaml +++ b/ci/testcases/vm.yaml @@ -1,4 +1,6 @@ -# Test cases for `vm`, drafted from regression.sh.in (review/collapse). +# Test cases for `vm`. +# VX_VM_ADDR_MODE is a resolved VX_types contract fixed at configure time, so +# BARE cannot be selected per case via CONFIGS; only the default mode is tested. category: vm defaults: xlen: @@ -11,24 +13,28 @@ tests: via: make-run drivers: - simx + - rtlsim dir: tests/regression/sgemm target: run-{driver} - id: isa-2 via: make-run drivers: - simx + - rtlsim dir: tests/regression/diverge target: run-{driver} - id: isa-3 via: make-run drivers: - simx + - rtlsim dir: tests/regression/dogfood target: run-{driver} - id: isa-4 via: make-run drivers: - simx + - rtlsim dir: tests/regression/raycast target: run-{driver} - id: isa-5 @@ -37,38 +43,47 @@ tests: - simx dir: tests/graphics/gfx_draw3d target: run-{driver} -- id: isa-6 +- id: stress via: make-run drivers: - simx - dir: tests/regression/sgemm + - rtlsim + dir: tests/regression/vm_stress target: run-{driver} - configs+: -DVX_CFG_VM_ADDR_MODE=BARE -- id: isa-7 - via: make-run +- id: xrt-cp drivers: - - simx - dir: tests/regression/diverge - target: run-{driver} - configs+: -DVX_CFG_VM_ADDR_MODE=BARE -- id: isa-8 + - xrt + app: sgemm + tier: full +- id: tlb-banks1 via: make-run drivers: - - simx - dir: tests/regression/dogfood - target: run-{driver} - configs+: -DVX_CFG_VM_ADDR_MODE=BARE -- id: isa-9 - via: make-run - drivers: - - simx - dir: tests/regression/raycast + - rtlsim + dir: tests/regression/sgemm target: run-{driver} - configs+: -DVX_CFG_VM_ADDR_MODE=BARE -- id: isa-10 - via: make-run + configs+: -DVX_CFG_TLB_NUM_BANKS=1 + tier: full +- id: multi drivers: - - simx - dir: tests/graphics/gfx_draw3d - target: run-{driver} - configs+: -DVX_CFG_VM_ADDR_MODE=BARE + - rtlsim + app: sgemm + shape: {cores: 2, clusters: 2, l2cache: true, l3cache: true} + tier: full + +# SimX<->rtlsim cycle parity with translation on (see model_parity.yaml for +# the contract; check gates are RV32-pinned like the rest of the parity +# suite). The SimX MMU charges the banked-TLB lookup pipeline +# (Mmu::TRANSLATE_LATENCY + one lookup per bank per cycle); these cases hold +# that model to the RTL. Both are steady-state sized; sgemm's cold TLB misses +# also cover the walk path (vm_stress itself is too small to gate — all boot +# skew). +- id: model_parity-sgemm + check: model_parity + via: blackbox + app: sgemm + tier: full +- id: model_parity-diverge + check: model_parity + via: blackbox + app: diverge + tier: full diff --git a/docs/designs/command_processor.md b/docs/designs/command_processor.md index d856d6ff6a..a91772c4b1 100644 --- a/docs/designs/command_processor.md +++ b/docs/designs/command_processor.md @@ -194,11 +194,10 @@ The doorbell commits atomically on the `Q_TAIL_HI` write ([`:333-336`](../../hw/rtl/cp/VX_cp_axil_regfile.sv#L333)); undecoded addresses return DECERR. -> **FPGA/sim divergence:** the RTL regfile has **no** `CP_SATP_LO/HI` -> registers. The Emulation CP *does* (`0x028/0x02C`, -> [`cmd_processor.cpp:75-76`](../../sim/common/cmd_processor.cpp#L75)) and -> the runtime always writes them -> ([`device.cpp:303-304`](../../sw/runtime/common/device.cpp#L303)). See §8. +The RTL regfile carries `CP_SATP_LO/HI` at `0x028/0x02C` and publishes +`VM_ENABLED` in `DEV_CAPS` bit 24, matching the Emulation CP +(`cmd_processor.cpp`); the runtime writes SATP on both paths +([`device.cpp`](../../sw/runtime/common/device.cpp)). --- @@ -380,11 +379,11 @@ the intent is not lost. ([`cmd_processor.cpp:452-462`](../../sim/common/cmd_processor.cpp#L452)). Non-cache-line-aligned transfers can over-write on FPGA. Needs tail `wstrb` on the last beat (review item C-2/P-W4). -2. **VM in RTL.** Add `CP_SATP_LO/HI` regfile decode + a hardware - page-table walker + TLB in `VX_cp_dma`, and route `F_MEM_PHYSICAL`, so - FPGA matches the simulator's MMU-aware DMA (cp_pure_v2 VM Phase 2; - review items P-W1/P-W3 and the SATP gap). Today VM works on - simx/rtlsim/gem5 and silently no-ops on FPGA. +2. **VM in RTL — done.** `CP_SATP_LO/HI` decode, the `DEV_CAPS` + `VM_ENABLED` bit, `F_MEM_PHYSICAL` routing, and a per-chunk + `VX_cp_mmu` walker in `VX_cp_dma` bring the FPGA path up to the + simulator's MMU-aware DMA (remaining: a per-transfer TLB beyond the + walker's 2-entry translation cache). 3. **Real `CMD_FENCE` semantics.** The engine retires FENCE as a NOP ([`VX_cp_engine.sv:109-112`](../../hw/rtl/cp/VX_cp_engine.sv#L109)); it should honor `FENCE_DMA_BIT` / `FENCE_GPU_BIT` ordering (C-7). diff --git a/docs/designs/virtual_memory_subsystem.md b/docs/designs/virtual_memory_subsystem.md index 030844a34a..f44ac51c6a 100644 --- a/docs/designs/virtual_memory_subsystem.md +++ b/docs/designs/virtual_memory_subsystem.md @@ -83,24 +83,47 @@ VM is real — consistent with the project's 32-bit-only RTL policy. ## 3. RTL components - [`VX_mmu.sv`](../../hw/rtl/mem/VX_mmu.sv) (top) — merges an - elastic-buffered TLB path, a bypass path, and the PTW through - `VX_mem_arb`. Takes `satp[31:0]`; `needs_translation()` bypasses only on - `satp[31]` (BARE) — there is no address-range bypass - ([`:38-44`](../../hw/rtl/mem/VX_mmu.sv#L38)). -- [`VX_mmu_tlb.sv`](../../hw/rtl/mem/VX_mmu_tlb.sv) — fully-associative CAM - TLB, MRU victim select, 4-state FSM (`IDLE/READY/PTW_WAIT/REPLAY`). It - can *match* superpages via `page_level`/`vpn_mask`, but fills always set - `page_level=0` ([`:282`](../../hw/rtl/mem/VX_mmu_tlb.sv#L282)) so - megapages are stored as 4 KB entries. -- [`VX_mmu_ptw.sv`](../../hw/rtl/mem/VX_mmu_ptw.sv) — **Sv32-only, - hardcoded 2-level** walker (`L1_REQ/RESP → L0_REQ/RESP → FILL`); 4-byte - PTE; one walk in flight; does **not** yet act on V/R/W/X/U flags (page - faults un-delivered, [`:113-120`](../../hw/rtl/mem/VX_mmu_ptw.sv#L113)). - -Instantiated per core in [`VX_core.sv:440`](../../hw/rtl/core/VX_core.sv#L440) -(dcache MMU, `DCACHE_NUM_REQS` ports) and `:461` (icache MMU, 1 port), -both under `#ifdef VX_CFG_VM_ENABLE`. The MMU sits **after** the -coalescer / LSU adapter (a single per-core MMU, not per-LSU-slice). + elastic-buffered TLB path and a bypass path through `VX_mem_bus_arb`. + Takes `satp[XLEN-1:0]`; `needs_translation()` bypasses only on the SATP + mode field (BARE) — there is no address-range bypass. TLB misses leave + the core on a `VX_ptw_bus_if` toward the shared walker. +- [`VX_mmu_tlb.sv`](../../hw/rtl/mem/VX_mmu_tlb.sv) — banked TLB wrapper: + `VX_CFG_TLB_SIZE` entries split over `VX_CFG_TLB_NUM_BANKS` banks + selected by the low VPN bits (the iTLB is always single-banked). Lanes + are distributed and gathered by `VX_stream_xbar`s so up to a bank-count + of translations proceed per cycle; each bank owns one outstanding walk, + identified on the walker bus by its bank index. +- [`VX_mmu_tlb_bank.sv`](../../hw/rtl/mem/VX_mmu_tlb_bank.sv) — + fully-associative CAM bank, MRU victim select, 3-state FSM + (`READY/PTW_WAIT/REPLAY`). Entries carry their page level: superpage + leaves (Sv32 megapages; Sv39 mega/gigapages) match on the VPN bits above + their level and translate with the matching offset width. A faulted walk + replays the access untranslated (identity), mirroring the CP walker's + defensive pass-through. +- [`VX_ptw_bus_if.sv`](../../hw/rtl/mem/VX_ptw_bus_if.sv) / + [`VX_ptw_arb.sv`](../../hw/rtl/mem/VX_ptw_arb.sv) — the walk + request/fill bus (`{vpn, root_ppn, tag}` / `{ppn, level, flags, fault, + tag}`) and its N→1 arbiter; one arb per level folds the source index + into the tag: banks → core (i/d) → socket → cluster → device. +- [`VX_mmu_ptw.sv`](../../hw/rtl/mem/VX_mmu_ptw.sv) — **one instance per + device**: a generic Sv32/Sv39 walker with `VX_CFG_PTW_NUM_WALKERS` + concurrent walk slots, fetching PTEs on a **dedicated L3 requestor + port** (`L3_PTW_IDX`). Leaf/validity checks per the privileged spec + (V, R/W combos, superpage alignment, no-leaf-at-level-0) report `fault` + on the fill; under simulation a fault also raises an error. Two + direct-mapped page-walk caches (`VX_mmu_pwc.sv`, + `VX_CFG_PTW_WALK_CACHE_SIZE` entries) cache the non-leaf entries of the + two upper levels so a warm walk starts one (Sv32) or two (Sv39) levels + below the root. + +The per-core MMUs are instantiated in +[`VX_core.sv`](../../hw/rtl/core/VX_core.sv) (dcache MMU with +`DCACHE_NUM_REQS` ports and `VX_CFG_TLB_NUM_BANKS` banks; icache MMU with +1 port), both under `#ifdef VX_CFG_VM_ENABLE`, **after** the coalescer / +LSU adapter. The TLBs are invalidated by a one-cycle pulse at the start of +the DCR cache flush (the flush request itself is translated, so the +invalidate cannot be held at the level of the pending flush), and the +device walker drops its walk caches on the same DCR event. --- @@ -181,8 +204,9 @@ are not mistaken for bugs: - **No A/D-bit writeback.** The runtime pre-sets `A=D=1`; the TLBs are read-only. - **Page-fault delivery is partial.** SimX aborts on a fault; the RTL PTW - does not check `V/R/W/X/U` yet, and no fault is routed to the LSU as an - exception. + checks `V/R/W/X` and superpage alignment and reports `fault` on the TLB + fill (the access then replays untranslated), but no fault is routed to + the LSU as an exception yet. - **No PMP / page protection enforcement.** - **SV48/SV57** enum values exist but are unimplemented. @@ -200,19 +224,21 @@ are not mistaken for bugs: invalidation, and a `VX_dma` block split out of the CP with its own TLB into the shared hierarchy. It is the forward roadmap (`feature_vm_v2`); the current MMU is a single flat 32-entry FA TLB per cache port. -2. **RTL PTW Sv39 + superpage fills** — the RTL walker is Sv32-only and - stores megapages as 4 KB entries. Generalizing it (as SimX already is) - is required for RV64 VM on FPGA. -3. **RTL page-fault delivery** — check PTE `V/R/W/X/U` and route a fault to - the LSU as an exception (`VX_mmu_ptw.sv:113` stub). -4. **RTL CP shared device-side MMU** — Phase 2 of `vm_sw_stack_redesign`, +2. **RTL page-fault delivery to the LSU** — faults are detected and + reported on the TLB fill, but not yet routed to the LSU as an + exception (`U` and PMP checks also remain open). +3. **RTL CP shared device-side MMU** — Phase 2 of `vm_sw_stack_redesign`, deferred past v3: add the SATP regfile decode + a hardware walker so the CP DMA honors VM in RTL, matching the SimX/CP-software path (see `command_processor.md` §10 item 2). -5. **`configure --vm` first-class flag** — VM is still forced per build via +4. **`configure --vm` first-class flag** — VM is still forced per build via `CONFIGS=-DVX_CFG_VM_ENABLE`. -6. **RTL VM in CI** — the `vm()` regression runs SimX-only; the rtlsim/xrt - lines are commented out pending RTL PTW completion. + +Delivered since the v3 baseline (previously items on this list): a +generic Sv32/Sv39 RTL walker with superpage fills, shared per device with +concurrent walk slots and page-walk caches on a dedicated L3 port; banked +per-core TLBs honoring `VX_CFG_TLB_SIZE`; rtlsim VM in CI on both XLENs +(`ci/testcases/vm.yaml`), including the `vm_stress` TLB-pressure test. **Superseded directions** (recorded to avoid revival): the per-LSU-slice MMU placement of `vm_migration` (replaced by a single per-core MMU after diff --git a/docs/vm.md b/docs/vm.md index 7e6c7b014b..e20021b978 100644 --- a/docs/vm.md +++ b/docs/vm.md @@ -1,128 +1,97 @@ # Virtual Memory -Vortex supports per-core virtual-to-physical address translation using the -RISC-V SV32 (XLEN=32) and SV39 (XLEN=64) page-table formats. VM is enabled -by the TOML setting `VM_ENABLE = true` in [VX_config.toml](../VX_config.toml). - -This document covers the runtime model, the environment-variable knobs -exposed for testing, and the perf counters surfaced through the -`vx_dump_perf` reporting. +Vortex supports virtual-to-physical address translation using the RISC-V +Sv32 (XLEN=32) and Sv39 (XLEN=64) page-table formats. VM is enabled per +build with `CONFIGS="-DVX_CFG_VM_ENABLE"`; the default in +[VX_config.toml](../VX_config.toml) is off. The authoritative architecture +description is +[docs/designs/virtual_memory_subsystem.md](designs/virtual_memory_subsystem.md); +this page covers usage, configuration knobs, and perf reporting. ## Components | Layer | Where | Role | |---|---|---| -| Page table | RAM at `PAGE_TABLE_BASE_ADDR` (0xF0000000 on 32‑bit, 0x0F0000000 on 64‑bit) | Multi-level table installed by the runtime; consumed by every PTW | -| Runtime `VMManager` | [sw/runtime/common/vm.{h,cpp}](../sw/runtime/common/vm.cpp) | Allocates the page table, mints VAs on `vx_mem_alloc`, walks the table on `mem_free` / `vx_copy_to_dev` / `vx_copy_from_dev` | +| Page table | RAM at `VX_MEM_PAGE_TABLE_BASE_ADDR` (0xF0000000) | Multi-level table installed by the runtime; consumed by every walker | +| Runtime `VMManager` | [sw/runtime/common/vm.{h,cpp}](../sw/runtime/common/vm.cpp) | Builds the page table (host-shadow + batched flush), mints VAs on `vx_buffer_create`, identity-maps system regions and `VX_MEM_PHYS` buffers | | Kernel SATP write | [sw/kernel/src/vx_start.S](../sw/kernel/src/vx_start.S) | Each core writes the SATP CSR with the PT base and addressing mode at boot | -| Per-core MMU (RTL) | [hw/rtl/core/VX_mmu.sv](../hw/rtl/core/VX_mmu.sv) + [VX_mmu_tlb.sv](../hw/rtl/core/VX_mmu_tlb.sv) + [VX_mmu_ptw.sv](../hw/rtl/core/VX_mmu_ptw.sv) | Two instances per core: dcache MMU and icache MMU. Each owns a 32-entry CAM TLB and an SV32/SV39 page-table walker that fetches PTEs through its own bus-level shim sitting between [VX_mem_unit](../hw/rtl/core/VX_mem_unit.sv) and the cache cluster | -| Per-core MMU (SimX) | [sim/simx/core.cpp](../sim/simx/core.cpp), uses `MemoryUnit` from [sim/common/mem.h](../sim/common/mem.h) | Functional translator. `LsuUnit::process_request_step` calls `core_->translate(va, type)` for each lane before stuffing the PA into `LsuReq.addrs`; the icache fetch path does the same for `trace->PC` | +| Per-core MMU (RTL) | [hw/rtl/mem/VX_mmu.sv](../hw/rtl/mem/VX_mmu.sv) + [VX_mmu_tlb.sv](../hw/rtl/mem/VX_mmu_tlb.sv) + [VX_mmu_tlb_bank.sv](../hw/rtl/mem/VX_mmu_tlb_bank.sv) | Two instances per core (dcache and icache side): a banked CAM TLB translating inline on hits, superpage-aware | +| Shared walker (RTL) | [hw/rtl/mem/VX_mmu_ptw.sv](../hw/rtl/mem/VX_mmu_ptw.sv) + [VX_mmu_pwc.sv](../hw/rtl/mem/VX_mmu_pwc.sv) | One per device: generic Sv32/Sv39 walker, `VX_CFG_PTW_NUM_WALKERS` concurrent walks, page-walk caches, PTE fetches on a dedicated L3 port | +| SimX model | [sim/simx/mem/mmu.{h,cpp}](../sim/simx/mem/mmu.cpp) + [ptw.{h,cpp}](../sim/simx/mem/ptw.cpp) | Timing twin of the RTL: banked per-core TLBs, one shared `Ptw` SimObject on the L3 port | +| CP DMA translation | [sim/common/cmd_processor.cpp](../sim/common/cmd_processor.cpp) | The command processor walks the table for every `CMD_MEM_*` operand, so the host API is VA-only | -## Address layout (XLEN=32, SV32) +Translation is gated only by the SATP mode (BARE bypasses); there is no +address-range bypass. The runtime identity-maps the IO region, kernel +image, page-table region, and `VX_MEM_PHYS` allocations (using superpage +leaves where alignment allows), so PA-addressed traffic still resolves +correctly through the table. -``` -0x00000000 ─┬── IO region (no translation) bypass - │ -0x00010000 ─┴── USER_BASE_ADDR - │ - │ Translated user VA range - │ -0x80000000 ─┬── STARTUP_ADDR bypass - │ (kernel code at boot) (40000 bytes) -0x80040000 ─┴── - │ - │ Translated user VA range (cont'd) - │ -0xF0000000 ─┬── PAGE_TABLE_BASE_ADDR bypass - │ (page tables themselves) - │ -0xFFFF0000 ─┴── STACK / LMEM (above PT base) bypass -``` +## Configuration -Anything in the bypass ranges flows through the MMU's bypass path with -zero translation overhead. Only addresses in the translated ranges incur -TLB lookups and (on miss) PTW walks. +| Knob | Default | Meaning | +|---|---|---| +| `VX_CFG_VM_ENABLE` | off | Enables the MMU/walker hardware and VM runtime | +| `VX_CFG_TLB_SIZE` | 32 | TLB entries per MMU (power of two) | +| `VX_CFG_TLB_NUM_BANKS` | 4 | dTLB lookup banks (power of two dividing `TLB_SIZE`); the iTLB is always single-banked | +| `VX_CFG_PTW_NUM_WALKERS` | 8 | Concurrent walk slots in the shared walker | +| `VX_CFG_PTW_WALK_CACHE_SIZE` | 64 | Entries per page-walk cache (direct-mapped, power of two) | +| `VX_CFG_VM_PINNED_REGION_SIZE` | 256 MB | Identity-mapped slab for `VX_MEM_PHYS` allocations | ## Environment variables `VORTEX_RANDOMIZE_VA` and `VORTEX_VA_SEED` are read by `VMManager`'s constructor — see [vm.cpp](../sw/runtime/common/vm.cpp). -- `VORTEX_RANDOMIZE_VA=0` (default) — identity mapping. `vx_mem_alloc` - returns a VA equal to the underlying PA. Useful as the baseline; verifies - the translation pipeline does not corrupt addresses. -- `VORTEX_RANDOMIZE_VA=1` — for each `vx_mem_alloc`, mint a random - page-aligned base VA in `[ALLOC_BASE_ADDR, PAGE_TABLE_BASE_ADDR)` (32-bit - bounded), reserve the contiguous range, and install per-page PTEs. The - user receives the random VA; PA stays in `global_mem_`. -- `VORTEX_VA_SEED=N` — seed for the `std::mt19937_64` RNG. Default - `0x12345678`. Same seed → same VA stream across runs. - -### Randomization algorithm - -The runtime allocates an entire contiguous VA range upfront, then maps each -page sequentially so multi-page buffers stay contiguous in VA space: - -```cpp -// 1. Find a random contiguous VA range -uint64_t candidate_va = random_address_in_range(); -if (virtual_mem_->reserve(candidate_va, size) == 0) { - base_vpn = candidate_va >> MEM_PAGE_LOG2_SIZE; -} -// 2. Map each PPN to a sequential VPN -for (uint64_t i = 0; i < num_pages; i++) { - update_page_table(base_ppn + i, base_vpn + i, flags); -} -``` - -After 1000 failed reservation attempts (heavily fragmented VA space), it -falls back to sequential allocation so progress is guaranteed. +- `VORTEX_RANDOMIZE_VA=0` (default) — sequential VA allocation. +- `VORTEX_RANDOMIZE_VA=1` — for each allocation, mint a random + page-aligned contiguous VA range. The user receives the random VA; the + PA stays wherever `global_mem_` placed it. +- `VORTEX_VA_SEED=N` — RNG seed (default `0x12345678`). Same seed → same + VA stream across runs. ## Perf counters -Six MMU-related counters live in the memory-subsystem MPM class -(`VX_DCR_MPM_CLASS_MEM`, alongside off-chip memory, lmem, and the -coalescer). The hardware sums the icache and dcache MMU counters into one -bank exposed via `pipeline_perf.mmu` in -[VX_gpu_pkg.sv](../hw/rtl/VX_gpu_pkg.sv). +The MMU counters live in the memory-subsystem MPM class +(`VX_DCR_MPM_CLASS_MEM`). The TLB counters are per core (icache + dcache +MMU summed); the walker and walk-cache counters belong to the shared +device-level walker and read the same value on every core. | CSR | Meaning | |---|---| | `VX_CSR_MPM_TLB_READS` | Total TLB lookups (icache + dcache MMU) | | `VX_CSR_MPM_TLB_HITS` | TLB hits | -| `VX_CSR_MPM_TLB_MISSES` | TLB misses (each triggers a PTW) | +| `VX_CSR_MPM_TLB_MISSES` | TLB misses (each triggers a walk) | | `VX_CSR_MPM_TLB_EVICTS` | TLB evictions on fill | -| `VX_CSR_MPM_PTW_WALKS` | Completed PTW walks | -| `VX_CSR_MPM_PTW_LATENCY` | Total PTW latency in cycles (avg = LATENCY / WALKS) | +| `VX_CSR_MPM_PTW_WALKS` | Walks started | +| `VX_CSR_MPM_PTW_LATENCY` | Sum of per-walk latencies (avg = LATENCY / WALKS) | +| `VX_CSR_MPM_PWC1_HITS` / `_MISSES` | Walks that skipped the top level via the walk cache | +| `VX_CSR_MPM_PWC2_HITS` / `_MISSES` | Sv39 only: walks that also skipped the middle level | -[common/legacy_perf.cpp](../sw/runtime/common/legacy_perf.cpp) reads these -(from the `VX_DCR_MPM_CLASS_MEM` class) and prints a per-core `vm:` line in -the memory report when `--perf=7` (MEM class) is passed to `blackbox.sh`. -Example: +[sw/runtime/common/perf.cpp](../sw/runtime/common/perf.cpp) prints these +with `--perf=7` (MEM class): ``` -PERF: vm: tlb_reads=96, hit=96%, evicts=0, ptw_walks=4, ptw_avg_lat=84.75 +PERF: core0: tlb: reads=2086, hit=75%, misses=522, evicts=487 +PERF: ptw: walks=522, avg_lat=27.15 cyc, pwc1_hit=99%, pwc2_hit=0% ``` ## Testing -The project ships a regression script at the repo root: +The CI catalog is [ci/testcases/vm.yaml](../ci/testcases/vm.yaml): compute +regressions on simx **and rtlsim** at both XLENs, the +[tests/regression/vm_stress](../tests/regression/vm_stress) TLB-pressure +test (strided page touches + a `VX_MEM_PHYS` buffer), and full-tier +configuration variants (single-banked TLB, multi-cluster with L2/L3). ```bash -./run_vm_regression.sh --driver=simx # identity mapping -./run_vm_regression.sh --driver=rtlsim --perf -./run_vm_regression.sh --driver=simx --randomize # randomized VAs -./run_vm_regression.sh --driver=simx --randomize --seed=1 # reproducible +./ci/regression.sh --test vm +# or a single case: +CONFIGS="-DVX_CFG_VM_ENABLE" ./ci/blackbox.sh --driver=rtlsim --app=vm_stress --perf=7 ``` -The script runs a 24-test common subset on the chosen driver. With -`--randomize` and a fixed `--seed`, two runs produce identical VA -sequences across all tests — useful for triaging VM bugs. - ## Disabling VM -Set `VM_ENABLE = false` in [VX_config.toml](../VX_config.toml) and -re-`./configure`. With VM disabled the per-core MMU paths in -[VX_core.sv](../hw/rtl/core/VX_core.sv) compile out (the dcache and -icache buses connect straight through), the SimX `Core::translate` path -becomes a no-op, and the runtime `VMManager` is never constructed. +Leave `VX_CFG_VM_ENABLE` unset (the default). The per-core MMU paths in +[VX_core.sv](../hw/rtl/core/VX_core.sv) compile out (the dcache and icache +buses connect straight through), the shared walker and its L3 port are not +instantiated, and the runtime `VMManager` is never constructed. diff --git a/hw/rtl/VX_cluster.sv b/hw/rtl/VX_cluster.sv index ec8b7ac340..656288f1c8 100644 --- a/hw/rtl/VX_cluster.sv +++ b/hw/rtl/VX_cluster.sv @@ -45,6 +45,11 @@ module VX_cluster import VX_gpu_pkg::*; VX_raster_launch_if.slave raster_launch_if[1], `endif +`ifdef VX_CFG_VM_ENABLE + // Page-table walker + VX_ptw_bus_if.master ptw_bus_if, +`endif + // Status output wire busy ); @@ -128,6 +133,24 @@ module VX_cluster import VX_gpu_pkg::*; .gbar_bus_if (gbar_bus_if) ); +`ifdef VX_CFG_VM_ENABLE + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_SOCKET_TAG_WIDTH) + ) per_socket_ptw_bus_if[NUM_SOCKETS](); + + VX_ptw_arb #( + .NUM_INPUTS (NUM_SOCKETS), + .TAG_WIDTH (PTW_SOCKET_TAG_WIDTH), + .REQ_OUT_BUF ((NUM_SOCKETS > 1) ? 3 : 0), + .RSP_OUT_BUF ((NUM_SOCKETS > 1) ? 3 : 0) + ) ptw_arb ( + .clk (clk), + .reset (reset), + .bus_in_if (per_socket_ptw_bus_if), + .bus_out_if (ptw_bus_if) + ); +`endif + // L2 input buses (post-arb tag width when DXA enabled) VX_mem_bus_if #( .DATA_SIZE (`VX_CFG_L1_LINE_SIZE), @@ -439,6 +462,10 @@ module VX_cluster import VX_gpu_pkg::*; .gbar_bus_if (per_socket_gbar_bus_if[socket_id]), + `ifdef VX_CFG_VM_ENABLE + .ptw_bus_if (per_socket_ptw_bus_if[socket_id]), + `endif + .busy (per_socket_busy[socket_id]) ); end diff --git a/hw/rtl/VX_gpu_pkg.sv b/hw/rtl/VX_gpu_pkg.sv index 6e2beb711e..625a0c2d9b 100644 --- a/hw/rtl/VX_gpu_pkg.sv +++ b/hw/rtl/VX_gpu_pkg.sv @@ -1066,14 +1066,23 @@ package VX_gpu_pkg; } coalescer_perf_t; `ifdef VX_CFG_VM_ENABLE + // Per-core TLB counters (summed over the i/d MMUs of a core). typedef struct packed { logic [PERF_CTR_BITS-1:0] tlb_reads; logic [PERF_CTR_BITS-1:0] tlb_hits; logic [PERF_CTR_BITS-1:0] tlb_misses; logic [PERF_CTR_BITS-1:0] tlb_evictions; - logic [PERF_CTR_BITS-1:0] ptw_walks; - logic [PERF_CTR_BITS-1:0] ptw_latency; } mmu_perf_t; + + // Device-level page-table walker counters (shared PTW + walk caches). + typedef struct packed { + logic [PERF_CTR_BITS-1:0] walks; + logic [PERF_CTR_BITS-1:0] latency; + logic [PERF_CTR_BITS-1:0] pwc1_hits; + logic [PERF_CTR_BITS-1:0] pwc1_misses; + logic [PERF_CTR_BITS-1:0] pwc2_hits; + logic [PERF_CTR_BITS-1:0] pwc2_misses; + } ptw_perf_t; `endif `ifdef VX_CFG_EXT_TCU_ENABLE @@ -1157,6 +1166,9 @@ package VX_gpu_pkg; lmem_perf_t lmem; coalescer_perf_t coalescer; mem_perf_t mem; + `ifdef VX_CFG_VM_ENABLE + ptw_perf_t ptw; + `endif `ifdef VX_CFG_EXT_DXA_ENABLE dxa_perf_t dxa; `endif @@ -1286,9 +1298,9 @@ package VX_gpu_pkg; localparam ICACHE_TAG_WIDTH_BASE = (ICACHE_FETCH_TAG_WIDTH + 1); `ifdef VX_CFG_VM_ENABLE localparam ICACHE_TLB_SOURCE_BITS = `UP(`CLOG2(1)); - // VX_mmu's internal merge_arb folds (2*NUM_REQS+1) inputs to NUM_REQS - // outputs, inserting CLOG2(CDIV(2*NUM_REQS+1, NUM_REQS)) sel bits. - localparam ICACHE_ARB_BITS = `CLOG2(`CDIV(2 * 1 + 1, 1)); + // VX_mmu's internal merge_arb folds the bypass and TLB streams + // (2*NUM_REQS inputs) onto NUM_REQS outputs. + localparam ICACHE_ARB_BITS = `ARB_SEL_BITS(2 * 1, 1); localparam ICACHE_TAG_WIDTH = (ICACHE_TAG_WIDTH_BASE + ICACHE_TLB_SOURCE_BITS + ICACHE_ARB_BITS); `else localparam ICACHE_TAG_WIDTH = ICACHE_TAG_WIDTH_BASE; @@ -1339,9 +1351,9 @@ package VX_gpu_pkg; localparam DCACHE_TAG_WIDTH_BASE = (DCACHE_CORE_TAG_WIDTH + 1); `ifdef VX_CFG_VM_ENABLE localparam DCACHE_TLB_SOURCE_BITS = `UP(`CLOG2(DCACHE_NUM_REQS)); - // VX_mmu's internal merge_arb folds (2*NUM_REQS+1) inputs to NUM_REQS - // outputs, inserting CLOG2(CDIV(2*NUM_REQS+1, NUM_REQS)) sel bits. - localparam DCACHE_ARB_BITS = `CLOG2(`CDIV(2 * DCACHE_NUM_REQS + 1, DCACHE_NUM_REQS)); + // VX_mmu's internal merge_arb folds the bypass and TLB streams + // (2*NUM_REQS inputs) onto NUM_REQS outputs. + localparam DCACHE_ARB_BITS = `ARB_SEL_BITS(2 * DCACHE_NUM_REQS, DCACHE_NUM_REQS); localparam DCACHE_TAG_WIDTH = (DCACHE_TAG_WIDTH_BASE + DCACHE_TLB_SOURCE_BITS + DCACHE_ARB_BITS); `else localparam DCACHE_TAG_WIDTH = DCACHE_TAG_WIDTH_BASE; @@ -1567,8 +1579,11 @@ package VX_gpu_pkg; // Sector = mem transaction granule (= line when 1 sector/line) localparam L3_SECTOR_SIZE = `VX_CFG_L3_SECTOR_SIZE; - // Input request size - localparam L3_NUM_REQS = `VX_CFG_NUM_CLUSTERS * L2_MEM_PORTS; + // Input request size: cluster memory ports plus, under VM, the shared + // page-table walker's dedicated port at index L3_PTW_IDX. + localparam L3_CLUSTER_REQS = `VX_CFG_NUM_CLUSTERS * L2_MEM_PORTS; + localparam L3_PTW_IDX = L3_CLUSTER_REQS; + localparam L3_NUM_REQS = L3_CLUSTER_REQS + `VX_CFG_VM_ENABLED; // Core request tag bits localparam L3_TAG_WIDTH = L2_MEM_TAG_WIDTH; @@ -1585,6 +1600,43 @@ package VX_gpu_pkg; // L3 is the LLC whenever it is enabled. localparam L3_IS_LLC = `VX_CFG_L3_ENABLED; + /////////////////////////////// VM Parameters ///////////////////////////// + +`ifdef VX_CFG_VM_ENABLE + // RISC-V Sv32 / Sv39 page-table geometry, derived from the VX_VM_* contract. + localparam VM_PAGE_OFFSET_BITS = `VX_VM_PAGE_LOG2_SIZE; + localparam VM_PT_LEVELS = `VX_VM_PT_LEVEL; + localparam VM_PTE_SIZE = `VX_VM_PTE_SIZE; + localparam VM_VPN_LEVEL_BITS = `CLOG2(`VX_VM_PT_SIZE / `VX_VM_PTE_SIZE); + localparam VM_VPN_WIDTH = VM_PT_LEVELS * VM_VPN_LEVEL_BITS; + localparam VM_PPN_WIDTH = `VX_CFG_MEM_ADDR_WIDTH - VM_PAGE_OFFSET_BITS; + localparam VM_LEVEL_BITS = `UP(`CLOG2(VM_PT_LEVELS)); + localparam VM_PTE_FLAGS_WIDTH = 8; + localparam VM_PTE_PPN_LSB = 10; + + localparam VM_PTE_FLAG_V = 0; + localparam VM_PTE_FLAG_R = 1; + localparam VM_PTE_FLAG_W = 2; + localparam VM_PTE_FLAG_X = 3; + + // PTW bus tag growth along the request path: + // TLB banks -> core (i/d select) -> socket -> cluster -> device PTW. + localparam PTW_TLB_TAG_WIDTH = `UP(`CLOG2(`VX_CFG_TLB_NUM_BANKS)); + localparam PTW_CORE_TAG_WIDTH = PTW_TLB_TAG_WIDTH + 1; + localparam PTW_SOCKET_TAG_WIDTH = PTW_CORE_TAG_WIDTH + `ARB_SEL_BITS(`VX_CFG_SOCKET_SIZE, 1); + localparam PTW_CLUSTER_TAG_WIDTH = PTW_SOCKET_TAG_WIDTH + `ARB_SEL_BITS(NUM_SOCKETS, 1); + localparam PTW_DEV_TAG_WIDTH = PTW_CLUSTER_TAG_WIDTH + `ARB_SEL_BITS(`VX_CFG_NUM_CLUSTERS, 1); + + function automatic logic vm_pte_valid(input logic [VM_PTE_FLAGS_WIDTH-1:0] flags); + // V=0, or the reserved W-without-R encoding, is an invalid PTE. + return flags[VM_PTE_FLAG_V] && !(flags[VM_PTE_FLAG_W] && !flags[VM_PTE_FLAG_R]); + endfunction + + function automatic logic vm_pte_is_leaf(input logic [VM_PTE_FLAGS_WIDTH-1:0] flags); + return flags[VM_PTE_FLAG_R] || flags[VM_PTE_FLAG_W] || flags[VM_PTE_FLAG_X]; + endfunction +`endif + /////////////////////////////////////////////////////////////////////////// localparam VX_MEM_PORTS = L3_MEM_PORTS; diff --git a/hw/rtl/VX_socket.sv b/hw/rtl/VX_socket.sv index d6d8891b21..23dbfabc77 100644 --- a/hw/rtl/VX_socket.sv +++ b/hw/rtl/VX_socket.sv @@ -70,6 +70,11 @@ module VX_socket import VX_gpu_pkg::*; // Global barrier VX_gbar_bus_if.master gbar_bus_if, +`ifdef VX_CFG_VM_ENABLE + // Page-table walker + VX_ptw_bus_if.master ptw_bus_if, +`endif + // Status output wire busy ); @@ -105,6 +110,24 @@ module VX_socket import VX_gpu_pkg::*; .bus_out_if (gbar_bus_if) ); +`ifdef VX_CFG_VM_ENABLE + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_CORE_TAG_WIDTH) + ) per_core_ptw_bus_if[`VX_CFG_SOCKET_SIZE](); + + VX_ptw_arb #( + .NUM_INPUTS (`VX_CFG_SOCKET_SIZE), + .TAG_WIDTH (PTW_CORE_TAG_WIDTH), + .REQ_OUT_BUF ((`VX_CFG_SOCKET_SIZE > 1) ? 3 : 0), + .RSP_OUT_BUF ((`VX_CFG_SOCKET_SIZE > 1) ? 3 : 0) + ) ptw_arb ( + .clk (clk), + .reset (reset), + .bus_in_if (per_core_ptw_bus_if), + .bus_out_if (ptw_bus_if) + ); +`endif + /////////////////////////////////////////////////////////////////////////// `ifdef PERF_ENABLE @@ -508,6 +531,10 @@ module VX_socket import VX_gpu_pkg::*; .gbar_bus_if (per_core_gbar_bus_if[core_id]), + `ifdef VX_CFG_VM_ENABLE + .ptw_bus_if (per_core_ptw_bus_if[core_id]), + `endif + .busy (per_core_busy[core_id]) ); end diff --git a/hw/rtl/Vortex.sv b/hw/rtl/Vortex.sv index 4b3d233abc..83f69e3be9 100644 --- a/hw/rtl/Vortex.sv +++ b/hw/rtl/Vortex.sv @@ -106,18 +106,25 @@ module Vortex import VX_gpu_pkg::*, VX_trace_pkg::*; ( `ifdef PERF_ENABLE cache_perf_t l3_perf; mem_perf_t mem_perf; +`ifdef VX_CFG_VM_ENABLE + ptw_perf_t ptw_perf; +`endif sysmem_perf_t sysmem_perf; always @(*) begin sysmem_perf = '0; sysmem_perf.l3cache = l3_perf; sysmem_perf.mem = mem_perf; + `ifdef VX_CFG_VM_ENABLE + sysmem_perf.ptw = ptw_perf; + `endif end `endif + // L3 requestors: the cluster memory ports, then the shared PTW port. VX_mem_bus_if #( .DATA_SIZE (L2_SECTOR_SIZE), .TAG_WIDTH (L3_TAG_WIDTH) - ) per_cluster_mem_bus_if[`VX_CFG_NUM_CLUSTERS * L2_MEM_PORTS](); + ) per_cluster_mem_bus_if[L3_NUM_REQS](); VX_mem_bus_if #( .DATA_SIZE (L3_SECTOR_SIZE), @@ -215,6 +222,51 @@ module Vortex import VX_gpu_pkg::*, VX_trace_pkg::*; ( .bus_out_if (per_cluster_dcr_bus_if) ); +`ifdef VX_CFG_VM_ENABLE + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_CLUSTER_TAG_WIDTH) + ) per_cluster_ptw_bus_if[`VX_CFG_NUM_CLUSTERS](); + + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_DEV_TAG_WIDTH) + ) ptw_bus_if(); + + VX_ptw_arb #( + .NUM_INPUTS (`VX_CFG_NUM_CLUSTERS), + .TAG_WIDTH (PTW_CLUSTER_TAG_WIDTH), + .REQ_OUT_BUF ((`VX_CFG_NUM_CLUSTERS > 1) ? 3 : 0), + .RSP_OUT_BUF ((`VX_CFG_NUM_CLUSTERS > 1) ? 3 : 0) + ) ptw_arb ( + .clk (clk), + .reset (reset), + .bus_in_if (per_cluster_ptw_bus_if), + .bus_out_if (ptw_bus_if) + ); + + // The walk caches hold page-table contents; drop them whenever the host + // flushes the caches, which it does before page tables change. + wire ptw_flush = dcr_bus_if.req_valid + && ~dcr_bus_if.req_data.rw + && (dcr_bus_if.req_data.addr == `VX_DCR_BASE_CACHE_FLUSH); + + VX_mmu_ptw #( + .NUM_WALKERS (`VX_CFG_PTW_NUM_WALKERS), + .PWC_SIZE (`VX_CFG_PTW_WALK_CACHE_SIZE), + .TAG_WIDTH (PTW_DEV_TAG_WIDTH), + .MEM_DATA_SIZE (L2_SECTOR_SIZE), + .MEM_TAG_WIDTH (L3_TAG_WIDTH) + ) ptw ( + .clk (clk), + .reset (reset), + .flush (ptw_flush), + `ifdef PERF_ENABLE + .ptw_perf (ptw_perf), + `endif + .ptw_bus_if (ptw_bus_if), + .mem_bus_if (per_cluster_mem_bus_if[L3_PTW_IDX]) + ); +`endif + // Generate all clusters for (genvar cluster_id = 0; cluster_id < `VX_CFG_NUM_CLUSTERS; ++cluster_id) begin : g_clusters @@ -241,6 +293,10 @@ module Vortex import VX_gpu_pkg::*, VX_trace_pkg::*; ( .raster_launch_if (per_cluster_raster_launch_if[cluster_id +: 1]), `endif + `ifdef VX_CFG_VM_ENABLE + .ptw_bus_if (per_cluster_ptw_bus_if[cluster_id]), + `endif + .busy (per_cluster_busy[cluster_id]) ); end diff --git a/hw/rtl/core/VX_core.sv b/hw/rtl/core/VX_core.sv index 7b036a7e5a..a22030c68f 100644 --- a/hw/rtl/core/VX_core.sv +++ b/hw/rtl/core/VX_core.sv @@ -68,6 +68,11 @@ module VX_core import VX_gpu_pkg::*; #( // Global barrier VX_gbar_bus_if.master gbar_bus_if, +`ifdef VX_CFG_VM_ENABLE + // Page-table walker + VX_ptw_bus_if.master ptw_bus_if, +`endif + // Status output wire busy ); @@ -523,24 +528,53 @@ module VX_core import VX_gpu_pkg::*; #( assign pipeline_perf.mmu.tlb_hits = dcache_mmu_perf.tlb_hits + icache_mmu_perf.tlb_hits; assign pipeline_perf.mmu.tlb_misses = dcache_mmu_perf.tlb_misses + icache_mmu_perf.tlb_misses; assign pipeline_perf.mmu.tlb_evictions = dcache_mmu_perf.tlb_evictions + icache_mmu_perf.tlb_evictions; - assign pipeline_perf.mmu.ptw_walks = dcache_mmu_perf.ptw_walks + icache_mmu_perf.ptw_walks; - assign pipeline_perf.mmu.ptw_latency = dcache_mmu_perf.ptw_latency + icache_mmu_perf.ptw_latency; `endif + // Both MMUs share the core's walker port. The TLBs are invalidated once + // when the DCR cache flush starts so a new launch never sees the previous + // launch's translations; the flush request itself is translated too, so + // the invalidate must be an edge, not the level of the pending flush. + reg dcr_flush_req_r; + always @(posedge clk) begin + if (reset) begin + dcr_flush_req_r <= 1'b0; + end else begin + dcr_flush_req_r <= dcr_flush_if.req; + end + end + wire mmu_flush = dcr_flush_if.req && ~dcr_flush_req_r; + + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_TLB_TAG_WIDTH) + ) mmu_ptw_bus_if[2](); + + VX_ptw_arb #( + .NUM_INPUTS (2), + .TAG_WIDTH (PTW_TLB_TAG_WIDTH) + ) ptw_arb ( + .clk (clk), + .reset (reset), + .bus_in_if (mmu_ptw_bus_if), + .bus_out_if (ptw_bus_if) + ); + // Per-core dcache MMU. VX_mmu #( .NUM_REQS (DCACHE_NUM_REQS), + .NUM_BANKS (`VX_CFG_TLB_NUM_BANKS), .DATA_SIZE (DCACHE_WORD_SIZE), .TAG_WIDTH (DCACHE_TAG_WIDTH_BASE) ) dcache_mmu ( .clk (clk), .reset (reset), + .flush (mmu_flush), `ifdef PERF_ENABLE .mmu_perf (dcache_mmu_perf), `endif .satp (sched_csr_if.csr_satp), .lsu_mem_if (mmu_dcache_if), - .dcache_mem_if (dcache_bus_if) + .dcache_mem_if (dcache_bus_if), + .ptw_bus_if (mmu_ptw_bus_if[0]) ); // Per-core icache MMU. NUM_REQS=1. @@ -551,17 +585,20 @@ module VX_core import VX_gpu_pkg::*; #( VX_mmu #( .NUM_REQS (1), + .NUM_BANKS (1), .DATA_SIZE (ICACHE_WORD_SIZE), .TAG_WIDTH (ICACHE_TAG_WIDTH_BASE) ) icache_mmu ( .clk (clk), .reset (reset), + .flush (mmu_flush), `ifdef PERF_ENABLE .mmu_perf (icache_mmu_perf), `endif .satp (sched_csr_if.csr_satp), .lsu_mem_if (mmu_icache_if), - .dcache_mem_if (icache_mmu_out_if) + .dcache_mem_if (icache_mmu_out_if), + .ptw_bus_if (mmu_ptw_bus_if[1]) ); `ASSIGN_VX_MEM_BUS_IF (icache_bus_if, icache_mmu_out_if[0]); diff --git a/hw/rtl/core/VX_csr_data.sv b/hw/rtl/core/VX_csr_data.sv index 696441f858..f1cdfdc253 100644 --- a/hw/rtl/core/VX_csr_data.sv +++ b/hw/rtl/core/VX_csr_data.sv @@ -359,13 +359,17 @@ import VX_fpu_pkg::*; // PERF: coalescer `CSR_READ_64(`VX_CSR_MPM_COALESCER_MISS, read_data_ro_w, sysmem_perf.coalescer.misses); `ifdef VX_CFG_VM_ENABLE - // PERF: VM/MMU (icache + dcache MMU summed) + // PERF: VM/MMU (per-core TLBs, device-level walker) `CSR_READ_64(`VX_CSR_MPM_TLB_READS, read_data_ro_w, pipeline_perf.mmu.tlb_reads); `CSR_READ_64(`VX_CSR_MPM_TLB_HITS, read_data_ro_w, pipeline_perf.mmu.tlb_hits); `CSR_READ_64(`VX_CSR_MPM_TLB_MISSES, read_data_ro_w, pipeline_perf.mmu.tlb_misses); `CSR_READ_64(`VX_CSR_MPM_TLB_EVICTS, read_data_ro_w, pipeline_perf.mmu.tlb_evictions); - `CSR_READ_64(`VX_CSR_MPM_PTW_WALKS, read_data_ro_w, pipeline_perf.mmu.ptw_walks); - `CSR_READ_64(`VX_CSR_MPM_PTW_LATENCY, read_data_ro_w, pipeline_perf.mmu.ptw_latency); + `CSR_READ_64(`VX_CSR_MPM_PTW_WALKS, read_data_ro_w, sysmem_perf.ptw.walks); + `CSR_READ_64(`VX_CSR_MPM_PTW_LATENCY, read_data_ro_w, sysmem_perf.ptw.latency); + `CSR_READ_64(`VX_CSR_MPM_PWC1_HITS, read_data_ro_w, sysmem_perf.ptw.pwc1_hits); + `CSR_READ_64(`VX_CSR_MPM_PWC1_MISSES, read_data_ro_w, sysmem_perf.ptw.pwc1_misses); + `CSR_READ_64(`VX_CSR_MPM_PWC2_HITS, read_data_ro_w, sysmem_perf.ptw.pwc2_hits); + `CSR_READ_64(`VX_CSR_MPM_PWC2_MISSES, read_data_ro_w, sysmem_perf.ptw.pwc2_misses); `endif default:; endcase diff --git a/hw/rtl/cp/VX_cp_axil_regfile.sv b/hw/rtl/cp/VX_cp_axil_regfile.sv index cbc96c6f71..0f22a81a4c 100644 --- a/hw/rtl/cp/VX_cp_axil_regfile.sv +++ b/hw/rtl/cp/VX_cp_axil_regfile.sv @@ -76,12 +76,16 @@ module VX_cp_axil_regfile output cpe_state_t q_state [NUM_QUEUES], // One-cycle reset pulse per queue when the host writes Q_CONTROL.reset. - output logic q_reset_pulse [NUM_QUEUES] + output logic q_reset_pulse [NUM_QUEUES], + + // CP_SATP — page-table root for the CP DMA's walker. + output logic [63:0] satp ); localparam int QID_W = (NUM_QUEUES > 1) ? $clog2(NUM_QUEUES) : 1; // ---- Per-queue programmable state ---- + logic [63:0] r_satp; logic [63:0] r_ring_base [NUM_QUEUES]; logic [63:0] r_head_addr [NUM_QUEUES]; logic [63:0] r_cmpl_addr [NUM_QUEUES]; @@ -174,7 +178,10 @@ module VX_cp_axil_regfile logic [5:0] off; if (is_global(addr, 8'h00)) return r_cp_ctrl; if (is_global(addr, 8'h04)) return {30'd0, cp_error, cp_busy}; - if (is_global(addr, 8'h08)) return {8'd0, + // DEV_CAPS bit 24 = VM_ENABLED: the runtime discovers VM support here + // and only then builds page tables and programs CP_SATP. + if (is_global(addr, 8'h08)) return {7'd0, + 1'(`VX_CFG_VM_ENABLED), 8'(AXI_TID_W), 8'(RING_SIZE_LOG2_MAX), 8'(NUM_QUEUES)}; @@ -184,6 +191,8 @@ module VX_cp_axil_regfile if (is_global(addr, 8'h1C)) return gpu_dev_caps[63:32]; if (is_global(addr, 8'h20)) return gpu_isa_caps[31:0]; if (is_global(addr, 8'h24)) return gpu_isa_caps[63:32]; + if (is_global(addr, 8'h28)) return r_satp[31:0]; + if (is_global(addr, 8'h2C)) return r_satp[63:32]; if (decode_queue(addr, qid, off)) begin case (off) 6'h00: return r_ring_base[qid][31:0]; @@ -218,6 +227,8 @@ module VX_cp_axil_regfile if (is_global(addr, 8'h1C)) return 1'b1; if (is_global(addr, 8'h20)) return 1'b1; if (is_global(addr, 8'h24)) return 1'b1; + if (is_global(addr, 8'h28)) return 1'b1; + if (is_global(addr, 8'h2C)) return 1'b1; if (decode_queue(addr, qid, off)) begin case (off) 6'h00, 6'h04, 6'h08, 6'h0C, 6'h10, 6'h14, @@ -290,6 +301,7 @@ module VX_cp_axil_regfile automatic logic [5:0] off; if (reset) begin r_cp_ctrl <= '0; + r_satp <= '0; for (int i = 0; i < NUM_QUEUES; ++i) begin r_ring_base[i] <= '0; r_head_addr[i] <= '0; @@ -306,7 +318,11 @@ module VX_cp_axil_regfile for (int i = 0; i < NUM_QUEUES; ++i) q_reset_pulse[i] <= 1'b0; if (wr_commit && is_decoded(wr_addr_buf)) begin - if (is_global(wr_addr_buf, 8'h00)) begin + if (is_global(wr_addr_buf, 8'h28)) begin + r_satp[31:0] <= wr_data_buf; + end else if (is_global(wr_addr_buf, 8'h2C)) begin + r_satp[63:32] <= wr_data_buf; + end else if (is_global(wr_addr_buf, 8'h00)) begin r_cp_ctrl <= wr_data_buf; if (wr_data_buf[1]) begin for (int i = 0; i < NUM_QUEUES; ++i) q_reset_pulse[i] <= 1'b1; @@ -402,4 +418,6 @@ module VX_cp_axil_regfile end endgenerate + assign satp = r_satp; + endmodule : VX_cp_axil_regfile diff --git a/hw/rtl/cp/VX_cp_core.sv b/hw/rtl/cp/VX_cp_core.sv index d5aa249255..e0c3bbdbeb 100644 --- a/hw/rtl/cp/VX_cp_core.sv +++ b/hw/rtl/cp/VX_cp_core.sv @@ -132,6 +132,8 @@ module VX_cp_core wire [`VX_DCR_DATA_BITS-1:0] dcr_last_rsp_data; + wire [63:0] cp_satp; + VX_cp_axil_regfile #( .NUM_QUEUES (NUM_QUEUES), .ADDR_W (AXIL_AW) @@ -146,7 +148,8 @@ module VX_cp_core .q_error (q_error_to_reg), .last_dcr_rsp (dcr_last_rsp_data), .q_state (q_state), - .q_reset_pulse (q_reset_pulse) + .q_reset_pulse (q_reset_pulse), + .satp (cp_satp) ); // ----- Per-CPE wires ----- @@ -339,6 +342,11 @@ module VX_cp_core ); `UNUSED_VAR (gpu_if_int.dcr_req_ready) + // CMD_CACHE_FLUSH sweeps a CACHE_FLUSH DCR read across the cores; the + // same event invalidates the DMA walker's translation cache. + wire cp_xlat_flush = gpu_if_int.dcr_req_valid && !gpu_if_int.dcr_req_rw + && (gpu_if_int.dcr_req_addr == `VX_DCR_ADDR_BITS'(`VX_DCR_BASE_CACHE_FLUSH)); + // ----- DMA (straddles host + dev xbars) ----- VX_mem_axi_if #(.ADDR_W(ADDR_W), .DATA_W(DATA_W), .ID_W(ID_W)) dma_host_axi (); VX_mem_axi_if #(.ADDR_W(ADDR_W), .DATA_W(DATA_W), .ID_W(ID_W)) dma_dev_axi (); @@ -348,6 +356,8 @@ module VX_cp_core VX_cp_dma u_dma ( .clk (clk), .reset (reset), + .satp (cp_satp), + .xlat_flush (cp_xlat_flush), .grant (any_dma_grant), .cmd (granted_dma_cmd), .done (dma_done), diff --git a/hw/rtl/cp/VX_cp_dma.sv b/hw/rtl/cp/VX_cp_dma.sv index bd80fd6524..2f4f01d4bd 100644 --- a/hw/rtl/cp/VX_cp_dma.sv +++ b/hw/rtl/cp/VX_cp_dma.sv @@ -46,6 +46,10 @@ module VX_cp_dma input wire clk, input wire reset, + input wire [63:0] satp, + // page tables may have changed (CACHE_FLUSH DCR): drop cached translations + input wire xlat_flush, + input wire grant, input cmd_t cmd, output logic done, @@ -60,13 +64,15 @@ module VX_cp_dma localparam int BIDX_W = 6; // beat index 0..63 localparam int BCNT_W = 7; // chunk length 1..64 - typedef enum logic [2:0] { - S_IDLE, S_SETUP, S_REQ_AR, S_READ, S_REQ_AW, S_WRITE, S_WAIT_B, S_DONE + typedef enum logic [3:0] { + S_IDLE, S_SETUP, S_XLAT_RD, S_REQ_AR, S_READ, S_XLAT_WR, S_REQ_AW, S_WRITE, S_WAIT_B, S_DONE } state_e; state_e state; logic [7:0] op_r; // latched opcode (host/dev routing) - logic [63:0] dst_r, src_r; + logic phys_r; // F_MEM_PHYSICAL: skip translation + logic [63:0] dst_r, src_r; // as issued (VAs under VM) + logic [63:0] src_pa_r, dst_pa_r; // per-chunk translated addresses logic [63:0] rem_beats; // 64 B beats still to move logic [BCNT_W-1:0] chunk_beats; // beats in the current chunk logic [BIDX_W-1:0] beat_idx; @@ -95,6 +101,59 @@ module VX_cp_dma wire rd_from_host = (cp_opcode_e'(op_r) == CMD_MEM_WRITE); // upload: read host wire wr_to_host = (cp_opcode_e'(op_r) == CMD_MEM_READ); // download: write host + // ---- Device-side address translation ---- + // Host addresses are host-physical and never translate; device-side + // operands are VAs under VM unless the command carries F_MEM_PHYSICAL. + // Chunks never cross a 4 KB boundary, so one translation per chunk per + // device-side operand suffices. +`ifdef VX_CFG_VM_ENABLE + wire xlat_enable = !phys_r; + wire xlat_rd_needed = xlat_enable && !rd_from_host; + wire xlat_wr_needed = xlat_enable && !wr_to_host; + wire xlat_req_valid = ((state == S_XLAT_RD) || (state == S_XLAT_WR)); + wire [63:0] xlat_req_vaddr = (state == S_XLAT_RD) ? src_r : dst_r; + wire xlat_rsp_valid; + wire [63:0] xlat_rsp_paddr; + wire xlat_req_ready; + `UNUSED_VAR (xlat_req_ready) + + wire mmu_arvalid; + wire [63:0] mmu_araddr; + wire mmu_rready; + wire xlat_active = xlat_req_valid; + + VX_cp_mmu u_mmu ( + .clk (clk), + .reset (reset), + .satp (satp), + .flush (xlat_flush), + .req_valid (xlat_req_valid && !xlat_rsp_valid), + .req_ready (xlat_req_ready), + .req_vaddr (xlat_req_vaddr), + .rsp_valid (xlat_rsp_valid), + .rsp_ready (1'b1), + .rsp_paddr (xlat_rsp_paddr), + .mem_arvalid (mmu_arvalid), + .mem_arready (axi_dev.arready), + .mem_araddr (mmu_araddr), + .mem_rvalid (axi_dev.rvalid), + .mem_rready (mmu_rready), + .mem_rdata (axi_dev.rdata) + ); +`else + wire xlat_rd_needed = 1'b0; + wire xlat_wr_needed = 1'b0; + wire xlat_rsp_valid = 1'b0; + wire [63:0] xlat_rsp_paddr = '0; + wire xlat_active = 1'b0; + wire mmu_arvalid = 1'b0; + wire [63:0] mmu_araddr = '0; + wire mmu_rready = 1'b0; + `UNUSED_VAR (satp) + `UNUSED_VAR (xlat_flush) + `UNUSED_VAR (phys_r) +`endif + // Last beat of the current chunk. wire last_beat = (BCNT_W'({1'b0, beat_idx}) == (chunk_beats - BCNT_W'(1))); @@ -113,6 +172,7 @@ module VX_cp_dma S_IDLE: begin if (grant) begin op_r <= cmd.hdr.opcode; + phys_r <= cmd.hdr.flags[F_MEM_PHYSICAL]; dst_r <= cmd.arg0; src_r <= cmd.arg1; // Round the byte count up to a whole cache line. @@ -126,7 +186,21 @@ module VX_cp_dma end else begin chunk_beats <= next_chunk; beat_idx <= '0; - state <= S_REQ_AR; + src_pa_r <= src_r; + dst_pa_r <= dst_r; + state <= xlat_rd_needed ? S_XLAT_RD : S_REQ_AR; + end + end + S_XLAT_RD: begin + if (xlat_rsp_valid) begin + src_pa_r <= xlat_rsp_paddr; + state <= S_REQ_AR; + end + end + S_XLAT_WR: begin + if (xlat_rsp_valid) begin + dst_pa_r <= xlat_rsp_paddr; + state <= S_REQ_AW; end end S_REQ_AR: begin @@ -140,7 +214,7 @@ module VX_cp_dma buf_r[beat_idx] <= rd_rdata; if (last_beat) begin beat_idx <= '0; - state <= S_REQ_AW; + state <= xlat_wr_needed ? S_XLAT_WR : S_REQ_AW; end else begin beat_idx <= beat_idx + BIDX_W'(1); end @@ -198,7 +272,7 @@ module VX_cp_dma always_comb begin // ----- axi_host ----- axi_host.arvalid = rd_arvalid & rd_from_host; - axi_host.araddr = src_r; + axi_host.araddr = src_pa_r; axi_host.arid = TID_PREFIX; axi_host.arlen = burst_len; axi_host.arsize = 3'd6; // 64 bytes per beat @@ -206,7 +280,7 @@ module VX_cp_dma axi_host.rready = rd_rready & rd_from_host; axi_host.awvalid = wr_awvalid & wr_to_host; - axi_host.awaddr = dst_r; + axi_host.awaddr = dst_pa_r; axi_host.awid = TID_PREFIX; axi_host.awlen = burst_len; axi_host.awsize = 3'd6; @@ -218,16 +292,18 @@ module VX_cp_dma axi_host.bready = wr_bready & wr_to_host; // ----- axi_dev ----- - axi_dev.arvalid = rd_arvalid & ~rd_from_host; - axi_dev.araddr = src_r; + // In a translate state the walker owns the AR/R channel; the DMA's own + // read states never overlap with it. + axi_dev.arvalid = xlat_active ? mmu_arvalid : (rd_arvalid & ~rd_from_host); + axi_dev.araddr = xlat_active ? mmu_araddr : src_pa_r; axi_dev.arid = TID_PREFIX; - axi_dev.arlen = burst_len; + axi_dev.arlen = xlat_active ? 8'd0 : burst_len; axi_dev.arsize = 3'd6; axi_dev.arburst = 2'b01; - axi_dev.rready = rd_rready & ~rd_from_host; + axi_dev.rready = xlat_active ? mmu_rready : (rd_rready & ~rd_from_host); axi_dev.awvalid = wr_awvalid & ~wr_to_host; - axi_dev.awaddr = dst_r; + axi_dev.awaddr = dst_pa_r; axi_dev.awid = TID_PREFIX; axi_dev.awlen = burst_len; axi_dev.awsize = 3'd6; diff --git a/hw/rtl/cp/VX_cp_mmu.sv b/hw/rtl/cp/VX_cp_mmu.sv new file mode 100644 index 0000000000..8ab598e625 --- /dev/null +++ b/hw/rtl/cp/VX_cp_mmu.sv @@ -0,0 +1,231 @@ +// Copyright © 2019-2023 +// Licensed under the Apache License, Version 2.0. + +`include "VX_define.vh" + +`ifdef VX_CFG_VM_ENABLE + +// Page-table walker for the CP DMA engine: translates one device-memory +// address per request, reading PTEs over the DMA's device AXI port +// (single-beat 64 B reads issued only while the DMA FSM is parked in a +// translate state, so the two never contend). A walk that faults returns +// the address untranslated — the defensive pass-through the software CP +// model uses. A small translation cache short-circuits the walk for the +// repeated per-chunk lookups of a large transfer; it is dropped whenever +// the page table may have changed (flush, new SATP) and bypassed in BARE +// mode. Page-table geometry and PTE decoding come from VX_gpu_pkg so this +// walker and VX_mmu_ptw translate identically. +module VX_cp_mmu + import VX_gpu_pkg::*; + import VX_cp_pkg::*; +( + input wire clk, + input wire reset, + + input wire [63:0] satp, + // page tables may have changed (CACHE_FLUSH DCR): drop cached translations + input wire flush, + + input wire req_valid, + output logic req_ready, + input wire [63:0] req_vaddr, + + output logic rsp_valid, + input wire rsp_ready, + output logic [63:0] rsp_paddr, + + // Single-beat 64 B read port, muxed onto the DMA's device AXI channel. + output logic mem_arvalid, + input wire mem_arready, + output logic [63:0] mem_araddr, + input wire mem_rvalid, + output logic mem_rready, + input wire [CL_BITS-1:0] mem_rdata +); + localparam int PTE_BITS = VM_PTE_SIZE * 8; + localparam int PTE_SHIFT = `CLOG2(VM_PTE_SIZE); + localparam int PTES_PER_CL = CL_BYTES / VM_PTE_SIZE; + localparam int PTE_SEL_BITS = `CLOG2(PTES_PER_CL); + localparam int TC_ENTRIES = 2; + +`ifdef VX_CFG_XLEN_64 + wire satp_translate = (satp[63:60] != 4'd0); +`else + wire satp_translate = satp[31]; +`endif + // Root PPN field only: the MODE/ASID bits above it must not fold into + // the table address (same slice as VX_mmu). + wire [VM_PPN_WIDTH-1:0] root_ppn = satp[VM_PPN_WIDTH-1:0]; + `UNUSED_VAR (satp) + + // a new page-table root invalidates every cached translation + logic [63:0] satp_r; + wire satp_changed = (satp != satp_r); + always_ff @(posedge clk) begin + if (reset) satp_r <= '0; + else satp_r <= satp; + end + wire tc_invalidate = flush || satp_changed; + + function automatic logic [VM_VPN_LEVEL_BITS-1:0] vpn_slice( + input logic [63:0] vaddr, + input logic [VM_LEVEL_BITS-1:0] level + ); + return vaddr[VM_PAGE_OFFSET_BITS + level * VM_VPN_LEVEL_BITS +: VM_VPN_LEVEL_BITS]; + endfunction + + function automatic logic [63:0] level_mask(input logic [VM_LEVEL_BITS-1:0] level); + return ~((64'd1 << (VM_PAGE_OFFSET_BITS + level * VM_VPN_LEVEL_BITS)) - 64'd1); + endfunction + + // A superpage leaf must have its low PPN bits clear (the page offset + // covers them); anything else is a misaligned superpage and faults. + function automatic logic superpage_misaligned( + input logic [VM_PPN_WIDTH-1:0] ppn, + input logic [VM_LEVEL_BITS-1:0] level + ); + logic [VM_PPN_WIDTH-1:0] mask; + mask = (VM_PPN_WIDTH'(1) << (level * VM_VPN_LEVEL_BITS)) - VM_PPN_WIDTH'(1); + return |(ppn & mask); + endfunction + + // translation cache: {vpn, leaf level} -> frame, replaced round-robin + typedef struct packed { + logic valid; + logic [VM_LEVEL_BITS-1:0] level; + logic [VM_VPN_WIDTH-1:0] vpn; + logic [63:0] base; // leaf frame base (page-aligned PA) + } tc_entry_t; + + tc_entry_t tc [TC_ENTRIES]; + logic tc_rr; + + logic tc_hit; + logic [63:0] tc_paddr; + always_comb begin + tc_hit = 1'b0; + tc_paddr = '0; + for (int i = 0; i < TC_ENTRIES; ++i) begin + automatic logic [63:0] mask = level_mask(tc[i].level); + if (tc[i].valid && + ((64'(tc[i].vpn) << VM_PAGE_OFFSET_BITS) & mask) == (req_vaddr & mask)) begin + tc_hit = 1'b1; + tc_paddr = (tc[i].base & mask) | (req_vaddr & ~mask); + end + end + end + + typedef enum logic [1:0] { + S_IDLE, S_FETCH, S_WAIT, S_RSP + } state_e; + + state_e state; + logic [63:0] vaddr_r; + logic [VM_PPN_WIDTH-1:0] cur_ppn_r; // table PPN during the walk + logic [VM_LEVEL_BITS-1:0] level_r; + logic [63:0] paddr_r; + + wire [63:0] pte_addr = (64'(cur_ppn_r) << VM_PAGE_OFFSET_BITS) + | (64'(vpn_slice(vaddr_r, level_r)) << PTE_SHIFT); + `UNUSED_VAR (pte_addr) + + // PTE select within the 64 B line + logic [PTE_BITS-1:0] pte_w; + always_comb begin + automatic logic [PTE_SEL_BITS-1:0] sel = pte_addr[PTE_SHIFT +: PTE_SEL_BITS]; + pte_w = mem_rdata[sel * PTE_BITS +: PTE_BITS]; + end + + wire [VM_PTE_FLAGS_WIDTH-1:0] pte_flags = pte_w[VM_PTE_FLAGS_WIDTH-1:0]; + wire [VM_PPN_WIDTH-1:0] pte_ppn = pte_w[VM_PTE_PPN_LSB +: VM_PPN_WIDTH]; + // D/A/G/U flag bits and the PBMT/N high PTE bits are not enforced here + `UNUSED_VAR (pte_w) + wire pte_is_leaf = vm_pte_is_leaf(pte_flags); + wire pte_fault = !vm_pte_valid(pte_flags) + || (!pte_is_leaf && (level_r == '0)) + || (pte_is_leaf && superpage_misaligned(pte_ppn, level_r)); + + wire [63:0] leaf_mask = level_mask(level_r); + wire [63:0] leaf_base = 64'(pte_ppn) << VM_PAGE_OFFSET_BITS; + + always_ff @(posedge clk) begin + if (reset) begin + state <= S_IDLE; + tc_rr <= 1'b0; + for (int i = 0; i < TC_ENTRIES; ++i) begin + tc[i].valid <= 1'b0; + end + end else begin + if (tc_invalidate) begin + for (int i = 0; i < TC_ENTRIES; ++i) begin + tc[i].valid <= 1'b0; + end + end + case (state) + S_IDLE: begin + if (req_valid) begin + if (!satp_translate) begin + paddr_r <= req_vaddr; + state <= S_RSP; + end else if (tc_hit && !tc_invalidate) begin + paddr_r <= tc_paddr; + state <= S_RSP; + end else begin + vaddr_r <= req_vaddr; + cur_ppn_r <= root_ppn; + level_r <= VM_LEVEL_BITS'(VM_PT_LEVELS - 1); + state <= S_FETCH; + end + end + end + S_FETCH: begin + if (mem_arready) begin + state <= S_WAIT; + end + end + S_WAIT: begin + if (mem_rvalid) begin + if (pte_fault) begin + // fault: pass the address through untranslated + paddr_r <= vaddr_r; + state <= S_RSP; + end else if (pte_is_leaf) begin + paddr_r <= (leaf_base & leaf_mask) | (vaddr_r & ~leaf_mask); + // a walk that straddled an invalidation resolved against the + // old table; hand its result back but do not cache it + if (!tc_invalidate) begin + tc[tc_rr].valid <= 1'b1; + tc[tc_rr].level <= level_r; + tc[tc_rr].vpn <= vaddr_r[VM_PAGE_OFFSET_BITS +: VM_VPN_WIDTH]; + tc[tc_rr].base <= leaf_base; + tc_rr <= ~tc_rr; + end + state <= S_RSP; + end else begin + cur_ppn_r <= pte_ppn; + level_r <= level_r - VM_LEVEL_BITS'(1); + state <= S_FETCH; + end + end + end + S_RSP: begin + if (rsp_ready) begin + state <= S_IDLE; + end + end + default: state <= S_IDLE; + endcase + end + end + + assign req_ready = (state == S_IDLE); + assign rsp_valid = (state == S_RSP); + assign rsp_paddr = paddr_r; + assign mem_arvalid = (state == S_FETCH); + // 64 B-aligned line fetch; the PTE is selected from the returned line. + assign mem_araddr = {pte_addr[63:6], 6'd0}; + assign mem_rready = (state == S_WAIT); + +endmodule + +`endif diff --git a/hw/rtl/cp/VX_cp_pkg.sv b/hw/rtl/cp/VX_cp_pkg.sv index a7f1d1ae70..cbb79a87fc 100644 --- a/hw/rtl/cp/VX_cp_pkg.sv +++ b/hw/rtl/cp/VX_cp_pkg.sv @@ -78,8 +78,10 @@ package VX_cp_pkg; // Header flag bits. // ------------------------------------------------------------------------ - localparam int F_PROFILE = 0; - localparam int F_FENCE_PRE = 1; + localparam int F_PROFILE = 0; + localparam int F_FENCE_PRE = 1; + // CMD_MEM_* device-side operand is a physical address (skip translation). + localparam int F_MEM_PHYSICAL = 2; typedef struct packed { logic [15:0] reserved; diff --git a/hw/rtl/mem/VX_mmu.sv b/hw/rtl/mem/VX_mmu.sv index 4cb2b7ea06..c5250fe4db 100644 --- a/hw/rtl/mem/VX_mmu.sv +++ b/hw/rtl/mem/VX_mmu.sv @@ -1,27 +1,32 @@ // Copyright 2024 -// MMU: TLB + PTW for VA→PA translation +// MMU: per-core TLB in front of a cache port; misses are walked by the +// device-level shared page-table walker over the VX_ptw_bus_if hierarchy. `include "VX_define.vh" module VX_mmu import VX_gpu_pkg::*; #( parameter NUM_REQS = DCACHE_NUM_REQS, + parameter NUM_BANKS = 1, parameter DATA_SIZE = DCACHE_WORD_SIZE, parameter TAG_WIDTH = DCACHE_TAG_WIDTH_BASE, parameter ADDR_WIDTH = `VX_CFG_MEM_ADDR_WIDTH - `CLOG2(DATA_SIZE), - parameter ATTR_WIDTH = MEM_ATTR_WIDTH, + parameter ATTR_WIDTH = MEM_ATTR_WIDTH, parameter EBUF_SIZE = 2 ) ( input wire clk, input wire reset, + input wire flush, `ifdef PERF_ENABLE output mmu_perf_t mmu_perf, `endif - input wire [31:0] satp, + input wire [`VX_CFG_XLEN-1:0] satp, VX_mem_bus_if.slave lsu_mem_if [NUM_REQS], - VX_mem_bus_if.master dcache_mem_if [NUM_REQS] + VX_mem_bus_if.master dcache_mem_if [NUM_REQS], + + VX_ptw_bus_if.master ptw_bus_if ); // ========================================================================= @@ -35,12 +40,23 @@ module VX_mmu import VX_gpu_pkg::*; #( // issued in BARE mode (SATP MSB cleared), which covers the few // instruction fetches between reset and the kernel's csrw satp. - function automatic logic needs_translation(input logic [31:0] full_addr); - // full_addr currently not consumed — only satp[31] gates translation. + // Only the SATP mode field gates translation: Sv32 keeps it in bit 31, + // Sv39/Sv48/Sv57 in bits 63:60 (mode 0 is BARE in both encodings). +`ifdef VX_CFG_XLEN_64 + wire satp_translate = (satp[`VX_CFG_XLEN-1:`VX_CFG_XLEN-4] != 4'd0); +`else + wire satp_translate = satp[`VX_CFG_XLEN-1]; +`endif + + // The root PPN rides along with every walk request; ASID is not used. + wire [VM_PPN_WIDTH-1:0] satp_root_ppn = satp[VM_PPN_WIDTH-1:0]; + `UNUSED_VAR (satp) + + function automatic logic needs_translation(input logic [`VX_CFG_MEM_ADDR_WIDTH-1:0] full_addr); + // full_addr currently not consumed — only the SATP mode gates translation. // Kept as a port to anticipate range-based bypass policies. `UNUSED_VAR (full_addr) - if (!satp[31]) return 1'b0; // BARE mode - return 1'b1; + return satp_translate; endfunction // ========================================================================= @@ -75,24 +91,12 @@ module VX_mmu import VX_gpu_pkg::*; #( .ATTR_WIDTH (ATTR_WIDTH) ) bypass_dcache_if[NUM_REQS](); + // [0..NUM_REQS-1]=bypass, [NUM_REQS..2*NUM_REQS-1]=TLB VX_mem_bus_if #( .DATA_SIZE (DATA_SIZE), .TAG_WIDTH (TAG_WIDTH_TLB), .ATTR_WIDTH (ATTR_WIDTH) - ) ptw_mem_if(); - -`ifdef PERF_ENABLE - mmu_perf_t mmu_perf_tlb; - `UNUSED_VAR (mmu_perf_tlb) - wire [PERF_CTR_BITS-1:0] ptw_latency_counter; -`endif - - // [0..NUM_REQS-1]=bypass, [NUM_REQS..2*NUM_REQS-1]=TLB, [2*NUM_REQS]=PTW - VX_mem_bus_if #( - .DATA_SIZE (DATA_SIZE), - .TAG_WIDTH (TAG_WIDTH_TLB), - .ATTR_WIDTH (ATTR_WIDTH) - ) merge_in_if[2 * NUM_REQS + 1](); + ) merge_in_if[2 * NUM_REQS](); // ========================================================================= // Elastic Buffers (TLB path only) @@ -107,7 +111,7 @@ module VX_mmu import VX_gpu_pkg::*; #( for (genvar i = 0; i < NUM_REQS; i++) begin : g_elastic_buffers - wire [31:0] full_addr_ebuf = {lsu_mem_if[i].req_data.addr, {`CLOG2(DATA_SIZE){1'b0}}}; + wire [`VX_CFG_MEM_ADDR_WIDTH-1:0] full_addr_ebuf = {lsu_mem_if[i].req_data.addr, {`CLOG2(DATA_SIZE){1'b0}}}; assign lane_needs_trans_ebuf[i] = needs_translation(full_addr_ebuf); wire [REQ_DATAW-1:0] req_data_in_packed; @@ -172,76 +176,31 @@ module VX_mmu import VX_gpu_pkg::*; #( end - // ========================================================================= - // TLB Miss/Fill Interface - // ========================================================================= - - wire tlb_miss_valid; - wire tlb_miss_ready; - wire [31:0] tlb_miss_vaddr; - - wire tlb_fill_valid; - wire tlb_fill_ready; - wire [31:0] tlb_fill_vaddr; - wire [31:0] tlb_fill_paddr; - wire [7:0] tlb_fill_flags; - // ========================================================================= // TLB Module // ========================================================================= VX_mmu_tlb #( .NUM_REQS (NUM_REQS), + .NUM_BANKS (NUM_BANKS), .DATA_SIZE (DATA_SIZE), .TAG_WIDTH_IN (TAG_WIDTH), .TAG_WIDTH_OUT (TAG_WIDTH_TLB), .ADDR_WIDTH (ADDR_WIDTH), - .ATTR_WIDTH (ATTR_WIDTH) + .ATTR_WIDTH (ATTR_WIDTH) ) tlb_unit ( .clk (clk), .reset (reset), + .flush (flush), + .root_ppn (satp_root_ppn), `ifdef PERF_ENABLE - .mmu_perf (mmu_perf_tlb). + .mmu_perf (mmu_perf), + `else + `UNUSED_PIN (mmu_perf_placeholder), `endif .tlb_in_if (buffered_if), .tlb_out_if (tlb_out_if), - .miss_valid (tlb_miss_valid), - .miss_ready (tlb_miss_ready), - .miss_vaddr (tlb_miss_vaddr), - .fill_valid (tlb_fill_valid), - .fill_ready (tlb_fill_ready), - .fill_vaddr (tlb_fill_vaddr), - .fill_paddr (tlb_fill_paddr), - .fill_flags (tlb_fill_flags) - ); - - // ========================================================================= - // PTW Module - // ========================================================================= - - VX_mmu_ptw #( - .DATA_SIZE (DATA_SIZE), - .TAG_WIDTH (TAG_WIDTH_TLB), - .ADDR_WIDTH (ADDR_WIDTH), - .ATTR_WIDTH (ATTR_WIDTH) - ) ptw_unit ( - .clk (clk), - .reset (reset), - .satp (satp), - .miss_valid (tlb_miss_valid), - .miss_ready (tlb_miss_ready), - .miss_vaddr (tlb_miss_vaddr), - .fill_valid (tlb_fill_valid), - .fill_ready (tlb_fill_ready), - .fill_vaddr (tlb_fill_vaddr), - .fill_paddr (tlb_fill_paddr), - .fill_flags (tlb_fill_flags), - .ptw_mem_if (ptw_mem_if), - `ifdef PERF_ENABLE - .perf_ptw_latency (ptw_latency_counter) - `else - `UNUSED_PIN (perf_ptw_latency_placeholder) - `endif + .ptw_bus_if (ptw_bus_if) ); // ========================================================================= @@ -252,7 +211,7 @@ module VX_mmu import VX_gpu_pkg::*; #( wire [NUM_REQS-1:0] lane_bypass; for (genvar i = 0; i < NUM_REQS; i++) begin : g_bypass_path - wire [31:0] full_addr = {lsu_mem_if[i].req_data.addr, {`CLOG2(DATA_SIZE){1'b0}}}; + wire [`VX_CFG_MEM_ADDR_WIDTH-1:0] full_addr = {lsu_mem_if[i].req_data.addr, {`CLOG2(DATA_SIZE){1'b0}}}; assign lane_needs_trans[i] = needs_translation(full_addr); assign lane_bypass[i] = ~lane_needs_trans[i]; @@ -291,19 +250,12 @@ module VX_mmu import VX_gpu_pkg::*; #( assign merge_in_if[NUM_REQS + i].rsp_ready = tlb_out_if[i].rsp_ready; end - assign merge_in_if[2 * NUM_REQS].req_valid = ptw_mem_if.req_valid; - assign merge_in_if[2 * NUM_REQS].req_data = ptw_mem_if.req_data; - assign ptw_mem_if.req_ready = merge_in_if[2 * NUM_REQS].req_ready; - assign ptw_mem_if.rsp_valid = merge_in_if[2 * NUM_REQS].rsp_valid; - assign ptw_mem_if.rsp_data = merge_in_if[2 * NUM_REQS].rsp_data; - assign merge_in_if[2 * NUM_REQS].rsp_ready = ptw_mem_if.rsp_ready; - // ========================================================================= // Merge Arbiter // ========================================================================= VX_mem_bus_arb #( - .NUM_INPUTS (2 * NUM_REQS + 1), + .NUM_INPUTS (2 * NUM_REQS), .NUM_OUTPUTS (NUM_REQS), .DATA_SIZE (DATA_SIZE), .TAG_WIDTH (TAG_WIDTH_TLB), @@ -376,17 +328,4 @@ module VX_mmu import VX_gpu_pkg::*; #( end - // ========================================================================= - // Performance Counters - // ========================================================================= - -`ifdef PERF_ENABLE - assign mmu_perf.tlb_reads = mmu_perf_tlb.tlb_reads; - assign mmu_perf.tlb_hits = mmu_perf_tlb.tlb_hits; - assign mmu_perf.tlb_misses = mmu_perf_tlb.tlb_misses; - assign mmu_perf.tlb_evictions = mmu_perf_tlb.tlb_evictions; - assign mmu_perf.ptw_walks = mmu_perf_tlb.ptw_walks; - assign mmu_perf.ptw_latency = ptw_latency_counter; -`endif - endmodule diff --git a/hw/rtl/mem/VX_mmu_ptw.sv b/hw/rtl/mem/VX_mmu_ptw.sv index 1b9d4432b4..f760a3aaf7 100644 --- a/hw/rtl/mem/VX_mmu_ptw.sv +++ b/hw/rtl/mem/VX_mmu_ptw.sv @@ -1,203 +1,414 @@ -// Copyright 2024 -// PTW: SV32 page table walker +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. `include "VX_define.vh" +// Shared page-table walker: one instance per device, fed by every core TLB +// through the VX_ptw_bus_if hierarchy. Up to NUM_WALKERS walks proceed +// concurrently, each identified by the bus tag of the requesting TLB bank. +// Page-table entries are fetched on a dedicated L3 port. Non-leaf entries of +// the upper levels are cached in page-walk caches so that a walk can start +// one (Sv32) or two (Sv39) levels below the root. module VX_mmu_ptw import VX_gpu_pkg::*; #( - parameter DATA_SIZE = DCACHE_WORD_SIZE, - parameter TAG_WIDTH = DCACHE_TAG_WIDTH + `UP(`CLOG2(DCACHE_NUM_REQS)), - parameter ADDR_WIDTH = DCACHE_ADDR_WIDTH, - parameter ATTR_WIDTH = MEM_ATTR_WIDTH + parameter NUM_WALKERS = `VX_CFG_PTW_NUM_WALKERS, + parameter PWC_SIZE = `VX_CFG_PTW_WALK_CACHE_SIZE, + parameter TAG_WIDTH = PTW_DEV_TAG_WIDTH, + parameter MEM_DATA_SIZE = L3_WORD_SIZE, + parameter MEM_TAG_WIDTH = L3_TAG_WIDTH ) ( input wire clk, input wire reset, - - input wire [31:0] satp, - - input wire miss_valid, - output wire miss_ready, - input wire [31:0] miss_vaddr, - - output wire fill_valid, - input wire fill_ready, - output wire [31:0] fill_vaddr, - output wire [31:0] fill_paddr, - output wire [7:0] fill_flags, - - VX_mem_bus_if.master ptw_mem_if, + input wire flush, `ifdef PERF_ENABLE - output wire [PERF_CTR_BITS-1:0] perf_ptw_latency -`else - output wire perf_ptw_latency_placeholder + output ptw_perf_t ptw_perf, `endif + + VX_ptw_bus_if.slave ptw_bus_if, + VX_mem_bus_if.master mem_bus_if ); + `STATIC_ASSERT(`IS_POW2(NUM_WALKERS), ("NUM_WALKERS must be a power of 2")) + `STATIC_ASSERT((VM_PT_LEVELS == 2) || (VM_PT_LEVELS == 3), ("only Sv32 and Sv39 page tables are supported")) + `STATIC_ASSERT(MEM_DATA_SIZE >= VM_PTE_SIZE, ("memory word must hold a PTE")) + + localparam SLOT_BITS = `LOG2UP(NUM_WALKERS); + localparam MEM_ADDR_WIDTH = `VX_CFG_MEM_ADDR_WIDTH - `CLOG2(MEM_DATA_SIZE); + localparam PTE_SHIFT = `CLOG2(VM_PTE_SIZE); + localparam PTE_BITS = VM_PTE_SIZE * 8; + localparam PTES_PER_WORD = MEM_DATA_SIZE / VM_PTE_SIZE; + localparam PTE_SEL_BITS = `LOG2UP(PTES_PER_WORD); + localparam PWC_KEY_WIDTH = VM_PPN_WIDTH + VM_VPN_LEVEL_BITS; + localparam TOP_LEVEL = VM_PT_LEVELS - 1; + + `STATIC_ASSERT(SLOT_BITS <= (MEM_TAG_WIDTH - UUID_WIDTH), ("walker id does not fit the L3 tag")) + + typedef enum logic [1:0] { + SLOT_IDLE = 2'd0, + SLOT_MEM_REQ = 2'd1, + SLOT_MEM_RSP = 2'd2, + SLOT_DONE = 2'd3 + } slot_state_t; + + typedef struct packed { + logic [VM_VPN_WIDTH-1:0] vpn; + logic [VM_PPN_WIDTH-1:0] root_ppn; + logic [VM_PPN_WIDTH-1:0] cur_ppn; // table being walked, then the leaf PPN + logic [VM_LEVEL_BITS-1:0] level; + logic [VM_PTE_FLAGS_WIDTH-1:0] flags; + logic fault; + logic [`UP(PTE_SEL_BITS)-1:0] pte_sel; // PTE index within the fetched word + logic [TAG_WIDTH-1:0] tag; + logic stale; // started before the last flush: no PWC fills + } slot_t; + + slot_state_t slot_state [NUM_WALKERS]; + slot_t slots [NUM_WALKERS]; + + wire [NUM_WALKERS-1:0] slot_idle, slot_mem_req, slot_done; + for (genvar s = 0; s < NUM_WALKERS; ++s) begin : g_slot_flags + assign slot_idle[s] = (slot_state[s] == SLOT_IDLE); + assign slot_mem_req[s] = (slot_state[s] == SLOT_MEM_REQ); + assign slot_done[s] = (slot_state[s] == SLOT_DONE); + end - // TAG_WIDTH/ATTR_WIDTH are part of the VX_mem_bus_if parameter - // signature; PTW issues with tag='0 and attr='0 and doesn't observe - // either on responses, so neither parameter is read here. - `UNUSED_PARAM (TAG_WIDTH) - `UNUSED_PARAM (ATTR_WIDTH) - - // SV32: PPN occupies satp[21:0] (Sv32 caps PPN at 22 bits); top bits - // 31:20 carry ASID + MODE — not consumed by the walker. - `UNUSED_VAR (satp[31:20]) - - localparam DATA_WIDTH = DATA_SIZE * 8; - - // SV32 parameters - localparam VPN_WIDTH = 20; - localparam PPN_WIDTH = VPN_WIDTH; - localparam PAGE_OFFSET_BITS = 12; - localparam VPN_LEVEL_BITS = 10; - localparam PTE_SIZE_BYTES = 4; - localparam PTE_SHIFT = `CLOG2(PTE_SIZE_BYTES); - - // State machine - typedef enum logic [2:0] { - PTW_IDLE = 3'd0, - PTW_L1_REQ = 3'd1, - PTW_L1_RESP = 3'd2, - PTW_L0_REQ = 3'd3, - PTW_L0_RESP = 3'd4, - PTW_FILL = 3'd5 - } ptw_state_t; - - ptw_state_t state, state_next; - - // PTW registers - reg [31:0] pending_vaddr; - reg [PPN_WIDTH-1:0] l1_ppn; - reg [PPN_WIDTH-1:0] final_ppn; - reg [7:0] final_flags; - reg [31:0] req_pte_addr_r; - - // VPN extraction: SV32 [31:22]=vpn1, [21:12]=vpn0, [11:0]=offset - wire [VPN_LEVEL_BITS-1:0] vpn1 = pending_vaddr[31:22]; - wire [VPN_LEVEL_BITS-1:0] vpn0 = pending_vaddr[21:12]; - - // PTE address: (base_ppn << 12) + (vpn << 2) - wire [31:0] l1_pte_addr = {satp[PPN_WIDTH-1:0], {PAGE_OFFSET_BITS{1'b0}}} + - {{(32-VPN_LEVEL_BITS-PTE_SHIFT){1'b0}}, vpn1, {PTE_SHIFT{1'b0}}}; - wire [31:0] l0_pte_addr = {l1_ppn, {PAGE_OFFSET_BITS{1'b0}}} + - {{(32-VPN_LEVEL_BITS-PTE_SHIFT){1'b0}}, vpn0, {PTE_SHIFT{1'b0}}}; - - // PTE parsing: [29:10]=PPN, [7:0]=flags - localparam NUM_WORDS = DATA_SIZE / 4; - localparam SEL_BITS = `CLOG2(NUM_WORDS); - - wire [DATA_WIDTH-1:0] rsp_data_full = ptw_mem_if.rsp_data.data; - - // Extract 32-bit PTE from cache line using registered address - wire [31:0] pte_data; - if (NUM_WORDS > 1) begin : g_pte_select - wire [SEL_BITS-1:0] word_sel = req_pte_addr_r[SEL_BITS+1:2]; - assign pte_data = rsp_data_full[word_sel * 32 +: 32]; - // Only the word-selector slice of req_pte_addr_r is consumed; the - // low two bits are byte-within-word and the high bits live above - // the cache-line size. - `UNUSED_VAR (req_pte_addr_r[1:0]) - `UNUSED_VAR (req_pte_addr_r[31:SEL_BITS+2]) - end else begin : g_pte_direct - assign pte_data = rsp_data_full[31:0]; - `UNUSED_VAR (req_pte_addr_r) + function automatic logic [VM_VPN_LEVEL_BITS-1:0] vpn_slice( + input logic [VM_VPN_WIDTH-1:0] vpn, + input logic [VM_LEVEL_BITS-1:0] level + ); + return vpn[level * VM_VPN_LEVEL_BITS +: VM_VPN_LEVEL_BITS]; + endfunction + + // A superpage leaf must have its low PPN bits clear (the page offset + // covers them); anything else is a misaligned superpage and faults. + function automatic logic superpage_misaligned( + input logic [VM_PPN_WIDTH-1:0] ppn, + input logic [VM_LEVEL_BITS-1:0] level + ); + logic [VM_PPN_WIDTH-1:0] mask; + mask = (VM_PPN_WIDTH'(1) << (level * VM_VPN_LEVEL_BITS)) - VM_PPN_WIDTH'(1); + return |(ppn & mask); + endfunction + + // ------------------------------------------------------------------------- + // Walk cache lookup on the incoming request + // ------------------------------------------------------------------------- + + wire [VM_VPN_WIDTH-1:0] req_vpn = ptw_bus_if.req_data.vpn; + wire [VM_PPN_WIDTH-1:0] req_root_ppn = ptw_bus_if.req_data.root_ppn; + + wire pwc1_hit; + wire [VM_PPN_WIDTH-1:0] pwc1_data; + wire pwc1_fill_valid; + wire [PWC_KEY_WIDTH-1:0] pwc1_fill_key; + wire [VM_PPN_WIDTH-1:0] pwc1_fill_data; + + VX_mmu_pwc #( + .KEY_WIDTH (PWC_KEY_WIDTH), + .DATA_WIDTH (VM_PPN_WIDTH), + .NUM_ENTRIES (PWC_SIZE) + ) pwc1 ( + .clk (clk), + .reset (reset), + .flush (flush), + .lookup_key ({req_root_ppn, vpn_slice(req_vpn, VM_LEVEL_BITS'(TOP_LEVEL))}), + .lookup_hit (pwc1_hit), + .lookup_data (pwc1_data), + .fill_valid (pwc1_fill_valid), + .fill_key (pwc1_fill_key), + .fill_data (pwc1_fill_data) + ); + + wire pwc2_hit; + wire [VM_PPN_WIDTH-1:0] pwc2_data; + wire pwc2_fill_valid; + wire [PWC_KEY_WIDTH-1:0] pwc2_fill_key; + wire [VM_PPN_WIDTH-1:0] pwc2_fill_data; + + if (VM_PT_LEVELS == 3) begin : g_pwc2 + // Level-1 tables are keyed by the level-2 table the first cache + // returned, so a double hit skips both upper fetches. + VX_mmu_pwc #( + .KEY_WIDTH (PWC_KEY_WIDTH), + .DATA_WIDTH (VM_PPN_WIDTH), + .NUM_ENTRIES (PWC_SIZE) + ) pwc2 ( + .clk (clk), + .reset (reset), + .flush (flush), + .lookup_key ({pwc1_data, vpn_slice(req_vpn, VM_LEVEL_BITS'(1))}), + .lookup_hit (pwc2_hit), + .lookup_data (pwc2_data), + .fill_valid (pwc2_fill_valid), + .fill_key (pwc2_fill_key), + .fill_data (pwc2_fill_data) + ); + end else begin : g_no_pwc2 + `UNUSED_VAR (pwc2_fill_valid) + `UNUSED_VAR (pwc2_fill_key) + `UNUSED_VAR (pwc2_fill_data) + assign pwc2_hit = 1'b0; + assign pwc2_data = '0; end - wire [PPN_WIDTH-1:0] pte_ppn = pte_data[29:10]; - wire [7:0] pte_flags = pte_data[7:0]; - // pte_data[31:30] are SV32 "reserved for SW" + N (not modelled); [9:8] - // is RSW (reserved for SW). The walker doesn't act on them. - `UNUSED_VAR (pte_data[31:30]) - `UNUSED_VAR (pte_data[9:8]) - - // PTE flag fields are parsed but not yet acted upon; fault handling is not implemented. - wire pte_valid = pte_flags[0]; - wire pte_invalid_combo = ~pte_flags[1] & pte_flags[2]; - wire pte_is_leaf = pte_flags[1] | pte_flags[2] | pte_flags[3]; - `UNUSED_VAR (pte_valid) - `UNUSED_VAR (pte_invalid_combo) - `UNUSED_VAR (pte_is_leaf) - - // State machine - wire mem_req_fire = ptw_mem_if.req_valid && ptw_mem_if.req_ready; - wire mem_rsp_fire = ptw_mem_if.rsp_valid && ptw_mem_if.rsp_ready; - - always_ff @(posedge clk) begin + wire start_skip2 = pwc1_hit && pwc2_hit; + wire start_skip1 = pwc1_hit && !pwc2_hit; + + wire [VM_LEVEL_BITS-1:0] start_level = start_skip2 ? VM_LEVEL_BITS'(TOP_LEVEL - 2) : + start_skip1 ? VM_LEVEL_BITS'(TOP_LEVEL - 1) : + VM_LEVEL_BITS'(TOP_LEVEL); + wire [VM_PPN_WIDTH-1:0] start_ppn = start_skip2 ? pwc2_data : + start_skip1 ? pwc1_data : + req_root_ppn; + + // ------------------------------------------------------------------------- + // Slot allocation + // ------------------------------------------------------------------------- + + wire [SLOT_BITS-1:0] free_slot; + wire free_valid; + + VX_priority_encoder #( + .N (NUM_WALKERS) + ) free_slot_enc ( + .data_in (slot_idle), + .index_out (free_slot), + .valid_out (free_valid), + `UNUSED_PIN (onehot_out) + ); + + assign ptw_bus_if.req_ready = free_valid; + wire req_fire = ptw_bus_if.req_valid && ptw_bus_if.req_ready; + + // ------------------------------------------------------------------------- + // Memory requests: one outstanding fetch per slot, round-robin issue + // ------------------------------------------------------------------------- + + wire [SLOT_BITS-1:0] mem_slot; + wire mem_slot_valid; + + VX_rr_arbiter #( + .NUM_REQS (NUM_WALKERS) + ) mem_arb ( + .clk (clk), + .reset (reset), + .requests (slot_mem_req), + .grant_index (mem_slot), + .grant_valid (mem_slot_valid), + .grant_ready (mem_bus_if.req_ready), + `UNUSED_PIN (grant_onehot) + ); + + wire [VM_VPN_LEVEL_BITS-1:0] mem_vpn_slice = vpn_slice(slots[mem_slot].vpn, slots[mem_slot].level); + wire [`VX_CFG_MEM_ADDR_WIDTH-1:0] mem_pte_addr = {slots[mem_slot].cur_ppn, {VM_PAGE_OFFSET_BITS{1'b0}}} + | `VX_CFG_MEM_ADDR_WIDTH'({mem_vpn_slice, {PTE_SHIFT{1'b0}}}); + + assign mem_bus_if.req_valid = mem_slot_valid; + assign mem_bus_if.req_data.rw = 1'b0; + assign mem_bus_if.req_data.addr = mem_pte_addr[`VX_CFG_MEM_ADDR_WIDTH-1 -: MEM_ADDR_WIDTH]; + `UNUSED_VAR (mem_pte_addr) + assign mem_bus_if.req_data.data = '0; + assign mem_bus_if.req_data.byteen = {MEM_DATA_SIZE{1'b1}}; + assign mem_bus_if.req_data.attr = '0; + assign mem_bus_if.req_data.tag = MEM_TAG_WIDTH'(mem_slot); + + wire mem_req_fire = mem_bus_if.req_valid && mem_bus_if.req_ready; + + // ------------------------------------------------------------------------- + // Memory responses: decode the PTE addressed by the slot + // ------------------------------------------------------------------------- + + wire [SLOT_BITS-1:0] rsp_slot = SLOT_BITS'(mem_bus_if.rsp_data.tag); + wire mem_rsp_fire = mem_bus_if.rsp_valid && mem_bus_if.rsp_ready; + // A slot that issued a fetch is always waiting for it. + assign mem_bus_if.rsp_ready = 1'b1; + + wire [PTE_BITS-1:0] rsp_pte; + if (PTES_PER_WORD > 1) begin : g_pte_select + assign rsp_pte = mem_bus_if.rsp_data.data[slots[rsp_slot].pte_sel * PTE_BITS +: PTE_BITS]; + end else begin : g_pte_single + assign rsp_pte = mem_bus_if.rsp_data.data[PTE_BITS-1:0]; + end + + wire [VM_PTE_FLAGS_WIDTH-1:0] rsp_flags = rsp_pte[VM_PTE_FLAGS_WIDTH-1:0]; + wire [VM_PPN_WIDTH-1:0] rsp_ppn = rsp_pte[VM_PTE_PPN_LSB +: VM_PPN_WIDTH]; + `UNUSED_VAR (rsp_pte) + + wire [VM_LEVEL_BITS-1:0] rsp_level = slots[rsp_slot].level; + wire rsp_is_leaf = vm_pte_is_leaf(rsp_flags); + wire rsp_fault = !vm_pte_valid(rsp_flags) + || (!rsp_is_leaf && (rsp_level == '0)) + || (rsp_is_leaf && superpage_misaligned(rsp_ppn, rsp_level)); + wire rsp_descend = !rsp_fault && !rsp_is_leaf; + + wire rsp_cacheable = !slots[rsp_slot].stale; + assign pwc1_fill_valid = mem_rsp_fire && rsp_cacheable && rsp_descend && (rsp_level == VM_LEVEL_BITS'(TOP_LEVEL)); + assign pwc1_fill_key = {slots[rsp_slot].root_ppn, vpn_slice(slots[rsp_slot].vpn, rsp_level)}; + assign pwc1_fill_data = rsp_ppn; + + assign pwc2_fill_valid = mem_rsp_fire && rsp_cacheable && rsp_descend && (rsp_level == VM_LEVEL_BITS'(1)) && (VM_PT_LEVELS == 3); + assign pwc2_fill_key = {slots[rsp_slot].cur_ppn, vpn_slice(slots[rsp_slot].vpn, rsp_level)}; + assign pwc2_fill_data = rsp_ppn; + + // ------------------------------------------------------------------------- + // Completion: hand finished walks back in round-robin order + // ------------------------------------------------------------------------- + + wire [SLOT_BITS-1:0] done_slot; + wire done_valid; + + VX_rr_arbiter #( + .NUM_REQS (NUM_WALKERS) + ) done_arb ( + .clk (clk), + .reset (reset), + .requests (slot_done), + .grant_index (done_slot), + .grant_valid (done_valid), + .grant_ready (ptw_bus_if.rsp_ready), + `UNUSED_PIN (grant_onehot) + ); + + assign ptw_bus_if.rsp_valid = done_valid; + assign ptw_bus_if.rsp_data.ppn = slots[done_slot].cur_ppn; + assign ptw_bus_if.rsp_data.level = slots[done_slot].level; + assign ptw_bus_if.rsp_data.flags = slots[done_slot].flags; + assign ptw_bus_if.rsp_data.fault = slots[done_slot].fault; + assign ptw_bus_if.rsp_data.tag = slots[done_slot].tag; + + wire rsp_fire = ptw_bus_if.rsp_valid && ptw_bus_if.rsp_ready; + + // ------------------------------------------------------------------------- + // Slot state + // ------------------------------------------------------------------------- + + always @(posedge clk) begin if (reset) begin - state <= PTW_IDLE; - pending_vaddr <= 32'b0; - l1_ppn <= 20'b0; - final_ppn <= 20'b0; - final_flags <= 8'b0; - req_pte_addr_r <= 32'b0; + for (integer s = 0; s < NUM_WALKERS; ++s) begin + slot_state[s] <= SLOT_IDLE; + end end else begin - state <= state_next; - - case (state) - PTW_IDLE: if (miss_valid && miss_ready) pending_vaddr <= miss_vaddr; - PTW_L1_REQ: if (mem_req_fire) req_pte_addr_r <= l1_pte_addr; - PTW_L0_REQ: if (mem_req_fire) req_pte_addr_r <= l0_pte_addr; - PTW_L1_RESP: if (mem_rsp_fire) l1_ppn <= pte_ppn; - PTW_L0_RESP: if (mem_rsp_fire) begin - final_ppn <= pte_ppn; - final_flags <= pte_flags; + if (req_fire) begin + slot_state[free_slot] <= SLOT_MEM_REQ; + slots[free_slot].vpn <= req_vpn; + slots[free_slot].root_ppn <= req_root_ppn; + slots[free_slot].cur_ppn <= start_ppn; + slots[free_slot].level <= start_level; + slots[free_slot].flags <= '0; + slots[free_slot].fault <= 1'b0; + slots[free_slot].tag <= ptw_bus_if.req_data.tag; + slots[free_slot].stale <= 1'b0; + end + if (flush) begin + // PTEs fetched by walks already in flight may predate the + // page-table update this flush publishes; keep them out of + // the caches just emptied (the TLB bank drops the result). + for (integer s = 0; s < NUM_WALKERS; ++s) begin + if (slot_state[s] != SLOT_IDLE) begin + slots[s].stale <= 1'b1; + end end - default: ; - endcase + end + if (mem_req_fire) begin + slot_state[mem_slot] <= SLOT_MEM_RSP; + slots[mem_slot].pte_sel <= mem_pte_addr[PTE_SHIFT +: `UP(PTE_SEL_BITS)]; + end + if (mem_rsp_fire) begin + if (rsp_descend) begin + slot_state[rsp_slot] <= SLOT_MEM_REQ; + slots[rsp_slot].cur_ppn <= rsp_ppn; + slots[rsp_slot].level <= rsp_level - VM_LEVEL_BITS'(1); + end else begin + slot_state[rsp_slot] <= SLOT_DONE; + slots[rsp_slot].cur_ppn <= rsp_ppn; + slots[rsp_slot].flags <= rsp_flags; + slots[rsp_slot].fault <= rsp_fault; + end + end + if (rsp_fire) begin + slot_state[done_slot] <= SLOT_IDLE; + end end end - always_comb begin - state_next = state; - case (state) - PTW_IDLE: if (miss_valid && miss_ready) state_next = PTW_L1_REQ; - PTW_L1_REQ: if (mem_req_fire) state_next = PTW_L1_RESP; - PTW_L1_RESP: if (mem_rsp_fire) state_next = PTW_L0_REQ; - PTW_L0_REQ: if (mem_req_fire) state_next = PTW_L0_RESP; - PTW_L0_RESP: if (mem_rsp_fire) state_next = PTW_FILL; - PTW_FILL: if (fill_valid && fill_ready) state_next = PTW_IDLE; - default: state_next = PTW_IDLE; - endcase +`ifdef SIMULATION + always @(posedge clk) begin + if (!reset && mem_rsp_fire && rsp_fault) begin + `ERROR(("%t: *** %s page fault: vpn=0x%0h level=%0d pte=0x%0h", $time, "ptw", slots[rsp_slot].vpn, rsp_level, rsp_pte)); + end end +`endif - // TLB interface - assign miss_ready = (state == PTW_IDLE); - assign fill_valid = (state == PTW_FILL); - assign fill_vaddr = pending_vaddr; - assign fill_paddr = {final_ppn, pending_vaddr[PAGE_OFFSET_BITS-1:0]}; - assign fill_flags = final_flags; - - // Memory interface - wire [31:0] pte_addr = (state == PTW_L1_REQ) ? l1_pte_addr : l0_pte_addr; - localparam ADDR_SHIFT = `CLOG2(DATA_SIZE); - wire [ADDR_WIDTH-1:0] pte_word_addr = pte_addr[31:ADDR_SHIFT]; - `UNUSED_VAR (pte_addr[ADDR_SHIFT-1:0]) - - assign ptw_mem_if.req_valid = (state == PTW_L1_REQ) || (state == PTW_L0_REQ); - assign ptw_mem_if.req_data.rw = 1'b0; - assign ptw_mem_if.req_data.addr = pte_word_addr; - assign ptw_mem_if.req_data.data = '0; - assign ptw_mem_if.req_data.byteen = {DATA_SIZE{1'b1}}; - assign ptw_mem_if.req_data.attr = '0; - assign ptw_mem_if.req_data.tag = '0; - assign ptw_mem_if.rsp_ready = (state == PTW_L1_RESP) || (state == PTW_L0_RESP); +`ifdef DBG_TRACE_MMU + always @(posedge clk) begin + if (req_fire) begin + `TRACE(2, ("%t: ptw-req: slot=%0d, vpn=0x%0h, root=0x%0h, level=%0d, tag=0x%0h\n", $time, free_slot, req_vpn, req_root_ppn, start_level, ptw_bus_if.req_data.tag)) + end + if (mem_req_fire) begin + `TRACE(2, ("%t: ptw-mem-req: slot=%0d, addr=0x%0h, level=%0d\n", $time, mem_slot, mem_pte_addr, slots[mem_slot].level)) + end + if (mem_rsp_fire) begin + `TRACE(2, ("%t: ptw-mem-rsp: slot=%0d, pte=0x%0h, leaf=%b, fault=%b\n", $time, rsp_slot, rsp_pte, rsp_is_leaf, rsp_fault)) + end + if (rsp_fire) begin + `TRACE(2, ("%t: ptw-rsp: slot=%0d, ppn=0x%0h, level=%0d, fault=%b, tag=0x%0h\n", $time, done_slot, slots[done_slot].cur_ppn, slots[done_slot].level, slots[done_slot].fault, slots[done_slot].tag)) + end + end +`endif + // ------------------------------------------------------------------------- // Performance counters + // ------------------------------------------------------------------------- + `ifdef PERF_ENABLE - reg [PERF_CTR_BITS-1:0] perf_ptw_latency_r; - wire ptw_active = (state != PTW_IDLE); + wire [NUM_WALKERS-1:0] slot_active = ~slot_idle; + wire [`CLOG2(NUM_WALKERS+1)-1:0] active_count; + `POP_COUNT(active_count, slot_active); + + reg [PERF_CTR_BITS-1:0] perf_walks_r, perf_latency_r; + reg [PERF_CTR_BITS-1:0] perf_pwc1_hits_r, perf_pwc1_misses_r; + reg [PERF_CTR_BITS-1:0] perf_pwc2_hits_r, perf_pwc2_misses_r; always @(posedge clk) begin if (reset) begin - perf_ptw_latency_r <= '0; - end else if (ptw_active) begin - perf_ptw_latency_r <= perf_ptw_latency_r + PERF_CTR_BITS'(1); + perf_walks_r <= '0; + perf_latency_r <= '0; + perf_pwc1_hits_r <= '0; + perf_pwc1_misses_r <= '0; + perf_pwc2_hits_r <= '0; + perf_pwc2_misses_r <= '0; + end else begin + perf_latency_r <= perf_latency_r + PERF_CTR_BITS'(active_count); + if (req_fire) begin + perf_walks_r <= perf_walks_r + PERF_CTR_BITS'(1); + if (pwc1_hit) begin + perf_pwc1_hits_r <= perf_pwc1_hits_r + PERF_CTR_BITS'(1); + if (VM_PT_LEVELS == 3) begin + if (pwc2_hit) perf_pwc2_hits_r <= perf_pwc2_hits_r + PERF_CTR_BITS'(1); + else perf_pwc2_misses_r <= perf_pwc2_misses_r + PERF_CTR_BITS'(1); + end + end else begin + perf_pwc1_misses_r <= perf_pwc1_misses_r + PERF_CTR_BITS'(1); + end + end end end - assign perf_ptw_latency = perf_ptw_latency_r; -`else - assign perf_ptw_latency_placeholder = 1'b0; + assign ptw_perf.walks = perf_walks_r; + assign ptw_perf.latency = perf_latency_r; + assign ptw_perf.pwc1_hits = perf_pwc1_hits_r; + assign ptw_perf.pwc1_misses = perf_pwc1_misses_r; + assign ptw_perf.pwc2_hits = perf_pwc2_hits_r; + assign ptw_perf.pwc2_misses = perf_pwc2_misses_r; `endif endmodule diff --git a/hw/rtl/mem/VX_mmu_pwc.sv b/hw/rtl/mem/VX_mmu_pwc.sv new file mode 100644 index 0000000000..3998d9043a --- /dev/null +++ b/hw/rtl/mem/VX_mmu_pwc.sv @@ -0,0 +1,69 @@ +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +`include "VX_define.vh" + +// Page-walk cache: direct-mapped cache of non-leaf page-table entries, keyed +// by {parent table PPN, child VPN slice} and holding the child table PPN. +// The lookup is combinational so a walk can skip a level in its first cycle; +// fills are registered. Entries are only ever invalidated on reset or flush. +module VX_mmu_pwc #( + parameter KEY_WIDTH = 30, + parameter DATA_WIDTH = 20, + parameter NUM_ENTRIES = 64 +) ( + input wire clk, + input wire reset, + input wire flush, + + input wire [KEY_WIDTH-1:0] lookup_key, + output wire lookup_hit, + output wire [DATA_WIDTH-1:0] lookup_data, + + input wire fill_valid, + input wire [KEY_WIDTH-1:0] fill_key, + input wire [DATA_WIDTH-1:0] fill_data +); + `STATIC_ASSERT(`IS_POW2(NUM_ENTRIES), ("NUM_ENTRIES must be a power of 2")) + + localparam IDX_BITS = `CLOG2(NUM_ENTRIES); + localparam TAG_BITS = KEY_WIDTH - IDX_BITS; + + reg [NUM_ENTRIES-1:0] valid_r; + reg [TAG_BITS-1:0] tags_r [NUM_ENTRIES]; + reg [DATA_WIDTH-1:0] data_r [NUM_ENTRIES]; + + wire [IDX_BITS-1:0] lookup_idx = lookup_key[IDX_BITS-1:0]; + wire [TAG_BITS-1:0] lookup_tag = lookup_key[KEY_WIDTH-1:IDX_BITS]; + wire [IDX_BITS-1:0] fill_idx = fill_key[IDX_BITS-1:0]; + wire [TAG_BITS-1:0] fill_tag = fill_key[KEY_WIDTH-1:IDX_BITS]; + + assign lookup_hit = valid_r[lookup_idx] && (tags_r[lookup_idx] == lookup_tag); + assign lookup_data = data_r[lookup_idx]; + + always @(posedge clk) begin + if (reset || flush) begin + valid_r <= '0; + end else if (fill_valid) begin + valid_r[fill_idx] <= 1'b1; + end + end + + always @(posedge clk) begin + if (fill_valid) begin + tags_r[fill_idx] <= fill_tag; + data_r[fill_idx] <= fill_data; + end + end + +endmodule diff --git a/hw/rtl/mem/VX_mmu_tlb.sv b/hw/rtl/mem/VX_mmu_tlb.sv index 69741a1dc6..aaf7f20f31 100644 --- a/hw/rtl/mem/VX_mmu_tlb.sv +++ b/hw/rtl/mem/VX_mmu_tlb.sv @@ -1,61 +1,75 @@ -// Copyright 2024 -// TLB: CAM-based address translation +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. `include "VX_define.vh" +// Banked TLB: NUM_REQS lanes are spread over NUM_BANKS banks by the low VPN +// bits, so up to NUM_BANKS translations proceed per cycle and a miss parked +// in one bank does not stall hits in the others. Every bank owns one +// outstanding walk on the shared PTW bus, identified by its bank index. module VX_mmu_tlb import VX_gpu_pkg::*; #( - parameter NUM_REQS = DCACHE_NUM_REQS, - parameter DATA_SIZE = DCACHE_WORD_SIZE, - parameter TAG_WIDTH_IN = DCACHE_TAG_WIDTH, - parameter TAG_WIDTH_OUT = TAG_WIDTH_IN + `UP(`CLOG2(NUM_REQS)), - parameter ADDR_WIDTH = DCACHE_ADDR_WIDTH, - parameter ATTR_WIDTH = MEM_ATTR_WIDTH + parameter NUM_REQS = DCACHE_NUM_REQS, + parameter NUM_BANKS = 1, + parameter NUM_ENTRIES = `VX_CFG_TLB_SIZE, + parameter DATA_SIZE = DCACHE_WORD_SIZE, + parameter TAG_WIDTH_IN = DCACHE_TAG_WIDTH_BASE, + parameter TAG_WIDTH_OUT = TAG_WIDTH_IN + `UP(`CLOG2(NUM_REQS)), + parameter ADDR_WIDTH = DCACHE_ADDR_WIDTH, + parameter ATTR_WIDTH = MEM_ATTR_WIDTH, + parameter PTW_TAG_WIDTH = PTW_TLB_TAG_WIDTH ) ( input wire clk, input wire reset, + input wire flush, -`ifdef PERF_ENABLE - output mmu_perf_t mmu_perf, -`endif + input wire [VM_PPN_WIDTH-1:0] root_ppn, VX_mem_bus_if.slave tlb_in_if [NUM_REQS], VX_mem_bus_if.master tlb_out_if [NUM_REQS], - output wire miss_valid, - input wire miss_ready, - output wire [31:0] miss_vaddr, + VX_ptw_bus_if.master ptw_bus_if, - input wire fill_valid, - output wire fill_ready, - input wire [31:0] fill_vaddr, - input wire [31:0] fill_paddr, - input wire [7:0] fill_flags +`ifdef PERF_ENABLE + output mmu_perf_t mmu_perf +`else + output wire mmu_perf_placeholder +`endif ); - // fill_vaddr's page-offset bits aren't stored in the TLB entry (only - // the VPN is); the lookup-side concatenation reuses the lookup_addr's - // offset to form the final paddr. - `UNUSED_VAR (fill_vaddr[11:0]) - - // ========================================================================= - // Local Parameters - // ========================================================================= + `STATIC_ASSERT(`IS_POW2(NUM_BANKS), ("NUM_BANKS must be a power of 2")) + `STATIC_ASSERT((NUM_ENTRIES % NUM_BANKS) == 0, ("NUM_ENTRIES must be a multiple of NUM_BANKS")) + `STATIC_ASSERT(`CLOG2(NUM_BANKS) <= PTW_TAG_WIDTH, ("bank index does not fit the PTW tag")) localparam DATA_WIDTH = DATA_SIZE * 8; + localparam SOURCE_BITS = `UP(`CLOG2(NUM_REQS)); localparam REQ_DATAW_IN = 1 + ADDR_WIDTH + DATA_WIDTH + DATA_SIZE + ATTR_WIDTH + TAG_WIDTH_IN; localparam REQ_DATAW_OUT = 1 + ADDR_WIDTH + DATA_WIDTH + DATA_SIZE + ATTR_WIDTH + TAG_WIDTH_OUT; - localparam RSP_DATAW_IN = DATA_WIDTH + TAG_WIDTH_IN; - localparam RSP_DATAW_OUT = DATA_WIDTH + TAG_WIDTH_OUT; - localparam SOURCE_BITS = `UP(`CLOG2(NUM_REQS)); - - // ========================================================================= - // Request Serialize (NUM_REQS-to-1) - // ========================================================================= - - wire [NUM_REQS-1:0] req_valid_in; + localparam BANK_SIZE = NUM_ENTRIES / NUM_BANKS; + localparam BANK_SEL_BITS = `CLOG2(NUM_BANKS); + localparam BANK_BITS = `UP(BANK_SEL_BITS); + localparam PAGE_OFFSET_BITS = VM_PAGE_OFFSET_BITS - `CLOG2(DATA_SIZE); + localparam MISS_DATAW = VM_VPN_WIDTH; + localparam FILL_DATAW = VM_PPN_WIDTH + VM_LEVEL_BITS + VM_PTE_FLAGS_WIDTH + 1; + + // ------------------------------------------------------------------------- + // Request distribution: lane -> bank by low VPN bits + // ------------------------------------------------------------------------- + + wire [NUM_REQS-1:0] req_valid_in; wire [NUM_REQS-1:0][REQ_DATAW_IN-1:0] req_data_in; - wire [NUM_REQS-1:0] req_ready_in; + wire [NUM_REQS-1:0] req_ready_in; + wire [NUM_REQS-1:0][BANK_BITS-1:0] req_bank_sel; - for (genvar i = 0; i < NUM_REQS; i++) begin : g_req_in + for (genvar i = 0; i < NUM_REQS; ++i) begin : g_req_in assign req_valid_in[i] = tlb_in_if[i].req_valid; assign req_data_in[i] = { tlb_in_if[i].req_data.rw, @@ -66,438 +80,254 @@ module VX_mmu_tlb import VX_gpu_pkg::*; #( tlb_in_if[i].req_data.tag[TAG_WIDTH_IN-1:0] }; assign tlb_in_if[i].req_ready = req_ready_in[i]; + if (NUM_BANKS > 1) begin : g_bank_sel + assign req_bank_sel[i] = tlb_in_if[i].req_data.addr[PAGE_OFFSET_BITS +: BANK_SEL_BITS]; + end else begin : g_single_bank + assign req_bank_sel[i] = '0; + end end - wire ser_req_valid; - wire [REQ_DATAW_IN-1:0] ser_req_data; - wire [SOURCE_BITS-1:0] ser_req_sel; - wire ser_req_ready; + wire [NUM_BANKS-1:0] bank_req_valid; + wire [NUM_BANKS-1:0][REQ_DATAW_IN-1:0] bank_req_data; + wire [NUM_BANKS-1:0][SOURCE_BITS-1:0] bank_req_sel; + wire [NUM_BANKS-1:0] bank_req_ready; - VX_stream_arb #( + VX_stream_xbar #( .NUM_INPUTS (NUM_REQS), - .NUM_OUTPUTS (1), + .NUM_OUTPUTS (NUM_BANKS), .DATAW (REQ_DATAW_IN), .ARBITER ("R"), .OUT_BUF (0) - ) req_serialize_arb ( + ) req_xbar ( .clk (clk), .reset (reset), + `UNUSED_PIN (collisions), .valid_in (req_valid_in), .data_in (req_data_in), + .sel_in (req_bank_sel), .ready_in (req_ready_in), - .valid_out (ser_req_valid), - .data_out (ser_req_data), - .sel_out (ser_req_sel), - .ready_out (ser_req_ready) + .valid_out (bank_req_valid), + .data_out (bank_req_data), + .sel_out (bank_req_sel), + .ready_out (bank_req_ready) ); - // ========================================================================= - // TLB Logic - // ========================================================================= - - localparam TLB_INDEX_BITS = 5; - localparam PAGE_OFFSET_BITS = 12 - `CLOG2(DATA_SIZE); - localparam VPN_WIDTH = 20; - localparam PPN_WIDTH = VPN_WIDTH; - localparam SUPERPAGE_OFFSET_BITS = 22 - `CLOG2(DATA_SIZE); - - typedef struct packed { - logic valid; - logic mru; - logic [1:0] page_level; - logic [VPN_WIDTH-1:0] vpn; - logic [PPN_WIDTH-1:0] ppn; - logic [7:0] flags; - } tlb_entry_t; - - tlb_entry_t tlb_entries [`VX_CFG_TLB_SIZE-1:0]; - - typedef enum logic [1:0] { - TLB_IDLE, - TLB_READY, - TLB_PTW_WAIT, - TLB_REPLAY - } tlb_state_t; - - tlb_state_t state; - - reg [REQ_DATAW_IN-1:0] miss_buffer; - reg [SOURCE_BITS-1:0] miss_sel; - reg [31:0] miss_fill_paddr; - // page-offset bits of the fill PA aren't replayed — the lookup - // contributes the offset directly. - `UNUSED_VAR (miss_fill_paddr[11:0]) - reg miss_sent; - reg [TLB_INDEX_BITS-1:0] victim_index; - - localparam ADDR_LSB_IN = TAG_WIDTH_IN + ATTR_WIDTH + DATA_SIZE + DATA_WIDTH; - localparam ADDR_LSB = TAG_WIDTH_OUT + ATTR_WIDTH + DATA_SIZE + DATA_WIDTH; - - wire use_miss_buffer = (state == TLB_REPLAY); - wire [REQ_DATAW_IN-1:0] lookup_data = use_miss_buffer ? miss_buffer : ser_req_data; - wire [SOURCE_BITS-1:0] lookup_sel = use_miss_buffer ? miss_sel : ser_req_sel; - wire [ADDR_WIDTH-1:0] lookup_addr = lookup_data[ADDR_LSB_IN +: ADDR_WIDTH]; - wire [VPN_WIDTH-1:0] lookup_vpn = lookup_addr[ADDR_WIDTH-1:PAGE_OFFSET_BITS]; - wire [TAG_WIDTH_IN-1:0] lookup_tag = lookup_data[TAG_WIDTH_IN-1:0]; - wire [TAG_WIDTH_OUT-1:0] lookup_tag_encoded; - - VX_bits_insert #( - .N (TAG_WIDTH_IN), - .S (SOURCE_BITS), - .POS (0) - ) tag_encode ( - .data_in (lookup_tag), - .ins_in (lookup_sel), - .data_out (lookup_tag_encoded) - ); + // ------------------------------------------------------------------------- + // Banks + // ------------------------------------------------------------------------- - wire [REQ_DATAW_OUT-1:0] lookup_data_encoded = { - lookup_data[REQ_DATAW_IN-1:TAG_WIDTH_IN], - lookup_tag_encoded - }; + wire [NUM_BANKS-1:0] bank_out_valid; + wire [NUM_BANKS-1:0][REQ_DATAW_OUT-1:0] bank_out_data; + wire [NUM_BANKS-1:0] bank_out_ready; - // CAM Lookup - function automatic [VPN_WIDTH-1:0] vpn_mask(input [1:0] level); - case (level) - 2'd0: vpn_mask = 20'hFFFFF; - 2'd1: vpn_mask = 20'hFFC00; - 2'd2: vpn_mask = 20'h00000; - default: vpn_mask = 20'hFFFFF; - endcase - endfunction - - wire [`VX_CFG_TLB_SIZE-1:0] cam_hit; - for (genvar i = 0; i < `VX_CFG_TLB_SIZE; i++) begin : g_cam - wire [VPN_WIDTH-1:0] mask_i = vpn_mask(tlb_entries[i].page_level); - assign cam_hit[i] = tlb_entries[i].valid && - ((tlb_entries[i].vpn & mask_i) == (lookup_vpn & mask_i)); - end - - wire tlb_hit = |cam_hit; + wire [NUM_BANKS-1:0] bank_miss_valid; + wire [NUM_BANKS-1:0][MISS_DATAW-1:0] bank_miss_data; + wire [NUM_BANKS-1:0] bank_miss_ready; + wire [NUM_BANKS-1:0] bank_fill_valid; + wire [NUM_BANKS-1:0][FILL_DATAW-1:0] bank_fill_data; + wire [NUM_BANKS-1:0] bank_fill_ready; - reg [TLB_INDEX_BITS-1:0] hit_index; - always_comb begin - hit_index = '0; - for (int j = `VX_CFG_TLB_SIZE-1; j >= 0; j--) begin - if (cam_hit[j]) hit_index = j[TLB_INDEX_BITS-1:0]; - end - end - - // Victim Selection (MRU-based) - reg [TLB_INDEX_BITS-1:0] victim_candidate; - reg found_invalid; - wire all_mru; - - always_comb begin - victim_candidate = '0; - found_invalid = 1'b0; - for (int j = `VX_CFG_TLB_SIZE-1; j >= 0; j--) begin - if (!tlb_entries[j].valid) begin - victim_candidate = j[TLB_INDEX_BITS-1:0]; - found_invalid = 1'b1; - end - end - if (!found_invalid) begin - for (int j = `VX_CFG_TLB_SIZE-1; j >= 0; j--) begin - if (tlb_entries[j].valid && !tlb_entries[j].mru) - victim_candidate = j[TLB_INDEX_BITS-1:0]; - end - end - end +`ifdef PERF_ENABLE + wire [NUM_BANKS-1:0][PERF_CTR_BITS-1:0] bank_perf_reads, bank_perf_hits, bank_perf_misses, bank_perf_evictions; +`endif - wire [`VX_CFG_TLB_SIZE-1:0] entry_mru; - for (genvar i = 0; i < `VX_CFG_TLB_SIZE; i++) begin : g_mru_check - assign entry_mru[i] = tlb_entries[i].valid ? tlb_entries[i].mru : 1'b0; - end - assign all_mru = &entry_mru; - - // Address Translation - wire [PPN_WIDTH-1:0] hit_ppn = tlb_entries[hit_index].ppn; - wire [1:0] hit_level = tlb_entries[hit_index].page_level; - - reg [ADDR_WIDTH-1:0] cam_translated_addr; - always_comb begin - case (hit_level) - 2'd0: cam_translated_addr = {hit_ppn, lookup_addr[PAGE_OFFSET_BITS-1:0]}; - 2'd1: cam_translated_addr = {hit_ppn[VPN_WIDTH-1:10], lookup_addr[SUPERPAGE_OFFSET_BITS-1:0]}; - 2'd2: cam_translated_addr = lookup_addr; - default: cam_translated_addr = {hit_ppn, lookup_addr[PAGE_OFFSET_BITS-1:0]}; - endcase + for (genvar b = 0; b < NUM_BANKS; ++b) begin : g_banks + VX_mmu_tlb_bank #( + .NUM_ENTRIES (BANK_SIZE), + .DATA_SIZE (DATA_SIZE), + .ADDR_WIDTH (ADDR_WIDTH), + .TAG_WIDTH_IN (TAG_WIDTH_IN), + .SOURCE_BITS (SOURCE_BITS), + .ATTR_WIDTH (ATTR_WIDTH), + .TAG_WIDTH_OUT (TAG_WIDTH_OUT) + ) bank ( + .clk (clk), + .reset (reset), + .flush (flush), + .req_valid (bank_req_valid[b]), + .req_data (bank_req_data[b]), + .req_sel (bank_req_sel[b]), + .req_ready (bank_req_ready[b]), + .out_valid (bank_out_valid[b]), + .out_data (bank_out_data[b]), + .out_ready (bank_out_ready[b]), + .miss_valid (bank_miss_valid[b]), + .miss_ready (bank_miss_ready[b]), + .miss_vpn (bank_miss_data[b]), + .fill_valid (bank_fill_valid[b]), + .fill_ready (bank_fill_ready[b]), + .fill_ppn (bank_fill_data[b][FILL_DATAW-1 -: VM_PPN_WIDTH]), + .fill_level (bank_fill_data[b][1 + VM_PTE_FLAGS_WIDTH +: VM_LEVEL_BITS]), + .fill_flags (bank_fill_data[b][1 +: VM_PTE_FLAGS_WIDTH]), + .fill_fault (bank_fill_data[b][0]) + `ifdef PERF_ENABLE + ,.perf_tlb_reads (bank_perf_reads[b]) + ,.perf_tlb_hits (bank_perf_hits[b]) + ,.perf_tlb_misses (bank_perf_misses[b]) + ,.perf_tlb_evictions (bank_perf_evictions[b]) + `else + ,`UNUSED_PIN (perf_placeholder) + `endif + ); end - wire [ADDR_WIDTH-1:0] replay_paddr = {miss_fill_paddr[31:12], lookup_addr[PAGE_OFFSET_BITS-1:0]}; - wire [ADDR_WIDTH-1:0] translated_addr = use_miss_buffer ? replay_paddr : cam_translated_addr; - - // State Machine - wire input_handshake = ser_req_valid && ser_req_ready; - - always_ff @(posedge clk) begin - if (reset) begin - state <= TLB_IDLE; - miss_buffer <= '0; - miss_sel <= '0; - miss_fill_paddr <= '0; - miss_sent <= 1'b0; - victim_index <= '0; - for (int i = 0; i < `VX_CFG_TLB_SIZE; i++) begin - tlb_entries[i].valid <= 1'b0; - tlb_entries[i].mru <= 1'b0; - tlb_entries[i].page_level <= 2'd0; - tlb_entries[i].vpn <= '0; - tlb_entries[i].ppn <= '0; - tlb_entries[i].flags <= '0; - end - end else begin - case (state) - TLB_IDLE: begin - state <= TLB_READY; - end - - TLB_READY: begin - if (input_handshake) begin - if (tlb_hit) begin - tlb_entries[hit_index].mru <= 1'b1; - if (all_mru) begin - for (int i = 0; i < `VX_CFG_TLB_SIZE; i++) begin - if (i[TLB_INDEX_BITS-1:0] != hit_index) - tlb_entries[i].mru <= 1'b0; - end - end - end else begin - miss_buffer <= ser_req_data; - miss_sel <= ser_req_sel; - victim_index <= victim_candidate; - state <= TLB_PTW_WAIT; - end - end - end - - TLB_PTW_WAIT: begin - if (miss_valid && miss_ready) miss_sent <= 1'b1; - - if (fill_valid && fill_ready) begin - miss_fill_paddr <= fill_paddr; - tlb_entries[victim_index].valid <= 1'b1; - tlb_entries[victim_index].mru <= 1'b1; - tlb_entries[victim_index].page_level <= 2'd0; - tlb_entries[victim_index].vpn <= fill_vaddr[31:12]; - tlb_entries[victim_index].ppn <= fill_paddr[31:12]; - tlb_entries[victim_index].flags <= fill_flags; - - if (all_mru) begin - for (int i = 0; i < `VX_CFG_TLB_SIZE; i++) begin - if (i[TLB_INDEX_BITS-1:0] != victim_index) - tlb_entries[i].mru <= 1'b0; - end - end - state <= TLB_REPLAY; - miss_sent <= 1'b0; - end - end - - TLB_REPLAY: begin - if (output_handshake) state <= TLB_READY; - end - - default: state <= TLB_IDLE; - endcase - end + // ------------------------------------------------------------------------- + // Output gather: bank -> originating lane (skid buffers decouple the + // banks' same-cycle hit path from the gather arbiter) + // ------------------------------------------------------------------------- + + wire [NUM_BANKS-1:0] bank_buf_valid; + wire [NUM_BANKS-1:0][REQ_DATAW_OUT-1:0] bank_buf_data; + wire [NUM_BANKS-1:0] bank_buf_ready; + wire [NUM_BANKS-1:0][SOURCE_BITS-1:0] bank_buf_sel; + + for (genvar b = 0; b < NUM_BANKS; ++b) begin : g_bank_out_buf + VX_elastic_buffer #( + .DATAW (REQ_DATAW_OUT), + .SIZE (2), + .OUT_REG (0) + ) out_buf ( + .clk (clk), + .reset (reset), + .valid_in (bank_out_valid[b]), + .data_in (bank_out_data[b]), + .ready_in (bank_out_ready[b]), + .valid_out (bank_buf_valid[b]), + .data_out (bank_buf_data[b]), + .ready_out (bank_buf_ready[b]) + ); + assign bank_buf_sel[b] = bank_buf_data[b][SOURCE_BITS-1:0]; end - // Control Signals - assign ser_req_ready = (state == TLB_READY) && deser_req_ready; - - wire tlb_out_valid = (state == TLB_READY && input_handshake && tlb_hit) || - (state == TLB_REPLAY); + wire [NUM_REQS-1:0] out_valid; + wire [NUM_REQS-1:0][REQ_DATAW_OUT-1:0] out_data; + wire [NUM_REQS-1:0] out_ready; - wire output_handshake = tlb_out_valid && deser_req_ready; - - wire [REQ_DATAW_OUT-1:0] tlb_out_data = { - lookup_data_encoded[REQ_DATAW_OUT-1], - translated_addr, - lookup_data_encoded[ADDR_LSB-1:0] - }; - - // ========================================================================= - // Request Deserialize (1-to-NUM_REQS) - // ========================================================================= - - wire [SOURCE_BITS-1:0] deser_req_sel = tlb_out_data[SOURCE_BITS-1:0]; - - wire deser_req_ready; - wire [NUM_REQS-1:0] deser_req_valid_out; - wire [NUM_REQS-1:0][REQ_DATAW_OUT-1:0] deser_req_data_out; - wire [NUM_REQS-1:0] deser_req_ready_out; - - VX_stream_switch #( - .NUM_INPUTS (1), + VX_stream_xbar #( + .NUM_INPUTS (NUM_BANKS), .NUM_OUTPUTS (NUM_REQS), .DATAW (REQ_DATAW_OUT), - .OUT_BUF (1) - ) req_deserialize_switch ( + .ARBITER ("R"), + .OUT_BUF (2) + ) out_xbar ( .clk (clk), .reset (reset), - .sel_in (deser_req_sel), - .valid_in (tlb_out_valid), - .data_in (tlb_out_data), - .ready_in (deser_req_ready), - .valid_out (deser_req_valid_out), - .data_out (deser_req_data_out), - .ready_out (deser_req_ready_out) + `UNUSED_PIN (collisions), + .valid_in (bank_buf_valid), + .data_in (bank_buf_data), + .sel_in (bank_buf_sel), + .ready_in (bank_buf_ready), + .valid_out (out_valid), + .data_out (out_data), + `UNUSED_PIN (sel_out), + .ready_out (out_ready) ); - for (genvar i = 0; i < NUM_REQS; i++) begin : g_req_out - assign tlb_out_if[i].req_valid = deser_req_valid_out[i]; - assign tlb_out_if[i].req_data.rw = deser_req_data_out[i][REQ_DATAW_OUT-1]; - assign tlb_out_if[i].req_data.addr = deser_req_data_out[i][REQ_DATAW_OUT-2 -: ADDR_WIDTH]; - assign tlb_out_if[i].req_data.data = deser_req_data_out[i][REQ_DATAW_OUT-2-ADDR_WIDTH -: DATA_WIDTH]; - assign tlb_out_if[i].req_data.byteen = deser_req_data_out[i][REQ_DATAW_OUT-2-ADDR_WIDTH-DATA_WIDTH -: DATA_SIZE]; - assign tlb_out_if[i].req_data.attr = deser_req_data_out[i][REQ_DATAW_OUT-2-ADDR_WIDTH-DATA_WIDTH-DATA_SIZE -: ATTR_WIDTH]; - assign tlb_out_if[i].req_data.tag = deser_req_data_out[i][TAG_WIDTH_OUT-1:0]; - assign deser_req_ready_out[i] = tlb_out_if[i].req_ready; + for (genvar i = 0; i < NUM_REQS; ++i) begin : g_req_out + assign tlb_out_if[i].req_valid = out_valid[i]; + assign { + tlb_out_if[i].req_data.rw, + tlb_out_if[i].req_data.addr, + tlb_out_if[i].req_data.data, + tlb_out_if[i].req_data.byteen, + tlb_out_if[i].req_data.attr, + tlb_out_if[i].req_data.tag + } = out_data[i]; + assign out_ready[i] = tlb_out_if[i].req_ready; end - // ========================================================================= - // Response Serialize (NUM_REQS-to-1) - // ========================================================================= - - wire [NUM_REQS-1:0] rsp_valid_in; - wire [NUM_REQS-1:0][RSP_DATAW_OUT-1:0] rsp_data_in; - wire [NUM_REQS-1:0] rsp_ready_in; - - for (genvar i = 0; i < NUM_REQS; i++) begin : g_rsp_in - assign rsp_valid_in[i] = tlb_out_if[i].rsp_valid; - assign rsp_data_in[i] = { - tlb_out_if[i].rsp_data.data, - tlb_out_if[i].rsp_data.tag[TAG_WIDTH_OUT-1:0] - }; - assign tlb_out_if[i].rsp_ready = rsp_ready_in[i]; + // Responses return on the lane that issued the request; only the lane + // bits folded into the tag need stripping. + for (genvar i = 0; i < NUM_REQS; ++i) begin : g_rsp + assign tlb_in_if[i].rsp_valid = tlb_out_if[i].rsp_valid; + assign tlb_in_if[i].rsp_data.data = tlb_out_if[i].rsp_data.data; + assign tlb_in_if[i].rsp_data.tag = tlb_out_if[i].rsp_data.tag[TAG_WIDTH_OUT-1:SOURCE_BITS]; + assign tlb_out_if[i].rsp_ready = tlb_in_if[i].rsp_ready; end - wire ser_rsp_valid; - wire [RSP_DATAW_OUT-1:0] ser_rsp_data; - wire ser_rsp_ready; + // ------------------------------------------------------------------------- + // Walker bus: bank misses arbitrated onto one request stream, fills + // routed back by the bank index carried in the tag + // ------------------------------------------------------------------------- + + wire miss_valid; + wire [MISS_DATAW-1:0] miss_data; + wire [BANK_BITS-1:0] miss_bank; + wire miss_ready; VX_stream_arb #( - .NUM_INPUTS (NUM_REQS), + .NUM_INPUTS (NUM_BANKS), .NUM_OUTPUTS (1), - .DATAW (RSP_DATAW_OUT), + .DATAW (MISS_DATAW), .ARBITER ("R"), .OUT_BUF (0) - ) rsp_serialize_arb ( + ) miss_arb ( .clk (clk), .reset (reset), - .valid_in (rsp_valid_in), - .data_in (rsp_data_in), - .ready_in (rsp_ready_in), - .valid_out (ser_rsp_valid), - .data_out (ser_rsp_data), - `UNUSED_PIN (sel_out), - .ready_out (ser_rsp_ready) - ); - - // ========================================================================= - // Response Deserialize (1-to-NUM_REQS) - // ========================================================================= - - wire [TAG_WIDTH_OUT-1:0] ser_rsp_tag = ser_rsp_data[TAG_WIDTH_OUT-1:0]; - wire [SOURCE_BITS-1:0] rsp_source; - wire [TAG_WIDTH_IN-1:0] rsp_tag_restored; - - VX_bits_remove #( - .N (TAG_WIDTH_OUT), - .S (SOURCE_BITS), - .POS (0) - ) rsp_tag_decode ( - .data_in (ser_rsp_tag), - .sel_out (rsp_source), - .data_out (rsp_tag_restored) + .valid_in (bank_miss_valid), + .ready_in (bank_miss_ready), + .data_in (bank_miss_data), + .data_out (miss_data), + .sel_out (miss_bank), + .valid_out (miss_valid), + .ready_out (miss_ready) ); - wire [RSP_DATAW_IN-1:0] ser_rsp_data_restored = { - ser_rsp_data[RSP_DATAW_OUT-1:TAG_WIDTH_OUT], - rsp_tag_restored + assign ptw_bus_if.req_valid = miss_valid; + assign ptw_bus_if.req_data.vpn = miss_data; + assign ptw_bus_if.req_data.root_ppn = root_ppn; + assign ptw_bus_if.req_data.tag = PTW_TAG_WIDTH'(miss_bank); + assign miss_ready = ptw_bus_if.req_ready; + + wire [BANK_BITS-1:0] fill_bank = BANK_BITS'(ptw_bus_if.rsp_data.tag); + wire [FILL_DATAW-1:0] fill_data = { + ptw_bus_if.rsp_data.ppn, + ptw_bus_if.rsp_data.level, + ptw_bus_if.rsp_data.flags, + ptw_bus_if.rsp_data.fault }; - wire deser_rsp_ready; - wire [NUM_REQS-1:0] deser_rsp_valid_out; - wire [NUM_REQS-1:0][RSP_DATAW_IN-1:0] deser_rsp_data_out; - wire [NUM_REQS-1:0] deser_rsp_ready_out; - VX_stream_switch #( .NUM_INPUTS (1), - .NUM_OUTPUTS (NUM_REQS), - .DATAW (RSP_DATAW_IN), + .NUM_OUTPUTS (NUM_BANKS), + .DATAW (FILL_DATAW), .OUT_BUF (0) - ) rsp_deserialize_switch ( + ) fill_switch ( .clk (clk), .reset (reset), - .sel_in (rsp_source), - .valid_in (ser_rsp_valid), - .data_in (ser_rsp_data_restored), - .ready_in (deser_rsp_ready), - .valid_out (deser_rsp_valid_out), - .data_out (deser_rsp_data_out), - .ready_out (deser_rsp_ready_out) + .sel_in (fill_bank), + .valid_in (ptw_bus_if.rsp_valid), + .ready_in (ptw_bus_if.rsp_ready), + .data_in (fill_data), + .data_out (bank_fill_data), + .valid_out (bank_fill_valid), + .ready_out (bank_fill_ready) ); - assign ser_rsp_ready = deser_rsp_ready; - - for (genvar i = 0; i < NUM_REQS; i++) begin : g_rsp_out - assign tlb_in_if[i].rsp_valid = deser_rsp_valid_out[i]; - assign tlb_in_if[i].rsp_data.data = deser_rsp_data_out[i][RSP_DATAW_IN-1 -: DATA_WIDTH]; - assign tlb_in_if[i].rsp_data.tag = deser_rsp_data_out[i][TAG_WIDTH_IN-1:0]; - assign deser_rsp_ready_out[i] = tlb_in_if[i].rsp_ready; - end - - // ========================================================================= - // Miss/Fill Interface - // ========================================================================= - - wire [ADDR_WIDTH-1:0] miss_buffer_addr = miss_buffer[ADDR_LSB_IN +: ADDR_WIDTH]; - wire [VPN_WIDTH-1:0] miss_buffer_vpn = miss_buffer_addr[ADDR_WIDTH-1:PAGE_OFFSET_BITS]; - // Only the VPN portion of miss_buffer_addr is used to drive miss_vaddr. - `UNUSED_VAR (miss_buffer_addr[PAGE_OFFSET_BITS-1:0]) - - assign miss_valid = (state == TLB_PTW_WAIT) && !miss_sent; - assign miss_vaddr = {miss_buffer_vpn, 12'b0}; - assign fill_ready = (state == TLB_PTW_WAIT) && miss_sent; - - // ========================================================================= - // Performance Counters - // ========================================================================= + // ------------------------------------------------------------------------- + // Performance counters + // ------------------------------------------------------------------------- `ifdef PERF_ENABLE - reg [PERF_CTR_BITS-1:0] perf_tlb_reads; - reg [PERF_CTR_BITS-1:0] perf_tlb_hits; - reg [PERF_CTR_BITS-1:0] perf_tlb_misses; - reg [PERF_CTR_BITS-1:0] perf_tlb_evictions; - wire victim_was_valid = tlb_entries[victim_index].valid; - - always @(posedge clk) begin - if (reset) begin - perf_tlb_reads <= '0; - perf_tlb_hits <= '0; - perf_tlb_misses <= '0; - perf_tlb_evictions <= '0; - end else begin - if (state == TLB_READY && input_handshake) - perf_tlb_reads <= perf_tlb_reads + PERF_CTR_BITS'(1); - if (state == TLB_READY && input_handshake && tlb_hit) - perf_tlb_hits <= perf_tlb_hits + PERF_CTR_BITS'(1); - if (miss_valid && miss_ready) - perf_tlb_misses <= perf_tlb_misses + PERF_CTR_BITS'(1); - if (fill_valid && fill_ready && victim_was_valid) - perf_tlb_evictions <= perf_tlb_evictions + PERF_CTR_BITS'(1); + reg [PERF_CTR_BITS-1:0] sum_reads, sum_hits, sum_misses, sum_evictions; + always @(*) begin + sum_reads = '0; + sum_hits = '0; + sum_misses = '0; + sum_evictions = '0; + for (integer b = 0; b < NUM_BANKS; ++b) begin + sum_reads = sum_reads + bank_perf_reads[b]; + sum_hits = sum_hits + bank_perf_hits[b]; + sum_misses = sum_misses + bank_perf_misses[b]; + sum_evictions = sum_evictions + bank_perf_evictions[b]; end end - - assign mmu_perf.tlb_reads = perf_tlb_reads; - assign mmu_perf.tlb_hits = perf_tlb_hits; - assign mmu_perf.tlb_misses = perf_tlb_misses; - assign mmu_perf.tlb_evictions = perf_tlb_evictions; - assign mmu_perf.ptw_walks = perf_tlb_misses; - assign mmu_perf.ptw_latency = '0; + assign mmu_perf.tlb_reads = sum_reads; + assign mmu_perf.tlb_hits = sum_hits; + assign mmu_perf.tlb_misses = sum_misses; + assign mmu_perf.tlb_evictions = sum_evictions; +`else + assign mmu_perf_placeholder = 1'b0; `endif endmodule diff --git a/hw/rtl/mem/VX_mmu_tlb_bank.sv b/hw/rtl/mem/VX_mmu_tlb_bank.sv new file mode 100644 index 0000000000..4fc22a0443 --- /dev/null +++ b/hw/rtl/mem/VX_mmu_tlb_bank.sv @@ -0,0 +1,352 @@ +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +`include "VX_define.vh" + +// One bank of the per-core TLB: a fully-associative CAM with MRU +// replacement and a lookup/miss/replay state machine. Hits pass through in +// the same cycle; a miss parks the request, sends one walk request to the +// shared PTW and replays the request once the translation has been filled. +// Superpage entries match on the VPN bits above their page level. +module VX_mmu_tlb_bank import VX_gpu_pkg::*; #( + parameter NUM_ENTRIES = 32, + parameter DATA_SIZE = DCACHE_WORD_SIZE, + parameter ADDR_WIDTH = DCACHE_ADDR_WIDTH, + parameter TAG_WIDTH_IN = DCACHE_TAG_WIDTH_BASE, + parameter SOURCE_BITS = 1, + parameter ATTR_WIDTH = MEM_ATTR_WIDTH, + parameter DATA_WIDTH = DATA_SIZE * 8, + parameter TAG_WIDTH_OUT = TAG_WIDTH_IN + SOURCE_BITS, + parameter REQ_DATAW_IN = 1 + ADDR_WIDTH + DATA_WIDTH + DATA_SIZE + ATTR_WIDTH + TAG_WIDTH_IN, + parameter REQ_DATAW_OUT = 1 + ADDR_WIDTH + DATA_WIDTH + DATA_SIZE + ATTR_WIDTH + TAG_WIDTH_OUT +) ( + input wire clk, + input wire reset, + input wire flush, + + // arbitrated request stream in, with the originating lane + input wire req_valid, + input wire [REQ_DATAW_IN-1:0] req_data, + input wire [SOURCE_BITS-1:0] req_sel, + output wire req_ready, + + // translated request stream out, lane folded into the tag + output wire out_valid, + output wire [REQ_DATAW_OUT-1:0] out_data, + input wire out_ready, + + // page-table walker + output wire miss_valid, + input wire miss_ready, + output wire [VM_VPN_WIDTH-1:0] miss_vpn, + + input wire fill_valid, + output wire fill_ready, + input wire [VM_PPN_WIDTH-1:0] fill_ppn, + input wire [VM_LEVEL_BITS-1:0] fill_level, + input wire [VM_PTE_FLAGS_WIDTH-1:0] fill_flags, + input wire fill_fault, + +`ifdef PERF_ENABLE + output wire [PERF_CTR_BITS-1:0] perf_tlb_reads, + output wire [PERF_CTR_BITS-1:0] perf_tlb_hits, + output wire [PERF_CTR_BITS-1:0] perf_tlb_misses, + output wire [PERF_CTR_BITS-1:0] perf_tlb_evictions +`else + output wire perf_placeholder +`endif +); + `STATIC_ASSERT(`IS_POW2(NUM_ENTRIES), ("NUM_ENTRIES must be a power of 2")) + + localparam INDEX_BITS = `LOG2UP(NUM_ENTRIES); + localparam PAGE_OFFSET_BITS = VM_PAGE_OFFSET_BITS - `CLOG2(DATA_SIZE); + localparam ADDR_LSB_IN = TAG_WIDTH_IN + ATTR_WIDTH + DATA_SIZE + DATA_WIDTH; + + typedef struct packed { + logic valid; + logic mru; + logic [VM_LEVEL_BITS-1:0] level; + logic [VM_VPN_WIDTH-1:0] vpn; + logic [VM_PPN_WIDTH-1:0] ppn; + logic [VM_PTE_FLAGS_WIDTH-1:0] flags; + } tlb_entry_t; + + tlb_entry_t entries [NUM_ENTRIES]; + + typedef enum logic [1:0] { + TLB_READY = 2'd0, + TLB_PTW_WAIT = 2'd1, + TLB_REPLAY = 2'd2 + } tlb_state_t; + + tlb_state_t state; + reg [REQ_DATAW_IN-1:0] miss_buffer; + reg [SOURCE_BITS-1:0] miss_sel; + reg miss_sent; + reg replay_identity; + reg [INDEX_BITS-1:0] victim_index; + + // ------------------------------------------------------------------------- + // Lookup + // ------------------------------------------------------------------------- + + wire is_replay = (state == TLB_REPLAY); + wire [REQ_DATAW_IN-1:0] lookup_data = is_replay ? miss_buffer : req_data; + wire [SOURCE_BITS-1:0] lookup_sel = is_replay ? miss_sel : req_sel; + wire [ADDR_WIDTH-1:0] lookup_addr = lookup_data[ADDR_LSB_IN +: ADDR_WIDTH]; + wire [VM_VPN_WIDTH-1:0] lookup_vpn = lookup_addr[PAGE_OFFSET_BITS +: VM_VPN_WIDTH]; + + // VPN bits below a superpage's level are part of its page offset. + function automatic logic [VM_VPN_WIDTH-1:0] vpn_mask(input logic [VM_LEVEL_BITS-1:0] level); + return ~((VM_VPN_WIDTH'(1) << (level * VM_VPN_LEVEL_BITS)) - VM_VPN_WIDTH'(1)); + endfunction + + wire [NUM_ENTRIES-1:0] cam_hit; + for (genvar i = 0; i < NUM_ENTRIES; ++i) begin : g_cam + wire [VM_VPN_WIDTH-1:0] mask = vpn_mask(entries[i].level); + assign cam_hit[i] = entries[i].valid && ((entries[i].vpn & mask) == (lookup_vpn & mask)); + end + + wire tlb_hit; + wire [INDEX_BITS-1:0] hit_index; + + VX_priority_encoder #( + .N (NUM_ENTRIES) + ) hit_enc ( + .data_in (cam_hit), + .index_out (hit_index), + .valid_out (tlb_hit), + `UNUSED_PIN (onehot_out) + ); + + // ------------------------------------------------------------------------- + // Victim selection: a free entry first, otherwise the first non-MRU one + // ------------------------------------------------------------------------- + + wire [NUM_ENTRIES-1:0] entry_free, entry_not_mru; + for (genvar i = 0; i < NUM_ENTRIES; ++i) begin : g_victim + assign entry_free[i] = ~entries[i].valid; + assign entry_not_mru[i] = ~entries[i].mru; + end + + wire free_valid; + wire [INDEX_BITS-1:0] free_index; + wire [INDEX_BITS-1:0] not_mru_index; + + VX_priority_encoder #( + .N (NUM_ENTRIES) + ) free_enc ( + .data_in (entry_free), + .index_out (free_index), + .valid_out (free_valid), + `UNUSED_PIN (onehot_out) + ); + + VX_priority_encoder #( + .N (NUM_ENTRIES) + ) not_mru_enc ( + .data_in (entry_not_mru), + .index_out (not_mru_index), + `UNUSED_PIN (valid_out), + `UNUSED_PIN (onehot_out) + ); + + wire [INDEX_BITS-1:0] victim_candidate = free_valid ? free_index : not_mru_index; + wire all_mru = ~(|entry_not_mru); + + // ------------------------------------------------------------------------- + // Translation + // ------------------------------------------------------------------------- + + wire [VM_PPN_WIDTH-1:0] hit_ppn = entries[hit_index].ppn; + wire [VM_LEVEL_BITS-1:0] hit_level = entries[hit_index].level; + + // A level-L entry keeps the low L*VPN_LEVEL_BITS VPN bits as offset. + wire [ADDR_WIDTH-1:0] hit_page_mask = ~((ADDR_WIDTH'(1) << (PAGE_OFFSET_BITS + hit_level * VM_VPN_LEVEL_BITS)) - ADDR_WIDTH'(1)); + wire [ADDR_WIDTH-1:0] hit_page_addr = ADDR_WIDTH'({hit_ppn, {PAGE_OFFSET_BITS{1'b0}}}); + wire [ADDR_WIDTH-1:0] cam_translated_addr = (hit_page_addr & hit_page_mask) | (lookup_addr & ~hit_page_mask); + + wire [ADDR_WIDTH-1:0] translated_addr = (is_replay && replay_identity) ? lookup_addr : cam_translated_addr; + + wire [TAG_WIDTH_OUT-1:0] lookup_tag_out; + VX_bits_insert #( + .N (TAG_WIDTH_IN), + .S (SOURCE_BITS), + .POS (0) + ) tag_insert ( + .data_in (lookup_data[TAG_WIDTH_IN-1:0]), + .ins_in (lookup_sel), + .data_out (lookup_tag_out) + ); + + assign out_data = { + lookup_data[REQ_DATAW_IN-1], + translated_addr, + lookup_data[ADDR_LSB_IN-1:TAG_WIDTH_IN], + lookup_tag_out + }; + + // ------------------------------------------------------------------------- + // Control + // ------------------------------------------------------------------------- + + wire req_fire = req_valid && req_ready; + wire out_fire = out_valid && out_ready; + wire miss_fire = miss_valid && miss_ready; + wire fill_fire = fill_valid && fill_ready; + wire replay_hit = tlb_hit || replay_identity; + + assign req_ready = (state == TLB_READY) && (out_ready || !tlb_hit); + assign out_valid = ((state == TLB_READY) && req_valid && tlb_hit) + || (is_replay && replay_hit); + + assign miss_valid = (state == TLB_PTW_WAIT) && !miss_sent; + assign miss_vpn = miss_buffer[ADDR_LSB_IN + PAGE_OFFSET_BITS +: VM_VPN_WIDTH]; + assign fill_ready = (state == TLB_PTW_WAIT) && miss_sent; + + // A walk issued before a flush resolved against the old page table: its + // fill is dropped and the replay re-walks (the entry is simply absent). + reg flush_pending; + wire install = fill_fire && !fill_fault && !flush && !flush_pending; + + always @(posedge clk) begin + if (reset) begin + state <= TLB_READY; + miss_sent <= 1'b0; + replay_identity <= 1'b0; + flush_pending <= 1'b0; + end else begin + if (fill_fire) begin + flush_pending <= 1'b0; + end else if (flush && (state == TLB_PTW_WAIT) && miss_sent) begin + flush_pending <= 1'b1; + end + case (state) + TLB_READY: begin + if (req_fire && !tlb_hit) begin + miss_buffer <= req_data; + miss_sel <= req_sel; + victim_index <= victim_candidate; + miss_sent <= 1'b0; + replay_identity <= 1'b0; + state <= TLB_PTW_WAIT; + end + end + TLB_PTW_WAIT: begin + if (miss_fire) begin + miss_sent <= 1'b1; + end + if (fill_fire) begin + replay_identity <= fill_fault && !flush_pending && !flush; + state <= TLB_REPLAY; + end + end + TLB_REPLAY: begin + if (out_fire) begin + state <= TLB_READY; + end else if (!replay_hit) begin + // the filled entry was flushed before the replay; walk again + miss_sent <= 1'b0; + state <= TLB_PTW_WAIT; + end + end + default:; + endcase + end + end + + // entry storage + always @(posedge clk) begin + if (reset || flush) begin + for (integer i = 0; i < NUM_ENTRIES; ++i) begin + entries[i].valid <= 1'b0; + entries[i].mru <= 1'b0; + end + end else begin + if ((state == TLB_READY) && req_fire && tlb_hit) begin + entries[hit_index].mru <= 1'b1; + if (all_mru) begin + for (integer i = 0; i < NUM_ENTRIES; ++i) begin + if (INDEX_BITS'(i) != hit_index) entries[i].mru <= 1'b0; + end + end + end + if (install) begin + entries[victim_index].valid <= 1'b1; + entries[victim_index].mru <= 1'b1; + entries[victim_index].level <= fill_level; + entries[victim_index].vpn <= miss_vpn; + entries[victim_index].ppn <= fill_ppn; + entries[victim_index].flags <= fill_flags; + if (all_mru) begin + for (integer i = 0; i < NUM_ENTRIES; ++i) begin + if (INDEX_BITS'(i) != victim_index) entries[i].mru <= 1'b0; + end + end + end + end + end + +`ifdef DBG_TRACE_MMU + always @(posedge clk) begin + if ((state == TLB_READY) && req_fire) begin + `TRACE(2, ("%t: tlb-lookup: vpn=0x%0h, hit=%b, lane=%0d\n", $time, lookup_vpn, tlb_hit, req_sel)) + end + if (fill_fire) begin + `TRACE(2, ("%t: tlb-fill: vpn=0x%0h, ppn=0x%0h, level=%0d, fault=%b, victim=%0d, install=%b, flush=%b\n", $time, miss_vpn, fill_ppn, fill_level, fill_fault, victim_index, install, flush)) + end + if (is_replay && !replay_hit) begin + `TRACE(2, ("%t: tlb-replay-miss: vpn=0x%0h, v0=%b, e0vpn=0x%0h, e0lvl=%0d, hit=%b\n", $time, lookup_vpn, entries[0].valid, entries[0].vpn, entries[0].level, tlb_hit)) + end + if (is_replay && out_fire) begin + `TRACE(2, ("%t: tlb-replay: vpn=0x%0h, paddr=0x%0h\n", $time, lookup_vpn, translated_addr)) + end + end +`endif + + // ------------------------------------------------------------------------- + // Performance counters + // ------------------------------------------------------------------------- + +`ifdef PERF_ENABLE + reg [PERF_CTR_BITS-1:0] perf_reads_r, perf_hits_r, perf_misses_r, perf_evictions_r; + + always @(posedge clk) begin + if (reset) begin + perf_reads_r <= '0; + perf_hits_r <= '0; + perf_misses_r <= '0; + perf_evictions_r <= '0; + end else begin + if ((state == TLB_READY) && req_fire) begin + perf_reads_r <= perf_reads_r + PERF_CTR_BITS'(1); + if (tlb_hit) perf_hits_r <= perf_hits_r + PERF_CTR_BITS'(1); + end + if (miss_fire) begin + perf_misses_r <= perf_misses_r + PERF_CTR_BITS'(1); + end + if (install && entries[victim_index].valid) begin + perf_evictions_r <= perf_evictions_r + PERF_CTR_BITS'(1); + end + end + end + + assign perf_tlb_reads = perf_reads_r; + assign perf_tlb_hits = perf_hits_r; + assign perf_tlb_misses = perf_misses_r; + assign perf_tlb_evictions = perf_evictions_r; +`else + assign perf_placeholder = 1'b0; +`endif + +endmodule diff --git a/hw/rtl/mem/VX_ptw_arb.sv b/hw/rtl/mem/VX_ptw_arb.sv new file mode 100644 index 0000000000..d32a415210 --- /dev/null +++ b/hw/rtl/mem/VX_ptw_arb.sv @@ -0,0 +1,156 @@ +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +`include "VX_define.vh" + +// N-to-1 arbiter for the PTW miss/fill bus. The input index is folded into +// the low tag bits on the way up and stripped on the way back, so every +// level of the core/socket/cluster hierarchy can stack one of these. +module VX_ptw_arb import VX_gpu_pkg::*; #( + parameter NUM_INPUTS = 1, + parameter TAG_WIDTH = 1, + parameter REQ_OUT_BUF = 0, + parameter RSP_OUT_BUF = 0, + parameter `STRING ARBITER = "R" +) ( + input wire clk, + input wire reset, + + VX_ptw_bus_if.slave bus_in_if [NUM_INPUTS], + VX_ptw_bus_if.master bus_out_if +); + localparam LOG_NUM_REQS = `ARB_SEL_BITS(NUM_INPUTS, 1); + localparam TAG_WIDTH_OUT = TAG_WIDTH + LOG_NUM_REQS; + localparam REQ_DATAW = VM_VPN_WIDTH + VM_PPN_WIDTH + TAG_WIDTH; + localparam RSP_DATAW = VM_PPN_WIDTH + VM_LEVEL_BITS + VM_PTE_FLAGS_WIDTH + 1 + TAG_WIDTH; + + // request path: arbitrate and stamp the source index into the tag + + wire [NUM_INPUTS-1:0] req_valid_in; + wire [NUM_INPUTS-1:0][REQ_DATAW-1:0] req_data_in; + wire [NUM_INPUTS-1:0] req_ready_in; + + for (genvar i = 0; i < NUM_INPUTS; ++i) begin : g_req_data_in + assign req_valid_in[i] = bus_in_if[i].req_valid; + assign req_data_in[i] = bus_in_if[i].req_data; + assign bus_in_if[i].req_ready = req_ready_in[i]; + end + + wire req_valid_out; + wire [REQ_DATAW-1:0] req_data_out; + wire [`UP(LOG_NUM_REQS)-1:0] req_sel_out; + wire req_ready_out; + + VX_stream_arb #( + .NUM_INPUTS (NUM_INPUTS), + .NUM_OUTPUTS (1), + .DATAW (REQ_DATAW), + .ARBITER (ARBITER), + .OUT_BUF (REQ_OUT_BUF) + ) req_arb ( + .clk (clk), + .reset (reset), + .valid_in (req_valid_in), + .ready_in (req_ready_in), + .data_in (req_data_in), + .data_out (req_data_out), + .sel_out (req_sel_out), + .valid_out (req_valid_out), + .ready_out (req_ready_out) + ); + + wire [TAG_WIDTH-1:0] req_tag_out; + assign bus_out_if.req_valid = req_valid_out; + assign { + bus_out_if.req_data.vpn, + bus_out_if.req_data.root_ppn, + req_tag_out + } = req_data_out; + assign req_ready_out = bus_out_if.req_ready; + + if (NUM_INPUTS > 1) begin : g_req_tag_sel + VX_bits_insert #( + .N (TAG_WIDTH), + .S (LOG_NUM_REQS), + .POS (0) + ) bits_insert ( + .data_in (req_tag_out), + .ins_in (req_sel_out), + .data_out (bus_out_if.req_data.tag) + ); + end else begin : g_req_tag + `UNUSED_VAR (req_sel_out) + assign bus_out_if.req_data.tag = req_tag_out; + end + + // response path: strip the source index and route back + + wire [NUM_INPUTS-1:0] rsp_valid_out; + wire [NUM_INPUTS-1:0][RSP_DATAW-1:0] rsp_data_out; + wire [NUM_INPUTS-1:0] rsp_ready_out; + + if (NUM_INPUTS > 1) begin : g_rsp_switch + + wire [LOG_NUM_REQS-1:0] rsp_sel_in; + wire [TAG_WIDTH-1:0] rsp_tag_in; + + VX_bits_remove #( + .N (TAG_WIDTH_OUT), + .S (LOG_NUM_REQS), + .POS (0) + ) bits_remove ( + .data_in (bus_out_if.rsp_data.tag), + .sel_out (rsp_sel_in), + .data_out (rsp_tag_in) + ); + + wire [RSP_DATAW-1:0] rsp_data_in = { + bus_out_if.rsp_data.ppn, + bus_out_if.rsp_data.level, + bus_out_if.rsp_data.flags, + bus_out_if.rsp_data.fault, + rsp_tag_in + }; + + VX_stream_switch #( + .NUM_INPUTS (1), + .NUM_OUTPUTS (NUM_INPUTS), + .DATAW (RSP_DATAW), + .OUT_BUF (RSP_OUT_BUF) + ) rsp_switch ( + .clk (clk), + .reset (reset), + .sel_in (rsp_sel_in), + .valid_in (bus_out_if.rsp_valid), + .ready_in (bus_out_if.rsp_ready), + .data_in (rsp_data_in), + .data_out (rsp_data_out), + .valid_out (rsp_valid_out), + .ready_out (rsp_ready_out) + ); + + end else begin : g_rsp_passthru + + assign rsp_valid_out[0] = bus_out_if.rsp_valid; + assign rsp_data_out[0] = bus_out_if.rsp_data; + assign bus_out_if.rsp_ready = rsp_ready_out[0]; + + end + + for (genvar i = 0; i < NUM_INPUTS; ++i) begin : g_bus_in_if + assign bus_in_if[i].rsp_valid = rsp_valid_out[i]; + assign bus_in_if[i].rsp_data = rsp_data_out[i]; + assign rsp_ready_out[i] = bus_in_if[i].rsp_ready; + end + +endmodule diff --git a/hw/rtl/mem/VX_ptw_bus_if.sv b/hw/rtl/mem/VX_ptw_bus_if.sv new file mode 100644 index 0000000000..eb4032a803 --- /dev/null +++ b/hw/rtl/mem/VX_ptw_bus_if.sv @@ -0,0 +1,65 @@ +// Copyright © 2019-2023 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +`include "VX_define.vh" + +// TLB miss / fill bus between the per-core TLBs and the shared page-table +// walker. Each requester owns at most one outstanding walk per tag value; +// the tag is widened by one arbiter select field per hierarchy level. +interface VX_ptw_bus_if import VX_gpu_pkg::*; #( + parameter TAG_WIDTH = 1 +) (); + + typedef struct packed { + logic [VM_VPN_WIDTH-1:0] vpn; + logic [VM_PPN_WIDTH-1:0] root_ppn; + logic [TAG_WIDTH-1:0] tag; + } req_data_t; + + typedef struct packed { + logic [VM_PPN_WIDTH-1:0] ppn; + logic [VM_LEVEL_BITS-1:0] level; + logic [VM_PTE_FLAGS_WIDTH-1:0] flags; + logic fault; + logic [TAG_WIDTH-1:0] tag; + } rsp_data_t; + + logic req_valid; + req_data_t req_data; + logic req_ready; + + logic rsp_valid; + rsp_data_t rsp_data; + logic rsp_ready; + + modport master ( + output req_valid, + output req_data, + input req_ready, + + input rsp_valid, + input rsp_data, + output rsp_ready + ); + + modport slave ( + input req_valid, + input req_data, + output req_ready, + + output rsp_valid, + output rsp_data, + input rsp_ready + ); + +endinterface diff --git a/hw/unittest/core/VX_core_top.sv b/hw/unittest/core/VX_core_top.sv index 7654efb367..6f5ca89803 100644 --- a/hw/unittest/core/VX_core_top.sv +++ b/hw/unittest/core/VX_core_top.sv @@ -77,6 +77,17 @@ module VX_core_top import VX_gpu_pkg::*; ); VX_gbar_bus_if gbar_bus_if(); + +`ifdef VX_CFG_VM_ENABLE + // Standalone core DUT has no shared walker; sink the port idle. A VM + // build of this DUT would stall on the first TLB miss by design. + VX_ptw_bus_if #( + .TAG_WIDTH (PTW_CORE_TAG_WIDTH) + ) ptw_bus_if(); + assign ptw_bus_if.req_ready = 1'b0; + assign ptw_bus_if.rsp_valid = 1'b0; + assign ptw_bus_if.rsp_data = '0; +`endif assign gbar_req_valid = gbar_bus_if.req_valid; assign gbar_req_id = gbar_bus_if.req_data.id; assign gbar_req_size_m1 = gbar_bus_if.req_data.size_m1; @@ -235,6 +246,10 @@ module VX_core_top import VX_gpu_pkg::*; .gbar_bus_if (gbar_bus_if), + `ifdef VX_CFG_VM_ENABLE + .ptw_bus_if (ptw_bus_if), + `endif + `ifdef VX_CFG_EXT_DXA_ENABLE .dxa_req_bus_if (dxa_req_bus_if), .dxa_lmem_bus_if(dxa_lmem_bus_if), diff --git a/hw/unittest/cp_axil_regfile/VX_cp_axil_regfile_top.sv b/hw/unittest/cp_axil_regfile/VX_cp_axil_regfile_top.sv index d19cdb2dd2..6a6fa1e3dc 100644 --- a/hw/unittest/cp_axil_regfile/VX_cp_axil_regfile_top.sv +++ b/hw/unittest/cp_axil_regfile/VX_cp_axil_regfile_top.sv @@ -53,7 +53,10 @@ module VX_cp_axil_regfile_top // q_state outputs (flattened) + reset pulses output wire [NUM_QUEUES*$bits(cpe_state_t)-1:0] q_state_packed, - output wire [NUM_QUEUES-1:0] q_reset_pulse + output wire [NUM_QUEUES-1:0] q_reset_pulse, + + // CP_SATP as seen by the DMA walker + output wire [63:0] satp ); VX_cp_axil_s_if #(.ADDR_W(ADDR_W)) s_if (); @@ -109,7 +112,8 @@ module VX_cp_axil_regfile_top .q_error (q_error_arr), .last_dcr_rsp (32'd0), .q_state (q_state_arr), - .q_reset_pulse (q_reset_arr) + .q_reset_pulse (q_reset_arr), + .satp (satp) ); endmodule : VX_cp_axil_regfile_top diff --git a/hw/unittest/cp_dma/VX_cp_dma_top.sv b/hw/unittest/cp_dma/VX_cp_dma_top.sv index da9faf2c7c..bf4dd88c2f 100644 --- a/hw/unittest/cp_dma/VX_cp_dma_top.sv +++ b/hw/unittest/cp_dma/VX_cp_dma_top.sv @@ -26,6 +26,8 @@ module VX_cp_dma_top input wire grant, input wire [$bits(cmd_t)-1:0] cmd_packed, + input wire [63:0] satp, + input wire xlat_flush, output wire done, // ---- Host-memory AXI master (flat) ---- @@ -180,6 +182,8 @@ module VX_cp_dma_top .reset (reset), .grant (grant), .cmd (cmd_typed), + .satp (satp), + .xlat_flush (xlat_flush), .done (done), .axi_host (axi_host), .axi_dev (axi_dev) diff --git a/sim/opaesim/Makefile b/sim/opaesim/Makefile index c07700cd6a..21a3b7efd6 100644 --- a/sim/opaesim/Makefile +++ b/sim/opaesim/Makefile @@ -22,6 +22,7 @@ LDFLAGS += -Wl,-rpath,$(THIRD_PARTY_DIR)/ramulator -L$(THIRD_PARTY_DIR)/ramulato # debug trace enable flags DBG_TRACE_FLAGS += -DDBG_TRACE_PIPELINE DBG_TRACE_FLAGS += -DDBG_TRACE_MEM +DBG_TRACE_FLAGS += -DDBG_TRACE_MMU DBG_TRACE_FLAGS += -DDBG_TRACE_CACHE DBG_TRACE_FLAGS += -DDBG_TRACE_AFU DBG_TRACE_FLAGS += -DDBG_TRACE_SCOPE diff --git a/sim/rtlsim/Makefile b/sim/rtlsim/Makefile index 616331b261..a6d2f57569 100644 --- a/sim/rtlsim/Makefile +++ b/sim/rtlsim/Makefile @@ -21,6 +21,7 @@ LDFLAGS += -Wl,-rpath,$(THIRD_PARTY_DIR)/ramulator -L$(THIRD_PARTY_DIR)/ramulat # control RTL debug tracing states DBG_TRACE_FLAGS += -DDBG_TRACE_PIPELINE DBG_TRACE_FLAGS += -DDBG_TRACE_MEM +DBG_TRACE_FLAGS += -DDBG_TRACE_MMU DBG_TRACE_FLAGS += -DDBG_TRACE_CACHE DBG_TRACE_FLAGS += -DDBG_TRACE_AFU DBG_TRACE_FLAGS += -DDBG_TRACE_SCOPE diff --git a/sim/simx/Makefile b/sim/simx/Makefile index 2e935fd9bb..4b1d956af5 100644 --- a/sim/simx/Makefile +++ b/sim/simx/Makefile @@ -57,7 +57,7 @@ endif # Add VM sources ifneq ($(filter -DVX_CFG_VM_ENABLE, $(XCONFIGS)),) - SRCS += $(SRC_DIR)/mem/mmu.cpp + SRCS += $(SRC_DIR)/mem/mmu.cpp $(SRC_DIR)/mem/ptw.cpp SRCS += $(SRC_DIR)/mem/mmu_tlb.cpp endif diff --git a/sim/simx/constants.h b/sim/simx/constants.h index 66e9472307..397bd6d194 100644 --- a/sim/simx/constants.h +++ b/sim/simx/constants.h @@ -66,7 +66,8 @@ inline constexpr uint32_t VX_CFG_DCACHE_NUM_REQS = (VX_CFG_NUM_LSU_BLOCKS * DCAC inline constexpr uint32_t NUM_SOCKETS = __UP(VX_CFG_NUM_CORES / VX_CFG_SOCKET_SIZE); inline constexpr uint32_t VX_CFG_L2_NUM_REQS = NUM_SOCKETS * VX_CFG_L1_MEM_PORTS; -inline constexpr uint32_t VX_CFG_L3_NUM_REQS = VX_CFG_NUM_CLUSTERS * VX_CFG_L2_MEM_PORTS; +// Cluster memory ports plus, under VM, the shared page-table walker's port. +inline constexpr uint32_t VX_CFG_L3_NUM_REQS = VX_CFG_NUM_CLUSTERS * VX_CFG_L2_MEM_PORTS + VX_CFG_VM_ENABLED; inline constexpr uint32_t PER_ISSUE_WARPS = VX_CFG_NUM_WARPS / VX_CFG_ISSUE_WIDTH; inline constexpr uint32_t ISSUE_WIS_BITS = log2ceil(PER_ISSUE_WARPS); diff --git a/sim/simx/core.cpp b/sim/simx/core.cpp index 294c00127d..8b2336e61a 100644 --- a/sim/simx/core.cpp +++ b/sim/simx/core.cpp @@ -194,7 +194,7 @@ class Core::Impl { // on miss, the embedded PTW FSM emits PTE fetches via ReqOut[0] // through the same cache hierarchy as regular loads. snprintf(sname, 100, "%s-dcache_mmu", name.c_str()); - dcache_mmu_ = Mmu::Create(sname, VX_CFG_DCACHE_NUM_REQS); + dcache_mmu_ = Mmu::Create(sname, VX_CFG_DCACHE_NUM_REQS, VX_CFG_TLB_NUM_BANKS); // Per-core icache MMU (1 port). Fetch reads/writes its upstream // channels (ReqIn[0]/RspOut[0]) directly; the downstream side is @@ -214,6 +214,12 @@ class Core::Impl { // (ReqIn[0]/RspOut[0]) is consumed directly by fetch() in on_tick. icache_mmu_->ReqOut.at(0).bind(&simobject_->icache_req_out.at(0)); simobject_->icache_rsp_in.at(0).bind(&icache_mmu_->RspIn.at(0)); + + // both MMUs' walk traffic surfaces on the core for the shared walker + dcache_mmu_->PtwReqOut.bind(&simobject_->ptw_req_out.at(0)); + simobject_->ptw_rsp_in.at(0).bind(&dcache_mmu_->PtwRspIn); + icache_mmu_->PtwReqOut.bind(&simobject_->ptw_req_out.at(1)); + simobject_->ptw_rsp_in.at(1).bind(&icache_mmu_->PtwRspIn); #else // No-VM: direct passthrough. for (uint32_t p = 0; p < VX_CFG_NUM_LSU_BLOCKS * DCACHE_CHANNELS; ++p) { @@ -913,6 +919,16 @@ class Core::Impl { dcache_mmu_->set_satp(satp); icache_mmu_->set_satp(satp); } + + void mmu_flush() { + dcache_mmu_->flush(); + icache_mmu_->flush(); + } + + uint64_t tlb_reads() const { return dcache_mmu_->tlb_reads() + icache_mmu_->tlb_reads(); } + uint64_t tlb_hits() const { return dcache_mmu_->tlb_hits() + icache_mmu_->tlb_hits(); } + uint64_t tlb_misses() const { return dcache_mmu_->tlb_misses() + icache_mmu_->tlb_misses(); } + uint64_t tlb_evictions() const { return dcache_mmu_->tlb_evictions() + icache_mmu_->tlb_evictions(); } #endif PoolAllocator& trace_pool() { return trace_pool_; } @@ -983,6 +999,10 @@ Core::Core(const SimContext& ctx, , icache_rsp_in(1, this) , dcache_req_out(VX_CFG_DCACHE_NUM_REQS, this) , dcache_rsp_in(VX_CFG_DCACHE_NUM_REQS, this) +#ifdef VX_CFG_VM_ENABLE + , ptw_req_out(2, this) + , ptw_rsp_in(2, this) +#endif , core_id_(core_id) , socket_(socket) , impl_(new Impl(ctx, this)) @@ -1113,4 +1133,9 @@ Word Core::flush_warp_pipeline(uint32_t wid) { #ifdef VX_CFG_VM_ENABLE void Core::set_satp(uint64_t satp) { impl_->set_satp(satp); } +void Core::mmu_flush() { impl_->mmu_flush(); } +uint64_t Core::tlb_reads() const { return impl_->tlb_reads(); } +uint64_t Core::tlb_hits() const { return impl_->tlb_hits(); } +uint64_t Core::tlb_misses() const { return impl_->tlb_misses(); } +uint64_t Core::tlb_evictions() const { return impl_->tlb_evictions(); } #endif diff --git a/sim/simx/core.h b/sim/simx/core.h index 3501f3a5fd..ce1af32d51 100644 --- a/sim/simx/core.h +++ b/sim/simx/core.h @@ -18,6 +18,9 @@ #include #include "types.h" #include "instr_trace.h" +#ifdef VX_CFG_VM_ENABLE +#include "mem/ptw.h" +#endif #include "VX_config.h" namespace vortex { @@ -76,6 +79,13 @@ class Core : public SimObject { std::vector> dcache_req_out; std::vector> dcache_rsp_in; +#ifdef VX_CFG_VM_ENABLE + // TLB walk requests of the two per-core MMUs ([0]=dcache, [1]=icache), + // routed by the processor to the shared page-table walker. + std::vector> ptw_req_out; + std::vector> ptw_rsp_in; +#endif + Core(const SimContext& ctx, const char* name, uint32_t core_id, @@ -142,6 +152,15 @@ class Core : public SimObject { // both per-core MMUs (dcache + icache). Translation itself happens // asynchronously inside the Mmu SimObject; LSU/fetch emit VAs. void set_satp(uint64_t satp); + + // Invalidate both MMU TLBs (DCR cache-flush path). + void mmu_flush(); + + // MMU perf counters (TLB, icache + dcache summed). + uint64_t tlb_reads() const; + uint64_t tlb_hits() const; + uint64_t tlb_misses() const; + uint64_t tlb_evictions() const; #endif diff --git a/sim/simx/csr_unit.cpp b/sim/simx/csr_unit.cpp index be663ba346..4e66475f9a 100644 --- a/sim/simx/csr_unit.cpp +++ b/sim/simx/csr_unit.cpp @@ -221,6 +221,18 @@ Word CsrUnit::get_csr(uint32_t addr, uint32_t wid, uint32_t tid) { CSR_READ_64(VX_CSR_MPM_LMEM_WRITES, lmem_perf.writes); CSR_READ_64(VX_CSR_MPM_LMEM_BANK_ST, lmem_perf.bank_stalls); CSR_READ_64(VX_CSR_MPM_COALESCER_MISS, coalescer_misses); + #ifdef VX_CFG_VM_ENABLE + CSR_READ_64(VX_CSR_MPM_TLB_READS, core_->tlb_reads()); + CSR_READ_64(VX_CSR_MPM_TLB_HITS, core_->tlb_hits()); + CSR_READ_64(VX_CSR_MPM_TLB_MISSES, core_->tlb_misses()); + CSR_READ_64(VX_CSR_MPM_TLB_EVICTS, core_->tlb_evictions()); + CSR_READ_64(VX_CSR_MPM_PTW_WALKS, proc_perf.ptw.walks); + CSR_READ_64(VX_CSR_MPM_PTW_LATENCY, proc_perf.ptw.latency); + CSR_READ_64(VX_CSR_MPM_PWC1_HITS, proc_perf.ptw.pwc1_hits); + CSR_READ_64(VX_CSR_MPM_PWC1_MISSES, proc_perf.ptw.pwc1_misses); + CSR_READ_64(VX_CSR_MPM_PWC2_HITS, proc_perf.ptw.pwc2_hits); + CSR_READ_64(VX_CSR_MPM_PWC2_MISSES, proc_perf.ptw.pwc2_misses); + #endif } } break; #ifdef VX_CFG_EXT_TCU_ENABLE diff --git a/sim/simx/mem/cache.cpp b/sim/simx/mem/cache.cpp index 041e5230bc..47accc2e3d 100644 --- a/sim/simx/mem/cache.cpp +++ b/sim/simx/mem/cache.cpp @@ -1569,9 +1569,12 @@ class CacheBank : public SimObject { void processFlush() { // Wait for in-flight requests to drain before walking lines, otherwise an // outstanding fill could install a fresh line behind our scan and leave - // a dirty victim un-evicted. + // a dirty victim un-evicted. pipe_req_ occupancy must come from size(): + // TFifo::empty() hides entries still inside the pipe's latency window, and + // a replayed store pushed this very tick would dirty its line behind the + // walk. if (pending_fill_reqs_ != 0 - || !pipe_req_->empty() + || pipe_req_->size() != 0 || !mshr_.empty()) { return; } diff --git a/sim/simx/mem/mmu.cpp b/sim/simx/mem/mmu.cpp index 02ba6026fa..9e14e8d0a6 100644 --- a/sim/simx/mem/mmu.cpp +++ b/sim/simx/mem/mmu.cpp @@ -14,31 +14,53 @@ #include "../debug.h" #include +#include namespace vortex { Mmu::Mmu(const SimContext& ctx, const char* name, - uint32_t num_ports) + uint32_t num_ports, + uint32_t num_banks) : SimObject(ctx, name) , ReqIn (num_ports, this) , RspOut(num_ports, this) , ReqOut(num_ports, this) , RspIn (num_ports, this) + , PtwReqOut(this) + , PtwRspIn(this) , num_ports_(num_ports) - , tlb_(VX_CFG_TLB_SIZE) + , num_banks_(num_banks) + , tlb_(VX_CFG_TLB_SIZE, num_banks) + , banks_(num_banks) {} Mmu::~Mmu() = default; void Mmu::on_reset() { - ptw_state_ = PTW_IDLE; - // SATP is set externally via set_satp(); don't clear on simulator reset. + for (auto& b : banks_) { + b.state = BankMiss::IDLE; + b.stale = false; + } + tlb_.reset_perf(); } void Mmu::set_satp(uint64_t satp) { + if (satp_ && satp_->get_satp() == satp) + return; satp_ = std::make_unique(satp); - tlb_.flush(); // sfence.vma + this->flush(); // sfence.vma +} + +void Mmu::flush() { + tlb_.flush(); + // A walk already in flight resolved (or is resolving) against the old + // page table; its fill must not be installed. The bank re-walks when it + // comes back. A not-yet-issued walk (WALK_REQ) starts after the flush. + for (auto& b : banks_) { + if (b.state == BankMiss::WALK_WAIT) + b.stale = true; + } } bool Mmu::needs_translation(uint64_t addr) const { @@ -53,150 +75,84 @@ bool Mmu::needs_translation(uint64_t addr) const { return true; } -void Mmu::start_ptw(uint64_t va, ACCESS_TYPE type, MemReq orig, uint32_t port) { - // Walk from the root table down: level VX_VM_PT_LEVEL-1 .. 0 - // (Sv32: L1->L0; Sv39: L2->L1->L0). The root table is at the SATP PPN. - ptw_state_ = PTW_REQ; - ptw_level_ = VX_VM_PT_LEVEL - 1; - ptw_cur_ppn_ = satp_->get_base_ppn(); - ptw_vaddr_ = va; - ptw_type_ = type; - ptw_orig_req_ = orig; - ptw_orig_port_ = port; - walk_start_cyc_= SimPlatform::instance().cycles(); - ++walks_; -} - -void Mmu::on_ptw_response(const MemRsp& rsp) { - // Extract the PTE from the cache-line payload at the recorded PTE - // address (low bits give the byte offset within the line). A PTE is - // VX_VM_PTE_SIZE bytes — 4 for Sv32, 8 for Sv39. - uint64_t pte_bytes = 0; - if (rsp.data) { - uint32_t byte_off = (uint32_t)(ptw_pte_addr_ & (VX_CFG_MEM_BLOCK_SIZE - 1)); - std::memcpy(&pte_bytes, - reinterpret_cast(rsp.data->data()) + byte_off, - VX_VM_PTE_SIZE); - } - PTE_t pte(pte_bytes); - - // Validity check per RISC-V privileged spec (Sv32/Sv39). - bool invalid = (pte.v == 0) | ((pte.r == 0) & (pte.w == 1)); - if (invalid) { - // Page fault — for now, abort the simulator with a clear message. - // TODO: route a page-fault exception back to the LSU. - std::cerr << "MMU: page fault on PTE at 0x" << std::hex << ptw_pte_addr_ - << " (vaddr 0x" << ptw_vaddr_ << ")" << std::dec << std::endl; - std::abort(); - } - - // A PTE with any of R/W/X set is a leaf; R=W=X=0 is a pointer to the - // next-level table. A leaf found at level L is a (super)page — L0 = - // 4 KB, L1 = megapage, L2 = gigapage; PTW_FILL composes the PA from - // ptw_leaf_level_, so the level just needs to be recorded here. - bool is_leaf = (pte.r != 0) | (pte.w != 0) | (pte.x != 0); - if (is_leaf) { - ptw_final_ppn_ = pte.ppn; - ptw_flags_ = pte.flags; - ptw_leaf_level_ = ptw_level_; - ptw_state_ = PTW_FILL; - return; - } - // Interior node — descend to the next level. A non-leaf at level 0 - // means the walk ran out of levels with no leaf: a page fault. - if (ptw_level_ == 0) { - std::cerr << "MMU: page fault — no leaf PTE for vaddr 0x" - << std::hex << ptw_vaddr_ << std::dec << std::endl; - std::abort(); +void Mmu::on_tick() { + // 1) responses flow back upstream unchanged (the walker has its own port). + for (uint32_t p = 0; p < num_ports_; ++p) { + if (RspIn.at(p).empty()) continue; + if (RspOut.at(p).full()) continue; + RspOut.at(p).send(RspIn.at(p).peek(), 1); + RspIn.at(p).pop(); } - ptw_cur_ppn_ = pte.ppn; - --ptw_level_; - ptw_state_ = PTW_REQ; -} -void Mmu::drive_ptw() { - // Bits of VA per page-table level. Derived from PT geometry: - // VX_VM_PT_SIZE / VX_VM_PTE_SIZE = entries per table = 2^VPN_BITS_PER_LEVEL. - const uint32_t VPN_BITS = log2ceil(VX_VM_PT_SIZE / VX_VM_PTE_SIZE); - const uint64_t VPN_MASK = (1ULL << VPN_BITS) - 1; - switch (ptw_state_) { - case PTW_REQ: { - // Index the page table at the current level by this level's VPN - // slice. The root level uses the SATP base PPN (set in start_ptw); - // deeper levels use the interior PTE's PPN recorded by the response. - uint32_t shift = VX_VM_PAGE_LOG2_SIZE + ptw_level_ * VPN_BITS; - uint64_t vpn = (ptw_vaddr_ >> shift) & VPN_MASK; - ptw_pte_addr_ = pte_addr(ptw_cur_ppn_, vpn); - MemReq req(MemOp::LD, ptw_pte_addr_, /*data*/nullptr, /*byteen*/0, - PTW_TAG_MARKER, /*hart_id*/0, /*uuid*/0); - if (ReqOut.at(0).try_send(req)) { - DT(4, this->name() << " ptw L" << (uint32_t)ptw_level_ - << "-req: addr=0x" << std::hex << ptw_pte_addr_ << std::dec); - ptw_state_ = PTW_WAIT; - } - break; - } - case PTW_FILL: { - // Compose the PA. For a leaf at level L the low 12 + L*VPN_BITS VA - // bits are the offset within the (super)page — L0 = 4 KB (pgoff - // only), L1 = megapage, L2 = gigapage — and come from the VA, not - // the leaf PPN. - uint32_t off_bits = VX_VM_PAGE_LOG2_SIZE + - ptw_leaf_level_ * log2ceil(VX_VM_PT_SIZE / VX_VM_PTE_SIZE); - uint64_t off_mask = (1ULL << off_bits) - 1; - uint64_t pa_base = (ptw_final_ppn_ << VX_VM_PAGE_LOG2_SIZE) & ~off_mask; - uint64_t pa = pa_base | (ptw_vaddr_ & off_mask); - // Cache the per-4KB sub-page mapping in the TLB. A megapage walk - // therefore only services the specific 4 KB that triggered the - // miss; subsequent VAs in the same megapage will re-walk (correct, - // just less optimal — fine for the rare system regions we identity-map). - uint64_t vpn = ptw_vaddr_ >> VX_VM_PAGE_LOG2_SIZE; - uint64_t ppn_4kb = pa >> VX_VM_PAGE_LOG2_SIZE; - tlb_.fill(vpn, ppn_4kb, ptw_flags_); - MemReq translated = ptw_orig_req_; - translated.addr = pa; - if (ReqOut.at(ptw_orig_port_).try_send(translated)) { - walk_latency_ += (SimPlatform::instance().cycles() - walk_start_cyc_); - ptw_state_ = PTW_IDLE; + // 2) walker fills: install the translation and let the bank replay. + if (!PtwRspIn.empty()) { + auto rsp = PtwRspIn.peek(); + PtwRspIn.pop(); + auto& bank = banks_.at(rsp.tag); + __assert(bank.state == BankMiss::WALK_WAIT, "unexpected walker fill"); + if (bank.stale) { + DT(3, this->name() << " tlb-fill-stale: bank=" << rsp.tag << " (re-walk)"); + bank.stale = false; + bank.state = BankMiss::WALK_REQ; + } else { + if (!rsp.fault) { + tlb_.fill(bank.req.addr >> VX_VM_PAGE_LOG2_SIZE, rsp.ppn, rsp.level, rsp.flags); + } + DT(3, this->name() << " tlb-fill: ppn=0x" << std::hex << rsp.ppn << std::dec << ", level=" << (int)rsp.level << ", fault=" << rsp.fault << ", bank=" << rsp.tag); + bank.rsp = rsp; + bank.state = BankMiss::REPLAY; } - break; - } - default: - break; } -} -void Mmu::on_tick() { - // 1) Drain responses. PTW responses are claimed by the FSM; everything - // else flows back upstream unchanged. - for (uint32_t p = 0; p < num_ports_; ++p) { - if (RspIn.at(p).empty()) continue; - const MemRsp& rsp = RspIn.at(p).peek(); - if (rsp.tag & PTW_TAG_MARKER) { - // PTW response. Only PTW state cares about it. - if (ptw_state_ == PTW_WAIT) { - on_ptw_response(rsp); - } - RspIn.at(p).pop(); - } else { - if (RspOut.at(p).full()) continue; - RspOut.at(p).send(rsp, 1); - RspIn.at(p).pop(); + // 3) replay parked accesses (translated, or as-is on a fault). The parked + // packet stays at the head of its ReqIn port until it leaves here, so the + // walk-in-flight access remains visible to SimChannel in-flight accounting + // (the processor's idle/flush decision) and later packets on that lane + // cannot overtake it — same as the RTL's per-lane elastic buffer. + for (auto& bank : banks_) { + if (bank.state != BankMiss::REPLAY) + continue; + MemReq translated = ReqIn.at(bank.port).peek(); + if (!bank.rsp.fault) { + translated.addr = (bank.rsp.ppn << VX_VM_PAGE_LOG2_SIZE) + | (translated.addr & ((1ULL << VX_VM_PAGE_LOG2_SIZE) - 1)); } + if (!ReqOut.at(bank.port).try_send(translated, TRANSLATE_LATENCY)) + continue; + ReqIn.at(bank.port).pop(); + bank.state = BankMiss::IDLE; } - // 2) Run PTW FSM — emit pending PTE fetches / fill. - if (ptw_state_ != PTW_IDLE && ptw_state_ != PTW_WAIT) { - drive_ptw(); + // 4) send one pending walk request (round-robin over banks). + if (!PtwReqOut.full()) { + for (uint32_t i = 0; i < num_banks_; ++i) { + uint32_t b = (miss_rr_ + i) % num_banks_; + auto& bank = banks_.at(b); + if (bank.state != BankMiss::WALK_REQ) + continue; + PtwReq req; + req.vpn = bank.req.addr >> VX_VM_PAGE_LOG2_SIZE; + req.root_ppn = satp_->get_base_ppn(); + req.tag = b; + PtwReqOut.send(req, 1); + bank.state = BankMiss::WALK_WAIT; + miss_rr_ = b + 1; + break; + } } - // 3) Forward incoming requests. Bypass for non-translated addresses; - // TLB-hit translates inline; TLB-miss kicks PTW (if free). - for (uint32_t p = 0; p < num_ports_; ++p) { + // 5) forward incoming requests. Bypass for non-translated addresses; + // TLB-hit translates and forwards after the lookup-pipeline latency; + // TLB-miss parks the access on its bank. Each bank is a single-ported CAM + // fed through a crossbar in the RTL, so it accepts at most one lookup per + // cycle; ports contend round-robin (mirrors VX_mmu_tlb's "R" arbiters). + uint64_t bank_taken = 0; + for (uint32_t i = 0; i < num_ports_; ++i) { + uint32_t p = (port_rr_ + i) % num_ports_; if (ReqIn.at(p).empty()) continue; const MemReq& req = ReqIn.at(p).peek(); - if (!needs_translation(req.addr)) { + if (!this->needs_translation(req.addr)) { if (ReqOut.at(p).try_send(req)) { ReqIn.at(p).pop(); } @@ -204,25 +160,32 @@ void Mmu::on_tick() { } uint64_t vpn = req.addr >> VX_VM_PAGE_LOG2_SIZE; - auto [hit, ppn] = tlb_.lookup(vpn); - if (hit) { + uint32_t b = tlb_.bank_of(vpn); + auto& bank = banks_.at(b); + if (bank.state != BankMiss::IDLE) { + // the bank is busy walking; this request waits at the head of ReqIn[p] + continue; + } + if (bank_taken & (1ULL << b)) + continue; // bank's lookup port already used this cycle + if (ReqOut.at(p).full()) + continue; // don't burn the bank slot (or the perf counters) on a stall + bank_taken |= 1ULL << b; + auto res = tlb_.lookup(vpn); + if (res.hit) { MemReq translated = req; - translated.addr = (ppn << VX_VM_PAGE_LOG2_SIZE) | - (req.addr & ((1ULL << VX_VM_PAGE_LOG2_SIZE) - 1)); - if (ReqOut.at(p).try_send(translated)) { - ReqIn.at(p).pop(); - } + translated.addr = (res.ppn << VX_VM_PAGE_LOG2_SIZE) + | (req.addr & ((1ULL << VX_VM_PAGE_LOG2_SIZE) - 1)); + ReqOut.at(p).send(translated, TRANSLATE_LATENCY); + ReqIn.at(p).pop(); } else { - // TLB miss — kick PTW if it's idle. Otherwise this request waits - // (stays at the head of ReqIn[p]) until the in-flight walk completes. - if (ptw_state_ == PTW_IDLE) { - // ACCESS_TYPE inferred from req op (no FETCH on dcache port). - ACCESS_TYPE type = req.is_write() ? ACCESS_TYPE::STORE : ACCESS_TYPE::LOAD; - start_ptw(req.addr, type, req, p); - ReqIn.at(p).pop(); - } + DT(3, this->name() << " tlb-miss: addr=0x" << std::hex << req.addr << std::dec << ", bank=" << b << ", port=" << p); + bank.req = req; + bank.port = p; + bank.state = BankMiss::WALK_REQ; } } + ++port_rr_; } } // namespace vortex diff --git a/sim/simx/mem/mmu.h b/sim/simx/mem/mmu.h index 3f4f0ab90d..9460078e2b 100644 --- a/sim/simx/mem/mmu.h +++ b/sim/simx/mem/mmu.h @@ -18,39 +18,21 @@ #include #include "../types.h" #include "mmu_tlb.h" +#include "ptw.h" namespace vortex { // Per-core MMU SimObject. Sits on the per-core dcache (or icache) request // path between the upstream (mem_unit/coalescer/lsu_dcache_adapter) and the -// downstream cache port. -// -// PTW PTE fetches go through this object's own downstream MemReq channel — -// i.e. through the cache hierarchy — exactly like a regular load. -// -// Architecture (per port p): -// ReqIn[p] --> [bypass | TLB lookup | PTW miss queue] --> ReqOut[p] -// RspOut[p] <-- [filter PTW responses, forward rest] <-- RspIn[p] -// -// PTW responses are distinguished from regular responses by a marker -// bit on the tag (`PTW_TAG_MARKER`) — the upstream caller never sees -// these because the MMU consumes them and walks the table. +// downstream cache port. The banked TLB translates inline on a hit; a miss +// parks the access on its bank and sends a walk request to the shared +// device-level Ptw, identified by the bank index. One outstanding walk per +// bank; hits in other banks proceed meanwhile. +// Mirrors hw/rtl/mem/VX_mmu.sv + VX_mmu_tlb.sv + VX_mmu_tlb_bank.sv. class Mmu : public SimObject { public: using Ptr = std::shared_ptr; - // PTW marker bit on MemReq/MemRsp tag — distinguishes PTW PTE - // fetches from regular upstream traffic on the shared dcache port. - // It must sit above every real LSU requestor tag (those are small — - // bounded by the coalescer/LSUQ depth) yet survive the dcache's - // non-cacheable bypass packing, which multiplexes the requestor port - // into the low bits with `tag = (tag << log2_num_inputs) | req_id` in - // a uint32_t MemReq::tag. Bit 31 overflows that shift (the marker is - // lost, so the PTE response is misrouted upstream and the PTW FSM - // hangs); bit 24 leaves 16M of real-tag headroom below it and 7 bits - // of shift headroom above it. - static constexpr uint32_t PTW_TAG_MARKER = 1u << 24; - // Upstream side (LSU/coalescer/fetch). std::vector> ReqIn; std::vector> RspOut; @@ -59,70 +41,60 @@ class Mmu : public SimObject { std::vector> ReqOut; std::vector> RspIn; + // Shared page-table walker. + SimChannel PtwReqOut; + SimChannel PtwRspIn; + Mmu(const SimContext& ctx, const char* name, - uint32_t num_ports); + uint32_t num_ports, + uint32_t num_banks = 1); ~Mmu(); // SATP CSR write — invoked from CsrUnit on `csrw satp`. Flushes the - // TLB on change (sfence.vma semantics). + // TLB when the value changes (sfence.vma semantics). void set_satp(uint64_t satp); + // Invalidate the TLB (DCR cache-flush path). + void flush(); + // Perf counter accessors. uint64_t tlb_reads() const { return tlb_.reads(); } uint64_t tlb_hits() const { return tlb_.hits(); } uint64_t tlb_misses() const { return tlb_.misses(); } uint64_t tlb_evictions()const { return tlb_.evictions(); } - uint64_t ptw_walks() const { return walks_; } - uint64_t ptw_latency() const { return walk_latency_; } protected: void on_reset(); void on_tick(); private: + // Round-trip cost of a translated access through the RTL lookup pipeline: + // per-lane elastic buffer -> lane/bank crossbar -> banked CAM lookup -> + // gather crossbar, plus the response-side elastic buffer (charged here on + // the request instead of the response path). Calibrated against rtlsim. + static constexpr uint64_t TRANSLATE_LATENCY = 5; + bool needs_translation(uint64_t addr) const; - // PTE address for a given level given the current SATP and walk VA. - uint64_t pte_addr(uint64_t base_ppn, uint64_t vpn_idx) const { - return (base_ppn * VX_VM_PT_SIZE) + (vpn_idx * VX_VM_PTE_SIZE); - } - - void start_ptw(uint64_t va, ACCESS_TYPE type, MemReq orig, uint32_t port); - void on_ptw_response(const MemRsp& rsp); - void drive_ptw(); - - uint32_t num_ports_; - std::unique_ptr satp_; - Tlb tlb_; - - // PTW FSM. One walk in flight at a time. - // The walk is VX_VM_PT_LEVEL-deep — Sv32 (2 levels) and Sv39 (3) — and - // is driven by a level counter rather than per-level states. - enum PtwState { - PTW_IDLE, - PTW_REQ, // need to emit the current level's PTE fetch - PTW_WAIT, // waiting for the current level's PTE response - PTW_FILL // ready to fill TLB and replay + // One parked access per TLB bank while its walk is in flight. + struct BankMiss { + enum State { IDLE, WALK_REQ, WALK_WAIT, REPLAY }; + State state = IDLE; + MemReq req; + uint32_t port = 0; + PtwRsp rsp; + bool stale = false; // walk issued before the last flush; discard its fill and re-walk }; - PtwState ptw_state_ = PTW_IDLE; - uint64_t ptw_vaddr_ = 0; - ACCESS_TYPE ptw_type_ = ACCESS_TYPE::LOAD; - uint64_t ptw_pte_addr_ = 0; // address of the most recent PTE fetch - uint64_t ptw_cur_ppn_ = 0; // base PPN of the page table at ptw_level_ - uint8_t ptw_level_ = 0; // current walk level: VX_VM_PT_LEVEL-1 .. 0 - uint64_t ptw_final_ppn_ = 0; - uint8_t ptw_flags_ = 0; - uint8_t ptw_leaf_level_= 0; // leaf level: 0 = 4KB, 1 = mega, 2 = giga - MemReq ptw_orig_req_; - uint32_t ptw_orig_port_ = 0; - - // Perf - uint64_t walks_ = 0; - uint64_t walk_latency_ = 0; - uint64_t walk_start_cyc_= 0; + uint32_t num_ports_; + uint32_t num_banks_; + std::unique_ptr satp_; + Tlb tlb_; + std::vector banks_; + uint32_t miss_rr_ = 0; + uint32_t port_rr_ = 0; friend class SimObject; }; diff --git a/sim/simx/mem/mmu_tlb.cpp b/sim/simx/mem/mmu_tlb.cpp index 42e03b2503..03a86b941e 100644 --- a/sim/simx/mem/mmu_tlb.cpp +++ b/sim/simx/mem/mmu_tlb.cpp @@ -5,58 +5,75 @@ // You may obtain a copy of the License at // http://www.apache.org/licenses/LICENSE-2.0 -#include +#include "mmu_tlb.h" #ifdef VX_CFG_VM_ENABLE -#include "mmu_tlb.h" -#include +#include namespace vortex { -Tlb::Tlb(uint32_t size) +static constexpr uint32_t VPN_BITS_PER_LEVEL = log2ceil(VX_VM_PT_SIZE / VX_VM_PTE_SIZE); + +Tlb::Tlb(uint32_t size, uint32_t num_banks) : entries_(size) + , num_banks_(num_banks) + , bank_size_(size / num_banks) {} -std::pair Tlb::lookup(uint64_t vpn) { +uint64_t Tlb::level_mask(uint8_t level) { + // VPN bits below a superpage's level belong to its page offset. + return ~((1ULL << (level * VPN_BITS_PER_LEVEL)) - 1); +} + +Tlb::LookupResult Tlb::lookup(uint64_t vpn) { ++reads_; - for (auto& e : entries_) { - if (e.valid && e.vpn == vpn) { + uint32_t base = this->bank_of(vpn) * bank_size_; + for (uint32_t i = base; i < base + bank_size_; ++i) { + auto& e = entries_[i]; + auto mask = level_mask(e.level); + if (e.valid && (e.vpn & mask) == (vpn & mask)) { e.mru = true; ++hits_; - return {true, e.ppn}; + // Resolve the (super)page to the specific 4 KB frame. + return {true, (e.ppn & mask) | (vpn & ~mask), e.level}; } } ++misses_; - return {false, 0}; + return {}; } -void Tlb::fill(uint64_t vpn, uint64_t ppn, uint8_t flags) { - // Prefer an invalid slot; fall back to a non-MRU victim. If all slots - // are valid AND every slot has mru=true, clear all MRU bits and evict slot 0. +void Tlb::fill(uint64_t vpn, uint64_t ppn, uint8_t level, uint8_t flags) { + // Prefer an invalid slot of the bank; fall back to a non-MRU victim. If + // every slot is valid AND MRU, clear the MRU bits and evict the first. + uint32_t base = this->bank_of(vpn) * bank_size_; int victim = -1; - for (size_t i = 0; i < entries_.size(); ++i) { + for (uint32_t i = base; i < base + bank_size_; ++i) { if (!entries_[i].valid) { victim = (int)i; break; } } if (victim < 0) { - for (size_t i = 0; i < entries_.size(); ++i) { + for (uint32_t i = base; i < base + bank_size_; ++i) { if (!entries_[i].mru) { victim = (int)i; break; } } } if (victim < 0) { - // All entries are valid + MRU. Clear MRU bits and pick slot 0. - for (auto& e : entries_) e.mru = false; - victim = 0; + for (uint32_t i = base; i < base + bank_size_; ++i) { + entries_[i].mru = false; + } + victim = (int)base; } if (entries_[victim].valid) ++evictions_; - entries_[victim].valid = true; - entries_[victim].mru = true; - entries_[victim].vpn = vpn; - entries_[victim].ppn = ppn; - entries_[victim].flags = flags; + entries_[victim] = Entry{true, true, level, vpn, ppn, flags}; +} + +void Tlb::reset_perf() { + reads_ = 0; + hits_ = 0; + misses_ = 0; + evictions_ = 0; } void Tlb::flush() { diff --git a/sim/simx/mem/mmu_tlb.h b/sim/simx/mem/mmu_tlb.h index f751a7cdd6..3b78723103 100644 --- a/sim/simx/mem/mmu_tlb.h +++ b/sim/simx/mem/mmu_tlb.h @@ -11,28 +11,50 @@ #ifdef VX_CFG_VM_ENABLE +#include #include #include namespace vortex { -// Per-core TLB. Small fully-associative CAM of {vpn → ppn} translations -// with MRU-style eviction. Tracks MMU perf counters (VX_DCR_MPM_CLASS_MEM). +// Per-core TLB, split into banks selected by the low VPN bits. Each bank is +// a small fully-associative CAM of {vpn -> ppn} translations with MRU-style +// eviction; entries carry their page level so superpage leaves match on the +// VPN bits above that level. Mirrors hw/rtl/mem/VX_mmu_tlb.sv. class Tlb { public: - explicit Tlb(uint32_t size = VX_CFG_TLB_SIZE); + // Same constraints VX_mmu_tlb.sv STATIC_ASSERTs: power-of-two bank count + // that divides the (power-of-two) entry count. + static_assert((VX_CFG_TLB_SIZE & (VX_CFG_TLB_SIZE - 1)) == 0, "VX_CFG_TLB_SIZE must be a power of two"); + static_assert((VX_CFG_TLB_NUM_BANKS & (VX_CFG_TLB_NUM_BANKS - 1)) == 0, "VX_CFG_TLB_NUM_BANKS must be a power of two"); + static_assert(VX_CFG_TLB_NUM_BANKS <= VX_CFG_TLB_SIZE, "VX_CFG_TLB_NUM_BANKS must not exceed VX_CFG_TLB_SIZE"); - // Returns {hit, ppn} for the given vpn. Increments `reads_` on every - // call and `hits_` on a successful lookup. - std::pair lookup(uint64_t vpn); + explicit Tlb(uint32_t size = VX_CFG_TLB_SIZE, uint32_t num_banks = 1); - // Install a new translation. Evicts a non-MRU entry when the TLB is - // full; updates `evictions_` if the chosen slot was previously valid. - void fill(uint64_t vpn, uint64_t ppn, uint8_t flags); + struct LookupResult { + bool hit = false; + uint64_t ppn = 0; // 4 KB-resolved + uint8_t level = 0; + }; + + uint32_t bank_of(uint64_t vpn) const { return vpn & (num_banks_ - 1); } + + // Returns the translation for the given vpn. Increments `reads_` on + // every call and `hits_` on a successful lookup. + LookupResult lookup(uint64_t vpn); - // Invalidate every entry (sfence.vma equivalent). + // Install a new translation at the given page level. Evicts a non-MRU + // entry of the bank when full; updates `evictions_` if the chosen slot + // was previously valid. + void fill(uint64_t vpn, uint64_t ppn, uint8_t level, uint8_t flags); + + // Invalidate every entry (sfence.vma equivalent). Counters are kept: a + // flush is part of normal operation, not a reset. void flush(); + // Zero the perf counters (kernel-launch reset, like the caches). + void reset_perf(); + uint64_t reads() const { return reads_; } uint64_t hits() const { return hits_; } uint64_t misses() const { return misses_; } @@ -42,14 +64,19 @@ class Tlb { struct Entry { bool valid = false; bool mru = false; + uint8_t level = 0; uint64_t vpn = 0; uint64_t ppn = 0; uint8_t flags = 0; }; - // Linear flat array; small enough (typ. 32 entries) for a per-cycle - // linear scan to model CAM lookup behavior. + static uint64_t level_mask(uint8_t level); + + // Flat array, bank-major; small enough for a linear scan per bank to + // model the CAM lookup. std::vector entries_; + uint32_t num_banks_; + uint32_t bank_size_; uint64_t reads_ = 0; uint64_t hits_ = 0; diff --git a/sim/simx/mem/ptw.cpp b/sim/simx/mem/ptw.cpp new file mode 100644 index 0000000000..7ed30364ff --- /dev/null +++ b/sim/simx/mem/ptw.cpp @@ -0,0 +1,224 @@ +// Copyright © 2019-2025 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 + +#include +#include + +#ifdef VX_CFG_VM_ENABLE + +#include "ptw.h" +#include "../debug.h" +#include +#include +#include +#include + +namespace vortex { + +static constexpr uint32_t VPN_BITS_PER_LEVEL = log2ceil(VX_VM_PT_SIZE / VX_VM_PTE_SIZE); +static constexpr uint32_t TOP_LEVEL = VX_VM_PT_LEVEL - 1; + +Ptw::Ptw(const SimContext& ctx, const char* name, uint32_t num_clients) + : SimObject(ctx, name) + , ReqIn (num_clients, this) + , RspOut(num_clients, this) + , MemReqOut(this) + , MemRspIn(this) + , num_clients_(num_clients) + , slots_(VX_CFG_PTW_NUM_WALKERS) + , pwc1_(VX_CFG_PTW_WALK_CACHE_SIZE) + , pwc2_(VX_CFG_PTW_WALK_CACHE_SIZE) +{} + +Ptw::~Ptw() = default; + +void Ptw::on_reset() { + for (auto& s : slots_) s = Slot{}; + perf_ = PerfStats(); + this->flush(); +} + +void Ptw::flush() { + for (auto& e : pwc1_) e.valid = false; + for (auto& e : pwc2_) e.valid = false; + // A walk already in flight read PTEs that may predate the page-table + // update this flush publishes; its upper-level results must not land in + // the caches just emptied (the requesting TLB bank discards its result). + for (auto& s : slots_) { + if (s.state != Slot::IDLE) s.stale = true; + } +} + +uint64_t Ptw::vpn_slice(uint64_t vpn, uint32_t level) const { + return (vpn >> (level * VPN_BITS_PER_LEVEL)) & ((1ULL << VPN_BITS_PER_LEVEL) - 1); +} + +bool Ptw::pwc_lookup(const std::vector& pwc, uint64_t key, uint64_t* ppn) const { + auto& e = pwc.at(key & (pwc.size() - 1)); + if (e.valid && e.key == key) { + *ppn = e.ppn; + return true; + } + return false; +} + +void Ptw::pwc_fill(std::vector& pwc, uint64_t key, uint64_t ppn) { + pwc.at(key & (pwc.size() - 1)) = PwcEntry{true, key, ppn}; +} + +void Ptw::on_tick() { + // 1) memory responses: decode the PTE addressed by the owning slot + if (!MemRspIn.empty()) { + auto& rsp = MemRspIn.peek(); + auto& slot = slots_.at(rsp.tag); + __assert(slot.state == Slot::MEM_RSP, "unexpected walker response"); + + uint64_t pte_addr = (slot.cur_ppn * VX_VM_PT_SIZE) + + (this->vpn_slice(slot.vpn, slot.level) * VX_VM_PTE_SIZE); + uint64_t pte_bytes = 0; + if (rsp.data) { + uint32_t byte_off = (uint32_t)(pte_addr & (VX_CFG_MEM_BLOCK_SIZE - 1)); + std::memcpy(&pte_bytes, + reinterpret_cast(rsp.data->data()) + byte_off, + VX_VM_PTE_SIZE); + } + PTE_t pte(pte_bytes); + DT(3, this->name() << " pte-rsp: slot=" << rsp.tag << ", pte=0x" << std::hex << pte_bytes << std::dec); + + bool invalid = (pte.v == 0) || ((pte.r == 0) && (pte.w == 1)); + bool is_leaf = (pte.r != 0) || (pte.w != 0) || (pte.x != 0); + uint64_t sp_mask = (1ULL << (slot.level * VPN_BITS_PER_LEVEL)) - 1; + bool misaligned = is_leaf && ((pte.ppn & sp_mask) != 0); + bool fault = invalid || misaligned || (!is_leaf && slot.level == 0); + + if (fault) { + // Match the RTL: report the fault to the TLB, which replays the + // access untranslated. Keep the simulation loud about it. + std::cerr << "PTW: page fault on PTE at 0x" << std::hex << pte_addr + << " (vpn 0x" << slot.vpn << ")" << std::dec << std::endl; + std::abort(); + } + + if (is_leaf) { + // Resolve the (super)page leaf to the faulting 4 KB frame. + slot.cur_ppn = (pte.ppn & ~sp_mask) | (slot.vpn & sp_mask); + slot.flags = pte.flags; + slot.fault = false; + slot.state = Slot::DONE; + } else { + if (slot.stale) { + // no PWC fill + } else if (slot.level == TOP_LEVEL) { + this->pwc_fill(pwc1_, (slot.root_ppn << VPN_BITS_PER_LEVEL) + | this->vpn_slice(slot.vpn, TOP_LEVEL), pte.ppn); + } else if (VX_VM_PT_LEVEL == 3 && slot.level == 1) { + this->pwc_fill(pwc2_, (slot.cur_ppn << VPN_BITS_PER_LEVEL) + | this->vpn_slice(slot.vpn, 1), pte.ppn); + } + slot.cur_ppn = pte.ppn; + --slot.level; + slot.state = Slot::MEM_REQ; + } + MemRspIn.pop(); + } + + // 2) issue one PTE fetch (round-robin over slots wanting memory) + if (!MemReqOut.full()) { + for (uint32_t i = 0; i < slots_.size(); ++i) { + uint32_t s = (mem_rr_ + i) % slots_.size(); + auto& slot = slots_.at(s); + if (slot.state != Slot::MEM_REQ) + continue; + uint64_t pte_addr = (slot.cur_ppn * VX_VM_PT_SIZE) + + (this->vpn_slice(slot.vpn, slot.level) * VX_VM_PTE_SIZE); + MemReq req(MemOp::LD, pte_addr, nullptr, 0, s, 0, 0); + DT(3, this->name() << " pte-fetch: addr=0x" << std::hex << pte_addr << std::dec << ", slot=" << s << ", level=" << (int)slot.level); + MemReqOut.send(req, 1); + slot.state = Slot::MEM_RSP; + mem_rr_ = s + 1; + break; + } + } + + // 3) hand one finished walk back (round-robin over done slots) + for (uint32_t i = 0; i < slots_.size(); ++i) { + uint32_t s = (done_rr_ + i) % slots_.size(); + auto& slot = slots_.at(s); + if (slot.state != Slot::DONE) + continue; + if (RspOut.at(slot.client).full()) + continue; + // slot.level stopped at the leaf level (0 = 4 KB page). + PtwRsp rsp{slot.cur_ppn, slot.level, slot.flags, slot.fault, slot.tag}; + DT(3, this->name() << " walk-done: ppn=0x" << std::hex << slot.cur_ppn << std::dec << ", client=" << slot.client << ", slot=" << s); + RspOut.at(slot.client).send(rsp, 1); + perf_.latency += (SimPlatform::instance().cycles() - slot.start_cycle); + slot.state = Slot::IDLE; + done_rr_ = s + 1; + break; + } + + // 4) accept one new walk (round-robin over clients) into a free slot + int free_slot = -1; + for (uint32_t s = 0; s < slots_.size(); ++s) { + if (slots_.at(s).state == Slot::IDLE) { free_slot = (int)s; break; } + } + if (free_slot >= 0) { + for (uint32_t i = 0; i < num_clients_; ++i) { + uint32_t c = (client_rr_ + i) % num_clients_; + if (ReqIn.at(c).empty()) + continue; + auto req = ReqIn.at(c).peek(); + ReqIn.at(c).pop(); + auto& slot = slots_.at(free_slot); + slot = Slot{}; + slot.vpn = req.vpn; + slot.root_ppn = req.root_ppn; + slot.stale = false; + slot.client = c; + slot.tag = req.tag; + slot.start_cycle = SimPlatform::instance().cycles(); + slot.state = Slot::MEM_REQ; + + // walk-cache lookups pick the starting level + uint64_t ppn1 = 0, ppn2 = 0; + bool hit1 = this->pwc_lookup(pwc1_, (req.root_ppn << VPN_BITS_PER_LEVEL) + | this->vpn_slice(req.vpn, TOP_LEVEL), &ppn1); + bool hit2 = hit1 && (VX_VM_PT_LEVEL == 3) + && this->pwc_lookup(pwc2_, (ppn1 << VPN_BITS_PER_LEVEL) + | this->vpn_slice(req.vpn, 1), &ppn2); + if (hit2) { + // hit2 implies Sv39 (TOP_LEVEL = 2); the guard keeps Sv32 folds sane + slot.level = (uint8_t)((TOP_LEVEL >= 2) ? (TOP_LEVEL - 2) : 0); + slot.cur_ppn = ppn2; + } else if (hit1) { + slot.level = TOP_LEVEL - 1; + slot.cur_ppn = ppn1; + } else { + slot.level = TOP_LEVEL; + slot.cur_ppn = req.root_ppn; + } + + DT(3, this->name() << " walk-start: vpn=0x" << std::hex << req.vpn << std::dec << ", client=" << c << ", slot=" << free_slot << ", level=" << (int)slot.level); + ++perf_.walks; + if (hit1) { + ++perf_.pwc1_hits; + if (VX_VM_PT_LEVEL == 3) { + if (hit2) ++perf_.pwc2_hits; else ++perf_.pwc2_misses; + } + } else { + ++perf_.pwc1_misses; + } + client_rr_ = c + 1; + break; + } + } +} + +} // namespace vortex + +#endif // VX_CFG_VM_ENABLE diff --git a/sim/simx/mem/ptw.h b/sim/simx/mem/ptw.h new file mode 100644 index 0000000000..7825cbe3e3 --- /dev/null +++ b/sim/simx/mem/ptw.h @@ -0,0 +1,117 @@ +// Copyright © 2019-2025 +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 + +#pragma once + +#include +#include + +#ifdef VX_CFG_VM_ENABLE + +#include +#include +#include "../types.h" + +namespace vortex { + +// TLB miss request / fill response between the per-core MMUs and the shared +// page-table walker. Mirrors hw/rtl/mem/VX_ptw_bus_if.sv; `tag` is owned by +// the requesting MMU (its TLB bank index). +struct PtwReq { + uint64_t vpn = 0; + uint64_t root_ppn = 0; + uint32_t tag = 0; +}; + +struct PtwRsp { + uint64_t ppn = 0; // 4 KB-resolved + uint8_t level = 0; + uint8_t flags = 0; + bool fault = false; + uint32_t tag = 0; +}; + +// Device-level shared page-table walker: one instance per processor, fed by +// every core MMU. Up to VX_CFG_PTW_NUM_WALKERS walks proceed concurrently; +// PTE fetches go out on a dedicated L3 port. Non-leaf entries of the upper +// levels are cached in direct-mapped page-walk caches so a walk can start +// one (Sv32) or two (Sv39) levels below the root. +// Mirrors hw/rtl/mem/VX_mmu_ptw.sv + VX_mmu_pwc.sv. +class Ptw : public SimObject { +public: + using Ptr = std::shared_ptr; + + struct PerfStats { + uint64_t walks = 0; + uint64_t latency = 0; // sum of per-walk latencies + uint64_t pwc1_hits = 0; + uint64_t pwc1_misses = 0; + uint64_t pwc2_hits = 0; + uint64_t pwc2_misses = 0; + }; + + // One request/response pair per client MMU. + std::vector> ReqIn; + std::vector> RspOut; + + // Dedicated L3 core port. + SimChannel MemReqOut; + SimChannel MemRspIn; + + Ptw(const SimContext& ctx, const char* name, uint32_t num_clients); + ~Ptw(); + + // Drop the page-walk caches (page tables are about to change). + void flush(); + + const PerfStats& perf_stats() const { return perf_; } + +protected: + void on_reset(); + void on_tick(); + +private: + struct Slot { + enum State { IDLE, MEM_REQ, MEM_RSP, DONE }; + State state = IDLE; + uint64_t vpn = 0; + uint64_t root_ppn = 0; + uint64_t cur_ppn = 0; // table being walked, then the leaf PPN + uint8_t level = 0; + uint8_t flags = 0; + bool fault = false; + uint32_t client = 0; + uint32_t tag = 0; + uint64_t start_cycle = 0; + bool stale = false; // started before the last flush; must not seed the PWCs + }; + + struct PwcEntry { + bool valid = false; + uint64_t key = 0; + uint64_t ppn = 0; + }; + + bool pwc_lookup(const std::vector& pwc, uint64_t key, uint64_t* ppn) const; + void pwc_fill(std::vector& pwc, uint64_t key, uint64_t ppn); + uint64_t vpn_slice(uint64_t vpn, uint32_t level) const; + + uint32_t num_clients_; + std::vector slots_; + std::vector pwc1_; + std::vector pwc2_; + uint32_t client_rr_ = 0; + uint32_t mem_rr_ = 0; + uint32_t done_rr_ = 0; + PerfStats perf_; + + friend class SimObject; +}; + +} // namespace vortex + +#endif // VX_CFG_VM_ENABLE diff --git a/sim/simx/processor.cpp b/sim/simx/processor.cpp index 0b5cf16df9..2a29aa8936 100644 --- a/sim/simx/processor.cpp +++ b/sim/simx/processor.cpp @@ -13,6 +13,7 @@ #include "processor.h" #include "processor_impl.h" +#include "core.h" #include #include @@ -61,6 +62,7 @@ ProcessorImpl::ProcessorImpl() // create L3 cache; when L3 is enabled it is the LLC, otherwise it is a // transparent bypass arbiter and the L2 (or L1) is the LLC. + constexpr uint32_t l3_num_inputs = VX_CFG_L3_NUM_REQS; l3cache_ = Cache::Create("l3cache", Cache::Config{ !VX_CFG_L3_ENABLED, log2ceil(VX_CFG_L3_SIZE), // C @@ -70,7 +72,7 @@ ProcessorImpl::ProcessorImpl() log2ceil(VX_CFG_L3_NUM_WAYS), // A log2ceil(VX_CFG_L3_NUM_BANKS), // B VX_CFG_XLEN, // address bits - VX_CFG_L3_NUM_REQS, // request size + (uint8_t)l3_num_inputs, // request size VX_CFG_L3_MEM_PORTS, // memory ports VX_CFG_L3_WRITEBACK, // write-back false, // write response @@ -114,6 +116,27 @@ ProcessorImpl::ProcessorImpl() memsim_->mem_rsp_out.at(i).bind(&l3cache_->mem_rsp_in.at(i)); } +#ifdef VX_CFG_VM_ENABLE + // shared page-table walker on its dedicated L3 port (the last L3 + // requestor slot; see VX_CFG_L3_NUM_REQS) + // a cluster instantiates NUM_SOCKETS * SOCKET_SIZE cores (rounded up) + constexpr uint32_t CORES_PER_CLUSTER = NUM_SOCKETS * VX_CFG_SOCKET_SIZE; + ptw_ = Ptw::Create("ptw", VX_CFG_NUM_CLUSTERS * CORES_PER_CLUSTER * 2); + constexpr uint32_t L3_PTW_IDX = VX_CFG_L3_NUM_REQS - 1; + ptw_->MemReqOut.bind(&l3cache_->core_req_in.at(L3_PTW_IDX)); + l3cache_->core_rsp_out.at(L3_PTW_IDX).bind(&ptw_->MemRspIn); + for (uint32_t i = 0; i < VX_CFG_NUM_CLUSTERS; ++i) { + for (uint32_t j = 0; j < CORES_PER_CLUSTER; ++j) { + auto core = clusters_.at(i)->get_core(j); + uint32_t client = (i * CORES_PER_CLUSTER + j) * 2; + for (uint32_t k = 0; k < 2; ++k) { + core->ptw_req_out.at(k).bind(&ptw_->ReqIn.at(client + k)); + ptw_->RspOut.at(client + k).bind(&core->ptw_rsp_in.at(k)); + } + } + } +#endif + // set up memory profiling for (uint32_t i = 0; i < VX_CFG_L3_MEM_PORTS; ++i) { memsim_->mem_req_in.at(i).tx_callback([&](const MemReq& req, uint64_t cycle){ @@ -154,6 +177,9 @@ void ProcessorImpl::attach_ram(RAM* ram) { } void ProcessorImpl::flush_caches() { +#ifdef VX_CFG_VM_ENABLE + ptw_->flush(); +#endif // Cache hierarchy is drained inside-out: issue all L1 flush_begin() calls // up-front so icache, dcache, and graphics caches flush in parallel, then // tick until all surfaces report flush_done(). @@ -161,6 +187,12 @@ void ProcessorImpl::flush_caches() { // L1 surfaces: dcache + icache + graphics caches. // Write-through surfaces early-exit in Cache::flush_begin(). for (auto& cluster : clusters_) { +#ifdef VX_CFG_VM_ENABLE + // page tables may change after this flush; drop the cached translations + for (uint32_t c = 0; c < NUM_SOCKETS * VX_CFG_SOCKET_SIZE; ++c) { + cluster->get_core(c)->mmu_flush(); + } +#endif cluster->dcache_flush_begin(); cluster->icache_flush_begin(); #ifdef VX_CFG_EXT_TEX_ENABLE @@ -334,6 +366,9 @@ ProcessorImpl::PerfStats ProcessorImpl::perf_stats() const { perf.mem_latency = perf_mem_latency_; perf.l3cache = l3cache_->perf_stats(); perf.memsim = memsim_->perf_stats(); +#ifdef VX_CFG_VM_ENABLE + perf.ptw = ptw_->perf_stats(); +#endif return perf; } diff --git a/sim/simx/processor_impl.h b/sim/simx/processor_impl.h index ea4120f74e..e4b23913e6 100644 --- a/sim/simx/processor_impl.h +++ b/sim/simx/processor_impl.h @@ -15,6 +15,9 @@ #include "memory.h" #include "cache.h" +#ifdef VX_CFG_VM_ENABLE +#include "mem/ptw.h" +#endif #include "constants.h" #include "cluster.h" #include "kmu.h" @@ -29,6 +32,9 @@ class ProcessorImpl { uint64_t mem_reads = 0; uint64_t mem_writes = 0; uint64_t mem_latency = 0; +#ifdef VX_CFG_VM_ENABLE + Ptw::PerfStats ptw; +#endif }; ProcessorImpl(); @@ -82,6 +88,9 @@ class ProcessorImpl { Memory::Ptr memsim_; RAM* ram_ = nullptr; // functional backing store (set by attach_ram) Cache::Ptr l3cache_; +#ifdef VX_CFG_VM_ENABLE + Ptw::Ptr ptw_; +#endif uint64_t perf_mem_reads_; uint64_t perf_mem_writes_; uint64_t perf_mem_latency_; diff --git a/sim/simx/types.h b/sim/simx/types.h index 9b3253ce31..e490b2b8b5 100644 --- a/sim/simx/types.h +++ b/sim/simx/types.h @@ -1463,7 +1463,7 @@ class TxArbiter : public SimObject> { , Inputs(num_inputs, this) , Outputs(num_outputs, this) , delay_(delay) - , lg2_num_reqs_(log2ceil(num_inputs / num_outputs)) + , lg2_num_reqs_(log2ceil((num_inputs + num_outputs - 1) / num_outputs)) , arbiters_(num_outputs, {type, 1u << lg2_num_reqs_}) { assert(num_inputs <= 64); @@ -1692,7 +1692,7 @@ class TxRxArbiter : public SimObject> { , RspIn(num_outputs, this) , arbiter_(nullptr) , rsp_delay_(rsp_delay) - , lg2_num_reqs_(log2ceil(num_inputs / num_outputs)) + , lg2_num_reqs_(log2ceil((num_inputs + num_outputs - 1) / num_outputs)) { if (num_inputs != num_outputs) { arbiter_ = ReqArb::Create(name, type, num_inputs, num_outputs, req_delay); diff --git a/sim/xrtsim/Makefile b/sim/xrtsim/Makefile index 48fe5d0976..39c9d6c839 100644 --- a/sim/xrtsim/Makefile +++ b/sim/xrtsim/Makefile @@ -22,6 +22,7 @@ LDFLAGS += -Wl,-rpath,$(THIRD_PARTY_DIR)/ramulator -L$(THIRD_PARTY_DIR)/ramulato # control RTL debug tracing states DBG_TRACE_FLAGS += -DDBG_TRACE_PIPELINE DBG_TRACE_FLAGS += -DDBG_TRACE_MEM +DBG_TRACE_FLAGS += -DDBG_TRACE_MMU DBG_TRACE_FLAGS += -DDBG_TRACE_CACHE DBG_TRACE_FLAGS += -DDBG_TRACE_AFU DBG_TRACE_FLAGS += -DDBG_TRACE_SCOPE diff --git a/sw/runtime/common/device.cpp b/sw/runtime/common/device.cpp index 91e6aab336..93270ad70a 100644 --- a/sw/runtime/common/device.cpp +++ b/sw/runtime/common/device.cpp @@ -301,6 +301,14 @@ vx_result_t Device::cp_init() { new vortex::VMManager(vm_io_.get())); if (vm_mgr_->init() != 0) return VX_ERR_DEVICE_LOST; + // The pinned slab is identity-mapped on demand, so its PA range must + // stay out of the VA allocator — otherwise an ordinary allocation + // could mint a VA inside it and a later VX_MEM_PHYS identity map + // would collide with that mapping. + if (pinned_size_ > 0) { + if (vm_mgr_->reserve_pinned_region(pinned_base_, pinned_size_) != 0) + return VX_ERR_DEVICE_LOST; + } const uint64_t satp = vm_mgr_->satp(); CP_WR(CP_SATP_LO, uint32_t(satp & 0xFFFFFFFFu)); CP_WR(CP_SATP_HI, uint32_t(satp >> 32)); diff --git a/sw/runtime/common/perf.cpp b/sw/runtime/common/perf.cpp index 5fd7a404dc..15ee90b2ac 100644 --- a/sw/runtime/common/perf.cpp +++ b/sw/runtime/common/perf.cpp @@ -573,18 +573,30 @@ extern "C" vx_result_t vx_device_dump_perf(vx_device_h hdevice, FILE *stream) { CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_COALESCER_MISS, core_id, &cm), { return err; }); perf_print_core(stream, core_id, "coalescer: misses=%" PRIu64, cm); } - // VM/MMU (per-core; hardware sums icache + dcache MMU counters). + // VM/MMU (per-core; hardware sums icache + dcache TLB counters). if (vm_enabled) { - uint64_t reads = 0, hits = 0, evicts = 0, walks = 0, lat = 0; - CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_READS, core_id, &reads), { return err; }); - CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_HITS, core_id, &hits), { return err; }); - CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_EVICTS, core_id, &evicts), { return err; }); - CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PTW_WALKS, core_id, &walks), { return err; }); - CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PTW_LATENCY, core_id, &lat), { return err; }); - perf_print_core(stream, core_id, "vm: tlb_reads=%" PRIu64 ", hit=%d%%, evicts=%" PRIu64 ", ptw_walks=%" PRIu64 ", ptw_avg_lat=%.2f", - reads, calc_percent(hits, reads), evicts, walks, safe_div((double)lat, (double)walks)); + uint64_t reads = 0, hits = 0, misses = 0, evicts = 0; + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_READS, core_id, &reads), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_HITS, core_id, &hits), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_MISSES, core_id, &misses), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_TLB_EVICTS, core_id, &evicts), { return err; }); + perf_print_core(stream, core_id, "tlb: reads=%" PRIu64 ", hit=%d%%, misses=%" PRIu64 ", evicts=%" PRIu64, + reads, calc_percent(hits, reads), misses, evicts); } } + // Shared page-table walker (device-level; same value on every core). + if (vm_enabled) { + uint64_t walks = 0, lat = 0, pwc1_h = 0, pwc1_m = 0, pwc2_h = 0, pwc2_m = 0; + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PTW_WALKS, 0, &walks), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PTW_LATENCY, 0, &lat), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PWC1_HITS, 0, &pwc1_h), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PWC1_MISSES, 0, &pwc1_m), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PWC2_HITS, 0, &pwc2_h), { return err; }); + CHECK_ERR(vx_device_mpm_query(hdevice, mpm_class, VX_CSR_MPM_PWC2_MISSES, 0, &pwc2_m), { return err; }); + perf_print(stream, "ptw: walks=%" PRIu64 ", avg_lat=%.2f cyc, pwc1_hit=%d%%, pwc2_hit=%d%%", + walks, safe_div((double)lat, (double)walks), + calc_percent(pwc1_h, pwc1_h + pwc1_m), calc_percent(pwc2_h, pwc2_h + pwc2_m)); + } // Global off-chip memory. { uint64_t r = 0, w = 0, lat = 0, bst = 0; diff --git a/sw/runtime/common/vm.cpp b/sw/runtime/common/vm.cpp index 84ecf652bf..6e77e85b8a 100644 --- a/sw/runtime/common/vm.cpp +++ b/sw/runtime/common/vm.cpp @@ -64,6 +64,15 @@ VMManager::~VMManager() { delete page_table_mem_; } +int VMManager::reserve_pinned_region(uint64_t base, uint64_t size) { + int err = this->virtual_mem_reserve(base, size, 0); + if (err != 0) + return err; + pinned_base_ = base; + pinned_size_ = size; + return 0; +} + int VMManager::virtual_mem_reserve(uint64_t dev_addr, uint64_t size, int /*flags*/) { CHECK_ERR(virtual_mem_->reserve(dev_addr, size), { return err; @@ -329,7 +338,9 @@ int VMManager::install_identity_map(uint64_t addr, uint64_t size) { // (set via mem_access) is the actual permission boundary. constexpr uint32_t IDENTITY_PTE_FLAGS = PTE_V | PTE_R | PTE_W | PTE_X; - (void)virtual_mem_->reserve(addr, size); + if (!this->in_pinned_region(addr, size)) { + (void)virtual_mem_->reserve(addr, size); + } uint64_t cur = addr; uint64_t end = addr + size; diff --git a/sw/runtime/common/vm.h b/sw/runtime/common/vm.h index ea265c9b3d..f13a650af0 100644 --- a/sw/runtime/common/vm.h +++ b/sw/runtime/common/vm.h @@ -80,6 +80,11 @@ class VMManager { uint64_t map_p2v(uint64_t ppn, uint32_t flags); int virtual_mem_reserve(uint64_t dev_addr, uint64_t size, int flags); + // Reserve the identity-mapped pinned slab out of the VA space once; the + // on-demand identity maps installed inside it later must not reserve + // again (the allocator reports the overlap as an error). + int reserve_pinned_region(uint64_t base, uint64_t size); + // Install an identity (VA == PA) mapping covering [addr, addr + size). // Uses megapage PTEs where alignment + size permit, leaf PTEs otherwise. int install_identity_map(uint64_t addr, uint64_t size); @@ -95,6 +100,13 @@ class VMManager { uint64_t satp() const { return satp_ ? satp_->get_satp() : 0; } private: + bool in_pinned_region(uint64_t addr, uint64_t size) const { + return pinned_size_ != 0 && addr >= pinned_base_ && (addr + size) <= (pinned_base_ + pinned_size_); + } + + uint64_t pinned_base_ = 0; + uint64_t pinned_size_ = 0; + uint8_t alloc_page_table(uint64_t* pt_addr); int16_t update_page_table(uint64_t ppn, uint64_t vpn, uint32_t flag, uint8_t leaf_level = 0); diff --git a/tests/regression/Makefile b/tests/regression/Makefile index be60b6363c..6d96bc1f7e 100644 --- a/tests/regression/Makefile +++ b/tests/regression/Makefile @@ -3,7 +3,7 @@ include $(ROOT_DIR)/config.mk # --- master list ------------------------------------------------------ TESTS := \ - basic demo dogfood dropout dotproduct dotproduct2 mstress io_addr \ + basic demo dogfood dropout dotproduct dotproduct2 mstress vm_stress io_addr \ arith printf diverge sort fence \ vecadd vecadd_v1 sgemm sgemmx sgemm_v1 conv3 relu sgemv \ sgemm2 sgemm2_v1 madmax stencil3d raycast bfs jacobi pathfinder \ diff --git a/tests/regression/basic/start.S b/tests/regression/basic/start.S index 6b8fc64571..57e00ed58e 100644 --- a/tests/regression/basic/start.S +++ b/tests/regression/basic/start.S @@ -27,17 +27,17 @@ _start: #ifdef VX_CFG_VM_ENABLE # Each core programs SATP from the runtime-installed page table at - # VX_CFG_PAGE_TABLE_BASE_ADDR. Must run before any user-VA load/store. -#if VX_CFG_VM_ADDR_MODE == SV39 - li t0, VX_CFG_PAGE_TABLE_BASE_ADDR - srli t0, t0, VX_CFG_MEM_PAGE_LOG2_SIZE + # VX_MEM_PAGE_TABLE_BASE_ADDR. Must run before any user-VA load/store. +#if VX_VM_ADDR_MODE == SV39 + li t0, VX_MEM_PAGE_TABLE_BASE_ADDR + srli t0, t0, VX_VM_PAGE_LOG2_SIZE li t1, 1 slli t1, t1, 63 or t0, t0, t1 csrw satp, t0 -#elif VX_CFG_VM_ADDR_MODE == SV32 - li t0, VX_CFG_PAGE_TABLE_BASE_ADDR - srli t0, t0, VX_CFG_MEM_PAGE_LOG2_SIZE +#elif VX_VM_ADDR_MODE == SV32 + li t0, VX_MEM_PAGE_TABLE_BASE_ADDR + srli t0, t0, VX_VM_PAGE_LOG2_SIZE li t1, 1 slli t1, t1, 31 or t0, t0, t1 diff --git a/tests/regression/vm_stress/Makefile b/tests/regression/vm_stress/Makefile new file mode 100644 index 0000000000..b36a07d74d --- /dev/null +++ b/tests/regression/vm_stress/Makefile @@ -0,0 +1,16 @@ +ROOT_DIR := $(realpath ../../..) +include $(ROOT_DIR)/config.mk + +PROJECT := vm_stress + +SRC_DIR := $(VORTEX_HOME)/tests/regression/$(PROJECT) + +SRCS := $(SRC_DIR)/main.cpp + +VX_SRCS := $(SRC_DIR)/kernel.cpp + +OPTS ?= -n256 + +KERNEL_LIB := vortex2 + +include ../common.mk diff --git a/tests/regression/vm_stress/common.h b/tests/regression/vm_stress/common.h new file mode 100644 index 0000000000..a63f80adec --- /dev/null +++ b/tests/regression/vm_stress/common.h @@ -0,0 +1,17 @@ +#ifndef _COMMON_H_ +#define _COMMON_H_ + +#define WORDS_PER_PAGE (4096 / 4) + +typedef struct { + uint32_t num_tasks; + uint32_t pages_per_task; + uint32_t total_pages; + uint32_t stride_pages; + uint32_t phys_words; + uint64_t src_addr; + uint64_t dst_addr; + uint64_t phys_addr; +} kernel_arg_t; + +#endif diff --git a/tests/regression/vm_stress/kernel.cpp b/tests/regression/vm_stress/kernel.cpp new file mode 100644 index 0000000000..39aea718ea --- /dev/null +++ b/tests/regression/vm_stress/kernel.cpp @@ -0,0 +1,20 @@ +#include +#include "common.h" + +// Each task touches one word in each of its pages, with a page stride that +// is odd so consecutive touches land in different TLB banks. The physical +// buffer is identity-mapped (VX_MEM_PHYS) and exercises the superpage path. +__kernel void kernel_main(kernel_arg_t* __UNIFORM__ arg) { + auto src_ptr = reinterpret_cast(arg->src_addr); + auto dst_ptr = reinterpret_cast(arg->dst_addr); + auto phys_ptr = reinterpret_cast(arg->phys_addr); + + uint32_t task_id = blockIdx.x * blockDim.x + threadIdx.x; + uint32_t bias = phys_ptr[task_id % arg->phys_words]; + + for (uint32_t k = 0; k < arg->pages_per_task; ++k) { + uint32_t page = ((task_id * arg->pages_per_task + k) * arg->stride_pages) % arg->total_pages; + uint32_t word = page * WORDS_PER_PAGE + (task_id % WORDS_PER_PAGE); + dst_ptr[word] = src_ptr[word] + bias; + } +} diff --git a/tests/regression/vm_stress/main.cpp b/tests/regression/vm_stress/main.cpp new file mode 100644 index 0000000000..d9633eb3cf --- /dev/null +++ b/tests/regression/vm_stress/main.cpp @@ -0,0 +1,178 @@ +#include +#include +#include +#include +#include +#include "common.h" + +#define RT_CHECK(_expr) \ + do { \ + int _ret = _expr; \ + if (0 == _ret) \ + break; \ + printf("Error: '%s' returned %d!\n", #_expr, (int)_ret); \ + cleanup(); \ + exit(-1); \ + } while (false) + +const char* kernel_file = "kernel.vxbin"; +uint32_t total_pages = 256; +uint32_t phys_words = 64; + +vx_device_h device = nullptr; +vx_buffer_h src_buffer = nullptr; +vx_buffer_h dst_buffer = nullptr; +vx_buffer_h phys_buffer = nullptr; +vx_queue_h queue = nullptr; +vx_module_h module_ = nullptr; +vx_kernel_h kernel = nullptr; +kernel_arg_t kernel_arg = {}; + +static void show_usage() { + std::cout << "Vortex VM stress test." << std::endl; + std::cout << "Usage: [-k: kernel] [-n pages] [-h: help]" << std::endl; +} + +static void parse_args(int argc, char **argv) { + int c; + while ((c = getopt(argc, argv, "n:k:h")) != -1) { + switch (c) { + case 'n': + total_pages = atoi(optarg); + break; + case 'k': + kernel_file = optarg; + break; + case 'h': { + show_usage(); + exit(0); + } break; + default: + show_usage(); + exit(-1); + } + } +} + +void cleanup() { + if (device) { + if (src_buffer) vx_buffer_release(src_buffer); + if (dst_buffer) vx_buffer_release(dst_buffer); + if (phys_buffer) vx_buffer_release(phys_buffer); + if (kernel) vx_kernel_release(kernel); + if (module_) vx_module_release(module_); + if (queue) vx_queue_release(queue); + vx_device_dump_perf(device, stdout); + vx_device_release(device); + } +} + +int main(int argc, char *argv[]) { + parse_args(argc, argv); + + std::cout << "open device connection" << std::endl; + RT_CHECK(vx_device_open(0, &device)); + + vx_queue_info_t qi = { sizeof(qi), nullptr, VX_QUEUE_PRIORITY_NORMAL, 0 }; + RT_CHECK(vx_queue_create(device, &qi, &queue)); + + uint64_t num_cores, num_warps, num_threads; + RT_CHECK(vx_device_query(device, VX_CAPS_NUM_CORES, &num_cores)); + RT_CHECK(vx_device_query(device, VX_CAPS_NUM_WARPS, &num_warps)); + RT_CHECK(vx_device_query(device, VX_CAPS_NUM_THREADS, &num_threads)); + + uint32_t num_tasks = num_cores * num_warps * num_threads; + uint32_t pages_per_task = (total_pages + num_tasks - 1) / num_tasks; + uint32_t num_words = total_pages * WORDS_PER_PAGE; + uint64_t buf_size = uint64_t(num_words) * sizeof(uint32_t); + + kernel_arg.num_tasks = num_tasks; + kernel_arg.pages_per_task = pages_per_task; + kernel_arg.total_pages = total_pages; + // odd stride so consecutive pages land in different TLB banks + kernel_arg.stride_pages = 17; + kernel_arg.phys_words = phys_words; + + std::cout << "pages: " << total_pages << ", tasks: " << num_tasks + << ", pages/task: " << pages_per_task << std::endl; + + std::cout << "allocate device memory" << std::endl; + RT_CHECK(vx_buffer_create(device, buf_size, VX_MEM_READ, &src_buffer)); + RT_CHECK(vx_buffer_address(src_buffer, &kernel_arg.src_addr)); + RT_CHECK(vx_buffer_create(device, buf_size, VX_MEM_WRITE, &dst_buffer)); + RT_CHECK(vx_buffer_address(dst_buffer, &kernel_arg.dst_addr)); + // physical (identity-mapped) buffer — exercises the superpage path + RT_CHECK(vx_buffer_create(device, phys_words * sizeof(uint32_t), + VX_MEM_READ | VX_MEM_PHYS, &phys_buffer)); + RT_CHECK(vx_buffer_address(phys_buffer, &kernel_arg.phys_addr)); + + std::cout << "upload buffers" << std::endl; + std::vector h_src(num_words); + std::vector h_dst(num_words, 0); + std::vector h_phys(phys_words); + for (uint32_t i = 0; i < num_words; ++i) { + h_src[i] = i * 2654435761u; + } + for (uint32_t i = 0; i < phys_words; ++i) { + h_phys[i] = 7 + i; + } + RT_CHECK(vx_enqueue_write(queue, src_buffer, 0, h_src.data(), buf_size, 0, nullptr, nullptr)); + RT_CHECK(vx_enqueue_write(queue, dst_buffer, 0, h_dst.data(), buf_size, 0, nullptr, nullptr)); + RT_CHECK(vx_enqueue_write(queue, phys_buffer, 0, h_phys.data(), phys_words * sizeof(uint32_t), 0, nullptr, nullptr)); + + std::cout << "load kernel module" << std::endl; + RT_CHECK(vx_module_load_file(device, kernel_file, &module_)); + RT_CHECK(vx_module_get_kernel(module_, "main", &kernel)); + + std::cout << "launch kernel" << std::endl; + vx_event_h launch_ev = nullptr, read_ev = nullptr; + { + vx_launch_info_t li = {}; + li.struct_size = sizeof(li); + li.kernel = kernel; + li.args_host = &kernel_arg; + li.args_size = sizeof(kernel_arg); + li.ndim = 1; + li.grid_dim[0] = num_tasks / num_threads; + li.block_dim[0] = num_threads; + RT_CHECK(vx_enqueue_launch(queue, &li, 0, nullptr, &launch_ev)); + } + + std::cout << "download destination buffer" << std::endl; + RT_CHECK(vx_enqueue_read(queue, h_dst.data(), dst_buffer, 0, buf_size, 1, &launch_ev, &read_ev)); + + std::cout << "wait for completion" << std::endl; + RT_CHECK(vx_event_wait_value(read_ev, 1, VX_TIMEOUT_INFINITE)); + vx_event_release(read_ev); + vx_event_release(launch_ev); + + std::cout << "verify result" << std::endl; + int errors = 0; + for (uint32_t t = 0; t < num_tasks; ++t) { + uint32_t bias = h_phys[t % phys_words]; + for (uint32_t k = 0; k < pages_per_task; ++k) { + uint32_t page = ((t * pages_per_task + k) * kernel_arg.stride_pages) % total_pages; + uint32_t word = page * WORDS_PER_PAGE + (t % WORDS_PER_PAGE); + uint32_t ref = h_src[word] + bias; + if (h_dst[word] != ref) { + if (errors < 20) { + printf("*** error: task=%u page=%u word=%u expected=0x%x actual=0x%x\n", + t, page, word, ref, h_dst[word]); + } + ++errors; + } + } + } + + std::cout << "cleanup" << std::endl; + cleanup(); + + if (errors != 0) { + std::cout << "Found " << std::dec << errors << " errors!" << std::endl; + std::cout << "FAILED!" << std::endl; + return errors; + } + + std::cout << "PASSED!" << std::endl; + return 0; +}