Reconstruct STM32 SVD files from CMSIS headers

.github/workflows/deploy-docs.yml

@@ -21,7 +21,13 @@ jobs:
 
       - name: Install modm-data with docs dependencies
         run: |
-          pip3 install -r tools/requirements.txt ".[docs]"
+          pip3 install -r tools/requirements.txt -e ".[docs]"
+
+      - name: Install ARM toolchain and sources for the SVD files
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y gcc-arm-none-eabi
+          make clone-sources-arm ext/stmicro/header/ ext/stmicro/cubehal/
 
 
       - name: Clone modm-ext/data.modm.io repository

.github/workflows/test-svd.yml

@@ -0,0 +1,67 @@
+name: Generate SVD Files
+
+on:
+  workflow_dispatch:
+  pull_request:
+  release:
+    types: [published]
+
+jobs:
+  generate-svd-stmicro:
+    name: Generate STM32 SVD Files
+    runs-on: ubuntu-latest
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+      cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}
+    steps:
+      - name: Check out Repository
+        uses: actions/checkout@v4
+
+      - name: Setup Python 3.12
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.12"
+
+      - name: Install Python Dependencies
+        run: |
+          pip3 install -r tools/requirements.txt -e .
+
+      - name: Install ARM Toolchain
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y gcc-arm-none-eabi
+
+      - name: Clone STMicro Sources
+        run: |
+          make clone-sources-arm ext/stmicro/header/ ext/stmicro/cubehal/
+
+      - name: Generate SVD Files
+        run: |
+          python3 -m modm_data.header2svd.stmicro --all --output svd-archive/
+
+      - name: Upload SVD Logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: svd-logs
+          path: log/stmicro/svd
+          retention-days: 1
+
+      - name: Upload SVD Files
+        uses: actions/upload-artifact@v4
+        with:
+          name: svd-archive
+          path: svd-archive/
+          retention-days: 7
+
+      - name: Zip SVD Files
+        if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
+        run: |
+          zip -r svd-archive.zip svd-archive/
+
+      - name: Upload to Github Release
+        uses: softprops/action-gh-release@v2
+        if: ${{ github.event_name == 'release' && github.event.action == 'published' }}
+        with:
+          files: |
+            svd-archive.zip

.github/workflows/test.yml

@@ -40,12 +40,14 @@ jobs:
         if: always()
         uses: chartboost/ruff-action@v1
         with:
+          version: "0.14.10"
           src: "./src"
           args: "format --check"
 
       - name: Run Ruff Linter
         if: always()
         uses: chartboost/ruff-action@v1
         with:
+          version: "0.14.10"
           src: "./src"
           args: "check"

docs/.gitignore

@@ -1,6 +1,7 @@
 # ignored since these are generated by tools/scripts/synchronize_docs.py
 src/api/
 src/pinout/
+src/svd/
 src/index.md
 src/source/overview.md
 src/pipeline/overview.md

docs/mkdocs.yml

@@ -89,3 +89,4 @@ nav:
       - Header to SVD: pipeline/header2svd.md
   - API Reference: api/modm_data.html
   - Pinouts: pinout/index.html
+  - SVD Explorer: svd/index.html

docs/src/pipeline/header2svd.md

@@ -1,15 +1,55 @@
 # CMSIS Header to SVD Pipeline
 
+The CMSIS device headers are compiled with `arm-none-eabi-gcc` to extract the
+numeric values of all macros and the layout of all peripheral structures. The
+bit field macros are then matched to the structure members to reconstruct the
+memory map of each device header, see `modm_data.header2svd.stmicro.memory_map`.
+
+The CMSIS headers are more accurate than the ST SVD files, since they are
+compiled and used by the HAL. The ST SVD files are therefore only used to find
+discrepancies, which need to be checked with the reference manual.
+
 ## Selective Conversion
 
-The resulting SVD files are found in `ext/stmicro/svd`.
-Only takes a few minutes.
+The resulting SVD files are found in `ext/stmicro/svd/header_*.svd` and the
+reports in `log/stmicro/svd/header_*.txt`. The extracted header data is cached
+in `ext/cache/cmsis/header2svd`.
+
+```bash
+# Convert all headers matching the pattern into SVD files
+python3 -m modm_data.header2svd.stmicro --header stm32f4
+# Convert all CMSIS headers and compare them with the ST SVD files
+python3 -m modm_data.header2svd.stmicro --all --compare
+```
+
+The CubeHAL source code in `ext/stmicro/cubehal` provides additional information,
+see `modm_data.header2svd.stmicro.cubehal`:
+
+- Register accesses like `SET_BIT(USARTx->CR1, USART_CR1_UE)` pair registers
+  with bit field macros when the naming heuristics fail.
+- The `IS_*_INSTANCE` macros documented in the LL functions remove the bit
+  fields and registers that are not supported by an instance of a shared
+  structure type, for example, the break and dead-time register of basic timers.
+- The LL function parameters are evaluated as enumerated values of bit fields.
+
+The interrupts and the descriptions of peripherals, registers and bit fields
+are taken from the CMSIS header. Overlapping bit fields with a different layout
+are placed into alternate registers, for example, the input capture bit fields
+of `TIM_CCMR1` in `CCMR1_ALT`.
+
+The report lists the bit field macros that could not be assigned to a register,
+the registers without bit fields, the overlapping bit fields that were removed,
+the alternate registers, the registers paired by CubeHAL, the bit fields not
+supported by an instance, the unassigned interrupts and the differences to the
+ST SVD file.
+
+The SVD files are also published on the homepage with the
+[SVD Explorer](https://gist.github.com/salkinium/12a18032caa303697c6583937f6fcd16),
+which lists them from the `svd-files.json` file:
 
 ```bash
-# Convert a group of devices into SVD files
-python3 -m modm_data.header2svd.stmicro --device stm32f030c6t6 --device stm32f030f4p6 --device stm32f030k6t6
-# Convert all CMSIS headers into SVD files
-python3 -m modm_data.header2svd.stmicro --all
+# Convert all CMSIS headers into the documentation folder
+python3 -m modm_data.header2svd.stmicro --all --output docs/src/svd/
 ```
 
 ## Automatic Conversion
@@ -20,5 +60,5 @@ To perform the steps automatically, you may also use `make`:
 # Using make
 make convert-stmicro-header-svd
 # Remove all svd files
-make clean-stmicro-svd
+make clean-stmicro-header-svd
 ```

pyproject.toml

@@ -28,6 +28,7 @@ classifiers = [
 dependencies = [
     "anytree==2.12.1",
     "CppHeaderParser>=2.7.4,<3",
+    "cxxheaderparser>=1.4.1,<2",
     "jinja2>=3.1.3,<4",
     "lxml>=5.2.0,<6",
     "kuzu>=0.11.3,<0.12",
@@ -47,7 +48,7 @@ docs = [
     "pdoc>=14,<15",
     "mkdocs>=1.5,<2",
     "mkdocs-material>=9.5,<10",
-    "modm-pinout @ git+https://github.com/modm-io/modm-pinout.git@main",
+    "modm-pinout @ git+https://github.com/modm-ext/modm-pinout.git@main",
 ]
 all = ["modm_data[docs]"]
 

src/modm_data/cubehal/__init__.py

@@ -11,5 +11,16 @@
 """
 
 from .dmamux_requests import read_request_map, read_bdma_request_map
+from .header import read_header
+from .registers import RegisterAccess, LLFunction, register_accesses, ll_functions, ll_descriptions
 
-__all__ = ["read_request_map", "read_bdma_request_map"]
+__all__ = [
+    "read_request_map",
+    "read_bdma_request_map",
+    "read_header",
+    "RegisterAccess",
+    "LLFunction",
+    "register_accesses",
+    "ll_functions",
+    "ll_descriptions",
+]