docs/install: add xCAT provisioner with rocky10/almalinux10 recipes

components/admin/docs/SPECS/docs.spec

@@ -96,6 +96,8 @@ for recipe_name in \
 	"almalinux10-aarch64-openchami-slurm" \
 	"openeuler24.03-x86_64-warewulf-slurm" \
 	"openeuler24.03-aarch64-warewulf-slurm" \
+	"rocky10-x86_64-xcat-slurm" \
+	"almalinux10-x86_64-xcat-slurm" \
 ; do
 	# Parse: distro-arch-provisioner-scheduler -> distro/arch/provisioner/scheduler
 	distro=$(echo "${recipe_name}" | cut -d- -f1)

docs/install/DESIGN.md

@@ -7,7 +7,7 @@ for OpenHPC installation recipes.
 
 - Make documentation easier to edit and contribute to
 - Normalize variable names and remove duplication
-- Support multiple provisioners (Warewulf, OpenCHAMI, Confluent) and schedulers (Slurm)
+- Support multiple provisioners (Warewulf, OpenCHAMI, Confluent, xCAT) and schedulers (Slurm)
 - Support multiple distros (Rocky, AlmaLinux, openEuler, SLES) and
   architectures (x86_64, aarch64)
 - Generate installation scripts directly from documentation
@@ -20,7 +20,7 @@ for OpenHPC installation recipes.
 A recipe is defined by two files in `recipes/`:
 
 - **`*.conf`** — ordered list of `config/` YAML files to merge
-- **`*.yaml`** — per-recipe overrides (Confluent only; omitted for other recipes)
+- **`*.yaml`** — per-recipe overrides (Confluent and xCAT recipes; omitted when not needed)
 
 The Makefile merges these into a single `build/*.yaml` using `yq` deep
 merge, which mkdoc.py then reads as its input.
@@ -35,8 +35,9 @@ config/provisioner/warewulf.yaml
 config/scheduler/slurm.yaml
 ```
 
-Confluent recipes additionally have a `.yaml` with two per-combination
-overrides that cannot be derived from the config hierarchy:
+Some recipes additionally have a `.yaml` with per-combination overrides that
+cannot be derived from the config hierarchy (Confluent carries `distro_id` and
+`distro_iso_image`; the xCAT recipes carry `distro_iso_image`):
 
 ```yaml
 # recipes/rocky10-x86_64-confluent-slurm.yaml
@@ -68,7 +69,8 @@ config/
 ├── provisioner/
 │   ├── warewulf.yaml            # is_warewulf: true, provisioner_name: "Warewulf"
 │   ├── openchami.yaml           # is_openchami: true, provisioner_name: "OpenCHAMI"
-│   └── confluent.yaml           # is_confluent: true, provisioner_name: "Confluent"
+│   ├── confluent.yaml           # is_confluent: true, provisioner_name: "Confluent"
+│   └── xcat.yaml               # is_xcat: true, provisioner_name: "xCAT"
 └── scheduler/
     └── slurm.yaml               # is_slurm: true, scheduler_name: "Slurm"
 ```
@@ -92,6 +94,25 @@ provisioners have fundamentally different workflows:
 - **Confluent**: boot nodes from Confluent → configure live nodes via nodeshell
 - **OpenCHAMI**: build layered container image (podman + yq) → cloud-init →
   boot nodes
+- **xCAT**: copycds ISO → define nodes → then one of
+  - *stateless*: genimage chroot → customize chroot → packimage → rsetboot/rpower
+  - *stateful*: install base OS to disk → configure live nodes via xdsh
+
+xCAT is a single recipe covering both provisioning modes. Unlike the other
+differences in this document, the mode is chosen at **run time** by
+`${enable_stateful}` (from `input.local`) rather than at build time, so one
+guide and one `recipe.sh` serve both. `templates/provisioner/xcat/` holds the
+shared steps plus the mode-specific ones, named `stateless-*` and `stateful-*`
+and gated with `ohpc_if`.
+
+The mode split is confined to how the compute environment is created and when
+the nodes first boot. Everything after that is shared: a "Select Provisioning
+Mode" section defines shell functions (`compute_exec`, `compute_install`,
+`compute_group_install`, `compute_upgrade`, `compute_clean`) that act on the
+image chroot in stateless mode and on the running nodes via `xdsh` in stateful
+mode, and the shared customization chapters call those helpers unchanged. This
+is why the xCAT column of the macro table below emits a helper call rather than
+a concrete command.
 
 Aggregator templates use `{% include %}` to compose sections:
 
@@ -157,7 +178,7 @@ InfiniBand and OmniPath compute-side go here.
 
 **deploy-*** — Cluster booted; compute nodes provisioned; Slurm started. Scope:
 maintenance-window actions (adding/removing nodes). Most provisioners boot here;
-Confluent boots during `provisioner-confluent`.
+Confluent boots during its `provisioner-*` chapter, as does xCAT in stateful mode.
 
 **dev-tools** — Login-node development tools: compilers, MPI, performance tools,
 third-party libraries.
@@ -217,12 +238,12 @@ yq -i '.packages += {{ packages | tojson }}' \
 These four macros abstract all provisioner differences for compute image
 operations. Templates use them without knowing which provisioner is active:
 
-| Macro | Warewulf | Confluent | OpenCHAMI |
-| ----- | -------- | --------- | --------- |
-| `compute_install(packages)` | `dnf install` in chroot | `nodeshell compute dnf install` | `yq` append to packages array |
-| `compute_sed(regex, file)` | `sed -i` on `$CHROOT/file` | `nodeshell compute sed -i` | `yq` append to cmds array |
-| `compute_echo(string, file)` | `echo` to `$CHROOT/file` | `nodeshell compute echo` | `yq` append to cmds array |
-| `compute_run(cmd)` | `wwctl image exec` | `nodeshell compute` | `yq` append to cmds array |
+| Macro | Warewulf | Confluent | OpenCHAMI | xCAT |
+| ----- | -------- | --------- | --------- | ---- | ------------- |
+| `compute_install(packages)` | `dnf install` in chroot | `nodeshell compute dnf install` | `yq` append to packages array | `compute_install` helper (mode-selected) |
+| `compute_sed(regex, file)` | `sed -i` on `$CHROOT/file` | `nodeshell compute sed -i` | `yq` append to cmds array | `compute_exec "sed -i ..."` |
+| `compute_echo(string, file)` | `echo` to `$CHROOT/file` | `nodeshell compute echo` | `yq` append to cmds array | `compute_exec "echo ..."` |
+| `compute_run(cmd)` | `wwctl image exec` | `nodeshell compute` | `yq` append to cmds array | `compute_exec` |
 
 `head_install(packages)` installs packages on the head node (uses
 `pkg_install`, consistent across provisioners).
@@ -512,10 +533,12 @@ docs/install/
 │   │   ├── provisioner-warewulf.md.j2
 │   │   ├── provisioner-confluent.md.j2
 │   │   ├── provisioner-openchami.md.j2
+│   │   ├── provisioner-xcat.md.j2
 │   │   ├── customize.md.j2
 │   │   ├── deploy-warewulf.md.j2
 │   │   ├── deploy-confluent.md.j2
 │   │   ├── deploy-openchami.md.j2
+│   │   ├── deploy-xcat.md.j2
 │   │   ├── dev-tools.md.j2
 │   │   ├── test.md.j2
 │   │   ├── post.md.j2
@@ -527,7 +550,8 @@ docs/install/
 │   ├── provisioner/
 │   │   ├── warewulf/
 │   │   ├── confluent/
-│   │   └── openchami/
+│   │   ├── openchami/
+│   │   └── xcat/
 │   ├── scheduler/
 │   │   └── slurm/
 │   ├── network/
@@ -551,9 +575,11 @@ docs/install/
 │   ├── el10-aarch64/
 │   ├── oe2403-x86_64/
 │   └── oe2403-aarch64/
-├── recipes/                     # Recipe YAML files (source only)
-│   ├── rocky10-x86_64-warewulf-slurm.yaml
-│   ├── almalinux10-x86_64-warewulf-slurm.yaml
+├── recipes/                     # Recipe definitions (source only)
+│   ├── rocky10-x86_64-warewulf-slurm.conf
+│   ├── almalinux10-x86_64-warewulf-slurm.conf
+│   ├── rocky10-x86_64-xcat-slurm.conf
+│   ├── rocky10-x86_64-xcat-slurm.yaml
 │   └── ...
 └── build/                       # Generated output (gitignored)
     ├── header-includes.tex      # Rendered from pandoc/header-includes.tex.j2
@@ -675,7 +701,7 @@ simple and correct.
 The Makefile injects `vc_revision` and `vc_date` (from `git log`) into
 each `build/*.yaml` via `yq` during the merge step, so mkdoc.py needs
 no subprocess calls. The `.yaml` prerequisite for `build/%.yaml` is
-optional via `.SECONDEXPANSION` — only Confluent recipes have one.
+optional via `.SECONDEXPANSION` — only some recipes (Confluent, xCAT) have one.
 
 ### RPM Packaging
 
@@ -717,10 +743,11 @@ python3 tests/ci/run_build.py $USER components/admin/docs/SPECS/docs.spec
 
 ### Recipe Naming
 
-Recipes are named `{distro}{version}-{arch}-{provisioner}-{scheduler}.yaml`
-and live in `recipes/`. See existing recipes for examples. The 14 current
-recipes cover Warewulf, Confluent, and OpenCHAMI across Rocky, AlmaLinux,
-and openEuler on x86\_64 and aarch64.
+Recipes are named `{distro}{version}-{arch}-{provisioner}-{scheduler}.conf`
+(with an optional `.yaml` override alongside) and live in `recipes/`. See
+existing recipes for examples. The 18 current recipes cover Warewulf,
+Confluent, OpenCHAMI, and xCAT across Rocky, AlmaLinux, and
+openEuler on x86\_64 and aarch64.
 
 ### Manifest Directory Naming
 

docs/install/Makefile

@@ -61,7 +61,7 @@ MANIFESTS     := $(shell find manifests -name '*.md.j2' -o -name '*.md')
 PANDOC_FILES  := $(PANDOC_DIR)/format-filters.lua \
                  $(PANDOC_DIR)/header-includes.tex.j2 \
                  $(PANDOC_DIR)/codeblock-styles.css
-DEPS          := mkdoc.py $(TEMPLATES) $(MANIFESTS) $(PANDOC_FILES)
+DEPS          := mkdoc.py templates/macros.j2 $(TEMPLATES) $(MANIFESTS) $(PANDOC_FILES)
 
 # Manifest directories for manifest/changelog targets
 MANIFEST_DIRS := $(wildcard manifests/*/pkg-ohpc.all)

docs/install/config/base.yaml

@@ -29,6 +29,7 @@ is_aarch64: false
 is_warewulf: false
 is_openchami: false
 is_confluent: false
+is_xcat: false
 
 ## Scheduler
 is_slurm: false

docs/install/config/provisioner/xcat.yaml

@@ -0,0 +1,35 @@
+# xCAT provisioner configuration
+#
+# A single recipe covers both xCAT provisioning modes. The mode is selected at
+# run time by ${xcat_mode} (set in input.local), not at document build
+# time, so one guide and one recipe script serve both:
+#
+#   xcat_mode=stateless  compute nodes boot a RAM-resident image built in a
+#                        chroot on the head node
+#   xcat_mode=stateful   a base OS is installed onto the local disk of each
+#                        compute node, then OpenHPC components are added to
+#                        the running nodes
+#
+# The recipe defines shell helpers (compute_exec, compute_install, ...) once the
+# mode is known; the shared customization chapters call those helpers and work
+# unchanged in both modes.
+
+provisioner: "xcat"
+provisioner_name: "xCAT"
+is_xcat: true
+
+# Compute-image package operations dispatch through the mode-selected shell
+# functions defined in the "Select Provisioning Mode" section.
+pkg_install_chroot: "compute_install"
+pkg_group_install_chroot: "compute_group_install"
+pkg_upgrade_chroot: "compute_upgrade"
+pkg_clean_chroot: "compute_clean"
+
+# xCAT release tree to install from ("latest" tracks the current stable
+# series; set a versioned directory such as "2.18" to pin). The repo
+# definitions are written locally: the .repo files published on xcat.org
+# point at a retired "devel" tree and are not usable directly.
+xcat_release: "latest"
+xcat_repo_base: "https://xcat.org/files/xcat/repos/yum"
+# OS-family component of the xcat-dep repository path (EL10 -> rh10)
+xcat_dep_dist: "rh10"

docs/install/input.local.template

@@ -57,9 +57,12 @@ node_disk=${node_disk:=/dev/sda}
 initialize_options="${initialize_options:-usklpta}"
 deployment_protocols="${deployment_protocols:-firmware}"
 dns_domain="${dns_domain:-local}"
-iso_path="${iso_path:-}"
 iso_url="${iso_url:-}"
 
+# Path to directory containing the base OS dvd iso
+# (Confluent and xCAT recipes; set to the directory, not the full filename)
+iso_path="${iso_path:-}"
+
 # Flags for optional installation/configuration
 enable_ib="${enable_ib:-0}"
 enable_opa="${enable_opa:-0}"
@@ -70,6 +73,10 @@ enable_ipoib="${enable_ipoib:-0}"
 enable_dracut="${enable_dracut:-1}"
 enable_todisk="${enable_todisk:-0}"
 
+# Provisioning mode (xCAT recipe only): "stateless" boots a netboot image,
+# "stateful" installs to the local disk of each compute node
+xcat_mode="${xcat_mode:-stateless}"
+
 enable_clustershell="${enable_clustershell:-0}"
 enable_ipmisol="${enable_ipmisol:-0}"
 enable_opensm="${enable_opensm:-0}"

docs/install/recipes/almalinux10-x86_64-xcat-slurm.conf

@@ -0,0 +1,6 @@
+config/base.yaml
+config/distro/el10.yaml
+config/distro/almalinux.yaml
+config/arch/x86_64.yaml
+config/provisioner/xcat.yaml
+config/scheduler/slurm.yaml

docs/install/recipes/almalinux10-x86_64-xcat-slurm.yaml

@@ -0,0 +1,4 @@
+# The osimage names (and the image directory layout under /install) are
+# derived by copycds from the distribution metadata inside the ISO, which
+# encodes the minor version; the recipe captures them at runtime via lsdef.
+distro_iso_image: "AlmaLinux-10-latest-x86_64-dvd.iso"

docs/install/recipes/rocky10-x86_64-xcat-slurm.conf

@@ -0,0 +1,6 @@
+config/base.yaml
+config/distro/el10.yaml
+config/distro/rocky.yaml
+config/arch/x86_64.yaml
+config/provisioner/xcat.yaml
+config/scheduler/slurm.yaml

docs/install/recipes/rocky10-x86_64-xcat-slurm.yaml

@@ -0,0 +1,4 @@
+# The osimage names (and the image directory layout under /install) are
+# derived by copycds from the distribution metadata inside the ISO, which
+# encodes the minor version; the recipe captures them at runtime via lsdef.
+distro_iso_image: "Rocky-10-latest-x86_64-dvd.iso"

docs/install/templates/appendices/upgrade.md.j2

@@ -36,6 +36,35 @@ updates are pre-configured.
 {{ pkg_clean }}
 {{ pkg_clean_chroot }}
 ```
+{% elif is_xcat %}
+1. (Optional) Ensure repo metadata is current (on the head node and in the
+   compute environment). Package managers will naturally do this on their own
+   over time, but if you are wanting to access updates immediately after a new
+   release, the following can be used to sync to the latest. The compute-side
+   command depends on how the cluster was provisioned. The helper functions
+   defined while installing are not available in a later shell, so the commands
+   are spelled out in full here.
+
+```bash
+{{ pkg_clean }}
+```
+
+   For a stateful cluster, act on the running compute nodes:
+
+```bash
+xdsh compute dnf clean expire-cache
+```
+
+   For a stateless cluster, act on the image chroot, re-deriving its location:
+
+```bash
+# Name the image explicitly if the head node holds more than one
+xcat_osimage=${xcat_osimage:-$(lsdef -t osimage | \
+    awk '/netboot-compute/{print $1; exit}')}
+CHROOT=$(lsdef -t osimage -o ${xcat_osimage} \
+    -i rootimgdir | awk -F= '/rootimgdir/{print $2}')/rootimg/
+dnf --installroot="${CHROOT}" clean expire-cache
+```
 {% else %}
 1. (Optional) Ensure repo metadata is current (on head node and in chroot
    location(s)). Package managers will naturally do this on their own over time,
@@ -93,6 +122,48 @@ apply the changes.
 
 Note that to update running services such as a resource manager, a service
 restart is required on the compute nodes.
+{% elif is_xcat %}
+3. Upgrade packages in the compute environment
+
+   For a stateful cluster, upgrade the running compute nodes. Note that to
+   update running services such as a resource manager, a service restart is
+   required on the compute nodes.
+
+```bash
+xdsh compute dnf -y upgrade "*-ohpc"
+
+# Any new compute-node Base OS provided dependencies can be installed by
+# updating the ohpc-base-compute metapackage
+xdsh compute dnf -y upgrade "ohpc-base-compute"
+```
+
+   For a stateless cluster, upgrade the image chroot, re-deriving its location
+   if it is not already set in the current shell:
+
+```bash
+# Name the image explicitly if the head node holds more than one
+xcat_osimage=${xcat_osimage:-$(lsdef -t osimage | \
+    awk '/netboot-compute/{print $1; exit}')}
+CHROOT=$(lsdef -t osimage -o ${xcat_osimage} \
+    -i rootimgdir | awk -F= '/rootimgdir/{print $2}')/rootimg/
+
+dnf -y --installroot="${CHROOT}" upgrade "*-ohpc"
+
+# Any new compute-node Base OS provided dependencies can be installed by
+# updating the ohpc-base-compute metapackage
+dnf -y --installroot="${CHROOT}" upgrade "ohpc-base-compute"
+```
+
+4. Rebuild and redeploy the stateless image
+
+   This step applies to stateless provisioning only. In stateful mode the
+   upgrade above already ran on the compute nodes and there is no image to
+   rebuild. Reboot the compute nodes when convenient to pick up the new image.
+
+```bash
+packimage ${xcat_osimage}
+nodeset compute osimage=${xcat_osimage}
+```
 {% else %}
 3. Upgrade packages in compute image
 

docs/install/templates/base-os/install.md.j2

@@ -22,7 +22,7 @@ echo ${sms_ip} ${sms_name} >> /etc/hosts
 {% endif %}
 
 While it is theoretically possible to enable SELinux on a cluster provisioned
-with {{ provisioner }}, doing so is beyond the scope of this document. Even the
+with {{ provisioner_name }}, doing so is beyond the scope of this document. Even the
 use of permissive mode can be problematic and we therefore recommend disabling
 SELinux on the *head node*. If SELinux components are installed locally, the
 `selinuxenabled` command can be used to determine if SELinux is currently

docs/install/templates/chapters/deploy-xcat.md.j2

@@ -0,0 +1,24 @@
+{# Deploy Aggregator #}
+# Deploy Cluster
+
+{% include "provisioner/deploy.md.j2" %}
+
+{# Slurm starts on the head node first so that stateless nodes, which boot
+   below, find the controller on their first try, and so that stateful nodes,
+   which are already running, have a controller to register with. #}
+{% include "scheduler/slurm/startup.md.j2" %}
+
+## Stateless Deployment
+
+Follow this section when `${xcat_mode}` is `0`. The customized image is
+packed and the compute nodes boot it over the network.
+
+{% include "provisioner/xcat/deploy-stateless-pack.md.j2" %}
+{% include "provisioner/xcat/deploy-stateless-boot.md.j2" %}
+
+## Stateful Deployment
+
+Follow this section when `${xcat_mode}` is `1`. The compute nodes are
+already running, so only their client daemons need starting.
+
+{% include "provisioner/xcat/deploy-stateful.md.j2" %}