From f9c0647567d3351f5f0574ddaa8fe198ea384298 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Vin=C3=ADcius=20Ferr=C3=A3o?= Date: Sun, 6 Sep 2026 23:16:17 -0300 Subject: [PATCH] docs/install: add xCAT provisioner with rocky10/almalinux10 recipes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds xCAT to the Markdown recipe system, covering both of its provisioning modes in a single guide. The mode is chosen at run time by ${xcat_mode} in input.local rather than at document build time, so one guide and one recipe.sh serve both, matching how the Warewulf recipe handles provision-to-disk: stateless compute nodes boot a RAM-resident image built in a chroot on the head node stateful a base OS is installed to each node's local disk, then OpenHPC components are added to the running nodes A "Select Provisioning Mode" section defines shell functions -- compute_exec, compute_install, compute_group_install, compute_upgrade, compute_clean -- that act on the image chroot when stateless and on the running nodes via xdsh when stateful. The shared customization chapters call those helpers and are otherwise untouched, so the InfiniBand, Omni-Path, memlock, GPU, syslog, oneAPI and admin-tool sections are written once rather than once per mode. Functions rather than variables because ${CHROOT} is only known after genimage runs, and a function re-resolves it at call time. Mode-specific steps live in templates/provisioner/xcat/ as stateless-* and stateful-*, gated with ohpc_if so the generated script takes one path. Both forked chapters group that work under a single heading -- "Stateless Provisioning"/"Stateful Provisioning" and "Stateless Deployment"/"Stateful Deployment" -- following the same aggregator-owns-the-grouping-header pattern as "Cluster Admin Tools", so a reader follows one group and skips the other outright. Nothing mode-specific sits outside those headings. The deploy chapter starts Slurm on the head node first. Stateless nodes boot afterwards and their slurmd reaches the controller on the first try, while the already-running stateful nodes are then told to start theirs. Details worth noting for review: - The mode is a value, not a boolean. Every other input.local flag named enable_* turns an optional feature on or off, but this selects a mandatory mode, so it is spelled xcat_mode=stateless|stateful rather than a lone enable_stateful with no counterpart. It fails closed: anything else aborts, so a half-configured input.local cannot quietly netboot a fleet meant to install to disk. - Image lookup refuses to guess. A head node holding images for several distributions or releases matches more than one, so the recipe stops unless exactly one matches or ${xcat_osimage} names it, and the chosen image is checked against the mode by its provmethod before anything is modified. - The stateful wait requires every node to positively report a finished boot and to have lost a marker written to /run before the reset, so neither an unreachable node nor the system being replaced counts as ready. It gives up after ${install_timeout} rather than waiting forever on a failed install. It proves a reboot, not a successful install: a failed PXE attempt falling back to the existing disk would pass. Proving the install would need a per-run token written from the kickstart. - The upgrade appendix spells out xdsh and --installroot commands per mode and re-derives ${CHROOT}, because the helper functions above exist only in the installing shell. Both modes were deployed end to end on AlmaLinux 10.1 with xCAT 2.18.2 and two compute nodes: the recipe ran start to finish in each, nodes provisioned to tmpfs and to local disk respectively, Slurm showed both nodes idle, and a two-rank MPI job ran across them. Signed-off-by: Vinícius Ferrão --- components/admin/docs/SPECS/docs.spec | 2 + docs/install/DESIGN.md | 69 +++++++---- docs/install/Makefile | 2 +- docs/install/config/base.yaml | 1 + docs/install/config/provisioner/xcat.yaml | 35 ++++++ docs/install/input.local.template | 9 +- .../almalinux10-x86_64-xcat-slurm.conf | 6 + .../almalinux10-x86_64-xcat-slurm.yaml | 4 + .../recipes/rocky10-x86_64-xcat-slurm.conf | 6 + .../recipes/rocky10-x86_64-xcat-slurm.yaml | 4 + .../templates/appendices/upgrade.md.j2 | 71 ++++++++++++ docs/install/templates/base-os/install.md.j2 | 2 +- .../templates/chapters/deploy-xcat.md.j2 | 24 ++++ .../templates/chapters/provisioner-xcat.md.j2 | 45 ++++++++ docs/install/templates/intro/inputs.md.j2 | 6 + docs/install/templates/intro/intro.md.j2 | 2 +- .../templates/intro/requirements.md.j2 | 4 +- docs/install/templates/macros.j2 | 24 ++++ .../provisioner/xcat/add-nodes.md.j2 | 63 ++++++++++ .../provisioner/xcat/deploy-stateful.md.j2 | 28 +++++ .../xcat/deploy-stateless-boot.md.j2 | 54 +++++++++ .../xcat/deploy-stateless-pack.md.j2 | 16 +++ .../provisioner/xcat/mode-select.md.j2 | 58 ++++++++++ .../provisioner/xcat/osimage-init.md.j2 | 66 +++++++++++ .../provisioner/xcat/post-add-user.md.j2 | 13 +++ .../xcat/stateful-diskconfig.md.j2 | 26 +++++ .../provisioner/xcat/stateful-install.md.j2 | 109 ++++++++++++++++++ .../provisioner/xcat/stateful-nfs.md.j2 | 29 +++++ .../provisioner/xcat/stateful-ohpc.md.j2 | 52 +++++++++ .../provisioner/xcat/stateful-setup.md.j2 | 55 +++++++++ .../provisioner/xcat/stateful-slurm.md.j2 | 33 ++++++ .../xcat/stateless-customize.md.j2 | 26 +++++ .../provisioner/xcat/stateless-genimage.md.j2 | 20 ++++ .../provisioner/xcat/stateless-ohpc.md.j2 | 54 +++++++++ .../provisioner/xcat/synclists.md.j2 | 29 +++++ .../provisioner/xcat/xcat-install.md.j2 | 22 ++++ .../provisioner/xcat/xcat-repo.md.j2 | 32 +++++ .../provisioner/xcat/xcat-setup.md.j2 | 31 +++++ .../templates/scheduler/slurm/test-job.md.j2 | 6 +- 39 files changed, 1109 insertions(+), 29 deletions(-) create mode 100644 docs/install/config/provisioner/xcat.yaml create mode 100644 docs/install/recipes/almalinux10-x86_64-xcat-slurm.conf create mode 100644 docs/install/recipes/almalinux10-x86_64-xcat-slurm.yaml create mode 100644 docs/install/recipes/rocky10-x86_64-xcat-slurm.conf create mode 100644 docs/install/recipes/rocky10-x86_64-xcat-slurm.yaml create mode 100644 docs/install/templates/chapters/deploy-xcat.md.j2 create mode 100644 docs/install/templates/chapters/provisioner-xcat.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/add-nodes.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/deploy-stateful.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/deploy-stateless-boot.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/deploy-stateless-pack.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/mode-select.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/osimage-init.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/post-add-user.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-diskconfig.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-install.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-nfs.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-ohpc.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-setup.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateful-slurm.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateless-customize.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateless-genimage.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/stateless-ohpc.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/synclists.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/xcat-install.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/xcat-repo.md.j2 create mode 100644 docs/install/templates/provisioner/xcat/xcat-setup.md.j2 diff --git a/components/admin/docs/SPECS/docs.spec b/components/admin/docs/SPECS/docs.spec index 206bbdc044..256154c53a 100644 --- a/components/admin/docs/SPECS/docs.spec +++ b/components/admin/docs/SPECS/docs.spec @@ -96,6 +96,8 @@ for recipe_name in \ "almalinux10-aarch64-openchami-slurm" \ "openeuler24.03-x86_64-warewulf-slurm" \ "openeuler24.03-aarch64-warewulf-slurm" \ + "rocky10-x86_64-xcat-slurm" \ + "almalinux10-x86_64-xcat-slurm" \ ; do # Parse: distro-arch-provisioner-scheduler -> distro/arch/provisioner/scheduler distro=$(echo "${recipe_name}" | cut -d- -f1) diff --git a/docs/install/DESIGN.md b/docs/install/DESIGN.md index e95c55b0da..3d45d4325c 100644 --- a/docs/install/DESIGN.md +++ b/docs/install/DESIGN.md @@ -7,7 +7,7 @@ for OpenHPC installation recipes. - Make documentation easier to edit and contribute to - Normalize variable names and remove duplication -- Support multiple provisioners (Warewulf, OpenCHAMI, Confluent) and schedulers (Slurm) +- Support multiple provisioners (Warewulf, OpenCHAMI, Confluent, xCAT) and schedulers (Slurm) - Support multiple distros (Rocky, AlmaLinux, openEuler, SLES) and architectures (x86_64, aarch64) - Generate installation scripts directly from documentation @@ -20,7 +20,7 @@ for OpenHPC installation recipes. A recipe is defined by two files in `recipes/`: - **`*.conf`** — ordered list of `config/` YAML files to merge -- **`*.yaml`** — per-recipe overrides (Confluent only; omitted for other recipes) +- **`*.yaml`** — per-recipe overrides (Confluent and xCAT recipes; omitted when not needed) The Makefile merges these into a single `build/*.yaml` using `yq` deep merge, which mkdoc.py then reads as its input. @@ -35,8 +35,9 @@ config/provisioner/warewulf.yaml config/scheduler/slurm.yaml ``` -Confluent recipes additionally have a `.yaml` with two per-combination -overrides that cannot be derived from the config hierarchy: +Some recipes additionally have a `.yaml` with per-combination overrides that +cannot be derived from the config hierarchy (Confluent carries `distro_id` and +`distro_iso_image`; the xCAT recipes carry `distro_iso_image`): ```yaml # recipes/rocky10-x86_64-confluent-slurm.yaml @@ -68,7 +69,8 @@ config/ ├── provisioner/ │ ├── warewulf.yaml # is_warewulf: true, provisioner_name: "Warewulf" │ ├── openchami.yaml # is_openchami: true, provisioner_name: "OpenCHAMI" -│ └── confluent.yaml # is_confluent: true, provisioner_name: "Confluent" +│ ├── confluent.yaml # is_confluent: true, provisioner_name: "Confluent" +│ └── xcat.yaml # is_xcat: true, provisioner_name: "xCAT" └── scheduler/ └── slurm.yaml # is_slurm: true, scheduler_name: "Slurm" ``` @@ -92,6 +94,25 @@ provisioners have fundamentally different workflows: - **Confluent**: boot nodes from Confluent → configure live nodes via nodeshell - **OpenCHAMI**: build layered container image (podman + yq) → cloud-init → boot nodes +- **xCAT**: copycds ISO → define nodes → then one of + - *stateless*: genimage chroot → customize chroot → packimage → rsetboot/rpower + - *stateful*: install base OS to disk → configure live nodes via xdsh + +xCAT is a single recipe covering both provisioning modes. Unlike the other +differences in this document, the mode is chosen at **run time** by +`${enable_stateful}` (from `input.local`) rather than at build time, so one +guide and one `recipe.sh` serve both. `templates/provisioner/xcat/` holds the +shared steps plus the mode-specific ones, named `stateless-*` and `stateful-*` +and gated with `ohpc_if`. + +The mode split is confined to how the compute environment is created and when +the nodes first boot. Everything after that is shared: a "Select Provisioning +Mode" section defines shell functions (`compute_exec`, `compute_install`, +`compute_group_install`, `compute_upgrade`, `compute_clean`) that act on the +image chroot in stateless mode and on the running nodes via `xdsh` in stateful +mode, and the shared customization chapters call those helpers unchanged. This +is why the xCAT column of the macro table below emits a helper call rather than +a concrete command. Aggregator templates use `{% include %}` to compose sections: @@ -157,7 +178,7 @@ InfiniBand and OmniPath compute-side go here. **deploy-*** — Cluster booted; compute nodes provisioned; Slurm started. Scope: maintenance-window actions (adding/removing nodes). Most provisioners boot here; -Confluent boots during `provisioner-confluent`. +Confluent boots during its `provisioner-*` chapter, as does xCAT in stateful mode. **dev-tools** — Login-node development tools: compilers, MPI, performance tools, third-party libraries. @@ -217,12 +238,12 @@ yq -i '.packages += {{ packages | tojson }}' \ These four macros abstract all provisioner differences for compute image operations. Templates use them without knowing which provisioner is active: -| Macro | Warewulf | Confluent | OpenCHAMI | -| ----- | -------- | --------- | --------- | -| `compute_install(packages)` | `dnf install` in chroot | `nodeshell compute dnf install` | `yq` append to packages array | -| `compute_sed(regex, file)` | `sed -i` on `$CHROOT/file` | `nodeshell compute sed -i` | `yq` append to cmds array | -| `compute_echo(string, file)` | `echo` to `$CHROOT/file` | `nodeshell compute echo` | `yq` append to cmds array | -| `compute_run(cmd)` | `wwctl image exec` | `nodeshell compute` | `yq` append to cmds array | +| Macro | Warewulf | Confluent | OpenCHAMI | xCAT | +| ----- | -------- | --------- | --------- | ---- | ------------- | +| `compute_install(packages)` | `dnf install` in chroot | `nodeshell compute dnf install` | `yq` append to packages array | `compute_install` helper (mode-selected) | +| `compute_sed(regex, file)` | `sed -i` on `$CHROOT/file` | `nodeshell compute sed -i` | `yq` append to cmds array | `compute_exec "sed -i ..."` | +| `compute_echo(string, file)` | `echo` to `$CHROOT/file` | `nodeshell compute echo` | `yq` append to cmds array | `compute_exec "echo ..."` | +| `compute_run(cmd)` | `wwctl image exec` | `nodeshell compute` | `yq` append to cmds array | `compute_exec` | `head_install(packages)` installs packages on the head node (uses `pkg_install`, consistent across provisioners). @@ -512,10 +533,12 @@ docs/install/ │ │ ├── provisioner-warewulf.md.j2 │ │ ├── provisioner-confluent.md.j2 │ │ ├── provisioner-openchami.md.j2 +│ │ ├── provisioner-xcat.md.j2 │ │ ├── customize.md.j2 │ │ ├── deploy-warewulf.md.j2 │ │ ├── deploy-confluent.md.j2 │ │ ├── deploy-openchami.md.j2 +│ │ ├── deploy-xcat.md.j2 │ │ ├── dev-tools.md.j2 │ │ ├── test.md.j2 │ │ ├── post.md.j2 @@ -527,7 +550,8 @@ docs/install/ │ ├── provisioner/ │ │ ├── warewulf/ │ │ ├── confluent/ -│ │ └── openchami/ +│ │ ├── openchami/ +│ │ └── xcat/ │ ├── scheduler/ │ │ └── slurm/ │ ├── network/ @@ -551,9 +575,11 @@ docs/install/ │ ├── el10-aarch64/ │ ├── oe2403-x86_64/ │ └── oe2403-aarch64/ -├── recipes/ # Recipe YAML files (source only) -│ ├── rocky10-x86_64-warewulf-slurm.yaml -│ ├── almalinux10-x86_64-warewulf-slurm.yaml +├── recipes/ # Recipe definitions (source only) +│ ├── rocky10-x86_64-warewulf-slurm.conf +│ ├── almalinux10-x86_64-warewulf-slurm.conf +│ ├── rocky10-x86_64-xcat-slurm.conf +│ ├── rocky10-x86_64-xcat-slurm.yaml │ └── ... └── build/ # Generated output (gitignored) ├── header-includes.tex # Rendered from pandoc/header-includes.tex.j2 @@ -675,7 +701,7 @@ simple and correct. The Makefile injects `vc_revision` and `vc_date` (from `git log`) into each `build/*.yaml` via `yq` during the merge step, so mkdoc.py needs no subprocess calls. The `.yaml` prerequisite for `build/%.yaml` is -optional via `.SECONDEXPANSION` — only Confluent recipes have one. +optional via `.SECONDEXPANSION` — only some recipes (Confluent, xCAT) have one. ### RPM Packaging @@ -717,10 +743,11 @@ python3 tests/ci/run_build.py $USER components/admin/docs/SPECS/docs.spec ### Recipe Naming -Recipes are named `{distro}{version}-{arch}-{provisioner}-{scheduler}.yaml` -and live in `recipes/`. See existing recipes for examples. The 14 current -recipes cover Warewulf, Confluent, and OpenCHAMI across Rocky, AlmaLinux, -and openEuler on x86\_64 and aarch64. +Recipes are named `{distro}{version}-{arch}-{provisioner}-{scheduler}.conf` +(with an optional `.yaml` override alongside) and live in `recipes/`. See +existing recipes for examples. The 18 current recipes cover Warewulf, +Confluent, OpenCHAMI, and xCAT across Rocky, AlmaLinux, and +openEuler on x86\_64 and aarch64. ### Manifest Directory Naming diff --git a/docs/install/Makefile b/docs/install/Makefile index 3a76326000..54b3483715 100644 --- a/docs/install/Makefile +++ b/docs/install/Makefile @@ -61,7 +61,7 @@ MANIFESTS := $(shell find manifests -name '*.md.j2' -o -name '*.md') PANDOC_FILES := $(PANDOC_DIR)/format-filters.lua \ $(PANDOC_DIR)/header-includes.tex.j2 \ $(PANDOC_DIR)/codeblock-styles.css -DEPS := mkdoc.py $(TEMPLATES) $(MANIFESTS) $(PANDOC_FILES) +DEPS := mkdoc.py templates/macros.j2 $(TEMPLATES) $(MANIFESTS) $(PANDOC_FILES) # Manifest directories for manifest/changelog targets MANIFEST_DIRS := $(wildcard manifests/*/pkg-ohpc.all) diff --git a/docs/install/config/base.yaml b/docs/install/config/base.yaml index a7cfbf57d7..80b74b9ca1 100644 --- a/docs/install/config/base.yaml +++ b/docs/install/config/base.yaml @@ -29,6 +29,7 @@ is_aarch64: false is_warewulf: false is_openchami: false is_confluent: false +is_xcat: false ## Scheduler is_slurm: false diff --git a/docs/install/config/provisioner/xcat.yaml b/docs/install/config/provisioner/xcat.yaml new file mode 100644 index 0000000000..92b64de99d --- /dev/null +++ b/docs/install/config/provisioner/xcat.yaml @@ -0,0 +1,35 @@ +# xCAT provisioner configuration +# +# A single recipe covers both xCAT provisioning modes. The mode is selected at +# run time by ${xcat_mode} (set in input.local), not at document build +# time, so one guide and one recipe script serve both: +# +# xcat_mode=stateless compute nodes boot a RAM-resident image built in a +# chroot on the head node +# xcat_mode=stateful a base OS is installed onto the local disk of each +# compute node, then OpenHPC components are added to +# the running nodes +# +# The recipe defines shell helpers (compute_exec, compute_install, ...) once the +# mode is known; the shared customization chapters call those helpers and work +# unchanged in both modes. + +provisioner: "xcat" +provisioner_name: "xCAT" +is_xcat: true + +# Compute-image package operations dispatch through the mode-selected shell +# functions defined in the "Select Provisioning Mode" section. +pkg_install_chroot: "compute_install" +pkg_group_install_chroot: "compute_group_install" +pkg_upgrade_chroot: "compute_upgrade" +pkg_clean_chroot: "compute_clean" + +# xCAT release tree to install from ("latest" tracks the current stable +# series; set a versioned directory such as "2.18" to pin). The repo +# definitions are written locally: the .repo files published on xcat.org +# point at a retired "devel" tree and are not usable directly. +xcat_release: "latest" +xcat_repo_base: "https://xcat.org/files/xcat/repos/yum" +# OS-family component of the xcat-dep repository path (EL10 -> rh10) +xcat_dep_dist: "rh10" diff --git a/docs/install/input.local.template b/docs/install/input.local.template index d04efd8c35..5de2a219b6 100644 --- a/docs/install/input.local.template +++ b/docs/install/input.local.template @@ -57,9 +57,12 @@ node_disk=${node_disk:=/dev/sda} initialize_options="${initialize_options:-usklpta}" deployment_protocols="${deployment_protocols:-firmware}" dns_domain="${dns_domain:-local}" -iso_path="${iso_path:-}" iso_url="${iso_url:-}" +# Path to directory containing the base OS dvd iso +# (Confluent and xCAT recipes; set to the directory, not the full filename) +iso_path="${iso_path:-}" + # Flags for optional installation/configuration enable_ib="${enable_ib:-0}" enable_opa="${enable_opa:-0}" @@ -70,6 +73,10 @@ enable_ipoib="${enable_ipoib:-0}" enable_dracut="${enable_dracut:-1}" enable_todisk="${enable_todisk:-0}" +# Provisioning mode (xCAT recipe only): "stateless" boots a netboot image, +# "stateful" installs to the local disk of each compute node +xcat_mode="${xcat_mode:-stateless}" + enable_clustershell="${enable_clustershell:-0}" enable_ipmisol="${enable_ipmisol:-0}" enable_opensm="${enable_opensm:-0}" diff --git a/docs/install/recipes/almalinux10-x86_64-xcat-slurm.conf b/docs/install/recipes/almalinux10-x86_64-xcat-slurm.conf new file mode 100644 index 0000000000..6e34a47cba --- /dev/null +++ b/docs/install/recipes/almalinux10-x86_64-xcat-slurm.conf @@ -0,0 +1,6 @@ +config/base.yaml +config/distro/el10.yaml +config/distro/almalinux.yaml +config/arch/x86_64.yaml +config/provisioner/xcat.yaml +config/scheduler/slurm.yaml diff --git a/docs/install/recipes/almalinux10-x86_64-xcat-slurm.yaml b/docs/install/recipes/almalinux10-x86_64-xcat-slurm.yaml new file mode 100644 index 0000000000..1d59427d1f --- /dev/null +++ b/docs/install/recipes/almalinux10-x86_64-xcat-slurm.yaml @@ -0,0 +1,4 @@ +# The osimage names (and the image directory layout under /install) are +# derived by copycds from the distribution metadata inside the ISO, which +# encodes the minor version; the recipe captures them at runtime via lsdef. +distro_iso_image: "AlmaLinux-10-latest-x86_64-dvd.iso" diff --git a/docs/install/recipes/rocky10-x86_64-xcat-slurm.conf b/docs/install/recipes/rocky10-x86_64-xcat-slurm.conf new file mode 100644 index 0000000000..0d8dd137f4 --- /dev/null +++ b/docs/install/recipes/rocky10-x86_64-xcat-slurm.conf @@ -0,0 +1,6 @@ +config/base.yaml +config/distro/el10.yaml +config/distro/rocky.yaml +config/arch/x86_64.yaml +config/provisioner/xcat.yaml +config/scheduler/slurm.yaml diff --git a/docs/install/recipes/rocky10-x86_64-xcat-slurm.yaml b/docs/install/recipes/rocky10-x86_64-xcat-slurm.yaml new file mode 100644 index 0000000000..7420235712 --- /dev/null +++ b/docs/install/recipes/rocky10-x86_64-xcat-slurm.yaml @@ -0,0 +1,4 @@ +# The osimage names (and the image directory layout under /install) are +# derived by copycds from the distribution metadata inside the ISO, which +# encodes the minor version; the recipe captures them at runtime via lsdef. +distro_iso_image: "Rocky-10-latest-x86_64-dvd.iso" diff --git a/docs/install/templates/appendices/upgrade.md.j2 b/docs/install/templates/appendices/upgrade.md.j2 index cb8bc530b3..a6b4ddb548 100644 --- a/docs/install/templates/appendices/upgrade.md.j2 +++ b/docs/install/templates/appendices/upgrade.md.j2 @@ -36,6 +36,35 @@ updates are pre-configured. {{ pkg_clean }} {{ pkg_clean_chroot }} ``` +{% elif is_xcat %} +1. (Optional) Ensure repo metadata is current (on the head node and in the + compute environment). Package managers will naturally do this on their own + over time, but if you are wanting to access updates immediately after a new + release, the following can be used to sync to the latest. The compute-side + command depends on how the cluster was provisioned. The helper functions + defined while installing are not available in a later shell, so the commands + are spelled out in full here. + +```bash +{{ pkg_clean }} +``` + + For a stateful cluster, act on the running compute nodes: + +```bash +xdsh compute dnf clean expire-cache +``` + + For a stateless cluster, act on the image chroot, re-deriving its location: + +```bash +# Name the image explicitly if the head node holds more than one +xcat_osimage=${xcat_osimage:-$(lsdef -t osimage | \ + awk '/netboot-compute/{print $1; exit}')} +CHROOT=$(lsdef -t osimage -o ${xcat_osimage} \ + -i rootimgdir | awk -F= '/rootimgdir/{print $2}')/rootimg/ +dnf --installroot="${CHROOT}" clean expire-cache +``` {% else %} 1. (Optional) Ensure repo metadata is current (on head node and in chroot location(s)). Package managers will naturally do this on their own over time, @@ -93,6 +122,48 @@ apply the changes. Note that to update running services such as a resource manager, a service restart is required on the compute nodes. +{% elif is_xcat %} +3. Upgrade packages in the compute environment + + For a stateful cluster, upgrade the running compute nodes. Note that to + update running services such as a resource manager, a service restart is + required on the compute nodes. + +```bash +xdsh compute dnf -y upgrade "*-ohpc" + +# Any new compute-node Base OS provided dependencies can be installed by +# updating the ohpc-base-compute metapackage +xdsh compute dnf -y upgrade "ohpc-base-compute" +``` + + For a stateless cluster, upgrade the image chroot, re-deriving its location + if it is not already set in the current shell: + +```bash +# Name the image explicitly if the head node holds more than one +xcat_osimage=${xcat_osimage:-$(lsdef -t osimage | \ + awk '/netboot-compute/{print $1; exit}')} +CHROOT=$(lsdef -t osimage -o ${xcat_osimage} \ + -i rootimgdir | awk -F= '/rootimgdir/{print $2}')/rootimg/ + +dnf -y --installroot="${CHROOT}" upgrade "*-ohpc" + +# Any new compute-node Base OS provided dependencies can be installed by +# updating the ohpc-base-compute metapackage +dnf -y --installroot="${CHROOT}" upgrade "ohpc-base-compute" +``` + +4. Rebuild and redeploy the stateless image + + This step applies to stateless provisioning only. In stateful mode the + upgrade above already ran on the compute nodes and there is no image to + rebuild. Reboot the compute nodes when convenient to pick up the new image. + +```bash +packimage ${xcat_osimage} +nodeset compute osimage=${xcat_osimage} +``` {% else %} 3. Upgrade packages in compute image diff --git a/docs/install/templates/base-os/install.md.j2 b/docs/install/templates/base-os/install.md.j2 index f5a7faa5e8..3109d4b9a2 100644 --- a/docs/install/templates/base-os/install.md.j2 +++ b/docs/install/templates/base-os/install.md.j2 @@ -22,7 +22,7 @@ echo ${sms_ip} ${sms_name} >> /etc/hosts {% endif %} While it is theoretically possible to enable SELinux on a cluster provisioned -with {{ provisioner }}, doing so is beyond the scope of this document. Even the +with {{ provisioner_name }}, doing so is beyond the scope of this document. Even the use of permissive mode can be problematic and we therefore recommend disabling SELinux on the *head node*. If SELinux components are installed locally, the `selinuxenabled` command can be used to determine if SELinux is currently diff --git a/docs/install/templates/chapters/deploy-xcat.md.j2 b/docs/install/templates/chapters/deploy-xcat.md.j2 new file mode 100644 index 0000000000..3b536995a3 --- /dev/null +++ b/docs/install/templates/chapters/deploy-xcat.md.j2 @@ -0,0 +1,24 @@ +{# Deploy Aggregator #} +# Deploy Cluster + +{% include "provisioner/deploy.md.j2" %} + +{# Slurm starts on the head node first so that stateless nodes, which boot + below, find the controller on their first try, and so that stateful nodes, + which are already running, have a controller to register with. #} +{% include "scheduler/slurm/startup.md.j2" %} + +## Stateless Deployment + +Follow this section when `${xcat_mode}` is `0`. The customized image is +packed and the compute nodes boot it over the network. + +{% include "provisioner/xcat/deploy-stateless-pack.md.j2" %} +{% include "provisioner/xcat/deploy-stateless-boot.md.j2" %} + +## Stateful Deployment + +Follow this section when `${xcat_mode}` is `1`. The compute nodes are +already running, so only their client daemons need starting. + +{% include "provisioner/xcat/deploy-stateful.md.j2" %} diff --git a/docs/install/templates/chapters/provisioner-xcat.md.j2 b/docs/install/templates/chapters/provisioner-xcat.md.j2 new file mode 100644 index 0000000000..70af089b58 --- /dev/null +++ b/docs/install/templates/chapters/provisioner-xcat.md.j2 @@ -0,0 +1,45 @@ +{# Provisioner Aggregator #} +# Install xCAT Provisioner + +{% include "provisioner/provisioner.md.j2" %} + +This guide covers both xCAT provisioning modes -- **stateless** (*netboot*), +where compute nodes boot a RAM-resident image served by the head node, and +**stateful** (*diskful*), where a base OS is installed onto each node's local +disk and OpenHPC components are added afterwards. The sections up to and +including *Identify Files for Synchronization* apply to both; after that, +follow the one section that matches the mode selected below and skip the other. + +{% include "provisioner/xcat/xcat-repo.md.j2" %} +{% include "provisioner/xcat/xcat-install.md.j2" %} +{% include "provisioner/xcat/xcat-setup.md.j2" %} + +{% include "provisioner/xcat/mode-select.md.j2" %} + +{% include "provisioner/xcat/osimage-init.md.j2" %} +{% include "provisioner/xcat/add-nodes.md.j2" %} +{% include "provisioner/xcat/synclists.md.j2" %} + +## Stateless Provisioning + +Follow this section when `${xcat_mode}` is `0`. The compute image is +assembled in a chroot on the head node; the nodes themselves do not boot until +the *Deploy Cluster* chapter. Skip ahead to *Stateful Provisioning* if you are +provisioning to local disk. + +{% include "provisioner/xcat/stateless-genimage.md.j2" %} +{% include "provisioner/xcat/stateless-ohpc.md.j2" %} +{% include "provisioner/xcat/stateless-customize.md.j2" %} + +## Stateful Provisioning + +Follow this section when `${xcat_mode}` is `1`. The compute nodes are +installed and booted here, and OpenHPC components are then added to the running +nodes. Skip this section entirely if you are provisioning stateless. + +{% include "provisioner/xcat/stateful-diskconfig.md.j2" %} +{% include "provisioner/xcat/stateful-install.md.j2" %} +{% include "provisioner/xcat/stateful-setup.md.j2" %} +{% include "provisioner/xcat/stateful-ohpc.md.j2" %} +{% include "provisioner/xcat/stateful-slurm.md.j2" %} +{% include "provisioner/xcat/stateful-nfs.md.j2" %} diff --git a/docs/install/templates/intro/inputs.md.j2 b/docs/install/templates/intro/inputs.md.j2 index babff1b9e9..ba52422243 100644 --- a/docs/install/templates/intro/inputs.md.j2 +++ b/docs/install/templates/intro/inputs.md.j2 @@ -48,7 +48,13 @@ Required variables: * `iso_path` - location of compute node base os dvd iso (default: `{{ distro_iso_image }}`) * `iso_url` - ISO download base URL (optional; overrides default upstream location) {% endif %} +{% if is_xcat %} +* `${domain_name}` - DNS domain name for cluster (e.g. `local`) +* `iso_path` - path to directory containing the base OS dvd iso (`{{ distro_iso_image }}`) +* `${xcat_mode}` - provisioning mode: `stateless` (netboot) or `stateful` + (install to local disk) +{% endif %} {% if is_x86_64 %} Optional variables: diff --git a/docs/install/templates/intro/intro.md.j2 b/docs/install/templates/intro/intro.md.j2 index 06cd7207ea..e4f0826ca5 100644 --- a/docs/install/templates/intro/intro.md.j2 +++ b/docs/install/templates/intro/intro.md.j2 @@ -5,7 +5,7 @@ including provisioning tools, resource management, I/O clients, development tools, and a variety of scientific libraries. These packages have been pre-built with HPC integration in mind while conforming to common Linux distribution standards. The documentation herein is intended to be reasonably generic, but -uses the underlying motivation of a small, 4-node stateless cluster installation +uses the underlying motivation of a small, 4-node cluster installation to define a step-by-step process. Several optional customizations are included and the intent is that these collective instructions can be modified as needed for local site customizations. diff --git a/docs/install/templates/intro/requirements.md.j2 b/docs/install/templates/intro/requirements.md.j2 index 5053afee21..344da56d5c 100644 --- a/docs/install/templates/intro/requirements.md.j2 +++ b/docs/install/templates/intro/requirements.md.j2 @@ -3,8 +3,8 @@ This installation recipe assumes the availability of a single *head node*, and four *compute* nodes. The *head node* serves as the overall system management server (SMS) and is provisioned with {{ distro_full_name }} and is subsequently -configured to provision the remaining *compute* nodes with {{ provisioner }} in -a stateless configuration. For power management, we assume that the compute node +configured to provision the remaining *compute* nodes with {{ provisioner_name }} in +{% if is_xcat %}either a stateless or a stateful configuration{% else %}a stateless configuration{% endif %}. For power management, we assume that the compute node baseboard management controllers (BMCs) are available via IPMI from the chosen *head node*. For file systems, we assume that the chosen *head node* will host an NFS file system that is made available to the compute nodes. diff --git a/docs/install/templates/macros.j2 b/docs/install/templates/macros.j2 index 5d9a0b2297..5e9c1cd265 100644 --- a/docs/install/templates/macros.j2 +++ b/docs/install/templates/macros.j2 @@ -48,6 +48,30 @@ wwctl image exec --build=false {{ baseos }} -- /bin/sh -c \ {% endif %} +{# xCAT Macros #} +{# One recipe serves both xCAT modes. The chroot/xdsh split is resolved at run + time by the shell functions defined in "Select Provisioning Mode", so these + macros emit a helper call rather than a mode-specific command. Paths are + always image-relative (compute_exec chroots for stateless, targets the live + node for stateful). #} +{% if is_xcat %} + +{%- macro compute_sed(regex, file) -%} +compute_exec "sed -i '{{ regex }}' \ + '{{ file }}'" +{%- endmacro -%} + +{%- macro compute_echo(string, file, append=true) -%} +compute_exec "echo '{{ string }}' {{ ">>" if append else ">" }} \ + '{{ file }}'" +{%- endmacro -%} + +{%- macro compute_run(cmd) -%} +compute_exec "{{ cmd }}" +{%- endmacro -%} + +{% endif %} + {# Confluent Macros #} {% if is_confluent %} diff --git a/docs/install/templates/provisioner/xcat/add-nodes.md.j2 b/docs/install/templates/provisioner/xcat/add-nodes.md.j2 new file mode 100644 index 0000000000..72e9b63a9a --- /dev/null +++ b/docs/install/templates/provisioner/xcat/add-nodes.md.j2 @@ -0,0 +1,63 @@ +## Add Compute Nodes into xCAT Database + +Add compute nodes and define their properties as objects in the xCAT database. +Hosts are grouped into a group named *compute* to allow group-level commands +later in the recipe. Note the use of variable names for hostnames, node IPs, MAC +addresses, and BMC login credentials — adjust to match local hardware. + + + +```bash +for ((i=0; i + + + + +```bash +# Define ib0 netmask +chdef -t network -o ib0 mask=${ipoib_netmask} net=${c_ipoib[0]} + +# Enable secondary NIC configuration +chdef compute -p postbootscripts=confignics + +# Register desired IPoIB IPs per compute +for ((i=0; i + + +Complete network service configuration: + + + +```bash +makehosts +makenetworks + +# makedns must run before makedhcp: xCAT's Kea DHCP-DDNS integration +# requires the DDNS key material that makedns generates +makedns -n +makedhcp -n +``` + diff --git a/docs/install/templates/provisioner/xcat/deploy-stateful.md.j2 b/docs/install/templates/provisioner/xcat/deploy-stateful.md.j2 new file mode 100644 index 0000000000..55fa8ad879 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/deploy-stateful.md.j2 @@ -0,0 +1,28 @@ +### Start Slurm on Compute Nodes + +With Slurm configured, we can now startup the resource manager services in +preparation for running user jobs. Generally, this is a two-step process that +requires starting up the controller daemons on the *head node* and the client +daemons on each of the *compute* nodes. Note that Slurm leverages the use of the +*munge* library to provide authentication services and this daemon also needs to +be running on all hosts within the resource management pool. + +We now start the Slurm daemon on the compute nodes. + + + +```bash +# Start munge and slurm client on compute nodes +{{ compute_run("systemctl enable --now munge") }} +{{ compute_run("systemctl enable --now slurmd") }} +``` + + + +After this, check status of the nodes within Slurm by using the `sinfo` command. +All compute nodes should be in an *idle* state (without asterisk). If the state +is reported as *unknown*, the following might help: + +```bash +scontrol update partition=normal state=idle +``` diff --git a/docs/install/templates/provisioner/xcat/deploy-stateless-boot.md.j2 b/docs/install/templates/provisioner/xcat/deploy-stateless-boot.md.j2 new file mode 100644 index 0000000000..711bc797e7 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/deploy-stateless-boot.md.j2 @@ -0,0 +1,54 @@ +### Boot Compute Nodes + +Boot the compute nodes; they load the packed image over the network into RAM. +Because the resource manager was started above, their `slurmd` finds the +controller as soon as they come up. After the initial PXE boot, subsequent +boots return to the default BIOS boot device. + + + + + +```bash +# Set compute group to boot from network on next boot +rsetboot compute net + +# Power cycle compute nodes +rpower compute reset +``` + + + + + + + + + + + + + + +The boot process typically completes in about 5 minutes. Monitor provisioning +with the `rcons` command, which displays the serial console for a selected node. +The escape sequence to exit `rcons` is `Ctrl-e c .` typed sequentially. + +Once booted, verify successful provisioning with a parallel command across the +compute nodes. xCAT provides `xdsh`, its native distributed shell, which +accepts xCAT node names and groups: + +```bash +xdsh compute uptime +c1: 12:56:50 up 14 min, 0 users, load average: 0.00, 0.01, 0.04 +c2: 12:56:50 up 13 min, 0 users, load average: 0.00, 0.02, 0.05 +c3: 12:56:50 up 14 min, 0 users, load average: 0.00, 0.02, 0.05 +c4: 12:56:50 up 14 min, 0 users, load average: 0.00, 0.01, 0.04 +``` + +The provisioner-independent `pdsh` utility, installed on the head node as part +of the `ohpc-base` metapackage, may be used as well: + +```bash +pdsh -w ${compute_prefix}[1-${num_computes}] uptime +``` diff --git a/docs/install/templates/provisioner/xcat/deploy-stateless-pack.md.j2 b/docs/install/templates/provisioner/xcat/deploy-stateless-pack.md.j2 new file mode 100644 index 0000000000..449b13292c --- /dev/null +++ b/docs/install/templates/provisioner/xcat/deploy-stateless-pack.md.j2 @@ -0,0 +1,16 @@ +### Pack the Compute Image + +The compute image is now complete. Pack it and associate it with the *compute* +group so the nodes have something to boot. + + + + +```bash +packimage ${xcat_osimage} + +# Associate provisioning image for compute group +nodeset compute osimage=${xcat_osimage} +``` + + diff --git a/docs/install/templates/provisioner/xcat/mode-select.md.j2 b/docs/install/templates/provisioner/xcat/mode-select.md.j2 new file mode 100644 index 0000000000..90da34ecc2 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/mode-select.md.j2 @@ -0,0 +1,58 @@ +## Select Provisioning Mode + +xCAT supports two ways of provisioning compute nodes, and this recipe covers +both. Choose one by setting `${xcat_mode}` in `input.local` before +running the recipe: + +| `xcat_mode` | Mode | How compute nodes run | +| ----------- | ---- | --------------------- | +| `stateless` (default) | **stateless** (*netboot*) | Nodes boot a RAM-resident image assembled in a chroot on the head node. Nothing is written to local disk, and the image is rebuilt and redeployed to change the nodes. | +| `stateful` | **stateful** (*diskful*) | A base OS is installed onto each node's local disk, then OpenHPC components are added to the running nodes. Nodes keep their state across reboots. | + +The difference is confined to how the compute environment is created and when +the nodes first boot. Everything afterwards -- InfiniBand, Omni-Path, memory +limits, GPU drivers, log forwarding, and the admin tooling -- is identical, and +is applied through the helper functions defined below. `compute_exec` runs a +command in the compute environment, and `compute_install` and its siblings +manage packages there; in stateless mode they act on the image chroot, and in +stateful mode they act on the running nodes through `xdsh`. + + + +```bash +# Fail closed on a missing or malformed mode. An input.local carried over from +# the separate stateful recipe has no xcat_mode at all, and silently defaulting +# it would netboot a fleet that should install to disk. +case "${xcat_mode:-}" in + stateless|stateful) ;; + *) echo "ERROR: xcat_mode must be 'stateless' or 'stateful'," >&2 + echo "not '${xcat_mode:-}'. Set it in input.local." >&2 + exit 1 ;; +esac + +if [ "${xcat_mode}" = "stateful" ]; then + # Stateful: act on the running compute nodes over xdsh + compute_exec() { xdsh compute "$@"; } + compute_install() { xdsh compute dnf -y install "$@"; } + compute_group_install() { xdsh compute "dnf -y groupinstall '$*'"; } + compute_upgrade() { xdsh compute dnf -y upgrade "$@"; } + compute_clean() { xdsh compute dnf clean expire-cache; } + xcat_image_type="install" +else + # Stateless: act on the chroot that backs the netboot image + compute_exec() { chroot "${CHROOT}" /bin/sh -c "$@"; } + compute_install() { dnf -y --installroot="${CHROOT}" install "$@"; } + compute_group_install() { dnf -y --installroot="${CHROOT}" groupinstall "$*"; } + compute_upgrade() { dnf -y --installroot="${CHROOT}" upgrade "$@"; } + compute_clean() { dnf --installroot="${CHROOT}" clean expire-cache; } + xcat_image_type="netboot" +fi + +# Synclists are registered per image type +xcat_synclist_dir="/install/custom/${xcat_image_type}" +``` + + +In stateless mode the helpers reference `${CHROOT}`, which is defined later by +the image build step; because they are shell functions, the value is resolved +when each helper is called rather than here. diff --git a/docs/install/templates/provisioner/xcat/osimage-init.md.j2 b/docs/install/templates/provisioner/xcat/osimage-init.md.j2 new file mode 100644 index 0000000000..865dbd7833 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/osimage-init.md.j2 @@ -0,0 +1,66 @@ +## Define Compute Image for Provisioning + +The following steps import the distribution installation media for use with +xCAT. A local copy of the distribution ISO is required; for this recipe the ISO +is `{{ distro_iso_image }}`. Initialize image creation using `copycds`, assuming +the ISO is available at `${iso_path}`: + + + +```bash +: "${iso_path:?ERROR: iso_path not set}" + +copycds ${iso_path}/{{ distro_iso_image }} +``` + + +`copycds` derives the OS image names from the distribution metadata inside the +ISO, which encodes the minor version (for example a Rocky 10.0 x86_64 ISO +creates `rocky10.0-x86_64-{install,netboot,statelite}-compute`). Stateless +provisioning uses the `netboot` image and stateful provisioning uses the +`install` image; `${xcat_image_type}`, set with the provisioning mode above, +selects the right one. Query the created images and capture the name. If the +head node already holds images for other distributions or releases the lookup +is ambiguous, so it fails rather than guessing; set `${xcat_osimage}` in +`input.local` to choose explicitly. Either way the image is then checked +against the selected mode, so a name that does not match fails before any +node or image is modified: + + + +```bash +# Query available images +lsdef -t osimage + +# Capture the compute image for the selected provisioning mode. A head node +# that holds images for several distributions or releases will match more +# than one; rather than silently taking the first, refuse to guess and let +# the operator name the image in input.local. +if [ -z "${xcat_osimage:-}" ]; then + xcat_matches=$(lsdef -t osimage 2>/dev/null | \ + awk -v t="${xcat_image_type}-compute" '$0 ~ t {print $1}') + xcat_count=$(printf '%s\n' "${xcat_matches}" | grep -c . || true) + if [ "${xcat_count}" -ne 1 ]; then + echo "ERROR: expected exactly one ${xcat_image_type}-compute osimage," >&2 + echo "found ${xcat_count}:" >&2 + printf ' %s\n' ${xcat_matches} >&2 + echo "Set xcat_osimage explicitly in input.local." >&2 + exit 1 + fi + xcat_osimage="${xcat_matches}" +fi + +# Validate the image against the requested mode before anything is changed, +# whether the name was discovered above or supplied in input.local. xCAT will +# otherwise happily provision a netboot image to local disk, or the reverse. +xcat_provmethod=$(lsdef -t osimage -o "${xcat_osimage}" -i provmethod 2>/dev/null \ + | awk -F= '/provmethod=/{print $2}') +if [ "${xcat_provmethod}" != "${xcat_image_type}" ]; then + echo "ERROR: osimage '${xcat_osimage}' has provmethod='${xcat_provmethod}'," >&2 + echo "but this run needs '${xcat_image_type}'. Check \${xcat_osimage} and" >&2 + echo "\${xcat_mode} in input.local." >&2 + exit 1 +fi +echo ${xcat_osimage} +``` + diff --git a/docs/install/templates/provisioner/xcat/post-add-user.md.j2 b/docs/install/templates/provisioner/xcat/post-add-user.md.j2 new file mode 100644 index 0000000000..6cba1fcb9c --- /dev/null +++ b/docs/install/templates/provisioner/xcat/post-add-user.md.j2 @@ -0,0 +1,13 @@ +## Add User + +Add a new user to the cluster. Distribute the updated credentials to compute +nodes using `updatenode`: + + +```bash +useradd -m test + +# Push updated credentials to compute nodes +updatenode compute -F +``` + diff --git a/docs/install/templates/provisioner/xcat/stateful-diskconfig.md.j2 b/docs/install/templates/provisioner/xcat/stateful-diskconfig.md.j2 new file mode 100644 index 0000000000..6cdf7fb566 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-diskconfig.md.j2 @@ -0,0 +1,26 @@ +### Disk Layout and Root Password + +The install applies a default disk layout to the first detected disk (a 1 GiB +`/boot` plus an LVM volume group holding swap and a `/` volume grown to fill +the disk). Sites requiring a different layout can register +a custom partition file in kickstart syntax with +`chdef -t osimage ${xcat_osimage} partitionfile=`; consult the +[xCAT documentation](https://xcat-docs.readthedocs.io/en/latest/) for details. + +The kickstart generated for stateful installs reads the compute root password +from the xCAT `passwd` table, so the entry must exist before the provisioning +image is associated with the nodes. Administrative access relies on the head +node root ssh keys that xCAT installs during provisioning, so the randomly +generated password below is intentionally throwaway; sites that want +interactive (console) root logins should substitute a chosen password for the +`openssl rand` call. Store the password as a hash: + + + + +```bash +chtab key=system passwd.username=root \ + passwd.password="$(openssl passwd -6 "$(openssl rand -base64 12)")" +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateful-install.md.j2 b/docs/install/templates/provisioner/xcat/stateful-install.md.j2 new file mode 100644 index 0000000000..3af8235e6c --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-install.md.j2 @@ -0,0 +1,109 @@ +### Install the Base OS onto Compute Nodes + +In stateful mode the compute nodes are provisioned before OpenHPC components +are added: associate the install image with the *compute* group, then boot the +nodes so they run the distribution installer over the network, perform an +unattended install onto local disk, and reboot into the freshly installed OS. + +The steps that follow install packages directly on the running compute nodes +with the package manager, which requires the nodes to reach the distribution, +EPEL, and OpenHPC package repositories. By default xCAT points the compute +default route at the head node; ensure this provides a path to the +repositories (for example via NAT on the head node or a site gateway +registered in the xCAT `networks` table), or host a local repository mirror +on the head node and register it on the computes instead. + + + + +```bash +# Associate provisioning image for compute group +nodeset compute osimage=${xcat_osimage} +``` + + +Before power cycling, mark the OS that is currently running, if any. `/run` is +a tmpfs, so the marker cannot survive a reboot. The wait further below refuses +any node that still carries it, which stops a slow BMC reset from letting the +system that is about to be replaced satisfy the check when nodes are being +reinstalled rather than installed for the first time. + + + + + + + + + +```bash +# Set compute group to boot from network on next boot +rsetboot compute net + +# Power cycle compute nodes +rpower compute reset +``` + + + + + + + + +The stateful installation typically completes in 5 to 10 minutes, depending on +BIOS post times and disk speed. Monitor provisioning with the `rcons` command, +which displays the serial console for a selected node; the escape sequence to +exit `rcons` is `Ctrl-e c .` typed sequentially. Deployment progress can also +be queried with `nodestat compute`, which reports the installation state of +each node until an `sshd` response indicates the node is up. + +Wait until every compute node has completed the installation and finished +booting before proceeding. `sshd` starts before the rest of the boot completes, +so waiting only for it lets the configuration steps below run against a node +that still refuses logins and has not finished starting systemd. The loop below +therefore requires every node to positively report a finished boot *and* to have +lost the pre-reset marker, so that neither an unreachable node nor one still +running the previous OS counts as ready. It gives up after `${install_timeout}` +seconds (two hours by default) rather than waiting forever on an installation +that failed: + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +Once installed, verify access with a parallel command across the compute nodes +using `xdsh`, xCAT's native distributed shell: + +```bash +xdsh compute uptime +c1: 12:56:50 up 4 min, 0 users, load average: 0.00, 0.01, 0.04 +c2: 12:56:50 up 3 min, 0 users, load average: 0.00, 0.02, 0.05 +``` diff --git a/docs/install/templates/provisioner/xcat/stateful-nfs.md.j2 b/docs/install/templates/provisioner/xcat/stateful-nfs.md.j2 new file mode 100644 index 0000000000..bf23e26f46 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-nfs.md.j2 @@ -0,0 +1,29 @@ +### Configure NFS Mounts on Compute Nodes + +Configure NFS mounts to access the *head node*'s shared filesystems on +*compute* nodes. First disable the xCAT-managed `/tftpboot` and `/install` +export entries so they do not conflict with the `/home` and `/opt/ohpc/pub` +exports configured earlier, then create the client mounts: + + + + +```bash +# Disable the xCAT-managed /tftpboot and /install export entries so +# they do not conflict with the /home and /opt/ohpc/pub exports +perl -pi -e "s|/tftpboot|#/tftpboot|" /etc/exports +perl -pi -e "s|/install|#/install|" /etc/exports +exportfs -a + +# Add NFS mounts for /home and /opt/ohpc/pub +NFS_HOME="${sms_ip}:/home /home nfs nfsvers=4,nodev,nosuid 0 0" +NFS_PUB="${sms_ip}:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=4,nodev 0 0" +{{ compute_run("echo ${NFS_HOME} >> /etc/fstab") }} +{{ compute_run("echo ${NFS_PUB} >> /etc/fstab") }} + +# Create mount point and mount filesystems +{{ compute_run("mkdir -p /opt/ohpc/pub") }} +{{ compute_run("mount -a") }} +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateful-ohpc.md.j2 b/docs/install/templates/provisioner/xcat/stateful-ohpc.md.j2 new file mode 100644 index 0000000000..af2f0e9608 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-ohpc.md.j2 @@ -0,0 +1,52 @@ +### Add OpenHPC Components to Compute Nodes + +The next step is adding OpenHPC components to the *compute* nodes, which at +this point are running basic OSes. This process leverages two xCAT-provided +mechanisms: `xdsh` to run the package installer on all the nodes in parallel +and `updatenode` to distribute configuration files from the head node to the +*compute* nodes. + +We first add the OpenHPC repository to the compute nodes: + + + + +```bash +# Add OpenHPC repo +{{ pkg_install_chroot }} \ + {{ ohpc_repo_server }}/OpenHPC/{{ ohpc_version_tree }}/{{ ostree }}/{{ arch }}/\ +ohpc-release-{{ ohpc_version_tree }}-1.{{ distro_tag }}.{{ arch }}.rpm +``` + + +Additionally, a workaround is needed for OpenHPC documentation files, which are +installed into a read-only NFS share /opt/ohpc/pub. Any package attempting to +write to that directory will fail to install. The following prevents that by +directing `rpm` not to install documentation files on the *compute* nodes: + + +```bash +{{ compute_echo("%_excludedocs 1", "$HOME/.rpmmacros")}} +``` + + +Now OpenHPC and other cluster-related software components can be installed on +the nodes. The first step is to install a base compute package on the compute +nodes: + + +```bash +# Install compute node base meta-package +{{ pkg_install_chroot }} ohpc-base-compute +``` + + +Next, we can include additional components: + + +```bash +# Include modules user environment +{{ pkg_install_chroot }} lmod-ohpc +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateful-setup.md.j2 b/docs/install/templates/provisioner/xcat/stateful-setup.md.j2 new file mode 100644 index 0000000000..4b5707fe11 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-setup.md.j2 @@ -0,0 +1,55 @@ +### Setup Compute Nodes + +With the compute nodes running their freshly installed base OS, configure them +for OpenHPC use. Commands are run on all nodes in parallel with `xdsh`, xCAT's +native distributed shell. + +The remaining steps install packages directly on the running compute nodes +with the package manager, which requires the nodes to reach the distribution, +EPEL, and OpenHPC package repositories. By default xCAT points the compute +default route at the head node; ensure this provides a path to the +repositories (for example via NAT on the head node or a site gateway +registered in the xCAT `networks` table), or host a local repository mirror +on the head node and register it on the computes instead. + +We first disable the firewall on the compute nodes. + + + + + + + + +```bash +# Disable firewall for computes +xdsh compute systemctl disable --now firewalld +``` + + +The *compute* nodes need access to the EPEL repository, a required dependency +for OpenHPC packages. + + +```bash +# Add EPEL repo +{{ pkg_install_chroot }} epel-release +# Enable CRB +{{ compute_run("/usr/bin/crb enable") }} +``` + + +And add additional packages. + + +```bash +# Add Network Time Protocol (NTP) support and point it at the head node +{{ pkg_install_chroot }} chrony +{{ compute_run("echo server ${sms_ip} iburst >> /etc/chrony.conf") }} +{{ compute_run("systemctl enable --now chronyd") }} + +# Include nfs-utils +{{ pkg_install_chroot }} nfs-utils +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateful-slurm.md.j2 b/docs/install/templates/provisioner/xcat/stateful-slurm.md.j2 new file mode 100644 index 0000000000..d71cd2aa58 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateful-slurm.md.j2 @@ -0,0 +1,33 @@ +### Install Slurm Client on Compute Nodes + +In an earlier section, the Slurm resource manager was installed and configured +on the *head node*. Now install the Slurm client components on the compute +nodes and register the head node as their configuration server (Slurm +"configless" mode): + + + + +```bash +# Add Slurm client support meta-package +{{ pkg_install_chroot }} ohpc-slurm-client + +# Register Slurm server with computes (using "configless" option) +{{ compute_run("echo SLURMD_OPTIONS=--conf-server ${sms_ip} > /etc/sysconfig/slurmd") }} +``` + + +Slurm relies on the *munge* authentication service, which requires an identical +key on every host in the resource management pool. The munge key was registered +for synchronization earlier and applied during the stateful install; re-run the +file synchronization so ownership and permissions are consistent now that the +`munge` user and service exist on the nodes: + + + +```bash +# Refresh synchronized files (passwd/group/shadow, munge key) +updatenode compute -F +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateless-customize.md.j2 b/docs/install/templates/provisioner/xcat/stateless-customize.md.j2 new file mode 100644 index 0000000000..1a0db1c7b2 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateless-customize.md.j2 @@ -0,0 +1,26 @@ +### Customize System Configuration + +Configure NFS client mounts, time synchronization, and disable the default +xCAT-managed export entries that conflict with the NFS exports set up earlier. + + + + +```bash +# Add NFS client mounts of /home and /opt/ohpc/pub to compute image +echo "${sms_ip}:/home /home nfs nfsvers=4,nodev,nosuid 0 0" \ + >> ${CHROOT}/etc/fstab +echo "${sms_ip}:/opt/ohpc/pub /opt/ohpc/pub nfs nfsvers=4,nodev 0 0" \ + >> ${CHROOT}/etc/fstab + +# Disable the xCAT-managed /tftpboot and /install export entries so +# they do not conflict with the /home and /opt/ohpc/pub exports above +perl -pi -e "s|/tftpboot|#/tftpboot|" /etc/exports +perl -pi -e "s|/install|#/install|" /etc/exports + +# Enable chrony time service on computes and identify head node as NTP server +chroot ${CHROOT} systemctl enable chronyd +echo "server ${sms_ip} iburst" >> ${CHROOT}/etc/chrony.conf +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateless-genimage.md.j2 b/docs/install/templates/provisioner/xcat/stateless-genimage.md.j2 new file mode 100644 index 0000000000..4a8ceef8d2 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateless-genimage.md.j2 @@ -0,0 +1,20 @@ +### Initialize the Image Chroot + +`genimage` creates a chroot that the compute image is built from. `${CHROOT}` +names that directory and is used by the customization steps throughout the +rest of this guide. + + + + +```bash +# Save chroot location for the compute image +CHROOT=$(lsdef -t osimage -o ${xcat_osimage} \ + -i rootimgdir | awk -F= '/rootimgdir/{print $2}')/rootimg/ +export CHROOT + +# Build initial chroot image +genimage ${xcat_osimage} +``` + + diff --git a/docs/install/templates/provisioner/xcat/stateless-ohpc.md.j2 b/docs/install/templates/provisioner/xcat/stateless-ohpc.md.j2 new file mode 100644 index 0000000000..e9bed725ed --- /dev/null +++ b/docs/install/templates/provisioner/xcat/stateless-ohpc.md.j2 @@ -0,0 +1,54 @@ +### Add OpenHPC Components to the Image + +The `genimage` process provides a minimal base OS configuration. Next, add +additional components to include resource management client services and other +packages to support the default OpenHPC environment. This process augments the +chroot-based install and requires access to the base OS and OpenHPC +repositories. + +Enable the necessary package repositories inside the chroot: + + + + +```bash +# Copy credential files first to ensure consistent uid/gids for slurm/munge +# at package install time. These will be kept in sync via the provisioning +# synclist after first boot. +cp /etc/passwd /etc/group ${CHROOT}/etc + +# Copy repo files into chroot +cp /etc/yum.repos.d/OpenHPC*.repo ${CHROOT}/etc/yum.repos.d +cp /etc/yum.repos.d/epel*.repo ${CHROOT}/etc/yum.repos.d + +# Install compute node base meta-package and disable firewall +{{ pkg_install_chroot }} ohpc-base-compute +chroot ${CHROOT} systemctl disable firewalld + +# Add Slurm client support meta-package +{{ pkg_install_chroot }} ohpc-slurm-client + +# Enable munge and slurmd to start on boot inside the compute image +chroot ${CHROOT} systemctl enable munge +chroot ${CHROOT} systemctl enable slurmd + +# Register Slurm server with computes (using "configless" option) +echo SLURMD_OPTIONS="--conf-server ${sms_ip}" > ${CHROOT}/etc/sysconfig/slurmd + +# Add Network Time Protocol (NTP) support +{{ pkg_install_chroot }} chrony + +# Add kernel drivers +{{ pkg_install_chroot }} kernel + +# Rebuild the netboot image with the kernel installed in the compute image +# (the head node kernel version may differ from the image kernel) +kver=$(rpm -q --root ${CHROOT} kernel \ + --qf "%{VERSION}-%{RELEASE}.%{ARCH}\n" | tail -1) +genimage ${xcat_osimage} -k ${kver} + +# Include modules user environment +{{ pkg_install_chroot }} lmod-ohpc +``` + + diff --git a/docs/install/templates/provisioner/xcat/synclists.md.j2 b/docs/install/templates/provisioner/xcat/synclists.md.j2 new file mode 100644 index 0000000000..7686cc8e1f --- /dev/null +++ b/docs/install/templates/provisioner/xcat/synclists.md.j2 @@ -0,0 +1,29 @@ +## Identify Files for Synchronization + +xCAT includes functionality to synchronize files from the head node to managed +hosts via *synclists*. This is one way to distribute user credentials to compute +nodes. In stateless mode the synclist is applied when the image boots; in +stateful mode the default `syncfiles` postscript applies it during the install, +and changes can be pushed on demand afterwards with `updatenode`. Enable the +synclist feature and register credential files: + + + +```bash +# Define path for xCAT synclist file +mkdir -p ${xcat_synclist_dir} +chdef -t osimage -o ${xcat_osimage} \ + synclists="${xcat_synclist_dir}/compute.synclist" + +# Add credential files to synclist +echo "/etc/passwd -> /etc/passwd" > ${xcat_synclist_dir}/compute.synclist +echo "/etc/group -> /etc/group" >> ${xcat_synclist_dir}/compute.synclist +echo "/etc/shadow -> /etc/shadow" >> ${xcat_synclist_dir}/compute.synclist +echo "/etc/munge/munge.key -> /etc/munge/munge.key" \ + >> ${xcat_synclist_dir}/compute.synclist +``` + + +The `updatenode compute -F` command distributes changes made to any defined +synchronization files. Use the uppercase `-F` flag; lowercase `-f` targets +service nodes. diff --git a/docs/install/templates/provisioner/xcat/xcat-install.md.j2 b/docs/install/templates/provisioner/xcat/xcat-install.md.j2 new file mode 100644 index 0000000000..bd229b1b13 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/xcat-install.md.j2 @@ -0,0 +1,22 @@ +## Install xCAT on *Head Node* + +With the OpenHPC and xCAT repositories enabled, install the xCAT provisioning +system: + + + +```bash +# Install xCAT +{{ pkg_install }} xCAT +``` + + + + + +Source the environment script to make the xCAT tools available in the current +shell (new login shells pick this up automatically): + +```bash +. /etc/profile.d/xcat.sh +``` diff --git a/docs/install/templates/provisioner/xcat/xcat-repo.md.j2 b/docs/install/templates/provisioner/xcat/xcat-repo.md.j2 new file mode 100644 index 0000000000..d0e6479534 --- /dev/null +++ b/docs/install/templates/provisioner/xcat/xcat-repo.md.j2 @@ -0,0 +1,32 @@ +## Enable xCAT Repository + +Enable the public xCAT package repositories. The `xcat-core` repo provides the +main xCAT packages; `xcat-dep` provides dependencies specific to the OS family +and architecture. The repository definitions are written locally (the `.repo` +files published on xcat.org reference a retired development tree and cannot be +used directly). + + + +```bash +repobase={{ xcat_repo_base }} + +cat << EOF > /etc/yum.repos.d/xcat-core.repo +[xcat-core] +name=xCAT core packages +baseurl=${repobase}/{{ xcat_release }}/xcat-core +enabled=1 +gpgcheck=1 +gpgkey=${repobase}/{{ xcat_release }}/xcat-core/repodata/repomd.xml.key +EOF + +cat << EOF > /etc/yum.repos.d/xcat-dep.repo +[xcat-dep] +name=xCAT dependencies +baseurl=${repobase}/xcat-dep/{{ xcat_dep_dist }}/{{ arch }} +enabled=1 +gpgcheck=1 +gpgkey=${repobase}/xcat-dep/{{ xcat_dep_dist }}/{{ arch }}/repodata/repomd.xml.key +EOF +``` + diff --git a/docs/install/templates/provisioner/xcat/xcat-setup.md.j2 b/docs/install/templates/provisioner/xcat/xcat-setup.md.j2 new file mode 100644 index 0000000000..299abfc38b --- /dev/null +++ b/docs/install/templates/provisioner/xcat/xcat-setup.md.j2 @@ -0,0 +1,31 @@ +## Complete Basic xCAT Setup for *Head Node* + +With xCAT installed, configure the internal provisioning interface and register +it with xCAT for DHCP service. Then set the cluster domain name used for +system-wide name resolution. + + + +```bash +# Verify required variables are set +: "${sms_eth_internal:?ERROR: sms_eth_internal not set}" +: "${sms_ip:?ERROR: sms_ip not set}" +: "${internal_netmask:?ERROR: internal_netmask not set}" +: "${domain_name:?ERROR: domain_name not set}" + +# Enable internal interface for provisioning +ip link set dev ${sms_eth_internal} up +ip address add ${sms_ip}/${internal_netmask} broadcast + \ + dev ${sms_eth_internal} + +# Register internal provisioning interface with xCAT for DHCP +chdef -t site dhcpinterfaces="xcatmn|${sms_eth_internal}" + +# Point provisioning services at the internal interface address. Without +# this, a multi-homed head node may auto-select its external address. +chdef -t site master=${sms_ip} nameservers=${sms_ip} + +# Define local domain name +chdef -t site domain=${domain_name} +``` + diff --git a/docs/install/templates/scheduler/slurm/test-job.md.j2 b/docs/install/templates/scheduler/slurm/test-job.md.j2 index 1965bc8521..6e2afcdefe 100644 --- a/docs/install/templates/scheduler/slurm/test-job.md.j2 +++ b/docs/install/templates/scheduler/slurm/test-job.md.j2 @@ -1,8 +1,10 @@ ## Run a Test Job With the resource manager enabled for production usage, users should now be able -to run jobs. To demonstrate this, we will add a "test" user on the **head** node -that can be used to run an example job. +{% if not is_xcat %}to run jobs. To demonstrate this, we will add a "test" user on the **head** node +that can be used to run an example job.{% else %}to run jobs. To demonstrate +this, the "test" user added in the Post Provisioning section can be used to run +an example job.{% endif %} OpenHPC includes a simple "hello-world" MPI application in the `/opt/ohpc/pub/examples` directory that can be used for this quick compilation