Integrate Open OnDemand with Magic Castle

[mwytang]

Scope:

• An Open OnDemand (OOD) puppet module
• Interactive plugins for Virtual Desktop, JupyterHub, and RStudio
• Automated provisioning where OOD puppet module deployment is integrated with Magic Castle
• Documentation

[cmd-ntrf]

I recently came across OSC Puppet module for OOD.
https://forge.puppet.com/modules/osc/openondemand/readme

It is probably a good starting point.

[mwytang]

thanks! We also recently discovered the GitHub repo for that puppet module - https://github.com/OSC/puppet-module-openondemand/blob/master/README.md

[misaell2]

Hello, I am trying to get SLURM as a plugin for OOD. I am following directions from this link:

https://osc.github.io/ood-documentation/latest/installation/resource-manager/slurm.html

But was wondering if anyone here was able to get SLURM on OOD and if they had any suggestions.

[cmd-ntrf]

Manually Installing Open OnDemand Behind the Magic Castle Reverse Proxy

Objective

The objective is to make Open OnDemand available in a Magic Castle deployment.

This procedure documents the manual installation and configuration steps before the feature is fully automated in Magic Castle.

The target architecture is:

User browser
    |
    v
https://ondemand.autoscale.calculquebec.cloud
    |
    v
Magic Castle reverse proxy
    |
    v
https://ood1:443
    |
    v
Open OnDemand

---

Step 1 — Add NFS and Slurm submitter profiles to the Open OnDemand role

Before creating the ood1 instance, add the required Puppet profiles for the ood role.

On mgmt1, edit the Puppet site data file:

/etc/puppetlabs/code/environments/production/data/site.yaml

Add the following profiles for the ood role:

ood:
  - profile::nfs
  - profile::slurm::submitter

The profile::nfs profile gives the Open OnDemand node access to shared filesystems, such as home directories or project storage.

The profile::slurm::submitter profile configures the Open OnDemand node as a Slurm submit host, allowing Open OnDemand to submit jobs to the cluster scheduler.

Adding these profiles before the instance is created allows Puppet to configure the Open OnDemand node correctly during its first configuration run.

---

Step 2 — Add the Open OnDemand route to the reverse proxy

On mgmt1, edit:

/etc/puppetlabs/code/environments/production/data/common.yaml

Add the Open OnDemand subdomain mapping:

profile::reverse_proxy::subdomains:
  ondemand: "https://ood1:443"

This configures the Magic Castle reverse proxy to route requests for:

https://ondemand.autoscale.calculquebec.cloud

to the internal Open OnDemand host:

https://ood1:443

The reverse proxy becomes the only public web entry point. The Open OnDemand instance remains internal.

---

Step 3 — Add an Open OnDemand instance

Add a new instance group for Open OnDemand using a dedicated ood tag.

ood = {
  type  = "p4-4gb"
  tags  = ["ood"]
  count = 1
}

This creates a dedicated VM for Open OnDemand. The ood tag identifies the instance for role assignment and future automation.

The instance does not need the public tag because users will reach Open OnDemand through the existing reverse proxy.

The resulting instance is expected to be named:

ood1

---

Step 4 — Wait for initial provisioning

Wait for the ood1 instance to complete its initial Magic Castle provisioning.

Before continuing, verify that the instance is reachable internally and that its base configuration has completed.

At this point, the node should already have received the profile::nfs and profile::slurm::submitter profiles from Puppet, and the reverse proxy should already know about the ood subdomain route.

---

Step 5 — Create the HTTP service principal in FreeIPA

Create a FreeIPA HTTP service principal for the internal Open OnDemand host.

Authenticate as an IPA administrator:

kinit admin

Create the HTTP service principal:

ipa service-add HTTP/ood1.int.autoscale.calculquebec.cloud

This creates the service principal:

HTTP/ood1.int.autoscale.calculquebec.cloud

The service principal is required so that the Open OnDemand host can request and manage an HTTPD TLS certificate through FreeIPA/certmonger.

---

Step 6 — Generate the internal SSL certificate on ood1

On ood1, request a certificate for Apache HTTPD:

sudo ipa-getcert request \
  -f /etc/pki/tls/certs/httpd.crt \
  -k /etc/pki/tls/private/httpd.key \
  -K HTTP/ood1.int.autoscale.calculquebec.cloud \
  -D ood1.int.autoscale.calculquebec.cloud

---

Step 7 — Install Open OnDemand manually

sudo dnf module enable ruby:3.3 nodejs:22
sudo dnf install https://yum.osc.edu/ondemand/4.2/ondemand-release-web-4.2-1.el9.noarch.rpm
sudo dnf install ondemand
sudo dnf install ondemand-dex
sudo dnf install mod_lua

More info on the official Open OnDemand installation documentation:

https://osc.github.io/ood-documentation/latest/installation/install-software.html

Install Open OnDemand on ood1 according to the operating system used by the Magic Castle deployment.

This procedure currently focuses on the Magic Castle-specific integration steps around the Open OnDemand installation. The exact package installation commands should be added once the target OS and Open OnDemand package source are finalized.

---

Step 8 — Configure Open OnDemand portal settings

On ood1, configure:

/etc/ood/config/ood_portal.yml

Add or update the following settings:

---
servername: ood.autoscale.calculquebec.cloud
dex_uri: /dex

oidc_settings:
  OIDCXForwardedHeaders: "X-Forwarded-Proto X-Forwarded-Host"

ssl:
  - 'SSLCertificateFile "/etc/pki/tls/certs/httpd.crt"'
  - 'SSLCertificateKeyFile "/etc/pki/tls/private/httpd.key"'
  - 'SSLCertificateChainFile "/etc/pki/tls/certs/ca-bundle.crt"'

server_aliases:
   - "ood.autoscale.calculquebec.cloud"

dex:
  storage:
    type: sqlite3
    config:
      file: "/etc/ood/dex/dex.db"
  ssl: false
  connectors:
    - type: ldap
      id: ldap
      name: LDAP
      config:
        host: ipa.int.autoscale.calculquebec.cloud:636
        insecureSkipVerify: false
        bindDN: "uid=admin,cn=users,cn=accounts,dc=int,dc=autoscale,dc=calculquebec,dc=cloud"
        bindPW: "<REDACTED>"
        userSearch:
          baseDN: cn=users,cn=accounts,dc=int,dc=autoscale,dc=calculquebec,dc=cloud
          filter: "(objectClass=posixAccount)"
          username: uid
          idAttr: uid
          emailAttr: mail
          nameAttr: gecos
          preferredUsernameAttr: uid
        groupSearch:
          baseDN: ou=Groups,dc=int,dc=autoscale,dc=calculquebec,dc=cloud
          filter: "(objectClass=posixGroup)"
          userMatchers:
            - userAttr: DN
              groupAttr: member
          nameAttr: cn

The servername must be the externally visible Open OnDemand hostname:

ondemand.autoscale.calculquebec.cloud

The following setting is required because Open OnDemand is accessed through the reverse proxy:

oidc_settings:
  OIDCXForwardedHeaders: "X-Forwarded-Proto X-Forwarded-Host"

This allows the OIDC layer to use the forwarded protocol and host headers from the reverse proxy. Without this, Open OnDemand or Dex may generate incorrect redirects using the internal hostname or wrong scheme.

The ssl section points Apache/Open OnDemand to the certificate generated by ipa-getcert.

The dex section configures Open OnDemand’s embedded Dex service to authenticate users against FreeIPA over LDAPS.

Important: the LDAP bind password must not be hardcoded in the final automated implementation. It should be generated, stored, and rendered through Magic Castle’s normal secret-management mechanism. Any password used during manual testing should be rotated before the environment is reused.

---

Step 9 — Create the Open OnDemand cluster definition

On ood1, create the directory where Open OnDemand expects cluster configuration files:

sudo mkdir -p /etc/ood/config/clusters.d

Then create the following file:

/etc/ood/config/clusters.d/cluster.yaml

Add the following content:

---
v2:
   metadata:
     title: "My Cluster"
   login:
     host: "autoscale.calculquebec.cloud"
   job:
     adapter: "slurm"
     cluster: "autoscale"
     bin: "/opt/software/slurm/bin"
     conf: "/etc/slurm/slurm.conf"
     copy_environment: false

---

Step 10 — Start Open OnDemand services

Restart Dex and Apache HTTPD so the new portal, Dex, and cluster configuration are applied:

sudo systemctl start ondemand-dex
sudo systemctl start httpd

At this point, Open OnDemand should use the configured server name, SSL certificate, reverse-proxy headers, LDAP authentication settings, Dex SQLite storage, and Slurm cluster definition.

Users should access Open OnDemand at:

https://ondemand.autoscale.calculquebec.cloud

---

Current manual workflow summary

The revised workflow is:

1. Add profile::nfs and profile::slurm::submitter to the ood role.
2. Add the ondemand subdomain route to the Magic Castle reverse proxy.
3. Add the dedicated internal ood instance.
4. Wait for ood1 provisioning.
5. Create the FreeIPA HTTP service principal for ood1.
6. Generate the internal HTTPD SSL certificate with ipa-getcert.
7. Install Open OnDemand manually on ood1.
8. Configure /etc/ood/config/ood_portal.yml with the public ondemand hostname, SSL certificate paths, forwarded OIDC headers, and FreeIPA LDAP connector.
9. Create /etc/ood/config/clusters.d/cluster.yaml.
10. Start ondemand-dex and httpd.

---

Items to automate

The following pieces will have to be automated:

• Assignment of profile::nfs to the ood role.
• Assignment of profile::slurm::submitter to the ood role.
• Creation of the reverse proxy subdomain entry.
• Creation of the internal ood instance group.
• Creation of the FreeIPA HTTP service principal.
• Request and tracking of the HTTPD certificate through certmonger.
• Installation of Open OnDemand packages.
• Generation of /etc/ood/config/ood_portal.yml.
• Generation of /etc/ood/dex/config.yaml.
• Secure handling of the LDAP bind credential.
• Creation of /etc/ood/config/clusters.d/cluster.yaml.
• Restarting ondemand-dex and httpd after configuration changes.
• Proper host-level firewall management for internal HTTPS access between the reverse proxy and ood1.