- Gateway: A Pangolin server running on a vps like Digital Ocean
- Homeserver: A server behind a nat
homeserver.ymlconfigures homeserver with the compose filesdns.ymlconfigures vps with blocky dns and letsencrypt
- create a simple butane configuration
- transpile it to an ignition file
- validate the ignition file
podman run --pull=always --rm -i quay.io/coreos/ignition-validate:release - < ucore.ign
This example uses podman, but docker can be used too
- start an http server to host the ingnition file and prepare for install
sudo podman run --rm --privileged -p 8080:5000 --security-opt label=disable -v $PWD/ucore.ign:/html/ucore.ign ghcr.io/patrickdappollonio/docker-http-server:v2 - boot up the fedora core installer and install with your ignition file:
sudo coreos-installer install /dev/sda --ignition-url https://192.168.0.100:8080/ucore.ignand then reboot
- ensure tailscaled is running:
sudo systemctl start tailescaled(ansible will ensure its enabled) and then sign intailscale up ansible-playbook -i server.tailscale.hostname, homeserver.yml(you will need--ask-become-passif there is a sudo password)- The server should be ready to go 🎉
Once a host has already been fully set up, routine changes (e.g. editing homeserver/docker/compose.yml or another file under homeserver/docker/) don't need the full first-time-setup run. Use --tags deploy to just copy the compose files/secrets and run docker compose up:
ansible-playbook -i server.tailscale.hostname, homeserver.yml --tags deploy
homeserver-backup.yml is a separate playbook, not part of ordinary deployment.
It layers restic and rsync with rpm-ostree, never applies them live, and never
reboots automatically. If the booted deployment lacks them, it stops with an
instruction to reboot in your own maintenance window and rerun. It does not run
the existing homeserver playbook's deployment or reboot handlers.
homeserver/backup.yml lists explicit service groups and sources. Each group gets
a live rsync pre-copy, then only its currently running containers stop. A final
rsync --checksum copies a consistent state with ownership, permissions, ACLs,
xattrs and hardlinks preserved. Containers restart in reverse stop order before
the next group. Only after every group succeeds does restic upload staged data.
Network upload time is never intentional container downtime. Related services
share a group: Immich/server + PostgreSQL, Karakeep + Meilisearch, and download
clients + their VPN. No whole-stack compose down or docker pause is used.
This follows PostgreSQL's documented two-pass rsync backup approach. For databases, native dumps or storage-level snapshots are also conventional; raw copies of actively changing database files are not sufficient. Existing storage is configured as ext4 with no managed snapshot layer, so this does not add one.
Important limits:
- Checksumming still reads every file while that group is stopped. Large Immich libraries or Syncthing trees may have significant downtime even with pre-copy. Measure first; use coordinated filesystem snapshots or application-native backups if that pause is unacceptable. Stopping around the B2 upload is the simpler alternative, but would keep that app offline for the upload duration.
- Groups represent different capture times, not one atomic whole-host snapshot.
- PostgreSQL uses a complete cold physical copy, not a portable SQL dump. Restore with the same PostgreSQL major version and compatible extensions/image. No external tablespace or external WAL paths are covered by the current scope.
- Recovery records container IDs durably before stopping;
finally, systemdExecStopPost, and an enabled boot recovery unit retry restarts. Originally stopped containers stay stopped. Failed restarts retain the recovery record. Recovery checks running state, not application readiness or long-term health. - Do not deploy, recreate containers, or run other backups against these sources concurrently. The lock serializes this runner only. All writers to a source must be in its group; update scope and its coverage test when adding services.
- Staging needs a full extra copy plus free space for rsync replacement files.
Default location is
/var/lib/homelab-backup/staging, with a 10 GiB pre-copy free-space guard, not a capacity reservation. Choose a sufficiently large local disk and add its mount torequired_mountsif moved. Missing mounts, missing sources, forced kills, copy errors, and restic failures fail the job. - Staging contains plaintext secrets and data under a root-only directory. It is a mutable working copy, not an independent backup or historical restore point.
Includes declared app state under /mnt/homelab_iscsi, Immich uploads and database,
Syncthing config/data, homeserver_9router-data resolved through Docker volume
inspection, deployed Compose files/secrets, and backup configuration/credentials.
Excludes all /mnt/media content (including audiobooks), Prometheus history,
LanguageTool downloaded models, Docker images/overlay layers, and the named log,
cache and temporary volumes. Loki has no persistent data mount in this Compose
file. Unlisted/orphaned directories and other projects are not automatically
included. This is an application-data backup, not a bare-metal OS image.
Timers use the host timezone, a random delay up to 30 minutes and missed-run catch-up. They remain disabled by default:
- Daily 03:30: stage and back up. Successful snapshot ID is recorded locally.
- Sunday 10:00: repository check plus a rotating 1/12 data read; twelve consecutive scheduled runs cycle all subsets, though changing repository contents can shift subset membership. This does not replace restore testing or a full check.
- First of month 12:00: check, retain 14 daily / 8 weekly / 12 monthly / 3 yearly,
prune, then check again. Retention requires a verified successful staged snapshot
less than 48 hours old matching the current repository and retention scope
(
host,tag,retention). No daily prune.
Adding or editing a group in homeserver/backup.yml does not invalidate that
marker, so rerunning homeserver-backup.yml to add a service never forces an
immediate backup. Changing host, tag or retention does, since those decide
what forget/prune delete. The new group is simply first captured by the next
scheduled run.
Prune limits repacking to 1 GiB per run and permits 20% unused space to reduce
rewrites. This trades higher stored bytes for less repacking; the repack cap can
leave more than 20% unused. Review billed storage and loosen the cap if needed.
Retention filters by homeserver and homelab-staged-v1; old docker-volumes
snapshots are intentionally untouched. Prune operates on the whole repository
but preserves data referenced by any remaining snapshot. Use a dedicated repo.
Backblaze B2 has no minimum-storage-duration/early-deletion fee. Retained data,
hidden object versions, and applicable egress matter more than churn itself.
This config preserves the existing native b2: backend and its credential names.
Use a private bucket and a bucket-scoped application key with list/read/write/delete
permissions, not the account master key. Store the repository password and recovery
credentials separately in a password manager; do not rely on their backed-up copy.
Do not change the password when reusing the existing repository.
Never add a lifecycle rule that expires live restic objects by age: old packs can still hold current backup data. Native B2 deletions remove versions directly; if switching to S3 later, configure hidden/noncurrent-version cleanup deliberately because S3 deletion hides objects. Bucket Object Lock is not enabled here; normal prune needs delete access, so host compromise can delete backups. Independent immutable/offline copies need a separate design, not an untested retention toggle.
- Reuse the existing vaulted
secrets.env, includingRESTIC_REPOSITORY,RESTIC_PASSWORD,B2_ACCOUNT_ID, andB2_ACCOUNT_KEY. No separate backup secrets file is needed. The playbook decrypts it into a private temporary file, usesdocker compose configto resolve the four values without starting any containers, then removes the temporary file even if parsing fails. Only those four values are written to root-only/etc/homelab-backup/environment.json; output/diffs are suppressed. Rerun this playbook after changing credentials. Check mode skips credential extraction and writing; real deployment validates that every required value is nonempty. - Review scope, free space, host timezone and retention in
homeserver/backup.yml. Keepbackup_enable_timers: false. On your chosen rollout date runansible-playbook homeserver-backup.yml. If packages need a reboot, schedule it yourself and rerun afterward. Neither install nor configuration starts a backup. - Wait for any legacy backup/check/prune to finish. Stop and remove only
restic-backup,restic-prune, andrestic-check; do not kill an active prune. Their Compose definitions are now behind the opt-inlegacy-backupprofile, so normal deployment will not create them again. Profiles do not stop already running containers. The native runner refuses to proceed while any of those legacy containers still exist, even if stopped. - On the host, run
sudo homelab-backup restic snapshotsto verify existing repository access. Only for an intentionally new repository runsudo homelab-backup restic init --repository-version 2. No automatic init, migration, unlock, or fallback repository is performed. - Run
sudo systemctl start homelab-backup.service, inspectsudo journalctl -u homelab-backup.service, then runsudo systemctl start homelab-check.service. Check container health and measure each group's downtime before enabling scheduling. - Perform an isolated restore drill as described below. Then set
backup_enable_timers: trueand rerun the backup playbook. Activation checks the successful snapshot marker, remote snapshot, sources and legacy retirement. With persistent timers, enabling them may trigger a missed scheduled run.
Do not use --profile legacy-backup after migration. Retire old snapshot history
only after the new backup has passed a restore drill; no old history is deleted
automatically by this migration.
Run sudo homelab-backup restic snapshots --tag homelab-staged-v1, select an exact
successful snapshot ID, and inspect it with sudo homelab-backup restic ls SNAPSHOT_ID.
Restore into a new, private scratch directory, never over running application data:
sudo install -d -m 0700 /var/tmp/homelab-restore-test
sudo homelab-backup restic restore SNAPSHOT_ID --target /var/tmp/homelab-restore-test --verifyThe tree contains the staging path followed by group name and original source
path; named volumes use GROUP/volumes/VOLUME_NAME. Use restic ls rather than
assuming a layout. For disaster recovery, reinstall restic and recover credentials
from your password manager first; direct restic works without this runner.
Test a SQLite-backed app and Immich/PostgreSQL in isolated containers with matching images, restored ownership, and no production mounts/networks. Confirm login, database reads and representative media retrieval. For actual recovery, stop only the affected group, preserve its current data for rollback, restore its data and permissions, then start dependencies before applications. Treat Immich DB and uploads as one group. Repeat drills quarterly and after major app/DB upgrades.
Review systemctl list-timers 'homelab-*', systemctl --failed, job journals and
/var/lib/homelab-backup/last-success.json. Wire an external monitor to failed jobs
and a last-success age over 48 hours before relying on this unattended. No alert
destination was supplied, so no email/webhook or false promise of notifications
is configured. A hung runner times out after 12 hours; boot/stop recovery retries
container starts. Check actual service health after any interrupted run.
Local-only validation (no Docker daemon, host connection, or B2 access):
python3 -m unittest discover -s homeserver/tests -v
ANSIBLE_ASK_VAULT_PASS=false ansible-playbook -i localhost, --connection local homeserver-backup.yml --syntax-checkTests require local Python, PyYAML and rsync; the installed runner uses only Python
standard-library modules plus host Docker, rsync and restic. The backup playbook
also needs the community.general Ansible collection. Use a long random repository
password for a new repository; existing repository passwords must remain unchanged.
References: PostgreSQL documentation, "File System Level Backup" (two-pass rsync); rpm-ostree administrator handbook (offline package layering); restic documentation, "Preparing a new repository" and "Removing backup snapshots"; Backblaze B2 pricing.