Skip to content

Repository files navigation

ilha - Git Worktrees with Isolated Docker Environments

Python Version Tests Status

A comprehensive Python CLI tool that creates isolated development environments using Git worktrees, Docker Compose, and Caddy reverse proxy. Each feature branch gets its own complete environment with isolated databases, Redis, and media storage.

🚀 Quick Start

Installation

# Install via pip
pip install ilha

# Or install from source
git clone https://github.com/catalpainternational/ilha.git
cd ilha
pip install -e .

Basic Usage

# Initialize ilha in your project
ilha setup

# Start global Caddy proxy
ilha start-proxy

# Create and start a worktree
ilha create feature-auth
ilha feature-auth up -d

# Access your isolated environment (note: domain includes project name)
# If your project is named "myapp", the URL will be:
open http://myapp-feature-auth.localhost

# Export environment as shareable package (includes code by default)
ilha packages export feature-auth

# Import environment from package
ilha packages import myapp-feature-auth-2024-01-15.ilha-package.tar.gz

# Stop and clean up
ilha feature-auth down
ilha remove feature-auth

🏗️ Architecture Overview

ilha CLI provides complete environment isolation through:

  • Git Worktrees: Isolated working directories for each branch
  • Docker Compose: Isolated containers with branch-specific volumes
  • Caddy Proxy: Dynamic routing based on branch names
  • Volume Isolation: Branch-specific databases, Redis, and media storage
  • Package Management: Export/import complete environments as shareable packages

System Architecture

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Global Caddy  │    │  Worktree A      │    │  Worktree B     │
│   (Port 80)     │    │  feature-auth    │    │  feature-pay    │
│                 │    │                  │    │                 │
│  Routes to:     │───▶│  • PostgreSQL    │    │  • PostgreSQL   │
│  • *.localhost  │    │  • Redis         │    │  • Redis        │
│                 │    │  • Web App       │    │  • Web App      │
└─────────────────┘    └──────────────────┘    └─────────────────┘

📋 Command Reference

Important: ilha uses a worktree-name-first pattern. For up and down commands, you can use:

  • ilha <worktree_name> up and ilha <worktree_name> down (explicit worktree name)
  • ilha up and ilha down (auto-detected from current directory or current git branch)

The shorthand ilha up and ilha down automatically resolve the worktree from your current directory (if you're in a worktree) or from your current git branch (if you're at the project root).

Setup Commands

Command Description Example
setup Initialize ilha for this project ilha setup [--monkey-patch]

Global Caddy Management

Command Description Example
start-proxy Start global Caddy proxy container ilha start-proxy
stop-proxy Stop global Caddy proxy container ilha stop-proxy
start Alias for start-proxy ilha start
stop Alias for stop-proxy ilha stop

Command Aliases

Alias Full Command Description
-D delete Delete worktree and branch completely
-r remove Remove worktree but keep git branch

Worktree Lifecycle

Command Description Example
create <branch> Create worktree ilha create feature-auth
<branch> up -d Start worktree environment ilha feature-auth up -d
up Start worktree environment (auto-detected) ilha up (from worktree directory or project root)
<branch> down Stop worktree environment ilha feature-auth down
down Stop worktree environment (auto-detected) ilha down (from worktree directory or project root)
remove <branch> Remove worktree (keep branch) ilha remove feature-auth
delete <branch> Delete worktree and branch ilha delete feature-auth

Note: The ilha up and ilha down commands automatically detect the worktree from:

  • Your current directory (if you're inside a worktree directory)
  • Your current git branch (if you're at the project root and a worktree exists for that branch)

Important: Branch names must match exactly. The delete and remove commands check worktrees, branches, and docker volumes for exact matches only. If no exact match is found, an error message will show what was checked.

Docker Compose Commands

ilha supports direct passthrough to Docker Compose commands using the pattern: ilha <worktree_name> <compose-command>

Command Pattern Description Example
<branch> exec <service> <cmd> Execute command in container ilha feature-auth exec web python manage.py migrate
<branch> logs <service> View container logs ilha feature-auth logs web
<branch> ps List containers ilha feature-auth ps
<branch> run <service> <cmd> Run one-off command ilha feature-auth run --rm web python manage.py test
<branch> build Build services ilha feature-auth build
<branch> restart <service> Restart service ilha feature-auth restart web
<branch> <compose-cmd> Any docker compose command ilha feature-auth pull, ilha feature-auth config

Supported Docker Compose Commands:

  • exec, logs, ps, run, build, pull, push, restart
  • start, stop, up, down, config, images, port
  • top, events, kill, pause, unpause, scale

Wildcard Operations

Command Description Example
remove <pattern> Remove worktrees matching pattern ilha remove test-*
delete <pattern> Delete worktrees and branches matching pattern ilha delete feature-*

Wildcard Patterns:

  • * - Matches any characters (e.g., test-* matches test-feature, test-bugfix)
  • ? - Matches single character (e.g., test-? matches test-1, test-a)
  • [abc] - Matches any character in brackets (e.g., test-[abc] matches test-a, test-b, test-c)
  • Case-insensitive matching (e.g., test-* matches Test-Feature, TEST-FEATURE)

Bulk Operations

Command Description Example
remove-all Remove all worktrees (keep branches) ilha remove-all
delete-all Delete all worktrees and branches ilha delete-all --force

Utility Commands

Command Description Example
list List active worktrees ilha list
prune Remove prunable worktrees ilha prune
help Show help information ilha help
clean-legacy Clean legacy ilha elements ilha clean-legacy

Volume Management

Command Description Example
volumes list List all worktree volumes ilha volumes list
volumes size Show volume sizes ilha volumes size
volumes backup <branch> Backup worktree volumes ilha volumes backup feature-auth
volumes restore <branch> <file> Restore from backup ilha volumes restore feature-auth backup.tar
volumes clean <branch> Clean up volumes ilha volumes clean feature-auth

Image Publishing

Command Description Example
image build [branch] Build and tag local Docker images for buildable compose services ilha image build staging --registry catalpa --save
image push [branch] Push already-built image tags to the configured registry ilha image push staging
image publish [branch] Build, push, and write .ilha/images/<branch>.json with exact image refs/digests ilha image publish staging
image list [branch] List published image manifests; add --remote to include DOCR tags ilha image list staging --remote
image destroy <branch> Delete published registry tags and local manifest (comma-separated branches supported) ilha image destroy staging --force

Image List Options:

  • branch_name (optional argument) - Filter to one branch manifest
  • --remote - Query DOCR via DigitalOcean API (requires DIGITALOCEAN_API_TOKEN)
  • --registry <name> - DigitalOcean Container Registry name (for --remote)
  • --repository <path> - Repository path prefix (e.g. bower/web); defaults from manifest when listing one branch
  • --json - Output as JSON

Image Destroy Options:

  • <branch> (required argument) - One or more comma-separated branch names
  • --service <name> - Apply registry deletion to matching compose service(s) only (repeatable)
  • --manifest-only - Remove .ilha/images/<branch>.json only; skip DOCR tag deletion
  • --registry-only - Delete DOCR tags only; keep local manifest
  • --force - Skip confirmation prompts
  • --registry <name> - Override DigitalOcean registry name for tag deletion

Image preferences are project-local and live in .ilha/config.yml:

image:
  provider: digitalocean
  registry: catalpa
  repository: bower
  platform: linux/amd64
  tag_template: "{branch}-{sha7}"

CLI flags override config for one run; add --save to persist reusable registry/repository/platform settings. DigitalOcean Container Registry images use registry.digitalocean.com/<registry>/<repository>/<service>:<tag> and authenticate via the DigitalOcean API (docker login credentials). Published manifests are written to .ilha/images/<branch>.json; use ilha image list --remote to query DOCR tags via the DO API. Worker deploy reads digest refs from those manifests via ImageManager.image_ref_for_service. Future deploy orchestration should consume the manifest digest when available rather than relying on mutable tags.

Do not bake secrets into images. ilha image build rejects sensitive-looking Docker build.args (for example DJANGO_SECRET_KEY, *_TOKEN, *_PASSWORD). It also refuses to build when the context contains .ilha/, secrets/, or backups/ unless those paths are listed in .dockerignore.

SOPS Secret Profiles

Store production secrets in encrypted SOPS files committed to git. Decrypt only at command boundaries (local compose, deploy). Worker env is copied over SSH after the droplet boots — never via cloud-init user_data.

Configure profiles in .ilha/config.yml:

secrets:
  provider: sops
  default_profile: staging
  profiles:
    staging:
      file: .ilha/env.staging.sops.env
    production:
      file: .ilha/env.production.sops.env
Command Description Example
secrets list List configured profiles ilha secrets list
secrets edit <profile> Edit encrypted file with sops ilha secrets edit staging
secrets print <profile> Show decrypted env (--redact to hide values) ilha secrets print staging --redact
secrets exec <profile> -- <cmd> Run a command with profile env ilha secrets exec staging -- ilha worker deploy staging
secrets materialize <profile> --output <path> Write plaintext env (requires --force to overwrite) ilha secrets materialize staging --output .ilha/env.ilha --force

Use --secrets <profile> on deploy/runtime commands:

ilha staging up -d --secrets staging
ilha droplet deploy staging --domain app.example.com --secrets production
ilha worker deploy staging --secrets staging --count 2 --central-droplet-name central

Requires sops and age keys on the machine running ilha. SOPS protects secrets at rest in git; runtime droplets still need plaintext env for Docker — restrict worker exposure with worker_pool.env_allowlist.

Ephemeral Worker Pool

Spin up short-lived RQ worker droplets that pull jobs from central Redis and report to central DB. Cloud-init only installs Docker and writes compose YAML (image refs, not secrets). App env and DOCR_READ_TOKEN are copied over SSH after the droplet is up — DigitalOcean stores user_data in plaintext metadata.

Command Description Example
worker bundle [branch] Generate minimal rq-worker compose + env for a droplet ilha worker bundle staging --central-droplet-name central
worker deploy [branch] Create worker droplet(s), then SSH-copy env and start the worker ilha worker deploy staging --count 2 --central-droplet-name central
worker list List worker-tagged droplets ilha worker list --branch staging
worker reap Destroy workers older than worker_pool.idle_destroy_after_seconds ilha worker reap --branch staging --dry-run
worker destroy Destroy worker droplets ilha worker destroy --all --branch staging --force

Recommended flow:

  1. ilha image publish staging — build, push, and write the image manifest
  2. Deploy central via ilha droplet deploy (db, redis, web on one droplet in a VPC)
  3. ilha worker deploy staging --count N --central-droplet-name <central> when jobs need capacity
  4. Autoscale via Django autoscale_workers + cron (see scripts/worker_autoscale.cron in consuming projects)
  5. ilha worker reap --branch staging or ilha worker destroy --all --branch staging when the queue is empty

Configure defaults in .ilha/config.yml under worker_pool (central droplet name, size, region, max_workers, ssh_keys). Requires DIGITALOCEAN_API_TOKEN on the operator machine only (never allowlisted onto workers), DOCR_READ_TOKEN for SSH docker login, SSH keys, and a published image manifest for the branch. Deploy fails before creating a droplet if the manifest, registry token, SSH keys, or max_workers cap is missing.

Droplet Management

Command Description Example
droplet deploy [branch] Create a droplet and deploy via rsync + remote docker compose (slim mode). Default restores local volumes onto a new droplet; pass --no-restore-volumes for empty volumes. Prefer ilha image publish so the server pulls digest refs; --build compiles on the droplet. ilha droplet deploy staging --domain app.example.com --build
droplet deploy --no-create Deploy to an existing server (skip droplet creation) ilha droplet deploy --no-create --scp-target root@1.2.3.4:/root
droplet put [branch] Update an existing deployment. Default is code + config only (does not copy the laptop DB). --restore-volumes replaces remote data. ilha droplet put staging --domain app.example.com
droplet dump-db [branch] Custom-format pg_dump from the remote db container into backups/ ilha droplet dump-db staging --domain app.example.com
droplet status [branch] Compare remote /app/.ilha/deploy.json to local HEAD ilha droplet status staging
droplet ssh [branch] Interactive SSH (or run a remote command) ilha droplet ssh staging
droplet exec [branch] SERVICE CMD… docker compose exec on the droplet ilha droplet exec staging web python manage.py migrate
droplet rename OLD NEW Rename a DigitalOcean droplet; --set-hostname updates the OS hostname ilha droplet rename staging-new staging --set-hostname
droplet logs [branch] Tail Docker Compose logs from a remote droplet ilha droplet logs staging or ilha droplet logs -s web -f
ilha doctor [branch] Diagnose a local worktree (not a remote droplet) ilha doctor staging
droplet list List all droplets (formatted table) ilha droplet list
droplet list --as-json List droplets as JSON ilha droplet list --as-json
droplet list --as-csv List droplets as CSV ilha droplet list --as-csv
droplet regions List available Digital Ocean regions (table) ilha droplet regions
droplet regions --show-all Include unavailable regions in region list ilha droplet regions --show-all
droplet regions --as-json List regions as JSON ilha droplet regions --as-json
droplet regions --as-csv List regions as CSV ilha droplet regions --as-csv
droplet info <id> Get droplet information ilha droplet info 12345678
droplet destroy <id> Destroy a droplet (supports ID or name, comma-separated) ilha droplet destroy 12345678 or ilha droplet destroy my-droplet or ilha droplet destroy 123,456,789

Droplet Deploy Options:

  • branch_name (optional argument) - Branch/worktree name (auto-detected from current directory if not provided)
  • --region <region> - Droplet region (e.g. nyc1, sfo3). Use ilha droplet regions to list available regions (defaults from env or nyc1)
  • --size <size> - Droplet size (defaults from env or s-1vcpu-1gb)
  • --image <image> - Droplet image (defaults from env or ubuntu-22-04-x64)
  • --ssh-keys <key> - SSH key names (comma-separated)
  • --tags <tag> - Tags for the droplet (can be specified multiple times)
  • --api-token <token> - Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)
  • --create-only - Only create droplet, skip deploy
  • --no-create - Skip droplet creation, deploy to existing server (requires --scp-target or --domain/--ip)
  • --scp-target <target> - SCP target (optional, defaults to root@:/root)
  • --vpc-uuid <uuid> - VPC UUID for the droplet (if not provided, uses default VPC for the region)
  • --central-droplet-name <name> - Name of central droplet to reuse VPC UUID from (for worker deployments)
  • --domain - Domain for HTTPS deployment
  • --ip - IP for HTTP-only deployment
  • --dns-token - DNS API token for domain management
  • --build - Rebuild Docker images on server (docker compose build before app start)
  • --no-build - Never run a remote compose build; fail if the web image is missing (mutually exclusive with --build)
  • --restore-volumes - Backup local worktree volumes and restore them on the server (opt-in; avoids accidental DB password mismatches on fresh droplets)
  • --force-password-reset - After --restore-volumes, skip the Postgres password vs .env validation (dangerous)
  • --debug - Enable DEBUG mode on server
  • --use-staging-certificates - Use Let's Encrypt staging certificates

Droplet Logs Options:

  • --service, -s <name> - Service name(s) to show logs for (can be repeated, default: all)
  • --tail, -n <lines> - Number of lines to show (default: 100)
  • --follow, -f - Follow log output (stream, Ctrl+C to stop)
  • --scp-target / --domain / --ip - Target override (falls back to stored config)

Droplet Name Auto-Detection:

  • If --domain is provided: uses subdomain as droplet name (e.g., app.example.com → app)
  • Otherwise: uses branch/worktree name as droplet name (auto-detected if not provided)

Droplet List Options:

  • --as-json or --json - Output as JSON format
  • --as-csv - Output as CSV format
  • --api-token <token> - Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)

Droplet Regions Options:

  • --show-all - Include unavailable regions
  • --as-json or --json - Output as JSON format
  • --as-csv - Output as CSV format
  • --api-token <token> - Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)

Droplet Destroy Options:

  • <id> - Single droplet ID or name, or comma-separated list of IDs/names (e.g., 123,456,789 or my-droplet,another-droplet,123)
  • --force - Skip confirmation (destroys without typing droplet name)
  • --only-droplet - Only destroy droplet, skip DNS deletion
  • --only-domain - Only destroy DNS records, skip droplet deletion
  • --domain <domain> - Domain name for DNS deletion (optional, auto-detects if not provided)
  • --api-token <token> - Digital Ocean API token (or use DIGITALOCEAN_API_TOKEN env var)
  • --dns-token <token> - DNS API token (if different from droplet token)
  • --json - Output as JSON format

Droplet Destroy Behavior:

  • Default: Destroys droplet only (backward compatible)
  • Multiple Droplets: Accepts comma-separated IDs or names (e.g., 123,456,789 or my-droplet,another-droplet,123) and processes all droplets sequentially
  • Name Resolution: Automatically resolves droplet names to IDs. If multiple droplets share the same name, use the droplet ID instead.
  • Error Handling: Continues destroying remaining droplets even if one fails
  • Summary Output: Shows summary of destroyed/failed droplets when processing multiple IDs
  • Confirmation: Requires typing the exact droplet name to confirm (unless --force) - each droplet requires confirmation separately
  • DNS Auto-detection: When destroying droplet, automatically finds and deletes DNS records pointing to droplet IP
  • DNS-only mode: Use --only-domain to delete DNS records without destroying the droplet
  • Domain confirmation: When deleting DNS records, requires typing the full domain name (e.g., "app.example.com") unless --force
  • Domain override: Use --domain <domain> to limit DNS search to specific domain

Package Management

Command Description Example
packages export <branch> Export worktree as shareable package (includes code by default) ilha packages export feature-auth
packages import <file> Import environment from package (auto-detects standalone mode) ilha packages import my-package.tar.gz
packages import <file> --domain <sub.domain.tld> Import with domain override (HTTPS via Caddy) ilha packages import pkg.tar.gz --domain myapp.example.com
packages import <file> --ip <x.x.x.x> Import with IP override (HTTP-only) ilha packages import pkg.tar.gz --ip 203.0.113.10
packages import <file> --standalone Force standalone import (create new project) ilha packages import my-package.tar.gz --standalone --target-dir ./myproject
packages list List available packages ilha packages list
packages validate <file> Validate package integrity ilha packages validate my-package.tar.gz

Standalone Package Import

ilha intelligently detects whether you're in an existing project and automatically switches between normal and standalone import modes.

Automatic Detection (Default):

# In an empty directory - automatically creates new project
cd /path/to/new-location
ilha packages import my-package.tar.gz
# Auto-detects: No git repo → standalone mode
# Creates: ./myproject-standalone/

# In existing ilha project - automatically imports as worktree
cd /path/to/existing-project
ilha packages import my-package.tar.gz
# Auto-detects: Existing .ilha → normal mode
# Creates: new worktree in project

Explicit Standalone Mode:

# Force standalone import with custom directory
ilha packages import my-package.tar.gz --standalone --target-dir ./my-new-project

# Force standalone even if in existing project
ilha packages import my-package.tar.gz --standalone

Requirements:

  • Standalone imports require packages exported with code (--include-code, which is the default)
  • Normal imports work with or without code

What Gets Created in Standalone Mode:

  1. Complete project structure extracted from package
  2. Project root .ilha/ configuration
  3. Worktree with its own .ilha/ (fractal structure)
  4. All code files
  5. Docker volumes (if --restore-data)
  6. Ready-to-use isolated environment

Note: Standalone import is simple - it extracts the project tar and applies domain/IP overrides. No git initialization or additional setup needed since the package contains the complete fractal structure.

Shell Completion Management

Command Description Example
completion install [shell] Install shell completion ilha completion install
completion uninstall Remove shell completion ilha completion uninstall
completion status Show completion status ilha completion status

🔧 Configuration

Django Compatibility Checks

During ilha setup, if a Django project is detected (manage.py present and a settings.py found), ilha validates your settings are environment-driven for reverse proxy use:

  • ALLOWED_HOSTS (comma-separated)
  • CSRF_TRUSTED_ORIGINS (space-separated)
  • USE_X_FORWARDED_HOST (boolean)
  • SECURE_PROXY_SSL_HEADER (tuple as HTTP_X_FORWARDED_PROTO,https)

If any are missing, ilha prints a nicely formatted guidance block and, when --monkey-patch is supplied, appends a safe snippet to settings.py to read these values from environment variables.

Suggested snippet (auto-added with --monkey-patch):

import os
ALLOWED_HOSTS = os.getenv("ALLOWED_HOSTS", "localhost,127.0.0.1").split(",")
CSRF_TRUSTED_ORIGINS = os.getenv("CSRF_TRUSTED_ORIGINS", "").split()
USE_X_FORWARDED_HOST = os.getenv("USE_X_FORWARDED_HOST", "False") == "True"
_hdr = os.getenv("SECURE_PROXY_SSL_HEADER")
if _hdr:
    SECURE_PROXY_SSL_HEADER = tuple(_hdr.split(",", 1))

SQLite Persistence (Automatic)

ilha automatically configures SQLite persistence when you run ilha setup:

Automatic Configuration:

  • ✅ Creates sqlite_data Docker volume
  • ✅ Mounts volume at /data in web containers
  • ✅ Copies SQLite databases between worktrees
  • ✅ Includes SQLite in export/import packages

Django Configuration: Set your database path to use the volume:

# settings.py
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': os.getenv('SQLITE_PATH', '/data/db.sqlite3'),  # Use volume path
    }
}

Migration: If your database is currently in the source tree (e.g., db/db.sqlite3), migrate it:

# Copy to Docker volume
docker compose cp db/db.sqlite3 web:/data/db.sqlite3

# Or use ilha setup --monkey-patch to auto-configure
ilha setup --monkey-patch

Why This Matters:

  • 📁 Each worktree gets its own isolated database
  • 📦 Databases are included in ilha packages
  • 🔄 Database state persists across container rebuilds
  • ❌ Without volume: all worktrees share the same database file

Project Setup

When you run ilha setup, it creates a .ilha/ directory with:

  • config.yml - Project configuration
  • docker-compose.worktree.yml - Transformed compose file (generated from existing docker-compose.yml)
  • Caddyfile.ilha - Caddy configuration
  • README.md - User guide optimized for coding agents
  • worktrees/ - Worktree directories

Docker Compose File Usage

Important: ilha always uses worktrees/{branch}/.ilha/docker-compose.worktree.yml for all runtime operations. ilha never looks for or uses other compose files (such as docker-compose.yml in the project root or worktree directory) during runtime operations.

  • Setup Phase: During ilha setup, the base docker-compose.yml in the project root is read and transformed into .ilha/docker-compose.worktree.yml
  • Runtime Phase: All commands (up, down, exec, logs, etc.) exclusively use worktrees/{branch}/.ilha/docker-compose.worktree.yml
  • No Fallback: There is no backward compatibility or fallback logic - ilha will only use the worktree compose file

Droplet Configuration

Droplet defaults can be configured in .env or .ilha/env.ilha:

# .env or .ilha/env.ilha
DROPLET_DEFAULT_REGION=nyc1
DROPLET_DEFAULT_SIZE=s-1vcpu-1gb
DROPLET_DEFAULT_IMAGE=ubuntu-22-04-x64
# SSH key names (comma-separated, e.g., anders,peter)
# Only key names are supported, not numeric IDs or fingerprints
DROPLET_DEFAULT_SSH_KEYS=anders,peter
DIGITALOCEAN_API_TOKEN=your_token_here

Configuration File

The .ilha/config.yml file contains:

project_name: myproject
caddy_network: ilha_caddy_proxy
worktree_dir: worktrees
services:
  web:
    container_name_template: ${COMPOSE_PROJECT_NAME}-web
  db:
    container_name_template: ${COMPOSE_PROJECT_NAME}-db
  redis:
    container_name_template: ${COMPOSE_PROJECT_NAME}-redis
volumes:
  - postgres_data
  - redis_data
  - media_files
environment:
  DEBUG: "True"
  ALLOWED_HOSTS: "localhost,127.0.0.1,*.localhost,web"

Volume Naming Convention

Worktree-specific volumes follow the pattern: {branch_name}_{volume_type}

  • feature-auth_postgres_data - Database data
  • feature-auth_redis_data - Cache data
  • feature-auth_media_files - User uploads

Network Configuration

  • Global Network: ilha_caddy_proxy (external)
  • Worktree Networks: {branch_name}_internal, {branch_name}_web

🛠️ Development Workflows

Feature Development

# 1. Initialize ilha in your project
ilha setup

# 2. Start global infrastructure
ilha start-proxy

# 3. Create feature branch environment
ilha create feature-new-auth
ilha feature-new-auth up -d

# 4. Develop and test
open http://feature-new-auth.localhost
# Make changes, test database migrations, etc.

# 5. Clean up when done
ilha feature-new-auth down
ilha remove feature-new-auth

Multiple Feature Branches

# Work on multiple features simultaneously
ilha create feature-auth
ilha create feature-payments
ilha create feature-notifications

# Start all environments
ilha feature-auth up -d
ilha feature-payments up -d
ilha feature-notifications up -d

# Access each independently
open http://feature-auth.localhost
open http://feature-payments.localhost
open http://feature-notifications.localhost

Database Testing

# Test database migrations in isolation
ilha create test-migration
ilha test-migration up -d

# Run migrations, test data changes
# Each worktree has its own database

# Clean up test environment
ilha delete test-migration

Docker Compose Integration

# Execute Django management commands
ilha feature-auth exec web python manage.py migrate
ilha feature-auth exec web python manage.py collectstatic

# View application logs
ilha feature-auth logs web
ilha feature-auth logs -f web  # Follow logs

# Run one-off commands
ilha feature-auth run --rm web python manage.py test
ilha feature-auth run --rm web bash

# Check container status
ilha feature-auth ps

# Restart services
ilha feature-auth restart web

🔍 Troubleshooting

Common Issues

Docker not running

Error: Docker is not running. Please start Docker and try again.

Solution: Start Docker Desktop or Docker daemon

PostgreSQL Database Corruption

Error: invalid primary checkpoint record
Error: could not locate a valid checkpoint record

Solution: This has been fixed! ilha now stops the original database container before copying volumes, then restarts it after copying. This ensures data consistency and prevents corruption.

Worktree already exists

Error: Worktree for branch 'feature-auth' already exists

Solution: Use ilha list to see existing worktrees, or remove the existing one

Port conflicts

Error: Port 80 is already in use

Solution: Stop other services using port 80, or check if global Caddy is already running

Volume creation fails

Error: Failed to create volume

Solution: Check Docker disk space, ensure Docker has sufficient permissions

Branch not found during deletion

Error: No exact match found for 'feature-auth'. Checked: worktrees, branches, and docker volumes.

Solution: Use ilha list to see exact branch names. Branch names are case-sensitive and must match exactly.

HTTPS hangs or certificate acquisition fails (Let's Encrypt rate limit)

Error: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many certificates (5) already issued for this exact set of identifiers in the last 168h0m0s

Symptoms: HTTPS connections hang or timeout, HTTP works fine. Browser shows connection timeout when accessing https://your-domain.com.

Root Cause: Let's Encrypt limits certificate issuance to 5 certificates per domain per 168 hours (7 days). If you've deployed the same domain multiple times, you may hit this limit.

Solution: Use Let's Encrypt staging certificates temporarily until the rate limit expires:

  1. Manual Fix (Immediate): SSH to your server and update Caddy configuration:

    # Get current config
    curl -s http://localhost:2019/config/ > /tmp/caddy_config.json
    
    # Edit config to use staging endpoint (add "ca" field to issuer)
    # Update the issuer in apps.tls.automation.policies[0].issuers[0]:
    # Add: "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"
    
    # Reload Caddy
    curl -X POST http://localhost:2019/load -H "Content-Type: application/json" -d @/tmp/caddy_config.json
  2. Automatic Fix (Long-term): ilha's dynamic configuration script (caddy-dynamic-config.py) now automatically detects rate limit errors and falls back to staging certificates. The script checks Caddy logs for rate limit patterns and automatically switches to staging when detected.

  3. Force Staging Mode for Testing: You can force staging certificates for testing without hitting rate limits by setting the USE_STAGING_CERTIFICATES environment variable:

    # Option 1: Set in your shell before running ilha commands
    export USE_STAGING_CERTIFICATES=1
    
    # Option 2: Add to your .ilha/env.ilha file (recommended for droplet deploy/put)
    echo "USE_STAGING_CERTIFICATES=1" >> .ilha/env.ilha

    For droplet deploy and droplet put: Add USE_STAGING_CERTIFICATES=1 to your .ilha/env.ilha file before deploying. This ensures the setting is included and used on the remote server. Perfect for testing environments where you don't want to consume your production certificate quota. Staging certificates don't count against Let's Encrypt's rate limits.

  4. Staging Certificates:

    • Staging certificates work for HTTPS but show browser security warnings
    • Users can proceed after accepting the warning
    • Connection is still encrypted, just not trusted by default
    • Switch back to production certificates after rate limit expires (check Caddy logs for "retry after" timestamp)
    • Staging certificates don't count against Let's Encrypt rate limits - perfect for testing!
  5. Monitor Certificate Health: The caddy-docker-monitor.py script monitors certificate health and logs warnings when rate limits are detected.

Note: Rate limits expire after 168 hours (7 days) from the first certificate issuance. Check Caddy logs for the exact expiration time.

Debug Commands

# Check Docker status
docker ps

# Check worktree status
ilha list

# Check volume status
ilha volumes list

# Check network status
docker network ls | grep ilha_caddy_proxy

Logs and Debugging

# View container logs
docker logs ilha_caddy_proxy
docker logs {branch-name}-web

# Check worktree directory
ls -la worktrees/{branch-name}/

# Verify environment files
cat worktrees/{branch-name}/.ilha/env.ilha

⌨️ Shell Completion

ilha CLI provides intelligent tab completion for Bash and Zsh shells, making it faster and easier to use.

Features

  • Command Completion: Tab complete all ilha commands and subcommands
  • Worktree Names: Auto-complete worktree branch names for commands that need them
  • Volume Operations: Complete worktree names for volume backup/restore/clean operations
  • Flag Completion: Auto-complete command flags like --force, -d, --detach

Installation

Automatic Installation (Recommended)

Shell completion is automatically offered during project setup:

ilha setup
# After successful setup, you'll be prompted:
# "Would you like to install bash completion? [Y/n]"

Manual Installation

# Install for current shell (auto-detected)
ilha completion install

# Install for specific shell
ilha completion install bash
ilha completion install zsh

# Check installation status
ilha completion status

# Uninstall completion
ilha completion uninstall

Usage Examples

# Tab complete commands
ilha <TAB>
# Shows: create delete down help list prune remove remove-all setup start-proxy stop-proxy start stop up volumes

# Tab complete worktree names
ilha feature-auth <TAB>
# Shows: up down exec logs ps run build restart

# Tab complete volume operations
ilha volumes backup <TAB>
# Shows: feature-auth feature-payments feature-notifications

# Tab complete flags
ilha delete <TAB>
# Shows: feature-auth --force

Troubleshooting

Completion not working after installation:

# Restart your shell or source the configuration
source ~/.bashrc  # For Bash
source ~/.zshrc   # For Zsh

Check completion status:

ilha completion status

Reinstall completion:

ilha completion uninstall
ilha completion install

🚀 Advanced Usage

Custom Configuration

Create custom environment files for specific worktrees:

# Create worktree with custom settings
ilha create feature-auth

# Edit environment file
vim worktrees/feature-auth/.ilha/env.ilha

# Start with custom configuration
ilha feature-auth up -d

Volume Management

# Backup before major changes
ilha volumes backup feature-auth

# Check volume sizes
ilha volumes size

# Clean up old volumes
ilha volumes clean feature-auth

Package Management

Auto-Detection (recommended):

# Automatically chooses correct mode based on current directory
ilha packages import myapp-feature-auth.tar.gz

Standalone Import (create new project):

# Create new project from package
mkdir ~/new-project && cd ~/new-project
ilha packages import myapp-feature-auth.tar.gz --standalone

# Or specify target directory
ilha packages import myapp-feature-auth.tar.gz --standalone --target-dir ./myproject

Normal Import (add to existing project):

# Import to existing project as new worktree
cd /path/to/existing-project
ilha packages import myapp-feature-auth.tar.gz

Import Options:

# Import without data (configuration and code only)
ilha packages import myapp-feature-auth.tar.gz --no-data

# Import to specific branch name
ilha packages import myapp-feature-auth.tar.gz --target-branch new-branch-name

# Domain/IP overrides for deployment
ilha packages import myapp-feature-auth.tar.gz --standalone --domain myapp.example.com  # HTTPS via Caddy
ilha packages import myapp-feature-auth.tar.gz --standalone --ip 203.0.113.10           # HTTP-only (no TLS)

Export Options:

# Export complete environment as shareable package (includes code by default)
ilha packages export feature-auth

# Export environment without code (smaller package)
ilha packages export feature-auth --no-code

# Export to specific directory
ilha packages export feature-auth --output-dir ./exports

Package Management:

# List available packages
ilha packages list

# Validate package integrity
ilha packages validate myapp-feature-auth.tar.gz

Bulk Operations

# Remove all worktrees (keeps git branches)
ilha remove-all

# Delete all worktrees and branches (destructive)
ilha delete-all --force

# Clean up prunable worktrees
ilha prune

Wildcard Operations

# Remove all test branches (keeps git branches)
ilha remove test-*

# Delete all feature branches and their worktrees
ilha delete feature-*

# Remove branches matching pattern with confirmation
ilha remove bugfix-*
# Output: Found 3 matching branch(es): bugfix-auth, bugfix-payment, bugfix-ui
# Remove 3 worktree(s) (keep branches)? [Y/n]: 

# Delete with force flag (skips confirmation)
ilha delete temp-* --force

📚 Additional Documentation

Agent skills: This repo includes an Agent Skill at ilha/skills/ilha/ (inside the ilha package). Symlinks from .agents/skills/ilha and .claude/skills/ilha point to it so Cursor, Claude Code, and Codex discover the skill when the repo is open; no installation needed.

🤝 Contributing

The ilha CLI follows a modular architecture designed for easy extension:

  • Core Infrastructure: core/ - Docker, Git, and environment management
  • Command Layer: commands/ - Business logic for each command
  • Utilities: utils/ - Shared functionality and validation
  • Configuration: config/ - Settings and Docker Compose files

Adding New Commands

  1. Create command class in commands/
  2. Add CLI interface in cli.py
  3. Add tests in tests/unit/
  4. Update documentation

Extending Functionality

  • New Volume Types: Modify EnvironmentManager.get_volume_names()
  • Custom Networks: Update DockerManager.create_network()
  • Additional Validation: Extend utils/validation.py

📊 Performance

  • Worktree Creation: ~30 seconds
  • Volume Copying: ~60 seconds (depending on data size)
  • Container Startup: ~45 seconds
  • Memory Usage: ~1GB per worktree
  • Disk Usage: ~5GB per worktree (varies with data)

✅ Status

Production Ready: All tests passing, 100% functional compatibility with original bash script.

Key Features:

  • ✅ Complete environment isolation
  • ✅ Dynamic routing with Caddy
  • ✅ Volume management and backup
  • ✅ Git worktree integration
  • ✅ Docker Compose commands
  • ✅ Comprehensive error handling
  • ✅ Rich console output
  • ✅ Type safety throughout
  • ✅ Project-agnostic setup
  • ✅ User guide in each .ilha directory
  • ✅ JSON output mode for programmatic access
  • ✅ Slim deploy via rsync + remote docker compose (no Python/git/ilha on server)

🚀 Deployment

Deploy to a New Droplet

# Create a DigitalOcean droplet and deploy via rsync + remote docker compose
# Branch name auto-detected from current directory
ilha droplet deploy --domain app.example.com --ssh-keys anders

# With explicit branch and custom sizing
ilha droplet deploy staging \
  --domain staging.example.com \
  --ssh-keys anders \
  --region syd1 --size s-1vcpu-2gb

# Create droplet only, deploy later
ilha droplet deploy staging --create-only --ssh-keys anders

Deploy to an Existing Server

# Skip droplet creation, deploy to a server you already have
ilha droplet deploy staging --no-create --domain app.example.com
ilha droplet deploy staging --no-create --scp-target root@1.2.3.4:/root

Recommended flows

  • First deploy to a new droplet (domain + TLS + published images):
    ilha image publish <branch> then
    ilha droplet deploy <branch> --domain <fqdn> --ssh-keys <key> --region syd1 --size s-1vcpu-2gb
    Pass --build only when there is no registry manifest. --restore-volumes is the deploy default so a fresh droplet gets local DB/media; use --no-restore-volumes for empty volumes.
  • Update an existing droplet (never copies the laptop DB unless asked):
    ilha droplet dump-db <branch> --domain <fqdn>
    ilha image publish <branch>
    ilha droplet put <branch> --domain <fqdn>
    ilha droplet status <branch>
    SOPS secrets.default_profile is applied automatically; --no-secrets skips it. --restore-volumes replaces remote volumes (requires --i-mean-it or a prompt when a --domain is set).
  • Fast redeploy after server prep and images exist:
    ilha droplet deploy <branch> --domain <fqdn> --no-create --no-prepare --skip-volume-backup --no-restore-volumes
  • Ship local Postgres/Redis volumes to the server (explicit):
    ilha droplet put <branch> --domain <fqdn> --restore-volumes --i-mean-it
    After restore, ilha checks that Postgres accepts POSTGRES_PASSWORD from the merged .env before starting the app.
  • Reinstall the CLI after pulling this repo: uv tool install --force --editable . (ilha --version must match git).

Update an Existing Deployment

# Push code + config only (default; does not copy the laptop database)
ilha droplet put staging --domain app.example.com

# Dump remote Postgres first, then update
ilha droplet dump-db staging --domain app.example.com
ilha droplet put staging --domain app.example.com

# Replace remote volumes with the laptop copies (destructive)
ilha droplet put staging --domain app.example.com --restore-volumes --i-mean-it

View Remote Logs

# Tail all service logs (default: 100 lines)
ilha droplet logs staging

# Follow logs in real-time (Ctrl+C to stop)
ilha droplet logs staging -f

# Specific services
ilha droplet logs staging -s web -s db

# More lines
ilha droplet logs staging -n 500

# With explicit target
ilha droplet logs --domain app.example.com -s web -f

Local worktree diagnostics: ilha doctor staging. ilha droplet doctor was removed.

Multi-Project Deployments

ilha supports deploying multiple independent projects to the same server, each with their own unique URL. This enables efficient resource usage while maintaining complete isolation between projects.

How It Works

Global Caddy Proxy:

  • A single global Caddy container (ilha_caddy_proxy) runs on the server
  • Monitors all Docker containers on the host via /var/run/docker.sock
  • Automatically discovers containers with caddy.proxy labels from any project
  • Routes traffic based on domain/IP specified in container labels

Container Discovery:

  • Caddy dynamically scans all containers on the Docker host
  • Finds containers with caddy.proxy labels regardless of which project directory they belong to
  • Each project's containers are automatically discovered and routed

Shared Network:

  • All project containers join the ilha_caddy_proxy network (external)
  • This enables the global Caddy to route to containers from any project
  • Projects remain isolated but share the routing infrastructure

Deployment Example

# Deploy Project 1 (e.g., Django app) to server
cd /path/to/project1
ilha droplet deploy --no-create --domain app1.example.com

# Deploy Project 2 (e.g., Flask API) to same server
cd /path/to/project2
ilha droplet deploy --no-create --domain api.example.com

# Deploy Project 3 (e.g., Node.js app) to same server
cd /path/to/project3
ilha droplet deploy --no-create --domain app3.example.com

All three projects will:

  • ✅ Run independently with isolated containers and volumes
  • ✅ Be accessible via their own unique domains
  • ✅ Share the same global Caddy proxy for routing
  • ✅ Not interfere with each other

Requirements

  1. Global Caddy Must Be Running: Ensure ilha_caddy_proxy container is running on the server

    # On the server, start global Caddy if not already running
    ilha start-proxy
  2. Unique Domains/IPs: Each project must have a unique domain or IP address

    • ✅ app1.example.com, app2.example.com, api.example.com (all different)
    • ❌ Cannot use the same domain for multiple projects
  3. Project Isolation: Projects can be in completely different directories

    • Standalone mode works perfectly for multi-project deployments
    • Each project maintains its own .ilha/ configuration
  4. Automatic Label Configuration: ilha automatically adds caddy.proxy labels to containers

    • Labels are set based on --domain or --ip flags during deploy
    • No manual configuration needed

Benefits

  • Resource Efficiency: One reverse proxy handles all projects
  • Complete Isolation: Each project has its own containers, volumes, and networks
  • Simple Management: Deploy each project independently
  • Flexible: Mix domain-based HTTPS and IP-based HTTP deployments
  • Scalable: Add new projects without affecting existing ones

Architecture Diagram

┌─────────────────────────────────────────────────────────┐
│              Global Caddy Proxy                         │
│         (ilha_caddy_proxy)                        │
│         Ports: 80, 443, 2019                           │
│                                                         │
│  Monitors ALL containers via Docker socket              │
│  Routes based on caddy.proxy labels                     │
└─────────────────────────────────────────────────────────┘
                    │
        ┌───────────┼───────────┐
        │           │           │
┌───────▼──────┐ ┌──▼──────┐ ┌──▼──────┐
│  Project 1   │ │Project 2│ │Project 3│
│              │ │         │ │         │
│ app1.example │ │api.exam │ │app3.exam│
│     .com     │ │  ple.com│ │  ple.com│
│              │ │         │ │         │
│ • PostgreSQL │ │• MongoDB│ │• Redis  │
│ • Redis      │ │• Web    │ │• Web    │
│ • Web App    │ │• API    │ │• App    │
└──────────────┘ └─────────┘ └─────────┘

DNS Management

ilha can automatically manage DNS records via Digital Ocean DNS API:

  • Automatic Domain Creation: If a subdomain doesn't exist, ilha will automatically create it when --domain is provided
  • Domain Validation: Checks if DNS records already exist and point to the correct server
  • Supported Provider: Digital Ocean DNS
  • API Token Configuration: Multiple options supported (priority order):
    1. CLI flag: --dns-token <token>
    2. Shell environment: export DIGITALOCEAN_API_TOKEN=token
    3. .env file: Add DIGITALOCEAN_API_TOKEN=token to project root .env file
    4. Global config: Add DIGITALOCEAN_API_TOKEN=token to ~/.ilha/env.ilha file

DNS Provider Setup

Digital Ocean

  1. Generate a personal access token from https://cloud.digitalocean.com/account/api/tokens
  2. Configure token using one of these methods:
    • Shell environment (recommended for CI/CD): export DIGITALOCEAN_API_TOKEN=your_token
    • Project .env file (recommended for project-specific tokens): Add DIGITALOCEAN_API_TOKEN=your_token to your project root .env file
    • Global config (recommended for personal tokens): Add DIGITALOCEAN_API_TOKEN=your_token to ~/.ilha/env.ilha file
    • CLI flag: Use --dns-token <token> when pushing

DNS Propagation

When ilha creates a DNS record, it is immediately available on Digital Ocean's authoritative nameservers, but it takes time to propagate to all DNS resolvers worldwide.

What is DNS Propagation? DNS propagation is the time it takes for DNS record changes to spread across all DNS servers on the internet. When you create a new DNS record, it's immediately available on the authoritative nameservers (Digital Ocean's in this case), but other DNS resolvers (like Google's 8.8.8.8 or Cloudflare's 1.1.1.1) cache DNS records and may take time to update.

Expected Propagation Times:

  • Authoritative nameservers: Immediate (Digital Ocean nameservers)
  • Public resolvers: 5-60 minutes typically
  • Global propagation: Up to 48 hours in rare cases

Verifying DNS Records:

You can verify DNS records in several ways:

  1. Check on Digital Ocean nameservers (immediate):

    dig your-domain.com A @ns1.digitalocean.com
    dig your-domain.com A @ns2.digitalocean.com
    dig your-domain.com A @ns3.digitalocean.com
  2. Check on public resolvers (may take time):

    dig your-domain.com A @8.8.8.8      # Google DNS
    dig your-domain.com A @1.1.1.1      # Cloudflare DNS
  3. Check from your local machine:

    dig your-domain.com A
    nslookup your-domain.com

Troubleshooting:

  • If the record exists on Digital Ocean nameservers but not on public resolvers, wait a few more minutes
  • If the record doesn't exist on any nameserver, check that the DNS record was created successfully
  • Browser DNS caches may need to be cleared or wait for TTL expiration

VPC Deployments (DigitalOcean)

ilha supports VPC (Virtual Private Cloud) deployments for secure private networking between droplets. For ephemeral RQ workers that connect to a central droplet over private IP, see Ephemeral Worker Pool above.

Central and Worker Droplet Setup:

# 1. Deploy central server with db/redis services
ilha droplet deploy test \
  --domain central.example.com \
  --ssh-keys anders

# 2. Deploy worker in same VPC (reuses VPC UUID from central)
ilha droplet deploy test \
  --central-droplet-name central \
  --ssh-keys anders

VPC Features:

  • Automatic VPC Detection: Extracts private IP addresses and VPC UUID from droplets
  • VPC UUID Reuse: Use --central-droplet-name to automatically reuse VPC UUID from central droplet
  • Port Binding: Configure .ilha/config.yml to automatically bind ports for VPC-accessible services:
    vpc:
      auto_bind_ports: true  # Enable automatic port binding (default: false)
      bind_to_private_ip: true  # Bind to private IP instead of 0.0.0.0 (default: true, recommended for security)
  • Security: Redis and database ports are automatically bound to private IP (not public) when available, preventing public internet exposure
  • Firewall Configuration: Optional automatic UFW firewall rules to restrict access to VPC network only
  • Worker Environment Configuration: When deploying workers with --exclude-deps, environment variables are automatically configured to point to central server's private IP
  • VPC Metadata: Package metadata includes VPC deployment information for automatic configuration

VPC Configuration Options:

  • --vpc-uuid <uuid> - Explicitly specify VPC UUID
  • --central-droplet-name <name> - Reuse VPC UUID from central droplet (for worker deployments)
  • --exclude-deps <services> - Exclude services from dependency resolution (indicates worker deployment)

Security Best Practices for VPC Deployments:

When deploying with --central-server and VPC networking, ilha implements multiple security layers:

  1. Private IP Binding (Default: Enabled)

    • Redis and database ports bind to private IP address (e.g., 10.x.x.x:6379) instead of 0.0.0.0
    • Prevents public internet exposure while maintaining VPC accessibility
    • Configure via .ilha/config.yml:
      vpc:
        bind_to_private_ip: true  # Default: true
  2. Firewall Rules (Optional: Opt-in)

    • Automatically configures UFW firewall rules to restrict access to VPC network (10.0.0.0/8)
    • Only allows connections from within the VPC
    • Configure via .ilha/config.yml:
      vpc:
        auto_configure_firewall: true  # Default: false (opt-in)
  3. Verification After deployment, verify security:

    # On central server, check port bindings
    docker ps --format "{{.Names}}\t{{.Ports}}" | grep -E "(redis|db)"
    # Should show: 10.x.x.x:6379->6379/tcp (not 0.0.0.0:6379)
    
    # Verify public access is blocked
    telnet <public-ip> 6379  # Should fail
    
    # Verify VPC access works
    telnet <private-ip> 6379  # Should succeed from worker droplet

Notes

  • When using --domain, ilha automatically enables HTTPS via Caddy's Let's Encrypt integration
  • When using --ip, deployments are HTTP-only. Certificate authorities do not issue certificates for IP addresses; use a domain for HTTPS.
  • DNS records are automatically created when --domain is provided (use --skip-dns-check to skip DNS management)
  • You can add defaults in .ilha/config.yml (optional):
deployment:
  default_server: user@server:/var/ilha/packages
  default_domain: myapp.example.com
  default_ip: 203.0.113.10
  ssh_key: ~/.ssh/deploy_key
vpc:
  auto_bind_ports: true  # Enable automatic port binding for VPC-accessible services
  bind_to_private_ip: true  # Bind to private IP only (default: true, recommended for security)
  auto_configure_firewall: false  # Auto-configure UFW rules (default: false, opt-in)

For detailed architecture information, see documentation/ARCHITECTURE.md

Additional Push Details:

  • Auto-import uses a robust remote script with strict mode and consistent quoting to avoid empty variables and broken chains.
  • The remote script resolves the ilha binary (prefers /opt/ilha-venv/bin/ilha, falls back to ilha).
  • Import runs with --non-interactive; older ilha versions safely ignore unknown flags.
  • If an existing ilha project is detected on the server, a normal import is used; otherwise standalone import is performed.

About

No description, website, or topics provided.

Resources

Stars

2 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages