Skip to content

Deployment Guide

Comprehensive guide for deploying the Freeze Design webshop to staging and production on Hetzner Cloud VPSes using Docker Compose, GitHub Actions CI/CD, and zero-downtime rolling deployments.

Production is live

Production runs on a Hetzner CX43 (16 GB / 8 vCPU) from tagged releases on main, deployed with scripts/deploy-production.sh (first deploy v1.26 on 2026-09-04, go-live on 2026-09-30, v1.60 on 2026-11-06). The shop answers on https://www.freezedesign.nl; freezedesign.nl and prod.freezedesign.nl redirect to www with a 301 (prod. still answers /api/ and /health for the post-deploy smoke and old Mollie webhook URLs). There is no basic auth on production any more. Staging is https://staging.freezedesign.nl, behind basic auth.

1. Overview

Architecture

The application runs as seven Docker containers orchestrated by Docker Compose:

Container Image Purpose
nginx nginx:1.31.6-alpine Reverse proxy, SSL termination, static file serving
backend GHCR (Django/Gunicorn) REST API, admin panel
frontend GHCR (Next.js) Server-side rendered storefront
db postgres:15-alpine PostgreSQL 15 database with SSL
redis redis:7-alpine Celery broker and cache (NOT RabbitMQ)
celery Same as backend Async task processing (payments, emails)
celery-beat Same as backend Scheduled task scheduler

External services:

  • DigitalOcean Spaces (ams3) -- media storage (freeze-production-media, freeze-staging-media, served directly from the bucket), encrypted database backups (BACKUP_S3_BUCKET) and the media copy made by scripts/backup-media.sh; separate keys per purpose
  • Openprovider -- registrar and DNS for freezedesign.nl. Cloudflare is not in front of the shop; the design for a later move is in Cloudflare CDN
  • Let's Encrypt -- SSL certificates via certbot (scripts/renew-ssl.sh)
  • Mollie -- payment processing (live key on production, test key on staging)
  • Sentry -- error tracking (projects per environment; PostHog for analytics via /ingest)
  • Discord -- deployment, backup, payment and error notifications
  • GHCR -- ghcr.io/voorman/webshop_freeze_design/{backend,frontend} images, public packages

Environment Comparison

Aspect Staging Production
Domain staging.freezedesign.nl www.freezedesign.nl — single domain, API under /api (no separate api. subdomain); freezedesign.nl and prod.freezedesign.nl redirect to www
Public access Basic auth (nginx .htpasswd, /api/ and /_allauth/ exempt) Public
VPS size Hetzner Cloud VPS, 8 GB RAM / 4 vCPU Hetzner CX43, 16 GB RAM / 8 vCPU (container mem_limits sum to ≈3.75 GB)
Compose file docker-compose.staging.yml docker-compose.prod.yml
Env file backend/.env.staging backend/.env.production
Image tag staging (IMAGE_TAG default) latest (IMAGE_TAG default), plus X.Y per release
CI/CD trigger Auto: deploy job in ci.yml on push to staging (after tests pass) Manual: scripts/deploy-production.sh tags main and dispatches deploy-production.yml
Celery worker --concurrency=1 --max-memory-per-child=262144 --concurrency=2 --max-memory-per-child=262144, mem_limit 768m (v1.54: a child above 256 MiB is replaced after its task)
Unknown host names nginx default_server answers 444 and refuses the TLS handshake (ssl_reject_handshake, v1.52), so scanners on the bare IP never reach Django No catch-all block in nginx/nginx.conf; an unknown name lands in the first server block
Celery-beat HeartbeatScheduler + healthcheck; cron restart via scripts/restart-unhealthy.sh Same
PostgreSQL tuning shared_buffers=512MB, effective_cache_size=1536MB shared_buffers=1GB, effective_cache_size=3GB
Mollie API Test key (test_...) Live key (live_...); deploy-rolling.sh refuses a test_ key via check --deploy
Sentry project (frontend) freeze-frontend-staging freeze-design-frontend
CSP (nginx, Next.js pages) Report-only Report-only (Content-Security-Policy-Report-Only in nginx/nginx.conf)
Search indexing X-Robots-Tag: noindex, nofollow Allowed
BACKUP_RETENTION_DAYS 7 30

Deployment Flow

flowchart TD
    A[PR squash-merged into staging] --> B["ci.yml: backend, frontend, build, E2E smoke + visual<br/>(skipped when the tree was already green in the PR)"]
    A --> C["build-images.yml: backend + frontend images<br/>pushed to GHCR as sha-&lt;sha&gt; (parallel to the tests)"]
    B -->|all green| D[deploy-staging.yml: retag sha-images as staging]
    C --> D
    D --> E["SCP scripts/*.sh, compose file and nginx config to /opt/webshop; deploy-rolling.sh pulls images"]
    E --> F["nginx -t on the new config, then migrations (pre-swap, throwaway container)"]
    F --> G[Rolling replace of backend and frontend with Docker health checks]
    G --> H[celery + celery-beat updated, nginx recreated]
    H --> I["Docker health check<br/>(backend + frontend containers)"]
    I --> J["Post-deploy functional smoke:<br/>image revision == pushed SHA,<br/>then /api/health/, /api/settings/, /api/products/ == 200"]
    J --> K["staging-perf-gate.yml started (k6 homepage, reports only)"]
    J --> L["e2e-staging.yml started (Playwright staging-smoke.spec.ts, Discord on failure)"]
    I -->|health check fails| R[Automatic rollback via scripts/rollback.sh]

Post-deploy verification order

After the containers report healthy, deploy-staging.yml runs a functional smoke on the live VPS: it reads the running backend image's OCI revision label and compares it with the pushed SHA (catches a silently-stale staging without publishing deployment metadata), then confirms /api/health/, /api/settings/, and /api/products/ all return HTTP 200. Only then does it start the k6 homepage performance gate (staging-perf-gate.yml) and, since v1.52 (#861), the E2E smoke (e2e-staging.yml with e2e/staging-smoke.spec.ts: homepage, product list, product page and cart behind the basic auth). Both run on their own, never roll back, and only report to Discord on failure.

Deployment Targets

Target Configuration Status
Local Docker docker-compose.yml / docker-compose.dev.yml Development only
Staging VPS docker-compose.staging.yml, .github/workflows/deploy-staging.yml Active: a green staging CI run calls the reusable deploy workflow
Production VPS docker-compose.prod.yml, .github/workflows/deploy-production.yml Active: tagged releases via scripts/deploy-production.sh, live on www.freezedesign.nl
Documentation .github/workflows/docs.yml Builds MkDocs, OpenAPI, Storybook and TypeDoc and deploys the combined site to Cloudflare Pages

2. Server Setup

Initial VPS Configuration

Provision a Hetzner Cloud VPS (production runs on a CX43 with 16 GB / 8 vCPU, staging on 8 GB / 4 vCPU) with Ubuntu 24.04 LTS in the Nuremberg (eu-central) or Falkenstein region.

Use scripts/provision-server.sh

The steps below are what scripts/provision-server.sh does on a fresh Ubuntu 24.04 server, idempotently: deploy user with your SSH key, ufw (22/80/443), fail2ban, unattended-upgrades, Docker Engine with DNS fix, a 4 GB swap file, /opt/webshop, and the cron entry that restarts an unhealthy celery-beat (scripts/restart-unhealthy.sh). Run it twice: once to set up, once with --harden after you have proven the key login works:

scp scripts/provision-server.sh root@<IP>:/tmp/
ssh root@<IP> "bash /tmp/provision-server.sh \"$(cat ~/.ssh/freeze_prod.pub)\""
ssh freeze-prod "whoami"   # must work before hardening
ssh root@<IP> "bash /tmp/provision-server.sh --harden \"$(cat ~/.ssh/freeze_prod.pub)\""

For staging pass COMPOSE_FILE=docker-compose.staging.yml so the cron line uses the right compose file.

# Connect as root
ssh root@<SERVER_IP>

# Update system
apt update && apt upgrade -y

# Create deploy user
adduser deploy
usermod -aG sudo deploy
usermod -aG adm deploy   # read system logs without sudo (journalctl -k, /var/log/syslog)

# Set up SSH key authentication
mkdir -p /home/deploy/.ssh
cp ~/.ssh/authorized_keys /home/deploy/.ssh/
chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys

# Disable password authentication
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Firewall

ufw allow OpenSSH
ufw allow 80/tcp
ufw allow 443/tcp
ufw enable

Install Docker

# Install Docker Engine
curl -fsSL https://get.docker.com | sh
usermod -aG docker deploy

# Configure DNS for Docker builds (required on some VPS providers)
echo '{"dns": ["8.8.8.8", "8.8.4.4"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

# Verify
docker --version
docker compose version

Install Certbot

sudo apt install -y certbot

Create Application Directory

CI/CD deploys to /opt/webshop, not the git clone directory:

sudo mkdir -p /opt/webshop
sudo chown deploy:deploy /opt/webshop

3. Staging Deployment

Prerequisites

  • Server setup completed (section 2)
  • DNS A record for staging.freezedesign.nl pointing to the server IP (DNS is at Openprovider)
  • GHCR packages set to public visibility (or configure auth on VPS)

Step 1: Clone the Repository

su - deploy
git clone git@github.com:Voorman/webshop_freeze_design.git /opt/webshop
cd /opt/webshop

Note: Private repos require SSH deploy keys. Use the git@github.com: URL, not HTTPS. Add a deploy key in GitHub repo settings.

Step 2: Configure Environment

cp backend/.env.staging.example backend/.env.staging
nano backend/.env.staging

Fill in all secrets. Critical variables:

Variable How to Generate / Where to Find
SECRET_KEY openssl rand -base64 32 \| tr -dc 'a-zA-Z0-9'
REDIS_PASSWORD openssl rand -base64 32 \| tr -dc 'a-zA-Z0-9' \| head -c 32
POSTGRES_PASSWORD Strong random password
DB_PASSWORD Same value as POSTGRES_PASSWORD
MOLLIE_API_KEY Mollie dashboard (use test_... key for staging)
DO_SPACES_ACCESS_KEY DigitalOcean dashboard > API > Spaces Keys
DO_SPACES_SECRET_KEY DigitalOcean dashboard > API > Spaces Keys
SENTRY_DSN Sentry project settings

Important notes on secrets:

  • SECRET_KEY must be alphanumeric only -- avoid special characters that require shell escaping.
  • REDIS_PASSWORD must be alphanumeric only -- it is embedded in URLs (CELERY_BROKER_URL, REDIS_URL). Characters like / break URL parsing and cause "Port could not be cast to integer" errors.
  • BACKEND_URL must NOT include /api -- the payment service builds webhook URLs as {BACKEND_URL}/api/payments/webhook/. Setting it to https://staging.freezedesign.nl/api would produce a double /api/api/ path.
  • ALLOWED_HOSTS should include localhost for Docker health checks to work.

Lock down the file:

chmod 600 backend/.env.staging

Step 3: Create the .env File for Docker Compose

The command: directives in docker-compose.staging.yml use ${REDIS_PASSWORD} which is interpolated at parse time from the .env file at the compose level, not from env_file:. Create a .env file next to the compose file:

cat > /opt/webshop/.env << 'EOF'
REDIS_PASSWORD=<same-password-as-in-backend/.env.staging>
IMAGE_TAG=staging
EOF

chmod 600 /opt/webshop/.env

Step 4: Generate PostgreSQL SSL Certificates

cd /opt/webshop
mkdir -p certs

# Generate self-signed certs for PostgreSQL SSL
openssl req -new -x509 -days 3650 -nodes \
  -out certs/server.crt \
  -keyout certs/server.key \
  -subj "/CN=db"

# PostgreSQL Alpine runs as UID 70
sudo chown 70:70 certs/server.key certs/server.crt
chmod 600 certs/server.key

Step 5: Set Up Basic Auth for Staging

Staging is protected with HTTP basic authentication:

sudo apt install -y apache2-utils
mkdir -p nginx
htpasswd -c nginx/.htpasswd <USERNAME>

Create a separate account for every staging tester; do not distribute one shared credential or implement a cookie bypass. After the initial file has been created, add users without -c so existing accounts are preserved:

htpasswd nginx/.htpasswd <TESTER_USERNAME>

To rotate the basic-auth login later (new name and password, plus the STAGING_BASIC_AUTH_BASE64 secret), follow the Staging-inlog wisselen runbook.

Step 6: Nginx Configuration

The staging nginx config (nginx/nginx.staging.conf) ships in the repository and is mounted into the nginx container by docker-compose.staging.yml — no manual copy is needed. Verify it is present:

ls -la /opt/webshop/nginx/nginx.staging.conf

Step 7: Generate SSL Certificates

Verify DNS is properly configured before running certbot:

# Verify DNS from multiple sources
dig @8.8.8.8 staging.freezedesign.nl A +short
dig @8.8.8.8 staging.freezedesign.nl AAAA +short

# If AAAA record exists, ensure it points to the correct VPS IPv6
ip -6 addr show | grep "inet6.*global"

Warning: Let's Encrypt prefers IPv6. If the domain has an AAAA record pointing to the wrong server, SSL validation will fail. Delete the AAAA record or update it before proceeding.

Generate the certificate:

chmod +x nginx/certbot-init.sh

# Test with Let's Encrypt staging first
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml --staging

# If staging works, generate real certificates
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml

Step 8: Create the Database

The POSTGRES_DB env var tells Django which database to connect to, but does not auto-create it. After starting the db container:

# Start only the database service first
docker compose -f docker-compose.staging.yml up -d db

# Wait for it to be healthy
docker compose -f docker-compose.staging.yml ps db

# Create the database (connect to 'postgres' default database first)
docker compose -f docker-compose.staging.yml exec db \
  psql -U webshop_staging -d postgres -c "CREATE DATABASE webshop_staging;"

Step 9: Start All Services

docker compose -f docker-compose.staging.yml up -d

Verify all services are running and healthy:

docker compose -f docker-compose.staging.yml ps

All services should show "Up (healthy)" status.

Step 10: Run Database Migrations and Create Staff User

docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py create_staff_user

Step 11: Verify Health

# Backend health (requires basic auth)
curl -u <USERNAME>:<PASSWORD> -f https://staging.freezedesign.nl/api/health/

# Nginx health (bypasses basic auth)
curl -f https://staging.freezedesign.nl/health

# Frontend loads
curl -u <USERNAME>:<PASSWORD> -sI https://staging.freezedesign.nl/ | head -5

# SSL certificate
openssl s_client -connect staging.freezedesign.nl:443 \
  -servername staging.freezedesign.nl < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates

4. Production Deployment

Initial setup — already done for the current production VPS

These steps describe how the production VPS (Hetzner CX43) was set up; it has run since 2026-09-04 and has served www.freezedesign.nl since the DNS cutover on 2026-09-30. Routine releases don't use this section; they use scripts/deploy-production.sh (section 7). Use this section only to rebuild the server (see also the Disaster Recovery runbook).

Prerequisites

  • Server setup completed (section 2)
  • DNS A records for freezedesign.nl, www.freezedesign.nl and prod.freezedesign.nl pointing to the server — no api. subdomain, the API is served under /api on the same domain. DNS is at the registrar (Openprovider); Cloudflare is not in use
  • GHCR packages set to public visibility

Step 1: Configure Environment

cd /opt/webshop
cp backend/.env.production.example backend/.env.production
nano backend/.env.production

Production-specific differences from staging (values from backend/.env.production.example):

Variable Production Value
ALLOWED_HOSTS freezedesign.nl,www.freezedesign.nl,prod.freezedesign.nl,localhost (keep prod.freezedesign.nl for the post-deploy smoke and old webhook URLs, and localhost because Docker health checks require it)
FRONTEND_URL https://www.freezedesign.nl — www is the host that serves the API, and the CSRF cookie is host-only, so login, account and admin only work there (nginx/nginx.conf, #373)
BACKEND_URL / API_URL Same value as FRONTEND_URL — no separate API host. Webhooks are built as {BACKEND_URL}/api/payments/webhook/, so this must not include an /api suffix.
MOLLIE_API_KEY Live key (live_...)
SENTRY_ENVIRONMENT production
SENTRY_TRACES_SAMPLE_RATE 0.1
CSP_REPORT_ONLY False
BACKUP_RETENTION_DAYS 30
chmod 600 backend/.env.production

Create the compose-level .env:

cat > /opt/webshop/.env << 'EOF'
REDIS_PASSWORD=<your-alphanumeric-redis-password>
IMAGE_TAG=latest
EOF

chmod 600 /opt/webshop/.env

NEXT_PUBLIC_API_URL is not needed on the server: the frontend image bakes it in at build time from the PRODUCTION_API_URL secret (https://www.freezedesign.nl/api).

Step 2: PostgreSQL SSL Certificates

Same process as staging (see section 3, step 4).

Step 3: Generate SSL Certificates

The production certificate is one lineage, freezedesign.nl, covering freezedesign.nl, www.freezedesign.nl and prod.freezedesign.nl (nginx/nginx.conf reads /etc/letsencrypt/live/freezedesign.nl/). It was first issued for prod. only and expanded to the apex and www on 2026-09-28 via a manual DNS-01 challenge; renewals use HTTP-01 through the nginx webroot (scripts/renew-ssl.sh forces --preferred-challenges http-01).

# Verify DNS
dig @8.8.8.8 www.freezedesign.nl A +short
curl -4 ifconfig.me  # confirm VPS IP matches

# Fresh issue (only when rebuilding the server; the volume name follows the
# compose project, i.e. the directory name /opt/webshop):
docker run --rm -p 80:80 \
  -v webshop_certbot_certs:/etc/letsencrypt \
  certbot/certbot certonly --standalone \
  --cert-name freezedesign.nl \
  -d freezedesign.nl -d www.freezedesign.nl -d prod.freezedesign.nl \
  --agree-tos --no-eff-email --email info@freezedesign.nl

Step 4: Create Database and Start Services

# Start database
docker compose -f docker-compose.prod.yml up -d db
docker compose -f docker-compose.prod.yml exec db \
  psql -U webshop_prod -d postgres -c "CREATE DATABASE webshop_prod;"

# Start all services
docker compose -f docker-compose.prod.yml up -d

# Run migrations
docker compose -f docker-compose.prod.yml exec backend python manage.py migrate

# Create staff user
docker compose -f docker-compose.prod.yml exec backend python manage.py create_staff_user

Step 5: Verify Health

curl -f https://www.freezedesign.nl/api/health/
curl -f https://www.freezedesign.nl/health
curl -sI https://www.freezedesign.nl/ | head -5
# Apex and prod. redirect to www
curl -sI https://freezedesign.nl/ | grep -i location
curl -sI https://prod.freezedesign.nl/ | grep -i location
openssl s_client -connect www.freezedesign.nl:443 \
  -servername www.freezedesign.nl < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates

Step 6: Cron entries

scripts/provision-server.sh installs /etc/cron.d/webshop-restart-unhealthy (every 5 minutes, restarts an unhealthy celery-beat and reports in Discord). Both servers also have a root crontab that runs scripts/renew-ssl.sh on Sundays 03:30 UTC as a safety net next to ssl-renewal.yml (see section 6). The media copy (scripts/backup-media.sh) is a separate cron job that Ivar sets up by hand (see scripts/backup/README.md).

5. CI/CD Pipeline

Staging: Automatic on Push to staging

The staging deploy is the final deploy job inside .github/workflows/ci.yml. On every push to the staging branch (typically a squash-merged feature PR), CI runs all tests first; only when backend-test, frontend-test, frontend-build, e2e-smoke-test, and e2e-visual-test are green does the deploy job call .github/workflows/deploy-staging.yml via workflow_call.

deploy-staging.yml itself only has workflow_call and workflow_dispatch triggers -- the old standalone workflow_run trigger no longer exists. Use gh workflow run deploy-staging.yml for a manual (re)deploy of the current staging tip.

What it does:

  1. Build phase -- On a staging push ci.yml calls build-images.yml in parallel with the tests: it builds the backend image, the staging frontend and the production frontend for that commit, with OCI revision metadata and GIT_SHA, and pushes them to GHCR (namespace ghcr.io/voorman/..., lowercase) under sha-<commit> tags only. Builds use the GitHub Actions cache with separate scopes (backend, frontend, frontend-production) so the images don't evict each other's layers. A manual deploy-staging.yml run builds the staging set itself
  2. Superseded check -- Reads the revision label of the backend container on staging; if staging already runs a newer commit, the deploy is skipped (superseded=true) instead of rolling staging back
  3. Retag -- Only now, with green tests, the sha- images get the staging tag that deploy-rolling.sh pulls
  4. Deploy phase -- Copies scripts/*.sh, docker-compose.staging.yml and nginx/nginx.staging.conf to /opt/webshop via SCP, then executes scripts/deploy-rolling.sh via SSH with DEPLOY_TAG=staging (with immediate scripts/rollback.sh if it fails)
  5. Health verification -- Checks Docker health status of backend and frontend containers. If either fails, triggers automatic rollback via scripts/rollback.sh
  6. Post-deploy functional smoke -- Over SSH on the VPS: reads the running backend image's OCI revision label and compares it with the pushed SHA (catches a silently-stale staging without exposing deployment metadata), then verifies /api/health/, /api/settings/, and /api/products/ all return HTTP 200
  7. k6 performance gate and E2E smoke -- Starts staging-perf-gate.yml (loadtests/homepage-load.js against STAGING_URL) and e2e-staging.yml (e2e/staging-smoke.spec.ts with the deployed sha); both run on their own, never block or roll back, and report a failure in Discord
  8. Notifications -- Sends Discord notifications on started/success/failure (and the same to ntfy topic freeze-deploy, see ntfy notifications); ci.yml's notify-staging job reports the overall result

Production: Tagged Releases via scripts/deploy-production.sh

.github/workflows/deploy-production.yml has only a workflow_dispatch trigger with a tag input. scripts/deploy-production.sh creates that tag on origin/main and dispatches the workflow, which:

  1. Looks for the sha- images of the tag commit (scripts/find-prebuilt-images.sh; for a docs-only tag commit it walks back to the last commit with code) and retags them as latest and X.Y; only when none exist does it build from the tag (with GIT_SHA and the revision label of the tag commit). It refuses to build a frontend whose PRODUCTION_NEXT_PUBLIC_SENTRY_DSN equals the staging DSN
  2. Determines the previous version tag for a rollback (PREVIOUS_VERSION)
  3. Copies scripts/*.sh, docker-compose.prod.yml and nginx/nginx.conf to /opt/webshop and runs scripts/deploy-rolling.sh with DEPLOY_TAG=latest and PRE_DEPLOY_BACKUP=true (check --deploy --tag security, safe_migrate --dry-run with a backup, then migrations)
  4. Verifies health, runs the same revision-label + endpoint smoke as staging against PRODUCTION_URL, and rolls back automatically on failure
  5. Notifies Discord via DISCORD_WEBHOOK_PRODUCTION (fallback DISCORD_WEBHOOK_CICD, then DISCORD_WEBHOOK_URL)

The GitHub environment production has no required reviewers; the confirmation prompt in the script is the gate. See the deployment runbook for the operator steps.

Required GitHub Secrets

Secret Staging Production
STAGING_HOST / PRODUCTION_HOST VPS IP VPS IP
STAGING_USER / PRODUCTION_USER deploy deploy
STAGING_SSH_KEY / PRODUCTION_SSH_KEY SSH private key SSH private key
STAGING_API_URL / PRODUCTION_API_URL https://staging.freezedesign.nl/api https://www.freezedesign.nl/api (baked into the frontend image)
STAGING_URL / PRODUCTION_URL https://staging.freezedesign.nl https://www.freezedesign.nl (smoke target and Discord link)
STAGING_BASIC_AUTH_BASE64 Base64 basic-auth credentials for the k6 gate, load-tests.yml and e2e-staging.yml n/a
NEXT_PUBLIC_SENTRY_DSN / PRODUCTION_NEXT_PUBLIC_SENTRY_DSN Frontend Sentry DSN (freeze-frontend-staging) Frontend Sentry DSN (freeze-design-frontend); must differ from staging
SENTRY_AUTH_TOKEN, NEXT_PUBLIC_POSTHOG_KEY Source-map upload and PostHog key for the frontend build Same
POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID Source-map upload to PostHog Error tracking (EU) in the frontend build; without them the build skips it Same
DISCORD_WEBHOOK_URL, DISCORD_WEBHOOK_CICD Deploy notifications DISCORD_WEBHOOK_PRODUCTION (fallback to the staging ones)

Compose File Requirements for CI/CD

The compose files used by CI/CD must use image: directives pointing to GHCR, not build:. Using build: causes "No image to be pulled" errors during docker compose pull.

# Correct (staging compose)
backend:
  image: ghcr.io/voorman/webshop_freeze_design/backend:${IMAGE_TAG:-staging}

# Wrong (will fail with CI/CD)
backend:
  build:
    context: ./backend

Use the IMAGE_TAG environment variable for versioning: set it in the .env file at the compose level or export it before running compose commands.

GHCR Package Visibility

GHCR packages must be set to public so the VPS can pull images without authentication. Change visibility in GitHub: Package settings > Change visibility > Public.

6. SSL Certificate Management

Automatic Renewal

Let's Encrypt certificates are valid for 90 days. Renewal is automated by .github/workflows/ssl-renewal.yml (weekly, Wednesday 04:00 UTC, via SSH), which runs a matrix over staging and production; both legs call the same idempotent scripts/renew-ssl.sh (certbot in a container, webroot challenge, nginx reload, check that port 443 serves the new certificate). Both VPSes also have a root crontab (Sundays 03:30 UTC) that runs the same script as a safety net (set up 2026-10-04, see .planning/todos/pending/production-vps-ssl-renewal-setup.md). The workflow's expiry-check job then checks www.freezedesign.nl, freezedesign.nl, prod.freezedesign.nl and staging.freezedesign.nl from outside and alerts in Discord below 14 days. Renewal only renews: adding names to a certificate is a separate certbot --expand step.

DOMAIN is the certbot lineage (directory under /etc/letsencrypt/live/), not the hostname: freezedesign.nl on production, staging.freezedesign.eu on staging (the staging certificate was created under the old name and expanded; certbot never renames a lineage).

The crontab line on each server (root):

# Production
30 3 * * 0 cd /opt/webshop && DOMAIN=freezedesign.nl COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh
# Staging
30 3 * * 0 cd /opt/webshop && DOMAIN=staging.freezedesign.eu COMPOSE_FILE=/opt/webshop/docker-compose.staging.yml bash scripts/renew-ssl.sh

Manual Renewal

# Check certificate expiry
openssl s_client -connect www.freezedesign.nl:443 \
  -servername www.freezedesign.nl < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates

# Renew now (production); the script reloads nginx itself
ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
    COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh'

# Force renewal when the normal renewal is blocked
#   via GitHub: gh workflow run ssl-renewal.yml -f force=true
#   on the server: add CERTBOT_EXTRA_FLAGS="--force-renewal" to the command above

Troubleshooting SSL

If Let's Encrypt validation fails:

  1. Verify DNS resolves correctly from external sources:
    dig @8.8.8.8 freezedesign.nl A +short
    dig @8.8.8.8 freezedesign.nl AAAA +short
    
  2. If an AAAA record exists, ensure it points to the correct VPS IPv6 address or delete it. Let's Encrypt prefers IPv6 and will fail if the AAAA record points elsewhere.
  3. Ensure port 80 is open (ufw allow 80/tcp).
  4. Ensure the certbot_webroot volume is not mounted as read-only (:ro) in the nginx service. Certbot needs to write challenge files.

7. Updating / Deploying New Versions

Staging: Merge a PR into the staging branch (feature PRs always target staging, squash-merged). CI runs the full test suite and, when green, the deploy job deploys to staging automatically.

gh pr merge <pr-number> --squash
# CI runs on the staging push; the deploy job handles the rest

# Manual (re)deploy of the current staging tip:
gh workflow run deploy-staging.yml

Promotion to main: Run scripts/release.sh --promote-only. It confirms the staging HEAD was deployed (from the CI push run on staging), opens or reuses the promotion PR, waits for its checks, and fast-forwards main to the PR's head SHA; GitHub then marks the PR as merged. main and staging keep the same SHAs. Promoting to main does not deploy anything.

Production: Run scripts/deploy-production.sh. It tags origin/main with the next vX.Y (or --tag), shows the commits and new migrations, asks for confirmation, and dispatches and follows deploy-production.yml. See the deployment runbook.

Manual Deployment

If CI/CD is unavailable, deploy manually from the VPS:

cd /opt/webshop

# Pull latest images
export IMAGE_TAG=staging  # or a specific tag like v1.20
docker compose -f docker-compose.staging.yml pull

# Run rolling deployment
export COMPOSE_FILE=docker-compose.staging.yml
export DEPLOYMENT_DIR=/opt/webshop
# Verplicht: bepaalt welke image-tag als rollback-referentie geldt.
# staging -> staging, productie -> latest.
export DEPLOY_TAG=staging
./scripts/deploy-rolling.sh

# Or for a simpler (non-rolling) update:
docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py collectstatic --noinput

After Changing Environment Variables

When you change values in .env.staging or .env.production, restarting containers is not enough. Containers cache environment variables at creation time. You must recreate them:

# Stop, remove, and recreate the affected service
docker compose -f docker-compose.staging.yml stop backend
docker compose -f docker-compose.staging.yml rm -f backend
docker compose -f docker-compose.staging.yml up -d backend

8. Rollback Procedures

Automatic Rollback

The CI/CD pipeline automatically rolls back when health checks fail after deployment. The flow:

  1. deploy-rolling.sh runs -- if it fails, rollback.sh is called immediately
  2. Post-deploy health verification checks Docker health status of backend and frontend
  3. If either service is unhealthy after multiple retries, rollback.sh restores the local predeploy snapshot, or else pulls the previous version tag (PREVIOUS_VERSION, e.g. 1.38) from GHCR, and restarts services
  4. A final fallback step runs if the health check step itself fails

Manual Rollback

To manually roll back to a previous version:

cd /opt/webshop

# Option 1: Roll back to a specific image tag
export IMAGE_TAG=1.38  # the known-good version (registry tag without 'v'; exists from 1.39 on)
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d

# Option 2: Use the rollback script
export COMPOSE_FILE=docker-compose.prod.yml
export DEPLOYMENT_DIR=/opt/webshop
export GITHUB_REPOSITORY=Voorman/webshop_freeze_design
export REGISTRY=ghcr.io
export ROLLBACK_VERSION=1.38  # optioneel: deze versie i.p.v. de lokale predeploy-snapshot
export DEPLOY_TAG=latest    # verplicht; productie=latest, staging=staging
./scripts/rollback.sh

# Verify services are healthy
docker compose -f docker-compose.prod.yml ps

9. Database Migration Rollback

Django migrations can be reversed if needed. This is a manual process.

Identify the Migration to Roll Back

# List applied migrations for an app
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py showmigrations <app_name>

Roll Back a Migration

# Roll back to a specific migration (the one BEFORE the problematic one)
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py migrate <app_name> <migration_number>

# Example: roll back products app to migration 0005
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py migrate products 0005

Full Database Restore

If migration rollback is insufficient, restore from a backup. The hourly backups are AES-256-GCM encrypted in DigitalOcean Spaces; restore_database downloads, decrypts (BACKUP_ENCRYPTION_KEY) and restores them:

# Stop services that use the database
docker compose -f docker-compose.prod.yml stop celery celery-beat

# List and restore (newest, or a specific filename instead of --latest)
docker compose -f docker-compose.prod.yml exec backend python manage.py restore_database --list
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py restore_database --latest --force

# Restart services
docker compose -f docker-compose.prod.yml up -d

See the Disaster Recovery runbook for the full procedure.

Manual Database Backup

# Create a backup before risky operations
docker compose -f docker-compose.prod.yml exec db \
  pg_dump -U webshop_prod webshop_prod > backup_$(date +%Y%m%d_%H%M%S).sql
gzip backup_*.sql

10. Troubleshooting

Service Health Checks

# Check all service statuses
docker compose -f docker-compose.prod.yml ps

# Check logs for a specific service
docker compose -f docker-compose.prod.yml logs backend --tail=50
docker compose -f docker-compose.prod.yml logs frontend --tail=50
docker compose -f docker-compose.prod.yml logs nginx --tail=50
docker compose -f docker-compose.prod.yml logs celery --tail=50

Health check details by container:

Container Health check tool Command
backend curl (Python image has curl) curl -f http://localhost:8000/api/health/
frontend wget (Node Alpine has wget, NOT curl) wget -q --spider http://127.0.0.1:3000/
nginx curl (Alpine has curl) curl -f http://127.0.0.1/health
db pg_isready pg_isready -U $POSTGRES_USER on production; staging and dev add -d $POSTGRES_DB (without it pg_isready uses the user name as database and logs a FATAL every 5 seconds, v1.54)
redis redis-cli redis-cli -a $REDIS_PASSWORD ping
celery celery inspect celery -A config inspect ping

Nginx 502 Bad Gateway After Deployment

Nginx caches upstream DNS at startup. When containers are recreated during deployment, they get new Docker network IPs but nginx still routes to the old IPs.

Fix: Restart nginx after deploying services:

docker compose -f docker-compose.prod.yml restart nginx

The rolling deploy script (deploy-rolling.sh) already does this as its final step.

Container Environment Variable Changes Not Taking Effect

Containers cache env vars at creation time. Restarting is not enough:

# Wrong: restart only re-reads env_file on some Docker versions
docker compose -f docker-compose.prod.yml restart backend

# Correct: stop, remove, and recreate
docker compose -f docker-compose.prod.yml stop backend
docker compose -f docker-compose.prod.yml rm -f backend
docker compose -f docker-compose.prod.yml up -d backend

Redis "Port Could Not Be Cast to Integer" Error

This means REDIS_PASSWORD contains special characters (like /) that break URL parsing in CELERY_BROKER_URL and REDIS_URL. Regenerate with alphanumeric characters only:

openssl rand -base64 32 | tr -dc 'a-zA-Z0-9' | head -c 32

Update both REDIS_PASSWORD in the backend env file and the compose-level .env file, then recreate all services that use Redis:

docker compose -f docker-compose.staging.yml stop redis backend celery celery-beat
docker compose -f docker-compose.staging.yml rm -f redis backend celery celery-beat
docker compose -f docker-compose.staging.yml up -d

SSL Certificate Errors

# Verify certificate files exist (the certs live in the Docker volume webshop_certbot_certs)
sudo ls -la "$(docker volume inspect webshop_certbot_certs --format '{{.Mountpoint}}')/live/freezedesign.nl/"

# Check what nginx serves
openssl s_client -connect localhost:443 -servername www.freezedesign.nl < /dev/null 2>/dev/null | \
  openssl x509 -noout -dates

# Re-generate certificates (staging example; see section 6 for renewals)
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml

Database Connection Issues

# Check PostgreSQL logs
docker compose -f docker-compose.prod.yml logs db --tail=50

# Verify database accepts connections
docker compose -f docker-compose.prod.yml exec db pg_isready -U webshop_prod

# Check connection count
docker compose -f docker-compose.prod.yml exec db \
  psql -U webshop_prod -c "SELECT count(*) FROM pg_stat_activity;"

Memory Issues

# System memory
free -h

# Docker container memory usage
docker stats --no-stream

# Redis memory usage
docker compose -f docker-compose.prod.yml exec redis \
  redis-cli -a "$REDIS_PASSWORD" --no-auth-warning info memory | grep used_memory_human

Celery Task Issues

# Check active tasks
docker compose -f docker-compose.prod.yml exec celery \
  celery -A config inspect active

# Check Celery Beat schedule
docker compose -f docker-compose.prod.yml logs celery-beat --tail=20

# Check Redis queue length
docker compose -f docker-compose.prod.yml exec redis \
  redis-cli -a "$REDIS_PASSWORD" --no-auth-warning llen celery

Disk Space

# Check disk usage
df -h

# Check Docker disk usage
docker system df

# Clean up unused Docker resources
docker system prune -f

# Remove unused images (use with caution in production)
docker image prune -a -f

Resetting Staging Data

To reset the staging database to a clean state:

docker compose -f docker-compose.staging.yml down
docker volume rm webshop_postgres_data   # compose project "webshop" (from /opt/webshop)
docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py create_staff_user

Debugging Webhooks (Mollie)

If payment webhooks return 404, check that BACKEND_URL does not include /api. The webhook URL is constructed as {BACKEND_URL}/api/payments/webhook/, so a BACKEND_URL of https://freezedesign.nl/api would result in the path /api/api/payments/webhook/.

Shell Escaping in Remote Debugging

These examples assume the server shell setup from the Alerting Runbook (cd /opt/webshop and export COMPOSE_FILE=…); a bare docker compose on the server fails without it.

When debugging via SSH into Docker containers, avoid embedding JSON in nested shell commands. Multi-layer escaping (local -> ssh -> docker exec -> bash -c -> curl -d) silently corrupts request bodies. Instead, write JSON to a file first:

# Wrong: body gets silently corrupted
docker compose exec backend bash -c 'curl -d "{\"key\":\"value\"}" ...'

# Correct: write JSON to file first
docker compose exec backend bash -c 'echo "{\"key\":\"value\"}" > /tmp/body.json && curl -d @/tmp/body.json ...'

Monitoring Checklist

After deployment, verify these items regularly (on the server, after the setup from the Alerting Runbook):

  • All services show "healthy" in docker compose ps
  • SSL certificate expiry > 30 days
  • Disk usage < 80%
  • Memory usage < 80%
  • Database connections < 40 (max 50 configured)
  • Redis memory < 200MB (256MB limit on production)
  • No errors in docker compose logs backend --tail=100
  • Sentry error count at acceptable level
  • Celery-beat healthy (docker compose ps celery-beat); journalctl -t restart-unhealthy empty

Operational procedures are in Monitoring, Alerting and Disaster Recovery.