Deployment Guide¶
Comprehensive guide for deploying the Freeze Design webshop to staging and production on Hetzner Cloud VPSes using Docker Compose, GitHub Actions CI/CD, and zero-downtime rolling deployments.
Production is live
Production runs on a Hetzner CX43 (16 GB / 8 vCPU) from tagged releases on main,
deployed with scripts/deploy-production.sh (first deploy v1.26 on 2026-09-04,
go-live on 2026-09-30, v1.60 on 2026-11-06). The shop answers on
https://www.freezedesign.nl; freezedesign.nl and prod.freezedesign.nl redirect
to www with a 301 (prod. still answers /api/ and /health for the post-deploy
smoke and old Mollie webhook URLs). There is no basic auth on production any more.
Staging is https://staging.freezedesign.nl, behind basic auth.
1. Overview¶
Architecture¶
The application runs as seven Docker containers orchestrated by Docker Compose:
| Container | Image | Purpose |
|---|---|---|
| nginx | nginx:1.31.6-alpine |
Reverse proxy, SSL termination, static file serving |
| backend | GHCR (Django/Gunicorn) | REST API, admin panel |
| frontend | GHCR (Next.js) | Server-side rendered storefront |
| db | postgres:15-alpine |
PostgreSQL 15 database with SSL |
| redis | redis:7-alpine |
Celery broker and cache (NOT RabbitMQ) |
| celery | Same as backend | Async task processing (payments, emails) |
| celery-beat | Same as backend | Scheduled task scheduler |
External services:
- DigitalOcean Spaces (
ams3) -- media storage (freeze-production-media,freeze-staging-media, served directly from the bucket), encrypted database backups (BACKUP_S3_BUCKET) and the media copy made byscripts/backup-media.sh; separate keys per purpose - Openprovider -- registrar and DNS for
freezedesign.nl. Cloudflare is not in front of the shop; the design for a later move is in Cloudflare CDN - Let's Encrypt -- SSL certificates via certbot (
scripts/renew-ssl.sh) - Mollie -- payment processing (live key on production, test key on staging)
- Sentry -- error tracking (projects per environment; PostHog for analytics via
/ingest) - Discord -- deployment, backup, payment and error notifications
- GHCR --
ghcr.io/voorman/webshop_freeze_design/{backend,frontend}images, public packages
Environment Comparison¶
| Aspect | Staging | Production |
|---|---|---|
| Domain | staging.freezedesign.nl |
www.freezedesign.nl — single domain, API under /api (no separate api. subdomain); freezedesign.nl and prod.freezedesign.nl redirect to www |
| Public access | Basic auth (nginx .htpasswd, /api/ and /_allauth/ exempt) |
Public |
| VPS size | Hetzner Cloud VPS, 8 GB RAM / 4 vCPU | Hetzner CX43, 16 GB RAM / 8 vCPU (container mem_limits sum to ≈3.75 GB) |
| Compose file | docker-compose.staging.yml |
docker-compose.prod.yml |
| Env file | backend/.env.staging |
backend/.env.production |
| Image tag | staging (IMAGE_TAG default) |
latest (IMAGE_TAG default), plus X.Y per release |
| CI/CD trigger | Auto: deploy job in ci.yml on push to staging (after tests pass) |
Manual: scripts/deploy-production.sh tags main and dispatches deploy-production.yml |
| Celery worker | --concurrency=1 --max-memory-per-child=262144 |
--concurrency=2 --max-memory-per-child=262144, mem_limit 768m (v1.54: a child above 256 MiB is replaced after its task) |
| Unknown host names | nginx default_server answers 444 and refuses the TLS handshake (ssl_reject_handshake, v1.52), so scanners on the bare IP never reach Django |
No catch-all block in nginx/nginx.conf; an unknown name lands in the first server block |
| Celery-beat | HeartbeatScheduler + healthcheck; cron restart via scripts/restart-unhealthy.sh |
Same |
| PostgreSQL tuning | shared_buffers=512MB, effective_cache_size=1536MB |
shared_buffers=1GB, effective_cache_size=3GB |
| Mollie API | Test key (test_...) |
Live key (live_...); deploy-rolling.sh refuses a test_ key via check --deploy |
| Sentry project (frontend) | freeze-frontend-staging |
freeze-design-frontend |
| CSP (nginx, Next.js pages) | Report-only | Report-only (Content-Security-Policy-Report-Only in nginx/nginx.conf) |
| Search indexing | X-Robots-Tag: noindex, nofollow |
Allowed |
BACKUP_RETENTION_DAYS |
7 | 30 |
Deployment Flow¶
flowchart TD
A[PR squash-merged into staging] --> B["ci.yml: backend, frontend, build, E2E smoke + visual<br/>(skipped when the tree was already green in the PR)"]
A --> C["build-images.yml: backend + frontend images<br/>pushed to GHCR as sha-<sha> (parallel to the tests)"]
B -->|all green| D[deploy-staging.yml: retag sha-images as staging]
C --> D
D --> E["SCP scripts/*.sh, compose file and nginx config to /opt/webshop; deploy-rolling.sh pulls images"]
E --> F["nginx -t on the new config, then migrations (pre-swap, throwaway container)"]
F --> G[Rolling replace of backend and frontend with Docker health checks]
G --> H[celery + celery-beat updated, nginx recreated]
H --> I["Docker health check<br/>(backend + frontend containers)"]
I --> J["Post-deploy functional smoke:<br/>image revision == pushed SHA,<br/>then /api/health/, /api/settings/, /api/products/ == 200"]
J --> K["staging-perf-gate.yml started (k6 homepage, reports only)"]
J --> L["e2e-staging.yml started (Playwright staging-smoke.spec.ts, Discord on failure)"]
I -->|health check fails| R[Automatic rollback via scripts/rollback.sh]
Post-deploy verification order
After the containers report healthy, deploy-staging.yml runs a functional smoke on the live VPS: it reads the running backend image's OCI revision label and compares it with the pushed SHA (catches a silently-stale staging without publishing deployment metadata), then confirms /api/health/, /api/settings/, and /api/products/ all return HTTP 200. Only then does it start the k6 homepage performance gate (staging-perf-gate.yml) and, since v1.52 (#861), the E2E smoke (e2e-staging.yml with e2e/staging-smoke.spec.ts: homepage, product list, product page and cart behind the basic auth). Both run on their own, never roll back, and only report to Discord on failure.
Deployment Targets¶
| Target | Configuration | Status |
|---|---|---|
| Local Docker | docker-compose.yml / docker-compose.dev.yml |
Development only |
| Staging VPS | docker-compose.staging.yml, .github/workflows/deploy-staging.yml |
Active: a green staging CI run calls the reusable deploy workflow |
| Production VPS | docker-compose.prod.yml, .github/workflows/deploy-production.yml |
Active: tagged releases via scripts/deploy-production.sh, live on www.freezedesign.nl |
| Documentation | .github/workflows/docs.yml |
Builds MkDocs, OpenAPI, Storybook and TypeDoc and deploys the combined site to Cloudflare Pages |
2. Server Setup¶
Initial VPS Configuration¶
Provision a Hetzner Cloud VPS (production runs on a CX43 with 16 GB / 8 vCPU, staging on 8 GB / 4 vCPU) with Ubuntu 24.04 LTS in the Nuremberg (eu-central) or Falkenstein region.
Use scripts/provision-server.sh
The steps below are what scripts/provision-server.sh does on a fresh Ubuntu 24.04 server,
idempotently: deploy user with your SSH key, ufw (22/80/443), fail2ban,
unattended-upgrades, Docker Engine with DNS fix, a 4 GB swap file, /opt/webshop, and the
cron entry that restarts an unhealthy celery-beat (scripts/restart-unhealthy.sh). Run it
twice: once to set up, once with --harden after you have proven the key login works:
scp scripts/provision-server.sh root@<IP>:/tmp/
ssh root@<IP> "bash /tmp/provision-server.sh \"$(cat ~/.ssh/freeze_prod.pub)\""
ssh freeze-prod "whoami" # must work before hardening
ssh root@<IP> "bash /tmp/provision-server.sh --harden \"$(cat ~/.ssh/freeze_prod.pub)\""
For staging pass COMPOSE_FILE=docker-compose.staging.yml so the cron line uses the right compose file.
# Connect as root
ssh root@<SERVER_IP>
# Update system
apt update && apt upgrade -y
# Create deploy user
adduser deploy
usermod -aG sudo deploy
usermod -aG adm deploy # read system logs without sudo (journalctl -k, /var/log/syslog)
# Set up SSH key authentication
mkdir -p /home/deploy/.ssh
cp ~/.ssh/authorized_keys /home/deploy/.ssh/
chown -R deploy:deploy /home/deploy/.ssh
chmod 700 /home/deploy/.ssh
chmod 600 /home/deploy/.ssh/authorized_keys
# Disable password authentication
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd
Firewall¶
Install Docker¶
# Install Docker Engine
curl -fsSL https://get.docker.com | sh
usermod -aG docker deploy
# Configure DNS for Docker builds (required on some VPS providers)
echo '{"dns": ["8.8.8.8", "8.8.4.4"]}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
# Verify
docker --version
docker compose version
Install Certbot¶
Create Application Directory¶
CI/CD deploys to /opt/webshop, not the git clone directory:
3. Staging Deployment¶
Prerequisites¶
- Server setup completed (section 2)
- DNS A record for
staging.freezedesign.nlpointing to the server IP (DNS is at Openprovider) - GHCR packages set to public visibility (or configure auth on VPS)
Step 1: Clone the Repository¶
Note: Private repos require SSH deploy keys. Use the
git@github.com:URL, not HTTPS. Add a deploy key in GitHub repo settings.
Step 2: Configure Environment¶
Fill in all secrets. Critical variables:
| Variable | How to Generate / Where to Find |
|---|---|
SECRET_KEY |
openssl rand -base64 32 \| tr -dc 'a-zA-Z0-9' |
REDIS_PASSWORD |
openssl rand -base64 32 \| tr -dc 'a-zA-Z0-9' \| head -c 32 |
POSTGRES_PASSWORD |
Strong random password |
DB_PASSWORD |
Same value as POSTGRES_PASSWORD |
MOLLIE_API_KEY |
Mollie dashboard (use test_... key for staging) |
DO_SPACES_ACCESS_KEY |
DigitalOcean dashboard > API > Spaces Keys |
DO_SPACES_SECRET_KEY |
DigitalOcean dashboard > API > Spaces Keys |
SENTRY_DSN |
Sentry project settings |
Important notes on secrets:
SECRET_KEYmust be alphanumeric only -- avoid special characters that require shell escaping.REDIS_PASSWORDmust be alphanumeric only -- it is embedded in URLs (CELERY_BROKER_URL,REDIS_URL). Characters like/break URL parsing and cause "Port could not be cast to integer" errors.BACKEND_URLmust NOT include/api-- the payment service builds webhook URLs as{BACKEND_URL}/api/payments/webhook/. Setting it tohttps://staging.freezedesign.nl/apiwould produce a double/api/api/path.ALLOWED_HOSTSshould includelocalhostfor Docker health checks to work.
Lock down the file:
Step 3: Create the .env File for Docker Compose¶
The command: directives in docker-compose.staging.yml use ${REDIS_PASSWORD} which is interpolated at parse time from the .env file at the compose level, not from env_file:. Create a .env file next to the compose file:
cat > /opt/webshop/.env << 'EOF'
REDIS_PASSWORD=<same-password-as-in-backend/.env.staging>
IMAGE_TAG=staging
EOF
chmod 600 /opt/webshop/.env
Step 4: Generate PostgreSQL SSL Certificates¶
cd /opt/webshop
mkdir -p certs
# Generate self-signed certs for PostgreSQL SSL
openssl req -new -x509 -days 3650 -nodes \
-out certs/server.crt \
-keyout certs/server.key \
-subj "/CN=db"
# PostgreSQL Alpine runs as UID 70
sudo chown 70:70 certs/server.key certs/server.crt
chmod 600 certs/server.key
Step 5: Set Up Basic Auth for Staging¶
Staging is protected with HTTP basic authentication:
Create a separate account for every staging tester; do not distribute one shared
credential or implement a cookie bypass. After the initial file has been created,
add users without -c so existing accounts are preserved:
To rotate the basic-auth login later (new name and password, plus the
STAGING_BASIC_AUTH_BASE64 secret), follow the
Staging-inlog wisselen runbook.
Step 6: Nginx Configuration¶
The staging nginx config (nginx/nginx.staging.conf) ships in the repository and is mounted into the nginx container by docker-compose.staging.yml — no manual copy is needed. Verify it is present:
Step 7: Generate SSL Certificates¶
Verify DNS is properly configured before running certbot:
# Verify DNS from multiple sources
dig @8.8.8.8 staging.freezedesign.nl A +short
dig @8.8.8.8 staging.freezedesign.nl AAAA +short
# If AAAA record exists, ensure it points to the correct VPS IPv6
ip -6 addr show | grep "inet6.*global"
Warning: Let's Encrypt prefers IPv6. If the domain has an AAAA record pointing to the wrong server, SSL validation will fail. Delete the AAAA record or update it before proceeding.
Generate the certificate:
chmod +x nginx/certbot-init.sh
# Test with Let's Encrypt staging first
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml --staging
# If staging works, generate real certificates
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml
Step 8: Create the Database¶
The POSTGRES_DB env var tells Django which database to connect to, but does not auto-create it. After starting the db container:
# Start only the database service first
docker compose -f docker-compose.staging.yml up -d db
# Wait for it to be healthy
docker compose -f docker-compose.staging.yml ps db
# Create the database (connect to 'postgres' default database first)
docker compose -f docker-compose.staging.yml exec db \
psql -U webshop_staging -d postgres -c "CREATE DATABASE webshop_staging;"
Step 9: Start All Services¶
Verify all services are running and healthy:
All services should show "Up (healthy)" status.
Step 10: Run Database Migrations and Create Staff User¶
docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py create_staff_user
Step 11: Verify Health¶
# Backend health (requires basic auth)
curl -u <USERNAME>:<PASSWORD> -f https://staging.freezedesign.nl/api/health/
# Nginx health (bypasses basic auth)
curl -f https://staging.freezedesign.nl/health
# Frontend loads
curl -u <USERNAME>:<PASSWORD> -sI https://staging.freezedesign.nl/ | head -5
# SSL certificate
openssl s_client -connect staging.freezedesign.nl:443 \
-servername staging.freezedesign.nl < /dev/null 2>/dev/null | \
openssl x509 -noout -dates
4. Production Deployment¶
Initial setup — already done for the current production VPS
These steps describe how the production VPS (Hetzner CX43) was set up; it has run
since 2026-09-04 and has served www.freezedesign.nl since the DNS cutover on
2026-09-30. Routine releases don't use this section; they use
scripts/deploy-production.sh (section 7). Use this section only to rebuild the
server (see also the Disaster Recovery runbook).
Prerequisites¶
- Server setup completed (section 2)
- DNS A records for
freezedesign.nl,www.freezedesign.nlandprod.freezedesign.nlpointing to the server — noapi.subdomain, the API is served under/apion the same domain. DNS is at the registrar (Openprovider); Cloudflare is not in use - GHCR packages set to public visibility
Step 1: Configure Environment¶
cd /opt/webshop
cp backend/.env.production.example backend/.env.production
nano backend/.env.production
Production-specific differences from staging (values from backend/.env.production.example):
| Variable | Production Value |
|---|---|
ALLOWED_HOSTS |
freezedesign.nl,www.freezedesign.nl,prod.freezedesign.nl,localhost (keep prod.freezedesign.nl for the post-deploy smoke and old webhook URLs, and localhost because Docker health checks require it) |
FRONTEND_URL |
https://www.freezedesign.nl — www is the host that serves the API, and the CSRF cookie is host-only, so login, account and admin only work there (nginx/nginx.conf, #373) |
BACKEND_URL / API_URL |
Same value as FRONTEND_URL — no separate API host. Webhooks are built as {BACKEND_URL}/api/payments/webhook/, so this must not include an /api suffix. |
MOLLIE_API_KEY |
Live key (live_...) |
SENTRY_ENVIRONMENT |
production |
SENTRY_TRACES_SAMPLE_RATE |
0.1 |
CSP_REPORT_ONLY |
False |
BACKUP_RETENTION_DAYS |
30 |
Create the compose-level .env:
cat > /opt/webshop/.env << 'EOF'
REDIS_PASSWORD=<your-alphanumeric-redis-password>
IMAGE_TAG=latest
EOF
chmod 600 /opt/webshop/.env
NEXT_PUBLIC_API_URL is not needed on the server: the frontend image bakes it in at build time
from the PRODUCTION_API_URL secret (https://www.freezedesign.nl/api).
Step 2: PostgreSQL SSL Certificates¶
Same process as staging (see section 3, step 4).
Step 3: Generate SSL Certificates¶
The production certificate is one lineage, freezedesign.nl, covering freezedesign.nl,
www.freezedesign.nl and prod.freezedesign.nl (nginx/nginx.conf reads
/etc/letsencrypt/live/freezedesign.nl/). It was first issued for prod. only and
expanded to the apex and www on 2026-09-28 via a manual DNS-01 challenge; renewals use
HTTP-01 through the nginx webroot (scripts/renew-ssl.sh forces
--preferred-challenges http-01).
# Verify DNS
dig @8.8.8.8 www.freezedesign.nl A +short
curl -4 ifconfig.me # confirm VPS IP matches
# Fresh issue (only when rebuilding the server; the volume name follows the
# compose project, i.e. the directory name /opt/webshop):
docker run --rm -p 80:80 \
-v webshop_certbot_certs:/etc/letsencrypt \
certbot/certbot certonly --standalone \
--cert-name freezedesign.nl \
-d freezedesign.nl -d www.freezedesign.nl -d prod.freezedesign.nl \
--agree-tos --no-eff-email --email info@freezedesign.nl
Step 4: Create Database and Start Services¶
# Start database
docker compose -f docker-compose.prod.yml up -d db
docker compose -f docker-compose.prod.yml exec db \
psql -U webshop_prod -d postgres -c "CREATE DATABASE webshop_prod;"
# Start all services
docker compose -f docker-compose.prod.yml up -d
# Run migrations
docker compose -f docker-compose.prod.yml exec backend python manage.py migrate
# Create staff user
docker compose -f docker-compose.prod.yml exec backend python manage.py create_staff_user
Step 5: Verify Health¶
curl -f https://www.freezedesign.nl/api/health/
curl -f https://www.freezedesign.nl/health
curl -sI https://www.freezedesign.nl/ | head -5
# Apex and prod. redirect to www
curl -sI https://freezedesign.nl/ | grep -i location
curl -sI https://prod.freezedesign.nl/ | grep -i location
openssl s_client -connect www.freezedesign.nl:443 \
-servername www.freezedesign.nl < /dev/null 2>/dev/null | \
openssl x509 -noout -dates
Step 6: Cron entries¶
scripts/provision-server.sh installs /etc/cron.d/webshop-restart-unhealthy (every 5
minutes, restarts an unhealthy celery-beat and reports in Discord). Both servers also have
a root crontab that runs scripts/renew-ssl.sh on Sundays 03:30 UTC as a safety net next
to ssl-renewal.yml (see section 6). The media copy (scripts/backup-media.sh) is a
separate cron job that Ivar sets up by hand (see scripts/backup/README.md).
5. CI/CD Pipeline¶
Staging: Automatic on Push to staging¶
The staging deploy is the final deploy job inside .github/workflows/ci.yml. On every push to the staging branch (typically a squash-merged feature PR), CI runs all tests first; only when backend-test, frontend-test, frontend-build, e2e-smoke-test, and e2e-visual-test are green does the deploy job call .github/workflows/deploy-staging.yml via workflow_call.
deploy-staging.yml itself only has workflow_call and workflow_dispatch triggers -- the old standalone workflow_run trigger no longer exists. Use gh workflow run deploy-staging.yml for a manual (re)deploy of the current staging tip.
What it does:
- Build phase -- On a staging push
ci.ymlcallsbuild-images.ymlin parallel with the tests: it builds the backend image, the staging frontend and the production frontend for that commit, with OCI revision metadata andGIT_SHA, and pushes them to GHCR (namespaceghcr.io/voorman/..., lowercase) undersha-<commit>tags only. Builds use the GitHub Actions cache with separate scopes (backend,frontend,frontend-production) so the images don't evict each other's layers. A manualdeploy-staging.ymlrun builds the staging set itself - Superseded check -- Reads the revision label of the backend container on staging; if staging already runs a newer commit, the deploy is skipped (
superseded=true) instead of rolling staging back - Retag -- Only now, with green tests, the
sha-images get thestagingtag thatdeploy-rolling.shpulls - Deploy phase -- Copies
scripts/*.sh,docker-compose.staging.ymlandnginx/nginx.staging.confto/opt/webshopvia SCP, then executesscripts/deploy-rolling.shvia SSH withDEPLOY_TAG=staging(with immediatescripts/rollback.shif it fails) - Health verification -- Checks Docker health status of backend and frontend containers. If either fails, triggers automatic rollback via
scripts/rollback.sh - Post-deploy functional smoke -- Over SSH on the VPS: reads the running backend image's OCI revision label and compares it with the pushed SHA (catches a silently-stale staging without exposing deployment metadata), then verifies
/api/health/,/api/settings/, and/api/products/all return HTTP 200 - k6 performance gate and E2E smoke -- Starts
staging-perf-gate.yml(loadtests/homepage-load.jsagainstSTAGING_URL) ande2e-staging.yml(e2e/staging-smoke.spec.tswith the deployedsha); both run on their own, never block or roll back, and report a failure in Discord - Notifications -- Sends Discord notifications on started/success/failure (and the same to ntfy topic
freeze-deploy, see ntfy notifications);ci.yml'snotify-stagingjob reports the overall result
Production: Tagged Releases via scripts/deploy-production.sh¶
.github/workflows/deploy-production.yml has only a workflow_dispatch trigger with a tag input. scripts/deploy-production.sh creates that tag on origin/main and dispatches the workflow, which:
- Looks for the
sha-images of the tag commit (scripts/find-prebuilt-images.sh; for a docs-only tag commit it walks back to the last commit with code) and retags them aslatestandX.Y; only when none exist does it build from the tag (withGIT_SHAand the revision label of the tag commit). It refuses to build a frontend whosePRODUCTION_NEXT_PUBLIC_SENTRY_DSNequals the staging DSN - Determines the previous version tag for a rollback (
PREVIOUS_VERSION) - Copies
scripts/*.sh,docker-compose.prod.ymlandnginx/nginx.confto/opt/webshopand runsscripts/deploy-rolling.shwithDEPLOY_TAG=latestandPRE_DEPLOY_BACKUP=true(check --deploy --tag security,safe_migrate --dry-runwith a backup, then migrations) - Verifies health, runs the same revision-label + endpoint smoke as staging against
PRODUCTION_URL, and rolls back automatically on failure - Notifies Discord via
DISCORD_WEBHOOK_PRODUCTION(fallbackDISCORD_WEBHOOK_CICD, thenDISCORD_WEBHOOK_URL)
The GitHub environment production has no required reviewers; the confirmation prompt in the script is the gate. See the deployment runbook for the operator steps.
Required GitHub Secrets¶
| Secret | Staging | Production |
|---|---|---|
STAGING_HOST / PRODUCTION_HOST |
VPS IP | VPS IP |
STAGING_USER / PRODUCTION_USER |
deploy |
deploy |
STAGING_SSH_KEY / PRODUCTION_SSH_KEY |
SSH private key | SSH private key |
STAGING_API_URL / PRODUCTION_API_URL |
https://staging.freezedesign.nl/api |
https://www.freezedesign.nl/api (baked into the frontend image) |
STAGING_URL / PRODUCTION_URL |
https://staging.freezedesign.nl |
https://www.freezedesign.nl (smoke target and Discord link) |
STAGING_BASIC_AUTH_BASE64 |
Base64 basic-auth credentials for the k6 gate, load-tests.yml and e2e-staging.yml |
n/a |
NEXT_PUBLIC_SENTRY_DSN / PRODUCTION_NEXT_PUBLIC_SENTRY_DSN |
Frontend Sentry DSN (freeze-frontend-staging) |
Frontend Sentry DSN (freeze-design-frontend); must differ from staging |
SENTRY_AUTH_TOKEN, NEXT_PUBLIC_POSTHOG_KEY |
Source-map upload and PostHog key for the frontend build | Same |
POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID |
Source-map upload to PostHog Error tracking (EU) in the frontend build; without them the build skips it | Same |
DISCORD_WEBHOOK_URL, DISCORD_WEBHOOK_CICD |
Deploy notifications | DISCORD_WEBHOOK_PRODUCTION (fallback to the staging ones) |
Compose File Requirements for CI/CD¶
The compose files used by CI/CD must use image: directives pointing to GHCR, not build:. Using build: causes "No image to be pulled" errors during docker compose pull.
# Correct (staging compose)
backend:
image: ghcr.io/voorman/webshop_freeze_design/backend:${IMAGE_TAG:-staging}
# Wrong (will fail with CI/CD)
backend:
build:
context: ./backend
Use the IMAGE_TAG environment variable for versioning: set it in the .env file at the compose level or export it before running compose commands.
GHCR Package Visibility¶
GHCR packages must be set to public so the VPS can pull images without authentication. Change visibility in GitHub: Package settings > Change visibility > Public.
6. SSL Certificate Management¶
Automatic Renewal¶
Let's Encrypt certificates are valid for 90 days. Renewal is automated by .github/workflows/ssl-renewal.yml (weekly, Wednesday 04:00 UTC, via SSH), which runs a matrix over staging and production; both legs call the same idempotent scripts/renew-ssl.sh (certbot in a container, webroot challenge, nginx reload, check that port 443 serves the new certificate). Both VPSes also have a root crontab (Sundays 03:30 UTC) that runs the same script as a safety net (set up 2026-10-04, see .planning/todos/pending/production-vps-ssl-renewal-setup.md). The workflow's expiry-check job then checks www.freezedesign.nl, freezedesign.nl, prod.freezedesign.nl and staging.freezedesign.nl from outside and alerts in Discord below 14 days. Renewal only renews: adding names to a certificate is a separate certbot --expand step.
DOMAIN is the certbot lineage (directory under /etc/letsencrypt/live/), not the
hostname: freezedesign.nl on production, staging.freezedesign.eu on staging (the staging
certificate was created under the old name and expanded; certbot never renames a lineage).
The crontab line on each server (root):
# Production
30 3 * * 0 cd /opt/webshop && DOMAIN=freezedesign.nl COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh
# Staging
30 3 * * 0 cd /opt/webshop && DOMAIN=staging.freezedesign.eu COMPOSE_FILE=/opt/webshop/docker-compose.staging.yml bash scripts/renew-ssl.sh
Manual Renewal¶
# Check certificate expiry
openssl s_client -connect www.freezedesign.nl:443 \
-servername www.freezedesign.nl < /dev/null 2>/dev/null | \
openssl x509 -noout -dates
# Renew now (production); the script reloads nginx itself
ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh'
# Force renewal when the normal renewal is blocked
# via GitHub: gh workflow run ssl-renewal.yml -f force=true
# on the server: add CERTBOT_EXTRA_FLAGS="--force-renewal" to the command above
Troubleshooting SSL¶
If Let's Encrypt validation fails:
- Verify DNS resolves correctly from external sources:
- If an AAAA record exists, ensure it points to the correct VPS IPv6 address or delete it. Let's Encrypt prefers IPv6 and will fail if the AAAA record points elsewhere.
- Ensure port 80 is open (
ufw allow 80/tcp). - Ensure the
certbot_webrootvolume is not mounted as read-only (:ro) in the nginx service. Certbot needs to write challenge files.
7. Updating / Deploying New Versions¶
Via CI/CD (Recommended)¶
Staging: Merge a PR into the staging branch (feature PRs always target staging, squash-merged). CI runs the full test suite and, when green, the deploy job deploys to staging automatically.
gh pr merge <pr-number> --squash
# CI runs on the staging push; the deploy job handles the rest
# Manual (re)deploy of the current staging tip:
gh workflow run deploy-staging.yml
Promotion to main: Run scripts/release.sh --promote-only. It confirms the staging HEAD was deployed (from the CI push run on staging), opens or reuses the promotion PR, waits for its checks, and fast-forwards main to the PR's head SHA; GitHub then marks the PR as merged. main and staging keep the same SHAs. Promoting to main does not deploy anything.
Production: Run scripts/deploy-production.sh. It tags origin/main with the next vX.Y (or --tag), shows the commits and new migrations, asks for confirmation, and dispatches and follows deploy-production.yml. See the deployment runbook.
Manual Deployment¶
If CI/CD is unavailable, deploy manually from the VPS:
cd /opt/webshop
# Pull latest images
export IMAGE_TAG=staging # or a specific tag like v1.20
docker compose -f docker-compose.staging.yml pull
# Run rolling deployment
export COMPOSE_FILE=docker-compose.staging.yml
export DEPLOYMENT_DIR=/opt/webshop
# Verplicht: bepaalt welke image-tag als rollback-referentie geldt.
# staging -> staging, productie -> latest.
export DEPLOY_TAG=staging
./scripts/deploy-rolling.sh
# Or for a simpler (non-rolling) update:
docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py collectstatic --noinput
After Changing Environment Variables¶
When you change values in .env.staging or .env.production, restarting containers is not enough. Containers cache environment variables at creation time. You must recreate them:
# Stop, remove, and recreate the affected service
docker compose -f docker-compose.staging.yml stop backend
docker compose -f docker-compose.staging.yml rm -f backend
docker compose -f docker-compose.staging.yml up -d backend
8. Rollback Procedures¶
Automatic Rollback¶
The CI/CD pipeline automatically rolls back when health checks fail after deployment. The flow:
deploy-rolling.shruns -- if it fails,rollback.shis called immediately- Post-deploy health verification checks Docker health status of backend and frontend
- If either service is unhealthy after multiple retries,
rollback.shrestores the localpredeploysnapshot, or else pulls the previous version tag (PREVIOUS_VERSION, e.g.1.38) from GHCR, and restarts services - A final fallback step runs if the health check step itself fails
Manual Rollback¶
To manually roll back to a previous version:
cd /opt/webshop
# Option 1: Roll back to a specific image tag
export IMAGE_TAG=1.38 # the known-good version (registry tag without 'v'; exists from 1.39 on)
docker compose -f docker-compose.prod.yml pull
docker compose -f docker-compose.prod.yml up -d
# Option 2: Use the rollback script
export COMPOSE_FILE=docker-compose.prod.yml
export DEPLOYMENT_DIR=/opt/webshop
export GITHUB_REPOSITORY=Voorman/webshop_freeze_design
export REGISTRY=ghcr.io
export ROLLBACK_VERSION=1.38 # optioneel: deze versie i.p.v. de lokale predeploy-snapshot
export DEPLOY_TAG=latest # verplicht; productie=latest, staging=staging
./scripts/rollback.sh
# Verify services are healthy
docker compose -f docker-compose.prod.yml ps
9. Database Migration Rollback¶
Django migrations can be reversed if needed. This is a manual process.
Identify the Migration to Roll Back¶
# List applied migrations for an app
docker compose -f docker-compose.prod.yml exec backend \
python manage.py showmigrations <app_name>
Roll Back a Migration¶
# Roll back to a specific migration (the one BEFORE the problematic one)
docker compose -f docker-compose.prod.yml exec backend \
python manage.py migrate <app_name> <migration_number>
# Example: roll back products app to migration 0005
docker compose -f docker-compose.prod.yml exec backend \
python manage.py migrate products 0005
Full Database Restore¶
If migration rollback is insufficient, restore from a backup. The hourly backups are
AES-256-GCM encrypted in DigitalOcean Spaces; restore_database downloads, decrypts
(BACKUP_ENCRYPTION_KEY) and restores them:
# Stop services that use the database
docker compose -f docker-compose.prod.yml stop celery celery-beat
# List and restore (newest, or a specific filename instead of --latest)
docker compose -f docker-compose.prod.yml exec backend python manage.py restore_database --list
docker compose -f docker-compose.prod.yml exec backend \
python manage.py restore_database --latest --force
# Restart services
docker compose -f docker-compose.prod.yml up -d
See the Disaster Recovery runbook for the full procedure.
Manual Database Backup¶
# Create a backup before risky operations
docker compose -f docker-compose.prod.yml exec db \
pg_dump -U webshop_prod webshop_prod > backup_$(date +%Y%m%d_%H%M%S).sql
gzip backup_*.sql
10. Troubleshooting¶
Service Health Checks¶
# Check all service statuses
docker compose -f docker-compose.prod.yml ps
# Check logs for a specific service
docker compose -f docker-compose.prod.yml logs backend --tail=50
docker compose -f docker-compose.prod.yml logs frontend --tail=50
docker compose -f docker-compose.prod.yml logs nginx --tail=50
docker compose -f docker-compose.prod.yml logs celery --tail=50
Health check details by container:
| Container | Health check tool | Command |
|---|---|---|
| backend | curl (Python image has curl) |
curl -f http://localhost:8000/api/health/ |
| frontend | wget (Node Alpine has wget, NOT curl) |
wget -q --spider http://127.0.0.1:3000/ |
| nginx | curl (Alpine has curl) |
curl -f http://127.0.0.1/health |
| db | pg_isready |
pg_isready -U $POSTGRES_USER on production; staging and dev add -d $POSTGRES_DB (without it pg_isready uses the user name as database and logs a FATAL every 5 seconds, v1.54) |
| redis | redis-cli |
redis-cli -a $REDIS_PASSWORD ping |
| celery | celery inspect |
celery -A config inspect ping |
Nginx 502 Bad Gateway After Deployment¶
Nginx caches upstream DNS at startup. When containers are recreated during deployment, they get new Docker network IPs but nginx still routes to the old IPs.
Fix: Restart nginx after deploying services:
The rolling deploy script (deploy-rolling.sh) already does this as its final step.
Container Environment Variable Changes Not Taking Effect¶
Containers cache env vars at creation time. Restarting is not enough:
# Wrong: restart only re-reads env_file on some Docker versions
docker compose -f docker-compose.prod.yml restart backend
# Correct: stop, remove, and recreate
docker compose -f docker-compose.prod.yml stop backend
docker compose -f docker-compose.prod.yml rm -f backend
docker compose -f docker-compose.prod.yml up -d backend
Redis "Port Could Not Be Cast to Integer" Error¶
This means REDIS_PASSWORD contains special characters (like /) that break URL parsing in CELERY_BROKER_URL and REDIS_URL. Regenerate with alphanumeric characters only:
Update both REDIS_PASSWORD in the backend env file and the compose-level .env file, then recreate all services that use Redis:
docker compose -f docker-compose.staging.yml stop redis backend celery celery-beat
docker compose -f docker-compose.staging.yml rm -f redis backend celery celery-beat
docker compose -f docker-compose.staging.yml up -d
SSL Certificate Errors¶
# Verify certificate files exist (the certs live in the Docker volume webshop_certbot_certs)
sudo ls -la "$(docker volume inspect webshop_certbot_certs --format '{{.Mountpoint}}')/live/freezedesign.nl/"
# Check what nginx serves
openssl s_client -connect localhost:443 -servername www.freezedesign.nl < /dev/null 2>/dev/null | \
openssl x509 -noout -dates
# Re-generate certificates (staging example; see section 6 for renewals)
./nginx/certbot-init.sh --domain staging.freezedesign.nl --compose-file docker-compose.staging.yml
Database Connection Issues¶
# Check PostgreSQL logs
docker compose -f docker-compose.prod.yml logs db --tail=50
# Verify database accepts connections
docker compose -f docker-compose.prod.yml exec db pg_isready -U webshop_prod
# Check connection count
docker compose -f docker-compose.prod.yml exec db \
psql -U webshop_prod -c "SELECT count(*) FROM pg_stat_activity;"
Memory Issues¶
# System memory
free -h
# Docker container memory usage
docker stats --no-stream
# Redis memory usage
docker compose -f docker-compose.prod.yml exec redis \
redis-cli -a "$REDIS_PASSWORD" --no-auth-warning info memory | grep used_memory_human
Celery Task Issues¶
# Check active tasks
docker compose -f docker-compose.prod.yml exec celery \
celery -A config inspect active
# Check Celery Beat schedule
docker compose -f docker-compose.prod.yml logs celery-beat --tail=20
# Check Redis queue length
docker compose -f docker-compose.prod.yml exec redis \
redis-cli -a "$REDIS_PASSWORD" --no-auth-warning llen celery
Disk Space¶
# Check disk usage
df -h
# Check Docker disk usage
docker system df
# Clean up unused Docker resources
docker system prune -f
# Remove unused images (use with caution in production)
docker image prune -a -f
Resetting Staging Data¶
To reset the staging database to a clean state:
docker compose -f docker-compose.staging.yml down
docker volume rm webshop_postgres_data # compose project "webshop" (from /opt/webshop)
docker compose -f docker-compose.staging.yml up -d
docker compose -f docker-compose.staging.yml exec backend python manage.py migrate
docker compose -f docker-compose.staging.yml exec backend python manage.py create_staff_user
Debugging Webhooks (Mollie)¶
If payment webhooks return 404, check that BACKEND_URL does not include /api. The webhook URL is constructed as {BACKEND_URL}/api/payments/webhook/, so a BACKEND_URL of https://freezedesign.nl/api would result in the path /api/api/payments/webhook/.
Shell Escaping in Remote Debugging¶
These examples assume the server shell setup from the Alerting Runbook (cd /opt/webshop and export COMPOSE_FILE=…); a bare docker compose on the server fails without it.
When debugging via SSH into Docker containers, avoid embedding JSON in nested shell commands. Multi-layer escaping (local -> ssh -> docker exec -> bash -c -> curl -d) silently corrupts request bodies. Instead, write JSON to a file first:
# Wrong: body gets silently corrupted
docker compose exec backend bash -c 'curl -d "{\"key\":\"value\"}" ...'
# Correct: write JSON to file first
docker compose exec backend bash -c 'echo "{\"key\":\"value\"}" > /tmp/body.json && curl -d @/tmp/body.json ...'
Monitoring Checklist¶
After deployment, verify these items regularly (on the server, after the setup from the Alerting Runbook):
- All services show "healthy" in
docker compose ps - SSL certificate expiry > 30 days
- Disk usage < 80%
- Memory usage < 80%
- Database connections < 40 (max 50 configured)
- Redis memory < 200MB (256MB limit on production)
- No errors in
docker compose logs backend --tail=100 - Sentry error count at acceptable level
- Celery-beat healthy (
docker compose ps celery-beat);journalctl -t restart-unhealthyempty
Operational procedures are in Monitoring, Alerting and Disaster Recovery.