Skip to content

Deployment Runbook

This document provides step-by-step procedures for deploying, rolling back, and managing the FreezeDesign webshop application.

Environments

Two environments are live. Staging (https://staging.freezedesign.nl, basic auth) deploys automatically from the staging branch. Production (https://www.freezedesign.nl, v1.60 since 2026-11-06; freezedesign.nl and prod.freezedesign.nl redirect to www) deploys tagged releases from main with scripts/deploy-production.sh.

On the server

Both servers run the stack in /opt/webshop as the deploy user, without a docker-compose.yml. Start every SSH session with cd /opt/webshop && export COMPOSE_FILE=docker-compose.prod.yml (staging: docker-compose.staging.yml); the examples below spell out -f for clarity.

Table of Contents

Overview

Architecture

The FreezeDesign application is deployed using a Docker-based infrastructure with the following characteristics:

  • Two live environments: staging (every green push to staging) and production (tagged releases from main)
  • Container registry: GitHub Container Registry (GHCR)
  • Deployment strategy: Zero-downtime rolling deployment
  • VPS hosting: Single VPS per environment with Docker Compose
  • Automatic rollback: Health check failures trigger automatic rollback

Deployment Flow

flowchart TD
    A[GitHub Actions: images in GHCR] --> B[VPS: docker compose pull]
    B --> C[deploy-rolling.sh<br/>nginx -t, migrate, roll backend then frontend,<br/>celery, recreate nginx]
    C --> D{Docker health<br/>backend + frontend}
    D -->|healthy| E[Smoke: image revision == expected SHA<br/>+ /api/health/, /api/settings/, /api/products/ = 200]
    D -->|unhealthy| R[rollback.sh<br/>predeploy image, or previous vX.Y on production]
    E --> F[Discord: success]
    R --> G[Discord: failure, workflow red]
    E -.->|staging only| K[staging-perf-gate.yml: k6 homepage]
    E -.->|staging only| S[e2e-staging.yml: Playwright smoke]

One deploy, from image to notification; the dotted steps are the staging-only k6 gate and E2E smoke that run in their own workflows after the deploy.

Images are pushed to GHCR under the lowercase namespace ghcr.io/voorman/webshop_freeze_design/{backend,frontend}.

Key Scripts

  • scripts/deploy-rolling.sh - Zero-downtime rolling deployment
  • scripts/rollback.sh - Automatic rollback on failure
  • scripts/health-check.sh - Shared Docker health-status check (sourced by the workflow)
  • scripts/notify-discord.sh - Discord notifications, also sent to ntfy (freeze-deploy) through scripts/notify-ntfy.sh since v1.59

Staging Deployment

Trigger: Automatic on push to the staging branch, as the final deploy job of the CI workflow (after all tests pass)

Workflows:

  • .github/workflows/ci.yml — contains the deploy job (uses: ./.github/workflows/deploy-staging.yml via workflow_call), gated on backend-test, frontend-test, frontend-build, e2e-smoke-test and e2e-visual-test
  • .github/workflows/deploy-staging.yml — the reusable deploy workflow itself (on: workflow_call + workflow_dispatch). A standalone "Deploy to Staging" run only exists for manual dispatch.

Flow

  1. Push to staging branch triggers the CI workflow
  2. Build images (build-images job, .github/workflows/build-images.yml) runs in parallel with the tests: backend and frontend images with the staging configuration, pushed to GHCR as sha-{commit} only. The frontend is built with NEXT_PUBLIC_API_URL from the staging secrets, the backend with GIT_SHA (its Sentry release) and the OCI revision label
  3. CI tests pass (linting, type checking, unit tests, E2E smoke + visual tests)
  4. Superseded guard: the deploy job reads the revision label of the backend image running on staging. If staging already runs a newer commit of staging, the deploy is skipped (notice in the run); the newest commit always wins
  5. Tag the tested images as staging (docker buildx imagetools create, seconds)
  6. Copy deployment scripts, compose file and nginx config to the staging VPS (appleboy/scp-action: scripts/*.sh, docker-compose.staging.yml, nginx/nginx.staging.conf)
  7. SSH to the staging VPS (appleboy/ssh-action) and run deploy-rolling.sh with DEPLOY_TAG=staging (rollback via rollback.sh on failure)
  8. Health checks verify backend and frontend via Docker health status (scripts/health-check.sh)
  9. Post-deploy functional smoke (see below) — deployed-SHA match + key endpoints
  10. Start the k6 gate: the deploy job dispatches staging-perf-gate.yml and does not wait for it. A k6 failure never rolls back; it posts to Discord (DISCORD_WEBHOOK_URL)
  11. Start the E2E smoke (#861): the deploy job dispatches e2e-staging.yml with spec=e2e/staging-smoke.spec.ts and the deployed sha. Playwright runs home, product page and cart against https://staging.freezedesign.nl behind the basic auth (STAGING_BASIC_AUTH_BASE64), without traces or video. Red means a Discord message "staging (E2E-smoke)" (DISCORD_WEBHOOK_URL), never a rollback. The same workflow runs any spec by hand: gh workflow run e2e-staging.yml --ref staging -f spec=e2e/<naam>.spec.ts
  12. Discord notification on success or failure (DISCORD_WEBHOOK_URL)

Only one deploy runs on the staging server at a time (concurrency group staging-server); a third queued run cancels the middle one.

Post-Deploy Functional Smoke

Before the k6 gate, the deploy-staging.yml workflow runs a fast functional smoke (step "Post-deploy functional smoke (deployed-SHA + key endpoints)"):

  1. Deployed-SHA match — reads the OCI revision label from the image used by the running backend container. It must equal the commit that was just pushed (github.sha); otherwise the deploy failed to roll the new code and the step fails. The public health response does not expose this deployment metadata.
  2. Key endpoints return 200 — GET /api/health/, GET /api/settings/, and GET /api/products/ are each expected to return HTTP 200.

Only if both checks pass does the workflow count as a successful deploy and start the k6 gate and the E2E smoke.

Manual Intervention

Staging deployment is fully automatic. To manually redeploy the current staging tip (without re-running CI):

gh workflow run "Deploy to Staging"

Promotion to main

Promote only after the retest on staging: Eefje has tested her feedback issues on staging and closed them, and Ivar has checked the todos on staging. Then promote staging to main with:

scripts/release.sh --promote-only

The script first confirms that the staging HEAD was deployed (it only counts the CI push run on staging; the promotion PR's own run has no deploy job). It then pauses for UAT and asks Promotie naar main starten? [y/N]; answer y only when that retest is done. After the confirmation it opens or reuses the promotion PR, waits for its checks, and then fast-forwards main to the PR's head SHA with a direct push. GitHub marks the PR as merged once main contains that commit. Nothing is rewritten, so main and staging keep the same SHAs; the script refuses if main has a commit that staging lacks. The staging branch has a protection rule (deletion and force-push forbidden); the restore and retarget steps in release.sh are safety nets.

Promoting to main does not deploy anything. Production deploys are a separate step (see below).

Monitoring

  • GitHub Actions workflow run status (the deploy job inside the CI run, or a manual "Deploy to Staging" run)
  • Discord notifications
  • Health check logs in workflow output

Production Deployment

Trigger: scripts/deploy-production.sh, which tags origin/main and dispatches .github/workflows/deploy-production.yml with that tag.

Environment protection: the GitHub environment production has no required reviewers (checked on 2026-09-28: the deploy job started right after the build). The deliberate confirmation is the [y/N] prompt in the script.

Pre-Deployment Checklist

  • main was updated with scripts/release.sh --promote-only, so its SHA ran on staging first
  • The latest production backup is recent (hourly; check the BackupRecord list or freeze-production-backups)
  • New migrations reviewed (the script lists them before it asks for confirmation)
  • Changes to backend/.env.production are in place (containers only read it when they are created, which the deploy does)
  • New secrets from the "Voor Ivar" issue (label voor-ivar) are set on production; see Voor Ivar

Deployment Steps

scripts/deploy-production.sh --dry-run    # show what would happen
scripts/deploy-production.sh              # next version: highest vX.Y tag + 1
scripts/deploy-production.sh --tag v1.28  # a specific version, or resume with an existing tag on main
  1. Version and target: the script picks the next vX.Y (numeric: v1.9 → v1.10) or takes --tag, and always tags origin/main, never the local HEAD (which usually sits on staging and runs ahead of main). It refuses if main already carries the latest tag, if the tag exists on another commit, or if main is not an ancestor of staging.
  2. Overview and confirmation: previous and new tag, commit, commits since the previous tag, and new migrations; then Productie uitrollen naar vX.Y? [y/N].
  3. Tag: annotated, with the commit list as message, pushed to origin.
  4. Workflow (deploy-production.yml):
  5. reuses the images that CI already built and tested for the tag commit (or, for a tag on a docs-only commit, for the last commit with code: scripts/find-prebuilt-images.sh) and retags them as latest and X.Y; only when nothing is found does it build from the tag itself (with GIT_SHA and the OCI revision label of the tag commit);
  6. looks up the previous vX.Y tag and passes it as PREVIOUS_VERSION for rollback.sh;
  7. copies the deployment scripts, compose file and nginx config to the VPS;
  8. runs scripts/deploy-rolling.sh with PRE_DEPLOY_BACKUP=true: first check --deploy --tag security --fail-level WARNING (stops on a test_ Mollie key without ALLOW_TEST_MOLLIE_KEY, or on E2E_TESTING=True), then safe_migrate --dry-run (flags destructive migrations and makes a database backup, and stops if that backup fails), then the migrations and the rolling update;
  9. verifies Docker health, then a smoke test that checks the running image's revision label against the tag and hits key endpoints;
  10. rolls back automatically if the health check fails, and notifies Discord on start, success and failure.
  11. Follow: the script finds the run and follows it with gh run watch until it finishes. On failure it prints the run URL.

If the tag was pushed but the workflow did not start, rerun with --tag vX.Y to resume; the script reuses a tag that already sits on main.

Post-Deployment Verification

ssh production-vps
cd /opt/webshop

# Running code: the revision label must be the tagged commit
docker image inspect ghcr.io/voorman/webshop_freeze_design/backend:latest \
  --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'

# Container health and migrations
docker compose -f docker-compose.prod.yml ps
docker compose -f docker-compose.prod.yml exec backend python manage.py showmigrations --plan | grep -c "\[ \]"   # 0 = nothing pending

# Logs
docker compose -f docker-compose.prod.yml logs -f --tail=100

Then check Sentry (environment:production) for new errors.

Update the CHANGELOG

Each production tag has its own section in CHANGELOG.md: tag v1.28 is ## [1.28] - <date>. After a successful deploy, scripts/deploy-production.sh prints a reminder as long as main has no section for the new tag. Open a small PR to staging that renames ## [Unreleased] to ## [1.28] - <date> and puts an empty ## [Unreleased] above it. It reaches main with the next promotion.

Known Failure: DNS Lookup Error on the Runner

Copy deployment scripts to server can fail with dial tcp: lookup *** on 127.0.0.53:53: no such host while the server itself is reachable. That is a DNS problem on the GitHub runner. Rerun with scripts/deploy-production.sh --tag vX.Y. If it keeps failing, run scripts/deploy-rolling.sh on the VPS with the same environment as the workflow: COMPOSE_FILE=docker-compose.prod.yml, DEPLOYMENT_DIR=/opt/webshop, GITHUB_REPOSITORY, REGISTRY=ghcr.io, DEPLOY_TAG=latest and PRE_DEPLOY_BACKUP=true (without it the script skips check --deploy and safe_migrate --dry-run, including its backup).

Zero-Downtime Deployment Details

The deploy-rolling.sh script updates services one at a time with an automatic per-service rollback to the previous image.

How It Works

The script refuses to run without DEPLOY_TAG (staging or latest), so a rollback reference can never silently point at the other environment's image.

  1. Pull images: docker compose pull fetches the images for DEPLOY_TAG from GHCR
  2. Test the nginx config: nginx -t runs in a throwaway container of the nginx service (docker compose run --rm --no-deps -T nginx nginx -t), with the config the workflow just copied, the real certificates and the freshly pulled image. If it fails, the deploy stops before migrations and container swaps, and the last config nginx ran with (nginx/nginx.conf.vorige-goed, or nginx/nginx.staging.conf.vorige-goed on staging) is copied back
  3. Production checks (only with PRE_DEPLOY_BACKUP=true, which deploy-production.yml sets): manage.py check --deploy --tag security --fail-level WARNING on the new image (stops on a test_ Mollie key or E2E_TESTING=True), then safe_migrate --dry-run (flags destructive migrations and makes a database backup; stops if that backup fails)
  4. Migrations: python manage.py migrate --noinput runs in a one-off backend container (run --rm) on the new image, before any container is swapped. A failing migration aborts the deploy with the old containers still running
  5. For each service (backend, then frontend):
    1. Backup image: The currently running image is tagged locally as predeploy
    2. Recreate: Container is recreated with the new image (up -d --no-deps --force-recreate)
    3. Health check: Docker health status is polled (30 attempts, 5-second intervals = 150 seconds max)
    4. Rollback on failure: If the health check fails, the predeploy image is restored and the container recreated again
  6. Celery services: the owner of the celerybeat_schedule volume (/app/celerybeat) is set to appuser in a short root container (since #811: a root-owned volume kept beat restarting), then celery and celery-beat are updated with a plain up -d (no wait for health; beat's own healthcheck shows up in docker compose ps after about a minute)
  7. Nginx recreate: Nginx is recreated (up -d --no-deps --force-recreate nginx) with the pinned image from the compose file (e.g. nginx:1.31.6-alpine), which also refreshes upstream DNS (new container IPs). A plain restart would keep the old image. Before that, the running image is tagged locally as nginx:predeploy. Once the new nginx is healthy, its config is saved as the .vorige-goed copy for step 2. If it does not become healthy, the nginx:predeploy image and the .vorige-goed config are put back and the deploy fails

If a deploy stops with ✗ nginx-config is ongeldig, the log shows the nginx -t error. The running nginx is untouched and the previous config is back on disk (unless this was the first deploy with the check, then nothing could be restored). Fix the config in the repo and deploy again. scripts/rollback.sh does not touch nginx.

If a deploy fails with ✗ Nieuwe nginx is niet healthy, nginx runs on the previous image and config again (✓ nginx teruggezet op het vorige image). Locally, the compose image name (e.g. nginx:1.31.6-alpine) then points to that old image until the next docker compose pull. Backend and frontend are already on the new version at that point. To update nginx, change the version in both docker-compose.prod.yml and docker-compose.staging.yml.

Resource Requirements

  • VPS capacity: during the roll two backend (or frontend) containers run side by side for a moment; the mem_limits in docker-compose.prod.yml add up to about 3.75 GB on a 16 GB machine, see Hosting
  • Duration: ~2-5 minutes for full deployment

Health Check Details

  • Mechanism: Docker's built-in health status (from the healthcheck definitions in the compose file), read via docker compose ps --format json
  • Backend healthcheck: curl -f http://localhost:8000/api/health/ (database + Redis); frontend: wget --spider http://127.0.0.1:3000/; nginx: curl -f http://localhost/health
  • Max attempts: 30 attempts × 5 seconds = 150 seconds maximum wait per service
  • The workflow-level verification step afterwards (scripts/health-check.sh, check_docker_health) uses 10 attempts × 5 seconds per service

Celery Services

Celery worker and beat scheduler are recreated with a plain up -d (not rolled, the deploy does not wait for their health) since they don't serve HTTP traffic. Both do have a Docker healthcheck: the worker answers celery inspect ping, and beat (since #798) touches /app/celerybeat/heartbeat on every tick; older than 10 minutes means unhealthy. A cron job on each server restarts an unhealthy beat and reports it in Discord — see Monitoring.

The worker command is celery -A config worker -l info --concurrency=2 --max-memory-per-child=262144 on production (--concurrency=1 on staging): a child above 256 MiB after a task is replaced. Production caps the service at mem_limit: 768m (#990). A recreate of celery aborts a running task; the deploy accepts that (tasks are acks_late, print files are regenerated by the hourly recover task).

Automatic Rollback

Trigger: Either deploy-rolling.sh fails on the VPS, or the workflow's health verification step (10 attempts × 5 seconds per service) fails afterwards

Script: scripts/rollback.sh

What Happens

  1. Deployment or health check fails
  2. Rollback script executes:
  3. Prefer the local predeploy backup image (snapshot taken by deploy-rolling.sh)
  4. Productie: otherwise pull the previous version tag (PREVIOUS_VERSION, e.g. 1.38) from GHCR — deploy-production.yml passes it in. Version tags exist from 1.39 on.
  5. Staging has no version tags: without a predeploy image the rollback fails
  6. Re-tag it as the deploy tag and recreate the services
  7. Discord notification sent with deployment failure status
  8. Workflow fails to signal deployment failure

Verification After Automatic Rollback

After automatic rollback, verify services are healthy (on the staging VPS):

ssh <user>@staging-vps
cd /opt/webshop
docker compose -f docker-compose.staging.yml ps   # all services healthy?
curl -f https://staging.freezedesign.nl/api/health/

Investigate Failure

  1. Check deployment logs in GitHub Actions workflow output (the deploy job in the CI run)
  2. Check application logs:
    docker compose -f docker-compose.staging.yml logs backend --tail=200
    docker compose -f docker-compose.staging.yml logs frontend --tail=200
    
  3. Identify root cause before attempting re-deployment

Manual Rollback Procedure

When to use:

  • Automatic rollback failed
  • Issues discovered after successful deployment
  • Emergency rollback needed outside of deployment workflow

Step-by-Step

  1. SSH into VPS:

    ssh <user>@staging-vps
    

  2. Navigate to deployment directory:

    cd /opt/webshop
    

  3. Set environment variables:

    export COMPOSE_FILE=docker-compose.staging.yml
    export GITHUB_REPOSITORY=Voorman/webshop_freeze_design
    export REGISTRY=ghcr.io
    

Which image is restored: - The local predeploy backup image on the VPS (the image that ran before the last deploy), if it still exists - Productie only: a specific version with ROLLBACK_VERSION=<X.Y> (e.g. 1.38, without v). This takes precedence over predeploy. Version tags exist from 1.39 on; the matching git tags are vX.Y. - Staging has no version tags (ROLLBACK_VERSION is refused with DEPLOY_TAG=staging)

  1. Run rollback script:

    # DEPLOY_TAG is verplicht: staging -> staging, productie -> latest.
    # Zonder deze variabele stopt het script met exit 2, zodat een rollback
    # nooit stilzwijgend het image van de ándere omgeving terugzet.
    DEPLOY_TAG=staging ./scripts/rollback.sh
    

  2. Verify health:

    docker compose -f docker-compose.staging.yml ps
    curl -f https://staging.freezedesign.nl/api/health/
    

  3. Check logs:

    docker compose -f docker-compose.staging.yml logs -f --tail=50
    

Last Resort: Simple Restart

If rollback.sh fails completely:

docker compose -f docker-compose.staging.yml restart backend frontend celery celery-beat

This restarts services with the currently tagged images (may not fully revert if images were already pulled).

Database Migration Rollback

Django migrations can be reversible or irreversible.

Reversible Migrations

Most migrations are automatically reversible:

  • AddField - removes the field
  • CreateModel - drops the table
  • AlterField - reverts field changes
  • AddIndex - drops the index

To rollback a reversible migration:

# SSH to VPS
ssh <user>@staging-vps
cd /opt/webshop

# Rollback to specific migration number
docker compose -f docker-compose.staging.yml exec backend \
  python manage.py migrate <app_name> <previous_migration_number>

# Example: rollback products app to migration 0015
docker compose -f docker-compose.staging.yml exec backend \
  python manage.py migrate products 0015

Verify rollback:

# Show current migration status
docker compose -f docker-compose.staging.yml exec backend \
  python manage.py showmigrations <app_name>

Irreversible Migrations

Some migrations cannot be automatically reversed:

  • RemoveField - data is lost when field is dropped
  • DeleteModel - data is lost when table is dropped
  • RunPython - custom Python code without reverse_code
  • RunSQL - custom SQL without reverse SQL

For irreversible migrations:

  1. Do not attempt migrate rollback (will fail or lose data)
  2. Restore from database backup (see Disaster Recovery)
  3. Choose backup timestamp before the migration ran

Best Practices

  1. Deploy migrations separately:
  2. Release 1: Deploy migration only (additive, safe)
  3. Verify migration succeeded
  4. Release 2: Deploy code that uses new schema

  5. Make migrations reversible:

  6. Provide reverse_code for RunPython operations
  7. Avoid RemoveField and DeleteModel when possible
  8. Use AlterField with null=True before removing

  9. Test migrations on staging before production

  10. Take backup before risky migrations:

    # Trigger manual backup
    docker compose exec backend python manage.py backup_database
    

Troubleshooting

Health Check Fails

Symptom: Deployment fails with "<service> failed health check after N attempts" or "<service> health check failed"

Diagnosis:

# Check logs
docker compose -f docker-compose.staging.yml logs backend --tail=200
docker compose -f docker-compose.staging.yml logs frontend --tail=200

# Check Docker health status
docker compose -f docker-compose.staging.yml ps

# Test health endpoint from inside the container
docker compose -f docker-compose.staging.yml exec backend \
  curl -v http://127.0.0.1:8000/api/health/

Common causes:

  • Database connection failure (check DB_HOST, DB_PASSWORD in .env)
  • Redis connection failure (check REDIS_URL)
  • Missing environment variables
  • Application startup errors (check logs for stack traces)
  • Port already in use (check docker compose ps)

Container Won't Start

Symptom: Container exits immediately or restart loop

Diagnosis:

# Check container status
docker compose -f docker-compose.staging.yml ps

# Check resource usage
docker stats

# Inspect container
docker compose -f docker-compose.staging.yml logs <service-name> --tail=100

Common causes:

  • Out of memory (check docker stats)
  • Syntax error in code (check logs for Python/JavaScript errors)
  • Missing dependencies (rebuild image)
  • Configuration error in docker-compose.staging.yml

Image Pull Fails

Symptom: "Error pulling image" during deployment

Diagnosis:

# Check GHCR authentication
docker login ghcr.io -u <username>

# Verify image exists
docker pull ghcr.io/<repo>/backend:<tag>

Common causes:

  • Invalid tag name
  • GHCR authentication expired
  • Image not pushed to registry
  • Network connectivity issues

Solution:

# Re-authenticate to GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u <username> --password-stdin

# Verify image tags
docker images | grep ghcr.io

Disk Space Issues

Symptom: "No space left on device"

Diagnosis:

# Check disk usage
df -h

# Check Docker disk usage
docker system df

Solution:

# Clean up old images and containers
docker system prune -f

# Remove unused volumes (careful!)
docker volume prune -f

# Remove specific old images
docker images | grep '<none>' | awk '{print $3}' | xargs docker rmi

Migration Fails

Symptom: "Migration failed" during deployment

Diagnosis:

# Check migration status
docker compose exec backend python manage.py showmigrations

# Check migration errors
docker compose logs backend | grep -i migration

Common causes:

  • Database schema conflict
  • Missing dependency migration
  • Custom SQL error in migration
  • Database connection interrupted

Solution:

# Fake migration if already applied manually
docker compose exec backend python manage.py migrate <app> <migration> --fake

# Or rollback and re-apply
docker compose exec backend python manage.py migrate <app> <previous-migration>
docker compose exec backend python manage.py migrate <app>

Performance Degradation After Deployment

Symptom: Application slower than before deployment

Diagnosis:

# Check resource usage
docker stats

# Check database connections
docker compose exec db psql -U $DB_USER -d $DB_NAME -c "SELECT count(*) FROM pg_stat_activity;"

# Check Redis memory
docker compose exec redis redis-cli INFO memory

Common causes:

  • Missing database indexes (new queries added)
  • Inefficient queries (check Django query logs)
  • Memory leak (check docker stats over time)
  • Cache not warming up (Redis empty after restart)

Solution:

  • Review and optimize new queries
  • Add database indexes for slow queries
  • Restart services if memory leak suspected
  • Warm up cache if needed