Deployment Runbook¶
This document provides step-by-step procedures for deploying, rolling back, and managing the FreezeDesign webshop application.
Environments
Two environments are live. Staging (https://staging.freezedesign.nl, basic auth)
deploys automatically from the staging branch. Production
(https://www.freezedesign.nl, v1.60 since 2026-11-06; freezedesign.nl and
prod.freezedesign.nl redirect to www) deploys tagged releases from main with
scripts/deploy-production.sh.
On the server
Both servers run the stack in /opt/webshop as the deploy user, without a
docker-compose.yml. Start every SSH session with
cd /opt/webshop && export COMPOSE_FILE=docker-compose.prod.yml (staging:
docker-compose.staging.yml); the examples below spell out -f for clarity.
Table of Contents¶
- Overview
- Staging Deployment
- Production Deployment
- Zero-Downtime Deployment Details
- Automatic Rollback
- Manual Rollback Procedure
- Database Migration Rollback
- Troubleshooting
Overview¶
Architecture¶
The FreezeDesign application is deployed using a Docker-based infrastructure with the following characteristics:
- Two live environments: staging (every green push to
staging) and production (tagged releases frommain) - Container registry: GitHub Container Registry (GHCR)
- Deployment strategy: Zero-downtime rolling deployment
- VPS hosting: Single VPS per environment with Docker Compose
- Automatic rollback: Health check failures trigger automatic rollback
Deployment Flow¶
flowchart TD
A[GitHub Actions: images in GHCR] --> B[VPS: docker compose pull]
B --> C[deploy-rolling.sh<br/>nginx -t, migrate, roll backend then frontend,<br/>celery, recreate nginx]
C --> D{Docker health<br/>backend + frontend}
D -->|healthy| E[Smoke: image revision == expected SHA<br/>+ /api/health/, /api/settings/, /api/products/ = 200]
D -->|unhealthy| R[rollback.sh<br/>predeploy image, or previous vX.Y on production]
E --> F[Discord: success]
R --> G[Discord: failure, workflow red]
E -.->|staging only| K[staging-perf-gate.yml: k6 homepage]
E -.->|staging only| S[e2e-staging.yml: Playwright smoke]
One deploy, from image to notification; the dotted steps are the staging-only k6 gate and E2E smoke that run in their own workflows after the deploy.
Images are pushed to GHCR under the lowercase namespace
ghcr.io/voorman/webshop_freeze_design/{backend,frontend}.
Key Scripts¶
scripts/deploy-rolling.sh- Zero-downtime rolling deploymentscripts/rollback.sh- Automatic rollback on failurescripts/health-check.sh- Shared Docker health-status check (sourced by the workflow)scripts/notify-discord.sh- Discord notifications, also sent to ntfy (freeze-deploy) throughscripts/notify-ntfy.shsince v1.59
Staging Deployment¶
Trigger: Automatic on push to the staging branch, as the final deploy job of the CI workflow (after all tests pass)
Workflows:
.github/workflows/ci.yml— contains thedeployjob (uses: ./.github/workflows/deploy-staging.ymlviaworkflow_call), gated onbackend-test,frontend-test,frontend-build,e2e-smoke-testande2e-visual-test.github/workflows/deploy-staging.yml— the reusable deploy workflow itself (on: workflow_call+workflow_dispatch). A standalone "Deploy to Staging" run only exists for manual dispatch.
Flow¶
- Push to
stagingbranch triggers the CI workflow - Build images (
build-imagesjob,.github/workflows/build-images.yml) runs in parallel with the tests: backend and frontend images with the staging configuration, pushed to GHCR assha-{commit}only. The frontend is built withNEXT_PUBLIC_API_URLfrom the staging secrets, the backend withGIT_SHA(its Sentry release) and the OCIrevisionlabel - CI tests pass (linting, type checking, unit tests, E2E smoke + visual tests)
- Superseded guard: the deploy job reads the
revisionlabel of the backend image running on staging. If staging already runs a newer commit ofstaging, the deploy is skipped (notice in the run); the newest commit always wins - Tag the tested images as
staging(docker buildx imagetools create, seconds) - Copy deployment scripts, compose file and nginx config to the staging VPS
(
appleboy/scp-action:scripts/*.sh,docker-compose.staging.yml,nginx/nginx.staging.conf) - SSH to the staging VPS (
appleboy/ssh-action) and rundeploy-rolling.shwithDEPLOY_TAG=staging(rollback viarollback.shon failure) - Health checks verify backend and frontend via Docker health status (
scripts/health-check.sh) - Post-deploy functional smoke (see below) — deployed-SHA match + key endpoints
- Start the k6 gate: the deploy job dispatches
staging-perf-gate.ymland does not wait for it. A k6 failure never rolls back; it posts to Discord (DISCORD_WEBHOOK_URL) - Start the E2E smoke (#861): the deploy job dispatches
e2e-staging.ymlwithspec=e2e/staging-smoke.spec.tsand the deployedsha. Playwright runs home, product page and cart againsthttps://staging.freezedesign.nlbehind the basic auth (STAGING_BASIC_AUTH_BASE64), without traces or video. Red means a Discord message "staging (E2E-smoke)" (DISCORD_WEBHOOK_URL), never a rollback. The same workflow runs any spec by hand:gh workflow run e2e-staging.yml --ref staging -f spec=e2e/<naam>.spec.ts - Discord notification on success or failure (
DISCORD_WEBHOOK_URL)
Only one deploy runs on the staging server at a time (concurrency group staging-server);
a third queued run cancels the middle one.
Post-Deploy Functional Smoke¶
Before the k6 gate, the deploy-staging.yml workflow runs a fast functional smoke
(step "Post-deploy functional smoke (deployed-SHA + key endpoints)"):
- Deployed-SHA match — reads the OCI revision label from the image used by the
running backend container. It must equal the commit that was just pushed
(
github.sha); otherwise the deploy failed to roll the new code and the step fails. The public health response does not expose this deployment metadata. - Key endpoints return 200 —
GET /api/health/,GET /api/settings/, andGET /api/products/are each expected to return HTTP 200.
Only if both checks pass does the workflow count as a successful deploy and start the k6 gate and the E2E smoke.
Manual Intervention¶
Staging deployment is fully automatic. To manually redeploy the current staging tip (without re-running CI):
Promotion to main¶
Promote only after the retest on staging: Eefje has tested her feedback issues on staging and closed them, and Ivar has checked the todos on staging. Then promote staging to main with:
The script first confirms that the staging HEAD was deployed (it only counts the CI push run on staging; the promotion PR's own run has no deploy job). It then pauses for UAT and asks Promotie naar main starten? [y/N]; answer y only when that retest is done. After the confirmation it opens or reuses the promotion PR, waits for its checks, and then fast-forwards main to the PR's head SHA with a direct push. GitHub marks the PR as merged once main contains that commit. Nothing is rewritten, so main and staging keep the same SHAs; the script refuses if main has a commit that staging lacks. The staging branch has a protection rule (deletion and force-push forbidden); the restore and retarget steps in release.sh are safety nets.
Promoting to main does not deploy anything. Production deploys are a separate step (see below).
Monitoring¶
- GitHub Actions workflow run status (the
deployjob inside the CI run, or a manual "Deploy to Staging" run) - Discord notifications
- Health check logs in workflow output
Production Deployment¶
Trigger: scripts/deploy-production.sh, which tags origin/main and dispatches .github/workflows/deploy-production.yml with that tag.
Environment protection: the GitHub environment production has no required reviewers (checked on 2026-09-28: the deploy job started right after the build). The deliberate confirmation is the [y/N] prompt in the script.
Pre-Deployment Checklist¶
-
mainwas updated withscripts/release.sh --promote-only, so its SHA ran on staging first - The latest production backup is recent (hourly; check the
BackupRecordlist orfreeze-production-backups) - New migrations reviewed (the script lists them before it asks for confirmation)
- Changes to
backend/.env.productionare in place (containers only read it when they are created, which the deploy does) - New secrets from the "Voor Ivar" issue (label
voor-ivar) are set on production; see Voor Ivar
Deployment Steps¶
scripts/deploy-production.sh --dry-run # show what would happen
scripts/deploy-production.sh # next version: highest vX.Y tag + 1
scripts/deploy-production.sh --tag v1.28 # a specific version, or resume with an existing tag on main
- Version and target: the script picks the next
vX.Y(numeric:v1.9→v1.10) or takes--tag, and always tagsorigin/main, never the local HEAD (which usually sits onstagingand runs ahead ofmain). It refuses ifmainalready carries the latest tag, if the tag exists on another commit, or ifmainis not an ancestor ofstaging. - Overview and confirmation: previous and new tag, commit, commits since the previous tag, and new migrations; then
Productie uitrollen naar vX.Y? [y/N]. - Tag: annotated, with the commit list as message, pushed to
origin. - Workflow (
deploy-production.yml): - reuses the images that CI already built and tested for the tag commit (or, for a tag on a docs-only commit, for the last commit with code:
scripts/find-prebuilt-images.sh) and retags them aslatestandX.Y; only when nothing is found does it build from the tag itself (withGIT_SHAand the OCIrevisionlabel of the tag commit); - looks up the previous
vX.Ytag and passes it asPREVIOUS_VERSIONforrollback.sh; - copies the deployment scripts, compose file and nginx config to the VPS;
- runs
scripts/deploy-rolling.shwithPRE_DEPLOY_BACKUP=true: firstcheck --deploy --tag security --fail-level WARNING(stops on atest_Mollie key withoutALLOW_TEST_MOLLIE_KEY, or onE2E_TESTING=True), thensafe_migrate --dry-run(flags destructive migrations and makes a database backup, and stops if that backup fails), then the migrations and the rolling update; - verifies Docker health, then a smoke test that checks the running image's
revisionlabel against the tag and hits key endpoints; - rolls back automatically if the health check fails, and notifies Discord on start, success and failure.
- Follow: the script finds the run and follows it with
gh run watchuntil it finishes. On failure it prints the run URL.
If the tag was pushed but the workflow did not start, rerun with --tag vX.Y to resume; the script reuses a tag that already sits on main.
Post-Deployment Verification¶
ssh production-vps
cd /opt/webshop
# Running code: the revision label must be the tagged commit
docker image inspect ghcr.io/voorman/webshop_freeze_design/backend:latest \
--format '{{index .Config.Labels "org.opencontainers.image.revision"}}'
# Container health and migrations
docker compose -f docker-compose.prod.yml ps
docker compose -f docker-compose.prod.yml exec backend python manage.py showmigrations --plan | grep -c "\[ \]" # 0 = nothing pending
# Logs
docker compose -f docker-compose.prod.yml logs -f --tail=100
Then check Sentry (environment:production) for new errors.
Update the CHANGELOG¶
Each production tag has its own section in CHANGELOG.md: tag v1.28 is ## [1.28] - <date>. After a successful deploy, scripts/deploy-production.sh prints a reminder as long as main has no section for the new tag. Open a small PR to staging that renames ## [Unreleased] to ## [1.28] - <date> and puts an empty ## [Unreleased] above it. It reaches main with the next promotion.
Known Failure: DNS Lookup Error on the Runner¶
Copy deployment scripts to server can fail with dial tcp: lookup *** on 127.0.0.53:53: no such host while the server itself is reachable. That is a DNS problem on the GitHub runner. Rerun with scripts/deploy-production.sh --tag vX.Y. If it keeps failing, run scripts/deploy-rolling.sh on the VPS with the same environment as the workflow: COMPOSE_FILE=docker-compose.prod.yml, DEPLOYMENT_DIR=/opt/webshop, GITHUB_REPOSITORY, REGISTRY=ghcr.io, DEPLOY_TAG=latest and PRE_DEPLOY_BACKUP=true (without it the script skips check --deploy and safe_migrate --dry-run, including its backup).
Zero-Downtime Deployment Details¶
The deploy-rolling.sh script updates services one at a time with an automatic per-service rollback to the previous image.
How It Works¶
The script refuses to run without DEPLOY_TAG (staging or latest), so a rollback
reference can never silently point at the other environment's image.
- Pull images:
docker compose pullfetches the images forDEPLOY_TAGfrom GHCR - Test the nginx config:
nginx -truns in a throwaway container of the nginx service (docker compose run --rm --no-deps -T nginx nginx -t), with the config the workflow just copied, the real certificates and the freshly pulled image. If it fails, the deploy stops before migrations and container swaps, and the last config nginx ran with (nginx/nginx.conf.vorige-goed, ornginx/nginx.staging.conf.vorige-goedon staging) is copied back - Production checks (only with
PRE_DEPLOY_BACKUP=true, whichdeploy-production.ymlsets):manage.py check --deploy --tag security --fail-level WARNINGon the new image (stops on atest_Mollie key orE2E_TESTING=True), thensafe_migrate --dry-run(flags destructive migrations and makes a database backup; stops if that backup fails) - Migrations:
python manage.py migrate --noinputruns in a one-off backend container (run --rm) on the new image, before any container is swapped. A failing migration aborts the deploy with the old containers still running - For each service (backend, then frontend):
- Backup image: The currently running image is tagged locally as
predeploy - Recreate: Container is recreated with the new image (
up -d --no-deps --force-recreate) - Health check: Docker health status is polled (30 attempts, 5-second intervals = 150 seconds max)
- Rollback on failure: If the health check fails, the
predeployimage is restored and the container recreated again
- Backup image: The currently running image is tagged locally as
- Celery services: the owner of the
celerybeat_schedulevolume (/app/celerybeat) is set toappuserin a short root container (since #811: a root-owned volume kept beat restarting), thenceleryandcelery-beatare updated with a plainup -d(no wait for health; beat's own healthcheck shows up indocker compose psafter about a minute) - Nginx recreate: Nginx is recreated (
up -d --no-deps --force-recreate nginx) with the pinned image from the compose file (e.g.nginx:1.31.6-alpine), which also refreshes upstream DNS (new container IPs). A plain restart would keep the old image. Before that, the running image is tagged locally asnginx:predeploy. Once the new nginx is healthy, its config is saved as the.vorige-goedcopy for step 2. If it does not become healthy, thenginx:predeployimage and the.vorige-goedconfig are put back and the deploy fails
If a deploy stops with ✗ nginx-config is ongeldig, the log shows the nginx -t error. The running nginx is untouched and the previous config is back on disk (unless this was the first deploy with the check, then nothing could be restored). Fix the config in the repo and deploy again. scripts/rollback.sh does not touch nginx.
If a deploy fails with ✗ Nieuwe nginx is niet healthy, nginx runs on the previous image and config again (✓ nginx teruggezet op het vorige image). Locally, the compose image name (e.g. nginx:1.31.6-alpine) then points to that old image until the next docker compose pull. Backend and frontend are already on the new version at that point. To update nginx, change the version in both docker-compose.prod.yml and docker-compose.staging.yml.
Resource Requirements¶
- VPS capacity: during the roll two backend (or frontend) containers run side by side
for a moment; the
mem_limits indocker-compose.prod.ymladd up to about 3.75 GB on a 16 GB machine, see Hosting - Duration: ~2-5 minutes for full deployment
Health Check Details¶
- Mechanism: Docker's built-in health status (from the healthcheck definitions in the compose file), read via
docker compose ps --format json - Backend healthcheck:
curl -f http://localhost:8000/api/health/(database + Redis); frontend:wget --spider http://127.0.0.1:3000/; nginx:curl -f http://localhost/health - Max attempts: 30 attempts × 5 seconds = 150 seconds maximum wait per service
- The workflow-level verification step afterwards (
scripts/health-check.sh,check_docker_health) uses 10 attempts × 5 seconds per service
Celery Services¶
Celery worker and beat scheduler are recreated with a plain up -d (not rolled, the deploy
does not wait for their health) since they don't serve HTTP traffic. Both do have a Docker
healthcheck: the worker answers celery inspect ping, and beat (since #798) touches
/app/celerybeat/heartbeat on every tick; older than 10 minutes means unhealthy. A cron
job on each server restarts an unhealthy beat and reports it in Discord — see
Monitoring.
The worker command is celery -A config worker -l info --concurrency=2
--max-memory-per-child=262144 on production (--concurrency=1 on staging): a child above
256 MiB after a task is replaced. Production caps the service at mem_limit: 768m (#990).
A recreate of celery aborts a running task; the deploy accepts that (tasks are
acks_late, print files are regenerated by the hourly recover task).
Automatic Rollback¶
Trigger: Either deploy-rolling.sh fails on the VPS, or the workflow's health verification step (10 attempts × 5 seconds per service) fails afterwards
Script: scripts/rollback.sh
What Happens¶
- Deployment or health check fails
- Rollback script executes:
- Prefer the local
predeploybackup image (snapshot taken bydeploy-rolling.sh) - Productie: otherwise pull the previous version tag (
PREVIOUS_VERSION, e.g.1.38) from GHCR —deploy-production.ymlpasses it in. Version tags exist from 1.39 on. - Staging has no version tags: without a
predeployimage the rollback fails - Re-tag it as the deploy tag and recreate the services
- Discord notification sent with deployment failure status
- Workflow fails to signal deployment failure
Verification After Automatic Rollback¶
After automatic rollback, verify services are healthy (on the staging VPS):
ssh <user>@staging-vps
cd /opt/webshop
docker compose -f docker-compose.staging.yml ps # all services healthy?
curl -f https://staging.freezedesign.nl/api/health/
Investigate Failure¶
- Check deployment logs in GitHub Actions workflow output (the
deployjob in the CI run) - Check application logs:
- Identify root cause before attempting re-deployment
Manual Rollback Procedure¶
When to use:
- Automatic rollback failed
- Issues discovered after successful deployment
- Emergency rollback needed outside of deployment workflow
Step-by-Step¶
-
SSH into VPS:
-
Navigate to deployment directory:
-
Set environment variables:
Which image is restored:
- The local predeploy backup image on the VPS (the image that ran before the last deploy), if it still exists
- Productie only: a specific version with ROLLBACK_VERSION=<X.Y> (e.g. 1.38, without v). This takes precedence over predeploy. Version tags exist from 1.39 on; the matching git tags are vX.Y.
- Staging has no version tags (ROLLBACK_VERSION is refused with DEPLOY_TAG=staging)
-
Run rollback script:
-
Verify health:
-
Check logs:
Last Resort: Simple Restart¶
If rollback.sh fails completely:
This restarts services with the currently tagged images (may not fully revert if images were already pulled).
Database Migration Rollback¶
Django migrations can be reversible or irreversible.
Reversible Migrations¶
Most migrations are automatically reversible:
AddField- removes the fieldCreateModel- drops the tableAlterField- reverts field changesAddIndex- drops the index
To rollback a reversible migration:
# SSH to VPS
ssh <user>@staging-vps
cd /opt/webshop
# Rollback to specific migration number
docker compose -f docker-compose.staging.yml exec backend \
python manage.py migrate <app_name> <previous_migration_number>
# Example: rollback products app to migration 0015
docker compose -f docker-compose.staging.yml exec backend \
python manage.py migrate products 0015
Verify rollback:
# Show current migration status
docker compose -f docker-compose.staging.yml exec backend \
python manage.py showmigrations <app_name>
Irreversible Migrations¶
Some migrations cannot be automatically reversed:
RemoveField- data is lost when field is droppedDeleteModel- data is lost when table is droppedRunPython- custom Python code withoutreverse_codeRunSQL- custom SQL without reverse SQL
For irreversible migrations:
- Do not attempt
migraterollback (will fail or lose data) - Restore from database backup (see Disaster Recovery)
- Choose backup timestamp before the migration ran
Best Practices¶
- Deploy migrations separately:
- Release 1: Deploy migration only (additive, safe)
- Verify migration succeeded
-
Release 2: Deploy code that uses new schema
-
Make migrations reversible:
- Provide
reverse_codeforRunPythonoperations - Avoid
RemoveFieldandDeleteModelwhen possible -
Use
AlterFieldwithnull=Truebefore removing -
Test migrations on staging before production
-
Take backup before risky migrations:
Troubleshooting¶
Health Check Fails¶
Symptom: Deployment fails with "<service> failed health check after N attempts" or "<service> health check failed"
Diagnosis:
# Check logs
docker compose -f docker-compose.staging.yml logs backend --tail=200
docker compose -f docker-compose.staging.yml logs frontend --tail=200
# Check Docker health status
docker compose -f docker-compose.staging.yml ps
# Test health endpoint from inside the container
docker compose -f docker-compose.staging.yml exec backend \
curl -v http://127.0.0.1:8000/api/health/
Common causes:
- Database connection failure (check
DB_HOST,DB_PASSWORDin.env) - Redis connection failure (check
REDIS_URL) - Missing environment variables
- Application startup errors (check logs for stack traces)
- Port already in use (check
docker compose ps)
Container Won't Start¶
Symptom: Container exits immediately or restart loop
Diagnosis:
# Check container status
docker compose -f docker-compose.staging.yml ps
# Check resource usage
docker stats
# Inspect container
docker compose -f docker-compose.staging.yml logs <service-name> --tail=100
Common causes:
- Out of memory (check
docker stats) - Syntax error in code (check logs for Python/JavaScript errors)
- Missing dependencies (rebuild image)
- Configuration error in
docker-compose.staging.yml
Image Pull Fails¶
Symptom: "Error pulling image" during deployment
Diagnosis:
# Check GHCR authentication
docker login ghcr.io -u <username>
# Verify image exists
docker pull ghcr.io/<repo>/backend:<tag>
Common causes:
- Invalid tag name
- GHCR authentication expired
- Image not pushed to registry
- Network connectivity issues
Solution:
# Re-authenticate to GHCR
echo $GITHUB_TOKEN | docker login ghcr.io -u <username> --password-stdin
# Verify image tags
docker images | grep ghcr.io
Disk Space Issues¶
Symptom: "No space left on device"
Diagnosis:
Solution:
# Clean up old images and containers
docker system prune -f
# Remove unused volumes (careful!)
docker volume prune -f
# Remove specific old images
docker images | grep '<none>' | awk '{print $3}' | xargs docker rmi
Migration Fails¶
Symptom: "Migration failed" during deployment
Diagnosis:
# Check migration status
docker compose exec backend python manage.py showmigrations
# Check migration errors
docker compose logs backend | grep -i migration
Common causes:
- Database schema conflict
- Missing dependency migration
- Custom SQL error in migration
- Database connection interrupted
Solution:
# Fake migration if already applied manually
docker compose exec backend python manage.py migrate <app> <migration> --fake
# Or rollback and re-apply
docker compose exec backend python manage.py migrate <app> <previous-migration>
docker compose exec backend python manage.py migrate <app>
Performance Degradation After Deployment¶
Symptom: Application slower than before deployment
Diagnosis:
# Check resource usage
docker stats
# Check database connections
docker compose exec db psql -U $DB_USER -d $DB_NAME -c "SELECT count(*) FROM pg_stat_activity;"
# Check Redis memory
docker compose exec redis redis-cli INFO memory
Common causes:
- Missing database indexes (new queries added)
- Inefficient queries (check Django query logs)
- Memory leak (check
docker statsover time) - Cache not warming up (Redis empty after restart)
Solution:
- Review and optimize new queries
- Add database indexes for slow queries
- Restart services if memory leak suspected
- Warm up cache if needed
Related Documentation¶
- Disaster Recovery - Database backup, restore, RTO/RPO targets
- Architecture Overview - System architecture overview