Skip to content

Disaster Recovery

Procedures for backup, restore, and recovery of the Freeze Design webshop.

Environments

The procedures below target production: docker-compose.prod.yml in /opt/webshop on the production VPS. ssh production-vps stands for an SSH session as the deploy user on that VPS. For staging, use the staging VPS and docker-compose.staging.yml. Backups of both environments go to DigitalOcean Spaces (ams3): freeze-production-backups and freeze-staging-backups.

Overview

RPO and RTO Targets

Metric Target Actual
RPO (Recovery Point Objective) 1 hour 1 hour (hourly automated backups via Celery Beat)
RTO (Recovery Time Objective) 4 hours 20-35 minutes (database restore from S3)

Storage and Retention

Property Value
Backup storage DigitalOcean Spaces, S3-compatible (ams3): freeze-production-backups and freeze-staging-backups, from BACKUP_S3_BUCKET + BACKUP_S3_ENDPOINT_URL. With BACKUP_S3_ENDPOINT_URL unset the code targets AWS S3 instead
Encryption Application-level AES-256-GCM (12-byte nonce) when BACKUP_ENCRYPTION_KEY (base64 32-byte) is set. Server-side AES256 and Standard-IA are only requested against AWS S3; Spaces rejects those headers
Storage class Spaces default (Standard-IA only when targeting AWS S3)
Media files DigitalOcean Spaces (ams3, separate from DB backups); versioning on since 2026-09-28. Copy outside the bucket with scripts/backup-media.sh (rclone to a second bucket, 12-month archive of deleted/overwritten files), see Media Backup
Backup format Full PostgreSQL pg_dump, gzip compressed, SHA-256 checksum (MD5 also computed for compatibility)
Retention 24 hourly + 7 daily + 4 weekly (~35 backups maintained), daily cleanup at 03:00 Amsterdam time. Until 2026-09-28 the cleanup only saw the oldest 100 objects (list_backups() bug, fixed in #345); the production bucket then held 964 objects
Tracking Every backup recorded in the BackupRecord Django model

No AWS services

Backups go to DigitalOcean Spaces through boto3's S3 API; nothing runs on AWS. The database runs in a Docker container on the Hetzner VPS, media also lives on Spaces, and DNS for freezedesign.nl (apex, www, prod, staging) is managed at Openprovider and points straight at the Hetzner VPSes; nothing sits in front of the shop (freezedesign.nl and prod.freezedesign.nl redirect to www in nginx). Cloudflare only hosts the docs site (Cloudflare Pages).

Backup Architecture

Components

  1. Celery Beat Scheduler (backend/config/celery.py) -- triggers hourly backups at minute 0
  2. BackupService (backend/apps/core/services/backup_service.py) -- orchestrates the backup pipeline
  3. DigitalOcean Spaces (S3-compatible) -- encrypted backup storage; retention is applied by the daily cleanup task

Process Flow

flowchart TD
    A[Celery Beat<br/>crontab minute=0] -->|dispatches task| B[Celery Worker<br/>backup.daily_backup]
    B -->|calls BackupService| C[BackupService]
    C --> S1[1. pg_dump — full SQL dump of PostgreSQL]
    C --> S2[2. gzip -9 — compress ~9:1]
    C --> S3[3. Encrypt — AES-256-GCM if BACKUP_ENCRYPTION_KEY set]
    C --> S4[4. SHA-256 checksum]
    C --> S5[5. Upload to Spaces — S3 API]
    C --> S6[6. BackupRecord — Django model tracks metadata]
    C --> S7[7. Cleanup — remove temp files]
    C --> D[(Spaces bucket<br/>BACKUP_S3_BUCKET, AES-256-GCM)]

Data Flow Summary

  1. pg_dump creates a plain-format SQL dump (~100-500 MB uncompressed)
  2. gzip level 9 compresses to ~50-100 MB
  3. Encrypted with AES-256-GCM when BACKUP_ENCRYPTION_KEY is configured (is_encrypted + nonce recorded on the BackupRecord)
  4. SHA-256 checksum calculated for integrity verification (MD5 also computed for backward compatibility)
  5. Uploaded to S3 with metadata (timestamp, database name, size, checksum)
  6. Temporary files cleaned up
  7. Retention enforced by daily cleanup task

Automated Schedule

Celery Beat times are Europe/Amsterdam (CELERY_TIMEZONE), defined in backend/config/celery.py. The weekly CI test below runs in UTC (GitHub Actions).

Hourly Backup

  • Schedule: Every hour at minute 0 (crontab(minute=0))
  • Task: backup.daily_backup
  • Timeout: Expires after 55 minutes (must complete before next hourly run)
  • Action: Full database backup, compress, upload to S3
'hourly-database-backup': {
    'task': 'backup.daily_backup',
    'schedule': crontab(minute=0),
    'options': {'expires': 3300},
}

Daily Cleanup

  • Schedule: 03:00 Amsterdam time daily (crontab(hour=3, minute=0))
  • Task: backup.cleanup_old_backups
  • Action: Enforces tiered retention policy

Retention logic:

  • Keep last 24 hourly backups
  • Keep 1 per day for the last 7 days
  • Keep 1 per week for the last 4 weeks
  • Delete everything else

Daily Verification

  • Schedule: 04:00 Amsterdam time daily (crontab(hour=4, minute=0))
  • Task: backup.verify_latest_backup
  • Timeout: Expires after 30 minutes

Verification checks:

  • Backup file exists in S3
  • File size is reasonable (> 1 MB)
  • SHA-256 checksum matches stored metadata
  • Backup timestamp is recent (< 2 hours old)

Monthly Restore Test

  • Schedule: 1st of each month at 05:00 Amsterdam time (crontab(day_of_month=1, hour=5, minute=0))
  • Task: backup.test_restore
  • Timeout: Expires after 2 hours

Test procedure:

  1. Download latest backup from S3
  2. Create temporary test database
  3. Restore backup to test database
  4. Verify record counts match
  5. Clean up test database
  6. Report success/failure

Weekly CI Backup Test

  • Schedule: Sunday at 05:00 UTC
  • Workflow: .github/workflows/backup-test.yml
  • Environment: Fresh PostgreSQL 15 + Redis in GitHub Actions

CI test steps:

  1. Create test database with seed data (create_test_fixture)
  2. Export backup using export_seed_data
  3. Create PostgreSQL dump using pg_dump -F c
  4. Verify backup file size and contents
  5. Flush database
  6. Restore from backup (import_seed_data)
  7. Verify data integrity (record counts match original)
  8. Restore pg_dump to a separate database and verify
  9. Upload backup artifacts for review (retained 7 days)

Manual Backup Procedure

When to Use

  • Before risky deployments
  • Before database migrations (especially irreversible ones)
  • Before major data operations (bulk updates, deletions)
  • As part of disaster recovery testing

Using Django Management Command

# SSH to VPS
ssh production-vps
cd /opt/webshop

# Trigger manual backup (creates dump, compresses, uploads to S3)
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py backup_database

# Verify backup was created (check logs for S3 URL and backup info)
docker compose -f docker-compose.prod.yml logs backend | grep -i backup

Expected output:

Backup completed successfully: {
  'filename': 'backup_20260201_143022.sql.gz',
  's3_url': 's3://freeze-production-backups/backup_20260201_143022.sql.gz',
  'size_mb': 87.43,
  'checksum_sha256': 'a1b2c3d4...',
  'timestamp': '2026-02-01T14:30:22Z'
}

Using pg_dump Directly

For more control or when the Django management command is unavailable:

ssh production-vps
cd /opt/webshop

# Load database credentials (DB_USER / DB_NAME, same values as POSTGRES_USER / POSTGRES_DB)
set -a; source backend/.env.production; set +a

# Custom format (recommended -- supports parallel restore)
docker compose -f docker-compose.prod.yml exec db \
  pg_dump -U $DB_USER -d $DB_NAME -F c -f /tmp/manual_backup.dump

# Or plain SQL format (human-readable, works with psql)
docker compose -f docker-compose.prod.yml exec db \
  pg_dump -U $DB_USER -d $DB_NAME -F p -f /tmp/manual_backup.sql

# Copy backup out of container
docker cp $(docker compose -f docker-compose.prod.yml ps -q db):/tmp/manual_backup.dump ./manual_backup.dump

# Compress manually
gzip manual_backup.dump

Backup format reference:

Flag Format Use Case
-F c Custom (binary) Recommended. Supports parallel restore with pg_restore
-F p Plain (SQL text) Human-readable. Restore with psql
-F t Tar archive Alternative archive format

Restore Procedure

Prerequisites

  • SSH access to production VPS
  • AWS CLI installed (it talks to Spaces with --endpoint-url), or download with BackupService().download_backup() from a Django shell as in Step 2
  • Database credentials (from backend/.env.production on the VPS; load them with set -a; source backend/.env.production; set +a so $DB_USER, $DB_NAME and the BACKUP_S3_* values below are in your shell)
  • Backup filename or timestamp to restore (python manage.py backup_database --list shows the bucket; restore_database --latest takes the newest)

Step 1: Download Backup from S3

ssh production-vps
cd /opt/webshop

# The AWS CLI reads AWS_*; take the values from the backup settings in backend/.env.production
export AWS_ACCESS_KEY_ID="$BACKUP_S3_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$BACKUP_S3_SECRET_KEY"
export AWS_DEFAULT_REGION="ams3"
S3="aws s3 --endpoint-url https://ams3.digitaloceanspaces.com"

# List available backups (names sort oldest first, so the newest is last)
$S3 ls s3://freeze-production-backups/ | grep backup_ | tail -5

# Download specific backup
$S3 cp s3://freeze-production-backups/backup_20260928_140000.sql.gz /tmp/

# Verify download
ls -lh /tmp/backup_*.sql.gz

Step 2: Decompress Backup

Encrypted backups: restore_database decrypts them itself

When BACKUP_ENCRYPTION_KEY is set (it is, on production and staging), uploaded backups are AES-256-GCM encrypted and cannot be gunzip-ed directly. Since 2026-10-19 restore_database, verify_backup and the Celery task backup.test_restore all detect this (via the BackupRecord, the S3 metadata encrypted flag, or the file header) and decrypt the download before decompressing it. The only requirement is that BACKUP_ENCRYPTION_KEY in the backend container is the key the backup was made with; if it is missing the command stops with a message saying so, instead of a gzip error.

For the manual gunzip/psql route below, let the command do the download, checksum and decryption for you and pick up the plain .sql.gz it leaves behind:

docker compose -f docker-compose.prod.yml exec backend \
  python manage.py restore_database backup_YYYYMMDD_HHMMSS.sql.gz --dry-run --force
# → "Encrypted: yes, decrypted OK"; the decrypted file is now at
#   /tmp/db_restore/backup_YYYYMMDD_HHMMSS.sql.gz inside the backend container
# Decompress the gzip backup (a plain dump, or the file left by --dry-run above)
gunzip /tmp/backup_20260201_143022.sql.gz

# Verify decompressed file
ls -lh /tmp/backup_20260201_143022.sql

For custom-format backups (.dump files), skip decompression -- use directly with pg_restore.

Step 3: Stop Application

Stop services that write to the database to prevent conflicts during restore:

docker compose -f docker-compose.prod.yml stop backend celery celery-beat

# Verify services are stopped (db and redis should still be running)
docker compose -f docker-compose.prod.yml ps

Step 4: Restore to Database

For custom-format backups (.dump):

docker compose -f docker-compose.prod.yml exec db \
  pg_restore -U $DB_USER -d $DB_NAME --clean --if-exists /tmp/backup.dump

Flags:

  • --clean -- drops database objects before recreating them
  • --if-exists -- uses IF EXISTS when dropping to avoid errors on missing objects

For SQL-format backups (.sql):

# Copy backup into the database container
docker cp /tmp/backup_20260201_143022.sql \
  $(docker compose -f docker-compose.prod.yml ps -q db):/tmp/backup.sql

# Restore using psql
docker compose -f docker-compose.prod.yml exec db \
  psql -U $DB_USER -d $DB_NAME -f /tmp/backup.sql

Step 5: Run Pending Migrations

After restoring, the backup may be from a schema version behind the current code:

docker compose -f docker-compose.prod.yml exec backend \
  python manage.py migrate

# Verify all migrations are applied
docker compose -f docker-compose.prod.yml exec backend \
  python manage.py showmigrations

Step 6: Restart Services

docker compose -f docker-compose.prod.yml up -d

# Verify all services are healthy
docker compose -f docker-compose.prod.yml ps

Step 7: Verify Health

# Check backend health endpoint
curl -f http://127.0.0.1:8000/api/health/

# Check frontend
curl -f http://127.0.0.1:3000/

# Tail logs for errors
docker compose -f docker-compose.prod.yml logs -f --tail=50

Step 8: Verify Data Integrity

docker compose -f docker-compose.prod.yml exec backend python manage.py shell
from apps.products.models import Product, Color, Size
from apps.orders.models import Order
from django.contrib.auth import get_user_model

User = get_user_model()

print(f"Products: {Product.objects.count()}")
print(f"Colors: {Color.objects.count()}")
print(f"Sizes: {Size.objects.count()}")
print(f"Orders: {Order.objects.count()}")
print(f"Users: {User.objects.count()}")

Compare counts with expected values or pre-incident counts.

Disaster Recovery Scenarios

Scenario A: Database Corruption

Situation: Database tables corrupted due to disk failure, software bug, or hardware issue.

Symptoms:

  • Database queries return errors
  • Application crashes with database errors
  • Data appears inconsistent or missing

Estimated downtime: 20-35 minutes Data loss: Up to 1 hour (RPO)

Procedure:

  1. Identify corruption scope -- determine which tables or data are affected
  2. Select restore point -- choose the most recent backup from before corruption occurred
  3. Follow the full Restore Procedure (steps 1-8)
  4. Verify data integrity in the Django shell
  5. Resume operations and monitor logs

Scenario B: Complete VPS Failure

Situation: VPS hardware failure, cloud provider outage, or catastrophic system failure.

Symptoms:

  • VPS unreachable via SSH
  • Application completely down
  • No response from any service

Estimated downtime: 2-4 hours (within RTO target) Data loss: Up to 1 hour (RPO)

Procedure:

  1. Provision new VPS (30-60 min): a Hetzner VPS with at least 4 GB RAM (the memory limits in docker-compose.prod.yml add up to about 3.75 GB; production today is a CX43 with 16 GB) and the public key of the deploy laptop. scripts/provision-server.sh does the rest in two passes: the deploy user, packages, swap, ufw (22, 80, 443), fail2ban, Docker, /opt/webshop, the cron job for restart-unhealthy.sh and unattended upgrades; the second pass (--harden) closes root login and passwords. Run it from your workstation:

    scp scripts/provision-server.sh root@<IP>:/tmp/
    ssh root@<IP> "bash /tmp/provision-server.sh \"\$(cat ~/.ssh/freeze_prod.pub)\""
    ssh freeze-prod "whoami"          # the new key works (update ~/.ssh/config first)
    ssh root@<IP> "bash /tmp/provision-server.sh --harden \"\$(cat ~/.ssh/freeze_prod.pub)\""
    
  2. Deploy infrastructure (15-30 min). The deploy workflow only copies scripts/*.sh, the compose file and nginx/nginx.conf; the rest is restored by hand from the repo and the password manager:

    ssh freeze-prod
    cd /opt/webshop
    
    # Files from the repo (checked out on your workstation): compose file, nginx config,
    # redis config and the scripts
    # scp docker-compose.prod.yml nginx/nginx.conf redis/redis.conf scripts/*.sh ... (keep the paths: nginx/, redis/, scripts/)
    
    # Secrets from the password manager (never in git): the env file and the basic-auth
    # file that nginx still mounts
    nano backend/.env.production          # chmod 600
    htpasswd -c nginx/.htpasswd <user>    # apache2-utils is installed by provisioning
    
    # Self-signed certificate for PostgreSQL (mounted from ./certs)
    bash scripts/generate-pg-certs.sh
    
    # Pull Docker images from GHCR (IMAGE_TAG defaults to latest; or X.Y for a specific release)
    docker compose -f docker-compose.prod.yml pull
    
    # Start infrastructure services first
    docker compose -f docker-compose.prod.yml up -d db redis
    
  3. Create the database (the container does not auto-create it):

    docker compose -f docker-compose.prod.yml exec db \
      psql -U $DB_USER -d postgres -c "CREATE DATABASE $DB_NAME;"
    
  4. Restore database from S3 backup (20-35 min) -- follow Restore Procedure steps 1-5

  5. Start application services:

    docker compose -f docker-compose.prod.yml up -d
    
  6. Configure DNS (5-30 min): at Openprovider, point the A records of www, the apex and prod to the new VPS IP. Check propagation:

    dig @8.8.8.8 www.freezedesign.nl A +short
    
  7. Configure SSL (5-10 min): issue the certificate with bash nginx/certbot-init.sh --alt www.freezedesign.nl --alt prod.freezedesign.nl (defaults: domain and lineage freezedesign.nl, docker-compose.prod.yml, webroot challenge; --staging first to test against the Let's Encrypt staging CA). Then install the renewal cron (see SSL Renewal → "Initial install") so the certificate does not silently expire 90 days later. The secrets PRODUCTION_HOST, PRODUCTION_USER and PRODUCTION_SSH_KEY of the GitHub environment production must point at the new server for the workflows (deploy-production.yml, ssl-renewal.yml) to reach it.

  8. Restore the other cron jobs: restart-unhealthy was installed by provisioning (/etc/cron.d/webshop-restart-unhealthy); the media backup needs rclone and its cron again (see Media Backup).

  9. Verify functionality: Full smoke test -- health endpoints, product listing, checkout flow, and docker compose -f docker-compose.prod.yml ps with every service healthy (celery-beat included). The next hourly backup should appear in #backups.

Downtime breakdown:

Step Time
VPS provisioning 30-60 min
Infrastructure deployment 15-30 min
Database restore 20-35 min
DNS propagation 5-30 min
SSL certificate 5-10 min
Verification 10-15 min
Total ~2-4 hours

Scenario C: Accidental Data Deletion

Situation: User data, orders, or products accidentally deleted (bulk delete, admin error, script bug).

Symptoms:

  • Reports of missing data
  • Empty tables or reduced record counts
  • User complaints about lost orders or designs

Estimated downtime: 15-35 minutes Data loss: Minimal (data between last backup and deletion)

Procedure:

  1. Identify deletion timestamp -- check admin logs, application logs, or user reports
  2. Stop application to prevent further changes:

    docker compose -f docker-compose.prod.yml stop backend celery celery-beat
    
  3. Select backup from before the deletion timestamp

  4. Choose restore approach:

    Option A -- Selective restore (if only specific tables affected):

    # Download and decompress backup, then restore specific tables only
    docker compose -f docker-compose.prod.yml exec db \
      pg_restore -U $DB_USER -d $DB_NAME \
        -t orders_order -t orders_orderitem \
        --data-only \
        /tmp/backup.dump
    

    Option B -- Full restore (if deletion scope is unclear):

    Follow the complete Restore Procedure.

  5. Verify that deleted data is restored

  6. Restart services

Scenario D: Failed Database Migration

Situation: Database migration fails partway through or causes data corruption.

Symptoms:

  • migrate command fails with an error
  • Application errors after migration
  • Data inconsistencies after schema change

Estimated downtime: 20-35 minutes (restore) + time to fix migration Data loss: Depends on approach

Procedure:

  1. Attempt migration rollback (if the migration is reversible):

    docker compose -f docker-compose.prod.yml exec backend \
      python manage.py migrate <app_name> <previous_migration_number>
    
  2. If rollback fails or migration is irreversible:

    • Stop the application
    • Restore from backup taken before the migration (this is why you take a manual backup before risky migrations)
    • Follow the full Restore Procedure
    • Verify schema and data integrity
  3. Fix the migration -- correct the migration code, test in a local or staging environment

  4. Re-apply the fixed migration

Prevention:

  • Always take a manual backup before irreversible migrations
  • Test migrations on staging first
  • Deploy migrations separately from application code changes
  • Write migrations to be reversible when possible

Monitoring and Alerting

Backup Task Logs

All backup tasks log to the apps.core.backup logger.

# View recent backup task logs
docker compose -f docker-compose.prod.yml logs celery | grep -i backup

# Follow backup logs in real time
docker compose -f docker-compose.prod.yml logs -f celery

Success output:

INFO apps.core.backup: Backup completed successfully: {
  'filename': 'backup_20260201_143022.sql.gz',
  's3_url': 's3://freeze-production-backups/backup_20260201_143022.sql.gz',
  'size_mb': 87.43, ...
}

Failure output:

ERROR apps.core.backup: Backup failed: pg_dump failed: ...
ERROR apps.core.backup: S3 upload failed: ...

Known failure cause until v1.54

The 219 backup failures up to 2026-09-26 and the ones of October 2026 had one cause (#992): apps/core/instrumentation.py installed a SlowQueryLogger wrapper on every database reconnect, so a long-lived Celery child stacked hundreds of them and every query ended in RecursionError: maximum recursion depth exceeded. The backup itself had already been uploaded; the BackupRecord insert failed, so the task reported a failure. Each redeploy reset the stack and hid it for a while. Fixed in v1.54 (one wrapper per connection); a RecursionError in #backups today is a new problem.

Discord Notifications

Backup task failures trigger Discord notifications to the #backups channel. Two mechanisms are involved:

  • Celery backup tasks notify via the Django DiscordService (backend/apps/core/services/backup_notifications.py, webhook env var DISCORD_WEBHOOK_BACKUPS, legacy BACKUP_DISCORD_WEBHOOK).
  • The weekly CI backup test notifies via scripts/notify-discord.sh (secret DISCORD_WEBHOOK_STAGING_BACKUPS, falling back to DISCORD_WEBHOOK_URL).

Since v1.59 both also reach ntfy: DiscordService forwards to topic freeze-backups (a red message also goes by e-mail to NOTIFY_FALLBACK_EMAIL), and notify-discord.sh to freeze-deploy. See ntfy notifications.

Daily Verification Task

Runs at 04:00 Amsterdam time every day. Checks:

  • Latest backup exists in S3
  • File size > 1 MB
  • Checksum matches metadata
  • Backup is less than 2 hours old

Failures are logged and should trigger manual investigation.

Weekly CI Backup Test

Runs every Sunday at 05:00 UTC in GitHub Actions (.github/workflows/backup-test.yml). Performs a full backup/restore cycle and reports results via GitHub Actions summary and Discord notification.

Health Check Commands

# List recent backups in Spaces (credentials as in "Step 1: Download Backup")
aws s3 --endpoint-url https://ams3.digitaloceanspaces.com ls s3://freeze-production-backups/ | tail -5

# Check the latest backup timestamp
aws s3 --endpoint-url https://ams3.digitaloceanspaces.com ls s3://freeze-production-backups/ | tail -1

# Check Celery Beat is alive (healthy = heartbeat < 10 min old) and scheduling tasks
docker compose -f docker-compose.prod.yml ps celery-beat
docker compose -f docker-compose.prod.yml logs celery-beat | tail -20

# Check recent backup task results
docker compose -f docker-compose.prod.yml logs celery | grep -i backup | tail -20

RTO/RPO Evidence

RPO: 1 Hour

  • Mechanism: Hourly automated backups via Celery Beat (crontab(minute=0))
  • Maximum data loss: 1 hour (worst case: failure occurs 1 minute before next hourly backup)
  • Verification: Daily verification task confirms latest backup is < 2 hours old

RTO: 20-35 Minutes (Actual)

Step Time Estimate Notes
Identify issue 5 min Sentry e-mail or Discord notification; no external uptime monitor yet (todo externe-uptime-monitor), so a dead server is noticed by hand
Download backup from S3 2-5 min ~50-100 MB compressed
Decompress backup 1-2 min gzip decompression
Stop application 1 min docker compose stop
Restore database (pg_restore) 5-15 min Depends on database size (100-500 MB)
Run migrations 1-2 min Usually fast (idempotent)
Restart services 2-3 min docker compose up -d
Health verification 2-5 min Smoke tests, check logs
Total 20-35 min Well within 4-hour target

Continuous Verification

Frequency Check Source
Hourly Backup creation and S3 upload Celery Beat task
Daily Verification of latest backup integrity backup.verify_latest_backup task
Weekly Full backup/restore cycle in CI .github/workflows/backup-test.yml
Monthly Automated restore test to temporary database backup.test_restore task

Media Backup

The database backups above do not contain the uploaded media (product photos, designs, fonts): those live in the media bucket (freeze-production-media, freeze-staging-media), and a database restore only brings back the file paths. Two layers protect the bytes:

Layer What it covers Retention Where it is configured
Spaces versioning on the media bucket An overwritten or deleted object stays retrievable as an old version 30 days once the lifecycle rule is set (NoncurrentVersionExpiration), see .planning/todos/pending/backup-versies-lifecycle.md Bucket setting, by Ivar with a full-access key
Copy in a second bucket (scripts/backup-media.sh) The whole bucket, outside the bucket the webshop writes to; survives an emptied bucket, a leaked key or a regional outage huidig/ mirrors the source; archief/<date>/ keeps what was deleted or overwritten for 12 months (MEDIA_BACKUP_ARCHIVE_DAYS=365) Cron on the VPS, by Ivar; the script is in the repo

How the copy works

One run of backup-media.sh (exit 0 = complete, 1 = sync failed or copy differs, 2 = usage error; details and variables in scripts/backup/README.md):

  1. rclone sync <source> <target>/huidig/ --backup-dir <target>/archief/<YYYY-MM-DD>/ --checksum --fast-list
  2. Delete archief/ folders older than 365 days
  3. Compare object count and bytes of the source with huidig/

Setting it up on the VPS (Ivar)

The script lives at scripts/backup-media.sh (top level, not in scripts/backup/) because the deploy workflows copy only scripts/*.sh to /opt/webshop on the VPS. Staging gets it with the next staging deploy; production only with the next release. To run it on a server before that, copy it by hand: scp scripts/backup-media.sh <vps>:/tmp/ and sudo install -m 755 /tmp/backup-media.sh /usr/local/sbin/backup-media.sh.

# 1. Second bucket with its own key: e.g. freeze-production-media-backup, a key
#    that can read the media bucket and read/write/delete the new one (rclone
#    moves files into archief/ and purges old archive folders). Never commit keys.
#    The production backup bucket is in sfo3 (set up 2026-10-17), so the copy
#    also survives an ams3 outage; staging's is in ams3.

# 2. rclone on the VPS: install, then write /root/.config/rclone/rclone.conf
#    (chmod 600) by hand rather than via `rclone config create`, which would
#    leave the secret in the shell history. Each remote has its own endpoint:
#    [spaces]  type = s3, provider = DigitalOcean, endpoint = ams3.digitaloceanspaces.com, acl = private
#    [backup]  same, endpoint = <region of the backup bucket>.digitaloceanspaces.com,
#              plus no_check_bucket = true (a per-bucket key may not create buckets;
#              without it the first write fails with 403).
#    With per-bucket keys `rclone lsd backup:` (all buckets) fails; always use
#    the full bucket path. "directory not found" on the bucket means wrong
#    name or wrong region endpoint.
sudo -v ; curl -fsSL https://rclone.org/install.sh | sudo bash
sudo rclone size spaces:freeze-production-media            # source readable
sudo rclone lsd  backup:freeze-production-media-backup     # target reachable

# 3. First run, dry first
sudo MEDIA_BACKUP_SOURCE=spaces:freeze-production-media \
     MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup \
     /opt/webshop/scripts/backup-media.sh --dry-run
sudo MEDIA_BACKUP_SOURCE=spaces:freeze-production-media \
     MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup \
     /opt/webshop/scripts/backup-media.sh

# 4. Nightly at 02:30 UTC. The VPS runs in UTC; the database cleanup is a Celery Beat
#    task at 03:00 Amsterdam time (02:00 UTC in winter, 01:00 UTC in summer), so the
#    media backup runs after it, not before (todo media-backup-tijd-na-database-cleanup)
sudo crontab -l 2>/dev/null > /tmp/crontab.bak || true
echo "30 2 * * * MEDIA_BACKUP_SOURCE=spaces:freeze-production-media MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup /opt/webshop/scripts/backup-media.sh >> /var/log/backup-media.log 2>&1" | sudo tee -a /tmp/crontab.bak
sudo crontab /tmp/crontab.bak

Check the first copy with rclone size backup:freeze-production-media-backup/huidig against rclone size spaces:freeze-production-media (the script prints both), and browse archief/ after the first day on which something was deleted or replaced.

Restoring media

# One file that was deleted or overwritten on <date>
rclone copy backup:freeze-production-media-backup/archief/<date>/media/products/foto.png spaces:freeze-production-media/media/products/

# The whole bucket after a disaster (into an empty or new media bucket)
rclone sync backup:freeze-production-media-backup/huidig spaces:freeze-production-media --checksum --fast-list

Within 30 days an overwritten object can also be restored from its Spaces version (aws s3api list-object-versions + copy-object ... ?versionId=), see the lifecycle todo above.

SSL Renewal

Let's Encrypt certificates expire every 90 days. The staging certificate expired silently on 2026-05-07 because no renewal automation was in place — this section documents the layered automation that prevents a repeat. Let's Encrypt no longer sends expiry e-mails (since June 2025), so the external monitor below is the only warning if both layers fail.

Environments

DOMAIN is the certbot lineage (the directory under /etc/letsencrypt/live/), not necessarily a hostname. renew-ssl.sh refuses to run without it.

Environment SSH alias DOMAIN (lineage) Names in the certificate COMPOSE_FILE
staging freeze-staging staging.freezedesign.eu staging.freezedesign.nl /opt/webshop/docker-compose.staging.yml
production freeze-prod freezedesign.nl freezedesign.nl, www.freezedesign.nl, prod.freezedesign.nl /opt/webshop/docker-compose.prod.yml

The staging lineage still carries the old .eu name; certbot renews it under that name, so leave DOMAIN as is.

Two-layer automation

The same script (scripts/renew-ssl.sh) runs from two independent triggers so a failure in one layer doesn't cause an outage:

  1. GitHub Actions workflow (.github/workflows/ssl-renewal.yml)
  2. Schedule: every Wednesday 04:00 UTC + manual workflow_dispatch (option force passes --force-renewal)
  3. One matrix job per environment, run with environment: staging / production, so it reads the STAGING_* / PRODUCTION_* SSH secrets from that GitHub environment (#1025; before that the job looked for repo secrets, found none and skipped both environments with only a notice). An environment whose *_HOST secret is still missing is skipped the same way. The schedule runs from main, so the deployment policy of both environments must allow main
  4. Discord notification (success or failure) via DISCORD_WEBHOOK_CICD

  5. VPS crontab (root)

  6. Schedule: weekly safety-net (Sundays 03:30 UTC)
  7. Same scripts/renew-ssl.sh invocation, with DOMAIN and COMPOSE_FILE set
  8. Survives GitHub-Actions outages

Certbot itself is idempotent — it skips renewal until the cert is within the 30-day window — so running both layers is safe; whichever fires first wins, the other is a no-op.

renew-ssl.sh always passes --preferred-challenges http-01. The production certificate was extended on 2026-09-28 via a manual DNS-01 challenge, which leaves pref_challs = dns-01 in its renewal file; without the flag certbot renew --webroot fails with "None of the preferred challenges are supported by the selected plugin". Extra certbot options go in CERTBOT_EXTRA_FLAGS (space-separated).

Dry run (after any change to the certificate or nginx)

Tests the full renewal against the Let's Encrypt staging CA without touching the live certificate:

ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
    COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml \
    CERTBOT_EXTRA_FLAGS="--dry-run --no-random-sleep-on-renew --cert-name freezedesign.nl" \
    bash scripts/renew-ssl.sh'

Expected: Congratulations, all simulated renewals succeeded and RESULT: skipped. Without --no-random-sleep-on-renew certbot first waits a random delay of up to 8 minutes (it does so for every non-interactive renew), which looks like a hang. The scheduled runs keep that delay on purpose; the workflow timeout (15 min) allows it.

Manual renewal

If both automations are down:

ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
    COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh'

The script reloads nginx automatically when a new certificate was installed. Use the staging values from the table for staging.

Initial install (after first VPS provisioning OR after disaster recovery)

# 1. Install the cron safety-net (production; use the staging values on staging)
sudo crontab -l 2>/dev/null > /tmp/crontab.bak || true
echo "30 3 * * 0 DOMAIN=freezedesign.nl COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml /opt/webshop/scripts/renew-ssl.sh >> /var/log/ssl-renewal.log 2>&1" | sudo tee -a /tmp/crontab.bak
sudo crontab /tmp/crontab.bak

# 2. Verify
sudo crontab -l | grep renew-ssl

# 3. The GitHub Actions side is automatic once the workflow file is on main and the
#    SSH secrets exist as environment secrets of the GitHub environment `production`
#    (PRODUCTION_HOST, PRODUCTION_USER, PRODUCTION_SSH_KEY; `staging` holds STAGING_*).
#    Check with a manual run of "SSL Renewal": a skipped leg prints
#    "<env>: secret <NAME> is niet gezet — overgeslagen".

A crontab line without DOMAIN does nothing: the script exits with code 2.

Health check before expiry

From any machine:

for h in www.freezedesign.nl freezedesign.nl prod.freezedesign.nl staging.freezedesign.nl; do
  printf '%-26s ' "$h"
  echo | openssl s_client -connect "$h:443" -servername "$h" 2>/dev/null \
    | openssl x509 -noout -enddate
done

Or with the same script the workflow uses (exit 1 below MIN_DAYS, default 14):

bash scripts/check-ssl-expiry.sh www.freezedesign.nl freezedesign.nl prod.freezedesign.nl staging.freezedesign.nl

ssl-renewal.yml runs this weekly as job expiry-check after the renewal jobs (also when they failed) and posts to Discord when a certificate expires within 14 days or a host returns no certificate. It runs on GitHub, so for a fully independent warning also add an external monitor (UptimeRobot or similar) on https://www.freezedesign.nl/api/health/ and https://staging.freezedesign.nl/api/health/, alerting at least 14 days before expiry.