Disaster Recovery¶
Procedures for backup, restore, and recovery of the Freeze Design webshop.
Environments
The procedures below target production: docker-compose.prod.yml in /opt/webshop
on the production VPS. ssh production-vps stands for an SSH session as the deploy
user on that VPS. For staging, use the staging VPS and docker-compose.staging.yml.
Backups of both environments go to DigitalOcean Spaces (ams3):
freeze-production-backups and freeze-staging-backups.
Overview¶
RPO and RTO Targets¶
| Metric | Target | Actual |
|---|---|---|
| RPO (Recovery Point Objective) | 1 hour | 1 hour (hourly automated backups via Celery Beat) |
| RTO (Recovery Time Objective) | 4 hours | 20-35 minutes (database restore from S3) |
Storage and Retention¶
| Property | Value |
|---|---|
| Backup storage | DigitalOcean Spaces, S3-compatible (ams3): freeze-production-backups and freeze-staging-backups, from BACKUP_S3_BUCKET + BACKUP_S3_ENDPOINT_URL. With BACKUP_S3_ENDPOINT_URL unset the code targets AWS S3 instead |
| Encryption | Application-level AES-256-GCM (12-byte nonce) when BACKUP_ENCRYPTION_KEY (base64 32-byte) is set. Server-side AES256 and Standard-IA are only requested against AWS S3; Spaces rejects those headers |
| Storage class | Spaces default (Standard-IA only when targeting AWS S3) |
| Media files | DigitalOcean Spaces (ams3, separate from DB backups); versioning on since 2026-09-28. Copy outside the bucket with scripts/backup-media.sh (rclone to a second bucket, 12-month archive of deleted/overwritten files), see Media Backup |
| Backup format | Full PostgreSQL pg_dump, gzip compressed, SHA-256 checksum (MD5 also computed for compatibility) |
| Retention | 24 hourly + 7 daily + 4 weekly (~35 backups maintained), daily cleanup at 03:00 Amsterdam time. Until 2026-09-28 the cleanup only saw the oldest 100 objects (list_backups() bug, fixed in #345); the production bucket then held 964 objects |
| Tracking | Every backup recorded in the BackupRecord Django model |
No AWS services
Backups go to DigitalOcean Spaces through boto3's S3 API; nothing runs on AWS. The
database runs in a Docker container on the Hetzner VPS, media also lives on Spaces,
and DNS for freezedesign.nl (apex, www, prod, staging) is managed at
Openprovider and points straight at the Hetzner VPSes; nothing sits in front of the
shop (freezedesign.nl and prod.freezedesign.nl redirect to www in nginx).
Cloudflare only hosts the docs site (Cloudflare Pages).
Backup Architecture¶
Components¶
- Celery Beat Scheduler (
backend/config/celery.py) -- triggers hourly backups at minute 0 - BackupService (
backend/apps/core/services/backup_service.py) -- orchestrates the backup pipeline - DigitalOcean Spaces (S3-compatible) -- encrypted backup storage; retention is applied by the daily cleanup task
Process Flow¶
flowchart TD
A[Celery Beat<br/>crontab minute=0] -->|dispatches task| B[Celery Worker<br/>backup.daily_backup]
B -->|calls BackupService| C[BackupService]
C --> S1[1. pg_dump — full SQL dump of PostgreSQL]
C --> S2[2. gzip -9 — compress ~9:1]
C --> S3[3. Encrypt — AES-256-GCM if BACKUP_ENCRYPTION_KEY set]
C --> S4[4. SHA-256 checksum]
C --> S5[5. Upload to Spaces — S3 API]
C --> S6[6. BackupRecord — Django model tracks metadata]
C --> S7[7. Cleanup — remove temp files]
C --> D[(Spaces bucket<br/>BACKUP_S3_BUCKET, AES-256-GCM)]
Data Flow Summary¶
pg_dumpcreates a plain-format SQL dump (~100-500 MB uncompressed)gziplevel 9 compresses to ~50-100 MB- Encrypted with AES-256-GCM when
BACKUP_ENCRYPTION_KEYis configured (is_encrypted+ nonce recorded on theBackupRecord) - SHA-256 checksum calculated for integrity verification (MD5 also computed for backward compatibility)
- Uploaded to S3 with metadata (timestamp, database name, size, checksum)
- Temporary files cleaned up
- Retention enforced by daily cleanup task
Automated Schedule¶
Celery Beat times are Europe/Amsterdam (CELERY_TIMEZONE), defined in backend/config/celery.py. The weekly CI test below runs in UTC (GitHub Actions).
Hourly Backup¶
- Schedule: Every hour at minute 0 (
crontab(minute=0)) - Task:
backup.daily_backup - Timeout: Expires after 55 minutes (must complete before next hourly run)
- Action: Full database backup, compress, upload to S3
'hourly-database-backup': {
'task': 'backup.daily_backup',
'schedule': crontab(minute=0),
'options': {'expires': 3300},
}
Daily Cleanup¶
- Schedule: 03:00 Amsterdam time daily (
crontab(hour=3, minute=0)) - Task:
backup.cleanup_old_backups - Action: Enforces tiered retention policy
Retention logic:
- Keep last 24 hourly backups
- Keep 1 per day for the last 7 days
- Keep 1 per week for the last 4 weeks
- Delete everything else
Daily Verification¶
- Schedule: 04:00 Amsterdam time daily (
crontab(hour=4, minute=0)) - Task:
backup.verify_latest_backup - Timeout: Expires after 30 minutes
Verification checks:
- Backup file exists in S3
- File size is reasonable (> 1 MB)
- SHA-256 checksum matches stored metadata
- Backup timestamp is recent (< 2 hours old)
Monthly Restore Test¶
- Schedule: 1st of each month at 05:00 Amsterdam time (
crontab(day_of_month=1, hour=5, minute=0)) - Task:
backup.test_restore - Timeout: Expires after 2 hours
Test procedure:
- Download latest backup from S3
- Create temporary test database
- Restore backup to test database
- Verify record counts match
- Clean up test database
- Report success/failure
Weekly CI Backup Test¶
- Schedule: Sunday at 05:00 UTC
- Workflow:
.github/workflows/backup-test.yml - Environment: Fresh PostgreSQL 15 + Redis in GitHub Actions
CI test steps:
- Create test database with seed data (
create_test_fixture) - Export backup using
export_seed_data - Create PostgreSQL dump using
pg_dump -F c - Verify backup file size and contents
- Flush database
- Restore from backup (
import_seed_data) - Verify data integrity (record counts match original)
- Restore
pg_dumpto a separate database and verify - Upload backup artifacts for review (retained 7 days)
Manual Backup Procedure¶
When to Use¶
- Before risky deployments
- Before database migrations (especially irreversible ones)
- Before major data operations (bulk updates, deletions)
- As part of disaster recovery testing
Using Django Management Command¶
# SSH to VPS
ssh production-vps
cd /opt/webshop
# Trigger manual backup (creates dump, compresses, uploads to S3)
docker compose -f docker-compose.prod.yml exec backend \
python manage.py backup_database
# Verify backup was created (check logs for S3 URL and backup info)
docker compose -f docker-compose.prod.yml logs backend | grep -i backup
Expected output:
Backup completed successfully: {
'filename': 'backup_20260201_143022.sql.gz',
's3_url': 's3://freeze-production-backups/backup_20260201_143022.sql.gz',
'size_mb': 87.43,
'checksum_sha256': 'a1b2c3d4...',
'timestamp': '2026-02-01T14:30:22Z'
}
Using pg_dump Directly¶
For more control or when the Django management command is unavailable:
ssh production-vps
cd /opt/webshop
# Load database credentials (DB_USER / DB_NAME, same values as POSTGRES_USER / POSTGRES_DB)
set -a; source backend/.env.production; set +a
# Custom format (recommended -- supports parallel restore)
docker compose -f docker-compose.prod.yml exec db \
pg_dump -U $DB_USER -d $DB_NAME -F c -f /tmp/manual_backup.dump
# Or plain SQL format (human-readable, works with psql)
docker compose -f docker-compose.prod.yml exec db \
pg_dump -U $DB_USER -d $DB_NAME -F p -f /tmp/manual_backup.sql
# Copy backup out of container
docker cp $(docker compose -f docker-compose.prod.yml ps -q db):/tmp/manual_backup.dump ./manual_backup.dump
# Compress manually
gzip manual_backup.dump
Backup format reference:
| Flag | Format | Use Case |
|---|---|---|
-F c |
Custom (binary) | Recommended. Supports parallel restore with pg_restore |
-F p |
Plain (SQL text) | Human-readable. Restore with psql |
-F t |
Tar archive | Alternative archive format |
Restore Procedure¶
Prerequisites¶
- SSH access to production VPS
- AWS CLI installed (it talks to Spaces with
--endpoint-url), or download withBackupService().download_backup()from a Django shell as in Step 2 - Database credentials (from
backend/.env.productionon the VPS; load them withset -a; source backend/.env.production; set +aso$DB_USER,$DB_NAMEand theBACKUP_S3_*values below are in your shell) - Backup filename or timestamp to restore (
python manage.py backup_database --listshows the bucket;restore_database --latesttakes the newest)
Step 1: Download Backup from S3¶
ssh production-vps
cd /opt/webshop
# The AWS CLI reads AWS_*; take the values from the backup settings in backend/.env.production
export AWS_ACCESS_KEY_ID="$BACKUP_S3_ACCESS_KEY"
export AWS_SECRET_ACCESS_KEY="$BACKUP_S3_SECRET_KEY"
export AWS_DEFAULT_REGION="ams3"
S3="aws s3 --endpoint-url https://ams3.digitaloceanspaces.com"
# List available backups (names sort oldest first, so the newest is last)
$S3 ls s3://freeze-production-backups/ | grep backup_ | tail -5
# Download specific backup
$S3 cp s3://freeze-production-backups/backup_20260928_140000.sql.gz /tmp/
# Verify download
ls -lh /tmp/backup_*.sql.gz
Step 2: Decompress Backup¶
Encrypted backups: restore_database decrypts them itself
When BACKUP_ENCRYPTION_KEY is set (it is, on production and staging), uploaded
backups are AES-256-GCM encrypted and cannot be gunzip-ed directly. Since
2026-10-19 restore_database, verify_backup and the Celery task
backup.test_restore all detect this (via the BackupRecord, the S3 metadata
encrypted flag, or the file header) and decrypt the download before
decompressing it. The only requirement is that BACKUP_ENCRYPTION_KEY in the
backend container is the key the backup was made with; if it is missing the
command stops with a message saying so, instead of a gzip error.
For the manual gunzip/psql route below, let the command do the download,
checksum and decryption for you and pick up the plain .sql.gz it leaves behind:
# Decompress the gzip backup (a plain dump, or the file left by --dry-run above)
gunzip /tmp/backup_20260201_143022.sql.gz
# Verify decompressed file
ls -lh /tmp/backup_20260201_143022.sql
For custom-format backups (.dump files), skip decompression -- use directly with pg_restore.
Step 3: Stop Application¶
Stop services that write to the database to prevent conflicts during restore:
docker compose -f docker-compose.prod.yml stop backend celery celery-beat
# Verify services are stopped (db and redis should still be running)
docker compose -f docker-compose.prod.yml ps
Step 4: Restore to Database¶
For custom-format backups (.dump):
docker compose -f docker-compose.prod.yml exec db \
pg_restore -U $DB_USER -d $DB_NAME --clean --if-exists /tmp/backup.dump
Flags:
--clean-- drops database objects before recreating them--if-exists-- usesIF EXISTSwhen dropping to avoid errors on missing objects
For SQL-format backups (.sql):
# Copy backup into the database container
docker cp /tmp/backup_20260201_143022.sql \
$(docker compose -f docker-compose.prod.yml ps -q db):/tmp/backup.sql
# Restore using psql
docker compose -f docker-compose.prod.yml exec db \
psql -U $DB_USER -d $DB_NAME -f /tmp/backup.sql
Step 5: Run Pending Migrations¶
After restoring, the backup may be from a schema version behind the current code:
docker compose -f docker-compose.prod.yml exec backend \
python manage.py migrate
# Verify all migrations are applied
docker compose -f docker-compose.prod.yml exec backend \
python manage.py showmigrations
Step 6: Restart Services¶
docker compose -f docker-compose.prod.yml up -d
# Verify all services are healthy
docker compose -f docker-compose.prod.yml ps
Step 7: Verify Health¶
# Check backend health endpoint
curl -f http://127.0.0.1:8000/api/health/
# Check frontend
curl -f http://127.0.0.1:3000/
# Tail logs for errors
docker compose -f docker-compose.prod.yml logs -f --tail=50
Step 8: Verify Data Integrity¶
from apps.products.models import Product, Color, Size
from apps.orders.models import Order
from django.contrib.auth import get_user_model
User = get_user_model()
print(f"Products: {Product.objects.count()}")
print(f"Colors: {Color.objects.count()}")
print(f"Sizes: {Size.objects.count()}")
print(f"Orders: {Order.objects.count()}")
print(f"Users: {User.objects.count()}")
Compare counts with expected values or pre-incident counts.
Disaster Recovery Scenarios¶
Scenario A: Database Corruption¶
Situation: Database tables corrupted due to disk failure, software bug, or hardware issue.
Symptoms:
- Database queries return errors
- Application crashes with database errors
- Data appears inconsistent or missing
Estimated downtime: 20-35 minutes Data loss: Up to 1 hour (RPO)
Procedure:
- Identify corruption scope -- determine which tables or data are affected
- Select restore point -- choose the most recent backup from before corruption occurred
- Follow the full Restore Procedure (steps 1-8)
- Verify data integrity in the Django shell
- Resume operations and monitor logs
Scenario B: Complete VPS Failure¶
Situation: VPS hardware failure, cloud provider outage, or catastrophic system failure.
Symptoms:
- VPS unreachable via SSH
- Application completely down
- No response from any service
Estimated downtime: 2-4 hours (within RTO target) Data loss: Up to 1 hour (RPO)
Procedure:
-
Provision new VPS (30-60 min): a Hetzner VPS with at least 4 GB RAM (the memory limits in
docker-compose.prod.ymladd up to about 3.75 GB; production today is a CX43 with 16 GB) and the public key of the deploy laptop.scripts/provision-server.shdoes the rest in two passes: thedeployuser, packages, swap, ufw (22, 80, 443), fail2ban, Docker,/opt/webshop, the cron job forrestart-unhealthy.shand unattended upgrades; the second pass (--harden) closes root login and passwords. Run it from your workstation: -
Deploy infrastructure (15-30 min). The deploy workflow only copies
scripts/*.sh, the compose file andnginx/nginx.conf; the rest is restored by hand from the repo and the password manager:ssh freeze-prod cd /opt/webshop # Files from the repo (checked out on your workstation): compose file, nginx config, # redis config and the scripts # scp docker-compose.prod.yml nginx/nginx.conf redis/redis.conf scripts/*.sh ... (keep the paths: nginx/, redis/, scripts/) # Secrets from the password manager (never in git): the env file and the basic-auth # file that nginx still mounts nano backend/.env.production # chmod 600 htpasswd -c nginx/.htpasswd <user> # apache2-utils is installed by provisioning # Self-signed certificate for PostgreSQL (mounted from ./certs) bash scripts/generate-pg-certs.sh # Pull Docker images from GHCR (IMAGE_TAG defaults to latest; or X.Y for a specific release) docker compose -f docker-compose.prod.yml pull # Start infrastructure services first docker compose -f docker-compose.prod.yml up -d db redis -
Create the database (the container does not auto-create it):
-
Restore database from S3 backup (20-35 min) -- follow Restore Procedure steps 1-5
-
Start application services:
-
Configure DNS (5-30 min): at Openprovider, point the A records of
www, the apex andprodto the new VPS IP. Check propagation: -
Configure SSL (5-10 min): issue the certificate with
bash nginx/certbot-init.sh --alt www.freezedesign.nl --alt prod.freezedesign.nl(defaults: domain and lineagefreezedesign.nl,docker-compose.prod.yml, webroot challenge;--stagingfirst to test against the Let's Encrypt staging CA). Then install the renewal cron (see SSL Renewal → "Initial install") so the certificate does not silently expire 90 days later. The secretsPRODUCTION_HOST,PRODUCTION_USERandPRODUCTION_SSH_KEYof the GitHub environmentproductionmust point at the new server for the workflows (deploy-production.yml,ssl-renewal.yml) to reach it. -
Restore the other cron jobs:
restart-unhealthywas installed by provisioning (/etc/cron.d/webshop-restart-unhealthy); the media backup needs rclone and its cron again (see Media Backup). -
Verify functionality: Full smoke test -- health endpoints, product listing, checkout flow, and
docker compose -f docker-compose.prod.yml pswith every servicehealthy(celery-beat included). The next hourly backup should appear in#backups.
Downtime breakdown:
| Step | Time |
|---|---|
| VPS provisioning | 30-60 min |
| Infrastructure deployment | 15-30 min |
| Database restore | 20-35 min |
| DNS propagation | 5-30 min |
| SSL certificate | 5-10 min |
| Verification | 10-15 min |
| Total | ~2-4 hours |
Scenario C: Accidental Data Deletion¶
Situation: User data, orders, or products accidentally deleted (bulk delete, admin error, script bug).
Symptoms:
- Reports of missing data
- Empty tables or reduced record counts
- User complaints about lost orders or designs
Estimated downtime: 15-35 minutes Data loss: Minimal (data between last backup and deletion)
Procedure:
- Identify deletion timestamp -- check admin logs, application logs, or user reports
-
Stop application to prevent further changes:
-
Select backup from before the deletion timestamp
-
Choose restore approach:
Option A -- Selective restore (if only specific tables affected):
# Download and decompress backup, then restore specific tables only docker compose -f docker-compose.prod.yml exec db \ pg_restore -U $DB_USER -d $DB_NAME \ -t orders_order -t orders_orderitem \ --data-only \ /tmp/backup.dumpOption B -- Full restore (if deletion scope is unclear):
Follow the complete Restore Procedure.
-
Verify that deleted data is restored
- Restart services
Scenario D: Failed Database Migration¶
Situation: Database migration fails partway through or causes data corruption.
Symptoms:
migratecommand fails with an error- Application errors after migration
- Data inconsistencies after schema change
Estimated downtime: 20-35 minutes (restore) + time to fix migration Data loss: Depends on approach
Procedure:
-
Attempt migration rollback (if the migration is reversible):
-
If rollback fails or migration is irreversible:
- Stop the application
- Restore from backup taken before the migration (this is why you take a manual backup before risky migrations)
- Follow the full Restore Procedure
- Verify schema and data integrity
-
Fix the migration -- correct the migration code, test in a local or staging environment
-
Re-apply the fixed migration
Prevention:
- Always take a manual backup before irreversible migrations
- Test migrations on staging first
- Deploy migrations separately from application code changes
- Write migrations to be reversible when possible
Monitoring and Alerting¶
Backup Task Logs¶
All backup tasks log to the apps.core.backup logger.
# View recent backup task logs
docker compose -f docker-compose.prod.yml logs celery | grep -i backup
# Follow backup logs in real time
docker compose -f docker-compose.prod.yml logs -f celery
Success output:
INFO apps.core.backup: Backup completed successfully: {
'filename': 'backup_20260201_143022.sql.gz',
's3_url': 's3://freeze-production-backups/backup_20260201_143022.sql.gz',
'size_mb': 87.43, ...
}
Failure output:
ERROR apps.core.backup: Backup failed: pg_dump failed: ...
ERROR apps.core.backup: S3 upload failed: ...
Known failure cause until v1.54¶
The 219 backup failures up to 2026-09-26 and the ones of October 2026 had one cause (#992):
apps/core/instrumentation.py installed a SlowQueryLogger wrapper on every database
reconnect, so a long-lived Celery child stacked hundreds of them and every query ended in
RecursionError: maximum recursion depth exceeded. The backup itself had already been
uploaded; the BackupRecord insert failed, so the task reported a failure. Each redeploy
reset the stack and hid it for a while. Fixed in v1.54 (one wrapper per connection); a
RecursionError in #backups today is a new problem.
Discord Notifications¶
Backup task failures trigger Discord notifications to the #backups channel. Two mechanisms are involved:
- Celery backup tasks notify via the Django
DiscordService(backend/apps/core/services/backup_notifications.py, webhook env varDISCORD_WEBHOOK_BACKUPS, legacyBACKUP_DISCORD_WEBHOOK). - The weekly CI backup test notifies via
scripts/notify-discord.sh(secretDISCORD_WEBHOOK_STAGING_BACKUPS, falling back toDISCORD_WEBHOOK_URL).
Since v1.59 both also reach ntfy: DiscordService forwards to topic freeze-backups (a red message also goes by e-mail to NOTIFY_FALLBACK_EMAIL), and notify-discord.sh to freeze-deploy. See ntfy notifications.
Daily Verification Task¶
Runs at 04:00 Amsterdam time every day. Checks:
- Latest backup exists in S3
- File size > 1 MB
- Checksum matches metadata
- Backup is less than 2 hours old
Failures are logged and should trigger manual investigation.
Weekly CI Backup Test¶
Runs every Sunday at 05:00 UTC in GitHub Actions (.github/workflows/backup-test.yml). Performs a full backup/restore cycle and reports results via GitHub Actions summary and Discord notification.
Health Check Commands¶
# List recent backups in Spaces (credentials as in "Step 1: Download Backup")
aws s3 --endpoint-url https://ams3.digitaloceanspaces.com ls s3://freeze-production-backups/ | tail -5
# Check the latest backup timestamp
aws s3 --endpoint-url https://ams3.digitaloceanspaces.com ls s3://freeze-production-backups/ | tail -1
# Check Celery Beat is alive (healthy = heartbeat < 10 min old) and scheduling tasks
docker compose -f docker-compose.prod.yml ps celery-beat
docker compose -f docker-compose.prod.yml logs celery-beat | tail -20
# Check recent backup task results
docker compose -f docker-compose.prod.yml logs celery | grep -i backup | tail -20
RTO/RPO Evidence¶
RPO: 1 Hour¶
- Mechanism: Hourly automated backups via Celery Beat (
crontab(minute=0)) - Maximum data loss: 1 hour (worst case: failure occurs 1 minute before next hourly backup)
- Verification: Daily verification task confirms latest backup is < 2 hours old
RTO: 20-35 Minutes (Actual)¶
| Step | Time Estimate | Notes |
|---|---|---|
| Identify issue | 5 min | Sentry e-mail or Discord notification; no external uptime monitor yet (todo externe-uptime-monitor), so a dead server is noticed by hand |
| Download backup from S3 | 2-5 min | ~50-100 MB compressed |
| Decompress backup | 1-2 min | gzip decompression |
| Stop application | 1 min | docker compose stop |
Restore database (pg_restore) |
5-15 min | Depends on database size (100-500 MB) |
| Run migrations | 1-2 min | Usually fast (idempotent) |
| Restart services | 2-3 min | docker compose up -d |
| Health verification | 2-5 min | Smoke tests, check logs |
| Total | 20-35 min | Well within 4-hour target |
Continuous Verification¶
| Frequency | Check | Source |
|---|---|---|
| Hourly | Backup creation and S3 upload | Celery Beat task |
| Daily | Verification of latest backup integrity | backup.verify_latest_backup task |
| Weekly | Full backup/restore cycle in CI | .github/workflows/backup-test.yml |
| Monthly | Automated restore test to temporary database | backup.test_restore task |
Media Backup¶
The database backups above do not contain the uploaded media (product photos, designs,
fonts): those live in the media bucket (freeze-production-media, freeze-staging-media),
and a database restore only brings back the file paths. Two layers protect the bytes:
| Layer | What it covers | Retention | Where it is configured |
|---|---|---|---|
| Spaces versioning on the media bucket | An overwritten or deleted object stays retrievable as an old version | 30 days once the lifecycle rule is set (NoncurrentVersionExpiration), see .planning/todos/pending/backup-versies-lifecycle.md |
Bucket setting, by Ivar with a full-access key |
Copy in a second bucket (scripts/backup-media.sh) |
The whole bucket, outside the bucket the webshop writes to; survives an emptied bucket, a leaked key or a regional outage | huidig/ mirrors the source; archief/<date>/ keeps what was deleted or overwritten for 12 months (MEDIA_BACKUP_ARCHIVE_DAYS=365) |
Cron on the VPS, by Ivar; the script is in the repo |
How the copy works¶
One run of backup-media.sh (exit 0 = complete, 1 = sync failed or copy differs, 2 =
usage error; details and variables in scripts/backup/README.md):
rclone sync <source> <target>/huidig/ --backup-dir <target>/archief/<YYYY-MM-DD>/ --checksum --fast-list- Delete
archief/folders older than 365 days - Compare object count and bytes of the source with
huidig/
Setting it up on the VPS (Ivar)¶
The script lives at scripts/backup-media.sh (top level, not in scripts/backup/) because
the deploy workflows copy only scripts/*.sh to /opt/webshop on the VPS. Staging gets it
with the next staging deploy; production only with the next release. To run it on a server
before that, copy it by hand: scp scripts/backup-media.sh <vps>:/tmp/ and
sudo install -m 755 /tmp/backup-media.sh /usr/local/sbin/backup-media.sh.
# 1. Second bucket with its own key: e.g. freeze-production-media-backup, a key
# that can read the media bucket and read/write/delete the new one (rclone
# moves files into archief/ and purges old archive folders). Never commit keys.
# The production backup bucket is in sfo3 (set up 2026-10-17), so the copy
# also survives an ams3 outage; staging's is in ams3.
# 2. rclone on the VPS: install, then write /root/.config/rclone/rclone.conf
# (chmod 600) by hand rather than via `rclone config create`, which would
# leave the secret in the shell history. Each remote has its own endpoint:
# [spaces] type = s3, provider = DigitalOcean, endpoint = ams3.digitaloceanspaces.com, acl = private
# [backup] same, endpoint = <region of the backup bucket>.digitaloceanspaces.com,
# plus no_check_bucket = true (a per-bucket key may not create buckets;
# without it the first write fails with 403).
# With per-bucket keys `rclone lsd backup:` (all buckets) fails; always use
# the full bucket path. "directory not found" on the bucket means wrong
# name or wrong region endpoint.
sudo -v ; curl -fsSL https://rclone.org/install.sh | sudo bash
sudo rclone size spaces:freeze-production-media # source readable
sudo rclone lsd backup:freeze-production-media-backup # target reachable
# 3. First run, dry first
sudo MEDIA_BACKUP_SOURCE=spaces:freeze-production-media \
MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup \
/opt/webshop/scripts/backup-media.sh --dry-run
sudo MEDIA_BACKUP_SOURCE=spaces:freeze-production-media \
MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup \
/opt/webshop/scripts/backup-media.sh
# 4. Nightly at 02:30 UTC. The VPS runs in UTC; the database cleanup is a Celery Beat
# task at 03:00 Amsterdam time (02:00 UTC in winter, 01:00 UTC in summer), so the
# media backup runs after it, not before (todo media-backup-tijd-na-database-cleanup)
sudo crontab -l 2>/dev/null > /tmp/crontab.bak || true
echo "30 2 * * * MEDIA_BACKUP_SOURCE=spaces:freeze-production-media MEDIA_BACKUP_TARGET=backup:freeze-production-media-backup /opt/webshop/scripts/backup-media.sh >> /var/log/backup-media.log 2>&1" | sudo tee -a /tmp/crontab.bak
sudo crontab /tmp/crontab.bak
Check the first copy with rclone size backup:freeze-production-media-backup/huidig
against rclone size spaces:freeze-production-media (the script prints both), and
browse archief/ after the first day on which something was deleted or replaced.
Restoring media¶
# One file that was deleted or overwritten on <date>
rclone copy backup:freeze-production-media-backup/archief/<date>/media/products/foto.png spaces:freeze-production-media/media/products/
# The whole bucket after a disaster (into an empty or new media bucket)
rclone sync backup:freeze-production-media-backup/huidig spaces:freeze-production-media --checksum --fast-list
Within 30 days an overwritten object can also be restored from its Spaces version
(aws s3api list-object-versions + copy-object ... ?versionId=), see the lifecycle
todo above.
SSL Renewal¶
Let's Encrypt certificates expire every 90 days. The staging certificate expired silently on 2026-05-07 because no renewal automation was in place — this section documents the layered automation that prevents a repeat. Let's Encrypt no longer sends expiry e-mails (since June 2025), so the external monitor below is the only warning if both layers fail.
Environments¶
DOMAIN is the certbot lineage (the directory under /etc/letsencrypt/live/),
not necessarily a hostname. renew-ssl.sh refuses to run without it.
| Environment | SSH alias | DOMAIN (lineage) |
Names in the certificate | COMPOSE_FILE |
|---|---|---|---|---|
| staging | freeze-staging |
staging.freezedesign.eu |
staging.freezedesign.nl |
/opt/webshop/docker-compose.staging.yml |
| production | freeze-prod |
freezedesign.nl |
freezedesign.nl, www.freezedesign.nl, prod.freezedesign.nl |
/opt/webshop/docker-compose.prod.yml |
The staging lineage still carries the old .eu name; certbot renews it under that
name, so leave DOMAIN as is.
Two-layer automation¶
The same script (scripts/renew-ssl.sh) runs from two independent triggers so a
failure in one layer doesn't cause an outage:
- GitHub Actions workflow (
.github/workflows/ssl-renewal.yml) - Schedule: every Wednesday 04:00 UTC + manual
workflow_dispatch(optionforcepasses--force-renewal) - One matrix job per environment, run with
environment: staging/production, so it reads theSTAGING_*/PRODUCTION_*SSH secrets from that GitHub environment (#1025; before that the job looked for repo secrets, found none and skipped both environments with only a notice). An environment whose*_HOSTsecret is still missing is skipped the same way. The schedule runs frommain, so the deployment policy of both environments must allowmain -
Discord notification (success or failure) via
DISCORD_WEBHOOK_CICD -
VPS crontab (root)
- Schedule: weekly safety-net (Sundays 03:30 UTC)
- Same
scripts/renew-ssl.shinvocation, withDOMAINandCOMPOSE_FILEset - Survives GitHub-Actions outages
Certbot itself is idempotent — it skips renewal until the cert is within the 30-day window — so running both layers is safe; whichever fires first wins, the other is a no-op.
renew-ssl.sh always passes --preferred-challenges http-01. The production
certificate was extended on 2026-09-28 via a manual DNS-01 challenge, which leaves
pref_challs = dns-01 in its renewal file; without the flag certbot renew --webroot
fails with "None of the preferred challenges are supported by the selected plugin".
Extra certbot options go in CERTBOT_EXTRA_FLAGS (space-separated).
Dry run (after any change to the certificate or nginx)¶
Tests the full renewal against the Let's Encrypt staging CA without touching the live certificate:
ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml \
CERTBOT_EXTRA_FLAGS="--dry-run --no-random-sleep-on-renew --cert-name freezedesign.nl" \
bash scripts/renew-ssl.sh'
Expected: Congratulations, all simulated renewals succeeded and RESULT: skipped.
Without --no-random-sleep-on-renew certbot first waits a random delay of up to
8 minutes (it does so for every non-interactive renew), which looks like a hang.
The scheduled runs keep that delay on purpose; the workflow timeout (15 min) allows it.
Manual renewal¶
If both automations are down:
ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh'
The script reloads nginx automatically when a new certificate was installed. Use the staging values from the table for staging.
Initial install (after first VPS provisioning OR after disaster recovery)¶
# 1. Install the cron safety-net (production; use the staging values on staging)
sudo crontab -l 2>/dev/null > /tmp/crontab.bak || true
echo "30 3 * * 0 DOMAIN=freezedesign.nl COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml /opt/webshop/scripts/renew-ssl.sh >> /var/log/ssl-renewal.log 2>&1" | sudo tee -a /tmp/crontab.bak
sudo crontab /tmp/crontab.bak
# 2. Verify
sudo crontab -l | grep renew-ssl
# 3. The GitHub Actions side is automatic once the workflow file is on main and the
# SSH secrets exist as environment secrets of the GitHub environment `production`
# (PRODUCTION_HOST, PRODUCTION_USER, PRODUCTION_SSH_KEY; `staging` holds STAGING_*).
# Check with a manual run of "SSL Renewal": a skipped leg prints
# "<env>: secret <NAME> is niet gezet — overgeslagen".
A crontab line without DOMAIN does nothing: the script exits with code 2.
Health check before expiry¶
From any machine:
for h in www.freezedesign.nl freezedesign.nl prod.freezedesign.nl staging.freezedesign.nl; do
printf '%-26s ' "$h"
echo | openssl s_client -connect "$h:443" -servername "$h" 2>/dev/null \
| openssl x509 -noout -enddate
done
Or with the same script the workflow uses (exit 1 below MIN_DAYS, default 14):
bash scripts/check-ssl-expiry.sh www.freezedesign.nl freezedesign.nl prod.freezedesign.nl staging.freezedesign.nl
ssl-renewal.yml runs this weekly as job expiry-check after the renewal jobs
(also when they failed) and posts to Discord when a certificate expires within
14 days or a host returns no certificate. It runs on GitHub, so for a fully
independent warning also add an external monitor (UptimeRobot or similar) on
https://www.freezedesign.nl/api/health/ and https://staging.freezedesign.nl/api/health/,
alerting at least 14 days before expiry.