Alerting Runbook¶
Step-by-step response procedures for every alert type in the Freeze Design monitoring system.
Environments
Staging (SENTRY_ENVIRONMENT=staging) and production (SENTRY_ENVIRONMENT=production,
https://www.freezedesign.nl) both report to Sentry; the alert rules filter on
production. Sentry is the working error tracker; PostHog is analytics-only.
On the server
The commands in this runbook run in a shell on the server, in the stack directory, with the right compose file selected. Do this first in every SSH session:
Without it, a bare docker compose … fails with no configuration file provided: neither
server has a docker-compose.yml. docker compose reads COMPOSE_FILE, so every command
below then works as written.
Where alerts come from (checked 2026-11-01)
Sentry sends e-mail, no Discord: four issue alerts for production (two high-priority
alerts, "nieuw of terugkerend issue (niet hoog)", and "fout in de betaal-webhook
(Mollie)"); the list with links is in Monitoring. There are
no Sentry metric alerts. Django posts to Discord through DiscordService
(DISCORD_WEBHOOK_{ERRORS,PAYMENTS,BACKUPS,ADMIN}; since v1.53 also the Informer
bookkeeping link, to #production-admin-actions), and the cron job
scripts/restart-unhealthy.sh posts to #errors. Since v1.59 the same messages also go
to ntfy, and healthchecks.io alerts when the beat pings stop (see
ntfy notifications). No external uptime monitor exists yet
(todo externe-uptime-monitor, status: ready). Celery times below are Dutch time
(Europe/Amsterdam).
Overview¶
Alert Routing Diagram¶
flowchart LR
S[Sentry: backend + frontend issues] -->|4 issue alerts, production| M[E-mail to Ivar]
B[Celery: backup.* tasks] --> BK[#backups]
P[Celery: payment anomaly, daily summary,<br/>recover paid orders] --> PAY[#payments]
D[Celery: disk space check] --> ERR[#errors]
C[cron: restart-unhealthy.sh<br/>celery-beat] --> ERR
A[Admin audit: delete / bulk_delete<br/>paid orders, stock, failed refunds,<br/>Informer booking failures] --> ADM[#production-admin-actions]
W[GitHub Actions: deploys, E2E smoke, SSL,<br/>backup test, k6, dagplanning, websitecheck] --> CI[CI/CD channels]
Every source and where its alert lands; the Discord channels are webhooks from the server env file, the CI/CD channels are GitHub secrets.
Quick Reference Table¶
| # | Alert | Arrives | Severity | Response Time | Key Command |
|---|---|---|---|---|---|
| 1 | New Issue | Sentry e-mail | Medium | Within 4 hours | docker compose logs backend --tail=100 |
| 2 | Regression | Sentry e-mail | HIGH | Within 1 hour | docker compose logs backend --tail=200 |
| 3 | Error Spike | Sentry high-priority e-mail, or many mails at once | Critical | Within 30 min | docker compose ps |
| 4 | Payment Error (Mollie webhook) | Sentry e-mail | CRITICAL | Within 15 min | docker compose logs backend --tail=100 \| grep -i "payment\|mollie" |
| 5 | Payment Anomaly | #payments | CRITICAL | Within 15 min | docker compose logs backend --tail=200 \| grep -i "payment\|mollie" |
| 6 | Backup Failure | #backups | High | Within 2 hours | docker compose exec backend python manage.py backup_health |
| 7 | Missed Backup | #backups | High | Within 2 hours | docker compose ps celery celery-beat |
| 8 | Disk Space Alert | #errors | Varies | Within 1 hour | df -h |
| 9 | Admin Delete | #production-admin-actions | Info | Next business day | Check the admin audit log |
| 10 | celery-beat restarted | #errors | Medium (High when repeated) | Within 4 hours | docker compose logs celery-beat --tail=100 |
Alert Type 1: New Issue Alert¶
Arrives: Sentry e-mail ("Productie: nieuw of terugkerend issue (niet hoog)", at most every 30 minutes; a high-priority issue has its own alert per project) Severity: Medium Source: Sentry issue alert rule What it means: Sentry detected an error it has never seen before in production.
Response Steps¶
-
Click the Sentry link in the e-mail to open the issue
-
Assess the impact:
- How many users affected? (check "Events" count in Sentry)
- Is it a 500 error visible to customers?
-
What page/endpoint is affected? (frontend issues carry the scrubbed URL,
:id/:key) -
Check recent deployments:
-
Check application logs:
-
If the error is user-facing:
- Determine if a hotfix is needed
-
If critical path (checkout, payment), escalate to Payment Error procedures
-
Resolve or assign in Sentry:
- If one-off / transient: mark as Ignored with note
- If needs fixing: leave open and create a todo in
.planning/todos/pending/(or let the weekly website check propose one) - If fixed by deploy: mark as Resolved in next release
Common Causes¶
- New code deployment introduced a bug
- External service API changed (Mollie, Resend, DigitalOcean Spaces, Google Places)
- Edge case in user input not previously encountered
- Browser-specific JavaScript error (frontend project)
Alert Type 2: Regression Alert¶
Arrives: Sentry e-mail (same rule as type 1: "new or regressed") Severity: HIGH Source: Sentry issue alert rule What it means: An issue that was previously marked as resolved has reappeared. This means a fix did not hold or was reverted.
Response Steps¶
-
Click the Sentry link to open the regressed issue
-
Check the issue history in Sentry:
- When was it originally resolved?
- What release resolved it?
-
What release caused the regression?
-
Compare releases:
-
Check application logs for the specific error:
-
If the regression is in a critical path, roll back to the previous release (production; the full procedure is in Deployment):
A rollback does not undo migrations; checkexport GITHUB_REPOSITORY=Voorman/webshop_freeze_design REGISTRY=ghcr.io export DEPLOY_TAG=latest ROLLBACK_VERSION=<X.Y> ./scripts/rollback.sh # e.g. 1.49; without it: the local predeploy snapshot docker compose pspython manage.py showmigrationsfor the previous version before you do this. -
Create a fix:
- Regressions indicate the original fix was incomplete or a new change conflicted
- Write a test that covers the regression scenario
- Deploy the fix and verify in Sentry that the issue resolves
Common Causes¶
- A new deployment overwrote or conflicted with the original fix
- The original fix was incomplete (fixed one case but not edge cases)
- Database migration changed data shape that the fix depended on
- Dependency update changed behavior
Alert Type 3: Error Spike¶
Arrives: several Sentry e-mails in a short time, or a high-priority alert; there is no metric alert with a threshold Severity: Critical What it means: The error rate is far above normal, indicating a systemic issue rather than isolated errors. As a rule of thumb for this low-traffic shop: more than 10 errors in an hour is elevated, more than 50 is a major incident (bad deploy, database down, external outage).
Response Steps¶
-
Check if the application is running:
All services should show "Up" and "healthy". -
Check for a recent deployment:
-
Check application logs for patterns:
-
Check external dependencies:
-
Check Sentry for the top issues:
- Go to Sentry > Issues > filter
environment:production> sort by "Events" > last hour -
The top issue is likely the cause of the spike
-
If caused by a deployment, consider rollback:
-
See rollback steps in Alert Type 2
-
If caused by external service failure:
- Check Mollie status: https://status.mollie.com
- Check DigitalOcean status: https://status.digitalocean.com
- If external, the spike will resolve when the service recovers
Common Causes¶
- Bad deployment (new bug affecting many requests)
- Database connection issues (pool exhausted, slow queries)
- Redis connection issues (cache failures cascading)
- External API outage (Mollie, DigitalOcean)
- DNS or certificate issues (SSL expiry)
Alert Type 4: Payment Error¶
Arrives: Sentry e-mail "Productie: fout in de betaal-webhook (Mollie)", at most every 5 minutes
Severity: CRITICAL
Source: Sentry issue alert rule on every error event with transaction:/api/payments/webhook/
What it means: The Mollie webhook raised an error while processing a payment status. Orders may stay "Wacht op betaling" although the customer paid. Errors elsewhere in the payment flow (creating the payment, the checkout) arrive as a normal new issue (type 1); treat them with the steps below as well.
Response Steps¶
-
Click the Sentry link immediately
-
Determine the error type:
- Webhook processing failure (payment status update failed). A temporary Mollie error (network, time-out, 429, 5xx) answers 503 so Mollie retries; those are logged and only need action when they keep coming
- Payment creation failure (Mollie API error)
- Checkout flow error (cart/order creation)
-
Invoice generation error
-
Check payment logs:
-
Check Mollie dashboard:
- Log in to https://my.mollie.com
- Go to Payments > check recent payment statuses
-
Are payments being created but failing at Mollie?
-
Check Mollie API status:
- Visit https://status.mollie.com
-
If Mollie is down, customers will see payment failures until service recovers
-
Verify webhook connectivity:
-
If the error is in our code:
- Check the traceback in Sentry for the exact failure point
- Common locations:
apps/payments/views.py(webhook),apps/payments/services.py(create_payment),apps/orders/ -
If a quick fix is possible, deploy immediately
-
Check for stuck orders (paid at Mollie, still pending here). The hourly task
monitoring.recover_incomplete_paid_orders(:45) re-runs invoice, mail, admin notice, print files and the shop mail for orders that are alreadypaid; it does not change an order that is stillpending:For an order that Mollie shows as paid, open the payment in Mollie and resend the webhook, or set the order to "Betaald" in the admin once the Mollie status is confirmed.docker compose exec backend python manage.py shell -c " from apps.orders.models import Order from django.utils import timezone from datetime import timedelta stuck = Order.objects.filter( status='pending', created_at__lt=timezone.now() - timedelta(hours=1) ) print('Pending > 1h:', stuck.count()) for o in stuck[:10]: print(o.order_number, o.created_at) " -
Customer communication:
- If payment failures are widespread, consider adding a banner to the site
Common Causes¶
- Mollie API outage or degradation
- Mollie API key expired or revoked
- Webhook URL not reachable (nginx misconfiguration, SSL issue)
- Code bug in payment flow (especially after deployment)
- Currency/amount formatting error
Alert Type 5: Payment Anomaly¶
Arrives: #payments
Severity: CRITICAL
Source: Celery task monitoring.check_payment_anomalies, every 15 minutes. It alerts when the last hour has at least 3 failed payments or a failure rate above 50 % (PAYMENT_ANOMALY_FAILURE_COUNT, PAYMENT_ANOMALY_FAILURE_RATE, PAYMENT_ANOMALY_WINDOW_HOURS), and stays silent under 3 payments in the window.
What it means: Multiple payment failures in a short period. This is a systemic payment failure, not an isolated error. The same channel gets the daily summary at 07:00 Dutch time (informational) and the "🔧 Hersteltaak" message when the recover task repaired an order.
Response Steps¶
-
Follow all steps from Alert Type 4 (Payment Error)
-
Check if ALL payments are failing:
docker compose exec backend python manage.py shell -c " from apps.payments.models import Payment from django.utils import timezone from datetime import timedelta since = timezone.now() - timedelta(hours=1) qs = Payment.objects.filter(created_at__gte=since) print('Last hour:', qs.count(), 'payments') for s in ('paid', 'failed', 'expired', 'canceled', 'open', 'pending'): print(f' {s}: {qs.filter(status=s).count()}') " -
If 100% failure rate:
- Likely a Mollie outage or API key issue
- Check
MOLLIE_API_KEYis set and is alive_key:docker compose exec backend sh -c 'echo ${MOLLIE_API_KEY%%_*}' -
Consider temporarily disabling checkout with a maintenance message
-
If partial failure rate:
- Likely a specific payment method or amount issue; with the "Alle betaalmethoden" switch on, a method can fail at Mollie while iDEAL works (see Betaalmethoden)
-
Check Sentry for common attributes across the failing payments
-
Monitor resolution:
- The task alerts every 15 minutes as long as the window is above the threshold
- Verify in Mollie that new payments succeed
Common Causes¶
- Same as Alert Type 4, but systemic
- API key rotation without updating env var
- Mollie account suspension (check email)
- Network issue between server and Mollie API
Alert Type 6: Backup Failure¶
Arrives: #backups
Severity: High
Source: Django backup task (Celery) Discord notification
What it means: A scheduled database backup failed. Data protection is compromised until the next successful backup.
Response Steps¶
-
Check the failure details in the Discord notification (includes error message and task ID)
-
Check backup task logs:
-
Try a manual backup:
If the manual backup succeeds, the failure was likely transient. -
Check disk space:
Backups will fail if the disk is full. -
Check database connectivity:
-
Check backup storage:
-
Verify backup schedule is running:
Look for the backup task being scheduled. -
If disk space is the issue:
Common Causes¶
- Disk full (backup volume)
- Database connection timeout during dump
- PostgreSQL in recovery mode (after crash)
- Celery worker crashed or restarted during backup
- File permission issue on backup directory
- Spaces key or bucket problem (
BACKUP_S3_*inbackend/.env.production) - A Celery child killed by the memory limit mid-task: the worker has
mem_limit 768mon production and--max-memory-per-child=262144since #990 (v1.54); checkdocker compose logs celery --tail=200 | grep -i "WorkerLostError\|SIGKILL" RecursionError: maximum recursion depth exceededat theBackupRecordinsert: the slow-query logger stacked one wrapper per database reconnect until #992 (v1.54). That explained the 219 failures up to 2026-09-26 and the ones in October 2026; on the current release this would be a new bug, not the old one
Alert Type 7: Missed Backup Warning¶
Arrives: #backups
Severity: High
Source: Daily backup summary task (backup.daily_summary, 06:00 Dutch time) with the field "Missed Backup Warning"
What it means: No successful backup has been recorded in the last 25 hours. The backup system may be silently broken.
Response Steps¶
-
Check if backups are running at all:
Anunhealthycelery-beat is restarted by cron within 5 minutes and reported in#errors(type 10). If it keeps failing, see that section. -
Check Celery worker health:
If the Celery worker is not running, backups cannot execute. -
Check backup monitor metrics:
docker compose exec backend python manage.py shell -c " from apps.core.services.backup_monitor import BackupMonitor monitor = BackupMonitor() metrics = monitor.get_metrics() print(f'Last backup: {metrics.get(\"last_backup_time\", \"Never\")}') print(f'Success rate: {metrics.get(\"success_rate\", \"Unknown\")}%') print(f'Total backups: {metrics.get(\"total_backups\", 0)}') " -
Check Redis (backup history is cached):
If Redis is down, backup history may be lost (but backups themselves still run). -
Run a manual backup to restore the schedule:
-
If Celery Beat is not scheduling tasks:
Common Causes¶
- Celery Beat stalled (now caught by its healthcheck) or not started
- Celery worker crashed (tasks scheduled but not executed)
- Redis down (task results not stored, but tasks may still run)
- Server reboot without docker compose up
Alert Type 8: Disk Space Alert¶
Arrives: #errors
Severity: Warning (≥ 80 %) or Critical (≥ 90 %)
Source: Celery task monitoring.check_disk_space (hourly at :30; thresholds DISK_SPACE_WARNING_PCT, DISK_SPACE_CRITICAL_PCT). It measures / as seen from the backend container.
What it means: Server disk usage has crossed a threshold. If it reaches 100%, the application will crash.
Response Steps¶
-
Check current disk usage:
-
Identify what is consuming space:
-
Common space consumers:
Docker images and containers:
docker system df
# Clean up unused images and containers (keeps the predeploy tags that are in use)
docker system prune -f
Local backup files (the backup_volume; the bucket is cleaned by the daily task):
Container logs are capped since #657 (5 × 10 MB per container). db, redis and
nginx only got that cap after a recreate; check with
sudo du -sh /var/lib/docker/containers/*/ and, if one is large,
docker compose up -d --no-deps db redis nginx (short restart of those three).
PostgreSQL WAL files:
- If above 90% (critical):
- Delete old Docker images:
docker image prune -a -f -
Consider increasing disk size at Hetzner
-
If above 95% (emergency):
- Delete the oldest local backup files in
/app/backups/ - Increase disk immediately at Hetzner Cloud console
Common Causes¶
- Docker images accumulating from deployments without cleanup
- PostgreSQL WAL files growing (missing checkpoint)
- A container created before the log cap (#657) with a huge json log
- Media uploads filling disk (should be on DigitalOcean Spaces, not local)
Alert Type 9: Admin Delete Action¶
Arrives: #production-admin-actions
Severity: Info
Source: Admin audit trail (apps/core/audit.py): every delete and bulk_delete through the admin API. The same channel carries the "paid order" message for every paid order, stock problems found after payment, and refunds that failed or were canceled at Mollie (apps/orders/tasks.py); those need no response beyond reading them. Since v1.53 it also gets "⚠️ Informer: apps/informer/notify.py) when a sales invoice could not be booked in Informer; the order in Admin → Bestellingen shows the reason and "Nu bijwerken" retries, and informer.sweep retries temporary errors every 5 minutes on its own.
What it means: An admin user performed a delete action in the admin interface. Deletes are irreversible and logged for audit purposes.
Response Steps¶
- Review the notification details:
- Who performed the delete?
- What model/object was deleted?
-
When did it happen?
-
Verify the delete was intentional:
- Check with the admin user if this was planned
-
If unexpected, check the admin audit log for more context:
docker compose exec backend python manage.py shell -c " from apps.core.models import AdminAuditLog recent = AdminAuditLog.objects.filter(action__in=['delete', 'bulk_delete']).order_by('-timestamp')[:5] for log in recent: print(f'{log.timestamp}: {log.user} {log.action} {log.model_name} (ID: {log.object_id}) {log.object_repr}') " -
If the delete was accidental:
- Check if a backup exists from before the delete
- Consult Disaster Recovery for restore instructions
-
Note: Restoring a full backup will overwrite all changes since the backup
-
No action needed if:
- The delete was intentional and expected
- This is informational only -- most delete notifications require no response
Common Causes¶
- Normal admin operations (removing test data, cleaning up old records)
- Accidental bulk delete (admin list view checkbox + delete action)
- Admin cleanup of expired/orphaned data
Alert Type 10: celery-beat restarted¶
Arrives: #errors, footer "restart-unhealthy op productie" (or "staging")
Severity: Medium; High when it repeats
Source: cron job /etc/cron.d/webshop-restart-unhealthy → scripts/restart-unhealthy.sh (every 5 minutes). Orange "🔁 celery-beat herstart": Docker reported the container unhealthy (no heartbeat for 10 minutes) and the script restarted it. Red "⚠️ celery-beat nog steeds ongezond": it was restarted less than 30 minutes ago and is unhealthy again; the script does not restart it again. Red "❌ Herstart … mislukt": docker compose restart failed.
What it means: The scheduler that dispatches backups, payment checks and mails stalled. One orange message with a healthy container afterwards needs no action.
Response Steps¶
-
Is it healthy now?
-
Red message, or orange messages every half hour: read the logs
Permission deniedon/app/celerybeat: the schedule volume is owned by root (#811). Fix:- Redis unreachable: beat cannot publish tasks; check
docker compose ps redis -
A corrupt
celerybeat-schedulefile: stop beat, delete the file in the volume (docker compose run --rm --no-deps --entrypoint sh celery-beat -c 'rm -f /app/celerybeat/celerybeat-schedule*') and start it again; the schedule is rebuilt fromconfig/celery.py -
Afterwards: check that the next scheduled task ran (
docker compose logs celery --tail=50), and look in#backupsfor the next hourly backup.
Background and the test procedure: Monitoring.
Escalation Procedures¶
For a solo-operator setup, "escalation" means increasing your own response urgency.
Severity Levels¶
| Severity | Response Time | Description |
|---|---|---|
| Info | Next business day | Informational, no action usually needed |
| Medium | Within 4 hours | Investigate, may need a fix in next deployment |
| High | Within 1 hour | Needs prompt attention, potential data or service impact |
| Critical | Within 15-30 min | Active incident affecting customers or revenue |
When to Consider Downtime¶
Take the site into maintenance mode if:
- Payment system is completely broken (100% failure rate for >30 min)
- Error rate exceeds 100/hour and rising (cascading failure)
- Database is corrupt or unreachable
- Security breach suspected (unauthorized admin access)
Maintenance Mode¶
# Stop the frontend to prevent new orders (nginx then answers 502 for pages; the API stays up)
docker compose stop frontend
# Fix the issue
# Restart
docker compose up -d frontend
Log Locations¶
Quick reference for finding logs during incident response. Every service logs to stdout only (Docker json-file, max 5 × 10 MB per container).
| Service | Log Command | What It Contains |
|---|---|---|
| Backend (Django) | docker compose logs backend --tail=N |
API requests, errors, application logs |
| Celery worker | docker compose logs celery --tail=N |
Task execution, backup results |
| Celery Beat | docker compose logs celery-beat --tail=N |
Task scheduling |
| Nginx | docker compose logs nginx --tail=N |
HTTP requests, proxy errors (/health is not logged) |
| PostgreSQL | docker compose logs db --tail=N |
Database queries, connections |
| Redis | docker compose logs redis --tail=N |
Cache operations, connection issues |
| restart-unhealthy (cron) | journalctl -t restart-unhealthy |
celery-beat restarts |
Filtering Logs by Time¶
# Last hour
docker compose logs backend --since=1h
# Last 30 minutes
docker compose logs backend --since=30m
# Since specific time (UTC)
docker compose logs backend --since="2026-01-15T10:00:00"