Skip to content

Alerting Runbook

Step-by-step response procedures for every alert type in the Freeze Design monitoring system.

Environments

Staging (SENTRY_ENVIRONMENT=staging) and production (SENTRY_ENVIRONMENT=production, https://www.freezedesign.nl) both report to Sentry; the alert rules filter on production. Sentry is the working error tracker; PostHog is analytics-only.

On the server

The commands in this runbook run in a shell on the server, in the stack directory, with the right compose file selected. Do this first in every SSH session:

cd /opt/webshop
export COMPOSE_FILE=docker-compose.prod.yml   # staging: docker-compose.staging.yml

Without it, a bare docker compose … fails with no configuration file provided: neither server has a docker-compose.yml. docker compose reads COMPOSE_FILE, so every command below then works as written.

Where alerts come from (checked 2026-11-01)

Sentry sends e-mail, no Discord: four issue alerts for production (two high-priority alerts, "nieuw of terugkerend issue (niet hoog)", and "fout in de betaal-webhook (Mollie)"); the list with links is in Monitoring. There are no Sentry metric alerts. Django posts to Discord through DiscordService (DISCORD_WEBHOOK_{ERRORS,PAYMENTS,BACKUPS,ADMIN}; since v1.53 also the Informer bookkeeping link, to #production-admin-actions), and the cron job scripts/restart-unhealthy.sh posts to #errors. Since v1.59 the same messages also go to ntfy, and healthchecks.io alerts when the beat pings stop (see ntfy notifications). No external uptime monitor exists yet (todo externe-uptime-monitor, status: ready). Celery times below are Dutch time (Europe/Amsterdam).

Overview

Alert Routing Diagram

flowchart LR
    S[Sentry: backend + frontend issues] -->|4 issue alerts, production| M[E-mail to Ivar]
    B[Celery: backup.* tasks] --> BK[#backups]
    P[Celery: payment anomaly, daily summary,<br/>recover paid orders] --> PAY[#payments]
    D[Celery: disk space check] --> ERR[#errors]
    C[cron: restart-unhealthy.sh<br/>celery-beat] --> ERR
    A[Admin audit: delete / bulk_delete<br/>paid orders, stock, failed refunds,<br/>Informer booking failures] --> ADM[#production-admin-actions]
    W[GitHub Actions: deploys, E2E smoke, SSL,<br/>backup test, k6, dagplanning, websitecheck] --> CI[CI/CD channels]

Every source and where its alert lands; the Discord channels are webhooks from the server env file, the CI/CD channels are GitHub secrets.

Quick Reference Table

# Alert Arrives Severity Response Time Key Command
1 New Issue Sentry e-mail Medium Within 4 hours docker compose logs backend --tail=100
2 Regression Sentry e-mail HIGH Within 1 hour docker compose logs backend --tail=200
3 Error Spike Sentry high-priority e-mail, or many mails at once Critical Within 30 min docker compose ps
4 Payment Error (Mollie webhook) Sentry e-mail CRITICAL Within 15 min docker compose logs backend --tail=100 \| grep -i "payment\|mollie"
5 Payment Anomaly #payments CRITICAL Within 15 min docker compose logs backend --tail=200 \| grep -i "payment\|mollie"
6 Backup Failure #backups High Within 2 hours docker compose exec backend python manage.py backup_health
7 Missed Backup #backups High Within 2 hours docker compose ps celery celery-beat
8 Disk Space Alert #errors Varies Within 1 hour df -h
9 Admin Delete #production-admin-actions Info Next business day Check the admin audit log
10 celery-beat restarted #errors Medium (High when repeated) Within 4 hours docker compose logs celery-beat --tail=100

Alert Type 1: New Issue Alert

Arrives: Sentry e-mail ("Productie: nieuw of terugkerend issue (niet hoog)", at most every 30 minutes; a high-priority issue has its own alert per project) Severity: Medium Source: Sentry issue alert rule What it means: Sentry detected an error it has never seen before in production.

Response Steps

  1. Click the Sentry link in the e-mail to open the issue

  2. Assess the impact:

  3. How many users affected? (check "Events" count in Sentry)
  4. Is it a 500 error visible to customers?
  5. What page/endpoint is affected? (frontend issues carry the scrubbed URL, :id/:key)

  6. Check recent deployments:

    # Which commit runs (the Sentry release is this SHA)
    docker image inspect ghcr.io/voorman/webshop_freeze_design/backend:latest \
      --format '{{index .Config.Labels "org.opencontainers.image.revision"}}'
    
    # Recent tags and commits
    git fetch --tags && git log --oneline -5 origin/main
    

  7. Check application logs:

    docker compose logs backend --tail=100 --since=1h
    

  8. If the error is user-facing:

  9. Determine if a hotfix is needed
  10. If critical path (checkout, payment), escalate to Payment Error procedures

  11. Resolve or assign in Sentry:

  12. If one-off / transient: mark as Ignored with note
  13. If needs fixing: leave open and create a todo in .planning/todos/pending/ (or let the weekly website check propose one)
  14. If fixed by deploy: mark as Resolved in next release

Common Causes

  • New code deployment introduced a bug
  • External service API changed (Mollie, Resend, DigitalOcean Spaces, Google Places)
  • Edge case in user input not previously encountered
  • Browser-specific JavaScript error (frontend project)

Alert Type 2: Regression Alert

Arrives: Sentry e-mail (same rule as type 1: "new or regressed") Severity: HIGH Source: Sentry issue alert rule What it means: An issue that was previously marked as resolved has reappeared. This means a fix did not hold or was reverted.

Response Steps

  1. Click the Sentry link to open the regressed issue

  2. Check the issue history in Sentry:

  3. When was it originally resolved?
  4. What release resolved it?
  5. What release caused the regression?

  6. Compare releases:

    # Find the commit that resolved the original issue
    git log --oneline origin/main | head -20
    
    # Check what changed between the resolving commit and now
    git diff <resolving-commit>..origin/main --stat
    

  7. Check application logs for the specific error:

    docker compose logs backend --tail=200 --since=2h | grep -i "error\|exception\|traceback"
    

  8. If the regression is in a critical path, roll back to the previous release (production; the full procedure is in Deployment):

    export GITHUB_REPOSITORY=Voorman/webshop_freeze_design REGISTRY=ghcr.io
    export DEPLOY_TAG=latest
    ROLLBACK_VERSION=<X.Y> ./scripts/rollback.sh   # e.g. 1.49; without it: the local predeploy snapshot
    docker compose ps
    
    A rollback does not undo migrations; check python manage.py showmigrations for the previous version before you do this.

  9. Create a fix:

  10. Regressions indicate the original fix was incomplete or a new change conflicted
  11. Write a test that covers the regression scenario
  12. Deploy the fix and verify in Sentry that the issue resolves

Common Causes

  • A new deployment overwrote or conflicted with the original fix
  • The original fix was incomplete (fixed one case but not edge cases)
  • Database migration changed data shape that the fix depended on
  • Dependency update changed behavior

Alert Type 3: Error Spike

Arrives: several Sentry e-mails in a short time, or a high-priority alert; there is no metric alert with a threshold Severity: Critical What it means: The error rate is far above normal, indicating a systemic issue rather than isolated errors. As a rule of thumb for this low-traffic shop: more than 10 errors in an hour is elevated, more than 50 is a major incident (bad deploy, database down, external outage).

Response Steps

  1. Check if the application is running:

    docker compose ps
    
    All services should show "Up" and "healthy".

  2. Check for a recent deployment:

    # When was the backend container last started?
    docker inspect --format='{{.State.StartedAt}}' $(docker compose ps -q backend)
    

  3. Check application logs for patterns:

    # Look for the most common errors in the last hour
    docker compose logs backend --tail=500 --since=1h | grep -c "ERROR"
    docker compose logs backend --tail=500 --since=1h | grep "ERROR" | sort | uniq -c | sort -rn | head -10
    

  4. Check external dependencies:

    # Database connection
    docker compose exec backend python manage.py dbshell -c "SELECT 1;"
    
    # Redis connection
    docker compose exec redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping'
    
    # Celery workers
    docker compose logs celery --tail=50 --since=1h
    

  5. Check Sentry for the top issues:

  6. Go to Sentry > Issues > filter environment:production > sort by "Events" > last hour
  7. The top issue is likely the cause of the spike

  8. If caused by a deployment, consider rollback:

  9. See rollback steps in Alert Type 2

  10. If caused by external service failure:

  11. Check Mollie status: https://status.mollie.com
  12. Check DigitalOcean status: https://status.digitalocean.com
  13. If external, the spike will resolve when the service recovers

Common Causes

  • Bad deployment (new bug affecting many requests)
  • Database connection issues (pool exhausted, slow queries)
  • Redis connection issues (cache failures cascading)
  • External API outage (Mollie, DigitalOcean)
  • DNS or certificate issues (SSL expiry)

Alert Type 4: Payment Error

Arrives: Sentry e-mail "Productie: fout in de betaal-webhook (Mollie)", at most every 5 minutes Severity: CRITICAL Source: Sentry issue alert rule on every error event with transaction:/api/payments/webhook/ What it means: The Mollie webhook raised an error while processing a payment status. Orders may stay "Wacht op betaling" although the customer paid. Errors elsewhere in the payment flow (creating the payment, the checkout) arrive as a normal new issue (type 1); treat them with the steps below as well.

Response Steps

  1. Click the Sentry link immediately

  2. Determine the error type:

  3. Webhook processing failure (payment status update failed). A temporary Mollie error (network, time-out, 429, 5xx) answers 503 so Mollie retries; those are logged and only need action when they keep coming
  4. Payment creation failure (Mollie API error)
  5. Checkout flow error (cart/order creation)
  6. Invoice generation error

  7. Check payment logs:

    docker compose logs backend --tail=100 --since=30m | grep -i "payment\|mollie\|checkout"
    

  8. Check Mollie dashboard:

  9. Log in to https://my.mollie.com
  10. Go to Payments > check recent payment statuses
  11. Are payments being created but failing at Mollie?

  12. Check Mollie API status:

  13. Visit https://status.mollie.com
  14. If Mollie is down, customers will see payment failures until service recovers

  15. Verify webhook connectivity:

    # Does Mollie reach our webhook endpoint?
    docker compose logs nginx --tail=200 | grep "/api/payments/webhook/"
    

  16. If the error is in our code:

  17. Check the traceback in Sentry for the exact failure point
  18. Common locations: apps/payments/views.py (webhook), apps/payments/services.py (create_payment), apps/orders/
  19. If a quick fix is possible, deploy immediately

  20. Check for stuck orders (paid at Mollie, still pending here). The hourly task monitoring.recover_incomplete_paid_orders (:45) re-runs invoice, mail, admin notice, print files and the shop mail for orders that are already paid; it does not change an order that is still pending:

    docker compose exec backend python manage.py shell -c "
    from apps.orders.models import Order
    from django.utils import timezone
    from datetime import timedelta
    stuck = Order.objects.filter(
        status='pending',
        created_at__lt=timezone.now() - timedelta(hours=1)
    )
    print('Pending > 1h:', stuck.count())
    for o in stuck[:10]:
        print(o.order_number, o.created_at)
    "
    
    For an order that Mollie shows as paid, open the payment in Mollie and resend the webhook, or set the order to "Betaald" in the admin once the Mollie status is confirmed.

  21. Customer communication:

  22. If payment failures are widespread, consider adding a banner to the site

Common Causes

  • Mollie API outage or degradation
  • Mollie API key expired or revoked
  • Webhook URL not reachable (nginx misconfiguration, SSL issue)
  • Code bug in payment flow (especially after deployment)
  • Currency/amount formatting error

Alert Type 5: Payment Anomaly

Arrives: #payments Severity: CRITICAL Source: Celery task monitoring.check_payment_anomalies, every 15 minutes. It alerts when the last hour has at least 3 failed payments or a failure rate above 50 % (PAYMENT_ANOMALY_FAILURE_COUNT, PAYMENT_ANOMALY_FAILURE_RATE, PAYMENT_ANOMALY_WINDOW_HOURS), and stays silent under 3 payments in the window. What it means: Multiple payment failures in a short period. This is a systemic payment failure, not an isolated error. The same channel gets the daily summary at 07:00 Dutch time (informational) and the "🔧 Hersteltaak" message when the recover task repaired an order.

Response Steps

  1. Follow all steps from Alert Type 4 (Payment Error)

  2. Check if ALL payments are failing:

    docker compose exec backend python manage.py shell -c "
    from apps.payments.models import Payment
    from django.utils import timezone
    from datetime import timedelta
    since = timezone.now() - timedelta(hours=1)
    qs = Payment.objects.filter(created_at__gte=since)
    print('Last hour:', qs.count(), 'payments')
    for s in ('paid', 'failed', 'expired', 'canceled', 'open', 'pending'):
        print(f'  {s}: {qs.filter(status=s).count()}')
    "
    

  3. If 100% failure rate:

  4. Likely a Mollie outage or API key issue
  5. Check MOLLIE_API_KEY is set and is a live_ key: docker compose exec backend sh -c 'echo ${MOLLIE_API_KEY%%_*}'
  6. Consider temporarily disabling checkout with a maintenance message

  7. If partial failure rate:

  8. Likely a specific payment method or amount issue; with the "Alle betaalmethoden" switch on, a method can fail at Mollie while iDEAL works (see Betaalmethoden)
  9. Check Sentry for common attributes across the failing payments

  10. Monitor resolution:

  11. The task alerts every 15 minutes as long as the window is above the threshold
  12. Verify in Mollie that new payments succeed

Common Causes

  • Same as Alert Type 4, but systemic
  • API key rotation without updating env var
  • Mollie account suspension (check email)
  • Network issue between server and Mollie API

Alert Type 6: Backup Failure

Arrives: #backups Severity: High Source: Django backup task (Celery) Discord notification What it means: A scheduled database backup failed. Data protection is compromised until the next successful backup.

Response Steps

  1. Check the failure details in the Discord notification (includes error message and task ID)

  2. Check backup task logs:

    docker compose logs celery --tail=100 --since=2h | grep -i "backup"
    

  3. Try a manual backup:

    docker compose exec backend python manage.py backup_database
    
    If the manual backup succeeds, the failure was likely transient.

  4. Check disk space:

    docker compose exec backend df -h /app/backups
    
    Backups will fail if the disk is full.

  5. Check database connectivity:

    docker compose exec backend python manage.py dbshell -c "SELECT 1;"
    

  6. Check backup storage:

    # Configuration and the latest records
    docker compose exec backend python manage.py backup_health
    # Backups in the bucket
    docker compose exec backend python manage.py backup_database --list
    

  7. Verify backup schedule is running:

    docker compose ps celery-beat          # healthy?
    docker compose logs celery-beat --tail=20
    
    Look for the backup task being scheduled.

  8. If disk space is the issue:

    # Make a backup and delete old ones from the bucket afterwards (--keep-count, default 30)
    docker compose exec backend python manage.py backup_database --cleanup-old
    

Common Causes

  • Disk full (backup volume)
  • Database connection timeout during dump
  • PostgreSQL in recovery mode (after crash)
  • Celery worker crashed or restarted during backup
  • File permission issue on backup directory
  • Spaces key or bucket problem (BACKUP_S3_* in backend/.env.production)
  • A Celery child killed by the memory limit mid-task: the worker has mem_limit 768m on production and --max-memory-per-child=262144 since #990 (v1.54); check docker compose logs celery --tail=200 | grep -i "WorkerLostError\|SIGKILL"
  • RecursionError: maximum recursion depth exceeded at the BackupRecord insert: the slow-query logger stacked one wrapper per database reconnect until #992 (v1.54). That explained the 219 failures up to 2026-09-26 and the ones in October 2026; on the current release this would be a new bug, not the old one

Alert Type 7: Missed Backup Warning

Arrives: #backups Severity: High Source: Daily backup summary task (backup.daily_summary, 06:00 Dutch time) with the field "Missed Backup Warning" What it means: No successful backup has been recorded in the last 25 hours. The backup system may be silently broken.

Response Steps

  1. Check if backups are running at all:

    docker compose ps celery celery-beat
    docker compose logs celery-beat --tail=50 | grep -i "backup"
    
    An unhealthy celery-beat is restarted by cron within 5 minutes and reported in #errors (type 10). If it keeps failing, see that section.

  2. Check Celery worker health:

    docker compose logs celery --tail=50
    
    If the Celery worker is not running, backups cannot execute.

  3. Check backup monitor metrics:

    docker compose exec backend python manage.py shell -c "
    from apps.core.services.backup_monitor import BackupMonitor
    monitor = BackupMonitor()
    metrics = monitor.get_metrics()
    print(f'Last backup: {metrics.get(\"last_backup_time\", \"Never\")}')
    print(f'Success rate: {metrics.get(\"success_rate\", \"Unknown\")}%')
    print(f'Total backups: {metrics.get(\"total_backups\", 0)}')
    "
    

  4. Check Redis (backup history is cached):

    docker compose exec redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping'
    
    If Redis is down, backup history may be lost (but backups themselves still run).

  5. Run a manual backup to restore the schedule:

    docker compose exec backend python manage.py backup_database
    

  6. If Celery Beat is not scheduling tasks:

    docker compose restart celery-beat
    docker compose logs celery-beat --tail=20
    

Common Causes

  • Celery Beat stalled (now caught by its healthcheck) or not started
  • Celery worker crashed (tasks scheduled but not executed)
  • Redis down (task results not stored, but tasks may still run)
  • Server reboot without docker compose up

Alert Type 8: Disk Space Alert

Arrives: #errors Severity: Warning (≥ 80 %) or Critical (≥ 90 %) Source: Celery task monitoring.check_disk_space (hourly at :30; thresholds DISK_SPACE_WARNING_PCT, DISK_SPACE_CRITICAL_PCT). It measures / as seen from the backend container. What it means: Server disk usage has crossed a threshold. If it reaches 100%, the application will crash.

Response Steps

  1. Check current disk usage:

    df -h
    

  2. Identify what is consuming space:

    sudo du -sh /var/lib/docker/volumes/* | sort -rh | head -10
    

  3. Common space consumers:

Docker images and containers:

docker system df
# Clean up unused images and containers (keeps the predeploy tags that are in use)
docker system prune -f

Local backup files (the backup_volume; the bucket is cleaned by the daily task):

docker compose exec backend ls -lah /app/backups/ | head -20

Container logs are capped since #657 (5 × 10 MB per container). db, redis and nginx only got that cap after a recreate; check with sudo du -sh /var/lib/docker/containers/*/ and, if one is large, docker compose up -d --no-deps db redis nginx (short restart of those three).

PostgreSQL WAL files:

docker compose exec db du -sh /var/lib/postgresql/data/pg_wal/

  1. If above 90% (critical):
  2. Delete old Docker images: docker image prune -a -f
  3. Consider increasing disk size at Hetzner

  4. If above 95% (emergency):

  5. Delete the oldest local backup files in /app/backups/
  6. Increase disk immediately at Hetzner Cloud console

Common Causes

  • Docker images accumulating from deployments without cleanup
  • PostgreSQL WAL files growing (missing checkpoint)
  • A container created before the log cap (#657) with a huge json log
  • Media uploads filling disk (should be on DigitalOcean Spaces, not local)

Alert Type 9: Admin Delete Action

Arrives: #production-admin-actions Severity: Info Source: Admin audit trail (apps/core/audit.py): every delete and bulk_delete through the admin API. The same channel carries the "paid order" message for every paid order, stock problems found after payment, and refunds that failed or were canceled at Mollie (apps/orders/tasks.py); those need no response beyond reading them. Since v1.53 it also gets "⚠️ Informer: voor order " (apps/informer/notify.py) when a sales invoice could not be booked in Informer; the order in Admin → Bestellingen shows the reason and "Nu bijwerken" retries, and informer.sweep retries temporary errors every 5 minutes on its own. What it means: An admin user performed a delete action in the admin interface. Deletes are irreversible and logged for audit purposes.

Response Steps

  1. Review the notification details:
  2. Who performed the delete?
  3. What model/object was deleted?
  4. When did it happen?

  5. Verify the delete was intentional:

  6. Check with the admin user if this was planned
  7. If unexpected, check the admin audit log for more context:

    docker compose exec backend python manage.py shell -c "
    from apps.core.models import AdminAuditLog
    recent = AdminAuditLog.objects.filter(action__in=['delete', 'bulk_delete']).order_by('-timestamp')[:5]
    for log in recent:
        print(f'{log.timestamp}: {log.user} {log.action} {log.model_name} (ID: {log.object_id}) {log.object_repr}')
    "
    

  8. If the delete was accidental:

  9. Check if a backup exists from before the delete
  10. Consult Disaster Recovery for restore instructions
  11. Note: Restoring a full backup will overwrite all changes since the backup

  12. No action needed if:

  13. The delete was intentional and expected
  14. This is informational only -- most delete notifications require no response

Common Causes

  • Normal admin operations (removing test data, cleaning up old records)
  • Accidental bulk delete (admin list view checkbox + delete action)
  • Admin cleanup of expired/orphaned data

Alert Type 10: celery-beat restarted

Arrives: #errors, footer "restart-unhealthy op productie" (or "staging") Severity: Medium; High when it repeats Source: cron job /etc/cron.d/webshop-restart-unhealthy → scripts/restart-unhealthy.sh (every 5 minutes). Orange "🔁 celery-beat herstart": Docker reported the container unhealthy (no heartbeat for 10 minutes) and the script restarted it. Red "⚠️ celery-beat nog steeds ongezond": it was restarted less than 30 minutes ago and is unhealthy again; the script does not restart it again. Red "❌ Herstart … mislukt": docker compose restart failed. What it means: The scheduler that dispatches backups, payment checks and mails stalled. One orange message with a healthy container afterwards needs no action.

Response Steps

  1. Is it healthy now?

    docker compose ps celery-beat
    journalctl -t restart-unhealthy --since "2 hours ago"
    

  2. Red message, or orange messages every half hour: read the logs

    docker compose logs celery-beat --tail=100
    

  3. Permission denied on /app/celerybeat: the schedule volume is owned by root (#811). Fix:
    docker compose run --rm --no-deps --user root --entrypoint sh celery-beat -c 'chown -R appuser:appuser /app/celerybeat'
    docker compose up -d celery-beat
    
  4. Redis unreachable: beat cannot publish tasks; check docker compose ps redis
  5. A corrupt celerybeat-schedule file: stop beat, delete the file in the volume (docker compose run --rm --no-deps --entrypoint sh celery-beat -c 'rm -f /app/celerybeat/celerybeat-schedule*') and start it again; the schedule is rebuilt from config/celery.py

  6. Afterwards: check that the next scheduled task ran (docker compose logs celery --tail=50), and look in #backups for the next hourly backup.

Background and the test procedure: Monitoring.


Escalation Procedures

For a solo-operator setup, "escalation" means increasing your own response urgency.

Severity Levels

Severity Response Time Description
Info Next business day Informational, no action usually needed
Medium Within 4 hours Investigate, may need a fix in next deployment
High Within 1 hour Needs prompt attention, potential data or service impact
Critical Within 15-30 min Active incident affecting customers or revenue

When to Consider Downtime

Take the site into maintenance mode if:

  • Payment system is completely broken (100% failure rate for >30 min)
  • Error rate exceeds 100/hour and rising (cascading failure)
  • Database is corrupt or unreachable
  • Security breach suspected (unauthorized admin access)

Maintenance Mode

# Stop the frontend to prevent new orders (nginx then answers 502 for pages; the API stays up)
docker compose stop frontend
# Fix the issue
# Restart
docker compose up -d frontend

Log Locations

Quick reference for finding logs during incident response. Every service logs to stdout only (Docker json-file, max 5 × 10 MB per container).

Service Log Command What It Contains
Backend (Django) docker compose logs backend --tail=N API requests, errors, application logs
Celery worker docker compose logs celery --tail=N Task execution, backup results
Celery Beat docker compose logs celery-beat --tail=N Task scheduling
Nginx docker compose logs nginx --tail=N HTTP requests, proxy errors (/health is not logged)
PostgreSQL docker compose logs db --tail=N Database queries, connections
Redis docker compose logs redis --tail=N Cache operations, connection issues
restart-unhealthy (cron) journalctl -t restart-unhealthy celery-beat restarts

Filtering Logs by Time

# Last hour
docker compose logs backend --since=1h

# Last 30 minutes
docker compose logs backend --since=30m

# Since specific time (UTC)
docker compose logs backend --since="2026-01-15T10:00:00"

Searching Logs

# Find all errors
docker compose logs backend --tail=500 | grep "ERROR"

# Find payment-related logs
docker compose logs backend --tail=500 | grep -i "payment\|mollie"

# Find backup-related logs
docker compose logs celery --tail=200 | grep -i "backup"