Skip to content

Monitoring

Monitoring runbook for the Freeze Design webshop: error tracking, scheduled tasks, health checks, analytics and alert routing for a solo-operator setup.

Environments

Staging (https://staging.freezedesign.nl, basic auth) and production (https://www.freezedesign.nl, v1.60 since 2026-11-06) are both live. Production is a single domain with the API under /api (no api. subdomain), the same model as staging; freezedesign.nl and prod.freezedesign.nl redirect to www. DNS is at Openprovider and points straight at the VPSes; nothing sits in front of the shop.

On the server

The docker compose commands below run in /opt/webshop with the compose file selected: export COMPOSE_FILE=docker-compose.prod.yml (staging: docker-compose.staging.yml). Neither server has a docker-compose.yml.

PostHog: instrumented, delivery depends on the env key

PostHog is instrumented on the frontend (PostHogProvider fires $pageview and other captures), but the provider self-disables when NEXT_PUBLIC_POSTHOG_KEY is unset — so events only reach PostHog in an environment where that key is configured. If a dashboard looks empty, verify the key is set for that environment rather than assuming the code is broken. PostHog is not an error tracker regardless — Sentry is the working error tracker.

Monitoring Stack

Tool Purpose Status (2026-11-01) Dashboard
Sentry Error tracking and performance, backend and frontend Live; alerts go by e-mail (see below) voorman.sentry.io
Django + Celery Beat Backups, disk space, payment anomalies, daily summaries → Discord webhooks, and since v1.59 also ntfy (ntfy notifications) Live Admin → Systeemstatus (/api/admin/health/)
Docker healthchecks + cron Container health; celery-beat restarted automatically when unhealthy Live (#798, #812) docker compose ps
healthchecks.io Dead man's switch: worker, beat and database (every 5 min) and ntfy reachable from outside (every 10 min); see Health pings In code since v1.59; pings only where HEALTHCHECK_PING_URL / NTFY_HEARTBEAT_PING_URL are set healthchecks.io
GitHub Actions Deploys, E2E smoke after every staging deploy (#861), weekly SSL renewal + expiry check, weekly backup test → Discord Live Actions tab
External uptime monitor Reachability from outside the VPS Not set up. Todo externe-uptime-monitor (status: ready); see External uptime monitor —
PostHog Product analytics and session replay (needs NEXT_PUBLIC_POSTHOG_KEY) Live eu.posthog.com

No paid monitoring services are required.


Sentry Setup

Sentry captures application errors from both the Django backend (project freeze-design-backend) and the Next.js frontend (freeze-design-frontend), org voorman.

Backend (Django)

backend/config/settings.py initialises sentry-sdk when SENTRY_DSN is set (and not under tests), with the Django, Celery (monitor_beat_tasks=False: no Sentry Crons, see below) and Redis integrations, traces_sample_rate from SENTRY_TRACES_SAMPLE_RATE (default 0.1), profiles_sample_rate=0.1 and send_default_pii=False. The noisy django.security.DisallowedHost logger (scanners hitting the bare IP) is ignored.

Sentry Crons is off. The plan includes one cron monitor, so every beat task Sentry created a monitor for was disabled or raised false "missed check-in" alerts. Errors in Celery tasks still reach Sentry. Beat tasks are watched by healthchecks.io (monitoring.healthcheck_ping, monitoring.ntfy_public_heartbeat; runbook ntfy-meldingen.md, Health pings) and by the celery-beat heartbeat with restart-unhealthy.sh (alerting.md).

Frontend (Next.js)

@sentry/nextjs captures browser errors (frontend/lib/sentryClient.ts, started by SentryProvider and the error pages), plus server and edge errors (sentry.server.config.ts, sentry.edge.config.ts). All three use tracesSampleRate: 0.1. Replays are recorded only when an error occurs (replaysSessionSampleRate: 0, replaysOnErrorSampleRate: 1.0): the replay budget is 50 per month. Password-reset keys, order ids and admin pages are scrubbed before an event leaves the browser (lib/privacyUrl.ts, #740).

Environment Variables

Set these in backend/.env.production (and backend/.env.staging with adjusted values):

Variable Example Purpose
SENTRY_DSN https://key@o0.ingest.de.sentry.io/0 Backend project DSN
SENTRY_ENVIRONMENT production Must be production for the production alert rules to fire
SENTRY_TRACES_SAMPLE_RATE 0.1 Performance sampling (10%)
SENTRY_RELEASE (unset) Optional override of the release; see below
SENTRY_API_TOKEN (read-only token) Admin "Website" tiles and the weekly website check read issue counts with it; not used for sending

The frontend DSN is baked into the image at build time: NEXT_PUBLIC_SENTRY_DSN (staging) and PRODUCTION_NEXT_PUBLIC_SENTRY_DSN (production) are GitHub secrets, and deploy-production.yml refuses to build a production frontend with the staging DSN.

Release tracking

Sentry needs a release to tell which deploy introduced or reintroduced a bug. The backend takes it from SENTRY_RELEASE if set, otherwise from GIT_SHA, which the Dockerfile stores as an env var from the build-arg that build-images.yml and deploy-production.yml pass (config/sentry.py, resolve_release). GIT_SHA=unknown (a local build without the arg) counts as no release. Nothing to configure on the server.

Check what the running backend reports:

docker compose exec backend python -c "import sentry_sdk; print(sentry_sdk.get_client().options.get('release'))"

Sentry Alerts

What is actually configured (checked 2026-10-17)

Sentry sends e-mail only; the Sentry Discord bot is not installed. These issue alerts exist in org voorman for production:

Alert Projects Fires on To
Productie backend: high priority issues freeze-design-backend new or escalated high-priority issue issue owners, else all members
Productie frontend: high priority issues freeze-design-frontend new or escalated high-priority issue issue owners, else all members
Productie: nieuw of terugkerend issue (niet hoog) both new or regressed issue below high priority, environment:production Ivar (e-mail), at most every 30 min
Productie: fout in de betaal-webhook (Mollie) freeze-design-backend every error event with transaction:/api/payments/webhook/, environment:production Ivar (e-mail), at most every 5 min

There are no metric alerts (error-rate thresholds). The weekly website check (skill .claude/skills/websitecheck/, Saturday 07:30, over the last full Monday–Sunday week) reviews open issues; issues older than 14 days stay open only with a todo or a reason. How to respond to each alert is in the Alerting runbook.

Changing or adding a rule: Sentry → Alerts → the rule → Edit. Keep the environment filter on production; staging reports to the same projects with environment:staging and must not page.

Optional: route Sentry alerts to Discord

Not done. If e-mail turns out to be too slow, install the Sentry Discord integration (Sentry → Settings → Integrations → Discord → Install, authorise the bot on the server), turn on Developer Mode in Discord (User Settings → Advanced) to copy numeric channel IDs, and add the action "Send a Discord notification" with that ID to the rules above. The bot needs Send Messages and Embed Links in the target channels. Separate channels for #errors and #payments already exist for the Django webhooks (see Discord Webhooks), so the bot could post there too.


Celery Beat schedule

All periodic work is scheduled in backend/config/celery.py (beat_schedule) and runs in the celery worker; celery-beat only dispatches. Times are Dutch time (CELERY_TIMEZONE = TIME_ZONE = "Europe/Amsterdam", so 06:00 is 06:00 on the clock in summer and winter; #962, #968).

When (Amsterdam) Task What it does Reports to
every hour :00 backup.daily_backup Database backup to Spaces #backups (success and failure)
every hour :20 cart.send_reminders Abandoned-cart mail, only when the admin switch is on —
every hour :25 informer.refresh_payments Reads the payment status of invoices sent to Informer (bookkeeping, feedback #57); does nothing while the link is off in Admin → Boekhouding #production-admin-actions on failure
every hour :30 monitoring.check_disk_space Disk usage of / in the backend container; warning ≥ 80 %, critical ≥ 90 % (DISK_SPACE_WARNING_PCT, DISK_SPACE_CRITICAL_PCT) #errors
every hour :40 newsletter.sync_pending Confirmed newsletter addresses that are not yet in the Resend contacts (/contacts, optional RESEND_SEGMENT_ID) —
every hour :45 monitoring.recover_incomplete_paid_orders Paid orders without invoice, paid mail, admin notice, print file or shop mail (when shop_order_email_enabled is on) are rescheduled (catches a missed Mollie webhook) #payments when it had to repair something
every 5 min monitoring.healthcheck_ping Pings healthchecks.io when the database answers, else /fail; does nothing without HEALTHCHECK_PING_URL healthchecks.io
every 10 min (:02, :12, …) monitoring.ntfy_public_heartbeat Silent message on freeze-ai-heartbeat through the public ntfy URL, then a ping; does nothing without NTFY_HEARTBEAT_PING_URL healthchecks.io
every 15 min monitoring.check_payment_anomalies Last hour: ≥ 3 failed payments or > 50 % failure rate (PAYMENT_ANOMALY_* settings); skipped under 3 payments #payments
every 5 min informer.sweep Retries Informer bookings after a temporary error and picks up paid orders that were never sent; does nothing while the link is off #production-admin-actions on failure
02:10 daily designs.purge_guest_emails Deletes the guest e-mail address from designs without an order after 12 months (apps/designs/retention.py) —
02:20 daily designs.purge_guest_designs Deletes guest designs without an order 90 days after their last change (#1018) —
02:30 daily cart.purge_stale Deletes guest carts without an order that were not updated for 30 days, then expired sessions (clearsessions; apps/orders/cart_cleanup.py, Synthetics) —
03:00 daily backup.cleanup_old_backups Retention: 24 hourly, 7 daily, 4 weekly #backups on failure
04:00 daily backup.verify_latest_backup Latest backup exists, > 1 MB, checksum matches, < 2 h old #backups on failure
06:00 daily backup.daily_summary Last 24 h of backups; Missed Backup Warning when the last success is > 25 h old #backups
07:00 daily monitoring.daily_payment_summary Orders paid in the last 24 h and their revenue: sign of life for the channel #payments
08:15 daily reviews.send_requests Google review request after delivery, only when the admin switch is on (max 100 per run) —
Monday 06:00 analytics.save_website_week Stores last week's PostHog and Sentry numbers (WebsiteWeekSnapshot) for the admin "Website" tiles; production only (WEBSITE_STATS_ENVIRONMENT, WEBSITE_STATS_HOST) —
1st of month 05:00 backup.test_restore Restores the latest backup into a temporary database #backups

Check that beat dispatched something recently: docker compose logs celery-beat --tail=20 (lines like Scheduler: Sending due task …) and that the worker ran it: docker compose logs celery --tail=50.

Worker memory. The worker runs with --max-memory-per-child=262144 (KiB) in both compose files: a child that is above 256 MiB after a task is replaced, so growth does not stack up. Production also has mem_limit: 768m on the celery service (#990, v1.54); with 512 MB the worker sat at 80 % and a print-file or invoice-PDF peak could kill a child mid-task.

Backup failures until v1.54 (#992): apps/core/instrumentation.py installed one SlowQueryLogger wrapper per database reconnect, so a long-lived Celery child stacked hundreds of layers and every query ended in RecursionError. That is what the hourly backup (at the BackupRecord insert, after the upload) and informer.sweep failed on, and what caused the 219 backup failures up to 2026-09-26; a redeploy hid it every time. The wrapper is now installed once per connection. A RecursionError in #backups on the current release is new and needs a look.


Celery-beat: healthcheck en automatische herstart

Beat plant alle periodieke taken (backups, betaalcontroles, winkelwagen-herinneringen, reviewverzoeken, nieuwsbrief-inhaalslag). Sinds #798 raakt de scheduler (config.beat:HeartbeatScheduler) bij elke tick /app/celerybeat/heartbeat aan en kijkt de healthcheck (scripts/check_beat_heartbeat.py, elke 60 s) hoe oud dat bestand is: ouder dan tien minuten (CELERY_BEAT_HEARTBEAT_MAX_AGE=600) is unhealthy in docker compose ps. Het schedule-bestand en de hartslag staan in het volume celerybeat_schedule; de deploy zet de eigenaar van dat volume op appuser (#811).

Docker herstart een ongezonde container niet vanzelf (restart: unless-stopped geldt alleen voor een container die stopt). Daarvoor draait op elke server een cronregel (/etc/cron.d/webshop-restart-unhealthy), elke vijf minuten, als deploy:

*/5 * * * * deploy cd /opt/webshop && [ -x scripts/restart-unhealthy.sh ] && scripts/restart-unhealthy.sh docker-compose.prod.yml celery-beat 2>&1 | logger -t restart-unhealthy

scripts/restart-unhealthy.sh (komt bij elke deploy mee) herstart de service bij unhealthy en meldt dat in Discord op de webhook DISCORD_WEBHOOK_ERRORS uit het env-bestand van de server (oranje: herstart; rood: nog steeds ongezond, of herstart mislukt), en sinds v1.59 ook op ntfy-topic freeze-errors (NTFY_URL/NTFY_TOKEN uit hetzelfde env-bestand). Hooguit één herstart per 30 minuten per service (RESTART_MIN_INTERVAL_MIN); daarna alleen melden, want een beat die direct weer omvalt is zelf een storing. Log op de server: journalctl -t restart-unhealthy.

Op een bestaande server zetten (nieuwe servers krijgen dit van scripts/provision-server.sh; op staging is de compose-file docker-compose.staging.yml):

ssh freeze-staging "printf '%s\n' '*/5 * * * * deploy cd /opt/webshop && [ -x scripts/restart-unhealthy.sh ] && scripts/restart-unhealthy.sh docker-compose.staging.yml celery-beat 2>&1 | logger -t restart-unhealthy' | sudo tee /etc/cron.d/webshop-restart-unhealthy > /dev/null && sudo chmod 644 /etc/cron.d/webshop-restart-unhealthy"

Testen zonder storing: bevries beat met docker compose kill -s STOP celery-beat. De heartbeat veroudert, na ruim tien minuten is de container unhealthy, en binnen vijf minuten daarna herstart cron hem en staat er een bericht in Discord. Daarna docker compose ps: healthy.

Blijft beat ongezond (rode melding "nog steeds ongezond"): kijk in docker compose logs --tail=100 celery-beat. Een Permission denied op /app/celerybeat is het eigenaarsprobleem uit #811; een nieuwe deploy zet het recht, of met de hand: docker compose run --rm --no-deps --user root --entrypoint sh celery-beat -c 'chown -R appuser:appuser /app/celerybeat' en dan docker compose up -d celery-beat.


Health Check Endpoints

Three levels, from cheap to complete:

Endpoint Answered by Checks Used by
GET /health nginx itself (return 200, not logged) only that nginx runs nginx container healthcheck
GET /api/health/ Django PostgreSQL (SELECT) and Redis read/write; < 50 ms; 503 on failure backend container healthcheck, deploy smoke test, external monitor (to do)
GET /api/health/detailed/ Django, staff only the above plus Celery worker, S3 and the other services Admin → Systeemstatus (/api/admin/health/) and hand checks

Request

curl https://www.freezedesign.nl/api/health/

Healthy Response (HTTP 200)

{
  "status": "healthy"
}

The public response intentionally contains no component status, infrastructure names, deployment metadata or exception details. The deploy workflows verify the running backend image's OCI revision label instead — see Deployment.

Unhealthy Response (HTTP 503)

{
  "status": "unhealthy"
}

Dependency failures are logged server-side. Use the staff-only detailed endpoint or the admin health dashboard when component-level diagnostics are required. Celery worker status is not part of /api/health/ (the Celery inspect call blocks for ~2 seconds, too slow for Docker health checks).


External uptime monitor (to do)

Nothing outside the VPS checks whether the shop is reachable (todo .planning/todos/pending/externe-uptime-monitor.md, status: ready, 2026-10-07). Sentry only sees errors from the app itself, the Docker healthchecks run on the server, and the weekly expiry-check in ssl-renewal.yml only looks at certificates. Until a monitor exists, a dead server is noticed when someone looks.

What the todo asks Ivar to set up (UptimeRobot or similar, free plan suffices):

  1. Account: check first whether one already exists.
  2. Monitors (HTTP(s), every 5 minutes, alert after 2 consecutive failures when the plan allows it):
    • https://www.freezedesign.nl/api/health/, expected status 200 — database and Redis;
    • https://www.freezedesign.nl/, expected status 200 — the frontend.
    • Optionally the same two on staging.freezedesign.nl (basic auth: the monitor needs the credentials, or use /health, which nginx answers without auth).
  3. Notification: e-mail, or a Discord webhook on a channel such as #monitoring (create the channel and webhook as in Discord Webhooks; the URL never goes into the repo). Check which notification channels the chosen plan includes.
  4. Test without an outage: add a temporary monitor on a URL that returns 404, e.g. https://www.freezedesign.nl/bestaat-niet, wait for the "down" notification, then delete that monitor.
  5. SSL expiry: only if the plan has it (UptimeRobot's free plan does not). Otherwise ssl-renewal.yml → expiry-check (weekly, Discord) stays the only warning; note the choice in production-vps-ssl-renewal-setup.
  6. Then update this runbook with the service, plan, monitors and where the alerts go, and the mentions in README.md, infrastructure/hosting.md, architecture/diagrams.md and Alerting.

PostHog Analytics

Status: instrumented; events require the env key

The frontend is instrumented (PostHogProvider captures $pageview and events), but the provider no-ops when NEXT_PUBLIC_POSTHOG_KEY is not set. If a project shows no events, confirm the key is configured for that environment. Use Sentry for errors regardless — PostHog is analytics only.

PostHog provides product analytics, session recording, feature flags, and A/B testing. It runs on the EU-hosted instance for GDPR compliance.

For full setup instructions, custom events, and dashboard configuration, see the PostHog Analytics Guide.

Quick Reference

Variable Value
NEXT_PUBLIC_POSTHOG_KEY Your project API key (build-time secret for the frontend image)
NEXT_PUBLIC_POSTHOG_HOST https://eu.i.posthog.com (default in frontend/lib/posthog.ts)
POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID Backend, read-only: the admin "Website" tiles and the weekly snapshot (POSTHOG_API_HOST defaults to https://eu.posthog.com)
WEBSITE_STATS_ENVIRONMENT, WEBSITE_STATS_HOST Backend; defaults production and www.freezedesign.nl. The tiles and the weekly snapshot count only events with that environment property and host (#1061)

Staging and production share one PostHog project. Since #1061 (v1.57) the frontend puts environment (NEXT_PUBLIC_ENVIRONMENT) on every event, not only on those after the loaded callback (frontend/lib/posthog.ts), and the weekly report and the admin tiles filter on environment = production. Older events may lack the property; the website check uses coalesce(properties.environment, 'production') = 'production' for that reason.

Two different PostHog hosts

The ingestion host (where the SDK sends events, api_host in frontend/lib/posthog.ts) is https://eu.i.posthog.com. The dashboard host where you view data in a browser is https://eu.posthog.com. They are not interchangeable — point the SDK at the ingestion host, and open the dashboard host to inspect events.

PostHog is a client-side analytics tool and does not generate operational alerts. Weekly (Saturday 07:30, cloud routine "Websitecheck"), the website check reads PostHog and Sentry over the last full Monday–Sunday week and writes .planning/weekrapporten/<week>.md in a PR claude/websitecheck-<datum>; websitecheck-discord.yml posts the "Voor Eefje" section to Discord (DISCORD_WEBHOOK_DAGPLANNING).

Monday check of the weekly snapshot. analytics.save_website_week runs Monday 06:00 (Dutch time) and writes the WebsiteWeekSnapshot of the previous week. Whether it did is visible without a Django shell or the 1Password SSH key:

docker compose exec -T backend python manage.py website_weekcijfers_status --aantal 2

Read-only; one line per snapshot (week_start, when it was written in UTC and Amsterdam time, status per tile; --aantal 1–12, default 2; "Geen weeksnapshots." when empty). The freeze-ro <server> weekcijfers wrapper for this (Secretive key, fixed read-only commands) lives in dotfiles-home; todo freeze-ro-weekcijfers-zonder-1password. Missing snapshot: docker compose logs celery --since=24h | grep -i website and, after a fix, manage.py website_weekcijfers_inlezen --weken 1 fills it in again.


Verification

After a change to the monitoring setup, verify each component end-to-end.

Test Sentry alerts

Open a Django shell on staging first (environment:staging: the issue appears in Sentry but no production rule fires), then on production if you want to see the e-mail:

docker compose exec backend python manage.py shell
import sentry_sdk
sentry_sdk.capture_message("Test alert: new issue verification", level="error")

Expected: a new issue in Sentry (Issues → filter on the environment) within a minute, with the release set to the running commit. On production, "Productie: nieuw of terugkerend issue (niet hoog)" mails Ivar within 30 minutes. Mark the test issue as Resolved afterwards. The Mollie-webhook alert cannot be tested this way: it filters on the transaction /api/payments/webhook/, so only a real webhook error triggers it.

Test the Discord webhooks

docker compose exec backend python manage.py shell -c "
from apps.core.services.discord_service import DiscordService
d = DiscordService()
for ch in ('errors', 'payments', 'backups', 'admin'):
    print(ch, d.is_configured(ch), d.send_embed(channel=ch, title='Test from Django', description='webhook ok', color=DiscordService.COLOR_INFO))
"

Each configured channel should print True True and show a blue embed.

Test the celery-beat restart

See "Testen zonder storing" above (kill -s STOP): an orange "🔁 celery-beat herstart" embed in #errors within about 15 minutes.

Verification Checklist

Component Test Method Expected Expected Timing
Sentry issue + e-mail capture_message on production issue in Sentry, mail to Ivar Within 30 min
Backup success wait for the next hour green embed in #backups At :00
Daily backup summary wait for 06:00 Dutch time summary embed in #backups 06:00 Amsterdam
Daily payment summary wait for 07:00 Dutch time embed in #payments 07:00 Amsterdam
Disk-space alert set DISK_SPACE_WARNING_PCT=1 on staging, recreate celery embed in #errors at :30, then restore the value Within 1 hour
celery-beat restart kill -s STOP celery-beat on staging embed in #errors, container healthy again Within 15 min
External monitor see External uptime monitor — —

Troubleshooting

Sentry alerts not firing

  1. Check the environment: rules filter on production.
docker compose exec backend python -c "import os; print(os.getenv('SENTRY_ENVIRONMENT'))"
  1. Check the DSN: docker compose exec backend python -c "import os; print(bool(os.getenv('SENTRY_DSN')))" must print True.
  2. Check the rule: Sentry → Alerts; each rule shows "Active", not "Disabled", and the notification action lists Ivar (or "issue owners").
  3. Check the mailbox: look in spam, and in Sentry → Settings → Account → Notifications that issue alerts are not muted for the projects.

Regressions not detected

Verify the release (see Release tracking). None or unknown means the image was built without GIT_SHA; a normal CI build sets it.

No Discord messages from Django

  1. docker compose exec backend env | grep DISCORD_WEBHOOK — the four URLs must be present. After editing the env file, recreate the containers (up -d backend celery celery-beat); a restart keeps the old environment.
  2. docker compose logs celery --tail=200 | grep -i discord — the service logs a warning instead of raising.
  3. Rate limit: at most 5 messages per channel per minute are sent; the rest is dropped with a warning.
  4. Test the webhook itself: curl -H "Content-Type: application/json" -d '{"content": "test"}' "$DISCORD_WEBHOOK_ERRORS" → HTTP 204.

Health Check Returns 503 But Site Loads

/api/health/ checks database and Redis, not just the process. If it returns 503 while the frontend still renders (cached pages):

  1. Backend logs: docker compose logs backend --tail=100
  2. Database: docker compose exec backend python manage.py dbshell
  3. Redis: docker compose exec redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping' (the password is in the container's environment, not in your shell)

Celery is not part of /api/health/ — check it with docker compose ps celery celery-beat or docker compose exec backend celery -A config inspect ping.

Alert Response Procedures

Site unreachable

  1. From outside: curl -sI https://www.freezedesign.nl/health (nginx) and curl -s https://www.freezedesign.nl/api/health/ (Django).
  2. On the server: docker compose ps — every service Up and healthy?
  3. Backend logs: docker compose logs backend --tail=100
  4. Common causes: database or Redis container down, backend crash loop, out of memory (free -h), disk full (df -h), certificate expired (curl -vI shows the date).
  5. If it started with a deploy, roll back: export DEPLOY_TAG=latest && ./scripts/rollback.sh (production; DEPLOY_TAG=staging on staging), see Deployment.

SSL certificate about to expire

Renewal is automated in two layers (.github/workflows/ssl-renewal.yml, Wednesdays 04:00 UTC, plus the root crontab safety net on each VPS, Sundays 03:30 UTC) — see the Disaster Recovery runbook, SSL Renewal section. The expiry-check job of that workflow posts to Discord (DISCORD_WEBHOOK_CICD) when a certificate expires within 14 days. If it fires:

  1. Check the recent runs of the "SSL Renewal" workflow in GitHub Actions
  2. SSH to the VPS and renew manually (production; use the staging values from the DR runbook on staging):
    ssh freeze-prod 'cd /opt/webshop && sudo env DOMAIN=freezedesign.nl \
        COMPOSE_FILE=/opt/webshop/docker-compose.prod.yml bash scripts/renew-ssl.sh'
    
    (the script reloads nginx automatically on success)
  3. Verify: curl -vI https://www.freezedesign.nl 2>&1 | grep "expire date"
  4. Check the cron safety net: sudo crontab -l | grep renew-ssl

Discord Channel Summary

All application messages go through DiscordService (backend/apps/core/services/discord_service.py), one webhook per channel from the env file. Since v1.59 it forwards every message to ntfy as well (NotifyService, see ntfy notifications). Sentry does not post to Discord.

Channel Env var / secret Source Messages
#errors DISCORD_WEBHOOK_ERRORS monitoring.check_disk_space; scripts/restart-unhealthy.sh (cron) Disk space ≥ 80 % / 90 %; celery-beat restarted or still unhealthy
#payments DISCORD_WEBHOOK_PAYMENTS monitoring.check_payment_anomalies, daily_payment_summary, recover_incomplete_paid_orders Payment anomaly, daily summary (07:00), repaired orders
#backups DISCORD_WEBHOOK_BACKUPS backup.* tasks, backup_database command Backup success/failure, cleanup, verification, restore test, daily summary (06:00)
#production-admin-actions DISCORD_WEBHOOK_ADMIN Admin audit (delete, bulk_delete), apps/orders/tasks.py, apps/informer/notify.py Admin deletes, paid orders, stock problems, failed or canceled refunds, Informer booking failures ("⚠️ Informer: …")
CI/CD channels GitHub secrets DISCORD_WEBHOOK_URL, DISCORD_WEBHOOK_CICD, DISCORD_WEBHOOK_PRODUCTION, DISCORD_WEBHOOK_STAGING_BACKUPS, DISCORD_WEBHOOK_DAGPLANNING Workflows Deploys, E2E smoke after a staging deploy, CI results, SSL renewal and expiry, weekly backup test, k6 gate, daily planning, website check
#monitoring — External uptime monitor Not set up yet, see above

Details and setup steps: Discord Webhooks.