Monitoring¶
Monitoring runbook for the Freeze Design webshop: error tracking, scheduled tasks, health checks, analytics and alert routing for a solo-operator setup.
Environments
Staging (https://staging.freezedesign.nl, basic auth) and production
(https://www.freezedesign.nl, v1.60 since 2026-11-06) are both live. Production is a
single domain with the API under /api (no api. subdomain), the same model as
staging; freezedesign.nl and prod.freezedesign.nl redirect to www. DNS is at
Openprovider and points straight at the VPSes; nothing sits in front of the shop.
On the server
The docker compose commands below run in /opt/webshop with the compose file
selected: export COMPOSE_FILE=docker-compose.prod.yml (staging:
docker-compose.staging.yml). Neither server has a docker-compose.yml.
PostHog: instrumented, delivery depends on the env key
PostHog is instrumented on the frontend (PostHogProvider fires $pageview
and other captures), but the provider self-disables when
NEXT_PUBLIC_POSTHOG_KEY is unset — so events only reach PostHog in an environment
where that key is configured. If a dashboard looks empty, verify the key is set for
that environment rather than assuming the code is broken. PostHog is not an
error tracker regardless — Sentry is the working error tracker.
Monitoring Stack¶
| Tool | Purpose | Status (2026-11-01) | Dashboard |
|---|---|---|---|
| Sentry | Error tracking and performance, backend and frontend | Live; alerts go by e-mail (see below) | voorman.sentry.io |
| Django + Celery Beat | Backups, disk space, payment anomalies, daily summaries → Discord webhooks, and since v1.59 also ntfy (ntfy notifications) | Live | Admin → Systeemstatus (/api/admin/health/) |
| Docker healthchecks + cron | Container health; celery-beat restarted automatically when unhealthy |
Live (#798, #812) | docker compose ps |
| healthchecks.io | Dead man's switch: worker, beat and database (every 5 min) and ntfy reachable from outside (every 10 min); see Health pings | In code since v1.59; pings only where HEALTHCHECK_PING_URL / NTFY_HEARTBEAT_PING_URL are set |
healthchecks.io |
| GitHub Actions | Deploys, E2E smoke after every staging deploy (#861), weekly SSL renewal + expiry check, weekly backup test → Discord | Live | Actions tab |
| External uptime monitor | Reachability from outside the VPS | Not set up. Todo externe-uptime-monitor (status: ready); see External uptime monitor |
— |
| PostHog | Product analytics and session replay (needs NEXT_PUBLIC_POSTHOG_KEY) |
Live | eu.posthog.com |
No paid monitoring services are required.
Sentry Setup¶
Sentry captures application errors from both the Django backend (project
freeze-design-backend) and the Next.js frontend (freeze-design-frontend), org voorman.
Backend (Django)¶
backend/config/settings.py initialises sentry-sdk when SENTRY_DSN is set (and not
under tests), with the Django, Celery (monitor_beat_tasks=False: no Sentry Crons, see below) and Redis integrations,
traces_sample_rate from SENTRY_TRACES_SAMPLE_RATE (default 0.1),
profiles_sample_rate=0.1 and send_default_pii=False. The noisy
django.security.DisallowedHost logger (scanners hitting the bare IP) is ignored.
Sentry Crons is off. The plan includes one cron monitor, so every beat task Sentry
created a monitor for was disabled or raised false "missed check-in" alerts. Errors in
Celery tasks still reach Sentry. Beat tasks are watched by healthchecks.io
(monitoring.healthcheck_ping, monitoring.ntfy_public_heartbeat; runbook
ntfy-meldingen.md, Health pings) and by the celery-beat heartbeat with
restart-unhealthy.sh (alerting.md).
Frontend (Next.js)¶
@sentry/nextjs captures browser errors (frontend/lib/sentryClient.ts, started by
SentryProvider and the error pages), plus server and edge errors
(sentry.server.config.ts, sentry.edge.config.ts). All three use tracesSampleRate: 0.1.
Replays are recorded only when an error occurs (replaysSessionSampleRate: 0,
replaysOnErrorSampleRate: 1.0): the replay budget is 50 per month. Password-reset keys,
order ids and admin pages are scrubbed before an event leaves the browser
(lib/privacyUrl.ts, #740).
Environment Variables¶
Set these in backend/.env.production (and backend/.env.staging with adjusted values):
| Variable | Example | Purpose |
|---|---|---|
SENTRY_DSN |
https://key@o0.ingest.de.sentry.io/0 |
Backend project DSN |
SENTRY_ENVIRONMENT |
production |
Must be production for the production alert rules to fire |
SENTRY_TRACES_SAMPLE_RATE |
0.1 |
Performance sampling (10%) |
SENTRY_RELEASE |
(unset) | Optional override of the release; see below |
SENTRY_API_TOKEN |
(read-only token) | Admin "Website" tiles and the weekly website check read issue counts with it; not used for sending |
The frontend DSN is baked into the image at build time: NEXT_PUBLIC_SENTRY_DSN
(staging) and PRODUCTION_NEXT_PUBLIC_SENTRY_DSN (production) are GitHub secrets, and
deploy-production.yml refuses to build a production frontend with the staging DSN.
Release tracking¶
Sentry needs a release to tell which deploy introduced or reintroduced a bug. The backend
takes it from SENTRY_RELEASE if set, otherwise from GIT_SHA, which the Dockerfile
stores as an env var from the build-arg that build-images.yml and
deploy-production.yml pass (config/sentry.py, resolve_release). GIT_SHA=unknown
(a local build without the arg) counts as no release. Nothing to configure on the server.
Check what the running backend reports:
docker compose exec backend python -c "import sentry_sdk; print(sentry_sdk.get_client().options.get('release'))"
Sentry Alerts¶
What is actually configured (checked 2026-10-17)
Sentry sends e-mail only; the Sentry Discord bot is not installed. These issue alerts
exist in org voorman for production:
| Alert | Projects | Fires on | To |
|---|---|---|---|
| Productie backend: high priority issues | freeze-design-backend |
new or escalated high-priority issue | issue owners, else all members |
| Productie frontend: high priority issues | freeze-design-frontend |
new or escalated high-priority issue | issue owners, else all members |
| Productie: nieuw of terugkerend issue (niet hoog) | both | new or regressed issue below high priority, environment:production |
Ivar (e-mail), at most every 30 min |
| Productie: fout in de betaal-webhook (Mollie) | freeze-design-backend |
every error event with transaction:/api/payments/webhook/, environment:production |
Ivar (e-mail), at most every 5 min |
There are no metric alerts (error-rate thresholds). The weekly website check
(skill .claude/skills/websitecheck/, Saturday 07:30, over the last full Monday–Sunday
week) reviews open issues; issues older than 14 days stay open only with a todo or a
reason. How to respond to each alert is in
the Alerting runbook.
Changing or adding a rule: Sentry → Alerts → the rule → Edit. Keep the environment filter
on production; staging reports to the same projects with environment:staging and must
not page.
Optional: route Sentry alerts to Discord¶
Not done. If e-mail turns out to be too slow, install the Sentry Discord integration
(Sentry → Settings → Integrations → Discord → Install, authorise the bot on the server), turn
on Developer Mode in Discord (User Settings → Advanced) to copy numeric channel IDs, and
add the action "Send a Discord notification" with that ID to the rules above. The bot
needs Send Messages and Embed Links in the target channels. Separate channels for
#errors and #payments already exist for the Django webhooks (see
Discord Webhooks), so the bot could post there too.
Celery Beat schedule¶
All periodic work is scheduled in backend/config/celery.py (beat_schedule) and runs in
the celery worker; celery-beat only dispatches. Times are Dutch time
(CELERY_TIMEZONE = TIME_ZONE = "Europe/Amsterdam", so 06:00 is 06:00 on the clock in
summer and winter; #962, #968).
| When (Amsterdam) | Task | What it does | Reports to |
|---|---|---|---|
| every hour :00 | backup.daily_backup |
Database backup to Spaces | #backups (success and failure) |
| every hour :20 | cart.send_reminders |
Abandoned-cart mail, only when the admin switch is on | — |
| every hour :25 | informer.refresh_payments |
Reads the payment status of invoices sent to Informer (bookkeeping, feedback #57); does nothing while the link is off in Admin → Boekhouding | #production-admin-actions on failure |
| every hour :30 | monitoring.check_disk_space |
Disk usage of / in the backend container; warning ≥ 80 %, critical ≥ 90 % (DISK_SPACE_WARNING_PCT, DISK_SPACE_CRITICAL_PCT) |
#errors |
| every hour :40 | newsletter.sync_pending |
Confirmed newsletter addresses that are not yet in the Resend contacts (/contacts, optional RESEND_SEGMENT_ID) |
— |
| every hour :45 | monitoring.recover_incomplete_paid_orders |
Paid orders without invoice, paid mail, admin notice, print file or shop mail (when shop_order_email_enabled is on) are rescheduled (catches a missed Mollie webhook) |
#payments when it had to repair something |
| every 5 min | monitoring.healthcheck_ping |
Pings healthchecks.io when the database answers, else /fail; does nothing without HEALTHCHECK_PING_URL |
healthchecks.io |
| every 10 min (:02, :12, …) | monitoring.ntfy_public_heartbeat |
Silent message on freeze-ai-heartbeat through the public ntfy URL, then a ping; does nothing without NTFY_HEARTBEAT_PING_URL |
healthchecks.io |
| every 15 min | monitoring.check_payment_anomalies |
Last hour: ≥ 3 failed payments or > 50 % failure rate (PAYMENT_ANOMALY_* settings); skipped under 3 payments |
#payments |
| every 5 min | informer.sweep |
Retries Informer bookings after a temporary error and picks up paid orders that were never sent; does nothing while the link is off | #production-admin-actions on failure |
| 02:10 daily | designs.purge_guest_emails |
Deletes the guest e-mail address from designs without an order after 12 months (apps/designs/retention.py) |
— |
| 02:20 daily | designs.purge_guest_designs |
Deletes guest designs without an order 90 days after their last change (#1018) | — |
| 02:30 daily | cart.purge_stale |
Deletes guest carts without an order that were not updated for 30 days, then expired sessions (clearsessions; apps/orders/cart_cleanup.py, Synthetics) |
— |
| 03:00 daily | backup.cleanup_old_backups |
Retention: 24 hourly, 7 daily, 4 weekly | #backups on failure |
| 04:00 daily | backup.verify_latest_backup |
Latest backup exists, > 1 MB, checksum matches, < 2 h old | #backups on failure |
| 06:00 daily | backup.daily_summary |
Last 24 h of backups; Missed Backup Warning when the last success is > 25 h old | #backups |
| 07:00 daily | monitoring.daily_payment_summary |
Orders paid in the last 24 h and their revenue: sign of life for the channel | #payments |
| 08:15 daily | reviews.send_requests |
Google review request after delivery, only when the admin switch is on (max 100 per run) | — |
| Monday 06:00 | analytics.save_website_week |
Stores last week's PostHog and Sentry numbers (WebsiteWeekSnapshot) for the admin "Website" tiles; production only (WEBSITE_STATS_ENVIRONMENT, WEBSITE_STATS_HOST) |
— |
| 1st of month 05:00 | backup.test_restore |
Restores the latest backup into a temporary database | #backups |
Check that beat dispatched something recently: docker compose logs celery-beat --tail=20
(lines like Scheduler: Sending due task …) and that the worker ran it:
docker compose logs celery --tail=50.
Worker memory. The worker runs with --max-memory-per-child=262144 (KiB) in both compose
files: a child that is above 256 MiB after a task is replaced, so growth does not stack up.
Production also has mem_limit: 768m on the celery service (#990, v1.54); with 512 MB the
worker sat at 80 % and a print-file or invoice-PDF peak could kill a child mid-task.
Backup failures until v1.54 (#992): apps/core/instrumentation.py installed one
SlowQueryLogger wrapper per database reconnect, so a long-lived Celery child stacked
hundreds of layers and every query ended in RecursionError. That is what the hourly
backup (at the BackupRecord insert, after the upload) and informer.sweep failed on, and
what caused the 219 backup failures up to 2026-09-26; a redeploy hid it every time. The
wrapper is now installed once per connection. A RecursionError in #backups on the
current release is new and needs a look.
Celery-beat: healthcheck en automatische herstart¶
Beat plant alle periodieke taken (backups, betaalcontroles, winkelwagen-herinneringen,
reviewverzoeken, nieuwsbrief-inhaalslag). Sinds #798 raakt de scheduler
(config.beat:HeartbeatScheduler) bij elke tick /app/celerybeat/heartbeat aan en kijkt de
healthcheck (scripts/check_beat_heartbeat.py, elke 60 s) hoe oud dat bestand is: ouder dan
tien minuten (CELERY_BEAT_HEARTBEAT_MAX_AGE=600) is unhealthy in docker compose ps.
Het schedule-bestand en de hartslag staan in het volume celerybeat_schedule; de deploy
zet de eigenaar van dat volume op appuser (#811).
Docker herstart een ongezonde container niet vanzelf (restart: unless-stopped geldt alleen
voor een container die stopt). Daarvoor draait op elke server een cronregel
(/etc/cron.d/webshop-restart-unhealthy), elke vijf minuten, als deploy:
*/5 * * * * deploy cd /opt/webshop && [ -x scripts/restart-unhealthy.sh ] && scripts/restart-unhealthy.sh docker-compose.prod.yml celery-beat 2>&1 | logger -t restart-unhealthy
scripts/restart-unhealthy.sh (komt bij elke deploy mee) herstart de service bij
unhealthy en meldt dat in Discord op de webhook DISCORD_WEBHOOK_ERRORS uit het
env-bestand van de server (oranje: herstart; rood: nog steeds ongezond, of herstart mislukt),
en sinds v1.59 ook op ntfy-topic freeze-errors (NTFY_URL/NTFY_TOKEN uit hetzelfde env-bestand).
Hooguit één herstart per 30 minuten per service (RESTART_MIN_INTERVAL_MIN); daarna alleen
melden, want een beat die direct weer omvalt is zelf een storing. Log op de server:
journalctl -t restart-unhealthy.
Op een bestaande server zetten (nieuwe servers krijgen dit van scripts/provision-server.sh;
op staging is de compose-file docker-compose.staging.yml):
ssh freeze-staging "printf '%s\n' '*/5 * * * * deploy cd /opt/webshop && [ -x scripts/restart-unhealthy.sh ] && scripts/restart-unhealthy.sh docker-compose.staging.yml celery-beat 2>&1 | logger -t restart-unhealthy' | sudo tee /etc/cron.d/webshop-restart-unhealthy > /dev/null && sudo chmod 644 /etc/cron.d/webshop-restart-unhealthy"
Testen zonder storing: bevries beat met docker compose kill -s STOP celery-beat. De
heartbeat veroudert, na ruim tien minuten is de container unhealthy, en binnen vijf minuten
daarna herstart cron hem en staat er een bericht in Discord. Daarna docker compose ps:
healthy.
Blijft beat ongezond (rode melding "nog steeds ongezond"): kijk in
docker compose logs --tail=100 celery-beat. Een Permission denied op
/app/celerybeat is het eigenaarsprobleem uit #811; een nieuwe deploy zet het recht, of
met de hand: docker compose run --rm --no-deps --user root --entrypoint sh celery-beat -c 'chown -R appuser:appuser /app/celerybeat' en dan docker compose up -d celery-beat.
Health Check Endpoints¶
Three levels, from cheap to complete:
| Endpoint | Answered by | Checks | Used by |
|---|---|---|---|
GET /health |
nginx itself (return 200, not logged) |
only that nginx runs | nginx container healthcheck |
GET /api/health/ |
Django | PostgreSQL (SELECT) and Redis read/write; < 50 ms; 503 on failure |
backend container healthcheck, deploy smoke test, external monitor (to do) |
GET /api/health/detailed/ |
Django, staff only | the above plus Celery worker, S3 and the other services | Admin → Systeemstatus (/api/admin/health/) and hand checks |
Request¶
Healthy Response (HTTP 200)¶
The public response intentionally contains no component status, infrastructure names, deployment metadata or exception details. The deploy workflows verify the running backend image's OCI revision label instead — see Deployment.
Unhealthy Response (HTTP 503)¶
Dependency failures are logged server-side. Use the staff-only detailed endpoint or the
admin health dashboard when component-level diagnostics are required. Celery worker status
is not part of /api/health/ (the Celery inspect call blocks for ~2 seconds, too slow
for Docker health checks).
External uptime monitor (to do)¶
Nothing outside the VPS checks whether the shop is reachable (todo
.planning/todos/pending/externe-uptime-monitor.md, status: ready, 2026-10-07). Sentry
only sees errors from the app itself, the Docker healthchecks run on the server, and the
weekly expiry-check in ssl-renewal.yml only looks at certificates. Until a monitor
exists, a dead server is noticed when someone looks.
What the todo asks Ivar to set up (UptimeRobot or similar, free plan suffices):
- Account: check first whether one already exists.
- Monitors (HTTP(s), every 5 minutes, alert after 2 consecutive failures when the plan
allows it):
https://www.freezedesign.nl/api/health/, expected status 200 — database and Redis;https://www.freezedesign.nl/, expected status 200 — the frontend.- Optionally the same two on
staging.freezedesign.nl(basic auth: the monitor needs the credentials, or use/health, which nginx answers without auth).
- Notification: e-mail, or a Discord webhook on a channel such as
#monitoring(create the channel and webhook as in Discord Webhooks; the URL never goes into the repo). Check which notification channels the chosen plan includes. - Test without an outage: add a temporary monitor on a URL that returns 404, e.g.
https://www.freezedesign.nl/bestaat-niet, wait for the "down" notification, then delete that monitor. - SSL expiry: only if the plan has it (UptimeRobot's free plan does not). Otherwise
ssl-renewal.yml→expiry-check(weekly, Discord) stays the only warning; note the choice inproduction-vps-ssl-renewal-setup. - Then update this runbook with the service, plan, monitors and where the alerts go,
and the mentions in
README.md,infrastructure/hosting.md,architecture/diagrams.mdand Alerting.
PostHog Analytics¶
Status: instrumented; events require the env key
The frontend is instrumented (PostHogProvider captures $pageview and events), but
the provider no-ops when NEXT_PUBLIC_POSTHOG_KEY is not set. If a project shows no
events, confirm the key is configured for that environment. Use Sentry for errors
regardless — PostHog is analytics only.
PostHog provides product analytics, session recording, feature flags, and A/B testing. It runs on the EU-hosted instance for GDPR compliance.
For full setup instructions, custom events, and dashboard configuration, see the PostHog Analytics Guide.
Quick Reference¶
| Variable | Value |
|---|---|
NEXT_PUBLIC_POSTHOG_KEY |
Your project API key (build-time secret for the frontend image) |
NEXT_PUBLIC_POSTHOG_HOST |
https://eu.i.posthog.com (default in frontend/lib/posthog.ts) |
POSTHOG_PERSONAL_API_KEY, POSTHOG_PROJECT_ID |
Backend, read-only: the admin "Website" tiles and the weekly snapshot (POSTHOG_API_HOST defaults to https://eu.posthog.com) |
WEBSITE_STATS_ENVIRONMENT, WEBSITE_STATS_HOST |
Backend; defaults production and www.freezedesign.nl. The tiles and the weekly snapshot count only events with that environment property and host (#1061) |
Staging and production share one PostHog project. Since #1061 (v1.57) the frontend puts
environment (NEXT_PUBLIC_ENVIRONMENT) on every event, not only on those after the
loaded callback (frontend/lib/posthog.ts), and the weekly report and the admin tiles
filter on environment = production. Older events may lack the property; the website check
uses coalesce(properties.environment, 'production') = 'production' for that reason.
Two different PostHog hosts
The ingestion host (where the SDK sends events, api_host in frontend/lib/posthog.ts)
is https://eu.i.posthog.com. The dashboard host where you view data in a browser
is https://eu.posthog.com. They are not interchangeable — point the SDK at the
ingestion host, and open the dashboard host to inspect events.
PostHog is a client-side analytics tool and does not generate operational alerts. Weekly
(Saturday 07:30, cloud routine "Websitecheck"), the website check reads PostHog and Sentry
over the last full Monday–Sunday week and writes .planning/weekrapporten/<week>.md in a PR
claude/websitecheck-<datum>; websitecheck-discord.yml posts the "Voor Eefje" section to
Discord (DISCORD_WEBHOOK_DAGPLANNING).
Monday check of the weekly snapshot. analytics.save_website_week runs Monday 06:00
(Dutch time) and writes the WebsiteWeekSnapshot of the previous week. Whether it did is
visible without a Django shell or the 1Password SSH key:
Read-only; one line per snapshot (week_start, when it was written in UTC and Amsterdam
time, status per tile; --aantal 1–12, default 2; "Geen weeksnapshots." when empty). The
freeze-ro <server> weekcijfers wrapper for this (Secretive key, fixed read-only commands)
lives in dotfiles-home; todo freeze-ro-weekcijfers-zonder-1password. Missing snapshot:
docker compose logs celery --since=24h | grep -i website and, after a fix,
manage.py website_weekcijfers_inlezen --weken 1 fills it in again.
Verification¶
After a change to the monitoring setup, verify each component end-to-end.
Test Sentry alerts¶
Open a Django shell on staging first (environment:staging: the issue appears in Sentry
but no production rule fires), then on production if you want to see the e-mail:
Expected: a new issue in Sentry (Issues → filter on the environment) within a minute, with
the release set to the running commit. On production, "Productie: nieuw of terugkerend
issue (niet hoog)" mails Ivar within 30 minutes. Mark the test issue as Resolved
afterwards. The Mollie-webhook alert cannot be tested this way: it filters on the
transaction /api/payments/webhook/, so only a real webhook error triggers it.
Test the Discord webhooks¶
docker compose exec backend python manage.py shell -c "
from apps.core.services.discord_service import DiscordService
d = DiscordService()
for ch in ('errors', 'payments', 'backups', 'admin'):
print(ch, d.is_configured(ch), d.send_embed(channel=ch, title='Test from Django', description='webhook ok', color=DiscordService.COLOR_INFO))
"
Each configured channel should print True True and show a blue embed.
Test the celery-beat restart¶
See "Testen zonder storing" above (kill -s STOP): an orange "🔁 celery-beat herstart"
embed in #errors within about 15 minutes.
Verification Checklist¶
| Component | Test Method | Expected | Expected Timing |
|---|---|---|---|
| Sentry issue + e-mail | capture_message on production |
issue in Sentry, mail to Ivar | Within 30 min |
| Backup success | wait for the next hour | green embed in #backups |
At :00 |
| Daily backup summary | wait for 06:00 Dutch time | summary embed in #backups |
06:00 Amsterdam |
| Daily payment summary | wait for 07:00 Dutch time | embed in #payments |
07:00 Amsterdam |
| Disk-space alert | set DISK_SPACE_WARNING_PCT=1 on staging, recreate celery |
embed in #errors at :30, then restore the value |
Within 1 hour |
| celery-beat restart | kill -s STOP celery-beat on staging |
embed in #errors, container healthy again |
Within 15 min |
| External monitor | see External uptime monitor | — | — |
Troubleshooting¶
Sentry alerts not firing¶
- Check the environment: rules filter on
production.
- Check the DSN:
docker compose exec backend python -c "import os; print(bool(os.getenv('SENTRY_DSN')))"must printTrue. - Check the rule: Sentry → Alerts; each rule shows "Active", not "Disabled", and the notification action lists Ivar (or "issue owners").
- Check the mailbox: look in spam, and in Sentry → Settings → Account → Notifications that issue alerts are not muted for the projects.
Regressions not detected¶
Verify the release (see Release tracking). None or unknown means
the image was built without GIT_SHA; a normal CI build sets it.
No Discord messages from Django¶
docker compose exec backend env | grep DISCORD_WEBHOOK— the four URLs must be present. After editing the env file, recreate the containers (up -d backend celery celery-beat); arestartkeeps the old environment.docker compose logs celery --tail=200 | grep -i discord— the service logs a warning instead of raising.- Rate limit: at most 5 messages per channel per minute are sent; the rest is dropped with a warning.
- Test the webhook itself:
curl -H "Content-Type: application/json" -d '{"content": "test"}' "$DISCORD_WEBHOOK_ERRORS"→ HTTP 204.
Health Check Returns 503 But Site Loads¶
/api/health/ checks database and Redis, not just the process. If it returns 503 while the
frontend still renders (cached pages):
- Backend logs:
docker compose logs backend --tail=100 - Database:
docker compose exec backend python manage.py dbshell - Redis:
docker compose exec redis sh -c 'redis-cli -a "$REDIS_PASSWORD" --no-auth-warning ping'(the password is in the container's environment, not in your shell)
Celery is not part of /api/health/ — check it with docker compose ps celery celery-beat
or docker compose exec backend celery -A config inspect ping.
Alert Response Procedures¶
Site unreachable¶
- From outside:
curl -sI https://www.freezedesign.nl/health(nginx) andcurl -s https://www.freezedesign.nl/api/health/(Django). - On the server:
docker compose ps— every serviceUpandhealthy? - Backend logs:
docker compose logs backend --tail=100 - Common causes: database or Redis container down, backend crash loop, out of memory (
free -h), disk full (df -h), certificate expired (curl -vIshows the date). - If it started with a deploy, roll back:
export DEPLOY_TAG=latest && ./scripts/rollback.sh(production;DEPLOY_TAG=stagingon staging), see Deployment.
SSL certificate about to expire¶
Renewal is automated in two layers (.github/workflows/ssl-renewal.yml, Wednesdays 04:00
UTC, plus the root crontab safety net on each VPS, Sundays 03:30 UTC) — see the
Disaster Recovery runbook, SSL Renewal section. The
expiry-check job of that workflow posts to Discord (DISCORD_WEBHOOK_CICD) when a
certificate expires within 14 days. If it fires:
- Check the recent runs of the "SSL Renewal" workflow in GitHub Actions
- SSH to the VPS and renew manually (production; use the staging values from the DR runbook on staging): (the script reloads nginx automatically on success)
- Verify:
curl -vI https://www.freezedesign.nl 2>&1 | grep "expire date" - Check the cron safety net:
sudo crontab -l | grep renew-ssl
Discord Channel Summary¶
All application messages go through DiscordService (backend/apps/core/services/discord_service.py),
one webhook per channel from the env file. Since v1.59 it forwards every message to ntfy as
well (NotifyService, see ntfy notifications). Sentry does not post to Discord.
| Channel | Env var / secret | Source | Messages |
|---|---|---|---|
#errors |
DISCORD_WEBHOOK_ERRORS |
monitoring.check_disk_space; scripts/restart-unhealthy.sh (cron) |
Disk space ≥ 80 % / 90 %; celery-beat restarted or still unhealthy |
#payments |
DISCORD_WEBHOOK_PAYMENTS |
monitoring.check_payment_anomalies, daily_payment_summary, recover_incomplete_paid_orders |
Payment anomaly, daily summary (07:00), repaired orders |
#backups |
DISCORD_WEBHOOK_BACKUPS |
backup.* tasks, backup_database command |
Backup success/failure, cleanup, verification, restore test, daily summary (06:00) |
#production-admin-actions |
DISCORD_WEBHOOK_ADMIN |
Admin audit (delete, bulk_delete), apps/orders/tasks.py, apps/informer/notify.py |
Admin deletes, paid orders, stock problems, failed or canceled refunds, Informer booking failures ("⚠️ Informer: …") |
| CI/CD channels | GitHub secrets DISCORD_WEBHOOK_URL, DISCORD_WEBHOOK_CICD, DISCORD_WEBHOOK_PRODUCTION, DISCORD_WEBHOOK_STAGING_BACKUPS, DISCORD_WEBHOOK_DAGPLANNING |
Workflows | Deploys, E2E smoke after a staging deploy, CI results, SSL renewal and expiry, weekly backup test, k6 gate, daily planning, website check |
#monitoring |
— | External uptime monitor | Not set up yet, see above |
Details and setup steps: Discord Webhooks.