ntfy notifications

The application alerts are moving from Discord to a self-hosted ntfy server on the Freeze AI VM (https://freeze-ai.tail56046.ts.net, public through Tailscale Funnel, deny-all: every sender needs a token). During the transition every message goes to both Discord and ntfy; removing Discord is a separate todo. The VM side (server, users, tokens, emergency line) is described in the Freeze AI repo, docs/plans/2026-11-02-ntfy-meldingen.md.

How it works

DiscordService (backend/apps/core/services/discord_service.py) forwards every message to NotifyService (backend/apps/core/services/notify_service.py), so all existing call sites (send_embed) reach ntfy without changes. restore_database posts its own message to the backups channel as well.

Channel	ntfy topic	Priority
errors	freeze-errors	red 4, yellow 3, green/blue 2
payments	freeze-payments	idem
backups	freeze-backups	idem
admin	freeze-admin	idem

The embed's title becomes the ntfy title (with NTFY_TITLE_PREFIX in front), and the description, fields and footer become a Markdown message. Messages are published as JSON, so titles with "—", "€" or emoji are fine. NotifyService never raises; a failed post is logged as a warning.

Safety nets

• ntfy unreachable (timeout, 5xx, network): the title only goes to the emergency topic on ntfy.sh (NTFY_FALLBACK_URL), at most once per title per 10 minutes. The emergency topic carries no customer data.
• Errors on payments and backups (red embeds) always also go by e-mail to NOTIFY_FALLBACK_EMAIL, whether or not ntfy is up. They are deduplicated per title for 10 minutes and capped at 20 mails per day.

Configuration (env on the VPS)

Variable	Production	Staging
NTFY_URL	https://freeze-ai.tail56046.ts.net	same
NTFY_TOKEN	token webshop-prod (1Password: "ntfy Freeze AI — tokens")	token webshop-staging
NTFY_TITLE_PREFIX	empty	[staging]
NTFY_FALLBACK_URL	the ntfy.sh emergency topic (1Password: "ntfy.sh noodtopic Freeze AI")	same
NOTIFY_FALLBACK_EMAIL	e.g. info@freezedesign.nl	empty or your own address
NTFY_TOPIC_<CHANNEL>	optional override of a topic	—

The webshop ntfy user may only write to freeze-errors, freeze-payments, freeze-backups, freeze-admin, freeze-approve, freeze-social and freeze-ai-heartbeat; it cannot read.

Check

# on the VPS, in the backend container
python manage.py shell -c "from apps.core.services.notify_service import NotifyService as N; print(N().send_embed('errors', title='Testmelding ntfy', description='Handmatige test'))"

True means ntfy accepted it; the message appears in the ntfy app on freeze-errors.

Health pings to healthchecks.io

Two Celery beat tasks (todo healthchecks-hetzner) let healthchecks.io notice when something goes silent. They do nothing without their URL.

Task	Every	Env	What a missing ping means
monitoring.healthcheck_ping	5 min	HEALTHCHECK_PING_URL	worker, beat or database is down (a database error sends /fail right away)
monitoring.ntfy_public_heartbeat	10 min	NTFY_HEARTBEAT_PING_URL (+ NTFY_URL, NTFY_TOKEN)	ntfy is not reachable from outside the tailnet (Funnel); the task publishes a silent message (priority 1) on freeze-ai-heartbeat first

Use one healthchecks.io check per environment for HEALTHCHECK_PING_URL (hetzner-staging, hetzner-prod) and one shared check ntfy-public. healthchecks.io alerts through the ntfy.sh emergency topic and e-mail, so the alert does not depend on the ntfy server it watches. Nobody subscribes to freeze-ai-heartbeat.

Scripts and GitHub workflows

scripts/notify-ntfy.sh <topic> <priority> <title> <message> [click-url] publishes to ntfy with NTFY_URL and NTFY_TOKEN; without them it does nothing, and it always exits 0 so a notification never fails a deploy. It needs only bash and curl.

Source	Topic	Token
scripts/notify-discord.sh (deploys, CI, backup test, SSL, E2E and k6 gate)	freeze-deploy, failure = priority 4	GitHub secret NTFY_TOKEN_CI (ntfy user ci)
scripts/restart-unhealthy.sh (cron on the VPS)	freeze-errors	NTFY_URL/NTFY_TOKEN from the same backend env file as the webhook

The workflows pass NTFY_URL and NTFY_TOKEN: ${{ secrets.NTFY_TOKEN_CI }} next to DISCORD_WEBHOOK_URL. Tests: scripts/tests/test_notify_ntfy.sh, scripts/tests/test_restart_unhealthy.sh.

Not covered here

• The dagplanning messages (dagbouw-bewaker, dagplanning-waakhond, websitecheck-discord) still post to their own Discord channel; they move when Discord is phased out.