Operations

Health

Check	Where	Healthy	Degraded
API	GET /health	{"status":"ok","db":"ready"}	503 with db: "down" while Mongo is unreachable. The API keeps serving; it does not crash-loop.
Worker	container healthcheck	pings Mongo and Redis	—
Redis	redis-cli ping	PONG	The API exits 1 at boot without Redis. A subscription write can wait for a database; a push accepted with 202 and dropped cannot un-disappear.

Logs

docker compose logs -f api worker

What you will never see in a log, at any level: a device token, an FCM registration token, a web push endpoint, .p8 contents, a service account, a VAPID private key. Failures are logged with the subscriptionId; if a token must be distinguished, it is the hash of its last six characters.

Restarts and drains

• docker compose stop worker sends SIGTERM. The worker stops taking jobs and finishes the in-flight ones, up to SHUTDOWN_TIMEOUT_MS (30 s). Compose waits stop_grace_period: 45s before SIGKILL. Nothing is lost either way: an unfinished job goes back to the queue.
• The API is stateless. Restart it whenever.
• A scheduled campaign survives restarts: it is a delayed BullMQ job in Redis, not a timer in a process. There is no scheduler service to keep alive. Redis runs with appendonly yes, so the job survives a Redis restart too.

Backups

What	How	Why it matters
Mongo mongo_data volume	docker compose --profile backup up -d	Subscriptions and users are the asset. Campaign history and events are nice to have.
ENCRYPTION_KEY	password manager	Without it a restored database has unreadable credentials.
Redis redis_data	optional	Only pending/scheduled jobs live there. Losing it loses queued sends, not subscribers.

The backup profile adds a mongo:7 sidecar that runs mongodump --gzip --archive every BACKUP_INTERVAL_HOURS (24) into the mongo_backups volume — or a host directory if BACKUP_DIR is set — and deletes archives older than BACKUP_KEEP_DAYS (14). Copy that directory off the host; a backup on the same disk as the database is not a backup. Restore with:

docker compose --profile backup run --rm backup \
  mongorestore --uri mongodb://mongo:27017 --gzip --archive=/backups/<file> --drop

Rotating ENCRYPTION_KEY means re-uploading every credential (there is no re-seal command). Rotating HMAC_SECRET breaks tracking for pushes already delivered but not yet opened — do it right after a campaign has settled.

Token hygiene (automatic)

• Every 410 Unregistered, BadDeviceToken, DeviceTokenNotForTopic, UNREGISTERED, SENDER_ID_MISMATCH, web 404/410 → the subscription is stamped invalidatedAt, a failed event is written with the code, and a subscription.invalidated webhook fires. It is never retried.
• The maintenance queue runs a daily sweep that sets optedIn: false on subscriptions with no activity for INACTIVE_SUBSCRIPTION_DAYS (90). A device that subscribes again is revived.
• Duplicate tokens cannot exist: (appId, token) and (appId, endpoint) are unique partial indexes. Re-subscribing from the same device updates the row.

Data retention

• events is a time-series collection with a TTL of EVENT_RETENTION_DAYS (180). Campaign counters (stats) are kept forever on the campaign.
• Idempotency records expire after 24 hours.
• Audit log entries are kept indefinitely.

Upgrading

git pull && docker compose build && docker compose up -d

Indexes are created idempotently on every API boot. The one thing that cannot be changed after the fact is the events collection's time-series options — it is created once; to change retention, set EVENT_RETENTION_DAYS and use collMod on the existing collection, or drop it (you lose events, not counters).

Monitoring ideas

• Alert on /health ≠ 200 for more than a minute.
• Alert on the worker log line count for failed events spiking per campaign — a credential expiring shows up as InvalidProviderToken / PERMISSION_DENIED across a whole platform at once.
• Watch Redis memory; a steadily growing bull:push-*:wait list means the worker is down or starved.
• The dashboard's campaign report has the per-code failure table — the fastest way to see why a platform is failing.