Sat Mailer

Transactional mail is where a festival stack fails quietly. An SMTP hiccup during a registration burst either blocks the request that triggered it or disappears into a swallowed promise, and nobody finds out until a few hundred people say their confirmation never arrived. So sending stopped being something the API does at all. Services write a document into a Firestore emails collection and move on; one worker is the only process in the system holding an SMTP connection.

The shape that buys is a durable outbox. Once the write lands, the mail is going to be attempted whether or not the API process survives the next minute, and the record of what happened to it lives next to the record that it was requested. The alternative — a mail helper imported into every service — spreads SMTP credentials, connection limits and retry logic across every codebase that ever needs to notify somebody.

The document is the queue

An email is one Firestore document: email, subject, html, a status of pending, processing, sent or failed, an attempts counter, createdAt, updatedAt, nextAttemptAt, sentAt and the last error string. There is no separate queue product and no separate state table. Producers only ever write the first three fields and the status; every other field is the worker's, which keeps the contract between them one line long.

What the worker does

• Polls Firestore every five seconds for up to ten pending documents ordered by createdAt.
• Claims each document inside a transaction, re-reading it and flipping pending to processing only if it is still claimable, incrementing attempts in the same write. A second replica racing for the same batch loses the transaction instead of sending a duplicate.
• Filters due time in code rather than in the query. Adding nextAttemptAt as a second where clause would force a composite index; skipping not-yet-due documents out of a small batch costs nothing at this volume.
• Processes the batch concurrently with Promise.all, so ten sends overlap rather than serialising behind the slowest one.
• Retries with capped exponential backoff at 2^attempts * 30 seconds, ceiling two hours, writing the error string onto the document each time and pushing the status back to pending so the next poll picks it up.
• Gives up after five attempts, marking the document failed so it stops consuming sends and stays visible for a human to look at.
• Heartbeats into an app/emailer document once an hour with status, running sent count, last check time and next check time, which makes "is mail still going out" answerable from the same Firebase console as everything else.
• Logs through pino to stdout and logs/mailer.log at once using multistream, falling back to console-only if the log directory cannot be created.
• Traps uncaughtException and unhandledRejection, flushes the logger and exits, so the container restart policy takes over instead of leaving a half-dead process running.
• Marks itself failed in Firestore if the main loop dies, so a dead worker looks different from an idle one.
• Takes its tuning from the environment — poll interval, fetch limit, max attempts and backoff base are all env vars with defaults baked in, so the cadence changes without a rebuild.

The index you cannot avoid

The committed log makes the composite-index decision concrete. Even the reduced query — one equality on status, one orderBy on createdAt — needs a composite index in Firestore, and the first eighteen poll cycles in the log are FAILED_PRECONDITION: the query requires an index. That index is currently building. The worker logs the error, sleeps five seconds and tries again, so the cold start is noisy but not fatal. Adding nextAttemptAt as a second range clause would have meant a second index and a second build; doing that filter in JavaScript over a ten-document batch costs a few microseconds.

The one-off blast

Next to the worker sits a standalone script for outbound campaigns. It reads a recipient CSV, splits on any of \r\n, \r or \n because the file came out of Excel, drops every line without an @ and a ., renders the Deans' Conclave invitation and sends with a fixed delay between messages to stay inside the provider's rate limit, printing a sent/failed/skipped summary at the end. It sends from outreach@ with a reply-to on the institute domain, where the worker sends from no-reply@, so replies to a campaign reach a human and replies to a confirmation do not.

The templates are hand-written table-layout HTML with inline styles and a Cloudinary-hosted banner, which is still the only reliable way to get a consistent render across Outlook and Gmail. The recipient list is not in the repository — Book1.csv is a header and 810 empty rows — so the address filter is the only thing standing between a checkout and an accidental run.

Deployment, and what the log shows

A node:20-alpine image with the service account key mounted read-only and the log directory bind-mounted to the host so logs survive restarts. Docker's json-file driver rotates three 10MB files behind it, and the service runs restart: always.

The log that ships with the repo covers six process starts, 243 claimed documents and 236 sends, with seven failures that went into backoff and came back. It is a small enough sample to read line by line, which is roughly the point of writing structured logs to a file as well as to stdout.

Two gaps are worth naming. A document that reaches processing and then loses its worker mid-send stays processing forever, because the poll query only asks for pending — there is no reaper for orphans, and the claim transaction's own "or the retry time has passed" branch is unreachable from this loop for that reason. And the fatal handler fires a Firestore update without awaiting it before process.exit(1), so the "worker died" marker is best-effort. Both are the kind of thing you fix when a second replica exists; with one process and a five-second poll, a restart is the reaper.