Send Email from a Background Job, Not the Request: Queues and Pacing

Calling an email API inside the signup handler works in development and fails in production in three ways: the request waits on a network call the user does not care about, a hiccup at the email provider turns into a 500 on your signup page, and a traffic spike turns into a burst that trips rate limits. Moving sends into a queue fixes all three. This guide shows the shape of that queue and the worker that drains it.

Quick answer

In the request, write what you want to send (recipient, template, data, idempotency key) to a job queue and return. A separate worker process pulls jobs, renders the email, POSTs to the send API, records the result, and retries failures with backoff. Limit the worker's concurrency and pace so you send a few messages per second, and treat 429 responses as a signal to slow down rather than an error.

What the request should do

• Commit the business change first (create the user, save the order). Never enqueue an email for something that might roll back.
• Enqueue a job with only identifiers and small data, not rendered HTML. The worker can load what it needs.
• Use the idempotency key as the job id so duplicate requests do not create duplicate jobs.
• Return immediately. The user sees "check your inbox" in under 100 ms.
• For the outbox pattern, write the job to a table in the same transaction and have a relay move it to the queue. This removes the window where the order exists but the job does not.

Choosing a queue

Use what your stack already has. Node: BullMQ on Redis. Python: Celery or RQ, or Django-Q. Ruby: Sidekiq or the built-in Solid Queue. PHP: Laravel queues with the database or Redis driver. Go: a worker pool reading from Postgres with SELECT ... FOR UPDATE SKIP LOCKED is enough. A plain database table plus a cron-driven worker is a perfectly good queue for a few thousand emails a day; only reach for a message broker when you need it.

The worker

// BullMQ worker, Node 18+
import { Worker } from "bullmq";

new Worker("email", async (job) => {
  const { to, from, subject, html, text } = await renderEmail(job.data);
  const res = await fetch("https://api.oqumail.com/api/v1/emails", {
    method: "POST",
    headers: { "Authorization": "Bearer " + process.env.OQUMAIL_API_KEY, "Content-Type": "application/json" },
    body: JSON.stringify({ from, to, subject, html, text }),
    signal: AbortSignal.timeout(15000)
  });
  if (res.status === 429 || res.status >= 500) throw new Error("retry later: " + res.status);
  if (!res.ok) { await markFailed(job.data.key, res.status); return; }
  const { id } = await res.json();
  await markSent(job.data.key, id);
}, {
  connection: { host: "127.0.0.1", port: 6379 },
  concurrency: 2,
  limiter: { max: 5, duration: 1000 } // at most 5 sends per second
});

Pacing and limits

Email providers rate limit, and OquMail returns 429 when you send too fast and 403 when a daily limit is reached. The worker above caps itself at 5 sends per second across 2 concurrent jobs, which is far more than a small product needs and low enough to never trigger a burst limit. On a 429, throw so the queue retries with backoff. On a 403 for a daily limit, pause the queue until the next day rather than hammering it; most queue libraries can pause and resume. Separate queues for urgent mail (codes, magic links) and bulk mail (digests) so a large digest run never delays a login code.

Failure handling

1. Configure the queue with 5 attempts and exponential backoff starting at 1 second.
2. After the last attempt, the job lands in a failed or dead-letter list. Alert on it; each one is a customer who did not get an email.
3. Record the API message id on success so support can look up the message in the delivery log.
4. Do not retry 400, 401, 403 (other than daily limit) or 422; mark failed and alert. They will not succeed on retry.
5. Keep the worker stateless so you can run two of them; the idempotency key protects against both processing the same job.

Why this also helps deliverability

Steady sending is what mailbox providers like. A queue that trickles messages out at a constant pace looks like a normal business; a burst of 2,000 identical messages in five seconds looks like a compromised account, whichever platform sends it. Pace also gives you a chance to see problems: if the delivery log starts showing deferrals from a particular receiver, you can pause before the whole batch is affected. On OquMail the per-message delivery log and the API's explicit 429 and 403 responses give the worker exactly the signals it needs.

Common questions

Is a queue overkill for a tiny app?

A database table and a cron job every minute is a queue and takes an hour to build. The behaviour you get (fast requests, retries, no double sends) is worth it from the first hundred users.

What about serverless functions?

Enqueue from the function to a managed queue (SQS, Cloud Tasks, QStash) and have a second function consume it. Do not send from the request-handling function unless you can tolerate lost sends on cold-start timeouts.

How do I test the worker locally?

Run it against a development API key with a from address on a test domain, and point recipients at your own mailbox. See the guide on testing email in development and staging for the full setup.