Retry transient failures, quarantine permanent ones and crash-looping poison pills in a DLQ with their metadata, never hold up the healthy messages, and replay once the bug is fixed.
Some messages fail. A database timeout will probably work next time (transient); a negative price will fail the same way forever (permanent); and a poison pill kills the consumer every time it is delivered (Day 4, sections 1.1 and 1.2). Retrying the permanent ones wastes time, and retrying a poison pill in place blocks everything behind it. The fix is a dead letter queue: after a bounded number of attempts the message is moved aside, with enough metadata to investigate it and replay it later (section 1.3).
You get a fake main queue with at-least-once delivery and a fake DLQ (see Given). A received message stays in flight until it is acked or retried; if the consumer dies, queue.recover() (the visibility timeout running out) puts its in-flight messages back. The queue counts deliveries itself: every message carries receive_count (1 on the first delivery) and first_received_at.
Implement DLQConsumer(queue, dlq, handler, clock, max_attempts=3):
| Method | Does |
|---|---|
process_one() |
receives one message and calls handler(body). Returns "ok", "retry", "dead", or None when the queue is empty |
replay(ids=None) |
moves DLQ entries (all of them, or those whose id is in ids) back onto the main queue as new messages, oldest first, and removes them from the DLQ. Returns how many it moved |
Rules the tests check:
queue.ack(id), return "ok".PermanentError: no retry. Dead-letter it now (reason "permanent").Exception (TransientError, or an unexpected TypeError) is treated as transient (the lesson's UNKNOWN category, section 2.4): if receive_count < max_attempts, queue.retry(id) sends it to the back of the queue and you return "retry"; otherwise dead-letter it (reason "exhausted").Crash is the process dying. Let it propagate (it is a BaseException, so except Exception won't catch it). The test then calls queue.recover() and starts a new consumer, and the message comes back with a higher receive_count. When a message arrives with receive_count > max_attempts, every earlier delivery died mid-handler: dead-letter it without calling the handler (reason "poison", error None, attempts = receive_count - 1).dlq.put(entry) and then queue.ack(id), and returns "dead". The entry is {"id", "body", "reason", "error", "attempts", "first_attempt_at", "failed_at"}: error is "ClassName: message" (f"{type(e).__name__}: {e}"), attempts the handler runs so far (receive_count), first_attempt_at the message's first_received_at, failed_at clock.now().dlq.put and before the ack, so the message is delivered again. If the DLQ already has its id, just ack it and return "dead".receive_count, never from the consumer's memory: a restarted consumer starts with empty memory, and a count that resets on restart is the infinite retry loop of section 6.3.In JavaScript and TypeScript: new DLQConsumer(queue, dlq, handler, clock, maxAttempts = 3) with processOne() and replay(ids?); messages are { id, body, receiveCount, firstReceivedAt }, entries are { id, body, reason, error, attempts, firstAttemptAt, failedAt } with error `${e.name}: ${e.message}` (or null), and the queue and DLQ methods are send, receive, ack, retry, recover, pending and put, has, remove. JavaScript has no BaseException, so Crash is an ordinary error: rethrow it.
A poison pill never reaches your except: it takes the process down, so nothing you keep in memory survives to count it. The broker does see every delivery. Amazon SQS works this way: its redrive policy moves a message to the dead-letter queue once its receive count exceeds maxReceiveCount.
Write your code, then run 3 example tests and 9 hidden tests. Each run is limited to 8 seconds: an infinite loop gets stopped and the runtime restarts, your code stays.