juniorData Processing

What is a dead letter queue and why does a pipeline need one?

Updated Feb 20, 2026

Short answer

A dead letter queue is a separate destination for messages a consumer cannot process after a bounded number of attempts. Without one, a single malformed message either blocks the partition behind it forever or is silently discarded. The DLQ takes the poison message out of the main flow so the pipeline keeps moving, while preserving the failure for inspection and replay.

Deep explanation

A poison message is one that fails every time — malformed JSON, a schema violation, a reference to a deleted row. Retrying it is futile, and the two naive responses are both bad:

  • Retry forever — on an ordered log like Kafka, the consumer never advances past the offset. Everything behind it stops. One bad message halts the partition.
  • Drop it — the pipeline moves on, but you have silently lost data with no record of what or why.

The DLQ is the third option: after N attempts, move the message aside with enough context to diagnose it, commit the offset, and carry on.

Python
MAX_ATTEMPTS = 3


def handle(msg):
    for attempt in range(1, MAX_ATTEMPTS + 1):
        try:
            process(msg)
            return
        except TransientError:
            if attempt == MAX_ATTEMPTS:
                break
            sleep(2 ** attempt)          # back off before retrying
        except PermanentError:
            break                        # no point retrying a schema violation
    dlq.send({
        'payload':      msg.value,
        'error':        traceback.format_exc(),
        'source_topic': msg.topic,
        'partition':    msg.partition,
        'offset':       msg.offset,      # so it can be replayed precisely
        'failed_at':    now(),
        'attempts':     attempt,
    })

Distinguishing transient from permanent failures is the design decision that matters. A database timeout is transient and deserves retries with backoff. A schema violation is permanent — retrying it three times just wastes time before the inevitable. Treating everything as transient turns a brief outage into a flood of DLQ entries; treating everything as permanent dead-letters messages that would have succeeded a second later.

A DLQ is only useful if someone looks at it. An unmonitored DLQ is a slower form of dropping messages. Alert on its depth and its rate of growth — a sudden spike usually means an upstream schema change rather than genuinely bad data.

Replay must be part of the design. Once the bug is fixed, you need to reprocess what accumulated. That requires storing enough context to reconstruct the original message, and it requires downstream consumers to be idempotent, since replay can duplicate work.

Real-world example

An order pipeline where an upstream service starts emitting a new field:

TypeScript
orders topic -> consumer -> validate schema -> write to warehouse
                              |
                              +-- schema violation, 3 attempts, all fail
                                     |
                                     v
                              orders.DLQ  { payload, error, offset, failed_at }

Without the DLQ the consumer retries offset 48,201 indefinitely and every later order stops being processed — the warehouse silently stops updating while the topic keeps growing. With it, the bad orders divert, the other 99.9% flow through, and an alert fires on DLQ depth. After the schema is fixed, the DLQ is replayed and nothing is lost.

Common mistakes

  • - Having no DLQ at all, so one poison message blocks a partition indefinitely or is silently dropped.
  • - Storing only the payload without the error, offset, and timestamp — leaving you unable to diagnose or replay accurately.
  • - Retrying permanent failures such as schema violations with the same backoff as transient ones, wasting time on a guaranteed failure.
  • - Never monitoring the DLQ, which makes it an elaborate way of losing data quietly.
  • - Replaying into a non-idempotent consumer, so recovery creates duplicate records.
  • - Setting no retention or size limit on the DLQ, so it grows unbounded and becomes its own operational problem.

Follow-up questions

  • How do you distinguish a transient failure from a permanent one?
  • Why does replaying from a DLQ require idempotent consumers?
  • What should be stored alongside the failed payload?
  • How does a DLQ relate to the retry topic pattern?

More Data Processing interview questions

View all →