The ingest queue backs up every Monday morning

The queue is empty by Thursday and 30,000 deep by Monday lunchtime, and nothing in the logs is red. That is not a defect. It is the shape of the arrivals meeting a pipeline sized for their average, and until both are plotted on one axis the team keeps hunting a bug that does not exist.

Do the arithmetic first. A weekly export drops 42,000 documents between 23:00 Sunday and 02:00 Monday; the pipeline sustains 1,400 an hour. Arrivals run at ten times the service rate, so the queue peaks near 37,800 and needs about 27 further hours to clear — the last document lands some time on Tuesday morning. Averaged over 168 hours the source sends 250 an hour, so the pipeline holds over five times the capacity it needs and is still a day late.

Three lines across seven days, before you change anything

The confirming observation is one chart covering a full week, ideally two, with time-of-week on the horizontal axis: arrivals per interval, completions per interval, and queue depth. Hour buckets are usually right; anything coarser than 4 hours hides the burst.

• Arrivals per hour. Group by hour-of-week, not by date, so the recurrence survives. If over 80 percent of the volume lands in 5 percent of the hours, the shape is the story.
• Completions per hour. The ceiling of this line is the real capacity, whatever the design document claims. A flat top under load is a limit, not a busy machine.
• Queue depth per stage, not per pipeline. A total says there is a backlog; a per-stage figure says which stage owns it, which is most of the diagnosis.
• Age of the oldest unprocessed item. Depth can fall while the oldest item gets older — a stalled ordered group, seen from outside.
• Distinct source identifiers against items handled. If 61,000 items came from 42,000 records, a third of the work was reprocessing.

Five things that turn a Monday into a wall

The chart narrows it to one of five causes, each with a distinguishing observable. Read for the row whose middle column matches what you plotted, not your suspicion.

The last row gets reached for last and should be reached for early: it is the only cause where total throughput looks healthy while a slice of the corpus is stuck. Ordered delivery is the mechanism. Amazon's SQS documents that in a FIFO queue, messages sharing a message group identifier are processed in strict sequence, and while one is in flight the rest of that group stays unavailable until it is deleted or its visibility timeout expires — 30 seconds by default, extendable, never beyond 12 hours from first receipt. A record failing after 25 seconds reappears every half minute indefinitely, and everything behind it waits.

What to do while the backlog is live

Containment means ordering the moves so the cheap, reversible ones happen first. Doubling the workers is the instinct and is third at best, because two of the five causes get worse when you do it.

1. Stop the reprocessing before adding anything. If the distinct-identifier ratio said retries, capping the receive count and routing failures to a dead-letter queue removes work instead of adding capacity. A pipeline retrying at scale competes with itself for the same limited dependency.
2. Clear the stalled partition by hand. Pull the head item, record its identifier, push it to the dead-letter path. One released record can take an hour off the drain estimate.
3. Raise concurrency only on the stage that is queued. Per-stage depth already named it. Raising it everywhere moves the bottleneck downstream and destroys the evidence.
4. Check the ceiling before adding workers against an external dependency. Where the limit is published in requests or tokens per minute, workers beyond that number become rejections. Size one shared token bucket to the documented figure.
5. Split the lane rather than shedding load. A cheap metadata pass on everything, with expensive derivation queued behind it, makes new arrivals findable in minutes while the heavy work stays a day behind.
6. Record the run identifier, the counts and every intervention before the backlog clears. Once the queue is empty the evidence for next Monday is gone — which is why derived records should carry the run and transform identifiers that make an answer traceable.

The pipeline is not undersized. It is sized for a week that never happens — an average nobody experiences, spread evenly across hours the source never uses.

Moving the burst instead of buying capacity to absorb it

Four durable options exist and they trade against each other. The cheapest is usually a conversation with whoever owns the export schedule.

The middle two rows get argued as if one were obviously right. They are not comparable without a number: the buffer wins where the business tolerates a documented worst-case age, the reserved capacity wins where it cannot. Where the choice is open, cost them out on one page — the discipline in modelling three options for one requirement side by side.

Whether the arrival shape can move depends on how the pipeline learns something changed. A source emitting notifications can be consumed continuously; a source that only produces a scheduled dump cannot, and the burst is then a property of the capture mechanism — the subject of change data capture applied to a document corpus.

Alerting on a queue that is supposed to fill

A depth threshold is the wrong alert, because depth is meant to spike and an alert that fires every Monday is unread by the third week. Two signals carry the information instead.

• Projected drain time. Depth divided by the completion rate of the last 15 minutes, alerting when the projection crosses the hour the work stops being useful. A depth of 30,000 is routine at 14,000 an hour and an incident at 400.
• Oldest unprocessed item, per partition. Catches the stalled group that healthy total throughput conceals.
• Arrival volume against the same hour in prior weeks. A Monday at triple its usual size is a source-side event, often a re-export of history.
• Rejection rate plotted next to worker count. Rising together is a rate limit, invisible in a success rate that counts the eventual retry.

All four want somewhere to live. An operator view showing per-stage depth, oldest-item age, dead-letter contents and a replay button is unglamorous internal tooling and operations software, and it pays for itself the first time someone answers "is Monday's export in yet" without opening a log. The wider question is the alerts that catch a pipeline lying about its own success.

What a shorter queue will not fix

A drained queue is a lateness fix, not a correctness fix, and the two get confused whenever a backlog resolves and a complaint does not. If a document is still wrong once the queue empties, the queue was never the cause — the failure isolated in an updated document that is still served as its old text.

Two neighbouring problems look like this one and are not. A job running for days is usually not progressing rather than progressing slowly — see the reindex that never finishes. A run that completes fast with a fraction of the records is silent failure, covered in a nightly sync that reports success while records are missing. Both sit in data readiness and pipelines, part of the engineering library.

• ingestion
• capacity
• queues
• diagnosis