Scheduled stories that go live late, or twice

Late and duplicate are one defect seen from two sides: a scheduler nobody verifies, driving a publish command with no idempotency key. Start by pulling four instants for the affected story — the slot it was scheduled for, the moment a worker claimed the job, the moment the record flipped to published, and the first request the edge served carrying the new object. Three gaps sit between those four numbers, and each belongs to a different part of the system. Whichever gap is large is the one to investigate; the others are noise.

Four instants, three gaps, three owners

If the third gap is the large one, the story did publish on time and the reader could not see it, which is a caching problem rather than a scheduling one and is traced separately in some readers still see the old version of a corrected story.

Why twice and late are the same defect

A publish job is a message. Almost every queue worth using delivers at least once, which means a message that is slow to acknowledge will be delivered again. Late publishing is that message arriving after its slot; duplicate publishing is the same message arriving twice because the first attempt did not acknowledge in time. The tell is arithmetic: measure the interval between the two publish events. If it is close to your queue's visibility or acknowledgement timeout, the job was claimed twice — the worker was still running when the queue decided it had died. Nobody scheduled the story twice, and looking in the CMS for a second schedule row will waste an afternoon.

Where the minutes go: five causes

1. Cron drift and cold starts. A scheduler polling once a minute cannot be more accurate than a minute, and a serverless worker adds its cold start on top. A story scheduled for 09:00 goes out somewhere before 09:02, which is invisible for a feature and unacceptable for a markets story timed to an exchange open. If the desk needs second-level accuracy, a per-minute poll is the wrong tool and a per-item timer is the right one.
2. A queue with no deduplication. The job is claimed, the worker takes longer than the visibility timeout, the queue re-delivers, and two workers publish. The fingerprint is the timeout arithmetic above. The fix is not a longer timeout — it is a publish command that a second attempt cannot double.
3. Time zone and daylight-saving handling in the schedule field. An editor in one zone, a server in UTC, and a field storing a naive local timestamp. Twice a year that field either names an instant that does not exist or one that occurs twice. Store the instant alongside the zone identifier the editor chose, from the IANA time zone database, and render back into that zone for display.
4. The worker publishes before the bundle is ready. The record flips live while image derivatives are still rendering, so the first Open Graph scraper and the first crawler both cache a story with a missing lead image — and Facebook, LinkedIn and Slack will each hold that cached preview for hours. Publishing should assert a readiness predicate over the whole bundle, not over the text record alone.
5. A downstream index never received the event. The article page is live and correct while the section front, the RSS feed, the news sitemap and the on-site search index do not have it. To everyone except a person holding the direct link, the story did not publish. Check each of the 4 consumers separately: they fail independently and none of them raises an alarm.

One cause is worth separating out because it looks like a scheduler fault and is not. Imported archives frequently carry a publication timestamp from the old system, and if the importer wrote that value into the field the scheduler reads, the scheduler will do exactly what it was told — publish something that is already live, or re-publish it. That is a field-mapping decision, and it belongs with the rest of the choices made in mapping legacy stories into a model that did not exist.

The key a publish command needs

Make publishing idempotent with a natural key rather than a generated one, so a retry reproduces the same key by construction instead of remembering it. The key is the story identifier, the scheduled instant and the content revision. Write it into a publications table with a unique index across those three columns, inside the same transaction that flips the record live. A second attempt collides with the index, catches the conflict, and returns the original result rather than performing the work again.

1. Compute the key from data the caller already has. If the key comes from a random identifier generated at call time, a retry generates a new one and the guard does nothing.
2. Enforce it with a unique index, not with a lookup before the insert. A check-then-write in an at-least-once system is the bug you are trying to fix, restated.
3. Include the revision. Re-publishing a corrected story is a legitimate second publish; re-publishing the identical revision is not, and the key should permit the first and refuse the second.
4. Make the whole publish one transaction over the bundle — record, derivatives, key set for invalidation — so a partial publish is not a state the system can be in.
5. Return the original outcome on conflict rather than an error. The caller that retried is not doing anything wrong, and an error here turns a handled case into a page for somebody.

A retry is not an exception. In any queue that delivers at least once, a retry is the normal case you have not written down yet.

Alert on the slot, not on the job

The standard alarm — the publish job failed — cannot detect the most common serious failure, which is that no job ran at all. A scheduler that has died emits nothing. The alarm that catches everything is a query over the schedule itself: any slot whose instant has passed by more than 2 minutes with no publication row against it. Run it every minute and route it to a person. It costs a few lines, and it catches a stopped scheduler, a poisoned message, a deploy that dropped the worker and a slot whose story was deleted underneath it.

• Add a second check for the inverse: more than one publication row for a story and slot. That is the duplicate alarm, and it fires before a reader finds the second URL.
• Gate the scheduler on open workflow states. A story with an unresolved obligation should not be publishable by a timer, which is why the legal read belongs in the state machine rather than in a conversation — the argument in the pre-publication legal read as a workflow state.
• Count how many stories are actually scheduled per day and at what accuracy the desk needs them. If most publishing is immediate and scheduling is a weekly newsletter, per-item timers are unnecessary; if the desk schedules 60 items a day against embargo times, they are not optional. That number is one of the facts a platform decision should rest on, alongside the rest of the content audit before anyone shortlists a platform.
• Keep the scheduler's own health visible: last poll time, queue depth, and the age of the oldest unclaimed slot. Three numbers on one panel.

None of this is difficult work; it is careful work, and it tends to get deferred because the failure is intermittent and the workaround is a person watching a clock. Building the publish path so it can be retried safely is the sort of thing we treat as core scope in MVP and product builds rather than as hardening to be done later. An embargoed release with a fixed lift time is a different obligation from a scheduled slot, and it is defined separately in embargoes as a field the platform enforces.

The rest of this silo sits under CMS and publishing platform engineering, and the platform work we do for newsrooms is described in our media and publishing practice.

• scheduling
• queues
• idempotency
• publishing platform