Eleven vehicles have not moved since morning, and the fleet is fine

Eleven out of four hundred is a diagnostic gift. A platform fault takes the whole fleet down; a subset that stalls while everything else reports normally is telling you the eleven share something the rest do not. So the first move is not to open the ingest logs. It is to bucket every vehicle in the fleet by the age of its newest position, then ask what the stale bucket has in common.

This page is about vehicles whose data genuinely stopped arriving or stopped being accepted. If the markers are frozen but fresh positions are landing on the server, that is a transport and rendering fault and belongs to the map is open and the vehicles have stopped moving. One query separates the two: compare the newest server-side timestamp for a stale vehicle against what its marker is showing.

Bucket the fleet by position age, then look for the shared attribute

1. For every vehicle, take the newest device timestamp and the newest server receipt timestamp. Two columns, not one — the difference between them is itself a finding.
2. Bucket into under 5 minutes, 5 to 30 minutes, 30 minutes to 4 hours, 4 to 24 hours, and older. Count vehicles per bucket.
3. Join the stale buckets against the vehicle master: device model, firmware build, install date, SIM or subscription batch, home depot, and whether the vehicle is on shift.
4. Look for a single attribute that most of the stale set shares and most of the healthy set does not. One model, one install week, one SIM range, one depot.
5. If no attribute stands out, stop looking at devices and go to the ingest side: count records received and records rejected per vehicle for the same window.

The correlations map almost one-to-one onto causes. All one device model points at firmware. All fitted in the same fortnight points at the install, usually a wiring or antenna practice that one engineer used. All in one depot points at something local — a workshop that isolates batteries over the weekend is the classic. A contiguous SIM range points at the subscription, not the hardware. And a spread across all attributes points at your own pipeline.

Five causes, and the evidence that separates them

The last two rows are the ones that survive longest, because both look exactly like silence from the outside. A vehicle whose records are being rejected is indistinguishable, on the map, from a vehicle with a dead unit. Fleet engineering goes to the depot, finds a working device, and the loop repeats.

Arriving, and being thrown away

Ingest validation is supposed to protect the state store from junk: coordinates at zero, timestamps in the future, a device identifier that matches no vehicle, an accuracy value outside the plausible range. All reasonable. The failure is doing it silently. A pipeline that drops a malformed record without a counter, a sample and an alert has converted a loud problem into a quiet one, and the quiet version costs a service visit.

• Count rejects per vehicle per hour, not just in aggregate. A fleet-wide reject rate of 0.2 percent hides one vehicle at 100 percent.
• Keep a sample of the rejected payload. 'Invalid record' with no example is not actionable at 2am or at any other time.
• Alert on the shape, not the volume. Any single vehicle whose records are entirely rejected for more than an hour is a page-worthy event, regardless of fleet size.
• Check the identifier join first. A device swapped between vehicles without the master record being updated produces perfect data attached to a vehicle that no longer exists — the class of problem the vehicle master record you need before a single position arrives is written to prevent.
• Watch the field the validator is strictest about. Rejecting on a missing optional field is a common own goal after a provider changes a payload.

Partners do this to each other too, and the pattern is identical: a record is transmitted, silently refused, and both sides believe the other is at fault. The same reasoning applies to the partner that keeps returning a rejected acknowledgement.

Backfill: the burst that arrives out of order

Most units buffer when they lose connectivity and flush when they get it back. That is the behaviour you want, and it breaks the map if last-known position is maintained by arrival order. A vehicle emerging from four hours in a dead zone dumps its queue; the pipeline writes each record as it arrives; and for the duration of the flush the marker walks the route again, hours behind reality, before jumping to the present. Meanwhile a dispatcher has assigned work against a position from lunchtime.

The staleness policy: when the map must say unknown

The durable fix is not finding these eleven vehicles. It is making the system incapable of presenting an old position as a current one. That needs a written policy with a threshold per vehicle class, because a threshold that is right for a delivery van in a city is absurd for a trailer tracker running on its own battery. Derive each threshold from the cadence you expect that class to report at in that state — which is why the policy has to be written alongside ping cadence set by what the vehicle is doing rather than after it.

Two rules make the table work rather than decorate a wiki. Every position rendered anywhere — map, list, API response, customer-facing tracking page — carries its age next to it, and the marker changes appearance at the ageing threshold and again at unknown. And unknown means unknown: the marker leaves the map or greys out entirely rather than sitting at the last coordinate looking authoritative.

A last-known position with no age attached is not visibility. It is a confident answer to a question the system stopped being able to answer, and dispatchers will act on it exactly as if it were current.

What to fix, in what order

1. Ship the staleness rendering first. It is the smallest change and it converts a silent wrong answer into a visible gap, which stops the operational harm while the rest is investigated.
2. Add per-vehicle receive and reject counters with retained samples, so the next partial outage is diagnosable from a dashboard rather than from a depot visit.
3. Fix the last-known selection rule to use device time, and mark buffered records, so backfill stops rewriting the present.
4. Only then chase the hardware. Devices are the slowest and most expensive thing on this list to touch, and by this point you know exactly which ones to touch.

Where a fleet runs more than one tracking vendor, staleness has to be evaluated per source before it is merged, or a healthy feed will mask a dead one behind a single canonical record — the adapter contract in one vehicle state from two telematics feeds is where that belongs. If none of this exists yet and the map is the only surface anyone has, a small, honest version of it is a sensible first build; that shape of work is what we describe under MVP and product builds. The neighbouring problems — drift, arrivals, cadence — sit together in telematics and the live vehicle data plane, part of our logistics and mobility work.

• telematics
• data quality
• fleet visibility
• monitoring