It keeps up on the bench and falls behind at line speed

Two different faults share the same complaint, and telling them apart takes ten minutes. Log the number of parts waiting to be processed once a second for ten minutes at full rate. If that queue grows steadily, the cell is slower than the line and no latency tuning will save it — you are short of throughput. If the queue stays flat but verdicts arrive after the part has passed the reject point, you are short of time inside a fixed deadline. The first is solved by doing less work or buying capacity; the second by moving a piece of steel.

Both are usually blamed on the model, and in most cells the model is not the largest term. Transfer of the image off the camera, and the format conversions that happen before inference sees anything, routinely consume more of the budget than the network everyone is arguing about.

The deadline is set by steel, not by software

Before measuring anything, work out how long you actually have. The verdict has to reach the actuator before the part reaches the reject point, so the budget is the distance between the camera and that point divided by the belt speed — and then several fixed costs come out of it. A camera 1.2 metres upstream of a blow-off nozzle on a belt running at 0.5 metres per second gives 2.4 seconds, which is generous. Move the nozzle to 300 millimetres from the camera and the same cell has 600 milliseconds, which is not.

This is why the cheapest fix for a latency deficit is frequently mechanical: move the reject point downstream, add a metre of conveyor, or divert into a queue emptied on a slower cycle. Nobody proposes it because it sounds like defeat, and it is usually the only change with no ongoing cost.

The per-part budget, to fill in with measured numbers

Log every stage per part against a shared identifier, and record percentiles rather than means. At 10 parts per second the 99.9th percentile arrives every 100 seconds, dozens of times a shift; a mean of 40 milliseconds with a tail at 400 misses parts all day while the dashboard reads healthy. That instrumentation is small, unglamorous software and it is the difference between a diagnosis and a debate — the sort of thing we treat as part of the build under internal tools and ops.

Transfer: usually already at the wire limit

A 5-megapixel monochrome frame at 8 bits is about 5 megabytes. A gigabit link carries roughly 110 to 118 megabytes per second of payload in practice, so that single frame occupies the wire for about 45 milliseconds and the theoretical ceiling is around 22 frames per second before anything else happens. Add a second camera on the same link and both numbers halve. A cell running 10 parts a second with two views is asking for 100 megabytes per second, which is the wire, with nothing left for headroom, retries or a burst.

• Check what the sensor is actually sending. Many cells transfer full resolution and then immediately resize; cropping a region of interest in the camera moves that saving onto the wire where it counts.
• Check the pixel format. Sending 12-bit or colour data when the pipeline converts to 8-bit mono is paying transfer cost for information that is discarded milliseconds later.
• Check for retries. Packet loss on an industrial network shows up as occasional frames taking several times longer, which is a tail problem and looks exactly like a slow model.
• Check whether the link is shared. A camera segment carrying plant traffic has a bandwidth you do not control, which is one of the reasons the ownership question in automation vendor, plant IT or product team has to be settled early rather than during an outage.
• Check the framing. Standard-sized packets add overhead on a link carrying thousands per frame, and the setting is worth several percent.

Pre-processing, and the per-call overhead nobody times

The model runs on a small normalised crop. Everything that turns a raw sensor frame into that crop — demosaicing, converting formats, resizing, normalising, copying between libraries — runs on the host, often on one core, on the full-resolution image. On a 5-megapixel frame that is tens of milliseconds per operation, and pipelines accumulate operations the way drawers accumulate cable.

The related surprise is that inference cost is dominated by per-call overhead rather than by the model. A small crop through a modern accelerator is a few milliseconds of compute wrapped in framework dispatch, synchronisation and memory copies that can be several times that. The diagnostic is simple: time the same model on a batch of one and a batch of eight. If eight parts take barely longer than one, you are paying overhead, not compute, and the fix is in the plumbing rather than in a smaller model.

A large share of pre-processing work exists only because the part could be anywhere in the frame. When presentation is repeatable, the pipeline can crop a known region instead of searching, and both transfer and processing shrink together. Positional repeatability bought in steel pays for itself in milliseconds forever after, which is the argument in fixturing a part so the camera sees the same thing twice.

Writing every image is a throughput decision

Archiving images is right — escape analysis is impossible without them — and doing it synchronously on the inspection thread is not. Compressed frames at roughly 400 kilobytes each, 20 a second, is about 8 megabytes per second, which any drive handles. The same rate in raw frames is 100 megabytes per second sustained, which a spinning disk cannot do at all and an SSD does while occasionally stalling for a flush. The median write is irrelevant; the 200-millisecond flush the inference thread waits on is the one that drops a part.

1. Move writes off the decision path entirely. Hand the frame to a queue and let a separate worker persist it; a verdict must never wait on a filesystem.
2. Bound the queue and decide what happens when it fills. Dropping archive images under pressure is a defensible choice; dropping inspections is not.
3. Compress in the worker, not in the hot path, and choose a setting that preserves the defect rather than one that looks fine on a screen.
4. Sample deliberately if you cannot keep everything: all rejects, all uncertain parts, and a fixed fraction of passes. Decide that fraction alongside the annotation standard, because it determines what a future training set can contain — the discipline in labelling defect images so two inspectors agree.
5. Watch for the log file nobody thinks about. Per-part debug logging written synchronously has ended more line trials than any model.

A cell that meets its deadline on average does not meet its deadline. The specification is the tail, because the part that arrives during the worst hundred milliseconds of the shift is a real part and it ships.

Three ways out, in the order to try them

1. Do less work. Crop in the camera, send 8-bit mono if that is what you use, remove redundant conversions, stop searching for a part you could fixture. This is free and it usually recovers more than the other two combined.
2. Move the deadline. Relocate the reject point downstream, or divert to a re-inspection lane that is cleared on a slower cycle. Cheap in capital, no ongoing cost, and it converts a hard real-time problem into a soft one.
3. Buy headroom. A faster interface, a second link, hardware decode, or an accelerator with room to spare. Correct when the first two are exhausted, and worth sizing against measured numbers rather than a vendor's frames-per-second figure.
4. Whatever you choose, re-run acceptance at full line speed with the tail measured, not on a bench with a tray of parts — the standard of proof set out in the acceptance run that proves an inspection cell.

One exclusion is worth stating. If images arrive damaged rather than late — clipped, smeared, duplicated — that is acquisition, and the checks are physical: half a part in frame, trigger and encoder faults. Fix that first, because a cell clipping half its frames produces a throughput measurement that means nothing. The rest of the cell's decisions sit in visual inspection and defect detection, part of our manufacturing and industrial vision work.

• throughput
• latency
• machine vision
• line speed