The model finds the scratch on day shift and misses it after dark

A detection rate that follows the clock is telling you the input changed, not the judgement. The model is a fixed function: feed it the same pixels and it returns the same number, today, tonight and next March. So before anyone opens the training set, take a few hundred archived images from the shift where the system works, push them through the machine that is currently failing, and compare the scores against what was recorded at the time. If they match, the model is exonerated and the investigation moves to the thing between the part and the sensor.

That leaves one question: what is different about the light after dark, or at eleven in the morning, or on the days the dock door is open. Almost always it is one of five things, and each leaves a different signature in the image statistics you are probably already storing and not plotting.

The test that separates optics from the model in one production day

1. Pick one part number that runs on both shifts in reasonable volume. Mixing part numbers ruins the comparison before it starts.
2. Pull 100 images of it from the good shift and 100 from the bad one, with their recorded scores and dispositions.
3. For every image, compute the mean and standard deviation of grey level in a fixed region of background — a patch of the fixture or the belt that contains no part. This is your illumination proxy and it costs a few lines of code.
4. Plot those two statistics against capture time across 48 hours. A daylight fault shows a smooth hump; a task lamp shows a square wave; lamp ageing shows nothing at this timescale at all.
5. Re-score the good-shift images now, on the live machine. Unchanged scores mean the model is fixed and the input is not.
6. If the image statistics are indistinguishable between shifts and the scores still differ, stop reading this page — you have a model or threshold problem, and the anomaly score and the threshold on a running line is the right place to go.

Why a scratch needs grazing light, and why fill light erases it

A fine surface defect is not darker or brighter than the material around it in any absolute sense. It becomes visible because it scatters light differently from the surface it sits in, and a station designed to find one puts a lamp at a low angle so that most light skims past the sensor and only the scattered light from the flaw reaches it. The defect appears as a bright line on a dark field. It is a contrast trick, and it is fragile in one specific direction.

This is why the instinct to add more light makes the station worse, and why a night shift with the overhead bays switched on can be harder than a night shift in the dark. It also explains a pattern people find baffling: the system starts passing subtle defects while still catching gross ones. A 3-millimetre dent survives the contrast loss; a 40-micrometre scratch does not. Measure the two conditions with a lux meter held at the part position and the ratio is often 5 to 1 or worse between a shrouded station and the same station with a roof light overhead at noon.

Five faults with a time-of-day signature

One signature does not belong on this table. If the variation is frame to frame rather than hour to hour — one image bright, the next dim, on the same parts inside a minute — you are looking at exposure timing against a light source that ripples with the mains supply, or at a strobe firing out of step with the trigger. A 200-microsecond exposure under lighting fed from a 50 Hz supply samples a different point of a 100 Hz brightness ripple on every frame, which reads as random noise and is not. That is acquisition, and it is covered in trigger and encoder faults that look like model faults.

The model did not get worse after dark. It received a different picture of the same part and answered the question it was actually asked, which is the one thing a model can be relied upon to do.

The decision tree, ending in steel or in a retrain

1. Do archived images re-score as they did before? No — the model or its runtime changed, and this page does not apply. Yes — continue.
2. Does the background statistic differ between the good and bad periods? No — the problem is in the part or the fixture, not the light; check presentation and handling. Yes — continue.
3. Is the difference a smooth daily curve? Shroud the station and re-test over 48 hours before anything else is considered.
4. Is it a step change? Find the date, then find what was done that day. Maintenance logs, packaging changes and housekeeping are all more likely than a software release.
5. Is it a slow drift over months? Lamp ageing. Replace on hours, and set an alarm on the background statistic so the next decay is caught at 10 percent rather than at the point of failure.
6. Fixed the light and still missing defects? Now retrain — but retrain on images captured under the corrected lighting, and treat the old set as a different domain rather than more data.

That last step is the one worth arguing about. Retraining on images from a broken lighting condition teaches the model to tolerate the fault, which sounds robust and is not: it widens the definition of normal until genuine defects fit inside it. The result is fewer false rejects and more escapes, and escapes are found by the customer rather than by you — the investigation described in tracing an escape backwards.

Instrument the station so this is never an argument again

• Store the background statistic with every image. Two numbers per part, indexed and plottable, turn a three-week argument into a chart.
• Alarm on illumination, not just on rejects. A background mean outside a stated band should stop the line or flag the batch before the reject rate tells you.
• Keep a commissioning photograph of the station from four angles, dated. It is the cheapest configuration management you will ever do, and it settles the 'nothing has changed' conversation.
• Count lamp hours and replace on a schedule, the way you would a filter. Waiting for visible dimming means waiting until output has fallen far enough for a human to notice, which is far past the point the camera noticed.
• Run acceptance across every shift and every weather condition you can get, not across one good week — the argument for a genuinely long parallel run made in a four-week parallel run modelled end to end.
• Watch the false reject rate on both shifts separately. A single fleet-wide figure averages the good shift and the bad one into a number that describes neither, as the false reject rate the line actually feels sets out.

One rival explanation is worth ruling out before you commit to optics: if the change coincides with a coil, resin or component changeover rather than with the clock, the variable is the material and not the light, and the checks are different — rejects doubling the morning a new lot goes on covers that path. It is also worth confirming the station is not simply out of time budget after a speed change, since shortening exposure to control blur is a common and invisible way to lose light, which sits in it keeps up on the bench and falls behind at line speed.

Getting the enclosure, the logging and the alarms right is unglamorous and it is most of the difference between an inspection cell that survives a year and one that gets switched off in month three. It is also the sort of scope we would rather size honestly up front than discover later, which is how we approach product builds of this shape. The neighbouring decisions — thresholds, defect classes, acceptance — sit together in visual inspection and defect detection, part of our manufacturing and industrial vision work.

• machine vision
• lighting
• inspection
• troubleshooting