The question nobody asks about a computer vision project

A US company that makes point of sale software for school cafeterias asked us whether AI could read a lunch tray.

The problem is real and it is small enough to describe in a sentence. A student slides a tray along the counter. Somebody has to look at it, work out that there is a chicken sandwich, a carton of milk and an apple on it, and tap three items into a till while a queue builds behind them. Do that a few hundred times over a forty minute lunch service. Every day.

They wanted to know if a camera could do the looking.

The answer is yes, and it has been yes for a couple of years now. That was not the interesting part of the conversation. The interesting part was the question underneath, which they had not asked and which decides whether a project like this survives its first term.

Who labels the training data, forever?

I should say up front that this engagement has not started. We delivered the architecture in March and they are choosing between deployment scenarios. I am writing about it anyway, because the thinking is the part that transfers and it does not depend on the outcome.

Every computer vision proposal I have seen underestimates the same thing, and it is never the model.

You need images of trays. Thousands of them, from the actual camera angle, under the actual lighting, containing the actual food that this district serves. Somebody has to draw a box around each item and name it. That is annotation, and it is the real cost of a vision system.

Then term ends. The menu changes. A supplier switches from a round roll to a square one. The kitchen introduces a new dessert. Your model has never seen any of it, and its accuracy on those items is zero.

So annotation is not a project cost. It is an operating cost, forever, and it is the reason most cafeteria vision pilots look excellent in week three and get quietly switched off in month five. Nobody budgeted for a person to keep drawing boxes around sandwiches until the end of time.

That is the question they had not asked. Not “can it see the tray”. It is “who keeps teaching it, after we have all moved on to something else”.

Here is the part I liked, and I want to be clear that the shape of it came from their system rather than from us.

The operator is already labelling.

Every transaction ends with a human confirming exactly what was on that tray, because that is how the student gets charged. The till record is a perfect, human-verified list of items. It is produced hundreds of times a day, by somebody doing their job, at zero additional cost, and it has been produced for years.

So the architecture stops treating annotation as a separate activity. The camera predicts. The operator does what they already do. At the end of the transaction the system compares the two, and every disagreement is a labelled training example with a human-verified answer attached to it.

Nobody draws a box. Nobody opens an annotation tool. Nobody is asked to do anything they were not already doing before the camera existed.

The pipeline that follows is unglamorous. The edge device keeps the image and its own prediction locally. Overnight it uploads both. A comparison job finds the disagreements. Open source auto-labelling models turn those into bounding boxes without a person in the loop. The training set grows on its own, the model retrains monthly at first and less often later, and updates push back to the devices over the air.

The projected accuracy curve runs from roughly 80 to 85 percent in the pilot months to above 95 percent within a year. Those are projections, not results, and I would not present them as anything else. But the mechanism producing them is not a projection. It is a till receipt.

The estimate. Specifically, I have spent years putting data collection into development estimates as though it were development.

In the scope for this system there is a line for collecting images at the pilot schools. Seven days. When I first wrote it, seven days sat in the same column as the seventeen days of application development, and to anyone reading the spreadsheet those two numbers looked like the same kind of thing.

They are not remotely the same kind of thing.

You cannot collect tray images faster by adding a developer. You cannot collect them at night. You cannot collect them at all during a school holiday. The camera has to be physically mounted above a real counter, during a real lunch service, which happens for about an hour a day, on school days only. Two or three weeks of calendar time, and no amount of money compresses it.

I have made this mistake in a way that was worse than a spreadsheet error, more than once. I have quoted an aggressive timeline, started work, and then discovered that the thing on the critical path was not code at all. It was waiting for the world to produce enough examples of itself.

What changed: data collection now comes out of the development estimate entirely and goes into the calendar as a dependency with its own start date, like a hardware lead time. It runs in parallel with the build, and it starts on day one whether or not the software is ready, because it is the only line item that cannot be recovered later.

If you take one practical thing from this post, take that one. In any project that has to learn from the physical world, find the line that is calendar time wearing a coding-time costume. There is always one, and it is usually on the critical path.

There was another one, and it belongs to whoever runs operations rather than whoever runs engineering.

What happens when the camera does not work?

Not philosophically. Concretely, at 11:40 on a Tuesday, with a queue of eleven year olds. A network switch fails, a device does not come back from a power cut, somebody knocks the mount and the camera is now pointing at the ceiling.

The answer has to be that nothing happens. The till keeps working exactly as it did before, in manual mode, and the person operating it does what they have always done. That requirement drives more of this architecture than the AI does. Detection runs entirely on the device with no internet dependency, because a system that stops working when the connection drops is worse than no system. The cloud manages the fleet and receives the overnight upload, and it is never in the path of a live transaction.

It also means the automation is only ever additive. Items the model is confident about are pushed through and the operator does not touch them. Everything else behaves exactly as it did before. There is no state in which the camera makes the job harder, which is the only version of this a cafeteria manager should agree to.

Strip out the trays and the schools and three things are left, and I now raise all three in the first conversation rather than the third.

Ask who produces the labels, and for how long. If the answer is a person hired to draw boxes, the project has an expiry date and it is roughly when that person’s contract ends.

Then look for the label you already have. Most operational processes end with a human confirming what actually happened, because some downstream system needs to be correct. An invoice. A till receipt. A signed form. That confirmation is ground truth, and it is usually free, already flowing, and being thrown away.

Then find the line in your estimate that is calendar time. It will be sitting quietly in a column of engineering days, looking exactly like them.

The best training data in most companies is already being produced by someone who has no idea they are producing it.