How a neural network recognizes food in a photo

A photo of a plate turns into a number on the screen in a few seconds, and from the outside, it looks like a measurement. There is no measurement at any step: there is recognition, there is weight estimation, and there is a database. It is worth breaking down the entire chain, if only to understand which of the steps is more prone to error.

Three steps, not one

The answer "pasta with chicken, 620 kcal" is assembled from three independent operations, and they should not be confused: they make different types of errors and require different solutions.

1. What is in the image. The model assigns the image to categories of dishes and ingredients it recognizes. This is a classification task, and it is solved better than the others.
2. How much of it there is. The volume, and through it the mass, is estimated from a flat image. This is where the main margin of error appears.
3. How much energy is in it. The identified dish is compared with reference values per 100 g. The error of the reference database is added to the two previous ones.

You cannot simply multiply the errors, but adding them linearly is also incorrect: they partially cancel each other out and partially amplify each other. The final result is usually determined by the second step — the weight.

First step: the model does not measure, it recognizes

Convolutional networks and their successors are trained on labeled sets of food photos: each image is matched to a category, the networks are shown hundreds of thousands of them, and they identify the features that distinguish the categories. It has no understanding of the recipe — only statistics of shapes, colors, and textures. Published systems of this kind show high recognition accuracy on the sets they were tested on.

This leads to the first limitation to keep in mind: 95% accuracy applies to the list of categories from the training set and to images similar to the training ones. Russian home cooking rarely appears in such sets: solyanka, draniki, cottage cheese casserole, or buckwheat with gravy are almost never included in well-known international datasets.

The second limitation is more serious. Within a single category, the range of caloric content is enormous, and recognition does not resolve it. "Salad" can be leaves with cucumber or Olivier with mayonnaise. "Soup" can be vegetable broth or kharcho with fatty lamb. A model that correctly identifies the category can be off by several times if it fails to see the dressing within that category.

Second step: weight from a flat image

A photograph is a projection. The height of a mound of rice, the depth of a bowl, or the thickness of a piece of meat on it are not visible, and they can only be reconstructed by making an assumption: a standard plate of such-and-such diameter, a standard portion of such-and-such height. It is precisely this assumption that is substituted.

How costly this is is shown by a study in which three large language models evaluated 52 standardized images — individual products and composed dishes in three portion sizes. For the two best models, the mean absolute error was about 36% by weight and about 36% by energy; for the third, it was 65% and 64%. For individual macronutrients, the spread was even wider: for protein, about 61% for the best models, and for fat, from 42% to 52%.

The numbers for weight and energy almost coincide, and this is no coincidence: the weight error is transferred to caloric content almost one-to-one. The reference database provides caloric content per 100 g; being off by a third in mass means being off by a third in calories.

How to estimate portion weight yourself — using a fork, your palm, or tableware — is covered in the article on portion weight without scales. Humans are no better at this task, which is the essence of the next section.

What is fair to compare this with

A 36% error sounds like a verdict until you ask: how much do the methods used instead provide?

The authors of the same study note that the achieved accuracy is comparable to traditional self-reporting methods — food diaries and questionnaires, which form the basis of most nutrition research. People systematically underestimate what they eat, and the magnitude of this underestimation is of the same order.

A direct comparison was also made on living people: thirty-six students kept both a weighed diary and a log via an image-recognition app for ten consecutive days. The correlation for energy and most nutrients turned out to be significant, but with a systematic bias: the app overestimated energy by about 154 kcal per day, protein by 8.9 g, fat by 10.4 g, carbohydrates by 33.8 g, and underestimated fiber by 4.5 g. The authors describe the limits of agreement for individual nutrients as wide.

A review of image-based methods describes the same picture in general: underestimation of intake intensity and a bias that grows along with the portion size.

The practical conclusion from this is not "do not use it," but "do not use it for that purpose." A tool with a one-third error is not suitable for calculating a 300 kcal deficit; it is perfectly suitable for seeing that a lunch cost 1100 kcal rather than 400 — that is, for distinguishing orders of magnitude, not tens. Why this picture is needed at all and what it provides is covered in the article why count calories.

Third step: the reference database

The found dish must be linked to something, and it is linked to a calorie table. The table is compiled from averaged recipes, which have their own variance: homemade borscht can vary twofold in caloric content between two cooks, and no model will see this. Where the tables themselves come from and why the same sausage has a different number of calories in different apps is the topic of the article on food databases.

A separate case is a packaged product. There is no need to calculate based on a photo of the dish at all: there is a barcode on the package, and the values declared by the manufacturer are retrieved from it. This is the only branch where a document appears instead of an estimate — with its own caveat about why the numbers on the package do not match reality.

What actually improves the answer

The error is distributed unevenly, and part of it is removed by how the photo is taken.

• Angle from above, not strictly from above and not from the side. Strictly from above, height is lost; from the side, area is lost. An intermediate angle preserves both.
• An object of known size in the frame. A fork, spoon, or standard mug provides scale. Without scale, a plate with a 20 cm diameter and a dish with a 30 cm diameter look the same in the image.
• Nothing extra in the frame. A second plate, phone, or napkin holder adds objects that need to be classified as food or not.
• Deconstruct layered items. Burgers, shawarma, and layered salads look the same from the outside despite different fillings. If the filling is visible, the answer is more accurate.
• Name what is not visible. A spoonful of oil in porridge, sugar in tea, or mayonnaise inside a salad leaves no trace in the photo but accounts for a hundred or two calories.

The last point is the general limit of the method. Anything not in the photograph is inaccessible to estimation in principle, and no model improvement will change that.

Count it from a photo

Frequently asked questions

How accurately does the neural network calculate calories from a photo?

In published tests, the mean absolute error of the best models was about 36% for both portion weight and caloric content. This is comparable to the accuracy of food diaries kept manually by people, but it is not sufficient for calculations where tens of calories matter.

Why does the same dish get a different assessment in two photos?

The volume is being estimated, and it is reconstructed from a projection. Changing the angle, lighting, or dishware alters the visible area and the estimated height, and with them, the calculated mass. The most stable results come from a top-down shot at an angle with an object of known size in the frame.

What can the model not see in principle?

Anything that leaves no trace in the image: oil added during frying, sugar in a drink, dressing inside a salad, or the fat content of ground meat in a patty. Such things cannot be inferred from an image and require a caption for the photo.

Does it help to take a picture with a ruler or a coin?

Any object of known size helps—a fork, a spoon, a mug. It sets the scale, without which a small plate close up and a large one from afar look the same. A special reference object is not needed for this.

Why is the answer for a packaged product more accurate than for a plate?

Because it is not an estimation that works there, but a search: the values declared by the manufacturer on the packaging are retrieved via the barcode. The volume estimation step is removed, and the only question remaining is how much the declared values can be trusted.

Read next

• Why AI confidently states incorrect figures — The unpleasant feature of machine responses is not that they can be incorrect—reference books and labels can be wrong too.
• Where do apps get calorie and macronutrient data and why do the numbers differ — The same cottage cheese gets different calorie counts in two apps, and people usually assume one of them is wrong.
• How to determine portion size without a scale — Portion weight is the most inaccurate variable in any calculation, including analysis by photo.

This article is for general information. It is not medical advice, a diagnosis, or a prescription for treatment or a diet, and it does not replace a consultation with your doctor. If you have a health condition, are pregnant, take medication, or follow a diet prescribed to you, decisions about food belong with your doctor.

Figures from regulations, guidelines and studies are given as they stood when this article was prepared and may since have changed; check them against the primary sources. This article is not advertising, an offer, or individual advice, and neither the author nor the site owner is responsible for decisions taken on the basis of it.