AI: how a model reads a menu page and evaluates each item
A photo of a menu page is transformed into a list of dishes with calorie and macronutrient counts. From the outside, it looks like a single action, but there are two distinct operations, and they fail in different ways: one reads the text from the photo, while the other guesses what lies behind the read name. The first rarely makes mistakes. The second always makes mistakes, and the only question is by how much.
Two operations, not one
Parsing a menu page breaks down into two independent steps, and they should not be confused—they are addressed differently.
- Read the names. The model sees the photo and writes out the lines: "Ukrainian borscht with sour cream," "Caesar salad with chicken." This is text recognition, and on a readable image, it works almost flawlessly.
- Evaluate each name. For each line, it estimates the calorie content, protein, fat, carbohydrates, and portion weight. There is no photo of the dish here, no recipe, and no scale: there is only the name and everything the model knows about dishes with that name.
A similar but not identical chain works when photographing a plate: there, the model recognizes the dish by its appearance and separately estimates the weight based on the volume in the frame. The breakdown of that chain is in the article on food recognition by photo.
Why names are not translated
The model writes out the names exactly as they are printed, in the language of the menu itself. This is not an effort-saving measure, but a requirement: a person holds this card in their hands and checks the answer against it line by line. "Caesar salad with chicken" turned into "Chicken salad" breaks the only way to compare the output with the paper—and it breaks silently, because the list remains truthful but useless.
For the same reason, the order of items and section headings are preserved: "Soups," "Main Courses." Re-sorting by calorie content might seem convenient, but you would no longer be able to flip through the card in your hands.
Where errors accumulate
The margin of error comes from the second step, and it has three sources of varying magnitude.
| Source | How significant |
|---|---|
| Specific kitchen recipe | The greatest. The amount of oil and cream is not defined by the name, yet it is exactly what shifts the number |
| Portion weight | Significant. "Pasta" can be 220 g or 400 g |
| Composition of a dish with an opaque name | Variable. "Chef's signature salad" provides no information at all |
The first two add up: a dish that is 1.5 times fattier than expected and 1.5 times larger in weight results in a twofold discrepancy. This is the limit of what should be expected from an estimate based on a name, and it cannot be reduced by improving the model—it can only be reduced by measurement, which is not present here.
The model states the portion weight explicitly, and this is done for the sake of verifiability: "600 kcal" cannot be challenged, but "600 kcal for 400 g" is visible and can be corrected. How to estimate weight yourself is in the article on portion weight.
What to do with opaque names
The line "Signature home-style dish" contains no information about the food. The honest answer to it is not a number, but an admission that there is no estimate: the item remains in the list with empty values instead of a plausible fabrication.
This is a conscious choice in favor of a gap in the list. Shortening the menu by removing an unrecognized line would mean lying about what is in it; substituting an average number would mean lying about the dish itself. An empty field reports exactly what happened.
What this analysis does not do
- Does not measure. There is not a single measurement in the chain—there is reading and estimation.
- Does not know about the kitchen. Ingredient substitutions, signature sauces, and the amount of oil used by a specific chef remain behind the scenes.
- Does not replace the question about allergens. Composition cannot be determined by name: "pasta carbonara" does not contain the word "egg," even though it contains eggs. How to ask about allergens is in a separate article, and this is a case where the cost of an error is incomparable to convenience.
- Does not replace photographing the plate. When the dish is served, it can be photographed—and that is a different, more accurate task.
How to photograph for readability
The readability of the image affects the first step, and only that step: if the line is recognized, its quality does not affect the subsequent estimation. What helps:
- photograph the entire page, not at an angle across the table;
- catch the light so there is no glare from the laminated page;
- a two-page spread is better shot in two frames: small text at the edge of the frame is harder to read;
- a long menu—by section, not all at once.
In short
- There are two operations: reading the names and evaluating them. The first is reliable, the second is an estimate.
- Names are kept as printed so the list can be checked against the card line by line.
- The main source of error is the kitchen recipe and portion weight, not the quality of recognition.
- An unevaluated item remains empty rather than being filled with a plausible number.
Frequently asked questions
- How does the app determine the calorie content of a dish from a menu photo?
- In two steps. First, the names of the items are read from the photo—this is text recognition, and on a clear image, it is almost error-free. Then, for each name, an estimate of the calorie content and macronutrients is provided: the model does not see the recipe or the portion weight; it answers the question of how much is typically in a dish with such a name.
- Why are dish names not translated into my language?
- Because a person holds the menu in their hands and compares the output with it line by line. A translated or shortened name stops matching the line in the menu, and the list remains truthful but useless. For the same reason, the original order of items and section headings are preserved.
- What is the accuracy of calorie estimation from a menu?
- This is an order-of-magnitude estimate. The greatest margin of error comes from the specific kitchen's recipe and the portion weight: a dish that is 1.5 times fattier than expected and 1.5 times larger in weight results in a twofold discrepancy. This is sufficient for comparing items with each other, but not for precise tracking.
- What happens if the name of a dish says nothing about its ingredients?
- The item remains in the list with empty values. Substituting a plausible number would mean lying about the dish, and removing the line would mean lying about what is on the menu. An empty field honestly indicates that there is no estimate.
Read next
- How a neural network recognizes food in a photo — A photo of a plate turns into a number on the screen in a few seconds, and from the outside, it looks like a measurement.
- How to determine the calorie content of a dish if it is not listed on the menu — Menus often list only the name and price.
- How to determine portion size without a scale — Portion weight is the most inaccurate variable in any calculation, including analysis by photo.
- Nutritional values in delivery apps: where the numbers in the dish card come from — In the dish card, below the photo, there are four numbers, and they look like a measurement.
This article is for general information. It is not medical advice, a diagnosis, or a prescription for treatment or a diet, and it does not replace a consultation with your doctor. If you have a health condition, are pregnant, take medication, or follow a diet prescribed to you, decisions about food belong with your doctor.
Figures from regulations, guidelines and studies are given as they stood when this article was prepared and may since have changed; check them against the primary sources. This article is not advertising, an offer, or individual advice, and neither the author nor the site owner is responsible for decisions taken on the basis of it.