Our Photo Scanner Runs a Better Model Now. That Was Never the Hard Part

On September 4th I made a one-line change that swapped which AI model reads your food photos. estimate-nutrition-image/index.ts used to call gpt-4o for every scan. Now, if ANTHROPIC_API_KEY is set, it calls Claude Sonnet 5 instead: same prompt, same JSON schema, same confidence field the model has to fill in as high, medium, or low. The key itself is the whole rollout plan. Set it and Sonnet runs with thinking disabled and no sampling parameters, since Sonnet 5 rejects those. Unset it and the code falls straight back to the old gpt-4o path, untouched. It’s about as safe a model swap as you can ship, and the paper trail behind the commit says exactly why we made it.

The paper is a Scientific Reports study published June 25th, titled, bluntly, “Model architecture dominates nutritional estimation accuracy in vision-language systems.” The researchers fed identical food photos through multiple vision-language models and varied everything else around them: camera angle, ingredient hints in the prompt, the wording of the prompt itself. Almost none of it moved the number. Multi-angle photography over a single photo showed no measurable benefit (RMSLE 0.627 versus 0.623). Prompt engineering showed no significant effect once you correct for testing several variants. Which model you picked, on the other hand, explained 99.6% of the variance in accuracy. That’s the actual finding behind our commit message calling Sonnet “the best-evidenced model for meal-photo nutrition,” not a hunch or a vendor’s benchmark slide. We picked the variable the research says is the only one that matters, and left everything else exactly as it was so the comparison stayed honest.

I’d like to end the story there, but the same paper keeps going past the part that justified our decision. It also ran every model in the test, including whichever one came out on top, against professional human nutritionists reading the same photos. The gap was not close. Best AI model: RMSLE 0.443. Nutritionists: 0.176. That’s a 152% gap in overall accuracy, and it gets worse on individual nutrients: the best model’s protein estimates were 672% worse than a trained human’s from the same picture. Whatever we just switched to is still, by this same study’s own numbers, nowhere near what a person who does this for a living gets from an identical photo.

So what did September 4th actually fix? Something real: which of the models actually available to us estimates more accurately, at roughly the same half-a-cent-per-scan cost gpt-4o already charged, so nobody’s food-logging budget line moved and nobody has to wait on a slower or pricier path to get a better number. That’s worth shipping, and I’d make the same call again for any model that comes along with better numbers behind it. What it isn’t is a fix for the limitation the paper is actually about. Swapping the engine in a car that tops out at 80 mph for a more efficient one that still tops out at 80 mph is a real improvement. It is not the same project as building a car that goes 120.

The honest description of what our photo scanner does, after today’s change, is this: it gives you a fast, reasonably good, self-rated estimate of what’s on your plate, using the best model anyone has benchmarked for the job, and that estimate should not be mistaken for what a dietitian would tell you looking at the same photo. Our own schema already has the confidence field built in for exactly this reason, we ask the model to flag when it isn’t sure. What we don’t do anywhere in the product yet is carry that same honesty through to the number itself. 420 calories from a photo scan shows up on screen with the exact same visual weight as 420 calories pulled from a barcode on a factory-sealed product, and those two numbers were never earned the same way. Fixing that display gap is a decision I don’t have made yet either. What I do have, in the same paper that told me which model to use, is a very direct answer to whether swapping models closes it. It doesn’t.

BITE of the Week: A Sharper Model on the Backend, a Cleaner Portion Dial Waiting on the Front

Version status is unchanged since last issue: 1.12.0, iOS build 107 and Android versionCode 138, is what’s live on both stores, and the OTA runtime is still 1.0.3. Three commits landed on main since, none tied to a new version bump yet, and they split cleanly by where they live.

Two are server-side, in supabase/functions/, so they went live the moment they were deployed, no store review and no OTA push required. The first, cut September 4th, is the model swap behind the essay above. The second, same day, is a real fix to something that was actively annoying new users. The first Meta ad cohort, on September 3rd, showed the proactive-messaging layer at its worst: a welcome message landing forty minutes after someone had already had a full conversation with Bite, a “no meal logged yet” nudge at twelve past local midnight, a “want to start by logging one thing?” ping thirty-five minutes after the user had already logged dinner through Bite directly, and four separate users told there was “no need to log on day one” the same week the app was nudging everyone else to log something. A new proactiveDeliveryGuard.ts now re-checks the condition against the live diary right before a message goes out. Has this person talked to Bite in the last 30 minutes? Are we inside the right local-time window? Is the thing we’re about to tell them actually still true? Early-tenure messages are now boxed into 09:00 to 21:00 local time, a tighter 19:00 to 22:00 for the day-one check-in, and the welcome message is skipped outright for anyone who has already started chatting. A logging bug rode along in the same commit: add_custom_meal_to_plan was writing consumed: false for every meal while the chat told the user it had been “logged,” so a meal someone described out loud to Bite could sit in the diary as merely planned. It now takes a real consumed flag and timestamp, and the diary card says Logged or Planned depending on which one actually happened.

The third change is client-side, in TodayScreen.tsx and LogCard.tsx, so it needs an OTA push or a store build to reach a phone rather than being live already like the two above. It removes the amount-in-grams row from a fresh photo-scan result, since the scanner weighs ingredients, not the whole plate, and that row used to sit there blank and scaling nothing until you saved the entry and reopened it. Portion count is now the one control, and it takes decimals: half a plate is 0.5, not a guess dressed up as grams. The same logic decides, per entry, whether you get that decimal stepper or the old whole-portion-and-grams pair, based on whether the thing you’re looking at actually carries a known serving size, so a fresh scan, a reopened voice log, and a manual “2 x 200g” entry each get the control that matches what’s actually known about them.

Three Quick Bites

1. A study published in EMBO Reports and covered August 30th found that obesity leaves a lasting epigenetic “memory” in CD4 T cells, a DNA methylation imprint that can persist for five to ten years after someone loses the weight. That lingering signature is one plausible reason obesity-related health risks, and the tendency to regain weight, can stay elevated long after the scale says otherwise, and it suggests real immune recovery needs years of sustained maintenance, not just a number on a chart returning to normal. ScienceDaily

2. Researchers from the University of Parma and Cranfield University tested 212 plant-based meat alternatives and plant-based drinks on UK shelves and found every single one contained at least one of 19 mycotoxins, with legume-based and mixed cereal-legume products carrying the most, including aflatoxins and several Fusarium toxins. Most measured levels sat below current EU safety limits for the same toxins in conventional cereals, but the researchers flagged that a diet built heavily around these products could mean a cumulative exposure nobody’s accounted for yet. ScienceDaily

3. A 10-year analysis of nearly 10,000 women aged 65 and older, drawn from the Study of Osteoporotic Fractures and published in Nutrients, found tea drinking was tied to modestly higher hip bone density over time, while drinking more than five cups of coffee a day was linked to lower bone density; moderate coffee intake showed no such harm. It’s an observational cohort, so it can’t prove either beverage is doing the causing, but a decade of repeated measurements in the same women is a sturdier signal than most diet-and-bone headlines get to work with. ScienceDaily

Tool of the Week: Nutrition5k

Since this issue is about how far a better vision model actually gets you, this week’s pick is for builders who want to check that for themselves instead of trusting a benchmark slide. Nutrition5k is Google Research’s open dataset of about 5,000 real cafeteria plates, each one weighed on a scale and broken down ingredient by ingredient into mass, calories, protein, carbs, and fat, captured with RGB photos, depth images, and multi-angle video, and released under a Creative Commons license free to use even commercially. It’s exactly the kind of ground truth the Scientific Reports paper behind today’s essay used to score every model it tested, and it’s sitting on GitHub for anyone who wants to run their own photo-logging pipeline against real measured plates instead of taking a vendor’s word for how close the estimate gets. Nutrition5k. Free, openly licensed, no account required.

Thanks for reading. If you’d rather your coach’s confidence field mean something, Mr BITE is free on iOS and Android.

Michael, building Mr BITE