How We Analyze Calorie Tracking App Reviews: Our Methodology

Sergey Oreshko - Co-founder and CEO of MyNetDiary

Our complete methodology: data sources, app selection, filtering, authenticity assessment, and the exact analysis prompt. Full transparency.

Transparency is rare in app comparison content. Most "best calorie tracker app" articles don't disclose how they select apps, filter reviews, or arrive at their conclusions. Many haven't been meaningfully updated even when they display a recent date at the top.

We think that should be different. The Diet App Scorecard publishes its complete methodology so that readers and other analysts can evaluate, replicate, or challenge our process. Everything is documented here: the data source (free and publicly accessible), the app selection criteria (verifiable on the App Store), the filtering criteria (explicit), and the analysis prompt (published verbatim below).

This methodology is applied equally to every app analyzed, including MyNetDiary. I'm the CEO of MyNetDiary and the author of this analysis, so yes — you should evaluate these findings with that context in mind. That's exactly why we publish the methodology in full: so you can judge for yourself whether the process introduces bias.

Why Monthly Reviews, Not Star Ratings

The scorecard analyzes one calendar month of user reviews at a time and recalculates the average rating from that month's reviews only. This is deliberate. The App Store has three problems that make its default view unreliable for evaluating apps.

The Review Summary skews positive

Apple generates an automatic summary of each app's reviews that appears prominently on the store page. Here's the problem: this summary is predominantly positive even when most recent reviews are negative. An app where 70% of last month's reviews are one- and two-star complaints can display a cheerful AI-generated summary suggesting users love it. It's often the first thing consumers see, and it misrepresents what users are actually saying.

"Most Helpful" shows old reviews

When you tap into Ratings & Reviews, the App Store defaults to "Most Helpful" — which typically means reviews from several years ago, heavily skewed positive. You might read the first five displayed reviews and see nothing but praise from 2020 or 2021, completely missing the current wave of complaints about bugs, redesigns, or billing problems.

The star rating is not a review rating

This one catches a lot of people off guard. The App Store displays an average star rating accumulated over the app's entire lifetime — which can span a decade or more. It's a star rating, not a review rating. It's not calculated from recent reviews. It's not weighted toward recent experience. An app that was excellent five years ago but has since raised prices, removed features, or pushed a bad redesign still carries those old five-star ratings in its average.

In our analysis, we've observed apps where the monthly review rating is 1.5 to 2.0 stars lower than the all-time average. That's a real decline in user experience that the displayed star rating completely hides.

The Diet App Scorecard exists to show you the current picture these App Store defaults can't.

How We Collect Reviews

The Diet App Scorecard is built from written customer reviews on the US App Store. This section describes exactly where those reviews come from, how we make sure we have all of them for the month, and what the source can and cannot tell us.

Primary source: App Store RSS feed

Reviews come from Apple's public customer-reviews feed for the US storefront, the same feed anyone can query for any app. For each review, the feed provides a unique review identifier, the reviewer's public display name and account identifier, the star rating, the title and text, the app version the reviewer was running, and a timestamp. We collect reviews for every app that qualifies for the month under the selection rule described above, and we collect them the same way for MyNetDiary as for every other app.

Why a single pull is not enough

Apple's feed returns a maximum of 500 reviews per app, fifty per page across ten pages, sorted most recent first. For most apps in most months, 500 is more than enough. It is not enough for the largest review bodies: MyFitnessPal produced 1,438 filtered reviews in April 2026 and 795 in May, and Simple produced 770 in July. A single pull on the last day of the month would capture only the most recent 500 and silently drop the earlier part of the month, and the reviews dropped would not be a random sample. Whatever happened in the first weeks of the month, an outage, a redesign, a billing wave, would be underweighted or missing, and the resulting average would be wrong in a direction no one could predict from the outside.

Incremental collection

To avoid that, we pull the feed for every qualifying app weekly throughout the month rather than once at month end, and have done so for every installment since the series began in February 2026. Each pull retrieves the current 500 most recent reviews; every review is stored under its unique identifier, and reviews already on file are updated in place rather than duplicated. No app in the category has come close to 500 written reviews in a single week, so no review scrolls out of the window before we have captured it. At month end, the dataset for each app is the union of every pull, restricted to reviews timestamped within the month.

Two consequences of this approach are worth stating. First, the review counts published in each installment are counts of all written reviews posted during the month that survived filtering, not a sample. Second, because the same identifiers persist across pulls, we can see when a reviewer edits a review or a rating after posting; the analysis uses the most recent version of each review, and where the same account has posted more than once, only its most recent review is counted.

Month assignment

A review belongs to the month of its timestamp in Coordinated Universal Time (UTC). Apple reports timestamps in the storefront's local time zone, so a small number of reviews posted late on the last evening of a month, US time, fall into the following month's dataset. The rule is applied identically to every app and every month.

What the feed does not include

The feed exposes written reviews only. The far larger number of star-only ratings that users submit through the in-app rating prompt never appear in it, which is one reason the scorecard's monthly averages differ from, and are generally lower and more volatile than, the App Store's displayed all-time rating. Reviews that a user deletes, or that Apple removes, after we have captured them remain in our dataset, because the feed does not report deletions; in practice this affects a handful of reviews a month. The feed covers the US storefront only, so reviews from other countries are not part of the scorecard.

Data fields

Filtering, then analysis

With the month's complete set in hand, we apply the filtering rules described below: duplicate submissions from the same account are collapsed to the most recent, exact duplicates across accounts are removed, and reviews unrelated to the app are excluded and counted. Only then is the average recalculated and the AI summary generated. Nothing in this section changes those rules; it describes how the input to them is assembled.

Alternative sources for verification

Reviews can also be obtained through third-party services like data.ai or Sensor Tower, or by manually copying and pasting from the App Store app on any Apple device. We use the RSS feed because it's free, public, and machine-readable, but we encourage readers to verify our data using whichever method they prefer.

Time window

Each scorecard analyzes reviews from one full calendar month. No partial months, no rolling windows. A clean, consistent time boundary that anyone can verify.

Why US App Store only

Two reasons.

First, verifiability. Anyone can open the App Store on a US-region Apple device and manually read the same reviews we analyze. That kind of verification is realistic for one market. It's not realistic across dozens of country-specific stores where review volume is sparse and the sheer number of stores makes it impractical.

Second, the US App Store is the largest English-language market and produces the highest review volume across all the apps we track. Even smaller apps generate enough reviews for meaningful comparison.

We may add non-US markets in the future. If we do, we'll document the change here.

Which Apps We Analyze and How We Select Them

We don't maintain a fixed, editorially chosen list. On the last day of each month, we check the US App Store's top 100 apps in the Health & Fitness category and identify all apps whose primary function is calorie tracking, food tracking, or diet tracking.

From that top 100, we exclude apps that aren't primarily calorie or food tracking: workout-only trackers, step counters, meditation apps, period trackers, sleep apps, and general fitness apps without food logging. What's left forms that month's scorecard.

Apps enter when they reach the top 100 and exit when they fall below it. If an app is popular enough to rank in the top 100 — whether organically or through paid marketing and advertising — it's relevant to consumers and belongs in the analysis. This also means we naturally capture apps that spend heavily on advertising to acquire downloads but may not deliver a satisfying user experience. The contrast between chart position and review rating is itself a finding.

The core of the calorie tracking category consistently appears in the top 100, so we have good longitudinal comparability. When apps enter or exit, we note it in that month's Trends section.

How We Filter Reviews

After pulling reviews for each app, we apply two filters:

• Random and unrelated reviews removed. Reviews that don't relate to the app experience: accidental posts, reviews about a completely different product, entries that are just random characters or test text. We report the count for each app.
• Duplicate reviews removed. Same user posting the same review multiple times, or substantially identical reviews from different accounts. The App Store RSS feed includes updated reviews alongside original submissions, so when a user edits their review, both versions may appear in the feed. We keep only the most recent version and remove earlier duplicates. Count reported.

What stays in? Everything else. We don't filter based on sentiment, star count, or tone. Negative reviews stay. Positive reviews stay. Competitor mentions, bug reports, emotional rants, feature requests — all of it stays. If MyNetDiary receives a one-star review describing a genuine frustration, it stays in and affects our calculated average exactly as it should.

The goal is simple: the filtered set should reflect what users actually think, with noise removed but no editorial thumb on the scale.

How We Calculate Ratings

After filtering, we recalculate the average star rating. Simple arithmetic mean: add up all the star ratings, divide by the number of filtered reviews. No weighting by review length, recency, or helpfulness votes. The simplest possible calculation, chosen so anyone can replicate it.

How this differs from the App Store's displayed rating

Three differences:

• Time window. The App Store displays an average accumulated over the app's entire lifetime. We calculate from one calendar month only.
• Star rating vs. review rating. The App Store includes all ratings ever given, including quick-tap ratings with no written review. We calculate from written reviews only, so every rating has context.
• Filtering. The App Store includes everything. We remove duplicates and unrelated reviews first.

These differences explain why our monthly rating can diverge significantly from the App Store's number. That divergence isn't an error — it's the point. The scorecard tells you what's happening now. The App Store star rating tells you what happened over the past decade.

How We Summarize Reviews

Each app gets a summary of about 100 to 120 words — a length we've found is the sweet spot for capturing what matters without losing nuance. Summaries don't compare apps to each other, so each one stands on its own. They mention other apps only when reviewers themselves do, since switching behavior ("I left MyFitnessPal because...") is a genuine part of the review record.

Every summary is generated by AI — currently Claude 4.6 Extended Thinking by Anthropic — using exactly the same prompt for every app in every month. The AI receives the raw reviews and the prompt below. It doesn't know which app it's analyzing. It gets the same instructions for MyNetDiary as for Cal AI, for Cronometer as for BitePal.

This is the cornerstone of the scorecard's methodology. A human summarizing nine apps' worth of reviews every month would inevitably bring varying attention, unconscious biases, and fatigue to the task. I know this because I tried it myself before switching to AI — my summaries of MyNetDiary were consistently gentler than my summaries of competitors. The AI doesn't have that problem. It applies the same analytical rigor to the 300th review as to the first. The prompt:

"Provide a summary of user reviews, about 100-120 words long, while filtering out random (unrelated) reviews. Report only the count of filtered random/unrelated and duplicate reviews, without listing details. In the summary, don't compare to other apps (mentioning other apps is OK if it's present in the reviews and warranted), so that the summary is independent. Represent positives and negatives proportionally to the total reviews body. Flag authenticity concerns only if review patterns appear suspicious (e.g., clusters of generic five-star reviews, signs of incentivized reviews, or ratings before meaningful use). If patterns appear organic, state so in one sentence. Calculate average review rating after filtering out the duplicate and random reviews."

We publish both the prompt and the AI model because transparency requires it. You can evaluate whether the instructions introduce bias. Other analysts can replicate the process with the same prompt, the same model, and the same publicly available review data, and compare their results to ours. If we change the AI model in the future, we'll document the change here with the date and rationale.

How We Assess Review Authenticity

Review authenticity is assessed by the same AI, using the same prompt, as part of each app's review analysis. The prompt instructs the AI to flag authenticity concerns only when review patterns appear suspicious, and to state that patterns appear organic when they do. The AI classifies each app into one of three designations:

• Organic. Review patterns look natural — a mix of ratings, specific feature references, proportional praise and criticism consistent with genuine use.
• Bimodal / Polarized. Strong split between high and low ratings. This isn't inherently suspicious: some apps genuinely divide users. We note the pattern so readers can evaluate it.
• Suspicious. Patterns suggesting incentivized, fake, or manipulated reviews.

What triggers Suspicious: clusters of brief, generic five-star reviews contrasting with detailed negatives; apps that prompt ratings before users have accessed any features; reviews from users who say they haven't tried the app yet; rating patterns that don't match what the reviews actually say.

What doesn't trigger it: lots of negative reviews (that's just a poorly received app, not a fake review issue), bimodal distributions where both ends have substance, or review volume differences between apps.

One thing worth addressing directly: negativity bias is real and applies to all apps equally. Frustrated users are more likely to leave reviews. But negativity bias doesn't explain why some apps show clusters of generic five-star reviews alongside detailed complaints. That's a different pattern, and it's the one we flag.

Why does this matter? Because calorie tracking apps influence health decisions. A user who downloads an app based on manipulated ratings and gets wildly inaccurate calorie estimates may make dietary choices that undermine their health goals. Review authenticity in health apps isn't academic. It has real consequences.

What the Scorecard Does Not Cover

We're upfront about the limitations:

• Google Play reviews. iOS App Store only for now. We may add Google Play in the future.
• Non-US markets. US App Store only. Manual verification works for one market; it doesn't scale across dozens of country-specific stores.
• Feature testing or analytical dimensions. The scorecard reports what users say in their reviews. We don't independently evaluate database quality, accuracy, free tier value, or ease of use here. Those analyses live in separate companion articles.
• Customer support quality. Some reviews mention support, but we don't test it independently.
• Paid review incentives. We can flag suspicious patterns, but we can't definitively prove reviews are paid.
• Apps outside the top 100. If an app doesn't make the top 100 Health & Fitness on the last day of the month, it's not included. This may miss niche apps with small but loyal user bases.

These limitations are deliberate. The scorecard is designed to do one thing well: report what users are saying right now, with a transparent and repeatable methodology. Other dimensions of app quality are addressed in separate content where they can get the depth they deserve.

Start with free,
accurate tracking today

Also available as www.plateai.com