AI Market Research Invents Numbers You Bet Money On

Blogger Ryan Robinson asked Claude a question about his own site and got back a search volume so clean and confident that he nearly planned a month of content around it. Then he checked the obvious thing: the model had no live keyword database behind it. The number was composed, not retrieved. That is AI market research working exactly as built.

Ask ChatGPT or Claude for a niche's search volume, a platform's fee schedule, an average cost per click, or what a side hustle typically earns, and you will usually get a precise-sounding digit delivered with total confidence. It sounds right because sounding right is what the model was built to do. So treat every number a chatbot hands you as a hypothesis, not data, until you can trace it to a named, dated primary source.

This piece closes three loops: why models do this even when they know better, what one invented figure costs a part-timer in hours and dollars, and the four-step verification stack that catches most fabrications in about ten minutes with free tools.

Why AI Market Research Invents Numbers

The model writes numbers, it does not look them up

A large language model predicts likely text. When you ask for the monthly search volume of dog birthday cakes, nothing in the model queries a keyword database, because there is no keyword database to query. The model completes your request with a digit that fits the rhythm of market research writing, which is one reason invented numbers land in believable ranges, 4,400 rather than 400,004. A widely cited survey of hallucination in natural language generation documents this failure mode across systems: fluent, confident output where factual grounding is optional. Public hallucination leaderboards show the same behavior in current frontier models, which still fabricate on a measurable share of open-ended tasks. Niche numerics are the worst case. There is little repeated training text about average candle seller revenue in Ohio, so the model fills the gap with something shaped like an answer.

So can ChatGPT give you accurate search volume? Only by accident. At best it repeats a third-party estimate it once saw, stripped of the tool, the date, and any way for you to check which estimate it was.

Precision is the disguise

Researchers file these failures under AI hallucinations, but the concept that matters most to a researcher is the precision effect. Research on the precision effect suggests that specific, precise numbers are often judged more credible than vague or round ones, all else equal. A figure like 14,800 searches a month feels more rigorous than a few thousand, even when the first number was invented and the second is closer to the truth. That inverts how most people vet human claims, where suspicious exactness invites scrutiny. Chatbot precision is a property of the prose, not evidence of a lookup.

Even the citations can be fabricated

Asking the model for its source does not end the audit, because a citation is also generated text. In Mata v. Avianca, lawyers filed a brief citing cases that did not exist. ChatGPT had invented them, quotes and internal citations included, and the court sanctioned the attorneys. A source named by a chatbot is a lead to check, never proof. Checking whether an AI citation is real is a five-minute job: search the exact title, open the claimed publication, and confirm the figure appears where it is said to appear.

The Five Numbers Side Hustlers Bet Money On

Cases of an AI chatbot inventing keyword statistics, fee percentages, or earnings averages almost all map to one of five claim types, and each type has a free authoritative instrument that can falsify it in minutes. The chatbot lines below are composites of the pattern, not quotes from any specific model.

Earnings and market size deserve the most suspicion. Are AI side hustle income estimates accurate? They are exactly as accurate as the survey underneath them, which the model usually cannot name. AI earnings estimates tend to blend stale surveys, content marketing copy, and platform PR, then strip the sample size and field dates on the way to your screen. For U.S. pay benchmarks, the BLS contingent worker data and Pew Research Center's gig work surveys are free, methodical, and dated, which is everything a chatbot average is not. The incentive problem compounds this: the FTC's earnings claims guidance exists because inflated earnings claims are a recurring enforcement theme in money making offers, and a model trained on that corpus will average the hype for you.

Market size figures get the roughest treatment. Most market projections circulating online trace back to press releases from research firms selling reports. When a model repeats one without naming the firm, the report, or the year, you are reading a vibe with a decimal point.

What One Fabricated Figure Costs in Hours and Dollars

Run the math on a single invented search volume, with the assumptions on the table. Say you pick a niche because a chatbot told you the main keyword gets 14,800 searches a month. You build for 8 weeks at 5 hours a week, which is 40 hours of content, formatting, and link outreach. Priced at the federal minimum wage of $7.25, that labor is worth $290, and most side hustlers value an hour of focused building well above that floor.

Hard costs stack on top. Assume $12 for a domain, one month of an SEO tool at $30 to $50, and a small test budget for ads. You are out roughly $100 in cash and 40 hours in time before real traffic data has anything to say. Then the second bill arrives: the traffic does not come, you spend two more weeks debugging a site that was never viable, and every downstream decision inherited the original error. One fabricated digit at the top of a plan propagates through everything built on top of it.

The proportion is the argument. Verifying that first number costs about ten minutes. Ten minutes of checking against eight weeks of building is the cheapest insurance available in side hustle research.

The Verification Stack, Step by Step

This is how to verify AI market research numbers without giving up the tools. Four steps, one log entry each, using free tools to verify search volume and ad costs along the way.

1. Classify the claim. Decide which of the five types you are holding. Classification tells you instantly which instrument can falsify it, which is the entire game.

2. Demand a source, then treat it as unverified. Ask the model to name the source, the publication date, and where the figure appears. If it cannot, that is your answer, and you just saved ten minutes. If it can, remember Mata: the citation itself may be decoration. A named source moves a claim from invented to checkable, never from invented to true.

3. Route the claim to its primary instrument. For demand, Google Keyword Planner shows volume ranges at no cost inside a Google Ads account, and understanding how Trends measures interest keeps you from reading a relative index as an absolute count. For fees, open the platform's official fee page on the day you decide, because fee schedules change more often than models update. For click costs, Keyword Planner's bid columns with your actual geography and keyword set will beat any remembered average. For earnings, government labor data beats any blended mean.

4. Reconcile and log. Keep a two-column log, AI figure next to verified figure, with the source URL and the date. After a month you own something no settings menu offers: your personal hallucination rate by claim type. That number, not the chatbot's confidence, is your trust setting.

Red Flags That a Number Is Fabricated

Triage heuristics catch most fabrications before any deep checking:

• Suspicious precision with no source. 14,800 instead of roughly 15k is the precision effect working on you. Make the digits prove something.
• Averages with no sample. Any earnings figure lacking a sample size, survey dates, and a definition of who was surveyed is an assertion. AI earnings estimates fail this test constantly.
• Sources that do not resolve. Search the exact title. If nothing loads, or the page loads and says something different, the claim is dead.
• Numbers that match nothing in the tool. The definitive test. If Keyword Planner, Trends, or the official fee page shows no version of the figure, the figure does not exist outside the chat window.
• Big round futures. Multi-billion-dollar projections for 2027 attached to no named report are press release echoes.

What about web search or browsing mode? Retrieval genuinely reduces fabrication, but it does not retire your audit. Retrieved snippets can be stale, can be SEO spam, or can be misread by the model, so the routing step stays. Open the primary source yourself.

Where AI Still Earns Its Place in Niche Research

None of this means closing the chat window. The failure is narrow: AI is unreliable as a data source. Reposition it as the operator on hypothesis duty, and keep the job of verifying AI answers for yourself.

AI niche research is genuinely strong at generating candidates. Ask for twenty angles on a niche, competitor positioning questions, customer interview scripts, or objections you have not considered, and the output is fast and cheap. Paste in documents you supply, a fee schedule, a labor statistics table, a competitor's pricing page, and ask questions about them. When you provide the data, the model cannot invent the underlying numbers, only misread them, and misreads are easy to catch.

You already run this playbook on advertised gig pay. Every earnings screenshot gets discounted until it survives scrutiny, because incentives bend claims. A chatbot has no malice, but it also has no brakes, which is the more dangerous combination: fluent numbers with nothing behind them. Extend the same audit standard to the machine, and AI converts from a fake data source into the fastest hypothesis generator you own.

The 10-Minute Checklist Before You Spend a Dollar

Run this before hours or money move. It takes about ten minutes for the two or three numbers that actually gate the decision.

1. List every number the chatbot gave you, including digits buried in prose. (1 minute)
2. Classify each into the five claim types. (1 minute)
3. Ask for named, dated sources and mark each answer as a lead to check, never as confirmation. (2 minutes)
4. Route the gating numbers to their instruments: Keyword Planner and Trends for demand, the official fee page for fees, bid columns for click costs, labor statistics for earnings. (4 minutes)
5. Log AI figure versus verified figure with the source URL and date. (2 minutes)

The decision rule is one sentence: no number survives without a primary source. If it cannot be traced, it cannot be budgeted. You will still run AI market research tomorrow, and you should. The habit that separates people who compound from people who restart every eight weeks is simply checking the numbers before betting on them.