Why most AI-generated insights are useless, and the four patterns that aren't

I write this as someone whose company sells AI-generated insights. I'm going to spend most of this post telling you that the category mostly produces garbage.

This isn't a hot take. It's the inevitable result of two facts: LLMs are very good at sounding like analysts, and most products shipping "AI insights" don't have the surrounding infrastructure to give the model anything new to say. The output reads well, conveys nothing, and trains users to scroll past.

After shipping AI summaries on tens of thousands of pulses, four patterns consistently deliver value. Four others consistently don't. This is the field guide.

The four patterns that work

1. Anomaly with cause

Bad version:

Daily orders spiked to 412 today, up 87% from the seven-day average.

Good version:

Daily orders spiked to 412 today, up 87% from the seven-day average. The driver is a Meta ads campaign launched at 11am — that campaign accounted for 58% of new orders before the spike normalized at 3pm.

The difference: the second version combines the order data with the ads data. The model is no longer narrating one chart; it's joining two. That's where the value is.

This requires the surrounding system to surface candidate causes to the model. Without cross-tool data, the model is reduced to guessing. Bad anomaly-with-cause is worse than no anomaly-with-cause, because hallucinated causes teach the reader to distrust the system.

2. Cross-tool correlation

The single most valuable thing an AI can do in an analytics context is notice that two signals from two different systems are related.

Last 14 days: support ticket volume up 22%, NPS down 8 points, churn signals up. The three are correlated at the account level — 60% of new tickets come from accounts that also dropped their NPS score this week.

No single dashboard catches this. The CSM tool shows tickets. The survey tool shows NPS. The billing tool shows churn signals. The AI is doing the connect-the-dots that an analyst would have done with three browser tabs and an hour.

This is the highest-bandwidth use of AI in analytics, and it's also the hardest to get right. It requires the model to have access to multiple sources and a notion of how to align them (usually by account or time window). It rewards investment in plumbing.

3. Threshold breach with action

Bad version:

AR over 60 days is now $312k.

Good version:

AR over 60 days is now $312k — above the $250k threshold the finance team set last quarter. Suggest escalation on the top three accounts: Acme ($82k), Globex ($48k), Initech ($31k). Owners auto-tagged.

The good version doesn't just flag the breach. It names the threshold (so the reader knows this is a deliberate alert, not a random call-out), identifies the accountable parties, and proposes a next step.

The mechanism: the model is grounding in team-specific configuration that nobody else has. The threshold isn't an LLM guess; it's a number the finance lead set. The model's job is to compose the breach + threshold + suggested action.

4. Peer benchmarking

Your AR aging > 60 days runs at 14% of total AR this quarter. For SaaS companies in your size bucket on Chartcastr, the median is 7% and the p75 is 11%. You are in the highest quartile of receivables aging risk.

This pattern requires aggregate-data access (anonymized cross-customer numbers — exactly what the /benchmarks hub publishes). It also requires the discipline not to claim more than the data supports. "You are above the median" is fine. "Other companies in your industry are doing better at collections" is not — you don't know what other companies are doing; you know what their AR numbers look like in your tool.

When done right, peer benchmarking is the most viral AI feature in analytics. People share their scores. They share the scores of accounts they don't like. The screenshots travel.

The four patterns that don't

1. Commodity rephrasing

Your MRR has grown by 3.2% this week, indicating strong revenue momentum.

This is what 80% of "AI insights" products ship. The model adds nothing. It is repeating the chart in a sentence and adding a feel-good adjective. Readers learn within three weeks that this layer contains no information, and then they ignore it on every future delivery.

The test: if you removed the AI line, would the reader's understanding decrease? If no, the line is commodity rephrasing.

2. Hallucinated comparisons

Your conversion rate is performing above industry benchmarks.

What benchmarks? Whose industry? Says who? When you trace this back, you usually find the model is making it up — there's no benchmark file, no peer-data plumbing, just the LLM doing what LLMs do when they don't know: generating plausible text.

The fix is structural: the model should never make a comparison claim unless it has access to the comparison data. If it doesn't, it should be instructed to say "trend only, no peer comparison available."

3. LLM-as-calculator

Your revenue is up 14.7% year-over-year.

You computed 14.7% with an LLM? Don't.

LLMs are unreliable at arithmetic at scale. Every production analytics failure I've seen at scale traced back to a team that piped raw numbers into a model and asked it to compute. Compute deterministically — SQL, dataframes, whatever — and then narrate the result. The model's job is the sentence, not the math.

This is the single most important architectural rule for AI-in-analytics. Violate it and you will be debugging "why did the AI say 14.7% when the dashboard says 12.4%" forever.

4. Insights nobody asked for

The temptation, especially for product teams shipping their first AI feature, is to generate as many insights as possible. "Look how smart it is!" So the product surfaces six insights per dashboard load, most of them trivial.

The reader's mental model adjusts: "insights are noise." They stop reading. Now your one actually important insight, when it appears next week, gets ignored along with the noise.

The discipline is editorial restraint. A good AI insight layer says "nothing notable today" most of the time. The silence is the signal that the system is being honest about what's worth your attention.

Three months ago we had an AI tool that surfaced eight 'insights' per report. I stopped reading them after the second week. Switched to Chartcastr because most days the AI section just says "nothing material this period" and when it does say something it's usually right. That's the bar.

What Chartcastr ships, specifically

To put proof-of-work behind the framework, here's what we run in production:

• Anomaly detection that fires only when the deviation exceeds a learned per-metric threshold, with a cited cause where available. Defaults to silent.
• Cross-tool insights (feature page) that surface a related-metric explanation when the model has confident grounding in two or more connected sources.
• Timeline insights (feature page) that highlight the moment something changed, with the change-source identified.
• Anomaly insights (feature page) — the highest-stakes pattern; we ship a "why" sentence only when the cause is in the joined data, otherwise we ship the anomaly with no narrative.

We do not ship "natural-language Q&A on your data" as a flagship feature. Other products do; we think it's the most overhyped pattern in the category right now. The bar for that to work reliably is much higher than vendors are admitting publicly.

How to evaluate an AI analytics feature in five minutes

Ask the vendor for three sample insights from a real customer (anonymized). For each one, ask:

1. Could a deterministic rule have produced this? If yes, the LLM is decoration.
2. Where did the comparison come from? If the answer is vague or "the model just knows", it's hallucinated.
3. Did the model do arithmetic? If yes, ask how they handle accuracy at scale. The answer will tell you whether they've thought about it.
4. What's the silence rate? How often does the system say "nothing notable"? If the answer is "rarely", it's noise.
5. Can the customer turn off categories of insight? Mature systems give users editorial control.

If a vendor can't answer all five, the AI layer is probably commodity rephrasing.

Further reading

• 10 Things That Make an AI Slack App Great — adjacent framework on UX patterns for AI bots in Slack specifically.
• AI Insight feature page — Chartcastr's cross-tool insight feature.
• The anatomy of a daily Slack update people actually read — why the one-line AI summary specifically matters for routine reporting.
• Push analytics — the broader frame: AI insights are a component of push-analytics, not a substitute for the model.

Most AI analytics features will get better at sounding insightful. The ones that survive will be the ones that bring new information to the chart. Anything else is narration in a more expensive coat.