Skip to content

AI answers with sources: why not trust AI-written SQL

Can you trust AI data analysis? Five ways AI-written SQL produces wrong numbers, and why official numbers should come from defined metrics with sources.

By The DataxLand team· · 6 min read

"Point ChatGPT at the database and ask it for revenue" sounds like the perfect shortcut for a business without a data analyst. Today's language models write SQL quite well: the query runs, the result looks plausible, and it comes with a confident explanation. The problem is that running is not the same as being right. A query with a bad join still returns a number, and nothing on screen tells you that number is twice the real one.

This post covers five common ways AI-written SQL goes wrong on business numbers, why "the AI picks the metric, the engine builds the query" is more trustworthy for official numbers, and when letting AI write SQL is still useful. It is the approach to AI data analysis that DataxLand chose for Si.

Five ways AI-written SQL goes wrong

The examples below are illustrations; the numbers are assumed to show the mechanism.

1. A join that double-counts

Shipping fees live at the order level, but the question is by product. The SQL looks reasonable:

SELECT p.category, SUM(o.shipping_fee) AS shipping
FROM orders o
JOIN order_items i ON i.order_id = o.order_id
JOIN products p ON p.sku = i.sku
GROUP BY p.category;

An order with 3 line items and a 30,000 ₫ shipping fee gets added three times: 90,000 ₫. Total shipping for the whole shop is inflated by the average number of lines per order. No error, no warning.

2. The wrong "revenue"

"October revenue" can mean GMV, revenue after discounts, net revenue after refunds, or cash received in the wallet. The AI picks the column with the most similar name — usually the order total. On the DataxLand sample data (a simulated scented-candle shop, 1 Aug – 29 Oct 2026), GMV is 831,421,000 ₫ and net revenue is 755,554,000 ₫: a gap of about 10% just from picking a column.

3. Forgetting to filter order status

Cancelled, unpaid and test orders sit in the same table as real ones, and each marketplace records status its own way. An unfiltered COUNT(*) counts cancelled orders too — in the sample data the cancellation rate is 7.42%, enough to skew every conversion rate.

4. Wrong date and time zone

Order timestamps are usually stored in UTC. Cut "1 October" in UTC and, for a shop in Vietnam (UTC+7), orders placed between midnight and 7 a.m. local time land on 30 September. Whether a week starts on Monday or Sunday also changes weekly reports.

5. A new definition every time you ask

Ask the same question twice and the AI may write two different queries: this time it subtracts refunds, next time it doesn't. The accountant sees one number, the director sees another, and nobody knows which is right because the definition lived in a query that has since scrolled away.

FailureDoes the SQL error?Does the number look plausible?
Double-counting joinNoYes, just higher
Wrong revenueNoYes
Missing status filterNoYes
Wrong time zoneNoYes, slightly off
Definition changes every timeNoYes, but they don't match

The middle column is the scary part: none of these failures raises an error. Checking every query by hand is a data analyst's job — exactly what the business is missing.

Another way: the AI picks the metric, the engine builds the query

Instead of letting the AI write SQL, split the question into two jobs:

  1. The AI understands the question: "profit after fees last month by channel, compared with the month before" becomes a structured combination — metric profit after fees, dimension channel, period last month, comparison previous period.
  2. The metric engine builds the query from that metric's approved definition: which tables, how to join them, which statuses to filter, which time zone to use for dates. This part is deterministic code, not a language model, so the same question always produces the same query.

The five failures above are blocked at the root: join paths and grain live in the semantic model; "net revenue" has exactly one versioned definition; order statuses are normalised during the build; business dates follow the workspace's time zone. The concept is explained in detail in What is a semantic layer.

What "with sources" means

A trustworthy answer lets you check it. With Si, every answer has:

  • Sources: the metric and version used, the period, and the date the data runs to;
  • A confirmation level: Standard (a pack metric, checked against expected numbers), Approved (the workspace's own metric, checked against data and activated) or Unconfirmed (an exploratory answer);
  • How the question was understood, shown as chips — metric, dimension, filter, period — that you can edit and run again without asking the AI again;
  • The number inspector, which shows the formula, the breakdown and even the SQL the metric engine ran.

When no metric fits or the data is not enough, Si says "Couldn't answer this question" instead of guessing.

When letting AI write SQL is still useful

Some exploratory questions go beyond the existing metrics: median order value, distributions, the correlation between daily ad spend and revenue, customer cohorts. For these, letting the AI write SQL or Python in an isolated environment makes sense, as long as the results are clearly labelled.

That is exactly what Si does: when a question goes beyond the metrics, Si writes SQL or Python and runs it in the exploration sandbox — a read-only copy with no personal data. Results always carry the Unconfirmed label, you can open the code Si ran, and sandbox results cannot be pinned as dashboard tiles. A calculation you use often becomes a workspace metric, gets checked against data, and only then becomes an official number.

Question typeHow Si answersConfirmation level
Revenue, profit, return rate, top N, vs. previous period, why it droppedMetrics through the semantic layerStandard or Approved
Median, distribution, correlation, cohorts, joins not yet in the modelSQL/Python in the sandboxUnconfirmed

How DataxLand does it

Si reads business numbers only through the semantic layer: the question becomes metrics, dimensions, filters, a period and a comparison; DataxLand's engine builds the query from approved definitions (see Si and Metrics and the semantic layer). Every number can be opened in the number inspector, including the SQL that ran. Exploratory questions go to the SQL/Python sandbox with the Unconfirmed label. The same set of tools is used in the app, Telegram, Slack and over MCP, so external agents don't write their own SQL against business data either. The overview is on the semantic layer page.

FAQ

Can the AI get numbers wrong?

It can, at the step of understanding the question: for example reading "revenue" as GMV when you meant net revenue. That's why the interpretation is shown as chips you can edit. The numbers themselves are computed by the metric engine from approved definitions, not by a language model.

Should I upload my revenue file to an AI chatbot and ask?

Fine for a one-off exploratory question, if the file contains no customer personal data and you check the result yourself. For numbers you use every week to make decisions, use metrics with fixed definitions so that asking twice gives the same number.

Where can I see the SQL that ran?

In the number inspector, the SQL tab shows the query the metric engine ran and lets you copy it. For sandbox answers, open View SQL code or View Python code under the answer.

Can I use Unconfirmed numbers?

Yes, for exploring and forming hypotheses. Before putting one in an official report, turn the calculation into a workspace metric, check it and activate it; it then carries the Approved label.

Read next: What is a semantic layer · Ask Si why profit dropped · Si on the platform page

Related posts

Verified dashboards from your own data, this afternoon

Free plan with no time limit · No card required · Sample workspace ready in minutes