Case study: wrong AI answers from 41% to 0%
An industrial company wanted to ask questions of its dealer and warranty data in plain language. We measured quality with 34 real business questions over four rounds.
The share of wrong answers fell from 41 percent to 0 percent. Where the data itself was faulty, the system declines to answer instead of guessing.
The starting point
The data sat in several systems. Anyone with a question needed a report or IT. The question was whether an AI answers reliably enough on this data to be used day to day.
What we did
- One engineer worked inside the customer's Microsoft environment, with its sign-in and permissions.
- On top of the database we built a semantic layer: terms, metrics and rules the way the business understands them.
- The AI accesses data through a dedicated gateway, read-only.
- We measured with 34 real business questions, the same every round, each answer checked individually.
The result
The share of wrong answers fell over four rounds from 41 to 0 percent. Round 2 was worse at first because the system answered more questions instead of declining. As the semantic layer was sharpened, errors then fell.
Where the source data itself was incomplete or wrong, the system declines to answer and says why. That is intended: an honest gap beats a plausible wrong number.
Everything built for the customer in the project, code, prompts and configuration, belongs to the customer.
Frequently asked questions
How do you measure whether AI answers on company data are right?
With a fixed set of real business questions whose correct answers are known. Every round asks all questions and checks each answer: correct, partly correct, correctly declined or wrong.
What is a semantic layer?
A description of the data in the business's own language: what a metric means, how it is calculated and which rules apply. The AI queries through this layer instead of guessing tables.
How long does a pilot like this take?
Eight weeks in this case. A pilot at Brutal AI usually takes less.
Read next
We do the maths with you.
Bring your licence list. In one call we see which tools can be replaced and what that means over five years.