Brutal AI

GuidesCase studies

Case study: wrong AI answers from 41% to 0%

In short

An industrial company wanted to ask questions of its dealer and warranty data in plain language. We measured quality with 34 real business questions over four rounds.

The share of wrong answers fell from 41 percent to 0 percent. Where the data itself was faulty, the system declines to answer instead of guessing.

Share of wrong answers per round, 34 business questions0 %20 %40 %60 %41 %Round 156 %Round 215 %Round 30 %Round 4
Benchmark in an industrial company's pilot (2026), 34 business questions per round: 41% wrong answers in round 1, 56% in round 2, 15% in round 3, 0% in round 4. In round 2 the system answered more questions instead of declining them, and at first the share of wrong answers rose too.

The starting point

The data sat in several systems. Anyone with a question needed a report or IT. The question was whether an AI answers reliably enough on this data to be used day to day.

What we did

  • One engineer worked inside the customer's Microsoft environment, with its sign-in and permissions.
  • On top of the database we built a semantic layer: terms, metrics and rules the way the business understands them.
  • The AI accesses data through a dedicated gateway, read-only.
  • We measured with 34 real business questions, the same every round, each answer checked individually.

The result

The share of wrong answers fell over four rounds from 41 to 0 percent. Round 2 was worse at first because the system answered more questions instead of declining. As the semantic layer was sharpened, errors then fell.

Where the source data itself was incomplete or wrong, the system declines to answer and says why. That is intended: an honest gap beats a plausible wrong number.

Everything built for the customer in the project, code, prompts and configuration, belongs to the customer.

Frequently asked questions

How do you measure whether AI answers on company data are right?

With a fixed set of real business questions whose correct answers are known. Every round asks all questions and checks each answer: correct, partly correct, correctly declined or wrong.

What is a semantic layer?

A description of the data in the business's own language: what a metric means, how it is calculated and which rules apply. The AI queries through this layer instead of guessing tables.

How long does a pilot like this take?

Eight weeks in this case. A pilot at Brutal AI usually takes less.

We do the maths with you.

Bring your licence list. In one call we see which tools can be replaced and what that means over five years.