FlowsBook a callSign inCreate free accountSign up
AI employees and automation

How to monitor AI agent quality over time, before customers notice a slide

A daily check of real answers against three rules and your own tests, a score to follow day by day, and a warning with the likely cause when the score drops.

Part of the guide AI employees

By the xMap team · · 5 min read

To monitor AI agent quality over time, score a sample of its real answers every day against the same rules, keep each day's score, and get told when the score falls well below its recent average. In xMap Flows that is one switch, Watch answer quality, in the Watch tab of a flow's Quality panel. Once it is on, xMap AI reads up to 20 real answers from the last day, scores them, draws the result on a chart, and sends a notice and an email when the score drops.

Testing before launch tells you an assistant was right on the day you checked. It says nothing about next month. This article explains why answers drift even when nobody touches the flow, what the daily watch checks, how to read its chart, and what to do on the morning it warns you. It belongs with the wider guide to AI employees.

Why good answers go bad

An agent's answers depend on more than its instructions. Any of these can change underneath it:

  • The facts it reads. A reference file with last season's opening hours, a table of branches that missed a new one, a price list that changed.
  • The questions people ask. A new product, a local event or a holiday brings a kind of message the agent never saw in testing.
  • The flow itself. A colleague shortens the instructions, removes a tool or widens the distance a search covers.
  • The places around you. Shops open and close, so an answer that named the nearest pharmacy last month may name a closed one now.

None of these breaks a run. The flow still passes, the chat still answers, and the first sign is often a customer complaint. A daily score is how you see the slide early.

What the daily watch checks

The flow's owner or an organization admin switches the watch on, for any flow that holds conversations or has test cases. Once a day it takes up to 20 answers the flow gave real people in the last 24 hours and judges each against three rules that apply to every assistant:

  1. Did it follow its instructions?
  2. Did it stay on the topics it is meant to cover?
  3. Did it avoid inventing facts that were not in what it read?

On top of those, it uses any test cases you ticked for the watch. A case that says must never quote a delivery date is then checked against real answers too, not just the test message. Each answer checked uses a few credits, and each organization has a daily limit on how many answers are judged, so a busy flow cannot run up a surprise.

Reading the chart

The Watch tab draws one score for each day, and under it the rules broken most often and the worst answers of the period, each with a link to its run. Read it like any trend line:

What you seeWhat it usually meansWhat to do
A flat line with small wobblesNormal day-to-day variety in the questionsNothing; glance at the worst answers weekly
A sudden step downSomething changed: the flow, a file it reads, or the questionsCheck History for the date and read the worst answers
A slow slide over weeksFacts going stale, or a new kind of question growingUpdate the reference data; add tests for the new questions
One rule broken again and againA gap in the instructions or the Protection settingsFix that rule's cause and add a test case

How to read the daily score of an assistant.

A quiet flow needs care. The watch compares each day with the seven days before it and only warns when at least five answers were judged, so an assistant that gets three messages a day will show a score but rarely trigger a warning. For those, look at the chart yourself every week or two.

The morning it warns you

When the score drops by 15 points or more against the week before, the owner gets an urgent notice in the bell and an email that names the likely cause from the answers that failed. Work through it in this order:

  1. Open the worst answers from the email or the Watch tab and press See how it answered on each.
  2. Check the flow's History for a change on or just before the day of the drop.
  3. Decide the cause: a change to undo, stale data to refresh, or a new kind of question to cover.
  4. Make the fix, then use the Compare tab to run the old version and the new one side by side on your tests or on the last real chats. The price is shown first.
  5. Turn one or two of the failed answers into test cases and tick them for the watch, so the next slide in the same place is caught sooner.

A made-up example

A city tourism office in Lisbon runs a chat page that suggests museums, viewpoints and places to eat near where a visitor is standing. Its score sits steady for two months. Then, over four days, it falls by 18 points and the email names invents facts as the rule broken most.

The worst answers are all about one museum. The trace shows the agent reading a reference file that still lists the museum's old ticket arrangements after a renovation, then filling in the rest from guesswork. Nobody had changed the flow at all; the world had moved. The team updates the file, adds a test that must give the new booking rule, ticks it for the watch and confirms with Compare that the new version answers correctly. The score is back the next day.

Habits that keep quality steady

  • Leave the watch on after a calm month. Calm is what it looks like when it is working.
  • Keep a few test cases ticked for the watch that cover the answers you would most hate to see go wrong.
  • Pair the watch with Check the work for messages that go out without a person reading them: the watch reports afterwards, Check the work stops a bad answer before it leaves.
  • Note the date when you change the instructions, so a later step on the chart is easy to explain.

Questions

Which flows can have the quality watch?

Flows that hold conversations, such as a chat page or an AI agent answering messages, and flows that have test cases. The flow's owner or an organization admin switches it on.

How many answers does it check?

Up to 20 real answers from the last day, once a day, plus the test cases you tick for the watch.

When does it send a warning?

When the day's score drops by 15 points or more against the seven days before, with at least five answers judged. The owner gets a notice and an email naming the likely cause.

Does the watch cost anything?

Each answer it checks uses a few credits, and the cost counts toward the flow's monthly budget like the rest of its work.

How long is the history kept?

Each day's score is kept for a year, so you can compare seasons and spot slow changes.

Templates that do this

Read next

More on ai employees and automation · Every article · Templates

Put it to work

An AI employee in xMap Flows does this on live data, on its own timetable. Start from a template or describe it in plain words.