FlowsBook a callSign inCreate free accountSign up
AI employees and automation

How to improve AI agent answers with feedback from the people it talks to

Thumbs up and down under each answer, a WhatsApp reply that says what went wrong, and one click to turn a bad answer into a test so the mistake cannot come back.

Part of the guide AI employees

By the xMap team · · 5 min read

The quickest way to improve AI agent answers with feedback is to let the people who read them rate each one, look at every thumbs down with the run that produced it, fix the cause, and turn that exchange into a test so the same slip is caught before it reaches anyone again. In xMap Flows the ratings arrive on their own from your chat page, the editor's chat and WhatsApp, and they all land in one place: the Feedback tab of the flow's Quality panel.

This is for whoever looks after an assistant that answers customers, staff or partners: a concierge chat, a store finder, a WhatsApp helper for field reps. It covers where ratings come from, how to sort them, how a single bad answer becomes a lasting fix, and what not to do with the numbers. For the bigger picture of what these assistants are, see AI employees.

Where the ratings come from

You do not need a survey or a separate form. Each place an agent answers already has a way to say whether the answer helped:

  • Your public chat page. Under every answer there is a thumbs up and a thumbs down. A visitor taps one; nothing else is asked of them.
  • Chat with this flow in the editor. The same two buttons, for you and your colleagues while you try it out.
  • WhatsApp. When a Chat flow answers on WhatsApp, the person can reply within half an hour with a thumbs emoji, or with words such as bad answer: that shop closed last year. The words become the reason.

Each answer takes one rating, so a visitor who changes their mind replaces their own vote rather than adding a second. Every rating stays linked to the run that wrote the answer, which is what makes it useful later.

Sorting the thumbs down

A thumbs down tells you something went wrong, not what. Before changing anything, read each one with its run open and put it in one of a few piles. Most fall into these:

What went wrongWhat it usually looks likeWhere the fix goes
A wrong factOpening hours or a phone number that is out of dateThe reference file or table the agent reads
A missed questionIt answered a nearby question instead of the one askedThe instructions: say what to do in that case
A bad searchPlaces from the wrong part of townThe tool's settings, such as the distance or the kinds of place
Out of boundsIt chatted about something you never meant it to coverThe Protection settings on the Chat block or the agent
ToneCorrect but curt, or far too longThe instructions, with a short example of the style you want

Five piles cover most bad answers; a few will need a closer look.

The piles matter because each has a different home. Rewording the instructions will not fix a stale phone number, and adding a reference file will not stop an agent from wandering off topic.

From one bad answer to a lasting fix

Take the thumbs down at the top of the list and work through it in order:

  1. Open the run from the rating and press See how it answered. You see the message, every tool the agent called with what it asked and what came back, the notes and passages it read, and the answer it wrote.
  2. Find the step where it went wrong. Often it is plain: the tool searched around the wrong point, or the passage it read did not hold the fact.
  3. Fix that one cause, in the place the table above points to.
  4. Back in the Feedback tab, press Turn into a test. The message, and the earlier messages when it was a follow-up, become a test case. Add what a right answer must say, what it must never say, or a check in plain words for xMap AI.
  5. Press Run all tests. The price is shown first; then each case runs as a test run, so nothing is sent or remembered. If the new case passes and the old ones still do, you are done.

Ratings that are not worth acting on, such as a thumbs down on a perfectly good answer, can be dismissed so the list stays short.

A made-up example

A garden centre group in Melbourne runs a chat page where shoppers ask which branch has a plant in stock and how to get there. The agent has the stock table from My data and Nearby places as its tools.

In its second week the Feedback tab shows six thumbs down. Three are about one branch: the agent keeps giving its old Saturday hours. The trace shows the hours came from a reference file nobody updated after the summer change. The manager fixes the file and turns one of the three into a test that must say the new closing time.

Two others are shoppers asking about lawn mowing services, which the group does not offer. The agent had suggested competitors. She adds if we do not offer it, say so and suggest what we do offer to the instructions and makes a test that must never name another business. The sixth was a shopper unhappy that a plant was out of stock, so she dismisses it. All tests pass and she deploys again.

What to do with the thumbs up

Positive ratings are easy to ignore and worth a glance. They show which kinds of question the agent handles well, which tells you what to keep stable when you next change its instructions. A good answer to an awkward question also makes a fine test case: turn it into one, so a later change cannot quietly break it.

Mistakes that waste good feedback

  • Reading the count, not the answers. Ten thumbs down about one stale fact is one fix, not ten problems.
  • Rewriting the instructions every time. Long, patched instructions start to contradict themselves; fix the data or the tool when that is the cause.
  • Fixing without a test. Without one, the same mistake can come back after an unrelated change and nobody notices until a customer does.
  • Expecting every visitor to rate. Most will not. A handful of ratings a week is enough to steer by, especially with tests doing the rest.

Questions

Do visitors need an account to rate an answer?

No. The thumbs under each answer on a public chat page work for anyone using the page, and on WhatsApp a reply with a thumbs emoji or a few words is enough.

Can someone rate the same answer many times?

No. Each answer keeps one rating, and a new tap replaces the earlier one.

Does a thumbs down change the agent by itself?

No. Ratings are listed for you to read; nothing in the agent changes until you fix the cause and save. Turning one into a test is one click, but it is still your choice.

How long are ratings kept?

A year, together with a link to the run that wrote each answer.

Who can turn feedback into tests?

People who may change the flow. Viewers can read the Feedback tab but not change it.

Templates that do this

Read next

More on ai employees and automation · Every article · Templates

Put it to work

An AI employee in xMap Flows does this on live data, on its own timetable. Start from a template or describe it in plain words.