FlowsBook a callSign inCreate free accountSign up
AI employees and automation

Measuring the work of AI employees: the right number for each kind of job

A lead finder, a watcher and a chat agent need different yardsticks. Which numbers to read for each, what hours saved really means, and a monthly routine that takes minutes.

Part of the guide AI employees

By the xMap team · · 4 min read

Measuring the work of AI employees starts by asking what each one is for, because a single number fits none of them well. A lead finder is judged on how many good, new results it brings; a watcher on whether it caught what mattered without crying wolf; a chat agent on whether its answers were right; a report writer on whether its reports passed your checks. xMap Flows gives you the raw material for each: a weekly report card, the run history, a Quality panel with tests and feedback, and the fields that checks add to every item.

This article is for managers who have a handful of AI employees running and want to know which are pulling their weight, without building a dashboard to find out.

Four kinds of job, four yardsticks

Kind of employeeWhat to measureWhere to read it
Finds things (leads, openings, sites)New results, strong fits among them, credits per strong fitThe report card and Results
Watches things (closures, rivals, pages)Alerts that led to action, and false alarmsRun history and the Team inbox
Answers people (chat, WhatsApp)Tests passing, thumbs up and down, the daily quality scoreThe Quality panel
Writes things (briefs, drafts, reports)Items that passed Check the work first timecheck_passed on each item, the run

Pick the row that matches the job, not the one that is easiest to count.

Finders: the report card

Every live AI employee gets a report card for the week against the week before: what it found, how many were strong fits (scored 75 or more by its AI step), an estimate of the hours it saved, and the credits it used. It sits on the employee's card on Flows, in its panel, and in the Monday email.

One derived number is worth working out yourself: credits divided by strong fits. It tells you what a good result costs this week. If it climbs while the market has not changed, the employee is probably scoring too much that a Filter or Only new should have removed first.

Watchers: alerts that mattered

Volume is a poor measure for a watcher. A road closure employee that sends ten alerts a day may be doing worse than one that sends two. Once a month, read back a sample of its messages in the Team inbox or its run history and mark each in your head: acted on, ignored, or wrong. Many ignored alerts mean its area or filters are too wide. A closure your team hit without warning means it is too narrow.

Agents that answer: the Quality panel

For employees that chat, the flow's Quality panel does the measuring. Tests are sample messages with what a right answer must and must never say; Run all tests gives a pass score you can rerun after any change. Feedback collects the thumbs people give under answers on the chat page, in the editor and on WhatsApp; a bad one becomes a test in one click. Watch answer quality, once switched on, scores up to 20 real answers a day against three rules (follows its instructions, stays on topic, invents no facts) and sends a notice when the score drops.

Writers: first-time pass rate

Put Check the work after the AI step and every item carries check_passed, check_issues and check_redone. The share of items that passed without being redone is a clean measure of how good the instructions are. If it falls after you change the prompt, the change made things worse.

What hours saved means

The hours saved on the report card is an estimate, worked out from the runs that found something, not a measurement of anyone's time. Use it to compare weeks and employees with each other, not to quote in a board pack. If you need a real figure, time how long the job took by hand for a week and compare that with what the employee now delivers.

A worked example (made up)

A marketing agency in Vancouver runs four AI employees for a restaurant group client. On the first Monday of the month the account lead spends fifteen minutes:

  1. The new-openings scout: strong fits rose from 6 to 9 week on week while credits stayed flat, so credits per strong fit fell. Keep.
  2. The rival watcher: its last 12 alerts include 5 the client ignored, all for chains outside the client's segment. Add a Filter on category.
  3. The reservations chat: tests pass 18 of 20; two thumbs-down last week were both about parking. Turn one into a test and add parking details to its reference file.
  4. The weekly brief writer: 4 of its 5 sections passed Check the work first time. Tighten the rule its fifth section kept failing.

The weekly review in the app turns the same reading into one decision per employee: keep it, raise its budget, change it, or let it go.

Mistakes to avoid

  • Judging every employee on results found; a quiet watcher may be the best one you have.
  • Comparing weeks across a change to the flow, such as a wider area, without noting the change.
  • Treating hours saved as fact in a business case.
  • Measuring a chat agent only by how many conversations it had rather than whether its answers were right.

To see a finder with scoring built in, start from new places every morning, only new; for an employee that writes a brief from the others' findings, try a weekly brief from your AI employees. The AI employees guide covers managing the whole team.

Questions

What counts as a strong fit?

A result its AI step scored 75 or more out of 100. The report card counts them separately from everything found.

Is hours saved accurate?

It is an estimate from the runs that found something, useful for comparing weeks and employees, not an exact count of time.

How do I measure a chat agent before customers use it?

Write test cases in the Quality panel, run them, and rerun them after every change. After launch, switch on Watch answer quality.

Can I get the numbers by email?

Yes. The Monday email carries a report card for each AI employee you own, unless you switch it off in Notifications.

Templates that do this

Read next

More on ai employees and automation · Every article · Templates

Put it to work

An AI employee in xMap Flows does this on live data, on its own timetable. Start from a template or describe it in plain words.