Comparing two AI employees that do the same job
Two versions of one AI employee, run side by side for a few weeks, show which finds more of what you want for less effort. How to set up a fair comparison and decide.
Part of the guide AI employees
To compare two AI employees that do the same job, run them side by side on the same area and the same timetable for a few weeks, change only one thing between them, and judge them on what you actually use: how many useful results each finds, how many of those the other one missed, how often each fails, and what each costs to run. At the end, keep the better one, fold its winning idea into the other, or keep both if they find different things.
This article is for anyone who has built an AI employee in xMap Flows and wondered whether a different data source, a different AI instruction or a different filter would do the job better. It covers when a side-by-side test is worth it, how to keep it fair, a made-up example with two sales scouts, how to read the result, the common mistakes, and where to find each figure in the app.
When a side-by-side test is worth it
Most changes to an AI employee are small and obvious: a new city, a tighter filter. You make them and move on. A side-by-side test earns its keep when the choice is genuinely uncertain and repeats every day, for example:
- Two data sources. Nearby places from xMap's places database against Map business scan of online maps, for the same type of business.
- Two AI instructions. A strict scoring instruction against a looser one, to see which produces strong fits your team actually calls.
- Two filters. A minimum rating of 4.0 against no rating filter, to see what the stricter one throws away.
- Two timetables. Every morning against twice a week, to see whether daily runs find anything the weekly one would not.
Keeping the comparison fair
- Copy, then change one thing. Duplicate the flow and change only the part you are testing. If both the source and the instruction differ, you will not know which made the difference.
- Same area, same time. Run both on the same Area of interest and the same schedule, so they see the same market on the same days.
- Name them clearly. Give each AI employee its own name, such as "Scout A, places database" and "Scout B, map scan", so the team inbox and report cards are easy to read.
- Do not send both to customers. During the test, send results to yourself or a test channel, not to prospects or the whole team.
- Run long enough. Two to four weeks of daily runs is usually enough to see a steady difference rather than one lucky morning.
- Decide the measure first. Write down what "better" means before you look: more strong fits, fewer duplicates, fewer failed runs, or lower cost for the same results.
A worked example (made up)
Say a made-up company selling kitchen equipment in Paris runs a sales scout that finds newly listed restaurants every morning. The sales lead wonders whether searching online maps directly would find more than the places database. She duplicates the flow, swaps the data block and runs both for three weeks. All figures below are invented to show how to read the result.
| Measure, over three weeks | Scout A, places database | Scout B, map scan |
|---|---|---|
| New restaurants found | 142 | 168 |
| Rated a strong fit by the AI (75 or more) | 38 | 41 |
| Strong fits the other scout did not find | 9 | 12 |
| Results with no phone or website | 18 | 35 |
| Failed runs | 0 | 1 |
A made-up three-week comparison for an imaginary company. Not measured.
Scout B finds more restaurants and a few more strong fits, but twice as many of its results have no phone or website, which slows the sales team down. Each scout also finds strong fits the other misses. The sales lead keeps both for now: Scout A as the daily list the team works from, and Scout B once a week, matched against Scout A's results so only the restaurants Scout A missed reach the team.
Where the figures come from in the app
- The weekly report card of each AI employee shows what it found, how many strong fits, an estimate of hours saved and what it cost this week against the week before.
- The weekly review puts each employee's card in front of you once a week, with a decision for each: keep it, raise its budget, change it or let it go.
- Run history shows every run, whether it passed, and what each block passed on, so you can see where one scout loses results the other keeps.
- A monthly budget on each employee keeps the test from costing more than you planned.
Measuring the overlap
The most useful single figure is what one finds that the other does not. Save Scout A's results as a list, then add a Match with my table step to Scout B, set to keep only items not in that list. Whatever comes out is Scout B's unique contribution. If that number is near zero for weeks, Scout B is a duplicate and can go.
Mistakes to avoid
- Changing several things at once. You learn which flow won, not why.
- Judging on volume alone. More results is worse if most are duplicates or cannot be contacted.
- Stopping after two days. Daily results vary. Give the test weeks, not days.
- Letting both reach the team. People get the same restaurant twice and start ignoring both.
- Forgetting to retire the loser. Two employees doing the same job double the noise and the cost. Pause the one you do not keep.
How to set it up with AI employees
Start from a template such as New places every morning, only new, then build the pair:
- Build Scout A: Schedule every morning, Area of interest on the city, Nearby places for the type, Only new, xMap LLM to score each place, Email to yourself.
- Duplicate it as Scout B and swap Nearby places for Map business scan, leaving everything else the same.
- Rename both AI employees so their names say what differs.
- Set a monthly budget on each.
- After a week, save Scout A's results as a list and add Match with my table to Scout B, keeping only items not in it.
- Read both report cards each week, and decide in the weekly review at the end of the test.
What lands in your inbox during the test is two morning emails, one from each scout, and later only Scout B's unique finds. Each Monday the report cards and the weekly review give the figures side by side.
Questions
How long should a comparison run?
Two to four weeks of the normal timetable is usually enough for a daily employee. Weekly ones need longer, often two months, to see enough runs.
Can I compare two AI instructions without duplicating the whole flow?
Duplicating is the clearest way, because each copy keeps its own results, report card and history. Change only the instruction in the copy.
Will running two employees cost twice as much?
Roughly, while the test runs, since each makes its own searches. A monthly budget on each keeps the cost within what you planned.
What if both find different things?
Keep both, but let only one reach the team and match the other against it, so people see only what the first one missed.
Templates that do this
Read next
More on ai employees and automation · Every article · Templates