Testing an AI employee before it goes live: a checklist

Test an AI employee in three passes before you deploy it. First a test run, which works on live data but sends nothing, to check what it finds and what it would send. Then a Run once with yourself as the only recipient, to see the real email or message arrive. Then read the cost of a run and a month, and decide how much the employee may do on its own. Only then deploy.

This is for anyone about to switch on a new AI employee, built from a template, by Ask xMap or by hand. It covers the difference between the two kinds of run, a checklist to go through, a made-up example, the mistakes that show up only after deploying, and the settings that make the first live week safe.

Two kinds of run

Both use the data services, so both are counted for the data they fetch. Only the estimates on the canvas while you edit cost nothing.

While you edit, blocks that find or filter things show an estimate of how many items they would pass on, marked est. Those numbers come from sample data and cost nothing; they are a guide to the shape of the flow, not a result.

The checklist

1. Does every block have what it needs? A block marked with ! on the canvas names what is missing: an account, an area, a setting. Fix those first.
2. Run a test. Read the count on each block. Did the search find a sensible number? Did each later block keep a sensible share?
3. Open a few items. Click a block in the run panel to see its fields and first items. Are the places in the right city? Are the fields you plan to use in the email actually filled?
4. Read the preview of what it would send. The run panel shows the first email or message as it would go out. Check the subject, the fields, the links.
5. If it scores, read the reasons for the top and bottom few. They tell you whether the instructions say what you meant.
6. Test with your own input if the flow starts from a webhook, a chat or new rows: Test with lets you give it the exact input you expect.
7. Run once with yourself as the recipient. Change the email's To, or the Slack channel, to your own for one run, and check what arrives on your phone and computer.
8. Read the cost. Run once shows the price of a run beside it, and the month's estimate sits beside Deploy. A schedule every 15 minutes costs very differently from one every morning.
9. Decide how much it does alone. Ask me first holds what it sends for your approval; Act and tell me sends and leaves a note.
10. Put the real recipients back, then deploy.

A worked example

A made-up case: a property manager in Dubai builds an employee that checks, every weekday, the road incidents near twelve buildings and emails the facilities team.

The test run finds 9 incidents across the city and keeps 2 near the buildings. Opening the items shows both have a street name and a distance, so the email will read well. The preview shows the subject as the flow's name; she changes it to say how many incidents and for which buildings. Run once sends to her own inbox; on her phone the map link opens at the right spot. The month's estimate is modest because it runs once a weekday. She sets it to Act and tell me, since a note to her own team needs no approval, puts the facilities team back as recipients and deploys.

Nothing here is unusual, and that is the point: each check took a minute, and each one would have been an awkward email to the whole team if skipped.

Mistakes that only show after deploying

• Testing only with test runs. A test never sends, so a wrong address or a Slack channel the account cannot post in is found on the first live morning.
• Forgetting that tests are not remembered. The first real run treats everything as new. If that would send hundreds of old items, run it once to yourself first.
• Leaving your own address as the only recipient. The team never hears from it. Put them back before deploying.
• Approving a schedule you did not read. Check the next runs listed under the schedule's time zone; a weekly run set to the wrong day is easy to miss.
• Skipping the reasons. Scores that look right in number can rest on the wrong facts.

The first live week

The first time you deploy a flow from a browser, a window called Ready to go live? sums up when it runs, where results go and what it costs, with Deploy and Not yet. After that, watch the employee's daily note in the Team inbox and its card on the Flows page for a week. If a run fails, Fix it with xMap AI on the notice opens the editor with the error explained. A monthly budget on the employee pauses it if it spends more than you expected.

Templates such as New places every morning, only new and Road closures in my city are good ones to practise this checklist on before your own flows.