FlowsBook a callSign inCreate free accountSign up
Business and places data

How to remove duplicate businesses from a list: by ID, name, phone or website

Remove duplicates from business list files by choosing the right key: the place's own ID, a cleaned name, the phone's digits or the web domain. Which to use when.

Part of the guide Location intelligence automation

By the xMap team · · 4 min read

The reliable way to remove duplicates from business list files is to decide first what makes two rows the same business, then compare on that one key in a cleaned form: the place's own ID when the rows came from map data, the phone number reduced to its digits when they came from different people, the website reduced to its domain when phones are missing, and a name stripped of capitals and spaces only as a last resort. Pick the key before you pick the tool, and most of the pain goes away.

Duplicates creep in from ordinary work: two overlapping searches, three regional managers each sending their own outlet sheet, an export from your CRM merged with a list from a trade fair. Spreadsheet tools that look for identical rows miss almost all of them, because "Cafe Lumen" with +33 1 42 00 00 00 and "CAFE LUMEN " with 01 42 00 00 00 are not identical text. This article walks through the keys, a made-up example, and a flow that cleans every incoming list the same way.

Choosing the key

KeyUse it whenWatch out for
The place's own IDBoth rows came from map or places dataRows typed by people have no ID; they fall back to name and position
Phone digitsLists from different sources, typed differentlyHead offices that share one number across branches
Web domainPhones are often missing, as with suppliersChains whose branches all list the brand's site
Cleaned nameNothing else is availableDifferent businesses with common names
Your own ID fieldTwo of your systems share a customer numberOld rows where the field was left blank

Which comparison to use when cleaning a list of businesses.

The rule of thumb: the more a key is shared by different businesses, the less you should trust it alone. A phone number shared by twenty franchise branches will merge twenty real places into one, which is worse than leaving a duplicate in.

What a cleaned comparison means

Comparing well matters as much as the key. In xMap Flows the Remove duplicates block cleans each key before comparing, so you do not have to:

  • Phones are compared by their last 9 digits, so a local format and an international one with the country code are the same number.
  • Websites are compared without http, www or the page, so a menu page and the home page of one domain match.
  • Names are compared without capitals, accents or spaces at the ends.
  • The recommended setting, the same place, uses the item's own ID first and falls back to name and position for rows without one.

Two more behaviours keep it honest. Rows with nothing to compare, such as no phone when you chose phone, are all kept rather than merged with each other. And you choose which copy survives: the first one, the best rated, or the one with the highest score from an AI block.

A made-up example: three outlet sheets in Nairobi

This example is invented. A beverage distributor's three area managers in Nairobi each send a sheet of outlets they visit. Together the sheets hold 1,150 rows. Say the sales operations lead uploads them to My data as one table and runs a flow with Remove duplicates set to the same phone, keeping the first.

Suppose 960 rows remain. Looking at what was merged, most pairs are the same kiosk written two ways, but one supermarket group shows up only once although it has five stores: all five listed the head office number. The lead adds a second pass for those rows using another field, the street address, and the five stores come back. The lesson is general: run the strong key first, then look at what it merged before trusting it.

Duplicates within a run, and across runs

Remove duplicates works inside one run. A different problem is the same business arriving again next week, because a daily search finds it every day. That is the job of Only new, which remembers what the flow has already found. A clean daily list usually has both: Remove duplicates straight after the searches, then Only new before anything is sent.

If the list will also be grouped and counted, Place classifier can merge places that came in twice (the same ID, or the same name within 60 metres) while it gives each one a clean category, so one block tidies two problems.

Making it a standing rule

  1. Bring the list in with My data, or connect the searches that feed it.
  2. Add Remove duplicates with the key from the table above.
  3. Add a second Remove duplicates on another field if the first key is often shared.
  4. Send the clean list on: a Match with my table step to drop customers you already have, then a sheet or your CRM.

Cleaning is not glamorous, but every count, map and call list downstream depends on it. It is a small, early part of location intelligence automation, and it is cheap to get right once and run for ever.

Questions

Can I remove duplicates without losing information from the copy I drop?

Remove duplicates keeps one item whole and drops the other. If both copies hold different useful fields, use Place classifier with merging on, which merges the two copies so no field is lost.

What if none of the rows has the field I choose?

The step fails and lists the fields that do reach it, so you can pick another key instead of getting a silently wrong list.

Does it work on lists I upload, not only on map data?

Yes. Uploaded tables, saved lists and the results of searches are all items to it; uploaded rows simply have no place ID, so name and position, phone or website do the work.

Should I remove duplicates before or after AI scoring?

Before. Scoring the same business twice wastes work, and two scores for one place confuse whoever reads the list.

Templates that do this

Read next

More on business and places data · Every article · Templates

Put it to work

An AI employee in xMap Flows does this on live data, on its own timetable. Start from a template or describe it in plain words.