Logic

Remove duplicates

Remove duplicates keeps one of each item when the same one arrives more than once: a shop in your own table and the same shop from a scan, one company hiring in three cities, a list someone uploaded twice. Choose what makes two items the same (the place itself, the name, the phone, the website or any field) and which one to keep.

In the editor: Keeps one of each item when the same one arrives more than once, for example from two searches that overlap. Items that have nothing to compare are all kept.

Settings

The same item when

Pick one

What makes two items the same.

Same place, job or incident (recommended) By the item’s own ID: the place ID on the map, the job posting’s address, the incident ID, or a ship’s MMSI. Items without one are compared by name and position.
Same name Names compared without capitals, accents or spaces at the ends. For jobs, the company name.
Same phone Phone numbers compared by their digits only, so +1 212 555 0182 and 00966-11-555-0182 are the same.
Same website Websites compared without http, www or the page: https://www.example.com/menu and example.com are the same.
Another field Any field by its name, such as company or a customer ID from your own table.

Field name

Text

Capitals, accents and spaces at the ends are ignored.

Needed before the flow can run.

Keep

Pick one

Which of the same items goes on.

The first one The one that arrived first.
The best rated The one with the highest rating; items without one count as lowest.
The highest fit score The one with the highest score from an xMap LLM or Jev block.

What comes out

Each item that leaves the block carries these fields, to use in the blocks after it as {{field}}.

One of each item

Nothing is added: the item kept goes on with all its fields, the others stop here.

every field: Unchanged, from the item that was kept.
name: For example the kept one of two spellings.

How to use it

Recruitment and B2B sales

One row per company hiring

A staffing agency searches sales manager jobs in three cities and wants each company once, not once per posting.

Before: The same company appeared three or four times, once per city.

With the flow: One row per company, with its most recent posting.

B2B sales

Clean an uploaded list before the CRM

A sales operations team merges outlet lists from three regional managers, full of repeats.

FMCG distribution

Your customers and new shops in one list

A beverage distributor puts its customer table next to a scan of grocery stores, so reps see every shop once on one map.

Two inputs reach the block; phone numbers are compared by their digits, however each list wrote them.

Delivery apps

Keep the better listing of the same restaurant

A delivery app finds the same restaurant twice under two names and wants the listing with the better rating.

Real estate

One score per site

A developer’s brokers sometimes send the same plot twice; the board wants one line per plot, with the best score.

Banking and payments

One master list from two systems

A payments company keeps merchants in its own database and in a CRM export, and wants one list with each merchant once.

Another field compares the ID both systems share, so a merchant spelled differently in each is still one row.

Good to know

Items with nothing to compare (no phone, no website, no value in the field you name) are all kept: they are never merged with each other.
Remove duplicates looks within one run. To skip items found in earlier runs, use Only new.
When two blocks feed one block, items with exactly the same name and position are already joined into one; Remove duplicates catches the rest, written differently.
Phones are compared by their last 9 digits, so 011 555 0182 and +1 212 555 0182, or 0509998888 and +971 50 999 8888, are the same phone.
If none of the items has the field you name, the step fails and lists the fields that do reach the block.
Remove duplicates is free and runs the same in the editor’s preview and in real runs.

Works well with

Only new: Lets through only the items this flow has not found before, so a daily flow sends only what is new.
Match with my table: Looks each incoming item up in one of your tables or saved lists (My data), for example your customers.
My data: Brings in a table you uploaded or a list you saved on the My data page: your stores, customers, target accounts or sites.
Map business scan: Finds businesses on online maps live: name, category, rating, address, phone, website and whether it is open.
Job postings: Live job postings: who is hiring, for what, and where.
Sort and limit: Puts the items in order, and can keep only the first ones: the 10 best rated, the 20 nearest, the highest fit scores.
Filter: Keeps only the items that match a rule.
HubSpot: Powerful CRM that offers tools for sales, customer service, and marketing automation.

Templates that use it

Companies hiring buyers, and who to call: Companies hiring buyers this week, sized and scored, with their purchasing managers in HubSpot.
More businesses like my best customers: Businesses that look like your best customers, near them, not yet in HubSpot.
How customers rate the cafes around me: How customers rate the best cafes around each of your branches, every week.
Retail tenants for my vacant units: Retailers hiring store managers, scored as tenants for your vacant units.
Monthly outlet census of my city: Every food outlet in your city, by channel, the ones not in your CRM, every month.
Companies hiring marketers need an agency: Companies hiring marketers, with the marketing lead and a pitch for agency support.
Companies hiring in bulk this week: Companies hiring frontline staff in bulk, with the person who hires and an intro.
Roles open for weeks: offer your help: Roles open for weeks, ranked by how urgently the employer needs help.
Where tech companies are hiring: Where tech companies are hiring, sized and mapped every week.
Hottest roles to source candidates for: Which roles employers in your city posted most this week, and which to source candidates for first.
Daily hiring signals in Slack: New warehouse employers posted to your Slack every morning.
Franchises hiring managers, scored as clients: Franchises hiring managers, scored as clients, with their HR lead in HubSpot.
Universities and colleges for graduate hiring: Universities and colleges with who to contact, for graduate hiring.
Market size of a category across cities: Every coffee shop in five cities, counted and summarised every month.
Monthly hiring index for a sector: A monthly hiring index for a sector across cities.
Weekly competitive intelligence for my cafes: Weekly competitive intelligence: every rival around your branches profiled, with a strategy note.
Staffing agency engine: who is hiring and why: A staffing agency engine: who is hiring, how fast and why, with the HR lead and a personal intro.
Market entry study, refreshed monthly: A market entry study refreshed every month: size, chains, demand, the best areas and the risks.
Launch targets: specialist clinics and their directors: Specialist clinics for a launch, prioritised, with the director, a visit note and a medical affairs check.
How guests rate the hotels next to mine: How guests rate the best hotels next to yours, every week.
Tour and attraction partners near my hotels: The best tours and attractions near each hotel, with a partnership pitch ready to send.
Destination report: the attractions visitors rate best: A monthly report on the attractions visitors rate best in three destinations.
Corporate travel buyers for my hotel sales team: Companies hiring travel and events staff, with the people who book, in HubSpot.
Growing small companies for my business loans team: Small companies that are hiring, sized and scored for a working capital loan.

And 83 more in the template gallery.

Stories with it

One clean prospect list, every shop once (Organic cosmetics)
Two thousand students, never stranded by a strike (Higher education)
Eight cabs that hear about a closure first (Airport taxis)
Six drivers, fair shares, no door twice (Local delivery)

Help articles

Clean categories and duplicates: Place classifier and Remove duplicates.

Guides and terms

Deduplication: Merging records that describe the same thing.

Try it

Build a flow with Remove duplicates on the home page, no account needed, or book a call. Try it live