Duplicate Candidate Detection: Why Your ATS Needs It

Last month, Greenhouse's CEO published something that made recruiters nod in unison: their duplicate tagging system got a complete overhaul. The old approach — flagging candidates when names looked similar — was generating so many false positives that teams had stopped trusting it entirely. The fix? Only flag duplicates when there's a strong signal match: same email, phone number, or LinkedIn profile.

It sounds obvious in hindsight. But most ATS platforms still rely on name-matching as their primary deduplication method. And that's costing agencies real money.

The Hidden Cost of Duplicate Records

Here's a number that should make every agency owner uncomfortable: automated pipeline QA typically flags 10-20% of candidate records as duplicates or stale. In a database of 50,000 profiles, that's up to 10,000 records creating confusion — double outreach, conflicting notes, broken analytics.

The downstream effects compound quickly:

• Double outreach: Two recruiters contact the same candidate about the same role. The candidate notices. Your agency looks disorganised.
• Split history: Half the notes and interactions sit on one profile, half on the other. Nobody has the full picture.
• Broken reporting: Pipeline conversion rates are inflated. Source attribution is wrong. You're making decisions on dirty data.
• GDPR exposure: A candidate requests data deletion. You delete one profile but miss the duplicate. That's a compliance violation.

We discovered we'd been running two separate interview processes for the same candidate — through two different recruiters. The client found out before we did. — Operations Director, London staffing agency

Why Name Matching Fails

Name-based duplicate detection sounds reasonable until you consider how names actually work in the real world:

The result: name matching produces both false positives (flagging different people as duplicates) and false negatives (missing actual duplicates with changed names). It's the worst of both worlds.

Strong Signal Matching: The Modern Approach

The shift Greenhouse announced — and that platforms like Yena have built natively — is deceptively simple: match on identifiers that are functionally unique to a person, not their name.

Three signals stand out:

1. Email Address

Email addresses are nearly unique to an individual and tend to persist for 5-10 years. A matching email between two profiles is one of the strongest duplicate indicators available. The key is normalisation: stripping whitespace, lowercasing, and handling Gmail dot-variations.

2. Phone Number

Mobile numbers persist for years and are rarely shared. But they come in wildly different formats: +49 172 1234567, 00491721234567, 0172-123-4567. Good deduplication normalises all formats to a canonical form before comparing — stripping country codes, spaces, dashes, and parentheses.

3. LinkedIn Profile URL or Member ID

LinkedIn profiles are globally unique. If two records share the same LinkedIn URL — even with formatting differences — they're the same person. Period. LinkedIn Member IDs are even more reliable since they never change, even when someone updates their profile URL.

Since switching to signal-based matching, our false positive rate dropped from about 30% to under 2%. The team actually trusts the duplicate flags now. — CTO, European recruitment technology company

How Modern Duplicate Detection Works in Practice

At import: When a CV is uploaded or a LinkedIn profile is imported, the system instantly checks email, phone, and LinkedIn against every existing record. If there's a match, the recruiter sees a clear alert: possible duplicate found with a link to the existing profile.

At manual creation: As a recruiter types a name, real-time suggestions surface similar records before they hit save. This prevents duplicates from being created in the first place.

In batch: For databases that have grown organically over years, a periodic batch scan identifies duplicate clusters across the entire database. The results go into a review queue for human confirmation.

The merge process matters too. A proper merge should preserve the complete activity history from both profiles — notes, emails, interview feedback, pipeline stages — and let the recruiter choose which data to keep when fields conflict. For a deeper look at database maintenance, check out our recruitment database cleanup guide.

GDPR and Duplicates: The Compliance Angle

Under GDPR Article 5(1)(d), personal data must be accurate. Duplicate records with conflicting information — different addresses, different consent records, different deletion requests — violate this principle directly.

The practical risk: a candidate exercises their right to erasure under Article 17. You delete their profile. But the duplicate persists with a different name or email, still containing their personal data. That's a compliance breach — and one that's entirely preventable with proper deduplication.

As AIHR's analysis of recruiting metrics highlights, data quality metrics are becoming a standard part of recruitment operations — not just a nice-to-have.

Building a Deduplication Culture

Technology solves most of the problem. But without process, new duplicates accumulate as fast as old ones are cleaned up. The agencies with the cleanest databases share a few habits:

• Search-before-create rule: Every new profile starts with a search. No exceptions.
• Import policies: Define who can bulk-import LinkedIn profiles and under what conditions. Mass imports without dedup checks are banned.
• Quarterly audits: Run the batch deduplication process every quarter. Document the results.
• Clear ownership: Someone is explicitly responsible for data quality — not as a side task, but as a core responsibility.

For teams choosing between an ATS and CRM approach, our ATS vs CRM comparison covers how each handles data quality differently. And if you're working with enriched candidate data, the database cleanup and enrichment guide dives deeper into keeping imported data clean.

We introduced a simple rule: if you create a new profile without searching first, you own the cleanup. Duplicate rate halved in six weeks. — Managing Director, Munich-based agency

FAQ: Duplicate Candidate Detection

How many duplicates does a typical recruitment database have?

Industry estimates suggest 10-20% of records in established databases are duplicates. Databases that have grown without structured import policies can have even higher rates. A RecruitCRM analysis of recruitment data management found that regular deduplication is one of the highest-ROI maintenance tasks agencies can perform.

Can duplicate detection work across different name spellings?

Name-based fuzzy matching can catch some variations, but it's unreliable for name changes like marriage or legal changes. That's exactly why strong signal matching — email, phone, LinkedIn — is more effective. These identifiers persist even when names change.

What happens to GDPR consent when merging duplicates?

The more restrictive consent status should always apply. If one profile has active consent and the duplicate does not, the merged profile should be treated as lacking consent. Always document the merge and the consent basis used.

Should duplicates be merged automatically or manually reviewed?

For very strong signals — identical email AND identical LinkedIn — automatic merging is defensible. For weaker matches like phone number only or name plus company, manual review is safer. The goal is high confidence, not high speed.

How often should I run deduplication on my database?

Real-time checks on every new record, plus a quarterly batch scan across the full database. If your database is growing rapidly with more than 100 new profiles per week, monthly batch scans are worth the effort.