Fuzzy identity matching and deduplication on top of a ~3.5M-record government-data pipeline.
Role: Backend / data engineer
~3.5M source records
Normalized & parsed
Candidate pairs
Scored matches
Reviewed merges
Role
Backend / data engineer
Ownership
Matcher and dedup workflows
Scale
~3.5M source records
Pipeline
Inherited, then extended
Overview
An existing multi-stage pipeline processed large government datasets for claimant outreach. I inherited it, then built the identity-matching engine and the deduplication and reconciliation workflows that sit on top of it.
Matching workflow
Government data~3.5M records · inherited pipeline
Normalization / parsing
Candidate matching
Levenshtein similarity
Token-set similarity
Haversine geo scoring
Rules & deduplicationCo-claimant safeguards
Merge / unmergeAuditable
Export tracking
Mail reconciliationReturn-to-sender
My work starts at normalization. Upstream ingestion stages were inherited. External Processing Service Data
What I owned
A pure-Python fuzzy matching engine combining Levenshtein distance, token-set similarity, and Haversine-based geographic scoring.
Normalization and parsing ahead of matching.
Rule-based deduplication with co-claimant safeguards and auditable merge/unmerge handling.
Batch-export tracking and mailed / return-to-sender reconciliation for outreach.
Engineering challenges
Similar is not the same
Names and addresses vary in spelling, order, and format. The matcher combines edit distance, token-set similarity, and geographic scoring rather than relying on a single signal.
Reversible decisions
A wrong merge affects real outreach. Merges are auditable and can be undone, and co-claimant safeguards stop related people from being collapsed into one record.