Data engineering case study
Refugee Data Integrity: Finding 165,000 Duplicate Records
A forensic data-quality programme used record linkage, operational validation, and governed remediation to identify duplicate refugee records and restore missing contact relationships.
- Organization
- UNHCR Jordan
- Period
- 2018-2024
- Role
- Head of Data Analysis Group
The challenge
Large operational databases accumulate duplicate people, broken relationships, and missing contact details. Those defects reduce trust in reporting and can prevent a family from being reached by a service.
The approach
- Combine deterministic rules and probabilistic matching to identify records requiring review.
- Validate candidate matches with operational teams rather than allowing an algorithm to merge records automatically.
- Treat remediation as a governed operational process with traceability, stewardship, and repeatable controls.
Measured outcomes
- Identified 165,000 possible duplicate records for controlled investigation.
- Recovered 63,000 missing refugee contacts through data integrity analysis.
- Established data-quality controls that could be reused across field systems and downstream services.
Governance and safeguards
- Potential matches were candidates for review, not automatic identity decisions.
- Remediation preserved an audit trail and required accountable validation.
- The work minimized exposure of identifiable data while supporting operational correction.
Evidence and sources
The links below distinguish independently published evidence from first-person programme records.
- DATAVERSITY: The Hunt for 165,000 Duplicates
Independent editorial publication describing the method, scale, and lessons from the work.