Document anonymization at scale
A large-scale anonymization pipeline for legal documents, producing usable training corpora without exposing personal data or identifying information.
Tech stack
- Python
- spaCy
- Presidio
- Named Entity Recognition
- Rules & regex
- PostgreSQL
- Batch processing
Data typology
Inventory the entities to mask (names, addresses, numbers, case references) and define consistency rules that keep documents readable.
Hybrid detection
Combine NER models with deterministic rules (regex, dictionaries) to maximize recall on sensitive data.
Consistent anonymization
Stable pseudonym substitution: the same entity gets the same replacement across the whole corpus, preserving meaning.
Quality control
Sample and manually review outputs, measure residual leakage rate, iterate on the rules.
Screenshots
Screenshots coming soon — project under NDA.