How to Remove Personally Identifiable Information from Datasets

Define what must be removed

PII removal is a detection and transformation task. It is narrower than establishing that a release is anonymous. Names, contact details and account identifiers are an initial list; a person’s identity can also emerge from context or combinations. Review terminology and current requirements with the privacy owner. The ICO guidance collection is a jurisdiction-specific starting point, not a substitute for that review.

Create a policy table with entity class, source locations, action, exception process and review owner. Include credentials and commercial secrets even when they are not personal information. Decide whether a record with unresolvable risk should be excluded instead of attempting increasingly destructive masking.

Inspect structured fields and hidden locations

Inventory profile columns, foreign keys, free text, HTML attributes, URLs, file paths, image text and attachment metadata. For HubSpot, notes may contain identity details absent from the contact table. For Outlook, signatures, recipient headers and quoted history require separate handling. For Zendesk, customer uploads and internal notes may be more sensitive than the ticket subject.

Do not rely on a field’s name to establish safety. A column called summary can contain a copied email address, and an opaque URL can contain an account token. Review source and derived fields rather than dropping one profile table and assuming the remaining corpus is safe.

Combine detectors and human review

Use deterministic patterns for well-structured entities, approved entity-recognition tools for context and source-specific dictionaries where appropriate. A regex can find common email shapes but miss names or misclassify technical strings. Avoid publishing a tiny detector as a complete privacy solution.

Build a reviewed sample across languages, record types, dates and high-risk categories. Include synthetic tests for split identifiers, quoted text, international phone formats and false positives such as version numbers. Keep the review set restricted and record how annotators handle uncertain cases.

Redact consistently while preserving meaning

Use typed placeholders such as [PERSON_1] or [ACCOUNT_2] when speaker continuity matters. Keep mappings internal and scope aliases appropriately; unnecessary cross-dataset linking creates additional exposure. Apply transformations to metadata and attachments as well as the visible body.

Synthetic example:

Before: Email Jordan at jordan@example.test about account AC-772.
After: Email [PERSON_1] at [EMAIL_1] about account [ACCOUNT_1].

This illustrates a policy decision, not proof of anonymity. If the remaining incident uniquely identifies the person, stronger transformation or exclusion may be necessary. See anonymization planning for the wider release-level assessment.

Measure redaction quality

Track false negatives (sensitive entities left behind) and false positives (useful content removed). Report results by entity and language rather than one reassuring aggregate. Count residual high-risk failures explicitly. A high precision score does not mean the detector found every identifier.

Inspect a post-transformation sample and rerun detection on the final serialized files. Escaped text, joined fields and attachment extraction can introduce values absent during an earlier scan. Quarantine uncertain examples and keep reason-coded exclusions.

Troubleshoot and release under review

Why did redaction break conversations? If every speaker becomes the same placeholder, identity continuity disappears. Use an approved scoped alias scheme and retest utility.

Why are model or regex results inconsistent? Pin tool and policy versions, record configuration and keep reproducible inputs. Investigate failures instead of silently accepting different outputs.

Removal of personal information does not establish rights to license employee, customer or third-party material. Keep privacy review and contractual review distinct. For a preliminary assessment, report the detection policy, review coverage and unresolved risks without sharing raw examples.