How to Deduplicate Large Text Datasets

Decide what a duplicate means

Repeated exports, mirrored repositories, quoted email and shared document templates create different kinds of duplication. Deduplicating a message by source ID removes repeated collection; deduplicating by text can remove legitimate observations. A customer saying “still broken” in two different incidents is not necessarily a duplicate event.

Define source identity, exact-content identity and near-duplicate family separately. For GitHub or GitLab, forks and copied files need family-level analysis. For Jira, duplicate issues may still contain distinct investigation evidence. Keep a policy describing what is retained and why.

Normalize conservatively

Retain original text and create a versioned comparison representation. Normalize line endings and agreed whitespace only where meaning survives. Lowercasing, deleting numbers or stripping code indentation can merge records whose difference is material. Record the normalizer version in your transformation log.

Separate recurring navigation or signatures from substantive content before comparison. A long common footer should not make two otherwise different documents appear identical. Inspect examples of both intended matches and intended non-matches before processing the entire corpus.

Group exact duplicates with hashes

The Python hashlib reference documents SHA-256 over bytes. A proposed comparison function is:

import hashlib

def comparison_key(text):
    normalized = text.replace("\r\n", "\n").replace("\r", "\n")
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

This function only normalizes line endings; it is not an anonymizer. Include schema and normalizer version in the grouping metadata. Compare canonical bytes within a candidate hash group before destructive removal, and retain the original source IDs and selection decision.

For a large corpus, stream records into bounded disk-backed partitions by hash prefix, then sort or group within each partition. Do not load every full document into a single in-memory dictionary. Store compact keys and offsets into restricted source files. Log failed parses instead of counting them as unique records.

Review near duplicates in stages

Near duplicates require a task-specific similarity method. A practical design is to tokenize a comparison representation, form overlapping token shingles, generate candidate groups with an approximate index and verify similarity within each candidate group. Tune thresholds using reviewed examples; no single cutoff works for code, mail and support text alike.

Synthetic example: two runbooks differ only in the port number and restart order. They may look nearly identical but have incompatible instructions. Keep them as versions with applicability context. Two copies of the same exported runbook can instead share a canonical record and multiple provenance edges.

Preserve context and evaluation integrity

Do not deduplicate messages independently if removing one breaks a conversation. Keep original event presence and use derived text representations when suppressing repeated quotes. Maintain a duplicate map from each removed source to the retained record and the rule used.

Create splits after grouping related copies. The same bug explanation in a ticket and a pull-request comment can leak evaluation answers even when the files are different. Check across platforms and document versions, then state residual overlap limitations honestly.

Troubleshoot counts and assess quality

Why did useful records disappear? Inspect whether aggressive normalization or template similarity merged materially different examples. Restore from the restricted originals and revise the policy.

Does fewer records mean better quality? Measure independent context and task usefulness, not just reduction percentage. Repetition can be legitimate evidence about frequency.

The dataset documentation paper supports making composition and processing transparent. Report the unique units, duplicate-family policy and excluded counts. Deduplication improves interpretability but does not establish rights clearance, privacy or buyer demand.