Dedupe

Deduplication reference — exact matching, fuzzy matching, hash-based dedup, bloom filters, and data quality. Use when removing duplicate records, files, or data entries.

Dedupe — Data Deduplication Reference

Quick-reference skill for deduplication strategies, algorithms, and data quality patterns.

When to Use

• Removing duplicate rows from datasets or databases
• Deduplicating files in storage systems
• Implementing fuzzy matching for near-duplicate detection
• Choosing between exact and probabilistic dedup methods
• Building ETL pipelines with deduplication stages

Commands

intro

bash

scripts/script.sh intro

Overview of deduplication — types, strategies, and tradeoffs.

exact

bash

scripts/script.sh exact

Exact deduplication — hash-based, key-based, and sorting approaches.

fuzzy

bash

scripts/script.sh fuzzy

Fuzzy deduplication — similarity measures, blocking, and record linkage.

files

bash

scripts/script.sh files

File-level deduplication — fdupes, jdupes, rdfind, and storage dedup.

algorithms

bash

scripts/script.sh algorithms

Dedup algorithms — bloom filters, HyperLogLog, MinHash, SimHash.

sql

bash

scripts/script.sh sql

SQL deduplication patterns — ROW_NUMBER, DISTINCT, GROUP BY strategies.

cli

bash

scripts/script.sh cli

Command-line dedup tools — sort, uniq, awk, and stream processing.

checklist

bash

scripts/script.sh checklist

Deduplication quality checklist and validation steps.

help

bash

scripts/script.sh help

version

bash

scripts/script.sh version

Configuration

Variable	Description
DEDUPE_DIR	Data directory (default: ~/.dedupe/)

---

Powered by BytesAgain | bytesagain.com | [e-mail]