How to remove duplicate rows from a CSV file

Duplicate rows creep into CSV files when lists are merged, exports are run twice, or the same person signs up with slightly different details. Before you remove anything, it’s worth deciding exactly what a duplicate means for your data.

Decide what counts as a duplicate

• Exact duplicates — every column is identical. These are almost always safe to remove.
• Key duplicates — rows share an identifier such as an email address or order number, but other columns differ. You need to choose which version to keep.
• Near duplicates — the same value written differently, like Ann@Example.com and ann@example.com with a trailing space. Normalising case and whitespace before comparing catches these.

Most tools keep the first occurrence and delete later ones. If you want to keep the most recent record instead, sort the file by date (newest first) before removing duplicates.

Option 1: in a spreadsheet

In Microsoft Excel, select your data and choose Data → Remove Duplicates, then tick the columns to compare. In Google Sheets, use Data → Data cleanup → Remove duplicates. Both work well, but opening a CSV in a spreadsheet can silently change data — long numbers may turn into scientific notation and leading zeros in codes such as ZIP codes or phone numbers can disappear when the file is saved again.

Option 2: on the command line

For exact duplicates in a simple file, this one-liner keeps the first occurrence of each line and preserves order:

awk '!seen[$0]++' input.csv > output.csv

It treats each line as plain text, so it can’t compare a single column and will break on quoted fields that contain line breaks.

Option 3: in your browser

The CSV duplicate remover parses the file properly (including quoted commas and multi-line cells), lets you pick the columns to compare, and can ignore case and extra spaces. It shows how many rows were removed and gives you a clean file to download. The file is processed on your device and never uploaded.

Check the result

• Compare the before and after row counts — does the number of duplicates seem plausible?
• Spot-check a few removed values by searching the original file.
• Keep the original file until you’re sure the cleaned version is right.