Remove duplicate CSV rows without losing distinct records

Choose a safe key, compare the removed records, and keep the first occurrence.

A duplicate depends on the key you choose

Two rows that look similar can describe different events. Before removing either, decide what one record represents. A repeated product code might be a duplicate in a product catalogue, but two legitimate purchases in a sales export. TableMender compares strings you choose as a key; it does not infer that business decision.

product_id,warehouse,quantity
00127,North,12
00127,South,8
00127,North,12
00904,North,3

These original teaching records have two occurrences of the same complete row. With all three columns as the key, the third data row is removed. Using only column 1 removes both later occurrences of 00127, including the South warehouse stock. That produces fewer rows but loses a distinct quantity. A smaller result is not automatically a better result.

Use the narrowest safe rule, then inspect its effect

1. Open CSV clean and preview the original with every cleaning rule off. Confirm the delimiter, header and number of records.
2. Enable duplicate removal. Leave key column numbers blank for a comparison of every field, or enter 1,2 only if product and warehouse together identify one record in your source.
3. Keep case comparison exact unless your destination treats uppercase and lowercase as equivalent. Enable trim separately only if edge whitespace is accidental.
4. Preview again. Read the duplicate count and source record examples. Confirm that a removed row represents the same event as the retained one.
5. Export All result records. Keep the original for a comparison; Restore original preview turns all cleaning rules off.

Review duplicate keys →

The first occurrence wins, without merging values

Microsoft describes a similar first-occurrence approach for Excel's Remove Duplicates workflow: selected columns establish the match, and the later whole row is removed. TableMender also keeps the first matching record in source order. It does not add quantities, combine notes, keep the newest date or prefer a nonempty field. If two matching keys contain conflicting values, resolve those conflicts in your source before deduplicating.

TableMender compares parsed field strings, whereas a spreadsheet may compare displayed or typed values. Here 00127 and 127 are different keys. Optional ignore case uses JavaScript lowercase conversion, without changing the exported spelling. It is not a locale-aware identity rule, and does not equate every linguistic variant.

Filtering changes the view; cleaning changes the result

Searching for North shows matching records but does not remove South from the result. The export selector controls whether you download every result record or every match across all preview pages. Duplicate removal, by contrast, changes the result before filtering. Always compare the result count, matching count and export count rather than judging a single page.

The header stays outside duplicate comparison. When a file has no header, turn that setting off so its first data record is eligible for cleaning and export. Column numbers are one-based; blank and repeated header labels do not change which numbered field is used.

Keep malformed records visible

Nonempty records with inconsistent field counts remain in the preview and block export. A missing key is not treated as a duplicate of an empty key. An entirely empty physical record can be removed only with the separate empty-record rule. Quoted line breaks are field contents, not extra duplicate records. TableMender does not pad missing fields or silently discard surplus ones.

← All guides