When are two similar-looking lines still different?
Exact deduplication answers one narrow question: has this complete line already appeared? It deliberately keeps the first occurrence, preserves order and does not decide whether capitalization, spacing or spelling differences are mistakes.
Similar is not always identical.
Follow each original line into the result. Capital letters, trailing spaces, and empty lines all have a visible role.
Below, [space], [tab], and [empty line] make whitespace visible. These labels are never added to your result.
7 original lines → 5 kept; 2 exact duplicates removed.
| Input line | Visible text | Decision |
|---|---|---|
| 1 | apple | Kept as result line 1 |
| 2 | Apple | Kept as result line 2 |
| 3 | apple[space] | Kept as result line 3 |
| 4 | apple | Matches input line 1; removed |
| 5 | [empty line] | Kept as result line 4 |
| 6 | [empty line] | Matches input line 5; removed |
| 7 | apple[tab] | Kept as result line 5 |
Every input line is shown. Counts and the cleaned result cover the full list.
The first occurrence wins. “apple”, “Apple”, and “apple[space]” are different lines. Repeated empty lines collapse to one; a spaces-only line is still different from an empty line. Windows, Mac, and Unix line endings are treated as the same separator.
Worked examples
- Starting point
- Four lines: item / item followed by one space / Item / item.
- Result
- 3 lines kept, 1 removed: item, item with its space, and Item.
- Why it happens
- Only the final item is an exact repeat. The trailing space and capital I distinguish the other lines, even if they look nearly identical in another application.
- Starting point
- Four lines: pear / empty line / pear / empty line.
- Result
- 2 lines kept and 2 removed: pear, then one empty line.
- Why it happens
- An empty line is a line value too. Its first occurrence survives, rather than every blank line being deleted. A trailing line break can represent that last empty line.
- Starting point
- Two lines: precomposed é / e followed by a combining acute accent.
- Result
- Both lines remain.
- Why it happens
- The code point sequences differ. Similar appearance does not establish exact equality, and this tool does not apply Unicode normalization.
Choose for the job
- Order records a priority or sequence.
- The first occurrence stays in place, and the remaining lines are not sorted.
- Apple and apple should become one item.
- Decide on a capitalization rule before deduplicating. Changing case discards a distinction, so preserve the original if that distinction might matter.
One row is not always one item
Spreadsheet rows may contain tabs and several fields. This tool compares the whole row; it does not select an email column or understand CSV quoting. Prepare one complete item per line for the comparison you intend.
How these examples are derived
Windows, Unix and lone carriage-return line endings are split into lines, then string values are kept in first-occurrence order. Output uses line-feed separators. Spaces and case remain unchanged; empty input has zero lines.
How to use remove duplicate lines
Remove repeated lines from a pasted list. The first occurrence is kept, and the remaining lines stay in their original order.
- Paste your list with one item per line.
- Check the unique result and the number of removed lines.
- Copy the result for use in your document or spreadsheet.
A short shopping list
For the three lines “apple”, “pear”, “apple”, the result contains “apple” followed by “pear”. “Apple” with a capital A is a different line, as is “apple” followed by a space.
Good to know
- Matching is exact: capitalization and leading or trailing spaces are preserved.
- Windows and Unix line endings are normalized to line breaks in the result.
- Repeated blank lines are duplicates too; the first blank line is kept.
Common questions
Does this tool sort the result?
No. The first occurrence of each exact line stays in place. This is useful when the original order has meaning.
Why are two similar lines still present?
They may differ in capitalization, whitespace or Unicode characters that look similar. This tool intentionally does not trim, lowercase or otherwise rewrite your items.