How to Remove Duplicate Lines While Keeping Their Original Order

To remove duplicate lines without disturbing a list, read it from top to bottom, keep each line the first time it appears and drop every later repeat. Nothing is sorted and nothing moves, so the lines that survive stay in their original order.

In the Text Cleaner this is a single option: open Advanced Cleaning Options, find the Lists group and tick Remove duplicate lines. What takes some thought is deciding when two lines count as the same, because lines can look identical on screen and still differ.

Keeping the first occurrence versus sorting

There are two common ways to de-duplicate, and they give different results.

Before

pears
apples
pears
plums
apples

After, keeping the first occurrence

pears
apples
plums

After, sorting and then removing repeats

apples
pears
plums

The sorting method exists because the simplest duplicate check only compares each line with the one before it. Sorting first brings identical lines together so that check can find them. That is fine when order means nothing and destructive when the order is the point: a ranked shortlist, the steps of a procedure, a playlist, entries in the sequence they arrived. The option in the Text Cleaner keeps the first occurrence exactly as written, preserves the original order and never sorts.

Three ways to compare lines

Under the checkbox, Compare lines by offers three choices. Take this list, in which · stands for a space, cleaned with the default General Clean preset:

Blue Mug
Red Mug
blue mug
··Red Mug
Blue Mug
Green Mug

Line	Exact match	Ignore case	Ignore surrounding whitespace
Blue Mug	kept	kept	kept
Red Mug	kept	kept	kept
blue mug	kept	removed	kept
··Red Mug	kept	kept	removed
Blue Mug (second copy)	removed	removed	removed
Green Mug	kept	kept	kept

• Exact match treats two lines as duplicates only when every character is the same, capitals and spaces included.
• Ignore case treats Blue Mug and blue mug as the same line. It suits names, tags and email addresses typed by different people. Avoid it where capitals matter, as in product codes or passwords.
• Ignore surrounding whitespace disregards spaces and tabs at the start and end of each line when comparing, so an indented copy matches one that is not indented.

The three are alternatives, so no single choice removes both the lowercase copy and the indented copy. Combine a comparison with a cleanup option instead. Whitespace cleanup runs before lines are compared, so ticking Trim each line and choosing Ignore case strips the indentation first and leaves three lines: Blue Mug, Red Mug, Green Mug.

Why lines that look identical survive

When a duplicate refuses to go, the two lines are not the same sequence of characters. Because cleanup runs first, the General Clean preset already deals with three of the usual causes before any comparison happens.

Hidden difference	Example	What deals with it
Spaces or tabs at the end of a line	Red Mug·	General Clean (Trim line endings)
Doubled spaces inside the line	Red··Mug	General Clean (Collapse multiple spaces)
A non-breaking space (U+00A0) in place of a space	Red[NBSP]Mug	General Clean (Normalize non-breaking spaces)
Indentation	··Red Mug	Trim each line, or the Ignore surrounding whitespace comparison
A tab in place of a space	Red[TAB]Mug	Tabs → spaces
A zero-width space (U+200B) or a byte order mark (U+FEFF)	Red Mug[ZWSP]	Remove zero-width spaces, Remove byte order marks
Different capitals	red mug	The Ignore case comparison
A different Unicode normalisation form	é stored as one code point in one copy and as two in the other	Normalize Unicode (NFC)
Different punctuation characters	A curly apostrophe in one copy, a straight one in the other	The Typography options, if straight punctuation is acceptable

A byte order mark sits at the very start of some files, so it is the first line that fails to match its own repeat further down. Invisible Unicode characters covers that family, and removing extra spaces and tabs covers the whitespace cases.

Normalisation is the least obvious cause. Unicode allows some characters to be stored in more than one way: é can be the single code point U+00E9, or the letter e followed by the combining acute accent U+0301. The two are canonically equivalent and are meant to look and behave the same, but a character-by-character comparison sees two different sequences. Normalize Unicode (NFC) rewrites text into the composed form, so both spellings become identical. NFC leaves compatibility characters alone, which means the single-character fi ligature (U+FB01) that turns up in some PDF text still differs from the two letters fi.

Line endings are not on the list. The Text Cleaner converts Windows and old Mac line endings to LF before doing anything else, so a stray carriage return cannot keep two lines apart there. In command-line tools it can.

A worked example

A list of office locations gathered from several sources, with ·, [NBSP] and [ZWSP] standing for a space, a non-breaking space and a zero-width space:

Before

New York
San Diego·
new york
San[NBSP]Diego
Santa Fe[ZWSP]
··Santa Fe

Compared character by character, no two of these six lines are the same. With General Clean and Remove duplicate lines on Exact match, only one line goes: the trailing space and the non-breaking space are cleaned up first, which makes the two San Diego lines identical. To catch the rest, also tick Trim each line and Remove zero-width spaces, set Compare lines by to Ignore case, and press Clean Text.

After

New York
San Diego
Santa Fe

The change summary reads “1 zero-width space removed”, “1 non-breaking space converted”, “2 lines trimmed” and “3 duplicate lines removed”. How it works lists the full order in which the options are applied.

Blank lines are never treated as duplicates, so the empty lines that separate groups in a list stay where they are. If removing a line leaves two blank lines side by side, they are reduced to one as long as Collapse multiple blank lines to one is on, which it is in every preset. Tick Remove empty lines if you do not need blank lines at all.

Other places to do it

In a spreadsheet

Excel has Data > Remove Duplicates, which lets you choose the columns to compare and keeps the first occurrence of each value. It compares what appears in the cell. Google Sheets has Data > Data cleanup > Remove duplicates, and treats values that differ only in letter case as duplicates. Remove stray spaces first: Excel’s TRIM function strips ordinary spaces from the ends of a value and reduces runs between words to one but, by itself, does not remove the non-breaking space. Cleaning text for spreadsheets and databases goes further.

On the command line

awk '!seen[$0]++' list.txt

This prints each line the first time it appears and keeps the original order. seen is a table of counters keyed by the whole line, and a line is printed only while its counter is still zero. Wrap the key in tolower() to ignore case:

awk '!seen[tolower($0)]++' list.txt

sort -u list.txt also removes duplicates, but it sorts first, so the original order is lost. uniq on its own only removes a line that repeats the line directly above it. In Python, list(dict.fromkeys(lines)) keeps first occurrences in order.

When not to remove duplicate lines

Repetition is sometimes the content.

• Lyrics and poems. A chorus or refrain is supposed to repeat.
• Logs. Each repeated line is an event, and how often it occurred is the information.
• Code and configuration. Closing braces, end statements and the same setting in different sections are structure. Removing them breaks the file.
• Records where repeats are legitimate. Two identical orders, votes or survey answers are two things that happened.
• Text read in blocks. In a list of postal addresses the same city line belongs to several entries, and de-duplicating would strip it from all but the first.

De-duplication also works on whole lines only. Two lines with the same email address but a different name beside it are different lines. For that, compare on a single column in a spreadsheet.