Remove Duplicate Lines

What This Tool Does

Simply click the tool to paste text in the box and the tool will compare each line to all others and return a cleaned version, with each duplicated line removed — only one of each line remaining. It does not upload or store anything that you paste, running completely in your browser.

This isn't a simple "find and replace", it's a single operation that removes all duplicates in a list, regardless of how many lines the list contains or where there are duplicated lines.

Remove Duplicate Lines (3 Steps)

1. Paste your text. List, log, or document — one item per line.
2. Choose your options. Choose whether or not to match case, whether to ignore leading or trailing spaces, and if there is more than one match, whether to retain the first or last match.
3. Click Remove Duplicates and paste the result. The cleaned list is immediately displayed — copy, download as a .txt file, or export as .csv.

For normal lists, there is no install, no sign up and no file size hassle.

Understanding Deduplication Options

This is a complete and comprehensive list of the options that are actually important—and what they do. These are the settings that are different, and affect how most tools that remove duplicate lines work, so be sure to get them right or you'll either leave behind duplicate lines or delete lines that weren't duplicates.

• Case sensitivity. If you have case sensitive matching turned on, then Apple and apple are two different lines. If you want them to be considered as the same duplicate (case insensitive) then turn it off — it might be useful for cleaning up lists of names, tags, or keywords from various sources with different capitalization.
• Whitespace trimming. This is because though you see two lines as identical, hello and hello, they are not identical to your computer because there is a space at the end of the hello line. Before comparing catches these "invisible" duplicates are usually removed by trimming whitespace — especially when data is copied from a spreadsheet or a PDF.
• Keep first vs. keep last occurrence. If a line is repeated, which one is left? "keep first" will keep the initial entries in the list, and "keep last" will be helpful when the latter entries in a list are more recent (e.g. a corrected email address listed twice).
• Blank line handling. Most tools collapse all blank lines to one and by default consider them to be identical lines. If blank lines are used as spacers, check for an "ignore blank lines" feature, which will leave the blank lines in place rather than converting them to a single line.
• Sorting the output. There are some tools that allow you to sort the deduplicated result alphabetically (A-Z or Z-A). If your data is in chronological order (a log, for example), then avoid using this function as it will change the original order, which is usually not desired.

Common Use Cases

Know the difference between Duplicate Lines and Duplicate Values.

A “duplicate line” tool compares full lines – the entire line, from the beginning to the line break has to match exactly (or match, according to the specified settings) for it to be considered as a duplicate. This is unlike a single column in a spreadsheet where just one column matches.

A line-based tool can only identify "true" duplicates (where the same row occurs more than once) when your data contains multiple columns separated by commas or tabs. If you only need to dedupe on one column, then you need a specific CSV/column deduplication tool, rather than a plain line remover.

Frequently Asked Questions