How to remove duplicate lines in Visual Studio Cod

Question

Say you have the following text:

abc
123
abc
456
789
abc
abc

I want to remove all "abc" lines and just keep one. I don't mind sorting. The result should be like this:

abc
123
456
789

Answer 1

Not actually in vscode, but if it works, it works.

1. Open a new Excel spreadsheet
2. Paste the data into a column
3. Go to the Data tab
4. Select the column of data (if you haven't already)
5. Click Remove Duplicates (somewhat in the middle of the bar)
6. Click OK to remove duplicates.

Not the best answer, as you specified vscode, but as I said: If it works, it works :)

Answer 2

Try find & replace with regexp.

• Find : ^(.+)((?:\r?\n.*)*)(?:\r?\n\1)$

• Replace : $1$2

It is possible to introduce some variance in the first group.

Answer 3

If the order of lines is not important

Sort lines alphabetically, if they aren't already, and perform these steps:
_{(based on this related question: How do I find and remove duplicate lines from a file using Regular Expressions?)}

1. Control+F

2. Toggle "Replace mode"

3. Toggle "Use Regular Expression" (the icon with the .* symbol)

4. In the search field, type ^(.*)(\n\1)+$

5. In the "replace with" field, type $1

6. Click ("Replace All").

If the order of lines is important so you can't sort

In this case, either resort to a solution outside VS Code (see here), or - if your document is not very large and you don't mind spamming the Replace All button - follow the previous steps, but in steps 4 and 5, enter these:
_{(based on Remove specific duplicate lines without sorting)}

Caution: Blocks for files with too many lines (1000+); may cause VS Code to crash; may introduce blank lines in some cases.

• search: ((^[^\S$]*?(?=\S)(?:.*)+$)[\S\s]*?)^\2$(?:\n)?

• replace with: $1

and then click the "Replace All" button as many times as there are duplicate occurrences.

You'll know it's enough when the line count stops decreasing when you click the button. Navigate to the last line of the document to keep an eye on that.

Answer 4

To add to @Marc.2377 s reply.

If the order is important and you don't care that you just keep the last of the duplicate lines, simply search for regexp

^(.*\n)(?=(?:.*\n)*?\1)

and replace with nothing.

This will take a line and try to find ahead some more (maybe 0) lines followed by the exact same line taken. It will remove the taken line.

This is just a one-shot regex. No need to spam the replace button.

Answer 5

Install DupChecker extension, hit F1, type "Check Duplicates".

It will check dupes and ask if you want to remove them.

Answer 6

Just had the same issue and found the VSCode package "Sort lines", see the VSCode market place for details (e.g. https://marketplace.visualstudio.com/items?itemName=Tyriar.sort-lines).

This package has the option "Sorting lines (unique)", which did it for me. Take care of any white spaces at the beginning/end of lines, they influence whether lines are considered unique or not.