This question already has an answer here:
I have a very simple JavaScript array that may or may not contain duplicates.
var names = ["Mike","Matt","Nancy","Adam","Jenny","Nancy","Carl"];
I need to remove the duplicates and put the unique values in a new array.
I could point to all the codes that I've tried but I think it's useless because they don't work. I accept jQuery solutions too.
A single line version using array filter and indexOf functions:
https://jsfiddle.net/2w0k5tz8/
Loop through, remove duplicates, and create a clone array place holder because the array index will not be updated.
Loop backward for better performance ( your loop wont need to keep checking the length of your array)
Vanilla JS: Remove duplicates using an Object like a Set
You can always try putting it into an object, and then iterating through its keys:
Vanilla JS: Remove duplicates by tracking already seen values (order-safe)
Or, for an order-safe version, use an object to store all previously seen values, and check values against it before before adding to an array.
ECMAScript 6: Use the new Set data structure (order-safe)
ECMAScript 6 adds the new
Set
Data-Structure, which lets you store values of any type.Set.values
returns elements in insertion order.Example usage:
The most concise way to remove duplicates from an array using native javascript functions is to use a sequence like below:
there's no need for
slice
norindexOf
within the reduce function, like i've seen in other examples! it makes sense to use it along with a filter function though:Yet another ES6(2015) way of doing this that already works on a few browsers is:
or even using the spread operator:
cheers!
Generic Functional Approach
Here is a generic and strictly functional approach with ES2015:
We can easily derive
unique
fromunqiueBy
or use the faster implementation utilizingSet
s:Benefits of this approach:
Performance Considerations
uniqueBy
isn't as fast as an imperative implementation with loops, but it is way more expressive due to its genericity.If you identify
uniqueBy
as the cause of a concrete performance penalty in your app, replace it with optimized code. That is, write your code first in an functional, declarative way. Afterwards, provided that you encounter performance issues, try to optimize the code at the locations, which are the cause of the problem.Memory Consumption and Garbage Collection
uniqueBy
utilizes mutations (push(x) (acc)
) hidden inside its body. It reuses the accumulator instead of throwing it away after each iteration. This reduces memory consumption and GC pressure. Since this side effect is wrapped inside the function, everything outside remains pure.tldr
===
"Smart" but naïve way
Basically, we iterate over the array and, for each element, check if the first position of this element in the array is equal to the current position. Obviously, these two positions are different for duplicate elements.
Using the 3rd ("this array") parameter of the filter callback we can avoid a closure of the array variable:
Although concise, this algorithm is not particularly efficient for large arrays (quadratic time).
Hashtables to the rescue
This is how it's usually done. The idea is to place each element in a hashtable and then check for its presence instantly. This gives us linear time, but has at least two drawbacks:
uniq([1,"1"])
will return just[1]
uniq([{foo:1},{foo:2}])
will return just[{foo:1}]
.That said, if your arrays contain only primitives and you don't care about types (e.g. it's always numbers), this solution is optimal.
The best from two worlds
A universal solution combines both approaches: it uses hash lookups for primitives and linear search for objects.
sort | uniq
Another option is to sort the array first, and then remove each element equal to the preceding one:
Again, this doesn't work with objects (because all objects are equal for
sort
). Additionally, we silently change the original array as a side effect - not good! However, if your input is already sorted, this is the way to go (just removesort
from the above).Unique by...
Sometimes it's desired to uniquify a list based on some criteria other than just equality, for example, to filter out objects that are different, but share some property. This can be done elegantly by passing a callback. This "key" callback is applied to each element, and elements with equal "keys" are removed. Since
key
is expected to return a primitive, hash table will work fine here:A particularly useful
key()
isJSON.stringify
which will remove objects that are physically different, but "look" the same:If the
key
is not primitive, you have to resort to the linear search:In ES6 you can use a
Set
:or a
Map
:which both also work with non-primitive keys.
Libraries
Both underscore and Lo-Dash provide
uniq
methods. Their algorithms are basically similar to the first snippet above and boil down to this:This is quadratic, but there are nice additional goodies, like wrapping native
indexOf
, ability to uniqify by a key (iteratee
in their parlance), and optimizations for already sorted arrays.If you're using jQuery and can't stand anything without a dollar before it, it goes like this:
which is, again, a variation of the first snippet.
Performance
Function calls are expensive in Javascript, therefore the above solutions, as concise as they are, are not particularly efficient. For maximal performance, replace
filter
with a loop and get rid of other function calls:This chunk of ugly code does the same as the snippet #3 above,
but an order of magnitude faster(as of 2017 it's only twice as fast - JS core folks are doing a great job!)ES6
ES6 provides the Set object, which makes things a whole lot easier:
or
Note that, unlike in python, ES6 sets are iterated in insertion order, so this code preserves the order of the original array.
However, if you need an array with unique elements, why not use sets right from the beginning?
Generators
A "lazy", generator-based version of
uniq
can be built on the same basis: