Get unique values from a list in python [duplicate

Question

This question already has an answer here:

• Removing duplicates in lists 43 answers

I want to get the unique values from the following list:

[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']

The output which I require is:

[u'nowplaying', u'PBS', u'job', u'debate', u'thenandnow']

This code works:

output = []
for x in trends:
    if x not in output:
        output.append(x)
print output

is there a better solution I should use?

Answer 1

This is simple solution-

list=[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
list=set(list)

Answer 2

set can help you filter out the elements from the list that are duplicates. It will work well for str, int or tuple elements, but if your list contains dict or other list elements, then you will end up with TypeError exceptions.

Here is a general order-preserving solution to handle some (not all) non-hashable types:

def unique_elements(iterable):
    seen = set()
    result = []
    for element in iterable:
        hashed = element
        if isinstance(element, dict):
            hashed = tuple(sorted(element.iteritems()))
        elif isinstance(element, list):
            hashed = tuple(element)
        if hashed not in seen:
            result.append(element)
            seen.add(hashed)
    return result

Answer 3

For long arrays

s = np.empty(len(var))

s[:] = np.nan

for  x in  set(var):

    x_positions = np.where(var==x)

    s[x_positions[0][0]]=x


sorted_var=s[~np.isnan(s)]

Answer 4

If you want to get unique elements from a list and keep their original order, then you may employ OrderedDict data structure from Python's standard library:

from collections import OrderedDict

def keep_unique(elements):
    return list(OrderedDict.fromkeys(elements).keys())

elements = [2, 1, 4, 2, 1, 1, 5, 3, 1, 1]
required_output = [2, 1, 4, 5, 3]

assert keep_unique(elements) == required_output

In fact, if you are using Python ≥ 3.6, you can use plain dict for that:

def keep_unique(elements):
    return list(dict.fromkeys(elements).keys())

It's become possible after the introduction of "compact" representation of dicts. Check it out here. Though this "considered an implementation detail and should not be relied upon".

Answer 5

Same order unique list using only a list compression.

> my_list = [1, 2, 1, 3, 2, 4, 3, 5, 4, 3, 2, 3, 1]
> unique_list = [
>    e
>    for i, e in enumerate(my_list)
>    if my_list.index(e) == i
> ]
> unique_list
[1, 2, 3, 4, 5]

enumerates gives the index i and element e as a tuple.

my_list.index returns the first index of e. If the first index isn't i then the current iteration's e is not the first e in the list.

Edit

I should note that this isn't a good way to do it, performance-wise. This is just a way that achieves it using only a list compression.

Answer 6

By using basic property of Python Dictionary:

inp=[u'nowplaying', u'PBS', u'PBS', u'nowplaying', u'job', u'debate', u'thenandnow']
d={i for i in inp}
print d

Output will be:

set([u'nowplaying', u'job', u'debate', u'PBS', u'thenandnow'])