How to remove duplicates from a list?

Question

I want to remove duplicates from a list but what I am doing is not working:

List<Customer> listCustomer = new ArrayList<Customer>();    
for (Customer customer: tmpListCustomer)
{
  if (!listCustomer.contains(customer)) 
  {
    listCustomer.add(customer);
  }
 }

Answer 1

The "contains" method searched for whether the list contains an entry that returns true from Customer.equals(Object o). If you have not overridden equals(Object) in Customer or one of its parents then it will only search for an existing occurrence of the same object. It may be this was what you wanted, in which case your code should work. But if you were looking for not having two objects both representing the same customer, then you need to override equals(Object) to return true when that is the case.

It is also true that using one of the implementations of Set instead of List would give you duplicate removal automatically, and faster (for anything other than very small Lists). You will still need to provide code for equals.

You should also override hashCode() when you override equals().

Answer 2

If that code doesn't work, you probably have not implemented equals(Object) on the Customer class appropriately.

Presumably there is some key (let us call it customerId) that uniquely identifies a customer; e.g.

class Customer {
    private String customerId;
    ...

An appropriate definition of equals(Object) would look like this:

    public boolean equals(Object obj) {
        if (obj == this) {
            return true;
        }
        if (!(obj instanceof Customer)) {
            return false;
        }
        Customer other = (Customer) obj;
        return this.customerId.equals(other.customerId);
    }

For completeness, you should also implement hashCode so that two Customer objects that are equal will return the same hash value. A matching hashCode for the above definition of equals would be:

    public int hashCode() {
        return customerId.hashCode();
    }

It is also worth noting that this is not an efficient way to remove duplicates if the list is large. (For a list with N customers, you will need to perform N*(N-1)/2 comparisons in the worst case; i.e. when there are no duplicates.) For a more efficient solution you should use something like a HashSet to do the duplicate checking.

Answer 3

As others have mentioned, you are probably not implementing equals() correctly.

However, you should also note that this code is considered quite inefficient, since the runtime could be the number of elements squared.

You might want to consider using a Set structure instead of a List instead, or building a Set first and then turning it into a list.

Answer 4

Assuming you want to keep the current order and don't want a Set, perhaps the easiest is:

List<Customer> depdupeCustomers =
    new ArrayList<>(new LinkedHashSet<>(customers));

If you want to change the original list:

Set<Customer> depdupeCustomers = new LinkedHashSet<>(customers);
customers.clear();
customers.addAll(dedupeCustomers);

Answer 5

Two suggestions:

• Use a HashSet instead of an ArrayList. This will speed up the contains() checks considerably if you have a long list

• Make sure Customer.equals() and Customer.hashCode() are implemented properly, i.e. they should be based on the combined values of the underlying fields in the customer object.

Answer 6

Nearly all of the above answers are right but what I suggest is to use a Map or Set while creating the related list, not after to gain performance. Because converting a list to a Set or Map and then reconverting it to a List again is a trivial work.

Sample Code:

Set<String> stringsSet = new LinkedHashSet<String>();//A Linked hash set 
//prevents the adding order of the elements
for (String string: stringsList) {
    stringsSet.add(string);
}
return new ArrayList<String>(stringsSet);