Python list of dictionaries search

Question

Assume I have this:

[
{"name": "Tom", "age": 10},
{"name": "Mark", "age": 5},
{"name": "Pam", "age": 7}
]

and by searching "Pam" as name, I want to retrieve the related dictionary: {name: "Pam", age: 7}

How to achieve this ?

Answer 1

dicts=[
{"name": "Tom", "age": 10},
{"name": "Mark", "age": 5},
{"name": "Pam", "age": 7}
]

from collections import defaultdict
dicts_by_name=defaultdict(list)
for d in dicts:
    dicts_by_name[d['name']]=d

print dicts_by_name['Tom']

#output
#>>>
#{'age': 10, 'name': 'Tom'}

Answer 2

My first thought would be that you might want to consider creating a dictionary of these dictionaries ... if, for example, you were going to be searching it more a than small number of times.

However that might be a premature optimization. What would be wrong with:

def get_records(key, store=dict()):
    '''Return a list of all records containing name==key from our store
    '''
    assert key is not None
    return [d for d in store if d['name']==key]

Answer 3

Have you ever tried out the pandas package? It's perfect for this kind of search task and optimized too.

import pandas as pd

listOfDicts = [
{"name": "Tom", "age": 10},
{"name": "Mark", "age": 5},
{"name": "Pam", "age": 7}
]

# Create a data frame, keys are used as column headers.
# Dict items with the same key are entered into the same respective column.
df = pd.DataFrame(listOfDicts)

# The pandas dataframe allows you to pick out specific values like so:

df2 = df[ (df['name'] == 'Pam') & (df['age'] == 7) ]

# Alternate syntax, same thing

df2 = df[ (df.name == 'Pam') & (df.age == 7) ]

I've added a little bit of benchmarking below to illustrate pandas' faster runtimes on a larger scale i.e. 100k+ entries:

setup_large = 'dicts = [];\
[dicts.extend(({ "name": "Tom", "age": 10 },{ "name": "Mark", "age": 5 },\
{ "name": "Pam", "age": 7 },{ "name": "Dick", "age": 12 })) for _ in range(25000)];\
from operator import itemgetter;import pandas as pd;\
df = pd.DataFrame(dicts);'

setup_small = 'dicts = [];\
dicts.extend(({ "name": "Tom", "age": 10 },{ "name": "Mark", "age": 5 },\
{ "name": "Pam", "age": 7 },{ "name": "Dick", "age": 12 }));\
from operator import itemgetter;import pandas as pd;\
df = pd.DataFrame(dicts);'

method1 = '[item for item in dicts if item["name"] == "Pam"]'
method2 = 'df[df["name"] == "Pam"]'

import timeit
t = timeit.Timer(method1, setup_small)
print('Small Method LC: ' + str(t.timeit(100)))
t = timeit.Timer(method2, setup_small)
print('Small Method Pandas: ' + str(t.timeit(100)))

t = timeit.Timer(method1, setup_large)
print('Large Method LC: ' + str(t.timeit(100)))
t = timeit.Timer(method2, setup_large)
print('Large Method Pandas: ' + str(t.timeit(100)))

#Small Method LC: 0.000191926956177
#Small Method Pandas: 0.044392824173
#Large Method LC: 1.98827004433
#Large Method Pandas: 0.324505090714

Answer 4

names = [{'name':'Tom', 'age': 10}, {'name': 'Mark', 'age': 5}, {'name': 'Pam', 'age': 7}]
resultlist = [d    for d in names     if d.get('name', '') == 'Pam']
first_result = resultlist[0]

This is one way...

Answer 5

Here is a comparison using iterating throuhg list, using filter+lambda or refactoring(if needed or valid to your case) your code to dict of dicts rather than list of dicts

import time

# Build list of dicts
list_of_dicts = list()
for i in range(100000):
    list_of_dicts.append({'id': i, 'name': 'Tom'})

# Build dict of dicts
dict_of_dicts = dict()
for i in range(100000):
    dict_of_dicts[i] = {'name': 'Tom'}


# Find the one with ID of 99

# 1. iterate through the list
lod_ts = time.time()
for elem in list_of_dicts:
    if elem['id'] == 99999:
        break
lod_tf = time.time()
lod_td = lod_tf - lod_ts

# 2. Use filter
f_ts = time.time()
x = filter(lambda k: k['id'] == 99999, list_of_dicts)
f_tf = time.time()
f_td = f_tf- f_ts

# 3. find it in dict of dicts
dod_ts = time.time()
x = dict_of_dicts[99999]
dod_tf = time.time()
dod_td = dod_tf - dod_ts


print 'List of Dictionries took: %s' % lod_td
print 'Using filter took: %s' % f_td
print 'Dict of Dicts took: %s' % dod_td

And the output is this:

List of Dictionries took: 0.0099310874939
Using filter took: 0.0121960639954
Dict of Dicts took: 4.05311584473e-06

Conclusion: Clearly having a dictionary of dicts is the most efficient way to be able to search in those cases, where you know say you will be searching by id's only. interestingly using filter is the slowest solution.