Python: Way to speed up a repeatedly executed eval

问题:

In my code, I'm using eval to evaluate a string expression given by the user. Is there a way to compile or otherwise speed up this statement?

import math
import random

result_count = 100000
expression = "math.sin(v['x']) * v['y']"

variable = dict()
variable['x'] = [random.random() for _ in xrange(result_count)]
variable['y'] = [random.random() for _ in xrange(result_count)]

# optimize anything below this line

result = [0] * result_count

print 'Evaluating %d instances of the given expression:' % result_count
print expression

v = dict()
for index in xrange(result_count):
    for name in variable.keys():
        v[name] = variable[name][index]
    result[index] = eval(expression) # <-- option ONE
    #result[index] = math.sin(v['x']) * v['y'] # <-- option TWO

For a quick comparison option ONE takes 2.019 seconds on my machine, while option TWO takes only 0.218 seconds. Surely Python has a way of doing this without hard-coding the expression.

回答1:

You can also trick python:

expression = "math.sin(v['x']) * v['y']"
exp_as_func = eval('lambda: ' + expression)

And then use it like so:

exp_as_func()

Speed test:

In [17]: %timeit eval(expression)
10000 loops, best of 3: 25.8 us per loop

In [18]: %timeit exp_as_func()
1000000 loops, best of 3: 541 ns per loop

---

As a side note, if v is not a global, you can create the lambda like this:

exp_as_func = eval('lambda v: ' + expression)

and call it:

exp_as_func(my_v)

回答2:

You can avoid the overhead by compiling the expression in advance using compiler.compile():

In [1]: import math, compiler

In [2]: v = {'x': 2, 'y': 4}

In [3]: expression = "math.sin(v['x']) * v['y']"

In [4]: %timeit eval(expression)
10000 loops, best of 3: 19.5 us per loop

In [5]: compiled = compiler.compile(expression, '<string>', 'eval')

In [6]: %timeit eval(compiled)
1000000 loops, best of 3: 823 ns per loop

Just make sure you do the compiling only once (outside of the loop). As mentioned in comments, when using eval on user submitted strings make sure you are very careful about what you accept.

回答3:

I think you are optimising the wrong end. If you want to perform the same operation for a lot of numbers you should consider using numpy:

import numpy
import time
import math
import random

result_count = 100000
expression = "sin(x) * y"

namespace = dict(
    x=numpy.array(
        [random.random() for _ in xrange(result_count)]),
    y=numpy.array(
        [random.random() for _ in xrange(result_count)]),
    sin=numpy.sin,
)
print ('Evaluating %d instances '
       'of the given expression:') % result_count
print expression

start = time.time()
result = eval(expression, namespace)
numpy_time = time.time() - start
print "With numpy:", numpy_time


assert len(result) == result_count
assert all(math.sin(a) * b == c for a, b, c in
           zip(namespace["x"], namespace["y"], result))

To give you an idea about the possible gain I've added a variant using generic python and the lambda trick:

from math import sin
from itertools import izip

start = time.time()
f = eval("lambda: " + expression)
result = [f() for x, y in izip(namespace["x"], namespace["y"])]
generic_time = time.time() - start
print "Generic python:", generic_time
print "Ratio:", (generic_time / numpy_time)

Here are the results on my aging machine:

$ python speedup_eval.py 
Evaluating 100000 instances of the given expression:
sin(x) * y
With numpy: 0.006098985672
Generic python: 0.270224094391
Ratio: 44.3063992807

The speed-up is not as high as I expected, but still significant.