I've accidentialy got a threaded application due to the use of web.py
and pyinotify
packages :)
The application keeps sorta state in JSON files, which it may change and which can be changed by anyone else. Changed JSON is reloaded back into a dict
by inotify
watcher thread.
The application runs a dozen of threads in quite simple conditions.
Well, finally I observe
RuntimeError: dictionary changed size during iteration.
I realized, I need a lock around each and every dictionary operation. On this (these actually) dictionary, I mean.
I tried to work it around in quite different ways (yes, digging stackoverflow mostly).
I faced deadlocks in case of non-reentrant locks (that looks obvious to me now).
I ended up with this approach:
DictLock = threading.RLock # use reentrant locks here!
class ThreadSafeDict(dict):
'''It must avoid things like
RuntimeError: dictionary changed size during iteration.
Inspired by https://stackoverflow.com/questions/13456735
and https://stackoverflow.com/questions/3278077
'''
def __init__(self, *av, **kw):
self._lock = kw.pop('_lock') if '_lock' in kw else DictLock()
super(ThreadSafeDict, self).__init__(*av, **kw)
self._thread = None
# do NO LOCKS here!
def replace(self, dictionary):
super(ThreadSafeDict, self).clear()
super(ThreadSafeDict, self).update(dictionary)
return self
def load_json(self, filename):
with open(filename) as fp:
super(ThreadSafeDict, self).clear()
super(ThreadSafeDict, self).update(json.load(fp))
return self
def save_json(self, filename, indent=2):
with open(filename, 'w') as fp:
json.dump(self, fp, indent=indent)
return self
# all locking is being performed below
def __getattribute__(self, name):
if name == '_lock':
return super(ThreadSafeDict, self).__getattribute__(name)
with self._lock:
self._thread = threading.current_thread()
item = super(ThreadSafeDict, self).__getattribute__(name)
if not callable(item):
log.debug('[%r]=>%r', name, item)
return item # return property value as is
def wrapper(*a, **k):
with self._lock:
return item(*a, **k)
log.debug('%r()=>{%r}', name, item.__name__)
return wrapper # return locked calls for methods
#end class ThreadSafeDict
The problem looks gone...
The questions is: Did I do it right? (I hope, this doesn't violate the "We prefer questions that can be answered, not just discussed." rule: it can be answered, anyway).