Python: How to update value of key value pair in n

Question

i am trying to make an inversed document index, therefore i need to know from all unique words in a collection in which doc they occur and how often.

i have used this answer in order two create a nested dictionary. The provided solution works fine, with one problem though.

First i open the file and make a list of unique words. These unique words i than want to compare with the original file. When there is a match, the frequency counter should be updated and its value be stored in the two dimensional array.

output should eventually look like this:

word1, {doc1 : freq}, {doc2 : freq} <br>
word2, {doc1 : freq}, {doc2 : freq}, {doc3:freq}
etc....

Problem is that i cannot update the dictionary variable. When trying to do so i get the error:

  File "scriptV3.py", line 45, in main
    freq = dictionary[keyword][filename] + 1
TypeError: unsupported operand type(s) for +: 'AutoVivification' and 'int'

I think i need to cast in some way the instance of AutoVivification to int....

How to go?

thanks in advance

my code:

#!/usr/bin/env python 
# encoding: utf-8

import sys
import os
import re
import glob
import string
import sets

class AutoVivification(dict):
    """Implementation of perl's autovivification feature."""
    def __getitem__(self, item):
        try:
            return dict.__getitem__(self, item)
        except KeyError:
            value = self[item] = type(self)()
            return value

def main():
    pad = 'temp/'
    dictionary  = AutoVivification()
    docID = 0
    for files in glob.glob( os.path.join(pad, '*.html') ):  #for all files in specified folder:
        docID = docID + 1
        filename = "doc_"+str(docID)
        text = open(files, 'r').read()                      #returns content of file as string
        text = extract(text, '<pre>', '</pre>')             #call extract function to extract text from within <pre> tags
        text = text.lower()                                 #all words to lowercase
        exclude = set(string.punctuation)                   #sets list of all punctuation characters
        text = ''.join(char for char in text if char not in exclude) # use created exclude list to remove characters from files
        text = text.split()                                 #creates list (array) from string
        uniques = set(text)                                 #make list unique (is dat handig? we moeten nog tellen)

        for keyword in uniques:                             #For every unique word do   
            for word in text:                               #for every word in doc:
                if (word == keyword and dictionary[keyword][filename] is not None): #if there is an occurence of keyword increment counter 
                    freq = dictionary[keyword][filename]    #here we fail, cannot cast object instance to integer.
                    freq = dictionary[keyword][filename] + 1
                    print(keyword,dictionary[keyword])
                else:
                    dictionary[word][filename] = 1

#extract text between substring 1 and 2 
def extract(text, sub1, sub2): 
    return text.split(sub1, 1)[-1].split(sub2, 1)[0]    

if __name__ == '__main__':
    main()

Answer 1

I agree you should avoid the extra classes, and especially __getitem__. (Small conceptual errors can make __getitem__ or __getattr__ quite painful to debug.)

Python dict seems quite strong enough for what you are doing.

What about straightforward dict.setdefault

    for keyword in uniques:                             #For every unique word do   
        for word in text:                               #for every word in doc:
            if (word == keyword):
                dictionary.setdefault(keyword, {})
                dictionary[keyword].setdefault(filename, 0)
                dictionary[keyword][filename] += 1

Of course this would be where dictionary is just a dict, and not something from collections or a custom class of your own.

Then again, isn't this just:

        for word in text:                               #for every word in doc:
            dictionary.setdefault(word, {})
            dictionary[word].setdefault(filename, 0)
            dictionary[word][filename] += 1

No reason to isolate unique instances, since the dict forces unique keys anyway.

Answer 2

if (word == keyword and dictionary[keyword][filename] is not None): 

that is not a correct usage i guess, instead try this:

if (word == keyword and filename in dictionary[keyword]): 

Because, checking the value of a non-existing key raise KeyError. :so You must check if key exists in dictionary...

Answer 3

#!/usr/bin/env python
# encoding: utf-8
from os.path import join
from glob import glob as glob_
from collections import defaultdict, Counter
from string import punctuation

WORKDIR  = 'temp/'
FILETYPE = '*.html'
OUTF     = 'doc_{0}'.format

def extract(text, startTag='<pre>', endTag='</pre>'):
    """Extract text between start tag and end tag

    Start at first char following first occurrence of startTag
      If none, begin at start of text
    End at last char preceding first subsequent occurrence of endTag
      If none, end at end of text
    """
    return text.split(startTag, 1)[-1].split(endTag, 1)[0]    

def main():
    DocWords = defaultdict(dict)

    infnames = glob_(join(WORKDIR, FILETYPE))
    for docId,infname in enumerate(infnames, 1):
        outfname = OUTF(docId)
        with open(infname) as inf:
            text = inf.read().lower()
        words = extract(text).strip(punctuation).split()
        for wd,num in Counter(words).iteritems():
            DocWords[wd][outfname] = num

if __name__ == '__main__':
    main()