Pyparsing infixNotation into a parse tree: Only on

Question

My eventual goal is to apply sql-where-clause-style queries to filter pandas dataframes. Some searching led me to pyparsing's infixNotation method.

I found an example of infix notation here: http://nullege.com/codes/show/src%40p%40y%40pyparsing-2.0.2%40examples%40simpleBool.py/15/pyparsing.infixNotation/python#

but that actually processes the notation, and I need to use the tree multiple times on different data. Looking for help with parse trees I found this: http://pyparsing.wikispaces.com/file/view/fourFn.py/30154950/fourFn.py

and I took the stack implementation from there. However, it isn't behaving as I expect so I am hoping someone can tell me what I am doing wrong.

Here is the simplest case I could come up with:

from pyparsing import Word, alphas, infixNotation, opAssoc

exprStack=[]
def pushFirst( strg, loc, toks ):
    exprStack.append( toks[0] )

def addAnd():
    exprStack.append("and")

varname = Word(alphas).setParseAction( pushFirst )

operators=[("and", 2, opAssoc.LEFT,addAnd)]

whereExpr = infixNotation(varname,operators)

exprStack=[]
teststring = "cheese and crackers and wine"
parsed=whereExpr.parseString(teststring)
for p in exprStack:
    print p

What I get from this code is:

cheese
crackers
wine
and

What I expect to get, from my understanding of how the infix notation method should work, is:

cheese
crackers
wine
and
and

I also tried running it with "cheese and crackers and wine and whine" but I still only got one "and" in my list.

What am I misunderstanding in using the infixNotation method?

Thank you

Answer 1

Start by decorating your parse action with pyparsing's traceParseAction diagnostic deccorator.

@traceParseAction
def addAnd():
    exprStack.append("and")

traceParseAction will show the matched source line, the starting location of the matched tokens, and tokens passed to the parse action, and the value that is returned by the parse action:

>>entering addAnd(line: 'cheese and crackers and wine', 0, 
                 ([(['cheese', 'and', 'crackers', 'and', 'wine'], {})], {}))
<<leaving addAnd (ret: None)

The tokens are a little confusing-looking, since what you are getting is a pyparsing ParseResults object, which has both list and dict semantics, so the Python repr of the object first shows its list contents, and then its named contents. What looks like a tuple with a list and a dict is really the ParseResults, and in this case, it is a ParseResults whose first element is another ParseResults, and this nested object is the one containing your list of matched tokens.

This is a little easier to see if you add a tokens argument to your parse action, and then print out tokens.dump():

def addAnd(tokens):
    print(tokens.dump())
    exprStack.append("and")

And you'll get the more readable:

[['cheese', 'and', 'crackers', 'and', 'wine']]
[0]:
  ['cheese', 'and', 'crackers', 'and', 'wine']

You can see that the matched tokens contains not just 'and', but all of the and-ed terms together, so you will need to push as many 'and's onto your exprStack as there are in the matched tokens.

def addAnd(tokens):
    exprStack.extend(tokens[0][1::2])

With this change, you should now see this as your returned stack:

cheese
crackers
wine
and
and