Parsing dictionary-like URL parameters in Python

Question

I'm working on implementing server-side filtering to serve KendoUI's Grid component, using Python.

The problem I'm facing is that the AJAX call that it generates by default seems to be incompatible with both Flask's built-in URL parser and Python's urlparse module.

Here's a contrived sample of the type of query string I'm having trouble with: a=b&c=d&foo[bar]=baz&foo[baz]=qis&foo[qis]=bar

Here's the result I'm going for:

{
    'a': 'b',
    'c': 'd',
    'foo': {
        'bar': 'baz',
        'baz': 'qis',
        'qis': bar'
    }
}

Unfortunately, here's the request.args you get from this, if passed to a Flask endpoint:

{
    'a': 'b',
    'c': 'd',
    'foo[bar]': 'baz'
    'foo[baz]': 'qis'
    'foo[qis]': 'bar'
}

Worse yet, in practice, the structure can be several layers deep. A basic call where you're filtering the column foo to only rows where the value is equal to 'bar' will produce the following:

{
    'filter[logic]': 'and',
    'filter[filters][0][value]': 'bar',
    'filter[filters][0][field]': 'foo',
    'filter[filters][0][operator]': 'eq'
}

I checked the RFC, and it requires that the query string contain only "non-hierarchical" data. While I believe it's referring to the object the URI represents, there is no provision for this type of data structure in the specification that I can find.

I begin to write a function that would take a dictionary of params and return the nested construct they represented, but I soon realized that it was nuanced problem, and that surely someone out there has had this trouble before.

Is anyone aware of either a module that will parse these parameters in the way I'm wanting, or an elegant way to parse them that I've perhaps overlooked?

Answer 1

Some time ago I found this project: https://github.com/bernii/querystring-parser

It is specifically aimed at doing what you wanted.

However, outside PHP world, GET (and POST) parameters behave differently. Like they usually are implemented using multi-value dictiomaries. So the better idea may be to fit that or find a way compatible with both worlds.

Alternatively, you can really use JSON-seralized data in the request body (POST) and just treat the accessed resource as controller (the resource that does something, in this case searching for something, after you pass some data to it).

Answer 2

You could send body in GET requests too. If all you have to send is some hierarchal data, might just json.dumps(data) at client and json.loads(data) on the server.

You can refer to such practices in httplib's documentation here: http://docs.python.org/2/library/httplib.html#httplib.HTTPConnection.request

Answer 3

I just wrote a little function to do this:

from collections import defaultdict
import re
params = {
    'a': 'b',
    'c': 'd',
    'foo[bar]': 'element1',
    'foo[baz]': 'element2',
    'foo[qis]': 'element3',
    'foo[borfarglan][bofgl]': 'element4',
    'foo[borfarglan][bafgl]': 'element5',
}

def split(string, brackets_on_first_result = False):
    matches = re.split("[\[\]]+", string)
    matches.remove('')
    return matches

def mr_parse(params):
    results = {}
    for key in params:
        if '[' in key:
            key_list = split(key)
            d = results
            for partial_key in key_list[:-1]:
                if partial_key not in d:
                    d[partial_key] = dict()
                d = d[partial_key]
            d[key_list[-1]] = params[key]
        else:
            results[key] = params[key]
    return results
print mr_parse(params)

This should work to any nest level.