qPython - Type conversion of kdb response data

Question

When I run a q query using qPython, I am able to return the data in a pandas data frame. What I am struggling with are the types of the "string" columns, i.e. columns that are presented as simple or mixed (character) lists in q. Their dtype is object and the values are represented in the form b'ab34knadke'. What I would like to have, however, is just the "ab34knadke"-part as a string.

I have looked at the docs for qPython but I am struggling to fully get the pandas and reader components.

Any thoughts are much appreciated!

Answer 1

In short you can fix the stringed columns using

data['stringcolumn']=data['stringcolumn'].str.decode('utf-8')

qpython maps the kdb byte arrays to python byte arrays in machine readable form. Consequently these need to be made into human readable strings using the above method.

Note: this solution solution is only valid if the strings/byte arrays coming in are in UTF-8 but this is a pretty safe bet.

You can read a little more about this here:https://www.geeksforgeeks.org/byte-objects-vs-string-python/