Stop at exception in my, not library code

Question

I'm developing an app using a Python library urllib and it is sometimes rising exceptions due to not being able to access an URL.

However, the exception is raised almost 6 levels into the standard library stack:

/home/user/Workspace/application/main.py in call(path)
     11                                  headers={'content-type': 'application/json'},
     12                                  data=b'')
---> 13     resp = urllib.request.urlopen(req)          ####### THIS IS MY CODE
     14     return json.loads(resp.read().decode('utf-8'))

/usr/lib/python3.4/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    159     else:
    160         opener = _opener
--> 161     return opener.open(url, data, timeout)
    162 
    163 def install_opener(opener):

/usr/lib/python3.4/urllib/request.py in open(self, fullurl, data, timeout)
    461             req = meth(req)
    462 
--> 463         response = self._open(req, data)
    464 
    465         # post-process response

/usr/lib/python3.4/urllib/request.py in _open(self, req, data)
    479         protocol = req.type
    480         result = self._call_chain(self.handle_open, protocol, protocol +
--> 481                                   '_open', req)
    482         if result:
    483             return result

/usr/lib/python3.4/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    439         for handler in handlers:
    440             func = getattr(handler, meth_name)
--> 441             result = func(*args)
    442             if result is not None:
    443                 return result

/usr/lib/python3.4/urllib/request.py in http_open(self, req)
   1208 
   1209     def http_open(self, req):
-> 1210         return self.do_open(http.client.HTTPConnection, req)
   1211 
   1212     http_request = AbstractHTTPHandler.do_request_

/usr/lib/python3.4/urllib/request.py in do_open(self, http_class, req, **http_conn_args)
   1182                 h.request(req.get_method(), req.selector, req.data, headers)
   1183             except OSError as err: # timeout error
-> 1184                 raise URLError(err)
   1185             r = h.getresponse()
   1186         except:

URLError: <urlopen error [Errno 111] Connection refused>

I usually run the code in ipython3 with the %pdb magic turned on so in case there is an exception I can inspect it immediately. However for this I have to go down the stack 6 levels to get to my code.

Is it achievable that my app crashes pointing to my code directly?

Answer 1

urllib can raise a lot of exceptions.

You need to put a try block around the call into urllib and figure how to handle the exceptions for example:

try:
    resp = urllib.request.urlopen(req)   

except URLError as e:
    # analyse e to figure out the detail
    ...

Certainly under python2's urllib lots of other exceptions are thrown. I'm not sure about python3's urllib.

Answer 2

It can be done with some hacking. These docs show how you can turn on post-mortem debugging with the following code in the entry point:

import sys
from IPython.core import ultratb
sys.excepthook = ultratb.FormattedTB(mode='Verbose',
                                     color_scheme='Linux', call_pdb=1)

Stepping through this hook after an exception is raised shows that we need to tinker with the debugger method. Unfortunately I can see no better way to do this other than to copy the entire method and modify it where needed (I tried modifying self.tb but traceback objects are read only and can't be used with copy.deepcopy). Here's a demo:

import json
import sys
from IPython.core import debugger, ultratb
from IPython.core.display_trap import DisplayTrap

class CustomTB(ultratb.FormattedTB):
    def debugger(self, force=False):
        if force or self.call_pdb:
            if self.pdb is None:
                self.pdb = debugger.Pdb(
                    self.color_scheme_table.active_scheme_name)
            # the system displayhook may have changed, restore the original
            # for pdb
            display_trap = DisplayTrap(hook=sys.__displayhook__)
            with display_trap:
                self.pdb.reset()
                # Find the right frame so we don't pop up inside ipython itself
                if hasattr(self, 'tb') and self.tb is not None:
                    etb = self.tb
                else:
                    etb = self.tb = sys.last_traceback

                # only modification is here -----+
                #                                |
                #                                V
                while self.tb is not None and '/lib/python3' not in self.tb.tb_next.tb_frame.f_code.co_filename:
                    self.tb = self.tb.tb_next
                if etb and etb.tb_next:
                    etb = etb.tb_next
                self.pdb.botframe = etb.tb_frame
                self.pdb.interaction(self.tb.tb_frame, self.tb)

        if hasattr(self, 'tb'):
            del self.tb

sys.excepthook = CustomTB(mode='Verbose',
                          color_scheme='Linux', call_pdb=1)

def foo():
    bar()

def bar():
    json.dumps(json)

foo()

As you can see it stops searching through the traceback when it's about to reach library code. Here's the result:

TypeErrorTraceback (most recent call last)
/Users/alexhall/Dropbox/python/sandbox3/sandbox.py in <module>()
     40     json.dumps(json)
     41 
---> 42 foo()
        global foo = <function foo at 0x1031358c8>

/Users/alexhall/Dropbox/python/sandbox3/sandbox.py in foo()
     35 
     36 def foo():
---> 37     bar()
        global bar = <function bar at 0x103135950>
     38 
     39 def bar():

/Users/alexhall/Dropbox/python/sandbox3/sandbox.py in bar()
     38 
     39 def bar():
---> 40     json.dumps(json)
        global json.dumps = <function dumps at 0x10168b268>
        global json = <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>
     41 
     42 foo()

/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py in dumps(obj=<module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw={})
    228         cls is None and indent is None and separators is None and
    229         default is None and not sort_keys and not kw):
--> 230         return _default_encoder.encode(obj)
        global _default_encoder.encode = <bound method JSONEncoder.encode of <json.encoder.JSONEncoder object at 0x10166e8d0>>
        obj = <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>
    231     if cls is None:
    232         cls = JSONEncoder

/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/encoder.py in encode(self=<json.encoder.JSONEncoder object>, o=<module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>)
    197         # exceptions aren't as detailed.  The list call should be roughly
    198         # equivalent to the PySequence_Fast that ''.join() would do.
--> 199         chunks = self.iterencode(o, _one_shot=True)
        chunks = undefined
        self.iterencode = <bound method JSONEncoder.iterencode of <json.encoder.JSONEncoder object at 0x10166e8d0>>
        o = <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>
        global _one_shot = undefined
    200         if not isinstance(chunks, (list, tuple)):
    201             chunks = list(chunks)

/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/encoder.py in iterencode(self=<json.encoder.JSONEncoder object>, o=<module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>, _one_shot=True)
    255                 self.key_separator, self.item_separator, self.sort_keys,
    256                 self.skipkeys, _one_shot)
--> 257         return _iterencode(o, 0)
        _iterencode = <_json.Encoder object at 0x1031296d8>
        o = <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>
    258 
    259 def _make_iterencode(markers, _default, _encoder, _indent, _floatstr,

/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/encoder.py in default(self=<json.encoder.JSONEncoder object>, o=<module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>)
    178 
    179         ""
--> 180         raise TypeError(repr(o) + " is not JSON serializable")
        global TypeError = undefined
        global repr = undefined
        o = <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'>
    181 
    182     def encode(self, o):

TypeError: <module 'json' from '/Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py'> is not JSON serializable
> /Users/alexhall/Dropbox/python/sandbox3/sandbox.py(40)bar()
     38 
     39 def bar():
---> 40     json.dumps(json)
     41 
     42 foo()

ipdb> down
> /Users/alexhall/.pyenv/versions/3.5.0/lib/python3.5/json/__init__.py(230)dumps()
    228         cls is None and indent is None and separators is None and
    229         default is None and not sort_keys and not kw):
--> 230         return _default_encoder.encode(obj)
    231     if cls is None:
    232         cls = JSONEncoder

ipdb> 

Basically the full traceback is still printed out but ipdb starts at your own code. If you enter the down command you find yourself in a library frame.

Answer 3

I think the answer is no.

pdb stops at the exception and shows you the stack.

Why would it be useful to hide the real source of the exception?

If it worked as you seem to be requesting and hides the 6 layers of stack how would you work out what to fix?

If this is still not on topic please add to your question.

Answer 4

I would go with modifying the code:

try:
    resp = urllib.request.urlopen(req)

except Exception as e:
    raise RuntimeError(e)

That way:

• %pdb moves you to your code,
• the original exception is preserved as argument of the "secondary" exception.

You may also monkeypatch urllib.request.urlopen() function:

class MonkeyPatchUrllib(object):
    def __enter__(self):
        self.__urlopen = urllib.request.urlopen
        urllib.request.urlopen = self
    def __exit__(self, exception_type, exception_value, traceback):
        urllib.request.urlopen = self.__urlopen
    def __call__(self, *args, **kwargs):
        try:                                  
            return self.__urlopen(*args, **kwargs)
        except Exception as e:
            raise RuntimeError(e)

Any time you have an exception raised in urlibopen() call within the context manager scope:

with MonkeyPatchUrllib():
    #your code here

%pdb will move you only 1 level away from your code.

[EDIT]

With sys.exc_info() it is possible to preserve a more verbose context of the original exception (like its traceback).

Answer 5

pdb has only incremental frame positioning (moving up or down the list of frames).

To get the feature you want, you can try trepan (github repository). It has an IPython extension here. You then use the command frame -1 once the exception shows up:

Frame (absolute frame positioning)

frame [thread-Name*|*thread-number] [frame-number]

Change the current frame to frame frame-number if specified, or the current frame, 0, if no frame number specified.

If a thread name or thread number is given, change the current frame to a frame in that thread. Dot (.) can be used to indicate the name of the current frame the debugger is stopped in.

A negative number indicates the position from the other or least-recently-entered end. So frame -1 moves to the oldest frame, and frame 0 moves to the newest frame. Any variable or expression that evaluates to a number can be used as a position, however due to parsing limitations, the position expression has to be seen as a single blank-delimited parameter. That is, the expression (5*3)-1 is okay while (5 * 3) - 1) isn’t.

Once you are in the desired frame, you can use edit to modify your code.

You may find the command backtrace useful too as it gives a stack trace with the less recent call at the bottom.

trepan depends on uncompyle6 available here.

pydb provides a similar feature but was unfortunately not ported to Python3.

Otherwise, you may decide to be patient and wait for improvements. In IPython/core/debugger.py:

"""
Pdb debugger class.

Modified from the standard pdb.Pdb class to avoid including readline, so that
the command line completion of other programs which include this isn't damaged.

In the future, this class will be expanded with improvements over the standard pdb.
[...]
"""