How do I get the current depth of the Python inter

Question

From the documentation:

sys.getrecursionlimit()

Return the current value of the recursion limit, the maximum depth of the Python interpreter stack. This limit prevents infinite recursion from causing an overflow of the C stack and crashing Python. It can be set by setrecursionlimit().

I am currently hitting the recursion limit when pickling an object. The object I am pickling only has a few levels of nesting, so I am a bit puzzled by what is happening.

I have been able to circumvent the issue with the following hack:

try:
    return pickle.dumps(x)
except:
    try:
        recursionlimit = getrecursionlimit()
        setrecursionlimit(2*recursionlimit)
        dumped = pickle.dumps(x)
        setrecursionlimit(recursionlimit)
        return dumped
    except:
        raise

Testing the above snippet on different contexts sometimes leads to success on the first try, and sometimes it leads to success on the second try. So far I have not been able to make it raise the exception.

To further debug my issue it would be helpful to have a way to obtain the current depth of the stack. That would allow me to verify if the entering stack depth is determining whether the snippet above will succeed on the first try or on the second.

Does the standard library provide a function to get the depth of the stack, or if not, how can I obtain it?

def get_stack_depth():
    # what goes here?

Answer 1

You can see the whole call stack from inspect.stack(), so currently taken depth would be len(inspect.stack(0)).

On the other hand, I guess you got the complete stack printed out when "maximum recursion depth exceeded" exception was raised. That stack trace should show you exactly what went wrong.

Answer 2

If speed is an issue, it's way faster to bypass inspect module.

testing depth: 50 (CPython 3.7.3)
stacksize4b()         | depth: 50   |    2.0 µs
stacksize4b(200)      | depth: 50   |    2.2 µs
stacksize3a()         | depth: 50   |    2.4 µs
stacksize2a()         | depth: 50   |    2.9 µs
stackdepth2()         | depth: 50   |    3.0 µs
stackdepth1()         | depth: 50   |    3.0 µs
stackdepth3()         | depth: 50   |    3.4 µs
stacksize1()          | depth: 50   |    7.4 µs  # deprecated
len(inspect.stack())  | depth: 50   |    1.9 ms

I shortened the name of my functions to stacksize() and for easier differentiation, I'm referring to @lunixbochs' functions as stackdepth().

---

Basic Algorithms:

That's probably the best compromise between code brevity, readability and speed for small stack sizes. For under ~10 frames, only stackdepth1() is slightly faster due to lower overhead.

from itertools import count

def stack_size2a(size=2):
    """Get stack size for caller's frame.
    """
    frame = sys._getframe(size)

    for size in count(size):
        frame = frame.f_back
        if not frame:
            return size

For achieving better timings for larger stack sizes, some more refined algorithms are possible. stacksize3a() is combining chained attribute lookup with a close range finish from stackdepth1() for a much more favorable slope in timings, starting to pay off for roughly > 70 frames in my benchmarks.

from itertools import count

def stack_size3a(size=2):
    """Get stack size for caller's frame.
    """
    frame = sys._getframe(size)
    try:
        for size in count(size, 8):
            frame = frame.f_back.f_back.f_back.f_back.\
                f_back.f_back.f_back.f_back
    except AttributeError:
        while frame:
            frame = frame.f_back
            size += 1
        return size - 1

Advanced Algorithms:

As @lunixbochs has brought up in an answer, sys._getframe() is basically stackdepth1() in C-code. While simpler algorithms always start their depth-search in Python from an existing frame at the top of stack, checking the stack downward for further existing frames, stacksize4b() allows starting the search from any level by its stack_hint-parameter and can search the stack down- or upward if needed.

Under the hood, calling sys._getframe() always means walking the stack from the top frame downward to a specified depth. Because the performance difference between Python and C is so huge, it can still pay off to call sys._getframe() multiple times if necessary to find a frame closer to the deepest one, before applying a basic close-range frame-by-frame search in Python with frame.f_back.

from itertools import count

def stack_size4b(size_hint=8):
    """Get stack size for caller's frame.
    """
    get_frame = sys._getframe
    frame = None
    try:
        while True:
            frame = get_frame(size_hint)
            size_hint *= 2
    except ValueError:
        if frame:
            size_hint //= 2
        else:
            while not frame:
                size_hint = max(2, size_hint // 2)
                try:
                    frame = get_frame(size_hint)
                except ValueError:
                    continue

    for size in count(size_hint):
        frame = frame.f_back
        if not frame:
            return size

The usage-idea of stacksize4b() is to place the size-hint at the lower bound of your expected stack depth for a jump start, while still being able to cope with every drastic and short-lived change in stack-depth.

The benchmark shows stacksize4b() with default size_hint=8 and adjusted size_hint=200. For the benchmark all stack depths in the range 3-3000 have been tested to show the characteristic saw pattern in timings for stacksize4b().