Ipython/Jupyter - can we program a “run all cell a

Question

I am using jupyter 4 with python and I would need my script to do a ' relaunch all the cells above' when a crash occurs. Is it possible ?

2nd question : If I need to relaunch all some cells, can I ask python to execute them according to some cell-id ? I could then create a list of the cells id which have to be re-executed when catching an exception...

Thx.

Answer 1

---

2nd question: NO

---

...at least not very reliably, because any ID of a cell would change if you insert or remove any other cell.

According to Execute specific cells through widgets and conditions on github:

We don't have the Ids of of cell in order to handle them programatically.

And further down on the same post:

There are some APIs which can run cells identified by numbers, but unfortunately the numbers change if you insert or delete a cell somewhere above.

---

1st question: YES

---

...but it's not 100% certain that it will solve your error handling needs as per the details in your question. But we'll get to that in a bit. Because the good news is that the answer to the question as it stands in the title

Ipython/Jupyter - can we program a “run all cell above”?

is YES WE CAN!

The hard (maybe even impossible) part of this question is to implement it as a robust error handling method. If you're only interested in that, skip to the section The hard part at the end of my answer. For now, let's go on with the easy part that is to programmatically run the menu option Cell > Run All (as described in the answer by Nic Cottrell). You have two options:

Option 1 - Run all cells above by executing a cell:

If you insert the following snippet in a cell and run it, all cells above will be executed:

from IPython.display import Javascript
display(Javascript('IPython.notebook.execute_cells_above()'))

Option 2 - Run all cells above by clicking a button:

If you insert the following snippet in a cell and run it, all cells above will be executed when you click the appearing button:

Snippet:

from IPython.core.display import display, HTML
HTML('''<script> </script> <form action="javascript:IPython.notebook.execute_cells_above()"><input type="submit" id="toggleButton" value="Run all"></form>''')

Output:

---

THE HARD PART

So, how can we set this up to handle an error when a crash occurs? I'm not an expert on this, but I think I've been able to make a setup that will work for you. But it will most likely depend on the type of error in question and the rest of your work flow.

The following example builds on two different error messages. The first is a NameError that occurs when you try to assign a value to a variable that does not exist. And this will be useful since re-running some cells after an error will need an iterator that resets only when the notebook is restarted completely, and not when a cell is re-run as part of an error handling method. The name error will only occur when the kernel is restarted upon a fresh restart of your notebook. As part of the error handling, the value 0 is assigned to x1. When the cell is only re-run x1 will increase by 1.

The second error will serve as a proxy for your error, and is an AssignmentError that occurs each time you try to delete an element from a list that does not exist. And this leads us to the real challenge, since if your error handler re-runs all cells above every time the error is triggered, you'll quickly end up in a bad loop. But we'll handle that with a counter that exits the looping execution of cells after a few runs.

It's also a bit problematic that there does not seem to exist a functionality to rerun your existing cell, or the cell from where the run cells above functionality is initialized. But we'll handle that with another suggestion from the same github post as earlier:

Doing the following helps me to execute the cell right below the code cell. You can also change the values to get cells in other parts of the notebook. display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+1, IPython.notebook.get_selected_index()+2)'))

Notebook with suggested work-flow:

Insert the four following snippets below in four cells. Click the menu option Cell > Run all once, and we're good to go!

Snippet 1 - Imports and setup

import sys
import os
from IPython.core.display import display, HTML
from IPython.display import Javascript
from random import randint

# Trigger to randomly raise en error in the next cell
ErrorTrigger = randint(0, 9)

# Assignment of variables at first run of the Norebook
try: x1
except NameError: x1 = None
if x1 is None:
    %qtconsole # opens a qtconsole (for variable inspection and debugging)
    x1 = 0 # counter for NameError
    x2 = 0 # counter for assignment error (used in cells below)
    mr = 0 # counter for manual relaunch by button 

    ErrorTriggers=[] # container for ErroTriggers    
    print('NameErrors = ', x1)
else:
    x1 = x1 + 1
    ErrorTriggers.append(ErrorTrigger)
#print('Executions:', x1, '||', 'Triggers:', ErrorTriggers)

Snippet 2 - Proxy for your error

# PROXY ERROR => INSERT YOUR CODE FROM HERE ################################################################
list1 = [1,2,3,4]

# 80 % chance of raising an error trying to delete an element that does not exist in the list
if ErrorTrigger > 2:
    elemDelete = 8 # error
else:
    elemDelete = 0 # not error
try:
    del list1[elemDelete]
    print('Executions:', x1, '||', 'Triggers:', ErrorTriggers)
    print('Routine success on attempt', x2 + 1)
    print('Error mesg: None')
    ErrorTriggers=[]
    x2 = 0 # reset error counter

# TO HERE #################################################################################################
except Exception:

    x2 = x2 + 1
    # Will end error handler after 5 attempts
    if x2 < 3:
        # As long as we're UNDER the attempt limit, the next cell executed by:
        display(Javascript('IPython.notebook.execute_cell_range(IPython.notebook.get_selected_index()+1,'+
                           ' IPython.notebook.get_selected_index()+2)'))
    else:
        # If we're OVER the attempt limit, it all ends here. The next cell is NOT run.
        # And NEITHER is the last cell with the button to relaunch the whole thing.

        print('Executions:', x1, '||', 'Triggers:', ErrorTriggers)
        print('Routine aborted after attempt', x2)
        print('Error msg:', sys.exc_info()[1]) # Returns a message describing the error
        # reset variables 
        ErrorTriggers = []
        x2 = 0

Snippet 3 - Cell to rerun all cells above as error handler

display(Javascript('IPython.notebook.execute_cells_above()'))

Snippet 4 - Cell to rerun the whole thing with en error probability of 20%

HTML('''<script> </script> <form action="javascript:IPython.notebook.execute_cells_above()"><input type="submit" id="toggleButton" value="Run again!"></form>''')

Screenshot after a few test runs:

I'll gladly add more details if the comments in the snippets are unclear. But if you run the notebook a few times by clicking Run Again! and at the same time have a look at the output of cell 3, you'll quickly grasp how the whole thing is put together:

Answer 2

I'm running Notebook server 5.4.0 and I have an option Cell > Run All Above which seems to do exactly this.