boost::python passing reference of python::list

I'd really like to know if there is a possibility to pass a reference of a python list to a boost::python c++ dll. What I want to achieve is that I have a list in python which can be read in c++ at any time. Let's say you'd have a variable in C++ that holds the reference to the list.

Is there any way to do this? So far I only found the ctypes in python where I can make references of primitive c types, which in this case, doesn't help.

I am happy for any suggestions or workarounds (a simple example would be great)

Greetings Chris

In short, Boost.Python maintains Python argument passing semantics with its TypeWrappers. Thus, when passing a list in Python to an exposed C++ function, a reference can be created and maintained by accepting the Python list as a boost::python::list object.

The detailed answer actually has a bit more depth to it. Before delving into it, let me expand upon some semantics to avoid confusion. With Python's garbage collection and pass-by-object semantics, the general rule of thumb is to treat the Boost.Python TypeWrappers as smart pointers.

If the function accepts the list as a boost::python::list object, then C++ now has a reference to the Python object. The Python list's lifetime will be extended to be at least as long as the booot::python::list object.
If the Python list is converted to a different type, such as std::vector, then C++ has constructed a copy to the Python list. This copy has no association with the original list.

Here is a simple example of a C++ module that can be passed a Python list, maintain a handle to it, and print its contents:

#include <iostream> // std::cout
#include <utility>  // std::make_pair
#include <boost/foreach.hpp>
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>

boost::python::list list;

/// @brief Store handle to the list.
///
/// @param pylist Python list for which a handle will be maintained.
void set(const boost::python::list& pylist)
{
  // As the boost::python::list object is smart-pointer like, this
  // creates a reference to the python list, rather than creating a 
  // copy of the python list.
  list = pylist;
}

// Iterate over the current list, printing all ints.
void display()
{
  std::cout << "in display" << std::endl;
  typedef boost::python::stl_input_iterator<int> iterator_type;
  BOOST_FOREACH(const iterator_type::value_type& data, 
                std::make_pair(iterator_type(list), // begin
                               iterator_type()))    // end
  {
    std::cout << data << std::endl;
  }
}

BOOST_PYTHON_MODULE(example) {
  namespace python = boost::python;
  python::def("set",     &set);
  python::def("display", &display);
}

And its usage:

>>> import example
>>>
>>> x = range(2)
>>> x
[0, 1]
>>> example.set(x)
>>> example.display()
in display
0
1
>>> x[:] = range(7, 10)
>>> example.display()
in display
7
8
9

One complexity introduced in the question is the desire to read the Python list in C++ at any time. In its most complicated case, any time can occur at any point in time, from within a C++ thread.

Lets start with the basics: Python's Global Interpreter Lock (GIL). In short, the GIL is a mutex around the interpreter. If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. Sometimes the reference counting is not very transparent, consider:

typedef boost::python::stl_input_iterator<int> iterator_type;
iterator_type iterator(list);

Although boost::python::stl_input_iterator accepts list as a constant reference, it creates a reference to the Python list from within the constructor.

In the previous example, as there were no C++ threads, all actions occurred while the GIL had been acquired. However, if display() could be invoked at any time from within C++, then some setup needs to occur.

First, the module needs to have Python initialize the GIL for threading.

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.
  ...
}

For convenience, lets create a simple class to help manage the GIL:

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

To show interactions with a C++ thread, lets add functionality to the module that will allow the application to schedule a delay for when the list's contents to be displayed.

/// @brief Entry point for delayed display thread.
///
/// @param Delay in seconds.
void display_in_main(unsigned int seconds)
{
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));
  gil_lock lock; // Acquire GIL.
  display();     // Can safely modify python objects.
  // GIL released when lock goes out of scope.
}

/// @brief Schedule the list to be displayed.
///
/// @param Delay in seconds.
void display_in(unsigned int seconds)
{
  // Start detached thread.
  boost::thread(&display_in_main, seconds).detach();
}

Here is the complete example:

#include <iostream> // std::cout
#include <utility>  // std::make_pair
#include <boost/foreach.hpp>
#include <boost/python.hpp>
#include <boost/python/stl_iterator.hpp>
#include <boost/thread.hpp>

boost::python::list list;

/// @brief Store handle to the list.
///
/// @param pylist Python list for which a handle will be maintained.
void set(const boost::python::list& pylist)
{
  list = pylist;
}

// Iterate over the current list, printing all ints.
void display()
{
  std::cout << "in display" << std::endl;
  typedef boost::python::stl_input_iterator<int> iterator_type;
  BOOST_FOREACH(const iterator_type::value_type& data, 
                std::make_pair(iterator_type(list), // begin
                               iterator_type()))    // end
  {
    std::cout << data << std::endl;
  }
}

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
}; 

/// @brief Entry point for delayed display thread.
///
/// @param Delay in seconds.
void display_in_main(unsigned int seconds)
{
  boost::this_thread::sleep_for(boost::chrono::seconds(seconds));
  gil_lock lock; // Acquire GIL.
  display();     // Can safely modify python objects.
  // GIL released when lock goes out of scope.
}

/// @brief Schedule the list to be displayed.
///
/// @param Delay in seconds.
void display_in(unsigned int seconds)
{
  // Start detached thread.
  boost::thread(&display_in_main, seconds).detach();
}

BOOST_PYTHON_MODULE(example) {
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::def("set",        &set);
  python::def("display",    &display);
  python::def("display_in", &display_in);
}

And its usage:

>>> import example
>>> from time import sleep
>>> 
>>> x = range(2)
>>> example.set(x)
>>> example.display()
in display
0
1
>>> example.display_in(3)
>>> x[:] = range(7, 10)
>>> print "sleeping"
sleeping
>>> sleep(6)
in display
7
8
9

While the Python console blocked for 6 seconds in the sleep(6) call, the C++ thread acquired the GIL, displayed the contents of list x, and released the GIL.