Boost.Python and Boost.Signals2: Segmentation faul

Question

I have a problem with integrating boost.signals2 in my existing C++ lib which I have exposed with boost.python.

I have a class that is exposed to python with a std::shared_ptr. This class should be able to raise some signals on certain events. Therefore I have exposed a connect_slot function which takes a boost::python::object as argument. If I raise a signal directly after connecting a slot, everything works fine, but if the class raises the signals lateron, I receive segmentation faults.

I think this may have something to do with threading in the c++ lib (It is also using boos::asio etc.)

Here are some code snippets:

MyClass.h:

public:
    typedef boost::signals2::signal<void (std::shared_ptr<int>)> signal_my_sig;
    void connect_slot(boost::python::object const & slot);

private:
    signal_my_sig    m_sig;

MyClass.cpp:

void MyClass::connect_slot(boost::python::object const & slot) { 
    std::cout << "register shd" << std::endl;
    m_sig.connect(slot);

    m_sig(12345); // this works
}


void MyClass::some_later_event() {
    m_sig(654321); // this does not work

}

I call the MyClass::connect_slot function in python with a custom python function like this:

def testfunc(some_int):
    print("slot called")

m = myext.MyClass()
m.connect_slot(testfunc)

The backtrace (using gdb) of the segmentation fault that is raised in MyClass::some_later_event looks like this:

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff3c37700 (LWP 20634)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff3c37700 (LWP 20634)]
0x00000000004f7480 in PyObject_Call ()
(gdb) 
(gdb) backtrace
#0  0x00000000004f7480 in PyObject_Call ()
#1  0x00000000004f7aa6 in PyEval_CallObjectWithKeywords ()
#2  0x000000000049bd84 in PyEval_CallFunction ()
#3  0x00007ffff5375d9f in boost::python::call<boost::python::api::object, int>
(callable=0x7ffff7ed4578, a0=@0x7ffff3c35b34: 5)
at /usr/local/boost_1_55_0/boost/python/call.hpp:66
#4  0x00007ffff5374b81 in boost::python::api::object_operators<boost::python::api::object>::operator()<int> (this=0x9e3bf0, a0=@0x7ffff3c35b34: 5)
at /usr/local/boost_1_55_0/boost/python/object_call.hpp:19
#5  0x00007ffff5373658 in boost::detail::function::void_function_obj_invoker1<boost::python::api::object, void, int>::invoke (function_obj_ptr=..., a0=5)
at /usr/local/boost_1_55_0/boost/function/function_template.hpp:153
#6  0x00007ffff5378a3c in boost::function1<void, int>::operator() (
this=0x9e3be8, a0=5)
at /usr/local/boost_1_55_0/boost/function/function_template.hpp:767
#7  0x00007ffff53781f9 in boost::signals2::detail::call_with_tuple_args<boost::signals2::detail::void_type>::m_invoke<boost::function<void (int)>, 0u, int&>(void*, boost::function<void (int)>&, boost::signals2::detail::unsigned_meta_array<0u>, std::tuple<int&>) const (this=0x7ffff3c35c7f, func=..., args=...)
at /usr/local/boost_1_55_0/boost/signals2/detail/variadic_slot_invoker.hpp:92

Any ideas?

Answer 1

If MyClass::some_later_event() is being invoked from a C++ thread that is not explicitly managing the Global Interpreter Lock (GIL), then that can result in undefined behavior.

---

Python and C++ threads.

Lets consider the case where C++ threads are interacting with Python. For example, a C++ thread can be set to invoke the MyClass's signal after a period of time via MyClass.event_in(seconds, value).

This example can become fairly involved, so lets start with the basics: Python's GIL. In short, the GIL is a mutex around the interpreter. If a thread is doing anything that affects reference counting of python managed object, then it needs to have acquired the GIL. In the GDB traceback, the Boost.Signals2 library was likely trying to invoke a Python object without the GIL, resulting in the crash. While managing the GIL is fairly straightforward, it can become complex rather quickly.

First, the module needs to have Python initialize the GIL for threading.

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.
  // ...
}

For convenience, lets create a simple class to help manage the GIL via scope:

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};

Lets identify when the C++ thread will need the GIL:

• boost::signals2::signal can make additional copies of connected objects, as is done when the signal is concurrently invoked.
• Invoking a Python objected connected through boost::signals2::signal. The callback will certainly affect python objects. For example, the self argument provided to __call__ method will increase and decrease an object's reference count.

The MyClass class.

Here is a basic mockup class based on the original code:

/// @brief Mockup class.
class MyClass
{
public:
  /// @brief Connect a slot to the signal.
  template <typename Slot>
  void connect_slot(const Slot& slot)
  {
    signal_.connect(slot);
  }

  /// @brief Send an event to the signal.
  void event(int value)
  {
    signal_(value);
  }

private:
  boost::signals2::signal<void(int)> signal_;
};

As a C++ thread may be invoking MyClass's signal, the lifetime of MyClass must be at least as long as the thread. A good candidate to accomplish this is by having Boost.Python manage MyClass with a boost::shared_ptr.

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass")
    .def("event", &MyClass::event)
    // ...
    ;
}

boost::signals2::signal interacting with python objects.

boost::signals2::signal may make copies when it is invoked. Additionally, there may be C++ slots connected to the signal, so it would be ideal to only lock the GIL when Python slots are being invoked. However, signal does not provide hooks to allow us to acquire the GIL before creating copies of slots or invoking the slot.

In order to avoid having signal create copies of boost::python::object slots, one can use a wrapper class that creates a copy of boost::python::object so that reference counts remain accurate, and manages the copy via shared_ptr. This allows signal to freely create copies of shared_ptr instead of copying boost::python::object without the GIL.

This GIL safety slot can be encapsulated in a helper class.

/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
///           * Caller must own GIL when constructing py_slot, as 
///             the python::object will be copy-constructed (increment
///             reference to the object)
///           * The newly constructed python::object will be managed
///             by a shared_ptr.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion.
///           * When py_slot is invoked (operator()), it will acquire
///             the GIL then delegate to the managed python::object.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(
        new boost::python::object(object),  // GIL locked, so copy.
        [](boost::python::object* object)   // Delete needs GIL.
        {
          gil_lock lock;
          delete object;
        }
      )
  {}

  // Use default copy-constructor and assignment-operator.
  py_slot(const py_slot&) = default;
  py_slot& operator=(const py_slot&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(args...); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

A helper function will be exposed to Python to help adapt the types.

/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
  MyClass& self,
  boost::python::object object)
{
  py_slot slot(object); // Adapt object to a py_slot for GIL management.

  // Using a lambda here allows for the args to be expanded automatically.
  // If bind was used, the placeholders would need to be explicitly added.
  self.connect_slot([slot](Args... args) mutable { slot(args...); });
}

And the updated binding expose the helper function:

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass")
  .def("connect_slot", &MyClass_connect_slot<int>)
  .def("event",        &MyClass::event)
  // ...
  ;

The thread itself.

The thread's functionality is fairly basic: it sleeps then invokes the signal. However, it is important to understand the context of the GIL.

/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // Sleep without the GIl.
  std::this_thread::sleep_for(std::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking or copying 
  // C++-specific slots connected to the signal.  Thus, it is the 
  // responsibility of python slots to manage the GIL via the 
  // py_slot wrapper class.
  self->event(args...);
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // The caller may or may not have the GIL.  Regardless, spawn off a 
  // thread that will sleep and then invoke an event on MyClass.  The
  // thread will not be joined so detach from it.  Additionally, as
  // shared_ptr is thread safe, copies of it can be made without the
  // GIL.
  std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
      .detach();
}

_{Note that MyClass_event_in_thread could be expressed as a lambda, but unpacking a template pack within a lambda does not work on some compilers.}

And the MyClass bindings are updated.

python::class_<MyClass, boost::shared_ptr<MyClass>,
               boost::noncopyable>("MyClass")
  .def("connect_slot", &MyClass_connect_slot<int>)
  .def("event",        &MyClass::event)
  .def("event_in",     &MyClass_event_in<int>)
  ;

---

The final solution looks like this:

#include <thread> // std::thread, std::chrono
#include <boost/python.hpp>
#include <boost/shared_ptr.hpp>
#include <boost/signals2/signal.hpp>

/// @brief Mockup class.
class MyClass
{
public:
  /// @brief Connect a slot to the signal.
  template <typename Slot>
  void connect_slot(const Slot& slot)
  {
    signal_.connect(slot);
  }

  /// @brief Send an event to the signal.
  void event(int value)
  {
    signal_(value);
  }

private:
  boost::signals2::signal<void(int)> signal_;
};

/// @brief RAII class used to lock and unlock the GIL.
class gil_lock
{
public:
  gil_lock()  { state_ = PyGILState_Ensure(); }
  ~gil_lock() { PyGILState_Release(state_);   }
private:
  PyGILState_STATE state_;
};    

/// @brief Helepr type that will manage the GIL for a python slot.
///
/// @detail GIL management:
///           * Caller must own GIL when constructing py_slot, as 
///             the python::object will be copy-constructed (increment
///             reference to the object)
///           * The newly constructed python::object will be managed
///             by a shared_ptr.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion.
///           * When py_slot is invoked (operator()), it will acquire
///             the GIL then delegate to the managed python::object.
struct py_slot
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_slot(const boost::python::object& object)
    : object_(
        new boost::python::object(object),  // GIL locked, so copy.
        [](boost::python::object* object)   // Delete needs GIL.
        {
          gil_lock lock;
          delete object;
        }
      )
  {}

  // Use default copy-constructor and assignment-operator.
  py_slot(const py_slot&) = default;
  py_slot& operator=(const py_slot&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    gil_lock lock;
    (*object_)(args...); 
  }

private:
  boost::shared_ptr<boost::python::object> object_;
};

/// @brief MyClass::connect_slot helper.
template <typename ...Args>
void MyClass_connect_slot(
  MyClass& self,
  boost::python::object object)
{
  py_slot slot(object); // Adapt object to a py_slot for GIL management.

  // Using a lambda here allows for the args to be expanded automatically.
  // If bind was used, the placeholders would need to be explicitly added.
  self.connect_slot([slot](Args... args) mutable { slot(args...); });
}

/// @brief Sleep then invoke an event on MyClass.
template <typename ...Args>
void MyClass_event_in_thread(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // Sleep without the GIL.
  std::this_thread::sleep_for(std::chrono::seconds(seconds));

  // We do not want to hold the GIL while invoking or copying 
  // C++-specific slots connected to the signal.  Thus, it is the 
  // responsibility of python slots to manage the GIL via the 
  // py_slot wrapper class.
  self->event(args...);
}

/// @brief Function that will be exposed to python that will create
///        a thread to call the signal.
template <typename ...Args>
void MyClass_event_in(
  boost::shared_ptr<MyClass> self,
  unsigned int seconds,
  Args... args)
{
  // The caller may or may not have the GIL.  Regardless, spawn off a 
  // thread that will sleep and then invoke an event on MyClass.  The
  // thread will not be joined so detach from it.  Additionally, as
  // shared_ptr is thread safe, copies of it can be made without the
  // GIL.
  // Note: MyClass_event_in_thread could be expressed as a lambda,
  //       but unpacking a template pack within a lambda does not work
  //       on some compilers.
  std::thread(&MyClass_event_in_thread<Args...>, self, seconds, args...)
      .detach();
}

BOOST_PYTHON_MODULE(example)
{
  PyEval_InitThreads(); // Initialize GIL to support non-python threads.

  namespace python = boost::python;
  python::class_<MyClass, boost::shared_ptr<MyClass>,
                 boost::noncopyable>("MyClass")
    .def("connect_slot", &MyClass_connect_slot<int>)
    .def("event",        &MyClass::event)
    .def("event_in",     &MyClass_event_in<int>)
    ;
}

And a testing script:

from time import sleep
import example

def spam1(x):
  print "spam1: ", x

def spam2(x):
  print "spam2: ", x

c = example.MyClass()
c.connect_slot(spam1)
c.connect_slot(spam2)
c.event(123)
print "Sleeping"
c.event_in(3, 321)
sleep(5)
print "Done sleeping"

Results in the following:

spam1:  123
spam2:  123
Sleeping
spam1:  321
spam2:  321
Done sleeping

Answer 2

Thanks to Tanner Sansbury for linking to his answer on this post. This solved my problem except that I could not call signals that accepted arguments.

I solved this by editing the py_slot class:

struct py_slot {
    public:
        /// @brief Constructor that assumes the caller has the GIL locked.
        py_slot(const boost::python::object& object)
            : object_(new boost::python::object(object),   // GIL locked, so     copy.
            py_deleter<boost::python::object>()) // Delete needs GIL.
            {}

        void operator()(SomeParamClass param) {
            // Lock the gil as the python object is going to be invoked.
            gil_lock lock;

            (*object_)(param);

    private:
        boost::shared_ptr<boost::python::object> object_;
};

The boost::bind call would look like this:

self->connect_client_ready(boost::bind(&py_slot<SomeParamClass>::operator(), py_slot<SomeParamClass>(object), _1)); // note the _1