How to return a std::vector<const> to python?

I have a c++11 function that returns a:

std::vector<const T*> f();

with T being a c++ class that I exposed to python with class_. All the T instances reside in static storage that will live throught the live of the python process.

I am trying to expose f as a python function

getAllTs()

that would return python objects wrappers around T. I chose T* to be the held type for class_.

I am converting std::vector to a python tuple, with this bad semi-generic function:

template <typename Cont>
struct stdcont_to_python_tuple
{
  static PyObject* convert(const Cont& container)
  {
    boost::python::list lst;
    for (const auto& elt: container)
      lst.append(elt);

    return boost::python::incref( boost::python::tuple(lst).ptr() );
  }

  static PyTypeObject const* get_pytype()
  {
    return &PyTuple_Type;
  }
};

I couldn't construct the tuple directory from the container. Is that possible?

I need to display these Ts in a UI table, perform sorting, filtering.

The max number of T instances is 30000 odd. In c++11:

sizeof(T) = 24 bytes

In python3:

sys.getsizeof(t) = 72 bytes

What is the return policy I can use where def'ing getAllTs to minimize duplication, ie to have the least extras added by python?

Thanks

Exposing `std::vector<const T*>` to Python

The easiest way to expose std::vector<...> is to expose it as a boost::python::class_, and use the vector_indexing_suite to provide a Python sequence like interface.

std::vector<const spam*> get_spams() { ... }

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose `spam`
  python::class_<spam, spam*>("Spam");

  // Expose `std::vector<const spam*>`
  python::class_<std::vector<const spam*>>("Spams")
    .def(python::vector_indexing_suite<std::vector<const spam*>>())
    ;

  python::def("get_spams", &get_spams);
}

However, using the type const spam* may not be as fruitful as one would hope. First, Python does not have the concept of const. Furthermore, when exposing the spam class as being held by spam*, the automatic to-python and from-python converter are for spam*, not const spam*. This may not be immediately apparent due to the indexing suite returning proxies during indexing, but it will become apparent when iterating, as the iterator's value will attempt to be converted to Python.

spams = example.get_spams()
# Access by index returns a proxy.  It does not perform a
# spam-to-python conversion.
spams[0].perform()
# The iterator's value will be returned, requiring a 
# spam-to-python conversion.
for spam in spams:
    pass

To resolve this, one can register an explicit to-python conversion for const spam*. I would strongly consider re-examining if const spam* is necessary, or if exposing spam as being held by a different type would be easier (e.g. boost::shared_ptr with a null deleter). Regardless, here is a complete example demonstrating this functionality:

#include <iostream>
#include <boost/python.hpp>
#include <boost/python/suite/indexing/vector_indexing_suite.hpp>

/// Mocks...
struct spam
{
  spam() { std::cout << "spam() " << this << std::endl; }
  ~spam() { std::cout << "~spam() " << this << std::endl; }
  void perform() { std::cout << "spam::perform() " << this << std::endl; }
};

namespace {
  std::array<spam, 3> spams;
} // namespace

std::vector<const spam*> get_spams()
{
  std::vector<const spam*> result;
  for (auto& spam: spams)
  {
    result.push_back(&spam);
  }
  return result;
}

/// @brief Convert for converting `const spam*` to `Spam`.
struct const_spam_ptr_to_python
{
  static PyObject* convert(const spam* ptr)
  {
    namespace python = boost::python;
    python::object object(python::ptr(ptr));
    return python::incref(object.ptr());
  }
};

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

   // Enable `const spam*` to `Spam` converter.
  python::to_python_converter<const spam*, const_spam_ptr_to_python>();

  // Expose `spam`.
  python::class_<spam, spam*>("Spam", python::no_init)
    .def("perform", &spam::perform)
    ;

  // Expose `std::vector<const spam*>`.
  python::class_<std::vector<const spam*>>("Spams")
    .def(python::vector_indexing_suite<std::vector<const spam*>>())
    ;

  python::def("get_spams", &get_spams);
}

Interactive usage:

>>> import example
spam() 0x7ffbec612218
spam() 0x7ffbec612219
spam() 0x7ffbec61221a
>>> spams = example.get_spams()
>>> for index, spam in enumerate(spams):
...     spams[index].perform()
...     spam.perform()
...
spam::perform() 0x7ffbec612218
spam::perform() 0x7ffbec612218
spam::perform() 0x7ffbec612219
spam::perform() 0x7ffbec612219
spam::perform() 0x7ffbec61221a
spam::perform() 0x7ffbec61221a
~spam() 0x7ffbec61221a
~spam() 0x7ffbec612219
~spam() 0x7ffbec612218

Construct Python tuple from C++ container

A boost::python::tuple can be constructed from a sequence. If a C++ object is provided, then it needs to be convertible to a Python type that implements the Python iterator protocol. For instance, in the above example, as std::vector<const spam*> is exposed and provides a sequence like interface via the vector_indexing_suite, one could write get_spams() as:

boost::python::tuple get_spams()
{
  std::vector<const spam*> result;
  for (auto& spam: spams)
  {
    result.push_back(&spam);
  }
  return boost::python::tuple(result);
}

Alternatively, one can use types and functions provided by the boost/python/iterator.hpp file to create Python iterators from C++ Containers or Iterators. The examples demonstrate exposing a std::pair of iterators, begin and end, to Python. As the std::pair will have a to-python conversion available, one could construct a boost::python::tuple from the std::pair.

Here is a compete example demonstrating this approach:

#include <boost/python.hpp>

/// @brief Returns a tuple, constructing it form a range.
template <typename Container>
boost::python::tuple container_to_tuple(Container& container)
{
  namespace python = boost::python;
  return python::tuple(std::make_pair(
      container.begin(), container.end()));
}

BOOST_PYTHON_MODULE(example)
{
  namespace python = boost::python;

  // Expose an int range.
  typedef std::vector<int>::iterator vector_int_iterator;
  typedef std::pair<vector_int_iterator, vector_int_iterator> vector_int_range;
  python::class_<vector_int_range>("IntRange")
    .def("__iter__", python::range(
        &vector_int_range::first, &vector_int_range::second))
      ;

   // Return a tuple of ints.
  python::def("get_ints", +[] {
    std::vector<int> result;
    result.push_back(21);
    result.push_back(42);
    return container_to_tuple(result);
  });
}

Interactive usage:

>>> import example
>>> ints = example.get_ints()
>>> assert(isinstance(ints, tuple))
>>> assert(ints == (21, 42))

Memory Footprint

If the C++ object already exists, one can have the python::object reference it via a pointer, which will reduce duplicating some overall memory usage. However, there are no options to reduce the base footprint of instances for Boost.Python classes, which comes from new-style class, size for variable-length data, C++ object, vtable pointer, pointer to instance holder, and padding for instance holder alignment. If you need a smaller footprint, then consider using the Python/C API directly for type creation, and using Boost.Python for interacting with the objects.