Removing Python overhead when wrapping C++ vectors

Question

from libcpp.algorithm cimport sort as stdsort
from libcpp.algorithm cimport unique
from libcpp.vector cimport vector
# from libcpp cimport bool
cimport cython

@cython.boundscheck(False)
@cython.wraparound(False)
@cython.initializedcheck(False)
cdef class Vector:
    cdef vector[cython.int] wrapped_vector

    # the easiest thing to do is add short wrappers for the methods you need
    def push_back(self, int num):
        self.wrapped_vector.push_back(num)

    def sort(self):
        stdsort(self.wrapped_vector.begin(), self.wrapped_vector.end())

    def unique(self):
        self.wrapped_vector.erase(unique(self.wrapped_vector.begin(), self.wrapped_vector.end()), self.wrapped_vector.end())


    def __str__(self):
        return "[" + ", ".join([str(i) for i in self.wrapped_vector]) + "]"

    def __repr__(self):
        return str(self)

    def __len__(self):
        return self.wrapped_vector.size()

    @cython.boundscheck(False)
    @cython.wraparound(False)
    @cython.initializedcheck(False)
    def __setitem__(self, int key, int item):
        self.wrapped_vector[key] = item

    @cython.boundscheck(False)
    @cython.wraparound(False)
    @cython.initializedcheck(False)
    def __getitem__(self, int key):
        return self.wrapped_vector[key]

I have tried to wrap vectors so that I can use them in Python dicts.

This seems to create crazy amounts of overhead. See line 72 and 75 for example. They just add an integer to the number already in the vector:

Is it possible to remove this overhead or is this the price I pay to wrap vectors?

Answer 1

This seems to be based on my answer to another question. The purpose of adding __getitem__ and __setitem__ to the cdef class Vector is purely so that it can be indexed from Python. From Cython you can index into the C++ vector directly for extra speed.

At the start of your files_to_bins add the line:

cdef Vector v

This will get Cython to make sure that anything assigned to v is a Vector object (it'll raise a TypeError if not) and thus you'll be allowed to access its cdef attributes directly.

Then change the line:

v[i] = v[i] + half_fragment_size

to:

v.wrapped_vector[i] = v.wrapped_vector[i] + half_fragment_size

(and similarly for the other indexing lines)

---

Be aware that boundscheck(False) and wraparound(False) is doing absolutely nothing for C++ objects. The C++ indexing operator performs no bounds checking (and Cython doesn't add it in) and it does not support negative indexing either. boundscheck and wraparound only apply to indexing memoryviews or numpy arrays.