OpenCV::solvePNP() - Assertion failed

Question

I am trying to get the pose of the camera with the help of solvePNP() from OpenCV.

After running my program I get the following errors:

OpenCV Error: Assertion failed (npoints >= 0 && npoints == std::max(ipoints.checkVector(2, CV_32F), ipoints.checkVector(2, CV_64F))) in solvePnP, file /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_graphics_opencv/opencv/work/OpenCV-2.4.2/modules/calib3d/src/solvepnp.cpp, line 55
libc++abi.dylib: terminate called throwing an exception

I tried to search how to solve these errors, but I couldn't resolve it unfortunately!

Here is my code, all comment/help is much appreciated:

enum Pattern { NOT_EXISTING, CHESSBOARD, CIRCLES_GRID, ASYMMETRIC_CIRCLES_GRID };

void calcBoardCornerPositions(Size boardSize, float squareSize, vector<Point3f>& corners,
                          Pattern patternType)
{
    corners.clear();

    switch(patternType)
    {
    case CHESSBOARD:
    case CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; ++i )
            for( int j = 0; j < boardSize.width; ++j )
                corners.push_back(Point3f(float( j*squareSize ), float( i*squareSize ), 0));
        break;

    case ASYMMETRIC_CIRCLES_GRID:
        for( int i = 0; i < boardSize.height; i++ )
            for( int j = 0; j < boardSize.width; j++ )
                corners.push_back(Point3f(float((2*j + i % 2)*squareSize), float(i*squareSize), 0));
        break;
    }
}

int main(int argc, char* argv[])
{
    float squareSize = 50.f;

    Pattern calibrationPattern = CHESSBOARD;

    //vector<Point2f> boardCorners;
    vector<vector<Point2f> > imagePoints(1);
    vector<vector<Point3f> > boardPoints(1);

    Size boardSize;
    boardSize.width = 9;
    boardSize.height = 6;

    vector<Mat> intrinsics, distortion;
    string filename = "out_camera_xml.xml";
    FileStorage fs(filename, FileStorage::READ);
    fs["camera_matrix"] >> intrinsics;
    fs["distortion_coefficients"] >> distortion;
    fs.release();

    vector<Mat> rvec, tvec;
    Mat img = imread(argv[1], CV_LOAD_IMAGE_GRAYSCALE); // at kell adnom egy kepet

    bool found = findChessboardCorners(img, boardSize, imagePoints[0], CV_CALIB_CB_ADAPTIVE_THRESH);

    calcBoardCornerPositions(boardSize, squareSize, boardPoints[0], calibrationPattern);
    boardPoints.resize(imagePoints.size(),boardPoints[0]);

    //***Debug start***
    cout << imagePoints.size() << endl << boardPoints.size() << endl << intrinsics.size() << endl << distortion.size() << endl;
    //***Debug end***

    solvePnP(Mat(boardPoints), Mat(imagePoints), intrinsics, distortion, rvec, tvec);

    for(int i=0; i<rvec.size(); i++) {
            cout << rvec[i] << endl;
    }

    return 0;
}

EDIT (some debug info):

I debugged it row by row. I stepped into all of the functions. I am getting the Assertion failed in SolvePNP(...). You can see below what I see when I step into the solvePNP function. First it jumps over the first if statement /if(vec.empty())/, and goes into the second if statement /if( !copyData )/, there when it executes the last line /*datalimit = dataend = datastart + rows*step[0]*/ jumps back to the first if statement and returns => than I get the Assertion failed error.

template<typename _Tp> inline Mat::Mat(const vector<_Tp>& vec, bool copyData)
: flags(MAGIC_VAL | DataType<_Tp>::type | CV_MAT_CONT_FLAG),
dims(2), rows((int)vec.size()), cols(1), data(0), refcount(0),
datastart(0), dataend(0), allocator(0), size(&rows)
{
    if(vec.empty())
        return;
    if( !copyData )
    {
        step[0] = step[1] = sizeof(_Tp);
        data = datastart = (uchar*)&vec[0];
        datalimit = dataend = datastart + rows*step[0];
    }
    else
        Mat((int)vec.size(), 1, DataType<_Tp>::type, (uchar*)&vec[0]).copyTo(*this);
}

Answer 1

Step into the function in a debugger and see exactly which assertion is failing. ( Probably it requires values in double (CV_64F) rather than float. )

OpenCVs new "inputarray" wrapper issuppsoed to allow you to call functions with any shape of mat, vector of points, etc - and it will sort it out. But a lot of functions assume a particular inut format or have obsolete assertions enforcing a particular format.

The stereo/calibration systems are the worst for requiring a specific layout, and frequently succesive operations require a different layout.

Answer 2

The types don't seem right, at least in the code that worked for me I used different types(as mentioned in the documentation).

objectPoints – Array of object points in the object coordinate space, 3xN/Nx3 1-channel or 1xN/Nx1 3-channel, where N is the number of points. vector can be also passed here.

imagePoints – Array of corresponding image points, 2xN/Nx2 1-channel or 1xN/Nx1 2-channel, where N is the number of points. vector can be also passed here.

cameraMatrix – Input camera matrix A = \vecthreethree{fx}{0}{cx}{0}{fy}{cy}{0}{0}{1} .

distCoeffs – Input vector of distortion coefficients (k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6]]) of 4, 5, or 8 elements. If the vector is NULL/empty, the zero distortion coefficients are assumed.

rvec – Output rotation vector (see Rodrigues() ) that, together with tvec , brings points from the model coordinate system to the camera coordinate system.

tvec – Output translation vector.

useExtrinsicGuess – If true (1), the function uses the provided rvec and tvec values as initial approximations of the rotation and translation vectors, respectively, and further optimizes them.

Documentation from here.

vector<Mat> rvec, tvec should be Mat rvec, tvec instead.

vector<vector<Point2f> > imagePoints(1) should be vector<Point2f> imagePoints(1) instead.

vector<vector<Point3f> > boardPoints(1) should be vector<Point3f> boardPoints(1) instead.

Note: I encountered the exact same problem, and this worked for me(It is a little bit confusing since calibrateCamera use vectors). Haven't tried it for imagePoints or boardPoints though.(but as it is documented in the link above, vector,vector should work, I thought I'd better mention it), but for rvec,trec I tried it myself.

Answer 3

I run in exactly the same problem with solvePnP and opencv3. I tried to isolate the problem in a single test case. I seams passing a std::vector to cv::InputArray does not what is expected. The following small test works with opencv 2.4.9 but not with 3.2.

And this is exactly the problem when passing a std::vector of points to solvePnP and causes the assert at line 63 in solvepnp.cpp to fail !

Generating a cv::mat out of the vector list before passing to solvePnP works.

//create list with 3 points
std::vector<cv::Point3f> vectorList;
vectorList.push_back(cv::Point3f(1.0, 1.0, 1.0));
vectorList.push_back(cv::Point3f(1.0, 1.0, 1.0));
vectorList.push_back(cv::Point3f(1.0, 1.0, 1.0));

//to input array
cv::InputArray inputArray(vectorList);
cv::Mat mat = inputArray.getMat();
cv::Mat matDirect = cv::Mat(vectorList);

LOG_INFO("Size vector: %d mat: %d matDirect: %d", vectorList.size(), mat.checkVector(3, CV_32F), matDirect.checkVector(3, CV_32F));

QVERIFY(vectorList.size() == mat.checkVector(3, CV_32F));

Result opencv 2.4.9 macos:

TestObject: OpenCV
Size vector: 3 mat: 3 matDirect: 3

Result opencv 3.2 win64:

TestObject: OpenCV
Size vector: 3 mat: 9740 matDirect: 3