What unit-testing framework should I use for Qt? [

Question

I am just starting up a new project that needs some cross-platform GUI, and we have chosen Qt as the GUI-framework.

We need a unit-testing framework, too. Until about a year ago we used an in-house developed unit-testing framework for C++-projects, but we are now transitioning to using Google Test for new projects.

Does anyone have any experience with using Google Test for Qt-applications? Is QtTest/QTestLib a better alternative?

I am still not sure how much we want to use Qt in the non-GUI parts of the project - we would probably prefer to just use STL/Boost in the core-code with a small interface to the Qt-based GUI.

EDIT: It looks like many are leaning towards QtTest. Is there anybody who has any experience with integrating this with a continous integration server? Also, it would seem to me that having to handle a separate application for each new test case would cause a lot of friction. Is there any good way to solve that? Does Qt Creator have a good way of handling such test cases or would you need to have a project per test case?

Answer 1

If you are using Qt, I would recommend using QtTest, because is has facilities to test the UI and is simple to use.

If you use QtCore, you can probably do without STL. I frequently find the Qt classes easier to use than the STL counterparts.

Answer 2

I've just been playing around with this. The main advantage of using Google Test over QtTest for us is that we do all our UI development in Visual Studio. If you use Visual Studio 2012 and install the Google Test Adapter you can get VS to recognise the tests and include them in its Test Explorer. This is great for developers to be able to use as they write code, and because Google Test is portable we can also add the tests to the end of our Linux build.

I'm hoping in the future that someone will add support for C++ to one of the concurrent testing tools that C# have, like NCrunch, Giles and ContinuousTests.

Of course, you might find someone writes another adapter for VS2012 that adds QtTest support to Test Adapter in which case this advantage goes away! If anyone is interested in this there's a good blog post Authoring a new Visual studio unit test adapter.

Answer 3

To append to Joe's answer.

Here's a small header I use (testrunner.h), containing an utility class spawning an event loop (which is, for example, needed to test queued signal-slot connections and databases) and "running" QTest-compatible classes:

#ifndef TESTRUNNER_H
#define TESTRUNNER_H

#include <QList>
#include <QTimer>
#include <QCoreApplication>
#include <QtTest>

class TestRunner: public QObject
{
    Q_OBJECT

public:
    TestRunner()
        : m_overallResult(0)
    {}

    void addTest(QObject * test) {
        test->setParent(this);
        m_tests.append(test);
    }

    bool runTests() {
        int argc =0;
        char * argv[] = {0};
        QCoreApplication app(argc, argv);
        QTimer::singleShot(0, this, SLOT(run()) );
        app.exec();

        return m_overallResult == 0;
    }
private slots:
    void run() {
        doRunTests();
        QCoreApplication::instance()->quit();
    }
private:
    void doRunTests() {
        foreach (QObject * test, m_tests) {
            m_overallResult|= QTest::qExec(test);
        }
    }

    QList<QObject *> m_tests;
    int m_overallResult;
};

#endif // TESTRUNNER_H

Use it like this:

#include "testrunner.h"
#include "..." // header for your QTest compatible class here

#include <QDebug>

int main() {
    TestRunner testRunner;
    testRunner.addTest(new ...()); //your QTest compatible class here

    qDebug() << "Overall result: " << (testRunner.runTests()?"PASS":"FAIL");

    return 0;
}

Answer 4

Why not using the unit-testing framework included in Qt? An example : QtTestLib Tutorial.

Answer 5

I started off using QtTest for my app and very, very quickly started running into limitations with it. The two main problems were:

1) My tests run very fast - sufficiently quickly that the overhead of loading an executable, setting up a Q(Core)Application (if needed) etc often dwarfs the running time of the tests themselves! Linking each executable takes up a lot of time, too.

The overhead just kept on increasing as more and more classes were added, and it soon became a problem - one of the goals of unit tests are to have a safety net that runs so fast that it is not a burden at all, and this was rapidly becoming not the case. The solution is to glob multiple test suites into one executable, and while (as shown above) this is mostly do-able, it is not supported and has important limitations.

2) No fixture support - a deal-breaker for me.

So after a while, I switched to Google Test - it is a far more featureful and sophisticated unit testing framework (especially when used with Google Mock) and solves 1) and 2), and moreover, you can still easily use the handy QTestLib features such as QSignalSpy and simulation of GUI events, etc. It was a bit of a pain to switch, but thankfully the project had not advanced too far and many of the changes could be automated.

Personally, I will not be using QtTest over Google Test for future projects - if offers no real advantages that I can see, and has important drawbacks.

Answer 6

QtTest is mostly useful for testing parts that require the Qt event loop/signal dispatching. It's designed in a way that each test case requires a separate executable, so it should not conflict with any existing test framework used for the rest of the application.

(Btw, I highly recommend using QtCore even for non-GUI parts of the applications. It's much nicer to work with.)