Testing with Qt's QTestLib module

Question

I started writing some tests with Qt's unit testing system.

How do you usually organize the tests? It is one test class per one module class, or do you test the whole module with a single test class? Qt docs suggest to follow the former strategy.

I want to write tests for a module. The module provides only one class that is going to be used by the module user, but there is a lot of logic abstracted in other classes, which I would also like to test, besides testing the public class.

The problem is that Qt's proposed way to run tests involved the QTEST_MAIN macro:

QTEST_MAIN(TestClass)
#include "test_class.moc"

and eventually one test program is capable of testing just one test class. And it kinda sucks to create test projects for every single class in the module.

Of course, one could take a look at the QTEST_MAIN macro, rewrite it, and run other test classes. But is there something, that works out of the box?

So far I do it by hand:

#include "one.h"
#include "two.h"

int main(int argc, char *argv[]) 
{ 
    QCoreApplication app(argc, argv); 
    TestOne one;
    QTest::qExec(&one, argc, argv);
    TestOne two;
    QTest::qExec(&two, argc, argv);
}

Answer 1

Usually I organize tests with one test executable per class under test.

and eventually one test program is capable of testing just one test class.

This is a good thing. It isolates your tests from each other, preventing things like a crash in one test from blocking all your other tests. That crash could be caused by a common component in several classes under test. The pattern of the failures would then tip you off to the underlying origin of the problem. Basically, you have better diagnostic information for failures if your tests are independent of each other.

Make it easy to set up multiple executables and run each test separately. Use a test runner to spawn off all the test processes.

Update:

I've changed my mind on this somewhat. Once you have a big program with lots of tests, linking hundreds of test executables becomes very slow. My new preference is to put all the tests for a library into an executable and choosing which tests to invoke using command-line arguments passed to the test executable.

That cuts down the number of executables from hundreds to dozens, but retains the advantages of running tests separately.

Answer 2

Yeah, QTest forces bit strange test structure and is generally inferior to Google Test/Mock Framework. For one project I'm forced to use QTest (client requirement), and here's how I use it:

1. I compile all test together as a subdir template project
2. To make creating new tests easier, I share a lot of project configuration by using common.pri file I include in every test .pro file
3. If possible I share the object files directory to speed up compilation
4. I run them all using a batch+awk+sed script.

Setting up this four points is very easy and makes usage of QTest almost pleasant. Do you have some problems with running multiple tests that are not solved by the config described above?

PS: running tests the way you do, i.e. calling multiple QTest::qExec causes problems with -o command line switch - you'll get only results for the last tested class.

Answer 3

In our setup with QTest, we did a few things to make it nicer.

• Define a subclass of QObject that is used as a base class for any new unit-test class.
• In the constructor for that class, we add the instance of the test to a static list of tests, and in the destructor we remove it.
• We then have a static function that loops through the tests and runs them using QTest::qExec(). (We accumulate the values returned each time, and return that from our function.)
• main() calls this function, and returns the result as the success/failure.
• Finally, in the compilation unit of the specific test itself, we usually include a static instance of that class.

This setup means that the class will be instantiated before main() is run, so it will be added to the list of classes to test when main runs. The framework requires that you just need to inherit your class properly, and instantiate a static instance if you always want it run.

We also occasionally create other optional tests, that are added based on command line switches.

Answer 4

Related to the answer posted by @cjhuitt

This is an example that removes the need of manually calling each test object

I TRY TO AVOID THINGS LIKE THIS:

MyTestClass1 t1;   t1.run();
MyTestClass2 t2;   t2.run();
//etc...

My solution is to let the test objects inherit from a base class that adds itself to a static list The main program then executes all the test objects in that list. In that way, none of the supporting framework code needs to be changed. The only things that change are the test classes themselves.

Here is how i do it:

qtestsuite.h - base class for the test objects

#ifndef QTESTSUITE_H
#define QTESTSUITE_H

#include <QObject>
#include <vector>

class QTestSuite : public QObject
{
    Q_OBJECT
public:
    static std::vector<QObject*> m_suites;

public:
    explicit QTestSuite();

};

#endif // QTESTSUITE_H

qtestsuite.cpp

#include "qtestsuite.h"
#include <iostream>

std::vector<QObject*> QTestSuite::m_suites;

QTestSuite::QTestSuite() : QObject()
{
    m_suites.push_back(this);
}

testall.cpp - runs the tests

#include "qtestsuite.h"

#include <QtTest/QtTest>
#include <iostream>

int main(int, char**)
{
    int failedSuitesCount = 0;
    std::vector<QObject*>::iterator iSuite;
    for (iSuite = QTestSuite::m_suites.begin(); iSuite != QTestSuite::m_suites.end(); iSuite++)
    {
        int result = QTest::qExec(*iSuite);
        if (result != 0)
        {
            failedSuitesCount++;
        }
    }
    return failedSuitesCount;
}

mytestsuite1.cpp - an example test object, create more of these

#include "qtestsuite.h"

#include <QtTest/QtTest>

class MyTestSuite1: public QTestSuite
{
     Q_OBJECT
private slots:
    void aTestFunction();
    void anotherTestFunction();
};

void MyTestSuite1::aTestFunction()
{
    QString str = "Hello";
    QVERIFY(str.toUpper() == "this will fail");
}

void MyTestSuite1::anotherTestFunction()
{
    QString str = "Goodbye";
    QVERIFY(str.toUpper() == "GOODBYE");
}

static MyTestSuite1 instance;  //This is where this particular test is instantiated, and thus added to the static list of test suites

#include "mytestsuite1.moc"

also, to create the .pro file

qmake -project "CONFIG += qtestlib"