What unit-testing framework should I use for Qt? [

Question

I am just starting up a new project that needs some cross-platform GUI, and we have chosen Qt as the GUI-framework.

We need a unit-testing framework, too. Until about a year ago we used an in-house developed unit-testing framework for C++-projects, but we are now transitioning to using Google Test for new projects.

Does anyone have any experience with using Google Test for Qt-applications? Is QtTest/QTestLib a better alternative?

I am still not sure how much we want to use Qt in the non-GUI parts of the project - we would probably prefer to just use STL/Boost in the core-code with a small interface to the Qt-based GUI.

EDIT: It looks like many are leaning towards QtTest. Is there anybody who has any experience with integrating this with a continous integration server? Also, it would seem to me that having to handle a separate application for each new test case would cause a lot of friction. Is there any good way to solve that? Does Qt Creator have a good way of handling such test cases or would you need to have a project per test case?

Answer 1

You don't have to create separate tests applications. Just use qExec in an independent main() function similar to this one:

int main(int argc, char *argv[])
{
    TestClass1 test1;
    QTest::qExec(&test1, argc, argv);

    TestClass2 test2;
    QTest::qExec(&test2, argc, argv);

    // ...

    return 0;
}

This will execute all test methods in each class in one batch.

Your testclass .h files would look as follows:

class TestClass1 : public QObject
{
Q_OBJECT

private slots:
    void testMethod1();
    // ...
}

Unfortunately this setup isn't really described well in the Qt documentation even though it would seem to be quite useful for a lot of people.

Answer 2

To extend mlvljr's and Joe's solution we can even support complete QtTest options per one test class and still run all in a batch plus logging:

usage: 
  help:                                        "TestSuite.exe -help"
  run all test classes (with logging):         "TestSuite.exe"
  print all test classes:                      "TestSuite.exe -classes"
  run one test class with QtTest parameters:   "TestSuite.exe testClass [options] [testfunctions[:testdata]]...

Header

#ifndef TESTRUNNER_H
#define TESTRUNNER_H

#include <QList>
#include <QTimer>
#include <QCoreApplication>
#include <QtTest>
#include <QStringBuilder>

/*
Taken from https://stackoverflow.com/questions/1524390/what-unit-testing-framework-should-i-use-for-qt
BEWARE: there are some concerns doing so, see  https://bugreports.qt.io/browse/QTBUG-23067
*/
class TestRunner : public QObject
{
   Q_OBJECT

public:
   TestRunner() : m_overallResult(0) 
   {
      QDir dir;
      if (!dir.exists(mTestLogFolder))
      {
         if (!dir.mkdir(mTestLogFolder))
            qFatal("Cannot create folder %s", mTestLogFolder);
      }
   }

   void addTest(QObject * test)
   {
      test->setParent(this);
      m_tests.append(test);
   }

   bool runTests(int argc, char * argv[]) 
   {
      QCoreApplication app(argc, argv);
      QTimer::singleShot(0, this, SLOT(run()));
      app.exec();

      return m_overallResult == 0;
   }

   private slots:
   void run() 
   {
      doRunTests();
      QCoreApplication::instance()->quit();
   }

private:
   void doRunTests() 
   {
      // BEWARE: we assume either no command line parameters or evaluate first parameter ourselves
      // usage: 
      //    help:                                        "TestSuite.exe -help"
      //    run all test classes (with logging):         "TestSuite.exe"
      //    print all test classes:                      "TestSuite.exe -classes"
      //    run one test class with QtTest parameters:   "TestSuite.exe testClass [options] [testfunctions[:testdata]]...
      if (QCoreApplication::arguments().size() > 1 && QCoreApplication::arguments()[1] == "-help")
      {
         qDebug() << "Usage:";
         qDebug().noquote() << "run all test classes (with logging):\t\t" << qAppName();
         qDebug().noquote() << "print all test classes:\t\t\t\t" << qAppName() << "-classes";
         qDebug().noquote() << "run one test class with QtTest parameters:\t" << qAppName() << "testClass [options][testfunctions[:testdata]]...";
         qDebug().noquote() << "get more help for running one test class:\t" << qAppName() << "testClass -help";
         exit(0);
      }

      foreach(QObject * test, m_tests)
      {
         QStringList arguments;
         QString testName = test->metaObject()->className();

         if (QCoreApplication::arguments().size() > 1)
         {
            if (QCoreApplication::arguments()[1] == "-classes")
            {
               // only print test classes
               qDebug().noquote() << testName;
               continue;
            }
            else
               if (QCoreApplication::arguments()[1] != testName)
               {
                  continue;
               }
               else
               {
                  arguments = QCoreApplication::arguments();
                  arguments.removeAt(1);
               }
         }
         else
         {
            arguments.append(QCoreApplication::arguments()[0]);
            // log to console
            arguments.append("-o"); arguments.append("-,txt");
            // log to file as TXT
            arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".log,txt");
            // log to file as XML
            arguments.append("-o"); arguments.append(mTestLogFolder % "/" % testName % ".xml,xunitxml");
         }
         m_overallResult |= QTest::qExec(test, arguments);
      }
   }

   QList<QObject *> m_tests;
   int m_overallResult;
   const QString mTestLogFolder = "testLogs";
};

#endif // TESTRUNNER_H

own code

#include "testrunner.h"
#include "test1" 
...

#include <QDebug>

int main(int argc, char * argv[]) 
{
    TestRunner testRunner;

    //your QTest compatible class here
    testRunner.addTest(new Test1);
    testRunner.addTest(new Test2);
    ...

    bool pass = testRunner.runTests(argc, argv);
    qDebug() << "Overall result: " << (pass ? "PASS" : "FAIL");

    return pass?0:1;
}

Answer 3

I unit tested our libraries using gtest and QSignalSpy. Use QSignalSpy to catch signals. You can call slots directly (like normal methods) to test them.

Answer 4

For Visual Studio test adapter tool support with the QtTest framework use this Visual Studio extension: https://visualstudiogallery.msdn.microsoft.com/cc1fcd27-4e58-4663-951f-fb02d9ff3653

Answer 5

I don't know that QTestLib is "better" than one framework for another in such general terms. There is one thing that it does well, and that's provide a good way to test Qt based applications.

You could integrate QTest into your new Google Test based setup. I haven't tried it, but based on how QTestLib is architected, it seems like it would not be too complicated.

Tests written with pure QTestLib have an -xml option that you could use, along with some XSLT transformations to convert to the needed format for a continuous integration server. However, a lot of that depends on which CI server you go with. I would imagine the same applies to GTest.

A single test app per test case never caused a lot of friction for me, but that depends on having a build system that would do a decent job of managing the building and execution of the test cases.

I don't know of anything in Qt Creator that would require a seperate project per test case but it could have changed since the last time I looked at Qt Creator.

I would also suggest sticking with QtCore and staying away from the STL. Using QtCore throughout will make dealing with the GUI bits that require the Qt data types easier. You won't have to worry about converting from one data type to another in that case.