How to test an EXE with Google Test?

Question

I have a C++ project in Visual Studio, and have added another project exclusively for testing. Both of these projects are EXEs (console apps). So how do I use the first project inside the second?

Just to clarify, the question here would be somewhat self-evident if the first project was a library that one could simply include in the second project but, being an EXE, this is where the problem lies.

Answer 1

Depends. Google Test is (primarily) a Unit Testing framework (oversimplifying, testing classes). You can absolutely use is for other types of tests, but it doesn't have "built in" functionality for other types of testing, you'll have to write it yourself.

If you are trying to system test your executable, than you can run the process. I suggest using Boost.Process if you are using a multi-platform system or already have a boost dependency. Else, look here: launch an exe/process with stdin stdout and stderr?

The "tests" you write will call the executable, and can input stdin or stdout accordingly.

For example:

std::string path_to_exectuable = "thepath";
TEST(FooTester,CheckHelpScriptReturns0)
{
 using bp =::boost::process; 
 std::vector<std::string> args; args.push_back("--help");
 bp::context ctx; 
 ctx.stdout_behavior = bp::capture_stream(); 

 bp::child c = bp::launch(exec, args, ctx); 
 bp::status s = c.wait(); 
 ASSERT_TRUE(s.exited())<<"process didn't exit!";
 ASSERT_EQ(s.exit_status(),0)<<"Help didn't return 0";
}

Answer 2

If you want to test a console app you can run a test that opens a console window and run the exe file of the first app. Then in your googletest catch the standard output from the exe you just ran.

[For more control over the first app you might need to have the first app parse arguments sent to it, e.g. some flags like -x or what ever you need.]

Answer 3

Per your comments, you have a C++ console application (MyApp) for which you have developed some application-specific classes that you want to unit-test with googletest in Visual Studio. How?

As you say, if you wanted to unit-test a library the way to do it would be obvious. You would:

• 1) Create a project to create a unit-testing application (UnitTest).
• 2) Configure the include-search directories so that the compiler can find the library's headers.
• 3) Configure the library-search directories so that the linker can find the library itself.
• 4) Add the library itself to the linker inputs.
• 5) Make the UnitTest project dependent on the library project, so that building UnitTest ensures MyApp is up-to-date.
• 6) Code the UnitTest app per googletest docs.

But since the classes you want to unit-test are specific to MyApp, you don't have any library.

A drill-sergeant answer to that is: You don't have a library containing the classes you want to unit-test? So make one!

That way you use 3 projects:-

• MyAppLib, generating library that contains all the functionality you want to unit-test.
• MyApp, generating the same executable as at present, but linking MyAppLib
• UnitTest, generating an executable that unit-tests MyAppLib, also linking MyAppLib

However if you don't like the drill-sergeant answer you can work around it.

From the usual build-system point of view (the one designed into Visual Studio), the important output of the MyApp project is the build-target - the .exe. The .obj files generated are just intermediate by-products. VS offers you no support for treating these by-products as automatic linker inputs of a dependent project, and if a dependent project was also an .exe of the same sort - as it is your case - then such automatic linkage would be impossible anyhow because the main entry point would be multiply defined.

But from the unit-testing point of view it's the other way round. The .exe is of no interest, whereas (some of) the .obj files wholly or partly contain the implementations of the classes you want to unit test. In the text-book case where class foo is defined in foo.h and implemented in foo.cpp, the object file foo.obj is needed in the linkage of UnitTest.

For simplicity, assume that MyApp employs just one application-specific class foo, defined in foo.h and implemented in foo.cpp. Then you have two options for building UnitTest.

• a) You can add foo.cpp to the source files of UnitTest. Don't copy it of course. Just Add an existing item from the source folder of MyApp. Then you're done, but this course has the downside that foo.cpp is exposed to untoward editing within the UnitTest project.

• b) You can treat foo.obj just like a static library required for the linkage of UnitTest and follow steps 1) - 6) above. This means in particular at step 3) that the {Debug|Release} build of UnitTest is configured with library-search directories that include \path\to\MyApp\{Debug|Release} (either in relative or absolute form).

In reality, for option b), there's very likely more than one .obj file from MyApp that you will have to link in UnitTest, and quite likely that their number will grow with time. Maintaining the right linkage of UnitTest could become a chore, and you might come to the conclusion that the drill-sergeant was right after all.

Answer 4

I was in a similar situation and I set this up in a way that effectively accomplishes the same goal of Mike Kinghan's answer as far as the compiler is concerned, but goes about it a different way from the user's perspective.

What I did was create a custom Configuration that I called "Testing". You create a new configuration by opening the project settings, choosing "Configuration Manager..." and selecting "New..." in the configuration selection box.

When prompted, I chose to copy the settings from the default "Debug" configuration, so that I can use the debugger with my tests just the same as if I was in the "Debug" configuration.

Under the new Testing configuration, I set the options for the compiler and linker to use google test as you normally would.

The important change in the properties is that I define a preprocessor variable which I have called "TESTING".

I rewrote my "main.cpp" to look something like this:

...
// includes
// functions
// whatever
...

#ifdef TESTING
#include <gtest/gtest.h>
#endif

int main(int argc, char **argv) {
   #ifdef TESTING
   ::testing::InitGoogleTest(&argc, argv);
   int val = RUN_ALL_TESTS();
   getchar();  // not necessary, but keeps the console open
   return val;
   #endif    

   // rest of main() as normal...
}

What I'm trying to indicate is that I only changed a few lines right around where main is defined, I don't have to make gross changes spread throughout the file.

Now that this is all set up I simply made a new source folder for my tests, and create ".cpp" files in there. To avoid bloating the normal executable, I wrap these files with a check for the TESTING variable, so I have something like this:

tests/Test.cpp:

#ifdef TESTING

#include <gtest/gtest.h>

#include "my_class_header.h"

TEST(TestMyClass, test_something) {
    // perform some test on class
} 

#endif

I think these files still get "hit" by the compiler under Debug and Release configurations, so having a ton of these might slow down the build, but the Debug and Release objects wont get bloated with testing code.

The two takeaways are:

• Using this method, the testing code is still organized separately from the application code, but it still resides in the same Visual Studio project, which may or may not be beneficial. Personally I like not having to manage/worry about a second project.
• Like Mike Kinghan said, managing and linking .obj files yourself can become a chore, but by using this method, the default Visual Studio settings manage this for you.

One downside is that effectively redundant copies of all object files will get created in the "Testing" output directory. With more configuration, surely there must be a way to "share" the Debug object files, but I didn't have a reason to go that far.

This is a very simple method which may be a lot easier than refactoring your application into separate libraries and a main. I don't love using preprocessor wankery, but in this case it's fairly straightforward, not too much code bloat, and accomplishes exactly what it needs to. You could always trigger the tests another way, without using the preprocessor.

Answer 5

If you are not very rigid about having the tests in a different project, you can write the tests in your application project. Then just make the application execute the tests when receiving certain command line arguments, and execute the normal application logic otherwise, i.e.

int main(int argc, char* argv[])
{
    if (argc >= 2 && std::string(argv[1]) == "--tests")
    {
        ::testing::InitGoogleTest(&argc, argv);
        return RUN_ALL_TESTS();
    }
    else
    {
        // Application logic goes here
    }
}

TEST(ExampleTests, TestSQRTCalculation) // assuming all the right headers are included
{
    EXPECT_NEAR(2.0, std::sqrt(4.0), 0.000001);
}

This avoids creating an unnecessary library for the sole purpose of testing, although you can still do it if it is correct structure-wise. The downside is that the testing code goes into the executable you are going to release. If you don't want that I guess you need an additional configuration which specifies a pre-processor directive to disable the tests.

Debugging the tests or running them automatically at post build is easy, simply by specifying "--tests" as debug args or at post build command line respectively.

Answer 6

I have prepared a github repo including Visual Studio 2015 solution in parralel of Mike's "drill-sergeant" suggestion. You can use it directly without any additional requirement or dependency.

https://github.com/fuatcoskun/GoogleTestVS2015

I hope it helps...