Benchmark C++ vs Java, Unrealistic results

问题:

I did a simple test, I know C++ is faster but the results of my test is unrealistic.

C++ code is:

#include <stdio.h>
#include <windows.h>

unsigned long long s(unsigned long long n)
{
    unsigned long long s = 0;

    for (unsigned long long i = 0; i < n; i++)
        s += i;

    return s;
}

int main()
{
    LARGE_INTEGER freq, start, end;
    QueryPerformanceFrequency(&freq);
    QueryPerformanceCounter(&start);

    printf("%llu\n", s(1000000000));

    QueryPerformanceCounter(&end);
    double d = (double) (end.QuadPart - start.QuadPart) / freq.QuadPart * 1000.0;

    printf("Delta: %f\n", d);

    return 0;
}

Java code is:

public class JavaApplication5 {

    public static long s(long n) {
        long s = 0;

        for (long i = 0; i < n; i++) {
            s += i;
        }

        return s;
    }

    public static void main(String[] args) {

        long start = System.nanoTime();

        System.out.println(s(1000000000));

        long end = System.nanoTime();

        System.out.println((end - start)/1000000);
    }
}

^{C++ compiler: gcc 4.4.0 and Java: jdk 1.6.0}

Java: 2795 ms

C++ : 0.013517 ms

It says C++ is 206777 times faster than Java! No way! What is wrong in my test?

回答1:

Show the compiler options you used. And your REAL code (#include <stdio> isn't your real code).

Your C++ compiler is much smarter than your Java compiler (this is true on average and in your case, but not every C++ compiler is smarter than every Java compiler), and it precomputed the result. The only thing you're timing is the printf call.

On most of the tasks Java is used for, it performs about as well as C++.

VM languages (Java, C#) have additional costs related to JIT compilation, but also benefit from more efficient memory allocation and inlining across shared libraries. And C++ is much much faster at accessing OS syscalls. Beyond that, C++ memory layouts can be carefully tuned for cache behavior; you don't get that level of control in managed languages.

Which of these factors has more influence is completely application-specific. Anyone making a blanket statement that "C++ is faster in general than Java" or "Java is faster in general than C++" is an idiot. Averages don't matter. Performance on YOUR application matters.

---

And here is my proof, that gcc is precomputing the answer.

On this code:

#include <stdio.h>
#include <windows.h>

unsigned long long s(unsigned long long n)
{
    unsigned long long s = 0;

    for (unsigned long long i = 0; i < n; i++)
        s += i;

    return s;
}

int main( int argc, char** argv )
{
    LARGE_INTEGER freq, start, end;
    QueryPerformanceFrequency(&freq);
    QueryPerformanceCounter(&start);

    printf("%llu\n", s(1000000000));

    QueryPerformanceCounter(&end);
    double d = (double) (end.QuadPart - start.QuadPart) / freq.QuadPart * 1000.0;

    printf("Delta: %f\n", d);

    QueryPerformanceCounter(&start);

    printf("%llu\n", s(atol(argv[1])));

    QueryPerformanceCounter(&end);
    d = (double) (end.QuadPart - start.QuadPart) / freq.QuadPart * 1000.0;

    printf("Delta: %f\n", d);
    return 0;
}

With gcc-4.3.4, using the command-line ./g++-4 -omasoud-gcc.exe -O3 masoud.cpp:

bash-3.2# ./masoud-gcc 1000000000
499999999500000000
Delta: 0.845755
499999999500000000
Delta: 1114.105866

By comparison, MSVC++ 16.00.40219.01 for x64 (2010 SP1), command-line cl /Ox masoud.cpp:

> masoud 1000000000
499999999500000000
Delta: 229.684364
499999999500000000
Delta: 354.275606

VC++ isn't precomputing the answer, but 64-bit code does execute the loop more than three times faster. This is the speed that Java ought to approach.

---

More fun facts: gcc precomputes the answer faster than the code it generates to calculate it out. Compile time for gcc:

real    0m0.886s
user    0m0.248s
sys     0m0.185s

回答2:

I guess that gcc compiled your s method into:

unsigned long long s(unsigned long long n)
{    
    return n*(n+1)/2;
}

...while the java JIT did not.

I'm no expert but gcc optimization process may contain more passes than what the JIT can do in a reasonable time. That's also why a gcc compilation may take ages while a java program launches itself immediatly (altough the JIT has to compile, link and optimize everything before the program can be run).

The java compiler (javac) does very few optimization, like constant folding, but that's pretty much all. Every major optimization (inlining, etc) is done by the JIT, so if it does not want the user to wait too long before launch, it must hurry. On the other hand, since gcc compiles and optimize everything statically, it can take as long as it needs.

EDIT:

It should be simple to know if there is a difference in optimizations: run your two programs with s(1000) and then with s(100000000000000). If I'm true, the c++ program may take the exact same time for both calls, while the java one while take longer to complete in the second case.

回答3:

First of all, you probably don't want the respective print functions to be part of your benchmark unless you really care how fast those are.

There's not a straight answer as to whether Java or C++ is faster...it depends. Java can be faster when the compiler can do some optimizations that would not be available for C++. So it depends on what specifically you are doing and what the compiler options are.