Fastest stdin/out IO in python 3?

Question

I've been solving a few problems on SPOJ.pl using python 3.1.2 and some peoples fast result on simple problems makes me wonder if there is a faster way to handle input and output.

I've tried using

input()
print()

and

sys.stdin.readline()
sys.stdout.write()

or rather

for line in sys.stdin:
    #Handle input
    sys.stdout.write(output)

to process each line. I've also tried to collect all output in lists and print all at once when everything is processed.

But all of these produce similar execution times.

Is there any faster way to handle input and output from stdin/out?

Answer 1

Probably not.

In the end, print will call sys.stdout.write(). But since print is a built in function, probably implemented in C, it might even be faster than calling sys.stdout.write().

Since all IO has to go through the object which sys.stdout returns, that's the bottleneck. The same is true for sys.stdin.

There are no magic tricks to make this faster.

If you need faster IO, try these things:

• Write to a file instead
• Use buffered IO (pipe stdout using the buffer command in a small shell script).
• Use a memory mapped file
• Make sure the process which reads your output can keep up. For example the DOS console is pretty slow. If you pipe the output through a slow command, that can block your python process.

[EDIT] Seems like SPOJ.pl is some kind of programmer shootout site. In this case, I/O speed is not the culprit: You have used a bad algorithm to solve the problem.

The speed difference between a good and a fair performance can easily be between 10 and 100'000 times. By changing a few lines of code, I could once make code run in less than 5 seconds that took 45 minutes before.

Answer 2

SPOJ lets you choose among a variety of programming languages. Are you comparing your execution time to other solutions written in other programming languages?

Just for fun, I submitted the following solutions to the first problem (codename TEST) to compare run-times.

C++ solution (G++ 4.3.2)

#include <iostream>
int main ( int, char** )
{
     for ( int number=0; (std::cin >> number) && (number != 42); ) {
         std::cout << number << std::endl;
     }
}

See the submission.

Python (2.5) solution

import sys
for line in sys.stdin:
    number = int(line)
    if number == 42:
        break
    print number

See the submission.

Conclusion

I'm not 100% sure that this gets the absolute best performance in either languages, but there's not so much code in there to optimize.

I get time 0.00 measurement for the C++ and 0.04 measurement for the Python code. Assuming the sequence of numbers submitted to both programs is the same, I think comparison of run-times against solutions in other languages is almost meaningless (see next paragraph).

Now, this is only true for simple problems. Most advanced problems require choosing the right algorithm for the problem and choosing the wrong one has drastic consequences. In those cases, carefully crafted Python solutions might still be slower than carefully crafted C++ solutions but the good Python solution will beat a naïve solution written in any other language.

Answer 3

The following will probably be fastest:

1. Read all the input at once using os.read(0, some_big_enough_number).

2. Process the output, collecting the results in a list results.

3. Write all the output at once using os.write(1, "".join(results)).

I remember one case where I noticed that os.read() and os.write() are sometimes faster than using Python I/O, but I don't remember the details.