IOError: [Errno 32] Broken pipe: Python

Question

I have a very simple Python 3 script:

f1 = open('a.txt', 'r')
print(f1.readlines())
f2 = open('b.txt', 'r')
print(f2.readlines())
f3 = open('c.txt', 'r')
print(f3.readlines())
f4 = open('d.txt', 'r')
print(f4.readlines())
f1.close()
f2.close()
f3.close()
f4.close()

But it always says:

IOError: [Errno 32] Broken pipe

I saw on the internet all the complicated ways to fix this, but I copied this code directly, so I think that there is something wrong with the code and not Python's SIGPIPE.

I am redirecting the output, so if the above script was named "open.py", then my command to run would be:

open.py | othercommand

Answer 1

The problem is due to SIGPIPE handling. You can solve this problem using the following code:

from signal import signal, SIGPIPE, SIG_DFL
signal(SIGPIPE,SIG_DFL) 

See here for background on this solution. Better answer here.

Answer 2

Closes should be done in reverse order of the opens.

Answer 3

I know this is not the "proper" way to do it, but if you are simply interested in getting rid of the error message, you could try this workaround:

python your_python_code.py 2> /dev/null | other_command

Answer 4

To bring Alex L.'s helpful answer, akhan's helpful answer, and Blckknght's helpful answer together with some additional information:

• Standard Unix signal SIGPIPE is sent to a process writing to a pipe when there's no process reading from the pipe (anymore).

  • This is not necessarily an error condition; some Unix utilities such as head by design stop reading prematurely from a pipe, once they've received enough data.

• By default - i.e., if the writing process does not explicitly trap SIGPIPE - the writing process is simply terminated, and its exit code is set to 141, which is calculated as 128 (to signal termination by signal in general) + 13 (SIGPIPE's specific signal number).

• By design, however, Python itself traps SIGPIPE and translates it into a Python IOError instance with errno value errno.EPIPE, so that a Python script can catch it, if it so chooses - see Alex L.'s answer for how to do that.

• If a Python script does not catch it, Python outputs error message IOError: [Errno 32] Broken pipe and terminates the script with exit code 1 - this is the symptom the OP saw.

• In many cases this is more disruptive than helpful, so reverting to the default behavior is desirable:

  • Using the signal module allows just that, as stated in akhan's answer; signal.signal() takes a signal to handle as the 1st argument and a handler as the 2nd; special handler value SIG_DFL represents the system's default behavior:

    from signal import signal, SIGPIPE, SIG_DFL
    signal(SIGPIPE, SIG_DFL) 
    

Answer 5

A "Broken Pipe" error occurs when you try to write to a pipe that has been closed on the other end. Since the code you've shown doesn't involve any pipes directly, I suspect you're doing something outside of Python to redirect the standard output of the Python interpreter to somewhere else. This could happen if you're running a script like this:

python foo.py | someothercommand

The issue you have is that someothercommand is exiting without reading everything available on its standard input. This causes your write (via print) to fail at some point.

I was able to reproduce the error with the following command on a Linux system:

python -c 'for i in range(1000): print i' | less

If I close the less pager without scrolling through all of its input (1000 lines), Python exits with the same IOError you have reported.

Answer 6

I haven't reproduced the issue, but perhaps this method would solve it: (writing line by line to stdout rather than using print)

import sys
with open('a.txt', 'r') as f1:
    for line in f1:
        sys.stdout.write(line)

---

You could catch the broken pipe? This writes the file to stdout line by line until the pipe is closed.

import sys, errno
try:
    with open('a.txt', 'r') as f1:
        for line in f1:
            sys.stdout.write(line)
except IOError as e:
    if e.errno == errno.EPIPE:
        # Handle error

You also need to make sure that othercommand is reading from the pipe before it gets too big - https://unix.stackexchange.com/questions/11946/how-big-is-the-pipe-buffer