Fatal Python error and `BufferedWriter`

Question

I've came across this paragraph in the Documentations which says:

Binary buffered objects (instances of BufferedReader, BufferedWriter, BufferedRandom and BufferedRWPair) protect their internal structures using a lock; it is therefore safe to call them from multiple threads at once.

I'm not sure why they need to "protect" their internal structures given the GIL is in action. Who cares? I didn't care much until I found out that this lock has some significance, consider this piece of code:

from _thread import start_new_thread
import time

def start():
    for i in range(10):
        print("SPAM SPAM SPAM!")

for i in range(10):
    start_new_thread(start, ())

time.sleep(0.0001)
print("main thread exited")

Output when run on Python 3.X:

...many SPAM...
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
main thread exited
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
SPAM SPAM SPAM!
Fatal Python error: could not acquire lock for 
<_io.BufferedWritername='<stdout>'> at interpreter shutdown, possibly due to daemon threads

Under Python 2.7, no errors. I'm not aware why would this happen, however, I've been looking around in bufferedio.c. Another code that behaves similarly to the above snippet that was tested on Python 3.X, sometimes I got Fatal Python error and sometimes I did not. Any threaded function with a loop plus std[out][err].write causes this fatal error. It's really hard to define the characteristics of this error and to the best of my knowledge the Documentation doesn't mention anything about it. I'm not sure even if it's a bug, I hope not.

My explanation of this behavior goes like this, *I could be totally wrong: The main thread exited while its holding the lock of sys.stdout.buffer. However, this seems contrary to the fact that threads are terminated when the main thread exits on the system on which I'm running Python, Linux.

---

I'm posting this as answer, it just can't be done in the comment section.

This behavior isn't just limited to write it affects read as well as flush calls on those objects BufferedReader, BufferedWriter, BufferedRandom and BufferedRWPair.

1) On Linux and probably on Windows too, when the main thread exits, its child threads are terminated. How does this affect the mentioned behavior in question? If the main thread was able to exit during its time slice, before being context switched with another thread, no fatal error occurs as all threads get terminated. Nothing guarantees however, that the main thread will exit as soon as it starts.

2) The fatal error takes place between the finalization process (shutdown) of the interpreter and the read, write, or flush call and possibly other operations on the Buffered* object. The finalization process acquires the lock of the those objects, any write for example to the BufferedWriter object results in Fatal Python error.

os._exit terminates the interpreter without the finalization steps and hence the interpreter will not own the lock of the object that we are talking about, this is another example:

from _thread import start_new_thread
import time, sys, os

def start(myId):
    for i in range(10):
        sys.stdout.buffer.write(b"SPAM\n")

for i in range(2):
    start_new_thread(start, (i,))

x = print("main thread")
print(x)

#os._exit(0)

In above code, if the main thread exits as soon as it starts, that's it, no fatal error occurs and all spawned threads are terminated immediately (at least in Linux) this is platform-dependent though. If you're unlucky enough and another thread started to play on the field before the main threads exits, without os._exit(0) call the interpreter goes through its normal cycle of finalization to acquire the lock of sys.stdout.buffer which results in fatal error. Run this code multiple times to notice its different behaviors.

Answer 1

I think you just have an erroneous understanding of GIL.

please think about when you have GIL and a list,then manipulate the list in different threads,what will happen?if you still confuse，test it.so does the BufferedWriter.

Answer 2

When I ran the first code on windows (cygwin), I got the error on python3, but I also got an error on python2

> Unhandled exception in thread started by 
> sys.excepthook is missing
> lost sys.stderr

So it is possible that on your platform python2.x may have silently exited the threads when they fail to acquire the lock. Also I believe that _thread module (thread in 2.7) is a low-level module and does not guarantee to avoid this behavior. From the module help

• When the main thread exits, it is system defined whether the other threads survive. On most systems, they are killed without executing try ... finally clauses or executing object destructors.
• When the main thread exits, it does not do any of its usual cleanup (except that try ... finally clauses are honored), and the standard I/O files are not flushed.

May be you should use higher level threading module with proper synchronization between main and other threads.

Answer 3

TL;DR

Your issue is not strictly related to lock stuff, but with the fact you are trying to write to a no-more-existent stdout with a daemon thread.

A bit of explain

When you run your main script the Python interpreter starts and execute your code opening the stdout file descriptor.

When your script ends without waiting for threads to finish:

• all the threads switch from non-daemons to daemons
• the interpreter exits calling a finalize function which wypes the threads' globals including the stdout
• the now-daemon threads try to aquire lock for stdout which is no more accessible due to the previous step

To avoid this issue you could write to file instead of stdout (as a daemon thread should do) or just wait for threads to finish with something like:

from threading import Thread
import time

def start():
    for i in range(10):
        print("SPAM SPAM SPAM!")

# create a thread list (you'll need it later)
threads = [Thread(target=start, args=()) for i in range(10)]

# start all the threads
for t in threads:
    t.start()
# or [t.start() for t in threads] if you prefer the inlines

time.sleep(0.0001)

# wait for threads to finish
for t in threads:
    t.join()
# or [t.join() for t in threads] for the inline version

print("main thread exited")