On windows, I want to copy a bunch of files over a network with Python. Sometimes, the network is not responding, and the copy is stalled. I want to check, if that happens, and skip the file in question, when that happens. By asking this related question here, I found out about the CopyFileEx function, that allows the use of a callback function, that can abort the file copy.
The implementation in Python looks like that:
import win32file
def Win32_CopyFileEx( ExistingFileName, NewFileName, Canc = False):
win32file.CopyFileEx(
ExistingFileName, # PyUNICODE | File to be copied
NewFileName, # PyUNICODE | Place to which it will be copied
Win32_CopyFileEx_ProgressRoutine, # CopyProgressRoutine | A python function that receives progress updates, can be None
Data = None, # object | An arbitrary object to be passed to the callback function
Cancel = Canc, # boolean | Pass True to cancel a restartable copy that was previously interrupted
CopyFlags = win32file.COPY_FILE_RESTARTABLE, # int | Combination of COPY_FILE_* flags
Transaction = None # PyHANDLE | Handle to a transaction as returned by win32transaction::CreateTransaction
)
From the documentation of the CopyFileEx function, I can see two possibilities of cancelation of a running copy.
pbCancel [in, optional] If this flag is set to TRUE during the copy operation, the operation is canceled. Otherwise, the copy
operation will continue to completion.
I could not figure out a way how to do that. I tried calling the same function with the same file handles again but with the cancel flag set to TRUE, but that leads in an error, because of the file in question being in use by another process.
Another possibility seems to be the callback function:
lpProgressRoutine [in, optional] The address of a callback function of
type LPPROGRESS_ROUTINE that is called each time another portion of
the file has been copied. This parameter can be NULL. For more
information on the progress callback function, see the
CopyProgressRoutine function.
The documentation of this ProgressRoutine states, that this callback is either called when the copy is started or when a junk of the file is finished copying. The callback function can cancel the copy process if it returns 1 or 2 ( cancel, stop). However, this callback function seems to not being called, when the copy of a junk is stalled.
So my question is: How I can cancel this copy on a per-file-basis when it is stalled?
win32file.CopyFileEx doesn't allow passing Cancel as anything but a boolean or integer value. In the API it's an LPBOOL pointer, which allows the caller to set its value concurrently in another thread. You'll have to use ctypes, Cython, or a C extension to get this level of control. Below I've written an example using ctypes.
If canceling the copy doesn't work because the thread is blocked on synchronous I/O, you can try calling CancelIoEx on the file handles that you're passed in the progress routine, or CancelSynchronousIo to cancel all synchronous I/O for the thread. These I/O cancel functions were added in Windows Vista. They're not available in Windows XP, in case you're still supporting it.
import ctypes
from ctypes import wintypes
kernel32 = ctypes.WinDLL('kernel32', use_last_error=True)
COPY_FILE_FAIL_IF_EXISTS = 0x0001
COPY_FILE_RESTARTABLE = 0x0002
COPY_FILE_OPEN_SOURCE_FOR_WRITE = 0x0004
COPY_FILE_ALLOW_DECRYPTED_DESTINATION = 0x0008
COPY_FILE_COPY_SYMLINK = 0x0800
COPY_FILE_NO_BUFFERING = 0x1000
CALLBACK_CHUNK_FINISHED = 0
CALLBACK_STREAM_SWITCH = 1
PROGRESS_CONTINUE = 0
PROGRESS_CANCEL = 1
PROGRESS_STOP = 2
PROGRESS_QUIET = 3
ERROR_REQUEST_ABORTED = 0x04D3
if not hasattr(wintypes, 'LPBOOL'):
wintypes.LPBOOL = ctypes.POINTER(wintypes.BOOL)
def _check_bool(result, func, args):
if not result:
raise ctypes.WinError(ctypes.get_last_error())
return args
LPPROGRESS_ROUTINE = ctypes.WINFUNCTYPE(
wintypes.DWORD, # _Retval_
wintypes.LARGE_INTEGER, # _In_ TotalFileSize
wintypes.LARGE_INTEGER, # _In_ TotalBytesTransferred
wintypes.LARGE_INTEGER, # _In_ StreamSize
wintypes.LARGE_INTEGER, # _In_ StreamBytesTransferred
wintypes.DWORD, # _In_ dwStreamNumber
wintypes.DWORD, # _In_ dwCallbackReason
wintypes.HANDLE, # _In_ hSourceFile
wintypes.HANDLE, # _In_ hDestinationFile
wintypes.LPVOID) # _In_opt_ lpData
kernel32.CopyFileExW.errcheck = _check_bool
kernel32.CopyFileExW.argtypes = (
wintypes.LPCWSTR, # _In_ lpExistingFileName
wintypes.LPCWSTR, # _In_ lpNewFileName
LPPROGRESS_ROUTINE, # _In_opt_ lpProgressRoutine
wintypes.LPVOID, # _In_opt_ lpData
wintypes.LPBOOL, # _In_opt_ pbCancel
wintypes.DWORD) # _In_ dwCopyFlags
@LPPROGRESS_ROUTINE
def debug_progress(tsize, ttrnsfr, stsize, sttrnsfr, stnum, reason,
hsrc, hdst, data):
print('ttrnsfr: %d, stnum: %d, stsize: %d, sttrnsfr: %d, reason: %d' %
(ttrnsfr, stnum, stsize, sttrnsfr, reason))
return PROGRESS_CONTINUE
def copy_file(src, dst, cancel=None, flags=0,
cbprogress=None, data=None):
if isinstance(cancel, int):
cancel = ctypes.byref(wintypes.BOOL(cancel))
elif cancel is not None:
cancel = ctypes.byref(cancel)
if cbprogress is None:
cbprogress = LPPROGRESS_ROUTINE()
kernel32.CopyFileExW(src, dst, cbprogress, data, cancel, flags)
Example
if __name__ == '__main__':
import os
import tempfile
import threading
src_fd, src = tempfile.mkstemp()
os.write(src_fd, os.urandom(16 * 2 ** 20))
os.close(src_fd)
dst = tempfile.mktemp()
cancel = wintypes.BOOL(False)
t = threading.Timer(0.001, type(cancel).value.__set__, (cancel, True))
t.start()
try:
copy_file(src, dst, cancel, cbprogress=debug_progress)
except OSError as e:
print(e)
assert e.winerror == ERROR_REQUEST_ABORTED
finally:
if os.path.exists(src):
os.remove(src)
if os.path.exists(dst):
os.remove(dst)
