Dos2unix not working when trying to silence comman

Question

I was calling dos2unix from within Python this way:

call("dos2unix " + file1, shell=True, stdout=PIPE)

However to silence the Unix output, I did this:

f_null = open(os.devnull, 'w')
call("dos2unix " + file1, shell=True, stdout=f_null , stderr=subprocess.STDOUT)

This doesn't seem to work. The command isn't being called anymore as the diff that I perform on the file1 against file2 (did a diff -y file1 file2 | cat -t and could see the line endings hadn't changed).

file2 is the file I am comparing file1 against. It has Unix line endings as it is generated on the box. However, there is a chance that file1 doesn't.

Answer 1

Not sure, why but I would try to get rid of the "noise" around your command & check return code:

check_call(["dos2unix",file1], stdout=f_null , stderr=subprocess.STDOUT)

• pass as list of args, not command line (support for files with spaces in it!)
• remove shell=True as dos2unix isn't a built-in shell command
• use check_call so it raises an exception instead of failing silently

At any rate, it is possible that dos2unix detects that the output isn't a tty anymore and decides to dump the output in it instead (dos2unix can work from standard input & to standard output). I'd go with that explanation. You could check it by redirecting to a real file instead of os.devnull and check if the result is there.

But I would do a pure python solution instead (with a backup for safety), which is portable and doesn't need dos2unix command (so it works on Windows as well):

with open(file1,"rb") as f:
   contents = f.read().replace(b"\r\n",b"\n")
with open(file1+".bak","wb") as f:
   f.write(contents)
os.remove(file1)
os.rename(file1+".bak",file1)

reading the file fully is fast, but could choke on really big files. A line-by-line solution is also possible (still using the binary mode):

with open(file1,"rb") as fr, open(file1+".bak","wb") as fw:
   for l in fr:
      fw.write(l.replace(b"\r\n",b"\n"))
os.remove(file1)
os.rename(file1+".bak",file1)