'^M' character at end of lines

Question

When I run a particular SQL script in Unix environments, I'm am seeing a '^M' character at the end of each line of the SQL script as it is echoed to the command-line. I don't know on which OS the SQL script was originally created.

What is causing this and how do I fix it?

Answer 1

To replace ^M characters in vi editor use below

open the text file say t1.txt

vi t1.txt

Enter command mode by pressing shift + :

then press keys as mentioned %s/^M/\r/g

in above ^M is not (shift + 6)M instead it is (ctrl + V)(ctrl + M)

Answer 2

An alternative to dos2unix command would be using standard utilities like sed.

For example, dos to unix:

sed 's/\r$//' dos.txt > unix.txt

unix to dos:

sed 's/$/\r/' unix.txt > dos.txt

Answer 3

The cause is the difference between how a Windows-based based OS and a Unix based OS store the end-of-line markers.

Windows based operating systems, thanks to their DOS heritage, store an end-of-line as a pair of characters - 0x0D0A (carriage return + line feed). Unix-based operating systems just use 0x0A (a line feed). The ^M you're seeing is a visual representation of 0x0D (a carriage return).

dos2unix will help with this. You probably also need to adjust the source of the scripts to be 'Unix-friendly'.

Answer 4

Try using dos2unix to strip off the ^M.

Answer 5

od -a $file is useful to explore those types of question on Linux (similar to dumphex in the above).

Answer 6

C:\tmp\text>dos2unix hello.txt helloUNIX.txt

Sed is even more widely available and can do this kind of thing also if dos2unix is not installed

C:\tmp\text>sed s/\r// hello.txt > helloUNIX.txt  

You could also try tr:

cat hello.txt | tr -d \r > helloUNIX2.txt  

Here are the results:

C:\tmp\text>dumphex hello.txt  
00000000h: 48 61 68 61 0D 0A 68 61 68 61 0D 0A 68 61 68 61 Haha..haha..haha  
00000010h: 0D 0A 0D 0A 68 61 68 61 0D 0A                   ....haha..  

C:\tmp\text>dumphex helloUNIX.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.  

C:\tmp\text>dumphex helloUNIX2.txt  
00000000h: 48 61 68 61 0A 68 61 68 61 0A 68 61 68 61 0A 0A Haha.haha.haha..  
00000010h: 68 61 68 61 0A                                  haha.