Why do people write the #!/usr/bin/env python sheb

Question

It seems to me like the files run the same without that line.

Answer 1

Expanding a bit on the other answers, here's a little example of how your command line scripts can get into trouble by incautious use of /usr/bin/env shebang lines:

$ /usr/local/bin/python -V
Python 2.6.4
$ /usr/bin/python -V
Python 2.5.1
$ cat my_script.py 
#!/usr/bin/env python
import json
print "hello, json"
$ PATH=/usr/local/bin:/usr/bin
$ ./my_script.py 
hello, json
$ PATH=/usr/bin:/usr/local/bin
$ ./my_script.py 
Traceback (most recent call last):
  File "./my_script.py", line 2, in <module>
    import json
ImportError: No module named json

The json module doesn't exist in Python 2.5.

One way to guard against that kind of problem is to use the versioned python command names that are typically installed with most Pythons:

$ cat my_script.py 
#!/usr/bin/env python2.6
import json
print "hello, json"

If you just need to distinguish between Python 2.x and Python 3.x, recent releases of Python 3 also provide a python3 name:

$ cat my_script.py 
#!/usr/bin/env python3
import json
print("hello, json")

Answer 2

It tells the interpreter which version of python to run the program with when you have multiple versions of python.

Answer 3

It seems to me like the files run the same without that line.

If so, then perhaps you're running the Python program on Windows? Windows doesn't use that line—instead, it uses the file-name extension to run the program associated with the file extension.

However in 2011, a "Python launcher" was developed which (to some degree) mimics this Linux behaviour for Windows. This is limited just to choosing which Python interpreter is run — e.g. to select between Python 2 and Python 3 on a system where both are installed. The launcher is optionally installed as py.exe by Python installation, and can be associated with .py files so that the launcher will check that line and in turn launch the specified Python interpreter version.

Answer 4

In order to run the python script, we need to tell the shell three things:

1. That the file is a script
2. Which interpreter we want to execute the script
3. The path of said interpreter

The shebang #! accomplishes (1.). The shebang begins with a # because the # character is a comment marker in many scripting languages. The contents of the shebang line are therefore automatically ignored by the interpreter.

The env command accomplishes (2.) and (3.). To quote "grawity,"

A common use of the env command is to launch interpreters, by making use of the fact that env will search $PATH for the command it is told to launch. Since the shebang line requires an absolute path to be specified, and since the location of various interpreters (perl, bash, python) may vary a lot, it is common to use:

#!/usr/bin/env perl  instead of trying to guess whether it is /bin/perl, /usr/bin/perl, /usr/local/bin/perl, /usr/local/pkg/perl, /fileserver/usr/bin/perl, or /home/MrDaniel/usr/bin/perl on the user's system...

On the other hand, env is almost always in /usr/bin/env. (Except in cases when it isn't; some systems might use /bin/env, but that's a fairly rare occassion and only happens on non-Linux systems.)

Answer 5

The exec system call of the Linux kernel understands shebangs (#!) natively

When you do on bash:

./something

on Linux, this calls the exec system call with the path ./something.

This line of the kernel gets called on the file passed to exec: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_script.c#L25

if ((bprm->buf[0] != '#') || (bprm->buf[1] != '!'))

This reads the very first bytes of the file, and compares them to #!.

If that is true, then the rest of the line is parsed by the Linux kernel, which makes another exec call with path /usr/bin/env python and current file as the first argument:

/usr/bin/env python /path/to/script.py

and this works for any scripting language that uses # as a comment character.

And yes, you can make an infinite loop with:

printf '#!/a\n' | sudo tee /a
sudo chmod +x /a
/a

Bash recognizes the error:

-bash: /a: /a: bad interpreter: Too many levels of symbolic links

#! just happens to be human readable, but that is not required.

If the file started with different bytes, then the exec system call would use a different handler. The other most important built-in handler is for ELF executable files: https://github.com/torvalds/linux/blob/v4.8/fs/binfmt_elf.c#L1305 which checks for bytes 7f 45 4c 46 (which also happens to be human readable for .ELF). This reads the ELF file, puts it into memory correctly, and starts a new process with it. See also: How does kernel get an executable binary file running under linux?

Finally, you can add your own shebang handlers with the binfmt_misc mechanism. For example, you can add a custom handler for .jar files. This mechanism even supports handlers by file extension. Another application is to transparently run executables of a different architecture with QEMU.

I don't think POSIX specifies shebangs however: https://unix.stackexchange.com/a/346214/32558 , although it does mention in on rationale sections, and in the form "if executable scripts are supported by the system something may happen".

Answer 6

This is meant as more of historical information than a "real" answer.

Remember that back in the day you had LOTS of unix like operating systems whose designers all had their own notion of where to put stuff, and sometimes didn't include Python, Perl, Bash, or lots of other GNU/Open Source stuff at all.

This was even true of different Linux distributions. On Linux--pre-FHS[1]-you might have python in /usr/bin/ or /usr/local/bin/. Or it might not have been installed, so you built your own and put it in ~/bin

Solaris was the worst I ever worked on, partially as the transition from Berkeley Unix to System V. You could wind up with stuff in /usr/, /usr/local/, /usr/ucb, /opt/ etc. This could make for some really long paths. I have memories of the stuff from Sunfreeware.com installing each package in it's own directory, but I can't recall if it symlinked the binaries into /usr/bin or not.

Oh, and sometimes /usr/bin was on an NFS server[2].

So the env utility was developed to work around this.

Then you could write #!/bin/env interpreter and as long as the path was proper things had a reasonable chance of running. Of course, reasonable meant (for Python and Perl) that you had also set the appropriate environmental variables. For bash/ksh/zsh it just worked.

This was important because people were passing around shell scripts (like perl and python) and if you'd hard coded /usr/bin/python on your Red Hat Linux workstation it was going to break bad on a SGI...well, no, I think IRIX put python in the right spot. But on a Sparc station it might not run at all.

I miss my sparc station. But not a lot. Ok, now you've got me trolling around on E-Bay. Bastages.

[1] File-system Hierarchy Standard. https://en.wikipedia.org/wiki/Filesystem_Hierarchy_Standard

[2] Yes, and sometimes people still do stuff like that. And no, I did not wear either a turnip OR an onion on my belt.