How to implement common bash idioms in Python? [cl

Question

I currently do my textfile manipulation through a bunch of badly remembered AWK, sed, Bash and a tiny bit of Perl.

I've seen mentioned a few places that python is good for this kind of thing. How can I use Python to replace shell scripting, AWK, sed and friends?

Answer 1

If your textfile manipulation usually is one-time, possibly done on the shell-prompt, you will not get anything better from python.

On the other hand, if you usually have to do the same (or similar) task over and over, and you have to write your scripts for doing that, then python is great - and you can easily create your own libraries (you can do that with shell scripts too, but it's more cumbersome).

A very simple example to get a feeling.

import popen2
stdout_text, stdin_text=popen2.popen2("your-shell-command-here")
for line in stdout_text:
  if line.startswith("#"):
    pass
  else
    jobID=int(line.split(",")[0].split()[1].lstrip("<").rstrip(">"))
    # do something with jobID

Check also sys and getopt module, they are the first you will need.

Answer 2

In the beginning there was sh, sed, and awk (and find, and grep, and...). It was good. But awk can be an odd little beast and hard to remember if you don't use it often. Then the great camel created Perl. Perl was a system administrator's dream. It was like shell scripting on steroids. Text processing, including regular expressions were just part of the language. Then it got ugly... People tried to make big applications with Perl. Now, don't get me wrong, Perl can be an application, but it can (can!) look like a mess if you're not really careful. Then there is all this flat data business. It's enough to drive a programmer nuts.

Enter Python, Ruby, et al. These are really very good general purpose languages. They support text processing, and do it well (though perhaps not as tightly entwined in the basic core of the language). But they also scale up very well, and still have nice looking code at the end of the day. They also have developed pretty hefty communities with plenty of libraries for most anything.

Now, much of the negativeness towards Perl is a matter of opinion, and certainly some people can write very clean Perl, but with this many people complaining about it being too easy to create obfuscated code, you know some grain of truth is there. The question really becomes then, are you ever going to use this language for more than simple bash script replacements. If not, learn some more Perl.. it is absolutely fantastic for that. If, on the other hand, you want a language that will grow with you as you want to do more, may I suggest Python or Ruby.

Either way, good luck!

Answer 3

One reason I love Python is that it is much better standardized than the POSIX tools. I have to double and triple check that each bit is compatible with other operating systems. A program written on a Linux system might not work the same on a BSD system of OSX. With Python, I just have to check that the target system has a sufficiently modern version of Python.

Even better, a program written in standard Python will even run on Windows!

Answer 4

• If you want to use Python as a shell, why not have a look at IPython ? It is also good to learn interactively the language.
• If you do a lot of text manipulation, and if you use Vim as a text editor, you can also directly write plugins for Vim in python. just type ":help python" in Vim and follow the instructions or have a look at this presentation. It is so easy and powerfull to write functions that you will use directly in your editor!

Answer 5

I have published a package on PyPI: ez.
Use pip install ez to install it.

It has packed common commands in shell and nicely my lib uses basically the same syntax as shell. e.g., cp(source, destination) can handle both file and folder! (wrapper of shutil.copy shutil.copytree and it decides when to use which one). Even more nicely, it can support vectorization like R!

Another example: no os.walk, use fls(path, regex) to recursively find files and filter with regular expression and it returns a list of files with or without fullpath

Final example: you can combine them to write very simply scripts:
files = fls('.','py$'); cp(files, myDir)

Definitely check it out! It has cost me hundreds of hours to write/improve it!

Answer 6

I will give here my opinion based on experience:

For shell:

• shell can very easily spawn read-only code. Write it and when you come back to it, you will never figure out what you did again. It's very easy to accomplish this.
• shell can do A LOT of text processing, splitting, etc in one line with pipes.
• it is the best glue language when it comes to integrate the call of programs in different programming languages.

For python:

• if you want portability to windows included, use python.
• python can be better when you must manipulate just more than text, such as collections of numbers. For this, I recommend python.

I usually choose bash for most of the things, but when I have something that must cross windows boundaries, I just use python.