How to implement common bash idioms in Python? [cl

Question

I currently do my textfile manipulation through a bunch of badly remembered AWK, sed, Bash and a tiny bit of Perl.

I've seen mentioned a few places that python is good for this kind of thing. How can I use Python to replace shell scripting, AWK, sed and friends?

Answer 1

I have built semi-long shell scripts (300-500 lines) and Python code which does similar functionality. When many external commands are being executed, I find the shell is easier to use. Perl is also a good option when there is lots of text manipulation.

Answer 2

As of 2015 and Python 3.4's release, there's now a reasonably complete user-interactive shell available at: http://xon.sh/ or https://github.com/scopatz/xonsh

The demonstration video does not show pipes being used, but they ARE supported when in the default shell mode.

Xonsh ('conch') tries very hard to emulate bash, so things you've already gained muscle memory for, like

env | uniq | sort -r | grep PATH

or

my-web-server 2>&1 | my-log-sorter

will still work fine.

The tutorial is quite lengthy and seems to cover a significant amount of the functionality someone would generally expect at a ash or bash prompt:

• Compiles, Evaluates, & Executes!
• Command History and Tab Completion
• Help & Superhelp with ? & ??
• Aliases & Customized Prompts
• Executes Commands and/or *.xsh Scripts which can also be imported
• Environment Variables including Lookup with ${}
• Input/Output Redirection and Combining
• Background Jobs & Job Control
• Nesting Subprocesses, Pipes, and Coprocesses
• Subprocess-mode when a command exists, Python-mode otherwise
• Captured Subprocess with $(), Uncaptured Subprocess with $[], Python Evaluation with @()
• Filename Globbing with * or Regular Expression Filename Globbing with Backticks

Answer 3

Yes, of course :)

Take a look at these libraries which help you Never write shell scripts again (Plumbum's motto).

• Plumbum
• Sarge
• sh

Also, if you want to replace awk, sed and grep with something Python based then I recommend pyp -

"The Pyed Piper", or pyp, is a linux command line text manipulation tool similar to awk or sed, but which uses standard python string and list methods as well as custom functions evolved to generate fast results in an intense production environment.

Answer 4

I suggest the awesome online book Dive Into Python. It's how I learned the language originally.

Beyond teaching you the basic structure of the language, and a whole lot of useful data structures, it has a good chapter on file handling and subsequent chapters on regular expressions and more.

Answer 5

Your best bet is a tool that is specifically geared towards your problem. If it's processing text files, then Sed, Awk and Perl are the top contenders. Python is a general-purpose dynamic language. As with any general purpose language, there's support for file-manipulation, but that isn't what it's core purpose is. I would consider Python or Ruby if I had a requirement for a dynamic language in particular.

In short, learn Sed and Awk really well, plus all the other goodies that come with your flavour of *nix (All the Bash built-ins, grep, tr and so forth). If it's text file processing you're interested in, you're already using the right stuff.