How can I shuffle the lines of a text file on the

Question

I want to shuffle the lines of a text file randomly and create a new file. The file may have several thousands of lines.

How can I do that with cat, awk, cut, etc?

Answer 1

A simple and intuitive way would be to use shuf.

Example:

Assume words.txt as:

the
an
linux
ubuntu
life
good
breeze

To shuffle the lines, do:

$ shuf words.txt

which would throws the shuffled lines to standard output; So, you've to pipe it to an output file like:

$ shuf words.txt > shuffled_words.txt

One such shuffle run could yield:

breeze
the
linux
an
ubuntu
good
life

Answer 2

If you have Scala installed, here's a one-liner to shuffle the input:

ls -1 | scala -e 'for (l <- util.Random.shuffle(io.Source.stdin.getLines.toList)) println(l)'

Answer 3

One liner for Python based on scai's answer, but a) takes stdin, b) makes the result repeatable with seed, c) picks out only 200 of all lines.

$ cat file | python -c "import random, sys; 
  random.seed(100); print ''.join(random.sample(sys.stdin.readlines(), 200))," \
  > 200lines.txt

Answer 4

Simple awk-based function will do the job:

shuffle() { 
    awk 'BEGIN{srand();} {printf "%06d %s\n", rand()*1000000, $0;}' | sort -n | cut -c8-
}

usage:

any_command | shuffle

This should work on almost any UNIX. Tested on Linux, Solaris and HP-UX.

Update:

Note, that leading zeros (%06d) and rand() multiplication makes it to work properly also on systems where sort does not understand numbers. It can be sorted via lexicographical order (a.k.a. normal string compare).

Answer 5

A one-liner for python:

python -c "import random, sys; lines = open(sys.argv[1]).readlines(); random.shuffle(lines); print ''.join(lines)," myFile

And for printing just a single random line:

python -c "import random, sys; print random.choice(open(sys.argv[1]).readlines())," myFile

But see this post for the drawbacks of python's random.shuffle(). It won't work well with many (more than 2080) elements.

Answer 6

In windows You may try this batch file to help you to shuffle your data.txt, The usage of the batch code is

C:\> type list.txt | shuffle.bat > maclist_temp.txt

After issuing this command, maclist_temp.txt will contain a randomized list of lines.

Hope this helps.