How can I shuffle the lines of a text file on the

Question

I want to shuffle the lines of a text file randomly and create a new file. The file may have several thousands of lines.

How can I do that with cat, awk, cut, etc?

Answer 1

We have a package to do the very job:

sudo apt-get install randomize-lines

Example:

Create an ordered list of numbers, and save it to 1000.txt:

seq 1000 > 1000.txt

to shuffle it, simply use

rl 1000.txt

Answer 2

Perl one-liner would be a simple version of Maxim's solution

perl -MList::Util=shuffle -e 'print shuffle(<STDIN>);' < myfile

Answer 3

Here is a first try that's easy on the coder but hard on the CPU which prepends a random number to each line, sorts them and then strips the random number from each line. In effect, the lines are sorted randomly:

cat myfile | awk 'BEGIN{srand();}{print rand()"\t"$0}' | sort -k1 -n | cut -f2- > myfile.shuffled

Answer 4

If like me you came here to look for an alternate to shuf for macOS then use randomize-lines.

Install randomize-lines(homebrew) package, which has an rl command which has similar functionality to shuf.

brew install randomize-lines

Usage: rl [OPTION]... [FILE]...
Randomize the lines of a file (or stdin).

  -c, --count=N  select N lines from the file
  -r, --reselect lines may be selected multiple times
  -o, --output=FILE
                 send output to file
  -d, --delimiter=DELIM
                 specify line delimiter (one character)
  -0, --null     set line delimiter to null character
                 (useful with find -print0)
  -n, --line-number
                 print line number with output lines
  -q, --quiet, --silent
                 do not output any errors or warnings
  -h, --help     display this help and exit
  -V, --version  output version information and exit

Answer 5

Another awk variant:

#!/usr/bin/awk -f
# usage:
# awk -f randomize_lines.awk lines.txt
# usage after "chmod +x randomize_lines.awk":
# randomize_lines.awk lines.txt

BEGIN {
  FS = "\n";
  srand();
}

{
  lines[ rand()] = $0;
}

END {
  for( k in lines ){
    print lines[k];
  }
}

Answer 6

Ruby FTW:

ls | ruby -e 'puts STDIN.readlines.shuffle'