How to choose a random line from a text file

Question

I am trying to make a lottery program for my school (we have an economic system).

My program generates numbers and saves it off into a text file. When I want to "pull" numbers out of my generator I want it to ensure that there is a winner.

Q: How do I have Python select a random line out of my text file and give my output as that number?

Answer 1

def random_line():
    line_num = 0
    selected_line = ''
    with open(filename) as f:
        while 1:
            line = f.readline()
            if not line: break
            line_num += 1
            if random.uniform(0, line_num) < 1:
                selected_line = line
    return selected_line.strip()

Although most of the approaches given here would work, but they tend to load the whole file in the memory at once. But not this approach. So even if the files are big, this would work.

The approach is not very intuitive at first glance. The theorem behind this states that when we have seen N lines in there is a probability of exactly 1/N that each of them is selected so far.

From Page no 123 of 'Python Cookbook'

Answer 2

If the file is very large - you could seek to a random location in the file given the file size and then get the next full line:

import os, random 
def get_random_line(file_name):
    total_bytes = os.stat(file_name).st_size 
    random_point = random.randint(0, total_bytes)
    file = open(file_name)
    file.seek(random_point)
    file.readline() # skip this line to clear the partial line
    return file.readline()

Answer 3

another approach:

import random, fileinput

text = None
for line in fileinput.input('data.txt'):
    if random.randrange(fileinput.lineno()) == 0:
        text = line
print text

Distribution:

$ seq 1 10 > data.txt

# run for 100000 times
$ ./select.py > out.txt

$ wc -l out.txt 
100000 out.txt

$ sort out.txt | uniq -c
  10066 1
  10004 10
  10023 2
   9979 3
   9926 4
   9936 5
   9878 6
  10023 7
  10154 8
  10011 9

I don't see the skewnes but perhaps the dataset is too small...

Answer 4

How do I have python select a random line out of my text file and give my output as that number?

Assuming the file is relatively small, the following is perhaps the easiest way to do it:

import random
line = random.choice(open('data.txt').readlines())

Answer 5

With a slight modification to your input file (store the number of items in the first line), you can choose a number uniformly without having to read the entire file into memory first.

import random
def choose_number( frame ):
    with open(fname, "r") as f:
        count = int(f.readline().strip())
        for line in f:
            if not random.randrange(0, count):
                return int(line.strip())
            count-=1

Say you have 100 numbers. The probability of choosing the first number is 1/100. The probability of choosing the second number is (99/100)(1/99) = 1/100. The probability of choosing the third number is (99/100)(98/99)(1/98) = 1/100. I'll skip the formal proof, but the odds of choosing any of the 100 numbers is 1/100.

It's not strictly necessary to store the count in the first line, but it saves you the trouble of having to read the entire file just to count the lines. Either way, you don't need to store the entire file in memory to choose any single line with equal probability.

Answer 6

I saw a python tutorials and found this snippet:

def randomLine(filename):
#Retrieve a  random line from a file, reading through the file once
        fh = open("KEEP-IMPORANT.txt", "r")
        lineNum = 0
        it = ''

        while 1:
                aLine = fh.readline()
                lineNum = lineNum + 1
                if aLine != "":
                        #
                        # How likely is it that this is the last line of the file ? 
                        if random.uniform(0,lineNum)<1:
                                it = aLine
                else:
                        break
        nmsg=it
        return nmsg
        #this is suposed to be a var pull = randomLine(filename)