Python append multiple files in given order to one

Question

I have up to 8 seperate Python processes creating temp files in a shared folder. Then I'd like the controlling process to append all the temp files in a certain order into one big file. What's the quickest way of doing this at an os agnostic shell level?

Answer 1

import os
str = os.listdir("./")

for i in str:
    f = open(i)
    f2 = open("temp.txt", "a")
    for line in f.readlines():
        f2.write(line)

We can use above code to read all the contents from all the file present in current directory and store into temp.txt file.

Answer 2

Use fileinput:

with open("bigfile.txt", "w") as big_file:
    with fileinput.input(files=tempfiles) as inputs:
        for line in inputs:
            big_file.write(line)

This is more memory efficient than @RafeKettler's answer as it doesn't need to read the whole file into memory before writing to big_file.

Answer 3

Simple & Efficient way to copy data from multiple files to one big file, Before that you need to rename your files to (int) eg. 1,2,3,4...etc, Code:

#Rename Files First

import os

path = 'directory_name'
files = os.listdir(path)
i = 1
for file in files:
    os.rename(os.path.join(path, file), os.path.join(path, str(i)+'.txt'))

    i = i+1

# Code For Copying Data from Multiple files

import os

i = 1
while i<50:

    filename = i
    for filename in os.listdir("directory_name"):

        # %s is your filename # .txt is file extension 
        f = open("%s.txt" % i,'r') 
        fout = open("output_filename", "a")

    for line in f:
        fout.write(line)
    i += 1

Answer 4

Not aware of any shell-level commands for appending one file to another. But appending at 'python level' is sufficiently easy that I am guessing python developers did not think it necessary to add it to the library.

The solution depends on the size and structure of the temp files you are appending. If they are all small enough that you don't mind reading each of them into memory, then the answer from Rafe Kettler (copied from his answer and repeated below) does the job with the least amount of code.

# tempfiles is an ordered list of temp files (open for reading)
f = open("bigfile.txt", "w")
for tempfile in tempfiles:
    f.write(tempfile.read())

If reading files fully into memory is not possible or not an appropriate solution, you will want to loop through each file and read them piece-wise. If your temp file contains newline-terminated lines which can be read individually into memory, you might do something like this

# tempfiles is an ordered list of temp files (open for reading)
f = open("bigfile.txt", "w")
for tempfile in tempfiles:
    for line in tempfile
        f.write(line)

Alternatively - something which will always work - you may choose a buffer size and just read the file piece-wise, e.g.

# tempfiles is an ordered list of temp files (open for reading)
f = open("bigfile.txt", "w")
for tempfile in tempfiles:
    while True:
        data = tempfile.read(65536)
        if data:
            f.write(data)
        else:
            break

The input/output tutorial has a lot of good info.

Answer 5

Just using simple file IO:

# tempfiles is a list of file handles to your temp files. Order them however you like
f = open("bigfile.txt", "w")
for tempfile in tempfiles:
    f.write(tempfile.read())

That's about as OS agnostic as it gets. It's also fairly simple, and the performance ought to be about as good as using anything else.

Answer 6

In this code, you can indicate the path and name of the input/output files, and it will create the final big file in that path:

import os

dir_name = "Your_Desired_Folder/Goes_Here"    #path
input_files_names = ["File1.txt", "File2.txt", "File3.txt"]     #input files
file_name_out = "Big_File.txt"     #choose a name for the output file
file_output = os.path.join(dir_name, file_name_out)
fout = open(file_output, "w")

for tempfile in input_files_names:
    inputfile = os.path.join(dir_name, tempfile)
    fin = open(inputfile, 'r')
    for line in fin:
        fout.write(line)

fin.close()    
fout.close()