Appending two CSV files column-wise

Question

Suppose I have two CSV files called A and B in Python.

A's head looks like:

 headerNameA1,headerNameA2
 1.12412424,1
 1,1
 1,1
 1,1

B's head looks like:

 headerNameB1,headerNameB2
 1,1
 1,1
 1,1
 1,1

My objective is to take B and append it onto A so that A will then look like:

 headerNameA1,headerNameA2,headerNameB1,headerNameB2
 1,1,1.12412424,1
 1,1,1,1
 1,1,1,1
 1,1,1,1

From another question I asked, here's code that will take A and B and combine them into a C:

 import csv
 with open('A','rb') as f1, open('B','rb') as f2, open('out.csv','wb') as w:
     writer = csv.writer(w)
     r1,r2 = csv.reader(f1),csv.reader(f2)
     while True:
         try:
             writer.writerow(next(r1)+next(r2))
         except StopIteration:
             break

However, the objective of this question is just to add B onto the back of A.

This would be necessary if the size of A is such that it is too expensive to disk space to make a copy of it as file C before deleting A afterwards.

A bash solution called through os.system is acceptable

Answer 1

If you get two file handles for the same file - one in 'read' mode, one in 'update' mode (r+b), the same strategy should work.

from itertools import izip
import csv
with open('A','rb') as f1, open('B','rb') as f2, open('A','r+b') as w:
    writer = csv.writer(w)
    for r1,r2 in izip(csv.reader(f1),csv.reader(f2)):
        writer.writerow(r1+r2)

When possible I'd recommend against this kind of thing and just explicitly write to a third file.

Answer 2

You might be able to get away with a named pipe. You have a Python process run which creates a pipe and opens it in write mode. It then outputs to that the column wise concatenation of the CSV files (similar to what you've got) already... When another process starts reading that file, it'll be able to consume the data, but no file is actually stored on the server, it's just on demand. When the "file" is consumed, then there'll be nothing in it, and any attempt to access it will block until another process writes to the other end.

Some dummy code - will need more thought out exception handling etc...:

import os
from itertools import izip

a = 'abcdef' # File A's rows
b = 'ghijkl' # File B's rows

outname = 'joined'

try:
    os.unlink(outname)
    os.mkfifo(outname)
except OSError:
    pass

with open(outname, 'w') as fout:
    for items in izip(a, b):
        fout.write(''.join(items) + '\n') # Do "real" write here instead...
    os.unlink(outname)

Something else opens that "file" in read mode and consumes it to retrieve the data. This should work unless that process has to have "physical files"...