I'm developing a script which takes the difference between 2 csv files and makes a new csv file as output with the differences BUT only if the same 2 rows (refers to row number) between the two input files contain different data e.g. row 3 has "mike", "basketball player" in file 1 and row 3 in file 2 has "mike", "baseball player". The output csv would grab these print them and write them to a csv. It works but there are some issues (I know that this question has also been asked several times before but others have done it differently to me and since I'm fairly new to programming I don't quite understand their codes).
The output in the new csv file has each letter of the output in each cell (see image below) and I believe its something to do with the delimiter/quotechar/quoting line 37. I want them in their own cells without any fullstops, multiple spaces, commas or "|".
Another issue is that it takes a long time to run. I'm working with datasets of up to 50,000 rows and it can take over an hour to run. Why is this and what advice would be useful to speed it up? Put something outside of the for loop maybe? I did try the difflib method earlier on but I was only able to print the entire "input_file1" but not compare that file with another.
# aim of script is to compare csv files and output difference as a new csv
# import necessary libraries
import csv
# File1 = open(raw_input("path:"),"r") #filename, mode
# File2 = open(raw_input("path:"),"r") #filename, mode
# selects the 2 input files to be compared
input_file1 = "G:/savestuffhereqwerty/electorate_meshblocks/teststuff/Book1.csv"
input_file2 = "G:/savestuffhereqwerty/electorate_meshblocks/teststuff/Book2.csv"
# creates the blank output csv file
output_path = "G:/savestuffhereqwerty/electorate_meshblocks/outputs/output2.csv"
a = open(input_file1, "r")
output_file = open(output_path,"w")
output_file.close()
count = 0
with open(input_file1) as fp1:
for row_number1, row_value1 in enumerate(fp1):
if row_number1 == count:
print "got to 1st point"
value1 = row_value1
with open(input_file2) as fp2:
for row_number2, row_value2 in enumerate(fp2):
if row_number2 == count:
print "got to 2nd point"
value2 = row_value2
if value1 == value2:
print value1, value2
else:
print value1, value2
with open(output_path, 'wb') as f:
writer = csv.writer(f, delimiter=',', quotechar='|', quoting=csv.QUOTE_MINIMAL)
# testing to see if the code writes text to the csv
writer.writerow(["test1"])
writer.writerow(["test2", "test3", "test4"])
writer.writerows([value1, value2])
print "code reached writing stage"
count += 1
print count
print "done"
# replace(",",".")