Want to print maxm entry of every email against it

问题:

I have a huge File around 2 GB having more then 20million rows

what i want is

Input File will be like this

[e-mail],1
[e-mail],1
[e-mail],1
[e-mail],2
[e-mail],3
[e-mail],4
[e-mail],5
[e-mail],6
[e-mail],7
[e-mail],1
[e-mail],1
[e-mail],1
[e-mail],2
[e-mail],3
[e-mail],1
[e-mail],2
[e-mail],1
[e-mail],1
[e-mail],2
[e-mail],3
[e-mail],1
[e-mail],1
[e-mail],2
[e-mail],3
[e-mail],1
[e-mail],2
[e-mail],1
[e-mail],1
[e-mail],1
[e-mail],1,1
[e-mail],2
[e-mail],3
[e-mail],4
[e-mail],5
[e-mail],6
[e-mail],7
[e-mail],1
[e-mail],2
[e-mail],3
[e-mail],4

Output File Needed:-

[e-mail],1,1
[e-mail],1,1
[e-mail],1,7
[e-mail],2,7
[e-mail],3,7
[e-mail],4,7
[e-mail],5,7
[e-mail],6,7
[e-mail],7,7
[e-mail],1,1
[e-mail],1,1
[e-mail],1,3
[e-mail],2,3
[e-mail],3,3
[e-mail],1,2
[e-mail],2,2
[e-mail],1,1
[e-mail],1,3
[e-mail],2,3
[e-mail],3,3
[e-mail],1,1
[e-mail],1,3
[e-mail],2,3
[e-mail],3,3
[e-mail],1,2
[e-mail],2,2
[e-mail],1,1
[e-mail],1,1
[e-mail],1,1
[e-mail],1,7
[e-mail],2,7
[e-mail],3,7
[e-mail],4,7
[e-mail],5,7
[e-mail],6,7
[e-mail],7,7
[e-mail],1,4
[e-mail],2,4
[e-mail],3,4
[e-mail],4,4

i,e 1 more column containing maximum no of entries corresponding to a particular email in each column so that every row now contains maximum occurence of each email. I am looking for a feasible soln for such a large file preferably in python or shell script and complexity of O(n) or O(nlogn) O(n**2) wont do in this case

回答1:

Lets try a python script since you may be more familiar with that language, doesn't require huge memory or hard disk space. Tested on Python 2.7 and 3.2

#!/usr/bin/python
email = "" # Initialize the email
count = 0  # and counter
import fileinput

for line in fileinput.input("word.txt"): # Interator: process a line at a time
  myArr = line.split(",")
  if (email != myArr[0]): # New email; print and reset count, email
    for n in range(0,count):
      print email + "," + str(n+1) + "," + str(count)
    email = myArr[0]
    count = 1  
  else: # Same email, increment count
    count = count + 1

# Print the final email
for n in range(0,count):
  print email + "," + str(n+1) + "," + str(count)

Anyone want to try an awk script?

回答2:

I understand that you want to have a third column containing the maximun value of the second column for each mail.

I that case, I would use a map to store the maximun second column value found for each mail:

Pseudo code:

1. Create a empty map where mail strings are the keys set -> M
2. For each line (l) of the input file:
   1. if (l.mail not in M) OR (l.mail in M AND l.secondColumn > M[l.mail].secondColumn) Then: l.thirdColumn = l.secondColunn AND M[l.mail] = l;
3. Create a new file -> fOut
4. Iterate over the map entries (for each map entry):
   1. Append M[entry] to fOut.