filtering out values from a column iteratively in

问题:

shell scipt newbie here. I have a set of csv files in a folder, What I want is to select 1000 distinct user Ids from each file in an iterative way so that the next set of user ids picked from the next file does not contain the user ids filtered from the previous files. I have selected distinct 1000 user id from the first file and stored it in a temp file. Below is the command:

sort -u -t, -k1,8 file1.csv|head -1000 > temp.txt

Here 8 is the user id column. Now I want next 1000 user ids from file2 such that the user ids from file1( stored in temp.txt) are excluded from file2. Is there an elegant way to achieve this?

回答1:

-k1,8 use 1st and then 8th column. Don't you want to use just -k8? According to your question, try:

cut -d"," -f 8 file2.csv | grep -v -f temp.txt | sort -u | head -1000 > temp2.txt 

BTW you can use wildcard in sort: sort -u -t, -k8 file*.csv | head ...