Random selection of columns using linux command

Question

I have a flat file (.txt) with 606,347 columns and I want to extract 50,000 RANDOM columns, with exception of the first column, which is sample identification. How can I do that using Linux commands? My file looks like:

ID  SNP1    SNP2    SNP3
1   0   0   2
2   1   0   2
3   2   0   1
4   1   1   2
5   2   1   0

It is TAB delimited.

Thank you so much.

Cheers,

Paula.

Answer 1

@karakfa 's answer is great, but the NF value can't be obtained in the BEGIN{} part of the awk script. Refer to: How to get number of fields in AWK prior to processing

I edited the code as:

head -4 10X.txt | awk '
function shuffle(a,n,k){
  for(i=1;i<=k;i++) {
    j=int(rand()*(n-i))+i
    if(j in a) a[i]=a[j]
    else a[i]=j
    a[j]=i;
  }
}
BEGIN{
  FS=" ";OFS="\t"; ncols=10;
  }NR==1{shuffle(tmp_array,NF,ncols);
    for(i=1;i<=ncols;i++){
      printf "%s", $(tmp_array[i]) OFS;
    }
    print "";
  }NR>1{
    printf "%s", $1 OFS;
    for(i=1;i<=ncols;i++){    
      printf "%s", $(tmp_array[i]+1) OFS;
    }
    print "";
    }' 

Because I am processing the single-cell gene expression profiles, so from the second row, the first column will be gene names. My output is:

D4-2_3095   D6-1_3010   D16-2i_1172 D4-1_337    iPSCs-2i_227    D4-2_170    D12-serum_1742  D4-1_1747   D10-2-2i_1373   D4-1_320    
Sox17   0   0   0   0   0   0   0   0   0   0   
Mrpl15  0.987862442831866   1.29176904082314    2.12650693025845    0   1.33257747910871    0   1.58815046312948    1.18541326956528    1.12103842107813    0.656789854017254   
Lypla1  0   1.29176904082314    0   0   0.443505832809852   0.780385141793088   0.57601629238987    0   0   0.656789854017254

Answer 2

Try this:

echo {2..606347} | tr ' ' '\n' | shuf | head -n 50000 | xargs -d '\n' | tr ' ' ',' | xargs -I {} cut -d $'\t' -f {} file

Update:

echo {2..606347} | tr ' ' '\n' | shuf | head -n 50000 | sed 's/.*/&p/' | sed -nf - <(tr '\t' '\n' <file) | tr '\n' '\t'

Answer 3

awk to the rescue!

$ cat shuffle.awk

   function shuffle(a,n,k) {
     for(i=1;i<=k;i++) {
       j=int(rand()*(n-i))+i
       if(j in a) a[i]=a[j]
       else a[i]=j
       a[j]=i;
     }
   }

   BEGIN{srand(); shuffle(ar,NF,ncols)}
        {for(i=1;i<=ncols;i++) printf "%s", $(ar[i]) FS; print ""}

general usage

$ echo $(seq 5) | awk -f shuffle.awk -v ncols=5
3 4 1 5 2

in your special case you can print $1 and start the function loop from 2.

i.e. change

for(i=1;i<=k;i++) to a[1]=1; for(i=2;i<=k;i++)