Scale Matrix in Scala/Spark

问题:

I had the following list

id1, column_index1, value1
id2, column_index2, value2
...

which I transformed to a indexed row matrix doing the following:

val data_mapped = data.map({ case (id, col, score) => (id, (col, score))})
val data_mapped_grouped = data_mapped.groupByKey
val indexed_rows = data_mapped_grouped.map({ case (id, vals) => IndexedRow(id, Vectors.sparse(nCols.value, vals.toSeq))})
val mat = new IndexedRowMatrix(indexed_rows)

I want to perform some preprocessing on this matrix: remove the sum of the columns from each column, standardize each column by its variance. I did try to use the built-in standard scaler

val scaler = new StandardScaler().fit(indexed_rows.map(x => x.features))

but this doesn't seem to be possible with IndexedRow type

thanks for your help!

回答1:

According to what I understood from your question, here is what you'll need to do to perform StandardScaler fit on your IndexedRow

import org.apache.spark.mllib.feature.{StandardScaler, StandardScalerModel}
import org.apache.spark.mllib.linalg.distributed.IndexedRow
import org.apache.spark.mllib.linalg.{Vector, Vectors}
import org.apache.spark.rdd.RDD

val data: RDD[(Int, Int, Double)] = ???

object nCol {
  val value: Int = ???
}

val data_mapped: RDD[(Int, (Int, Double))] = 
    data.map({ case (id, col, score) => (id, (col, score)) })
val data_mapped_grouped: RDD[(Int, Iterable[(Int, Double)])] = 
    data_mapped.groupByKey

val indexed_rows: RDD[IndexedRow] = data_mapped_grouped.map { 
       case (id, vals) => 
       IndexedRow(id, Vectors.sparse(nCol.value, vals.toSeq)) 
}

You can get your vectors from your IndexedRow with a simple map

val vectors: RDD[Vector] = indexed_rows.map { case i: IndexedRow => i.vector }

Now that you have an RDD[Vector] you can try to fit it with your scaler.

val scaler: StandardScalerModel = new StandardScaler().fit(vectors)

I hope this helps!