format a (K, (v, w)) pair in spark rdd

问题:

I have an rdd like this:

val custFile = sc.textFile("custInfo.txt").map(line => line.split('|'))

val custPrd = custFile.map(a => (a(0), ((a(1)), (a(2), a(3), a(4), a(5), a(6), a(7), a(8)))))

val custGrp = custPrd.groupByKey

custGrp.saveAsTextFile("custinfo2")

that produces this:

(1104,CompactBuffer((S_SAVG,(1,1,1,1,1,1,1)), (CN_SAVG,(4,4,1,1,4,1,1))))

how can I use something like this:

custPrdGrp.map{case (k, vals) => {val valsString = vals.mkString(", "); s"{$k:, {$valsString}}" }}

to format a (k, (v, w)) pair...I tried this but got an error:

val custPrdRep = custPrdGrp.map({case (k, (v, w)) => {val valsString = v.mkString(", "); val valsPrvcy = w.mkString(", "); s"'${k}'| [$valsString]" }})
<console>:27: error: constructor cannot be instantiated to expected type;
 found   : (T1, T2)
 required: Iterable[(String, (String, String, String, String, String, String, String))]
       val custPrdRep = custPrdGrp.map({case (k, (v, w)) => {val valsString = v.mkString(", "); val valsPrvcy = w.mkString(", "); s"'${k}'| [$valsString]" }})
                                                 ^ 


<console>:27: error: not found: value v
           val custPrdRep = custPrdGrp.map({case (k, (v, w)) => {val valsString = v.mkString(", "); val valsPrvcy = w.mkString(", "); s"'${k}'| [$valsString]" }})
                                                                                  ^
    <console>:27: error: not found: value w
           val custPrdRep = custPrdGrp.map({case (k, (v, w)) => {val valsString = v.mkString(", "); val valsPrvcy = w.mkString(", "); s"'${k}'| [$valsString]" }})

I'd want the array to look like this:

('1104'|{'S_SAVG': {a: '1', b: '1', c: '1', d: '1', e: '1', f: '1', g: '1'}, 'CN_SAVG': {a: '4', b: '4', c: '1', d: '1', e: '4', f: '1', g: '1'}})

回答1:

Well, there is quite a lot of details here but something like this should work:

val keys = List("a", "b", "c", "d", "e", "f", "g")

custGrp.map{case (k, vals) => {
    val valsString = vals map {
        case (val1, val2) => {
            val pairs = keys
                // Create someLetter: 'someNumber' pairs
                .zip(val2.productIterator.map{case (x: String)  => x}.toSeq)
                .map{case (k, v) => s"$k: '$v'"}
                // Join into a single string
                .mkString(", ")
            // Add "key"
            s"'$val1': {$pairs}"
        }
    }
    // Combine above
    val valsComb = valsString.mkString(", ")
    // Create final string
    s"('$k'|{$valsComb})"
}}

You could simplify things by creating a correct data structure in the first place. For example by using Maps instead of tuples:

 Map("S_SAVG" -> Map("a" -> "1", "b" -> "1", ...), ...)