I'm getting type mismatch error for below scenario
//scores sample file
0000006,Danny,6.2
0000002,Danny,7.0
Code.
case class Person(id: String, dob: String, country: String, score: Double)
def getResultRecords(persons: RDD[List[String]]): RDD[Person] = {
val personUS = persons.map(rec => Person(rec.head, rec(1),"US"), java.lang.Double.parseDouble(cpg.getOrElse(rec.head,"0").toString())
val personMX = persons.map(rec => Person(rec.head, rec(1),"MX"),java.lang.Double.parseDouble(cpg.getOrElse(rec.head,"0").toString()))
val personAll = personUS.union(personMX)
//return personAll
}//Getting an error saying " type mismach found : Unit expected : RDD[Person]"
What am I doing wrong here? How can I fix it?
Please help me.
YOur method signature is def getResultRecords(persons: RDD[List[String]]): RDD[Person]; that means the method should return an instance of RDD[Person].
However, the last line of that same method says - val personAll = personUS.union(personMX) - which returns nothing.
Uncomment return personAll line - and this will erase your exception.
Also, I found some issue with your syntax.
Here's is your code snippet.
case class Person(id: String, dob: String, country: String, score: Double)
def getResultRecords(persons: RDD[List[String]]): RDD[Person] = {
val personUS = persons.map(rec => Person(rec.head, rec(1),"US", java.lang.Double.parseDouble(cpg.getOrElse(rec.head,"0").toString()))
val personMX = persons.map(rec => Person(rec.head, rec(1),"MX",java.lang.Double.parseDouble(cpg.getOrElse(rec.head,"0").toString()))
val personAll = personUS.union(personMX)
return personAll
}
An extra parenthesis ) is given here - Person(rec.head, rec(1),"US".
And return statement was commented out.
In Spark, transformations are lazy. You need to execute an action returning an RDD[Person]. Otherwise, your function won't return anything. That's why the Type Mismatch error.
