Read a Text File Based on key as a Column from ano

问题:

I am new to Spark I am Trying to load Table into Spark as a textFIle

I want to read the textfile based on another Text file Column eg:: Id as A key If B.id Matches A.id Then I have to read File B into Spark

val file2=sc.textFile("path")

回答1:

One way would be read both the files & then join them based on id field and select only those columns from table b, some thing like below

val df1 = Seq((1, "Anu"),(2, "Suresh"),(3, "Usha"), (4, "Nisha")).toDF("id","name")
val df2 = Seq((1, 23),(2, 24),(3, 24), (4, 25), (5, 30), (6, 32)).toDF("id","age")

df1.as("df1").join(df2.as("df2"), df1("id") === df2("id"), "inner").select("df2.*").show()

output:

+---+---+
| id|age|
+---+---+
|  1| 23|
|  2| 24|
|  3| 24|
|  4| 25|
+---+---+