Spark GraphX - How can I read from a JSON file in

Question

I'm new to Spark and Scala, and I am trying to read a bunch of tweeter data from a JSON file and turn that into a graph where a vertex represents a tweet and the edge connects to tweets which are a re-tweet of the original posted item. So far I have managed to read from the JSON file and figure out the Schema of my RDD. Now I believe I need to somehow take the data from the SchemaRDD object and create an RDD for the Vertices and an RDD for the edges. Is this the way to approach this or is there an alternative solution? Any help and suggestions would be highly appreciated.

Answer 1

This really depends on your json file. You need to parse the data from the json file and create your vertices and edges based on the parsed data. There isnt a certain way to implement this, its really up to the programmer. One approach is to create a vertices array and edges array (again based on the parsed data) and parallelize those (create VertexRDD and EdgeRDD), and then create the graph you need. Hope I helped.