How to convert arbitrary JSON into a usable struct

Question

I'm trying to use gson to convert this returned JSON into some kind of data structure such that I can extract useful data.

For Example:

http://search.twitter.com/search.json?q=test&rpp=1

Returns:

{
    "completed_in":0.028,
    "max_id":196386333906837504,
    "max_id_str":"196386333906837504",
    "next_page":"?page=2&max_id=196386333906837504&q=test&rpp=1",
    "page":1,
    "query":"test",
    "refresh_url":"?since_id=196386333906837504&q=test",
       "results":[
          {
             "created_at":"Sat, 28 Apr 2012 23:52:05 +0000",
             "from_user":"della_ky",
             "from_user_id":525641596,
             "from_user_id_str":"525641596",
             "from_user_name":"kydella modeste",
             "geo":null,
             "id":196386333906837504,
             "id_str":"196386333906837504",
             "iso_language_code":"en",
             "metadata":{
                "result_type":"recent"
             },
             "profile_image_url":"http:\/\/a0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
             "profile_image_url_https":"https:\/\/si0.twimg.com\/profile_images\/2159990525\/webcam-toy-photo3_20_2__normal.jpg",
             "source":"<a href="http:\/\/mobile.twitter.com" rel="nofollow">Mobile Web<\/a>",
             "text":"RT @Y__U__NOOO: #SongsIKnowOffByHeart ALL SONGS I LISTEN TO. BRAIN, Y U NO REMEMBER TEST ANSWERS LIKE THAT?!?",
             "to_user":null,
             "to_user_id":null,
             "to_user_id_str":null,
             "to_user_name":null
          }
       ],
       "results_per_page":1,
       "since_id":0,
       "since_id_str":"0"
    }

Ultimately, I would like to be able to output a list of tweets with the name of the sender and the date/time of the tweet.

I have read through the gson documentation but it's going over my head to be honest - lots of new concepts there for me.

Do I need to define a class which maps exactly to the structure of the JSON in order to then populate an instance of that class? If so this seems very inflexible/laborious. Ideally I'm looking for something which will handle JSON in any form and give me a structure I can use automatically...

Is anyone able to give me some pointers? Being new to this - the more detailed and in words of the fewest syllables the better!

Update - Thanks to the responses I've already had on this I've had a go at putting a class together to capture the twitter JSON. However, since the JSON has an embedded ArrayList of Objects I'm struggling a bit... So far I have

public class tweetData {
    private double completed_in;
    private long max_id;
    private long max_id_str;
    private String next_page;
    private int page;
    private String query;
    private String refresh_url;
    private List<tweetDetails> tweets = new ArrayList<tweetDetails>();
}

and

public class tweetDetails {
    private String created_at;
    private String from_user;
    private long from_user_id;
    private long from_user_id_str;
    private String from_user_name;
    private String geo;
    private long id;
    private long id_str;
    private String iso_language_code;
//  "metadata":
//  {
//  "result_type":"recent"
//  },
    private String profile_image_url;
    private String profile_image_url_https;
    private String source;
    private String text;
    private String to_user;
    private String to_user_id;
    private String to_user_id_str;
    private String to_user_name;
}

Which I'm instantiating with

URI uri = new URI("http", "search.twitter.com", "/search.json", "q="+ searchTerms + "&rrp=" + RRP, null);
URL twitterSearch = uri.toURL();
URLConnection yc = twitterSearch.openConnection();
JsonReader reader = new JsonReader(new InputStreamReader(yc.getInputStream()));
Gson gson = new Gson();
tweetData data = gson.fromJson(reader, tweetData.class);
System.out.println(data);

The basic name:values are being populated correctly but the ArrayList is not.

tweetData : 0.17196614959919140865196614959919140865?page=2&max_id=196614959919140865&q=test1test?since_id=196614959919140865&q=testSIZE 0[]

So, I'm still struggling a bit - any more tips hugely appreciated!

Tia, Tom

Answer 1

Gson is a slick beast! Or at least it became so over the years that have passed since the question had been asked.

You can pass it an Object.class as a second parameter to the fromJson() method and it will parse your Json into a reasonable structure of LinkedTreeMaps and ArrayLists.

Object result = (new Gson()).fromJson(jsonString, Object.class)

More than that, you can really do partial parsing and leave loose ends at any level of your object structure by defining a certain field as Object!

Gson will then parse Json into your structure and your field of type Object will contain the above mentioned structure of LinkedTreeMaps and ArrayLists.

E.g., you may define a class

Person {
    String name;
    Object details;
}

(Imagine, you care mostly about the person's name but may want the details also somewhere. To log them, for instance.)

Then you can pass the following Json to the fromJson(input, Person.class) method as a first parameter

{ 
    "name": "Carlsson", 
    "details": {
        "address": "Stockholm",
        "phones": [
            "work": "233-322-233-322",
            "home": "none"
        ]
    }
}

The result will have the name field filled with "Carlsson" string and details field will contain a LinkedTreeMap with keys "address" and "phones", etc.

Answer 2

Do I need to define a class which maps exactly to the structure of the JSON in order to then populate an instance of that class? If so this seems very inflexible/laborious.

Yes. GSON is a library that can be used to convert Java Objects into their JSON representation. It can also be used to convert a JSON string to an equivalent Java object. This is really powerful because you can automagically instantiate your Java objects from the JSON representation. Assuming your JSON doesn't change its structure, you only have to define the appropriate Java object representation once.

Ideally I'm looking for something which will handle JSON in any form and give me a structure I can use automatically...

However, if you don't want automagical serialisation/deserialisation, then try looking at a simpler library such as java.net/projects/jsonp.

You can extract stuff from it just by querying the keys:

final JSONObject json = new JSONObject(theJsonString);
final String id = json.getString("max_id");
final JSONArray results = json.getJSONArray("results");
final String user = results.getJSONObject(2).getString("from_user");

Answer 3

Gson actually does all the serialization for you. So yes, you would have to write the classes yourself. To you, this seams inflexible and laborious, but that's only because that library isn't made for what you're asking for (it doesn't parse 'arbitrary' JSON).

I would suggest at least considering writing the classes and using gson. The reason I say that is because either way your application's logic will have to expect a very specific format, and writing out that format in a Java class will make things tidier. Here's a nice guide that will help you get started that way.

If you want to simply decode the JSON without serializing it into a Java class (IMHO the only way to use 'arbitrary' JSON), you'll want to use another library. Try this one. It allows you to decode the JSON, and use it by getting values from it (as described in this question: Convert a JSON string to object in Java ME?).

Answer 4

There are some tools that do gson to schema mapping. You give some sample JSON responses, and the java classes to access them are created for you.

http://www.jsonschema2pojo.org/