Android: Parsing large JSON file

Question

I'm creating an android application which should parse a Json from a file or url to a jsonarray and jsonobjects. The problem is that my json is 3.3 mb and when i use a simple code, something like this: (can't acces my real code now because im at work, copied some code from tutorial; so there might be some errors in it)

(assuming i already have my inputstream content)

InputStream content = entity.getContent();
                BufferedReader reader = new BufferedReader(new InputStreamReader(content));
                String line;
                while ((line = reader.readLine()) != null) {
                    builder.append(line);
String twitterfeed = builder.toString();
}

JSONArray jsonArray = new JSONArray(twittefeed);
            Log.i(ParseJSON.class.getName(),
                    "Number of entries " + jsonArray.length());
            for (int i = 0; i < jsonArray.length(); i++) {
                JSONObject jsonObject = jsonArray.getJSONObject(i);
                Log.i(ParseJSON.class.getName(), jsonObject.getString("text"));

When i run this code on my android device, i get an OutOfMemory error when parsing the string to the jsonArray. I logged some things i found that my total string is 17 mb (of a 3.3 mb json file?!) When i use a small json file, like a twitterfeed or so, the code works fine. When i got this 17 mb string in my memory i can't parse the json, because then i run out of memory.

After a lot of research i found that jackson might be my solution, because i understood that it is possible to parse an inputstream. This should help, because than i don't need the 17 mb string in my memory; and this is not the most efficient way i gues.... But i can't get it clear of this really will work, and didn't get it running myself. Does anyone know of this is really will work, and where i can find a tutorial?

I found the "createJsonParser -- public JsonParser createJsonParser(InputStream in)" and think this is my way to go... but i don't know how to implement this in my code, and can't find an example. Does anyone know how this works?

Answer 1

You should use json streaming either with gson or jackson. With Jackson you can use a hybrid approach as well. This would reduce your memory consumption significantly as only the portion of json being parsed is loaded into memory.

https://sites.google.com/site/gson/gson-user-guide http://jackson.codehaus.org/

A jackson example http://www.mkyong.com/java/jackson-streaming-api-to-read-and-write-json/

Answer 2

The hybrid streaming approach with Jackson is a good option. That way, you can advance a pointer through your raw JSON input (validating it as you go) and, when you detect an input chunk that needs to be processed, you can read it into a tree-hierarchy so that you can pull out any data you want from it.