Download large data from JSON using Volley in andr

Question

I am building an android application in which I need to parse the data from the Database through JSON using Volley. I have 17,000+ rows in the table, I need to fetch all the row from the table via JSON and I need to store it in SQLite. Once downloaded the user can modify the fields of all row. Once the fields have been modified. The whole data need to be sent back to the server(Like synchronize).

What my problem is, If I attempt to download this huge data to my app. It's getting crashed. Is there any way that I can do to make this app work. Also tell me how to parse huge data from the server using volley.

Answer 1

1- Use Gson for parsing data and store in ormlite database.
2-After the modification to database tables you should be synchronize table row bye row using sync adapter to server.
3-Make Service for synchronize to server

Answer 2

I am not sure what type of server technology you are using. Downloading 17K + data and uploading the same amount overtime is I guess not the optimum use of technology.

Try some technology which reduces your burden about data sync and parsing. I am sure you are aware about Firebase by google. that one of the solution in this case.

Let me know if I misunderstood your question in this case.

Answer 3

I think you are getting OutOfMemoryException, am i right ? Make paging for huge data, you can pull 100 by 100 or 1000 by 1000 from server,then you can store them in sqlite directly

Answer 4

I think it crashes because of timeOutError. When volley has to parse a large amount of data, you should set the timeout for your volley request. Default timeout is 2500ms, set it to 10000ms or higher it'll work for ex.

strReq.setRetryPolicy(new DefaultRetryPolicy(10000, DefaultRetryPolicy.DEFAULT_MAX_RETRIES, DefaultRetryPolicy.DEFAULT_BACKOFF_MULT));

It will not consume all the time that you allotted, but it will be disconnected once all the data is fetched

timeoutError will not give you crash, but onErrorResponse would be called, please make sure that tables that you are going to insert into were created before making insert operation.

Answer 5

Enable largheap in application manifest. Use this link to know how to enable it

I am using retrofit to download large file. Try to parse using jsonreader for large data.