Split JSON into multiple files

Question

I have json file exported from mongodb which looks like:

{"_id":"99919","city":"THORNE BAY"}
{"_id":"99921","city":"CRAIG"}
{"_id":"99922","city":"HYDABURG"}
{"_id":"99923","city":"HYDER"}

there are about 30000 lines, I want to split each line into it's own .json file. (I'm trying to transfer my data onto couchbase cluster)

I tried doing this:

cat cities.json | jq -c -M '.' | \
while read line; do echo $line > .chunks/cities_$(date +%s%N).json; done

but I found that it seems to drop loads of line and the output of running this command only gave me 50 odd files when I was expecting 30000 odd!!

Is there a logical way to make this not drop any data using anything that would suite?

Answer 1

Assuming you don't care about the exact filenames, if you want to split input into multiple files, just use split.

jq -c . < cities.json | split -l 1 --additional-suffix=.json - .chunks/cities_

Answer 2

In general to split any text file into separate files per-line using any awk on any UNIX system is simply:

awk '{close(f); f=".chunks/cities_"NR".json"; print > f}' cities.json