How to merge 2 json file using jq?

Question

I'm using the jq tools (jq-json-processor) in shell script to parse json.

I've got 2 json files and want to merge them into one unique file

Here the content of files:

file1

{
    "value1": 200,
    "timestamp": 1382461861,
    "value": {
        "aaa": {
            "value1": "v1",
            "value2": "v2"
        },
        "bbb": {
            "value1": "v1",
            "value2": "v2"
        },
        "ccc": {
            "value1": "v1",
            "value2": "v2"
        }
    }
}

file2

{
    "status": 200,
    "timestamp": 1382461861,
    "value": {
        "aaa": {
            "value3": "v3",
            "value4": 4
        },
        "bbb": {
            "value3": "v3"
        },      
        "ddd": {
            "value3": "v3",
            "value4": 4
        }
    }
}

expected result

{
    "value": {
        "aaa": {
            "value1": "v1",
            "value2": "v2",
            "value3": "v3",
            "value4": 4
        },
        "bbb": {
            "value1": "v1",
            "value2": "v2",
            "value3": "v3"
        },
        "ccc": {
            "value1": "v1",
            "value2": "v2"
        },
        "ddd": {
            "value3": "v3",
            "value4": 4
        }
    }
}

I try a lot of combinaison but the only result i get is the following, which is not the expected result:

{
  "ccc": {
    "value2": "v2",
    "value1": "v1"
  },
  "bbb": {
    "value2": "v2",
    "value1": "v1"
  },
  "aaa": {
    "value2": "v2",
    "value1": "v1"
  }
}
{
  "ddd": {
    "value4": 4,
    "value3": "v3"
  },
  "bbb": {
    "value3": "v3"
  },
  "aaa": {
    "value4": 4,
    "value3": "v3"
  }
}

Using this command:

jq -s '.[].value' file1 file2

Answer 1

Use jq -s add:

$ echo '{"a":"foo","b":"bar"} {"c":"baz","a":0}' | jq -s add
{
  "a": 0,
  "b": "bar",
  "c": "baz"
}

This reads all JSON texts from stdin into an array (jq -s does that) then it "reduces" them.

(add is defined as def add: reduce .[] as $x (null; . + $x);, which iterates over the input array's/object's values and adds them. Object addition == merge.)

Answer 2

First, {"value": .value} can be abbreviated to just {value}.

Second, the --argfile option (available in jq 1.4 and jq 1.5) may be of interest as it avoids having to use the --slurp option.

Putting these together, the two objects in the two files can be combined in the specified way as follows:

$ jq -n --argfile o1 file1 --argfile o2 file2 '$o1 * $o2 | {value}'

The '-n' flag tells jq not to read from stdin, since inputs are coming from the --argfile options here.

Answer 3

Here's a version that works recursively (using *) on an arbitrary number of objects:

echo '{"A": {"a": 1}}' '{"A": {"b": 2}}' '{"B": 3}' | jq --slurp 'reduce .[] as $item ({}; . * $item)'
{
  "A": {
    "a": 1,
    "b": 2
  },
  "B": 3
}

Answer 4

Who knows if you still need it, but here is the solution.

Once you get to the --slurp option, it's easy!

--slurp/-s:
    Instead of running the filter for each JSON object in the input,
    read the entire input stream into a large array and run the filter just once.

Then the + operator will do what you want:

jq -s '.[0] + .[1]' config.json config-user.json

(Note: if you want to merge inner objects instead of just overwriting the left file ones with the right file ones, you will need to do it manually)

Answer 5

Since 1.4 this is now possible with the * operator. When given two objects, it will merge them recursively. For example,

jq -s '.[0] * .[1]' file1 file2

Would get you:

{
  "value1": 200,
  "timestamp": 1382461861,
  "value": {
    "aaa": {
      "value1": "v1",
      "value2": "v2",
      "value3": "v3",
      "value4": 4
    },
    "bbb": {
      "value1": "v1",
      "value2": "v2",
      "value3": "v3"
    },
    "ccc": {
      "value1": "v1",
      "value2": "v2"
    },
    "ffffd": {
      "value3": "v3",
      "value4": 4
    }
  },
  "status": 200
}

If you also want to get rid of the other keys (like your expected result), one way to do it is this:

jq -s '.[0] * .[1] | {value: .value}' file1 file2

Or the presumably somewhat more efficient (because it doesn't merge any other values):

jq -s '.[0].value * .[1].value | {value: .}' file1 file2

Answer 6

for a completeness of options posting here how to achieve the same with jtc utility:

1. as per ask, there's a requirement to extract (but preserve structure) of value object from file1:

bash $ cat file1 | jtc -w '<value>l' -pp
{
   "value": {
      "aaa": {
         "value1": "v1",
         "value2": "v2"
      },
      "bbb": {
         "value1": "v1",
         "value2": "v2"
      },
      "ccc": {
         "value1": "v1",
         "value2": "v2"
      }
   }
}
bash $

2. now insert/merge value from file2 into the resulted json:

bash $ cat file1 | jtc -w '<value>l' -pp | jtc -w'<value>l' -i"`jtc -w'<value>l' file2`"
{
   "value": {
      "aaa": {
         "value1": "v1",
         "value2": "v2",
         "value3": "v3",
         "value4": 4
      },
      "bbb": {
         "value1": "v1",
         "value2": "v2",
         "value3": "v3"
      },
      "ccc": {
         "value1": "v1",
         "value2": "v2"
      },
      "ffffd": {
         "value3": "v3",
         "value4": 4
      }
   }
}
bash $