jq json parser hash the field value

问题:

I have a JSON file as the below:

[{
  "macAddress": "ac:5f:3e:87:d7:1a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:2a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:3a",
  "ip": "1.2.3.4"
}]

use jq to hash the macAddress field like so:

jq .[] | hash(.macAddress)

Can I define my own hash function and let jq to run the hash during the parsing process?

My expected hash function can be simple as using native linux command md5sum

echo -n "my_salt""42:12:20:2e:2b:ca" | md5sum
d973ea7c353e78ba1724efbc8054dfdc -

So the output json will be

[{
  "macAddress": "d973ea7c353e78ba1724efbc8054dfdc",
  "ip": "1.2.3.4"
},
{
  "macAddress": "d973ea7c353e78ba1724efbc8054d2er",
  "ip": "1.2.3.4"
},
{
  "macAddress": "d973ea7c353e78ba2324efbc8054d123",
  "ip": "1.2.3.4"
}]

回答1:

My expected hash function can be simple as using native linux command md5sum

This stays native and might be suitable;

Invocation :

jq -c .[] "$jsonfile" |
while read -r jsonline ; do
  hashmac="$(jq --arg mysalt "$mysalt" -s -j '
        .[] | "\($mysalt)" + .macAddress' <<<"$jsonline" |
      md5sum | cut -d ' ' -f1)"
  jq --arg hashmac "$hashmac" -s -r '
      .[] | .macAddress |= "\($hashmac)"' <<<"$jsonline"
done

---

Example file - /tmp/testfile:

[{
  "macAddress": "ac:5f:3e:87:d7:1a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:2a",
  "ip": "1.2.3.4"
},
{
  "macAddress": "ac:5f:3e:87:d7:3a",
  "ip": "1.2.3.4"
},
{
 "macAddress": "42:12:20:2e:2b:ca",
 "ip": "1.2.3.4"
}]

Result Output:

{
  "macAddress": "1f960fe4d24684ca44e5e67b6259362c",
  "ip": "1.2.3.4"
}
{
  "macAddress": "3527422754ecbfdd01d48b17fce87842",
  "ip": "1.2.3.4"
}
{
  "macAddress": "9bc8da72324448c3032a20fb67a31466",
  "ip": "1.2.3.4"
}
{
  "macAddress": "d973ea7c353e78ba1724efbc8054dfdc",
  "ip": "1.2.3.4"
}

Comments:

• -j causes jq to not output a newline, equivalent to your echo -n example

• Variables in this example are sent to jq as strings using -arg, and referenced as "\($var)" as opposed to escaping the variable directly, for example:

  • "\($mysalt)" + .macAddress' (jq variable)

    Instead of:

  • "'"$mysalt"'" + .macAddress' (direct shell substitution)

• This example uses cut -d ' ' -f1 to trim off the -, but there's probably a better way

Alternate:

jq --arg hashmac "$hashmac" -s -r '.[] |= . + {"hashAddress":"\($hashmac)"}'

Would append the json

[
  {
    "macAddress": "ac:5f:3e:87:d7:1a",
    "ip": "1.2.3.4",
    "hashAddress": "1f960fe4d24684ca44e5e67b6259362c"
  }
]

etc.

回答2:

The accepted answer entails 2n + 1 invocations of jq, where n is the number invocations of the message digest function.

To reduce the number of invocations to just 2, see shell-out value to md5 (crypto) function