Unix & Linux
bash grep scripting json jq
Updated Sat, 20 Aug 2022 22:54:58 GMT

Convert json file to a "key-path" with the resulting value at the end of each "key-path"

I have a large and relatively complex json configuration file that I want to search for keys and/or values using simple tools like "grep". When I 'grep' the file I would like to have in the output contain the full path of the key and every sub-key/array leading up to the final value.

This would be similar to searching for a file/directory in a large file system and able to see the full path to the file/directory when it is found such as you have with the 'find' command.

Also as a similar comparison to what I'm trying to accomplish is like the xml2 utility which converts an xml file into a key/value path for easy searching and reporting.

I have been working with the 'jq' utility to parse the json file using the 'keys' command. I've hobbled together a crude bash script to walk down the first branch of the json key-path -- but cannot find an easy way to recursively walk up and down the full tree of the json structure.

Here is a manual illustration of what I'm trying to do in my bash script. Yes, it's terribly inefficient, but I figure I'll make it work before improving performance!

Given file.json containing a complex configuration, I use the keys to grab the first item, once I have that, I use it in another iteration to get the next key, and so on until I reach the end of the branch...

cat file.json | jq '. | keys | .[]'
cat file.json | jq '.Blueprints | keys | .[]'
cat file.json | jq '.Blueprints.security | keys | .[]'
cat file.json | jq '.Blueprints.security.kerberos_descriptor | keys | .[]'
cat file.json | jq '.Blueprints.security.kerberos_descriptor.identities | keys | .[]'

The final result ultimately looks like this:


But of course this is only the first branch of a very large tree of key/values.


jq has a few suitable builtins to help out. You don't need much Bash trickery and it's not very well suited for the problem. Here is a pretty explicit version in jq that you should be able to modify for whatever purpose you need:

jq -r '. as $root |
       path(..) | . as $path |
       $root | getpath($path) as $value |
       select($value | scalars) |
       ([$path[] | @json] | join(".")) + " = " + ($value | @json)
    ' < file.json

It uses the variable binding operator ... as $identifier | several times to remember calculated values by name - some of those are unnecessary, but they make it easier to talk about. Each of those lines binds a variable $x for the remainder of the program to the value on the left.

The path/1 function is the key here, and does basically what you want already: path(..) produces an array of all the keys you'd need to traverse to get to every value nested in the object. Each path is in the form

[ "Blueprints", "security", "kerberos_descriptor" ]

and they can be used in the same way as other arrays, as well as with special functions that interpret paths.

path(..) | . as $path |

in particular is sort of defining a loop: for each path in the file, call it $path and run the rest of the program as the loop body. The remainder of the program is selecting and outputting, so for each path, it is checked and an output line possibly generated.

getpath reads one of those path arrays and plucks out the value it identifies. select lets us filter to only values passing a test - here, it selects only values that are scalars (numbers, strings, or booleans, or nulls), so intermediate objects and arrays are left out (as are nulls).

The final line formats the output as

"abc"."def".3."xyz" = true

for every value in the file, one per line, and you can adjust that as you wish. Redirect it into a file and you can grep through it repeatedly.

@json quotes values correctly, and the rest should be easy to change to suit the format you need. It doesn't use square brackets for arrays, because manually replicating the functionality of join to put dots in for the other cases is surprisingly complex. The parentheses are needed on both sides.

Comments (2)

  • +1 – Wow, that is a fantastic and elegant solution! Exactly what I needed and wanted. And also my appreciation for a good/clear explanation! Thank You! — Apr 26, 2019 at 14:32  
  • +1 – I regret that I can upvote this only once, excellent work — Feb 16, 2021 at 08:31  

External Links

External links referenced by this document: