jq-front: JSON with inheritance and templating

Design

Following is a diagram that illustrates jq-front 's processing pipeline design.

Figure 1. jq-front’s pipeline

As shown, it consists of four main components, which are "file-level inheritance", "local node materialization", "node-level inheritance", and "templating".

Designs of those components will be described in the rest of this section.

For both of file-level and node-level inheritances, the inheritance mechanism (the large box in the diagram) is invoked recursively. That is, when a file which has $extends attribute, the mechanism will be applied repeatedly until it reaches a file that doesn’t have any. In case a cyclic inheritance is found, jq-front will report it and abort.

File-level Inheritance

'File-level Inheritance' composes a new JSON file from a given one by expanding the files provided through the top-level $extends attribute.

At the end of this process, the attribute is removed and copied to the output, "Work(1)" in the diagram.

Local node Materialization

If the file, "Work(1)" has $local attribute whose value is an object node, nodes associated with keys under the attribute are dumped under a temporary directory. The temporary directory is called 'local node directory'. The 'local node directory' and its contents are utilized by the "node-level inheritance' mechanism.

Node-level Inheritance

Node-level Inheritance mechanism is a bit more complicated than the file-level one.

Create a base JSON object by the following procedure.
1. Scan paths of all internal nodes.
2. For each path, if it ends with "$extends", expand the files specified by the attribute.
3. Assign the JSON node created in step b. to the parent of the $extends attribute.
Overlay the JSON object created in the step. 1 with an original JSON object.
Remove nodes, i.e., $extends and $local nodes from the JSON object created in the step 2.

'Local node directory' is inserted before the first entry of JF_PATH environment variable when a file to be expanded is searched during the step 1. b.

Templating

Templating is executed by performing eval on every text node whose value starts with eval: in JSON objects Work(2 a) and Work(2 b). The order where it is applied is determined by jq 's path(..) function. Note that it is not defined in jq 's manual[jq] how keys are sorted within an object node, although they seem to be sorted by dictionary order.

This happens in two steps. The first one processes "keys" in the JSON object, while the second only processes the other (values and arrays).

See Templating to know more about it.

Temporary Files

jq-front creates temporary files under a directory specified by TMPDIR environment variable.

TMPDIR/
  .jq-front/             (1)
    session-XXXXXX/
      inprogress/        (2)
      localnodes-XXXXX/  (3)
      nodepool/          (4)
      source_files/      (5)
      templating/        (6)
      misc/              (7)

Tools used by jq-front write their temporary files.
a directory to store "mark" files to manage on-going inheritance processing.
a directory to store paths to "local nodes" as files.
a directory to store cached files for inheritance processing.
a directory to store script files to be "sourced", when SOURCE specified in $extends or $includes.
a directory used during "templating"
other temporary files generated by jq-front itself.

`inprogress` directory

In this directory, empty files of the following names are created.:

inheritance-$(hashcode _filename)
reference-$(hashcode _filename)

Those are to detect cyclic dependencies during jq-front 's processing. The hashcode part is computed by md5sum command from the absolute path of a file. If jq-front finds another file /your/another/file is inherited by a currently processed file, it will create a file inheritance-$(hashcode /your/another/file). If a file to be created found existing already, jq-front will consider that there is a cyclic dependency and aborts the process with an error message. jq-front will then process /your/another/file recursively and once its process is finished, the temporary file will be removed.

`nodepool` directory

This is a directory to store files whose all inheritances are all expanded. Under this directory, $(hashcode _filename) is created with the expanded content. Following is an example of the file:

/tmp/.jq-front/session-XXXXXX/nodepool/$(hashcode parent.json)

{"a":{"a":"eval:string:$(echo 'A')","b":"eval:string:$(echo 'B')","o":"eval:string:$(ref $(cur).a)-$(ref $(cur).b)"}}

The content of the file parent.json and the file it extends before the inheritance expansion may look like following:

parent.json

{
  "a": {
    "$extends": [
      "input/A.json"
    ]
  }
}

A.json

{
  "a": "eval:string:$(echo 'A')",
  "b": "eval:string:$(echo 'B')",
  "o": "eval:string:$(ref $(cur).a)-$(ref $(cur).b)"
}

Note that the references are not processed in the step of inheritance expansion yet as illustrated in the jq-front’s pipeline.

`localnodes-XXXXX` directory

The node-level inheritance feature allows you to extend not only local nodes, which are defined inside JSON files, but also independent files[Node level Inheritance Example].

Node level Inheritance Example

{
  "$local": {
    "localNode": {
      "k": "v"
    }
  },
  "A": {
    "$extends": [
      "localNode", "externalFile.json"
    ]
  }
}

This is a main difference between file level inheritance and node level inheritance mechanism. To make this possible, jq-front "materializes" the local nodes, that is, those local nodes are turned into files and then the files will be processed indifferently.

top

open this page in a window

FAQ

Why is jq-front so slow?

It is because it’s written as a bash script and creates quite a lot of process an input file. If somebody starts a project to rewrite this in Java, for instance, I would be contributing to it!

How is error handling designed?

jq-front is designed and implemented so that it stops processing on errors. However, due to the specification of bash, if you use command substitution in templating and an inner call occurs an error, it CANNOT stop.

Except for that, if you think it should stop but it doesn’t, please file a ticket here.

On errors, you should see an output like following.

ERROR: Malformed JSON was given:'/home/hiroshi/Documents/jq-front/tests/negative/malformed-node-inheritance/filelevel/M.json'='// MALFORMED'
  at 36 abort /home/hiroshi/Documents/jq-front/lib/shared.sh
  at 559 run_jqfront /home/hiroshi/Documents/jq-front/jq-front
  at 472 expand_filelevel_inheritances /home/hiroshi/Documents/jq-front/jq-front

Others

Please also check: FAQ on GitHub

top

open this page in a window

Features

With jq-front, you can build a JSON object reusing other JSON objects by its inheritance mechanism. To specify JSON objects to be inherited, jq-front searches for keys which have special names in a given object. $extends and $local are keywords used for this purpose.

Not only that, it allows you to construct a node by referencing other nodes and executing commands.

Basic features offered by the product are following

Inheritance
- File-level Inheritance
- Node-level Inheritance
- Reverse Inheritance
- Script-output Inheritance
- Yaml inheritance
Templating ("eval" feature)
Validation

And usage of them will be discussed in this section.

In this section, we introduce features of the product using following data.

A.json
```
{
  "a": "A",
  "o": "A"
}
```

AA.json

{
  "$extends": [ "A.json" ],
  "aa": "AA"
}

B.json
```
{
  "a": "B",
  "o": "B"
}
```

T.json

{
  "versionStem": "2.12.0",
  "snapshotVersion": "eval:$(ref .versionStem)-SNAPSHOT"
}

Inheritance

File-level Inheritance

In programming languages, inheritance is an indispensable technique to reuse a component. jq-front offers it for the purpose. A usage example is as follows.

I.json

{
  "$extends": [ "A.json" ],
  "o": "hello world"
}

jq-front I.json

I.json will be rendered into the following JSON object with this command line.

File-level Inheritance output

{
  "a": "A",
  "o": "hello world"
}

As you see in the example, the attribute o whose value is set to "A" in A.json, is overridden by the value in I.json.

Multiple inheritance is also supported by jq-front. Just by listing file names of JSONs to be inherited, multiple inheritance happens, like "$extends": ["A.json", "B.json"]. When both A.json and B.json have attributes at the same path, A.json side’s value will be used.

J.json

{
  "$extends": [ "A.json", "B.json" ]
}

That is, J.json will be rendered into following output.

File-level Multiple Inheritance output

{
  "a": "A",
  "b": "B",
  "o": "A"
}

As it is so in normal programming language that supports multiple inheritance, avoiding diamond inheritance is a good idea. Multiple inheritance was implemented to cope with a situation where you want to reuse two JSON objects defined for completely different purposes. For instance, one is for authentication information and the other is for GUI flavor.

JSON objects that are inherited can also inherit some other JSON files.

K.json

{
  "$extends": [ "AA.json", "B.json" ]
}

That is, K.json is rendered into a following JSON file.

File-level Inheritance output (2)

{
  "a": "A",
  "aa": "AA",
  "b": "B",
  "o": "A"
}

Caution

Ensure that inheritance hierarchy does not have any cyclic dependencies. It will be checked and result in an error.

Node-level Inheritance

"Node-level Inheritance" refers to an inheritance happens on an internal (object) node of a given JSON file. Although it is implemented as a separate mechanism from the file-level one as it will be discussed in "Design" section, it behaves almost the same as the "file-level" one.

L.json

{
  "a": {
    "$extends": [ "A.json" ],
    "a": "L"
  }
}

Node-level Inheritance output

{
  "a": {
    "a": "L",
    "o": "A"
  }
}

As it worked for File-level Inheritance, multiple inheritance works also for Node-level Inheritance.

However, for internal nodes, you can also reference "local" nodes not only external files.

P.json

{
  "$local": {
    "nodeA": {
       "aa": "aa"
    },
    "nodeB": {
    }
  },
  "a": {
    "$extends": ["nodeA"],
    "a": "a"
  }
}

These nodes can be referenced through "node-level inheritance feature" as shown in the example. Note that you do not need to specify .json extension. And P.json will result in following output.

Local Node Inheritance output

{
  "a": {
    "aa": "aa",
    "a": "a"
  }
}

Reverse Inheritance

When you design a data structure using JSON (or YAML), you often find that you want to define a template, where user custom files are inserted.

{
  "company": "SPQR",
  "laptopSpec": {
    "cpu": "M1",
    "mem": "16GB",
    "storage": "512GB"
  },
  "userConfig" : {
    "userName": "eval:$(whoami)",
    "preferredShell": "zsh",
    "preferredWindowManager": "twm"
  }
}

Suppose that your users have their user preference files at /userhome/.yourapp/config as a JSON file. You are thinking of overriding the elements under userConfig by the JSON config file. But using the normal inheritance ($extends) here will not help. Because the values you define in the base file as default values will override the user specific configuration. Not the other way around. What you can do here is "reverse inheritance" with the $includes keyword.

{
  "company": "SPQR",
  "laptopSpec": {
    "cpu": "M1",
    "mem": "16GB",
    "storage": "512GB"
  },
  "userConfig" : {
    "$includes": [ "/userhome/.yourapp/config" ],
    "userName": "eval:$(whoami)",
    "preferredShell": "zsh",
    "preferredWindowManager": "twm"
  }
}

Script Inheritance

jq-front can use the output from your shell as a file to be extended if it is a JSON node.

For instance, if you have a following script file: S.sh, which prints something like {"S":"shell"}.

S.sh

echo '{"S":"shell-'${1}'"}'

The output from the script can be extended by a following file, for instance.

{
  "i": {
    "$extends": [
      "S.sh;bash -eu;hello"
    ],
    "o": "hello world"
  }
}

This results in a file as follows.

{
  "i": {
    "S": "hello",
    "o": "hello world"
  }
}

The component bash -eu is a program with which the script (S.sh) is executed. This feature is still experimental.

Yaml file inheritance

jq-front can handle YAML files also. If you have following two files,

A.yml

---
a: A
o: A
y: Y

{
  "$extends": [ "A.yml" ],
  "o": "hello world"
}

The output will be like following

{
  "a": "A",
  "o": "hello world",
  "y": "Y"
}

This feature is still experimental.

Templating

Sometimes we need to compose a value of text node from a value of another. Following is such an example.

Version file

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "2.12.0-SNAPSHOT"
}

In this example, the version to be released next is 2.12.0, however the version under the development for it has a suffix -SNAPSHOT

To follow the principle of D-R-Y, how should we fix it? Templating is a feature to offer a solution to this challenge.

We can describe this relationship by using the templating feature of jq-front.

T.json

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "eval:$(ref .releaseVersion)-SNAPSHOT"
}

Once you render this file with jq-front, you will get the first file (Version file).

$ref is a built-in function of jq-front, which expands the value of the node specified by the path given as an argument. Not only built-in functions but also any commands (bash expressions) valid on a platform on which jq-front is running can be used here.

For instance, following is a valid input to jq-front.

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "eval:$(ref .releaseVersion)-$(date \"+%Y-%m-%d\")"
}

And this will result in an output below.

{
  "releaseVersion": "2.12.0",
  "snapshotVersion": "2.12.0-2019-08-28"
}

This feature can be disabled by -d (--disable-templating) option. And to enable it explicitly, you can use -e (--enable-templating) in case JF_TEMPLATING_ENABLED is set to no.

This feature can be used for the key side, also.

That is, if the following JSON object is given as input:

{
  "eval:$(echo helloBase)": {
    "a": [
      "Hello"
    ],
    "b": [
      "World"
    ],
    "arr": "eval:array:$(array_append \"$(ref $(cur).a)\" \"$(ref $(cur).b)\")"
  }
}

it will be converted as as follows:

{
  "helloBase": {
    "a": [
      "Hello"
    ],
    "b": [
      "World"
    ],
    "arr": [
      "Hello",
      "World"
    ]
  }
}

Note that the key-side templating happens first and then the value-side templating will follow. Also note that you need to use key-side templating carefully, because it may confuse you sometimes, otherwise. For instance, if you create a key which results in the same string as another key, the outcome isn’t specified.

top

open this page in a window

Installation

Ubuntu

Download an archive or clone the repository. And place the file jq-front, lib, and schema in a directory which is on your PATH. Following dependencies will be required by jq-front

bash
jq
ajv-cli
- npm
yq
- python-pip
- python

Warning

The version of yq installed through, snap install yq doesn’t work for jq-front. Visit yq site to know it more.

Note	All the direct and indirect dependencies are found in the `Dockerfile`.

With Docker (recommended)

If you are already using docker and bash, this is the easiest way to use jq-front. To install docker, visit

Add a following entry to your .bashrc or a file sourced through it (such as .profile on Mac OSX).

function jq-front() {
  docker run --rm -i \
    -v "${HOME}:/var/lib/jf/${HOME}" \
    -v "${HOME}/.jq-front.rc:/root/.jq-front.rc" \
    -e JF_PATH_BASE="/var/lib/jf" \
    -e JF_PATH="${JF_PATH}" \
    -e JF_DEBUG=${JF_DEBUG:-disabled} \
    -e JF_CWD="$(pwd)" \
    dakusui/jq-front:"${JF_DOCKER_TAG:-v0.57}" "${@}"
}

NOTE: This approach only allows you to process files under $HOME directory.

To install bash on windows, install-bash-on-windows will be helpful.

top

open this page in a window

Limitations and Future Work

Issues

File ends with `?`

With Issue-138, now a question mark ? in a file name has a special semantics, where it means that it is considered an empty JSON object file if the file is not found by the `jq-front’s file searching mechanism.

For instance,

{
  "key": {
     "$extends": [ "missingFile.json?" ]
  }
}

Now, this produces

{
  "key": {}
}

Instead of giving you an error. And currently there is no way to specify a file whose name really ends with ?. This semantics is only introduced in inheritances of JSON files, any other usages are not considered as of now.

Referencing a JSON path containing `.`

It is tricky to reference a path containing ..

{
  "key": "eval:string:$(ref '.\"key-2.suffix\"')",
  "key-2.suffix": "Hello, world!"
}

Note that you first need the double quotes for the path component (key-2.suffix). Then, you need the single quotes for the entire path expression ('.\"key-2.suffix\"') to prevent the double quotes going away.

Finally, the input renders into a following JSON content.

{
  "key": "Hello, world!",
  "key-2.suffix": "Hello, world!"
}

The quotings are necessary because the jq-front relies on eval built-in of bash for implementing the eval: syntax.

Make it faster

The largest weakpoint of jq-front is its performance. It takes seconds to process even a relatively simple and small file.

However, trying to make jq-front faster sacrificing the readability of it doesn’t seem to me a good idea, especially in case it is written in a language which is generally considered "hard to read and debug".

Instead, we should think of implement it in another language, which is performance-wise more powerful and optimized, such as Java, C#, or whatsoever.

Design consideration

Path in JSON
Implementing the 'templating' feature.

Path in JSON

To implement a processor like jq-front requires a notation to specify a certain point in a JSON node as a string.

jq has such as feature out-of-box.

{ "a": { "b": 123, "c": ["HELLO"]
} }

The string HELLO in the array in the example above can be specified by a string .a.c[0].

We need to choose a library that can do this sort of work or implement such a functionality by ourselves.

Implementing the 'templating' feature

In order to implement the 'templating' feature, we need to be able to handle a string like following.

    "eval:object:{"hello":"$(ref .root.greeting[0].english)"}

top

open this page in a window

Usage

Command Line Interface

jq-front [-h|--help] [--validation=no|strict|lenient] [--nested-templating-levels=num] [--version] [TARGET]

-h, --help: Shows this help
--validation: Validation mode. no, strict, and lenient are available. The default is no.
--nested-templating-levels: Number of times templating happens by default. The default is 5. If templating doesn’t finish within num times, an error will be reported.
--version: Shows a version.
TARGET: A file to be processed. If not given, stdin will be processed.

Environment variables

You can control behaviours of jq-front by setting environment variables described in this section.

JF_PATH

From directories listed in JF_PATH, jq-front searches for requested file. Entries in the variable are separated by colons(:).

When it is searching for a file during node-level inheritance resolution, it first searches for local node directory, which is created for temporarily, and if nothing is found, it will then traverses the variable.

Default value

.

JF_DEBUG

If this variable is set to enabled, debug information will be printed to stderr.

Default value

disabled

JF_INFO

If this variable is set to enabled, "INFO"(information) level log will be printed to stderr.

Default value

disabled

JF_PERF

If this variable is set to enabled, "PERF"(performance) level log will be printed to stderr.

Default value

disabled

top

open this page in a window

Syntax

`$extends` keyword

This keyword can be used as a key whose associated value is an array. Each element in the array must be a text node.

The string can be one of

A file in JF_PATH.
When it is placed NOT at the top level of an object node file, a name of a "local node".
A script invocation directive.

To create a JSON file from an existing one A.json, you can do following.

{
  "$extends": [ "A.json" ]
}

As you see in the example, the name of the file is placed inside an array and it means you can do so called "multiple-inheritance".

{
  "$extends": [ "A.json", "B.json" ]
}

If you do a multiple inheritance, an element appeared in the array earlier is more prioritized(similar to the multiple inheritance in python).

In case you have nodes at the same path in A.json and B.json, value from the A.json wins.

File in `JF_PATH`

As long as your file is under a directory specified by an element in JF_PATH, you can use it. That is, suppose that you have JF_PATH and it has a value .:dir1.

dir1
 |
 `--- child
      |
      `--- J.json

The file J.json can be referenced by

"$extends": ["child/J.json"]

In case you have child/J.json in multiple places under elements in JF_PATH, the first one will be used. For instance, in the following example, dir1/child/J.json will be referenced for the example abobe.

dir1
 +--- child
 |    |
 |     `--- J.json
dir2
 |
 `--- child
      |
      `--- J.json

You can also specify names of "local nodes". (See the section for $local keyword)

Inheriting YAML files

You can specify a file on JF_PATH environment variable. If the name ends with .yaml or .yml, it will be treated as a YAML file and converted into a JSON file by yq command.

This feature is still experimental.

Script Invocation Directive

You can specify a program which generates a JSON object with a following syntax.

{
  "$extends": [
    "SS.sh;bash -eu;dir1/J.json"
  ]
}

SS.sh is a script file to be executed. bash -eu is a program which executes the script. dir1/J.json is an argument which is passed to the program SS.sh.

The string is split by semicolons and the first token is treated as a name of a program to be executed. The program is searched from JF_PATH. The second toke is a shell with which the program is executed. And the rest will be passed to the program as arguments.

Note	By inserting one or more semicolons, this syntax is triggred.

`$local` keyword

This keyword can be used as a key whose associated value is an object. A value in the object must be an object.

This keyword can only be placed at the top-level of a file.

{
  "$local": {
    "A": {
      "a": "valueInA"
    },
    "B": {
      "b": "valueInB"
    },
    "C": {
      "$extends": ["A","B"]
    }
  },
  "D": {
    "$extends": ["C"]
  }
}

In this example, local nodes A, B, and C are defined. And a node at the top level, D extends C, which then extends A and B. This results in a following JSON object.

{
  "D": {
    "a": "valueInA",
    "b": "valueInB"
  }
}

Note	In case you have a local node and a file with the same name, `jq-node` picks up a local node, although you do not need to mind it usually because you do not want to give a suffix `.json` to a local node.

`eval:` keyword

This keyword can be used in a text node. The syntax can be defined as follows.

  eval:[TYPE:]STRING
  TYPE ::= object array string number boolean

If TYPE: is omitted, in other words eval: is followed by anything else than the defined TYPE`s, `jq-front behaves as if string is specified.

The STRING is evaluated by a following command line.

    eval "echo \"${_body}\"" 2>"${_error}"

As seen in the above fragment, stderr is redirected to an internal file and the file is checked if a string ERROR: is contained in it. If the string is found in it, jq-front considers that something went wrong during the evaluation and aborts the rest of the process. Such a string is printed to stderr by error function (See its definition in Built-in jq-front functions section).

Not only variables, functions, and commands visible to a bash shell on which jq-front runs, you can use functions provided by the processor. For more details, refer to Built-in jq-front functions section.

Caution

There are similar keywords template:, but it is only kept for compatibility and will be removed in the future version of this product. Please refrain from using it. In case you want to define a text node that starts with the string itself, you can do raw:template:…

`raw:` keyword

You may sometimes want to define a text node which starts with other keywords such as eval: itself. In such cases you can use raw: keyword to escape it.

   raw:eval:hello

This results in a following output.

   eval:hello

`template:` keyword

Deprecated. A keyword that has similar effects to eval: keyword. This is kept only for compatibility.

Built-in `jq-front` functions

In addition to commands and functions visible to a bash shell on which jq-front runs, you can use functions listed in this section.

`ref` function

A function that returns a value of a node specified by an argument. This function can only work from inside "Work(2)" file.

In case this function references a text node that starts with eval:, it performs templating on the node. This means, the ref function may be applied recursively. In case cyclic reference is found during this process, it will be reported and the process will be aborted.

parameter:
- _path: path to a node in the file "Work(2)"
returned value (stdout):
- A value of a node specified by _path

Examples

Input Output

Input	Output
A.json `{ "a": { "b": { "c": "hello" } }, "r": "eval:string:$(ref .a.b.c), world" }`	`{ "a": { "b": { "c": "hello" } }, "r": "hello, world" }`
B.JSON `{ "$extends": ["A.json"], "r": "eval:string:$(ref .a.b.c), world" }`	`{ "a": { "b": { "c": "hello" } }, "r": "hello, world" }`
C.JSON `{ "$extends": ["A.json"], "r": "eval:object:$(ref .a.b)" }`	`{ "a": { "b": { "c": "hello" } }, "r": { "c": "hello" } }`

A.json

{
  "a": {
    "b": {
       "c": "hello"
    }
  },
  "r": "eval:string:$(ref .a.b.c), world"
}

{
  "a": {
    "b": {
       "c": "hello"
    }
  },
  "r": "hello, world"
}

B.JSON

{
  "$extends": ["A.json"],
  "r": "eval:string:$(ref .a.b.c), world"
}

{
  "a": {
    "b": {
       "c": "hello"
    }
  },
  "r": "hello, world"
}

C.JSON

{
  "$extends": ["A.json"],
  "r": "eval:object:$(ref .a.b)"
}

{
  "a": {
    "b": {
       "c": "hello"
    }
  },
  "r": {
    "c": "hello"
  }
}

`self` function

A function that prints the entire file content before templating. This function is intended for internal use.

parameter: (none)
returned value (stdout):
- Content of the processed file before any templating happens.

`curn` function

A function that returns a path to the node that makes a call to this function.

parameter: (none)
returned value (stdout):
- A path to the string node that makes the call to this function.

Examples

Input Output

Input	Output
A.json `{ "a": { "b": [ "eval:$(curn)" ] } }`	`{ "a": { "b": [ ".a.b[0]" ] } }`
A.json `{ "a": { "b": { "c": "eval:$(cur)" } } }`	`{ "a": { "b": { "c": ".a.b.c" } } }`

A.json

{
  "a": {
    "b": [
       "eval:$(curn)"
    ]
  }
}

{
  "a": {
    "b": [
       ".a.b[0]"
    ]
  }
}

A.json

{
  "a": {
    "b": {
       "c": "eval:$(cur)"
    }
  }
}

{
  "a": {
    "b": {
       "c": ".a.b.c"
    }
  }
}

Note	The actual outputs of this function are escaped.

`cur` function

A function that returns a path to a container node to which the current "entry" belongs. An "entry" means a string element in an array or a pair of key and value in an object.

parameter: (none)
returned value (stdout):
- A path to the container the node belongs to

Examples

Input Output

Input	Output
A.json `{ "a": { "b": [ "eval:$(cur)" ] } }`	`{ "a": { "b": [ ".a.b" ] } }`
A.json `{ "a": { "b": { "c": "eval:$(cur)" } } }`	`{ "a": { "b": { "c": ".a.b" } } }`
X.JSON `{ "x": { "$extends": ["A.json"] } }`	`{ "x": { "a": { "b": { "c": ".x.a.b" } } } }`

A.json

{
  "a": {
    "b": [
       "eval:$(cur)"
    ]
  }
}

{
  "a": {
    "b": [
       ".a.b"
    ]
  }
}

A.json

{
  "a": {
    "b": {
       "c": "eval:$(cur)"
    }
  }
}

{
  "a": {
    "b": {
       "c": ".a.b"
    }
  }
}

X.JSON

{
  "x": {
    "$extends": ["A.json"]
  }
}

{
  "x": {
    "a": {
      "b": {
         "c": ".x.a.b"
      }
    }
  }
}

Note	Notice that jq-front first expands all the inheritances in the input and then invokes the 'templating' mechanism. Thus, `cur` function calls in inherited files are evaluated based on paths where they appear in the expanded file.

`parent` function

A function that prints a path to a parent node of a given path.

parameter:
- A path to a node
returned value (stdout):
- A path to a parent of the node.

Examples

Input Output

Input	Output
`"eval:$(parent .hello.world)"`	`".hello"`

"eval:$(parent .hello.world)"

".hello"

`error` function

A function that prints a given error message and returns a non-zero value.

parameter:
- An error message
returned value:
- stdout
  - (none)
- sterr
  - A string starts with ERROR: {given error message}. A stack trace follows it.
- exit code
  - A non-zero value.

Note

If you nest a call to a function or command that fails by another using a command substitution of bash, the next call will not be aborted immediately, in general. That is, "eval:$(echo $(cat missing-file))-$(echo hello)" will result in -hello. This is a behaviour of bash and its command substitution. However, if you use this function, "eval:$(echo $(cat missing-file || error 'something went wrong'))-$(echo hello)", jq-front will abort the process after evaluating this string because it finds a keyword ERROR: in the stderr.

Note

Functions discussed in this section check whether $? is zero at the beginning. If it is not zero, the function will abort and the string evaluation will be aborted in general. However, a user is still able to nest the call with another function that not necessarily performs such a check. In this case, the evaluation will not stop at the point and jq-front will abort the rest of its execution after handling the string since it will find the ERROR: keyword.

Examples

Input Output

Input	Output
`"eval:$(error hello)"`	`ERROR: hello at 36 abort /home/who/Documents/jq-front/lib/shared.sh at 439 _check_cyclic_dependency /home/who/Documents/jq-front/jq-front at 111 _expand_nodelevel_inheritances /home/who/Documents/jq-front/jq-front ...`

"eval:$(error hello)"

ERROR: hello
  at 36 abort /home/who/Documents/jq-front/lib/shared.sh
  at 439 _check_cyclic_dependency /home/who/Documents/jq-front/jq-front
  at 111 _expand_nodelevel_inheritances /home/who/Documents/jq-front/jq-front
...

Defining a user function for templating

You can define your own function for the templating stage by following syntax. First, you can create a file that contains definitions of your functions.

SS.sh

function hello_world() {
  echo "Hello, world. My Function!"
}

Next you reference the file inside the $extends syntax.

{
  "$extends": [
    "SS.sh;SOURCE"
  ],
  "key": "eval:string:hello_world=$(hello_world),$(echo HELLO)"
}

The string SOURCE is the keyword that tells jq-front to import the file. And as you see, you are now able to call the function you defined, hello_world.

{
  "key": "hello_world=Hello, world. My Function!,HELLO"
}

The file will be rendered as you see above.

top

open this page in a window

`./` jq-front: JSON with inheritance and templating

jq-front is a simple tool to give your JSON files a power of inheritance and templating. It was named after Cfront[2], which is an old component of C++ language that converts C++ source code into C language.

Despite that there are criticisms to use JSON[5] as system configuration information’s format[3], JSON still has its beauty such as rich tool supports, good balance between machine’s and human’s readability and writability, clarity and simplicity in syntax, etc.

However, surely it has some weaknesses when we try to define data structure. For instance, under a situation, where we need to generate similar but slightly different system configurations repeatedly, it is quite difficult to remove redundant pieces since JSON itself does not define anything about relationships between files or nodes inside files.^[1]

For what is it useful?

Haven’t you ever created configuration files, each of which are slightly different?

config for foo config for bar

config for foo	config for bar
`{ "foo_database": { "server": { "ip": "192.168.1.5", "port": 2001 }, "db_name": "foo", "user": { "name": "root", "password": "foo_root" } } }`	`{ "bar_database": { "server": { "ip": "192.168.1.5", "port": 2001 }, "db_name": "bar", "user": { "name": "root", "password": "bar_root" } } }`

{
  "foo_database": {
    "server": {
      "ip": "192.168.1.5",
      "port": 2001
    },
    "db_name": "foo",
    "user": {
      "name": "root",
      "password": "foo_root"
    }
  }
}

{
  "bar_database": {
    "server": {
      "ip": "192.168.1.5",
      "port": 2001
    },
    "db_name": "bar",
    "user": {
      "name": "root",
      "password": "bar_root"
    }
  }
}

Probably you want to define a JSON, which defines boring default values like this,

database_default

{
    "server": {
      "ip": "192.168.1.5",
      "port": 2001
    },
    "user": {
      "name": "root"
    }
}

And reuse it from other files as follows

foo.json bar.json

foo.json	bar.json
`{ "foo_database": { "$extends": [ "database_default.json" ], "db_name": "foo", "user": { "password": "foo_root" } } }`	`{ "foo_database": { "$extends": [ "database_default.json" ], "db_name": "bar", "user": { "password": "bar_root" } } }`

{
  "foo_database": {
    "$extends": [ "database_default.json" ],
    "db_name": "foo",
    "user": {
      "password": "foo_root"
    }
  }
}

{
  "foo_database": {
    "$extends": [ "database_default.json" ],
    "db_name": "bar",
    "user": {
       "password": "bar_root"
    }
  }
}

jq-front does it for you. Try jq-front foo.json from your command line and you will see the config for foo in the first matrix.

YAML?

There is a criticism to use JSON as a configuration language[3]. Although I am a bit skeptical at the discussion from some view points (e.g., users will eventually end up in desiring GUI not only YAML), jq-front can also be used to make your YAML based configuration language more powerful.

INPUT OUTPUT

INPUT	OUTPUT
`$local: database_default: server: ip: 192.168.1.5 port: 2000 db_name: test user: name: root password: root # database foo differs from default by only its port and user password foo_database: $extends: [ database_default ] server: port: 2001 db_name: foo user: password: foo_root`	`foo_database: server: ip: 192.168.1.5 port: 2001 db_name: foo user: name: root password: foo_root`

$local:
  database_default:
    server:
      ip: 192.168.1.5
      port: 2000
    db_name: test
    user:
      name: root
      password: root

# database foo differs from default by only its port and user password
foo_database:
  $extends: [ database_default ]
  server:
    port: 2001
  db_name: foo
  user:
    password: foo_root

foo_database:
  server:
    ip: 192.168.1.5
    port: 2001
  db_name: foo
  user:
    name: root
    password: foo_root

The command line to render the input is following.[7]

$ yq . -j in.yaml | jq-front  | yq . -y

HOCON?

HOCON[hocon] is another notation intended for human readability and writability. It also supports the data inheritance, and the node reference feature but you have still have good reasons to use jq-front.

The input and output of jq-front are both JSON, which are widely accepted standard.
jq-front allows you to define custom operations very easily by "templating".
The capability to structure data (data inheritance and node inheritance) and the improvements in human readability are two separate concerns, which shouldn’t be confused.

The last point tends to be looked over frequently but important in context of automating the system configuration, which is necessary in CI/CD. In order to configure your system automatically, it is important to generate the configuration file for the system programmatically.

However, HOCON is not so machine generatable format as JSON because it is a superset of JSON. Even if it is possible, still you end up in a problem, where you will potentially get an object looking differently after a round trip generation. That is, you first write a HOCON file by hand, then process it by your program. After that, you may notice that the output of your program doesn’t respect your style. Style is not important for computers, but it is for human’s readability and wasn’t it the intention of the notation? This situation may come from the design of your pipeline, where you are trying to resolve two problems in a single stage by HOCON. One is human readability and the other is value resolution to structure your data. Instead, you can separate it into two stages, where those are executed independently.

You can use jq-front for the second stage. With that, you can utilize any tools and libraries that are designed for JSON. (Remember, jq itself equips with a very powerful query language after which the tool was named). This approach delivers you another benefit, where you can select another solution to improve human readability of your configuration such as YAML.

References

[1] jq-front project in github.org. jq-front: 2019
[2] Cfront article in en.wikipedia.org. Cfront: 2019
[3] Thayne McCombs. Why JSON isn’t a Good Configuration Language: 2018.
[4] YAML article in en.wikipedia.org. YAML: 2019
[5] json.org. JSON:2019
[6] jq jq manual:2019
[7] More complex inheritance in YAML? stackoverflow:2019
HOCON (Human-Optimized Config Object Notation) HOCON:2020

top

open this page in a window

Design

File-level Inheritance

Local node Materialization

Node-level Inheritance

Templating

Temporary Files

inprogress directory

nodepool directory

localnodes-XXXXX directory

FAQ

Why is jq-front so slow?

How is error handling designed?

Others

Features

Inheritance

File-level Inheritance

Node-level Inheritance

Reverse Inheritance

Script Inheritance

Yaml file inheritance

Templating

Installation

Ubuntu

With Docker (recommended)

Limitations and Future Work

File ends with ?

Referencing a JSON path containing .

Make it faster

Design consideration

Path in JSON

Implementing the 'templating' feature

Usage

Command Line Interface

Environment variables

JF_PATH

Default value

JF_DEBUG

Default value

JF_INFO

Default value

JF_PERF

Default value

Syntax

$extends keyword

File in JF_PATH

Inheriting YAML files

Script Invocation Directive

$local keyword

eval: keyword

raw: keyword

template: keyword

Built-in jq-front functions

ref function

Examples

self function

curn function

Examples

cur function

Examples

parent function

Examples

error function

Examples

Defining a user function for templating

./ jq-front: JSON with inheritance and templating

For what is it useful?

YAML?

HOCON?

References

`inprogress` directory

`nodepool` directory

`localnodes-XXXXX` directory

File ends with `?`

Referencing a JSON path containing `.`

`$extends` keyword

File in `JF_PATH`

`$local` keyword

`eval:` keyword

`raw:` keyword

`template:` keyword

Built-in `jq-front` functions

`ref` function

`self` function

`curn` function

`cur` function

`parent` function

`error` function

`./` jq-front: JSON with inheritance and templating