YAML, not Gherkin

Gherkin is one notation widely used in "Cucumber"'s ecosystem.

A Gherkin (Cucumber) Example
Feature: Guess the word

  # The first example has two steps
  Scenario: Maker starts a game
    When the Maker starts a game
    Then the Maker waits for a Breaker to join

  # The second example has three steps
  Scenario: Breaker joins a game
    Given the Maker has started a game with the word "silky"
    When the Breaker joins the Maker's game
    Then the Breaker must guess a word with 5 characters

If we write this in YAML, it will be like this:

YAML version of Gherkin Example
---
Feature:
  Name: Guess the word (1)
  Scenarios:
  # The first example has two steps
  - Name: Maker starts a game
    When: the Maker starts a game
    Then: the Maker waits for a Breaker to join
  # The second example has three steps
  - Name: Breaker joins a game (1)
    Given: the Maker has started a game with the word "silky"
    When: the Breaker joins the Maker's game
    Then: the Breaker must guess a word with 5 characters

<1>: These attributes are introduced to define names of a feature and a scenario, which is defined without explicit attribute name in Gherkin.

Gherkin seems slightly simpler and cleaner since you will need fewer keywords, however, is it really worth adapting a dedicated notation not used anything but it? By using Gherkin, you will need a set of plugins to support Gherkin for every tool used in a software development project such as browser, editor, or IDE.

If so, wouldn’t YAML be good enough to define your test cases? isn’t YAML good enough, if we have a way to avoid repeating the same thing over and over again?

Given-When-Then structure

The commandunit has a similar syntax to other BDD (Beahvior Driven Development) testing tools such as "Cucumber"[2]. It has three sections.

Given

A step that defines a procedure to prepare a state of system under test before starting a test’s main part. This corresponds to "set up" phase in "four phase testing" model

When

A step that interacts with the SUT. This corresponds to "execute" phase in "four phase testing" model

Then

A step that verifies the output and the state of the SUT. This corresponds to "verify" phase in "four phase testing" model

Following is one example of a test definition for commandunit in plain JSON.

JSON example
{
  "type": "NORMAL",
  "description": [],
  "given": {
    "description": [
      "This test should always be executed."
    ],
    "stdin": [],
    "shell": {
      "name": "bash",
      "options": [
        "-eu",
        "-E"
      ]
    },
    "source": [],
    "environmentVariables": {
      "COMMANDUNIT_DEPENDENCIES_ROOT": "/home/hiroshi/Documents/github/commandunit/out/main/scripts/dependencies"
    },
    "cmd": ":",
    "args": []
  },
  "when": {
    "description": [
      "Run 'array_contains' with the first argument contained by the rest."
    ],
    "stdin": [],
    "shell": {
      "name": "bash",
      "options": [
        "-eu",
        "-E"
      ]
    },
    "source": [
      "${COMMANDUNIT_DEPENDENCIES_ROOT}/bud/lib/arrays.rc"
    ],
    "environmentVariables": {
      "COMMANDUNIT_DEPENDENCIES_ROOT": "/home/hiroshi/Documents/github/commandunit/out/main/scripts/dependencies"
    },
    "cmd": "array_contains",
    "args": [
      "hello",
      "hello",
      "world"
    ]
  },
  "then": {
    "description": [],
    "exitCode": [
      "EQUAL",
      0
    ],
    "stdout": {
      "present": [],
      "absent": [
        "REGEX:.+"
      ]
    },
    "stderr": {
      "present": [],
      "absent": [
        "REGEX:.+"
      ]
    }
  }
}

Powered by jq-front

jq-front [1] is a tool to enable your JSON files to extend other JSON files. Not only that it allows you to reference another node in the same file to compute a node’s value.

Test case definition tends to be repetitive, however, if we use jq-front to define common attribute values in base/normal.json, which can be used by other JSON files, it can be as simple as the following example.

JSON++ example
{
  "$extends": [
    "base/normal.json"
  ],
  "when": {
    "description": [
      "Run 'array_contains' with the first argument contained by the rest."
    ],
    "source": [
      "${COMMANDUNIT_DEPENDENCIES_ROOT}/bud/lib/arrays.rc"
    ],
    "cmd": "array_contains",
    "args": [
      "hello",
      "hello",
      "world"
    ]
  },
  "then": {
    "exitCode": [
      "EQUAL",
      0
    ],
    "stdout": {
      "absent": [
        "REGEX:.+"
      ]
    },
    "stderr": {
      "absent": [
        "REGEX:.+"
      ]
    }
  }
}

jq-front renders this file into a normal JSON file, where the values defined inside base/normal.json are expended and then overridden by the values in foo.json++ file.

It is a very flexible, yet still your files are JSON, as you see. This means that you don’t need to find supports of the new format. You can just keep using your favorite tools (editor, browser, IDE, etc) because in general they support popular formats such as JSON, YAML, or the good old XML.

YAML → JSON Pipeline

Let’s go one step forward. We use YAML in order to ensure the test case definitions readable for human. We also use jq-front, which processes JSON files(.json++) and renders into normal JSON files.

In short, instead of coming up with a single notation that solves all the problems at once, it applies tools that solve them one by one.

Following is an example of a file that written in YAML, using jq-front 's feature.

YAML++ example, test-contains_true.yaml++
"$extends":
  - base/normal.json
when:
  description:
    - Run 'array_contains' with the first argument contained by the rest.
  source:
    - ${COMMANDUNIT_DEPENDENCIES_ROOT}/bud/lib/arrays.rc
  cmd: array_contains
  args:
    - hello
    - hello
    - world
then:
  exitCode:
    - EQUAL
    - 0
  stdout:
    absent:
      - REGEX:.+
  stderr:
    absent:
      - REGEX:.+

This can be converted into a JSON file (a .json++ file) that uses jq-front feature, which is shown as the JSON++ example. Then it can be converted into normal JSON file, shown as JSON example

References

top

open this page in a new window

The wrapper script

The commandunit comes with a "wrapper" script that defines a bash function for you. The function automatically downloads the specified version of commandunit from its repo and execute it. If you specify a dockerized version of it, it downloads an image for it, if you specify non-dockerized ("native") version of it, it shallow-clones a branch in the repo for it.

It has two classes of options, which are "wrapper" options and normal options. The former controls the behavior of the wrapper and the latter controls the commandunit self’s behavior.

For instance, you can specify a version of commandunit 's main part by --version=…​, which is one of wrapper options. To specify tests to be executed by a regular expression you can ue --filter=…​ option which is a normal option.

Check Command Line Interface Handbook for more detail.

Directory Layout

The default directory layout of commandunit-based project is designed to be compatible with Maven-based project[1].

Default Directory Layout
src/                                    (1)
  main/
    dependencies                        (2)
  test/                                 (3)
    .commandunit/                       (4)
      jsonpp/                           (5)
      testdrivers/                      (6)
    scripts/
target/
  commandunit/
    report/                             (7)
    work/                               (8)
      scripts/
        core/
          {testcase-1}.json
          {testcase-1}.json++
          {testcase-1}.yaml++

  1. Current directory (project root directory)

  2. A directory under which "dependencies" are downloaded by the wrapper script. Typically used for downloading "native" versions of commandunit.

  3. A directory to store test scripts. (test script directory, can be changed through --test-srcdir= option)

  4. A directory to store commandunit 's configuration. (test project configuration directory) exported as COMMANDUNIT_DIR environment variable. This can be configured through --commandunit-dir option.

  5. A base directory from which files specified by $extends attribute are searched..

  6. A base directory from which "driver" files are searched.

  7. A directory to which commandunit writes report. (report directory, can be changed through --test-reportdir= option)

  8. A directory to which commandunit stores processed test scripts and intermediate files. (working directory, can be changed through --test-workdir option)

commandunit 's Pipeline

The following diagram illustrates data flows of the commandunit.

Diagram
Figure 1. The Processing Pipeline of the commandunit
preprocess subcommand (2.1)

This step converts test case definitions into JSON files, which can be executed by the step that comes the next: run step.

run subcommand (2.2)

This step runs the test case definitions. Test result is accumulated in Result Repository

report subcommand (2.3)

A step to compile human readable report from the output of the previous step (run).

Test Repository

Specified by --test-srcdir. A directory under which users store test definition files, typically yaml++ files, as source.

Compiled Test Repository

Specified by --test-workdir. A directory under which files generated during the execution of preprocessor subcommand.

Result Repository

Specified by --test-reportdir. A directory under which test report files are stored. Executor stores testreport.json here and report renderer reads it and generates testreport.adoc hre from the JSON file.

Test Report

testreport.adoc

References

top

open this page in a new window

commandunit is developed on Linux Platform (Ubuntu22.04)

Requirements

build.sh

Building, testing, generating doc, and releasing can be done with a script build.sh. It takes an array of subcommands. Subcommands can be classified into two groups, which are "grouped subcommands" and "basic subcommands". A "grouped subcommand" is interpreted as a sequence of subcommands and executed by build.sh one by one.

If you don’t specify any subcommand:

$ ./build.sh

It will be interpreted that BUILD grouped subcommand is specified.

Basic Subcommands

Parameters

A basic subcommand can take parameters. There are three targets: source, snapshot, and release. source means that the subcommand handles source files. For instance, if the target is source, the test subcommand uses source files for test. If it is snapshot, a snapshot docker image will be used. If release, a docker image to be released will be used.

build

Builds a docker image from a source.

Only snapshot and release can be specified for the first parameter.

clean

Removes all the working files generated by this build script. Note that files created by generate-src will not be removed since they are a part of the source tree, not working files.

This subcommand doesn’t take any parameters.

dummy

This subcommand does nothing. Useful for testing a change made in build.sh.

This subcommand doesn’t take any parameters.

fail

This subcommand fails always. Useful for testing a change made in build.sh.

This subcommand doesn’t take any parameters.

coverage

This command executes test and generates a coverage report. Generated coverage report will be found under out/testreport-coverage directory.

This subcommand doesn’t take any parameters.

doc

Generates documentation under working directory from asciidoc (.adoc) files under OUTDIR/site/adoc directory, whose contents are copied from src/site/adoc directory. The generated documentation (.html files) will be found under OUTDIR/doc directory.

This subcommand doesn’t take any parameters.

generate-src

A subcommand to generate "source" files based on project information such as the current target development version, repository URL, etc. Typically executed at the last of RELEASE subcommand.

This subcommand doesn’t take any parameters.

prepare

Copies the source tree (SRCDIR) to the working directory (OUTDIR).

This subcommand doesn’t take any parameters.

publish-doc

Pushes the generated documents to a branch of the git repository, the branch (.repo.documentation.docbranch attribute) and repository URL (.repo.url attribute) can be specified in build_info.json.

This subcommand doesn’t take any parameters.

push

Pushes a docker image to the docker repository.

The first parameter specifies snapshot or released image to be pushed.

test

Executes unit test suites stored under out/test/scripts.

The first parameter specifies the type of the commandunit executable used for the test execution. source means the scripts before dockerized, snapshot is a docker image for snapshot, and release is a docker image to be released.

release-precheck

Performs a "pre-check" for the entire release procedure. It checks if there is no uncommitted changes, un-pushed commits, and you are on the main branch.

release-postmortem

Some files in the source tree of commandunit should be generated in an automated way, otherwise they will become redundant.

Grouped Subcommands

Grouped subcommands do not take any parameters.

BUILD

A basic "build" command. It generates documentation and execute tests.

$ ./build.sh BUILD
Included Subcommands

clean, prepare, doc, and test

DOC

$ ./build.sh DOC

A subcommand to build documentation under working directory. Generated HTML files will be found under out/doc directory.

Included Subcommands

clean, prepare, and doc

TEST

$ ./build.sh TEST

Executes the test suite.

Included Subcommands

clean, prepare, and test

COVERAGE

$ ./build.sh COVERAGE
Included Subcommands

clean, prepare, and coverage

PACKAGE

$ ./build.sh PACKAGE
Included Subcommands

clean prepare, test:::true, build:snapshot, and test:snapshot::true

DEPLOY

$ ./build.sh DEPLOY
Included Subcommands

PACKAGE and push:snapshot

RELEASE

$ ./build.sh RELEASE

A subcommand for releasing. Note that a check will be done at the beginning of this subcommand. If something not pushed on your local, this subcommand fails (release-precheck). Also, note that documentation is not published by this subcommand, and you need to do it separately using PUBLISH_DOC subcommand.

Included Subcommands

clean, release-precheck, prepare, test:::true, build:release, test:release::true, push:release, and release-postmortem.

PUBLISH_DOC

$ ./build.sh PUBLISH_DOC

Repository

The source code repository of commandunit is found here.

Tags

Each released commit has a tag: v{Major Version}.{Minor Version}. For the commit whose "wrapper script" was verified most recently has a tag wrapper-verified.

Branches

The documentation is stored in docs branch and exposed to the public through dakusui.github.io/commandunit. Check also PUBLISH_DOC subcommand of the build script.

Directory Structure

PROJECT_BASEDIR/                                      (1)
  buildtools/          A directory for build tools.
  out/                 A directory for working files. (2)
    doc/                                              (3)
    main/
      scripts/                                        (4)
    site/
      adoc/                                           (5)
  src/                 A directory for source tree    (6)
    docker/
    main/
    site/
    test/
  build.sh             A build script for this project.
  build_info.json      A project definition file.

  • <1>: project base directory.

  • <2>: Specified by environment variable OUTDIR.

  • <3>: DOCDIR.

  • <4>: EXEC_BASEDIR.

  • <5>: DOC_SRCDIR.

  • <6>: Specified by environment variable SRCDIR.

build_info.json

build_info.json example
{
  "projectName": "commandunit",
  "homepage": "https://github.com/dakusui/commandunit",
  "docker": {
    "user": "dakusui",
    "hostFsRootMountPoint": "/var/lib/commandunit"
  },
  "version": {
    "latestReleased": {
      "major": 1,
      "minor": 25
    },
    "target": {
      "major": 1,
      "minor": 26
    }
  },
  "repo": {
    "url": "https://github.com/dakusui/commandunit.git",
    "documentation": {
      "docbranch": "docs"
    },
    "release": {
      "branch": "main"
    }
  }
}

Build-time Environment Variables

LATEST_RELEASED_VERSION

The most recent released version of this product.

TARGET_VERSION

The version currently under development.

PROJECT_NAME

The name of this project.

PROJECT_BASEDIR

The path to the root directory of this project.

HOMEPAGE

The URL of the project’s homepage.

SRCDIR

The path to the source directory (i.e., ${PROJECT_BASEDIR}/src) OUTDIR: The path to the working directory (i.e., ${PROJECT_BASEDIR}/out)

EXEC_BASEDIR

The directory that holds executables of this product during the build-time. (i.e., ${PROJECT_BASEDIR}/out/main/scripts)

DOCDIR

The directory to store generated documentations. (i.e., ${PROJECT_BASEDIR}/out/doc)

DOCSRCDIR

The directory to store documentation source files during a build. (i.e., ${PROJECT_BASEDIR}/out/doc)

DOCKER_REPO_NAME

A name of docker repository name. (i.e., {.docker.user}/{.project.name}).

BUILD_HOSTFSROOT_MOUNTPOINT

A path under which the root directory of the host side file system "seems" mounted. By prefixing this path to a host side path, you can calculate the corresponding path of it inside the docker container.

Known Problems

  • A build on macOS halts during test.

open this page in a new window

commandunit doesn’t work

I am on a macOS and using commandunit, but a permission problem is reported

Currently, minikube + dockerized combination seems sometimes not working on macOS (Issue-26). Please try native mode, if you see an error such as following:

INFO: Local repository path that has SSH Git URL ...
mkdir: cannot create directory '/tmp/bud': No space left on device
ERROR:(exit code:1, pipe status: 1):
  at /app/dependencies/bud/lib/core.rc:54 (abort)
I am on a macOS and using commandunit in "native" mode but it doesn’t work

In some conditions, yaml2json installed through npm doesn’t work macOS at all. There are two workarounds.

  1. Install it through brew

  2. Install yq and create a wrapper script that invokes it with the name yaml2json. The wrapper script should look like following:

#!/usr/bin/env bash
yq -o=json "${@}"

Place this file somewhere on the PATH environment variable. (t.b.d.)

if macOS yaml2json doesn’t work. → Workaround

/opt/homebrew/bin/yaml2json 0.3.0[1]

none subcommand.

macOS yaml2json doesn’t work.

References

top

open this page in a new window

Command Line Options

Following is the description of command line options of commandunit.

Command Line Options
$ commandunit --help
Usage: commandunit [WRAPPER OPTION]... [--] [OPTION]... [SUBCOMMAND]...

A wrapper function for 'commandunit' to invoke its docker image.
Followings are switching options to control the wrapper's behaviour.
Options not listed here or ones after the separator (--) are passed to the docker image directly.

Wrapper options: (1)
--native            Use the 'native' version of commandunit.  (2)
--version={VERSION} Use the specified version of commandunit. If 'snapshot' is given, a version under development is used (default: v1.24).
--debug-shell       Get the shell of the docker image. Type Ctrl-D to quit it. Development purpose only.
                    (3)
--show-image-name   Print the image name. Useful to figure out the version.
--quit              Quit before running the image. With --show-image-name, useful to figure out the image version
--help              Show this help and pass the --help option to the docker image.
--                  A separator to let this wrapper know the options after it should be passed directly to the image

Usage: commandunit [OPTION]... [SUBCOMMAND]...

Runs tests.

Sub-commands: (4)
  preprocess:
    Preprocesses test definition files (yaml++, yaml, and json++) and convert them into executable JSON test files
  run:
    Runs tests under a directory specified by --test-workdir and writes a report file: testreport.json under a
    directory specified by --test-reportdir.
  report:
    Reads a file testreport.json under directory specified by --test-reportdir and renders a report file (testreport.adoc)
    under the same directory.
  none:
    Does nothing.

Options: (* - for development use only)

 -h, --help            show this help
 -p, --parallel        execute the tests in parallel
 -f, --filter          filter tests with the specified regular expression(default:'.*')
     --commandunit-dir directory to store config and data files (default: {test-srcdir}/.commandunit)
     --project-name    set project name of the test execution. used as the report's title(default:'unknown')
     --test-srcdir     specify directory under which tests are stored (default: current directory/src/test)
     --test-workdir    specify directory under which commandunit writes internal data* (default: current directory + /target/commandunit/work)
     --test-reportdir  write test reports under the specified directory* (default: current directory + /target/commandunit/report)
     --ignore-mtime    ignore mtime and forcibly compile tests (5)
     --clean           clean working directory
     --tapview         show test progress with 'tapview'

Examples:
  commandunit                      Run tests found under current directory in sequential mode.
  commandunit --test-srcdir=DIR    Run tests found under DIR in sequential mode.
  commandunit --test-srcdir=DIR -p Run tests found under DIR in parallel mode.

- Test Anything Protocol: <https://testanything.org/>
- documentation: <https://dakusui.github.io/commandunit/>
- github project: <https://github.com/dakusui/bud>
- tapview: <https://gitlab.com/esr/tapview>

  • <1>: Check Wrapper Options and Normal Options section for more detail.

  • <2>: "native" mode means an execution mode of commandunit, where it is executed without using docker. Check "Native" execution for more Detail.

  • <3>: This option gives you a shell inside the docker image of commandunit. Values environment variables are guaranteed to be the same as the ones, when you execute it without this position.

  • <4>: Check Sub-commands section for more detail.

  • <5>: Forcibly commandunit preprocesses the test source files and intermediate files.

Wrapper Options and Normal Options

As described in the Design Detail, commandunit comes with its "wrapper" script. The wrapper has its own options, and it interprets command line arguments as wrapper options, first. If -- is present and any non-valid wrapper option is found before it, an error will be reported and commandunit will quit with non-0 exit code. The arguments that cannot be interpreted as wrapper options will be passed to the commandunit 's main part. The main part parses them as "normal" options using getopt.

"Native" execution

It is possible to use commandunit without Docker solutions. The mode is called "native" mode, although it is executed as a bash script, not a binary executable.

To execute commandunit in native mode, you can just give --native wrapper option to its command line.

Sub-commands

Diagram
Figure 1. The Processing Pipeline of the commandunit

As described in the Command Line Options, commandunit has four subcommands, which are:

  • preprocess

  • execute

  • report

  • none

preprocess, execute, and report correspond to the actions 2.1, 2.2, and 2.3 respectively in the diagram:The Processing Pipeline of the commandunit, none does nothing, and it is useful to use in combination with --clean option, which removes all the generated files in the project.

$ commandunit --clean none

top

open this page in a new window

commandunit: Installation Handbook

Place the following script in one of directories on your PATH environment variable and give 755 permission (chmod 755).

commandunit
#!/usr/bin/env bash

function __commandunit_exec_commandunit() {
  export PROJECT_BASE_DIR="${PWD}"
  export COMMANDUNIT_PWD="${PROJECT_BASE_DIR}" # Referenced by commandunit itself.
  export COMMANDUNIT_SOURCE_DIR="${PROJECT_BASE_DIR}/src/dependencies/commandunit"
  export COMMANDUNIT_DEFAULT_VERSION="${COMMANDUNIT_DEFAULT_VERSION:-v1.29}"
  export BUD_DEBUG=enabled
  function __commandunit_user_option() {
    case "$(uname -sr)" in

    Darwin*)
      echo -n "-u" "1000:1000"
      ;;

    Linux*Microsoft*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    Linux*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    CYGWIN* | MINGW* | MSYS*)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;

    # Add here more strings to compare
    # See correspondence table at the bottom of this answer

    *)
      echo -n "-u" "$(id -u):$(id -g)"
      ;;
    esac
  }
  function __commandunit_exec_commandunit_docker() {
    local _image_name="${1}"
    shift
    # shellcheck disable=SC2086
    # shellcheck disable=SC2046
    docker run \
      $(__commandunit_user_option) \
      --env COMMANDUNIT_PWD="${_project_basedir}" \
      --env COMMANDUNIT_LOGLEVEL="${_loglevel}" \
      --env TMPDIR="/tmp" \
      -v "${_project_basedir}:${_hostfsroot_mountpoint}${_project_basedir}" \
      ${_entrypoint} \
      -i "${_image_name}" \
      "${@}"
  }
  function __commandunit_clean_cloned_commandunit() {
    local _version_name="${1}" _source_dir="${2:-${COMMANDUNIT_SOURCE_DIR}}"
    #      +-- Safeguard for a bug, where the variable becomes empty.
    #     |    Because this function removes everything under the dir.
    #     V
    if [[ "${_source_dir}/${_version_name}" == *"/src/dependencies/"* ]]; then
      if [[ -d "${_source_dir}/${_version_name}" ]]; then
        rm -fr "${_source_dir"?"}/${_version_name:?}"
      fi
    fi
  }
  function __commandunit_clone_commandunit() {
    local _git_tag_option="${1}" _version_name="${2}"
    local _out
    # shellcheck disable=SC2086
    _out="$(git clone --depth 1 ${_git_tag_option} https://github.com/dakusui/commandunit.git "${COMMANDUNIT_SOURCE_DIR}/${_version_name}" 2>&1)" || {
      echo "Failed to clone<: ${_out}>" >&2
      return 1
    }
  }
  function __commandunit_exec_commandunit_native() {
    local _version_name="${1}"
    shift
    "${COMMANDUNIT_SOURCE_DIR}/${_version_name}/src/main/scripts/bin/commandunit-main" "${@}"
  }

  local _project_basedir="${PROJECT_BASE_DIR}"
  local _hostfsroot_mountpoint="/var/lib/commandunit"
  local _docker_repo_name="dakusui/commandunit"
  local _entrypoint=""
  local _native="no"
  local _loglevel="${COMMANDUNIT_LOGLEVEL:-ERROR}"
  local _image_name _image_version="${COMMANDUNIT_DEFAULT_VERSION}" _branch_name="main" _version_name=${COMMANDUNIT_DEFAULT_VERSION} _args _show_image_name=false _i _s _quit=false _help=false
  _args=()
  _s=to_func
  for _i in "${@}"; do
    if [[ "${_s}" == to_func ]]; then
      if [[ $_i == "--version="* ]]; then
        local _v="${_i#*=}"
        if [[ "${_v}" == "snapshot" ]]; then
          # _image_version="${COMMANDUNIT_VERSION%.*}.$((${COMMANDUNIT_VERSION##*.} + 1))"
          local _default_version="${COMMANDUNIT_DEFAULT_VERSION}" # e.g. v1.24
          _image_version="${_default_version%.*}.$((${_default_version##*.} + 1))"
        else
          _branch_name="${_v}"
          _image_version="${_v}"
        fi
        _version_name="${_v}"
        # --version=      snapshot               v1.99          (none)
        # _version_name   snapshot               v1.99           v1.24
        # _image_version  v1.25                  v1.99           v1.24          (private)
        # _image_name     dakusui:v1.25-snapshot dakusui:v1.99   dakusui:v1.24
        # _branch_name    main                   v1.99           v1.24
      elif [[ $_i == "--native" ]]; then
        _native="yes"
      elif [[ $_i == "--debug-shell" ]]; then
        _entrypoint="--entrypoint=/bin/bash"
      elif [[ $_i == "--show-image-name" ]]; then
        _show_image_name=true
      elif [[ $_i == "--quit" ]]; then
        _quit=true
      elif [[ $_i == "--help" ]]; then
        _help=true
        _args+=("${_i}")
      elif [[ $_i == "--" ]]; then
        _s=to_container
      else
        _args+=("${_i}")
      fi
    else
      _args+=("${_i}")
    fi
  done
  _image_name="${_docker_repo_name}:${_image_version}"
  if [[ "${_version_name}" == "snapshot" ]]; then
    _image_name="${_image_name}-${_v}"
  fi
  if ${_show_image_name}; then
    echo "${_image_name}"
  fi
  if ${_help}; then
    echo "Usage: commandunit [WRAPPER OPTION]... [--] [OPTION]... [SUBCOMMAND]..."
    echo ""
    echo "A wrapper function for 'commandunit' to invoke its docker image.".
    echo "Followings are switching options to control the wrapper's behaviour."
    echo "Options not listed here or ones after the separator (--) are passed to the commandunit implementation directly."
    echo ""
    echo "Wrapper options:"
    echo "--native            Use the 'native' version of commandunit."
    echo "--version={VERSION} Use the specified version of commandunit. If 'snapshot' is given, a version under development is used (default: ${COMMANDUNIT_DEFAULT_VERSION})."
    echo "--debug-shell       Get the shell of the docker image. Type Ctrl-D to quit it. Development purpose only."
    echo "--show-image-name   Print the image name. Useful to figure out the version."
    echo "--quit              Quit before running the image. With --show-image-name, useful to figure out the image version"
    echo "--help              Show this help and pass the --help option to the docker image."
    echo "--                  A separator to let this wrapper know the options after it should be passed directly to the image"
    echo ""
  fi
  ${_quit} && return 0
  if [[ "${_native}" == "no" ]]; then
    __commandunit_exec_commandunit_docker "${_image_name}" "${_args[@]}"
  else
    which jq-front > /dev/null || {
      function jq-front() {
        docker run --rm -i \
          -v "${HOME}:/var/lib/jf/${HOME}" \
          -e JF_PATH_BASE="/var/lib/jf" \
          -e JF_PATH="${JF_PATH}" \
          -e JF_DEBUG=${JF_DEBUG:-disabled} \
          -e JF_CWD="$(pwd)" \
          -e COMMANDUNIT_DEPENDENCIES_ROOT="${COMMANDUNIT_DEPENDENCIES_ROOT:?Required environment variable is not set.}" \
          -e COMMANDUNIT_BUILTIN_ROOT="${COMMANDUNIT_DEPENDENCIES_ROOT:?Required environment variable is not set.}" \
          dakusui/jq-front:"${JF_DOCKER_TAG:-v0.53}" "${@}"
      } &&
      export -f jq-front
    }
    if [[ "${_version_name}" ==  "snapshot" ]]; then
      # Remove already loaded one, then... do clone
      __commandunit_clean_cloned_commandunit "${_version_name}"
    fi
    if [[ ! -e "${COMMANDUNIT_SOURCE_DIR}/${_version_name}" ]]; then
      __commandunit_clone_commandunit "--branch ${_branch_name}" "${_version_name}"
    fi
    __commandunit_exec_commandunit_native "${_version_name}" "${_args[@]}"
  fi
}

function main() {
  __commandunit_exec_commandunit "${@}"
}

main "${@}"

Or download it from here.

For more details, check commandunit-installer

top

open this page in a new window

This is a manual that provides useful information to write your own test cases using commandunit 's functionalities.

Syntax

given, when, and then structure

given

You can define an action which should be executed before when clause. If the execution of this clause fails, i.e., if it results in non-0 exit code, the rest of the test (when and then clauses) will not be executed. This corresponds to the "set-up" phase in "four phase testing" model.

when

You can define an action whose behavior should be tested in this clause. This section has the same structure as the given clause. This corresponds to the "execute" phase in "four phase testing" model.

then

This clause defines a check to be exercised for the result of when clause. This corresponds to the "verify" phase in "four phase testing" model.

As discussed later in this section, it is advised to a built-in test template: base/normal.json directly or indirectly always. If no preparation is necessary for your test, you do not need to define anything for given clause explicitly as long as you are extending it.

type attribute

commandunit specifies the structure of a test case, which has given, when, and then. However, the syntax inside the given and when clauses is user customizable, although it doesn’t need to be modified under usual usages. If you specify a value of type attribute NORMAL, it searches for handler functions from json_NORMAL.rc and scripttest.rc files. These .rc files are placed under test drivers directories. Test driver directories are under the {commandunit-dir}/.commandunit/testdrivers and {commandunit-home}/src/main/scripts/lib/testdrivers. commandunit searches for the files from them in this order.

Therefore, syntax inside given and when clauses can be differently defined based on the type attribute value. However, the only built-in currently available (base/normal.json) uses the NORMAL type, and it has sufficiently general-purposed design, we discuss the syntax defined by the NORMAL type in the rest of this document unless otherwise explicitly noted.

given and when clauses

given and when clauses have the same structure, and you can use the same set of attributes for them.

description (array)

This part is included as a comment in the auto-generated test script.

environmentVariables (object)

An object that specifies environment variables and their values. Those environment variables can be referenced from inside the elements in args array.

source (array)

Files to be sourced by the shell.

stdin (array)

An array of string, which gives to the executed command from stdin.

shell (object)

A shell with which a command specified by cmd and args.

cmd (string)

A command to be executed. A function in sourced files (see source) can also be specified.

args (array)

Arguments to be passed to the command specified by cmd attribute.

The given clause should be used to "set up" test fixture, while when clause should be used for executing the command under target itself.

then clause

exitCode (array)

An array that defines a condition which the exit code of the command line specified by when clause should satisfy. The first element is a name of a predicate and the rest of the element are arguments to be passed to the predicate. Currently available predicates are, EQUAL and NOT_EQUAL. [EQUAL, 0] is satisfied if and only if the exit code of the command in the when clause was 0. [NOT_EQUAL, 0] is satisfied if and only if the exit code of the command in the when clause was not 0.

stdout and stderr (object)

Under these attributes, you can define regular expressions that should and should not be present in them. To define a check which examines if stdout contains a regular expression: Hello world, you can do:

  stdout:
    present:
      - REGEX:Hello world

Also, to define a check, stderr doesn’t contain anything, you can do:

  stderr:
    absent:
      - REGEX:.+

Built-ins

Built-in Test Templates: base/normal.json

base/normal.json

A basic test base file.

commandunit has a built-in JSON file, base/normal.json, which you can extend to define your own tests.

Details
base/normal.json
{
  "type": "NORMAL",
  "description": [
  ],
  "given": {
    "description": [
      "This test should always be executed."
    ],
    "stdin": [
    ],
    "shell": {
      "name": "bash",
      "options": [
        "-eu",
        "-E"
      ]
    },
    "source": [
    ],
    "environmentVariables": {
      "COMMANDUNIT_BUILTIN_ROOT": "eval:string:${COMMANDUNIT_DEPENDENCIES_ROOT}"
    },
    "cmd": ":",
    "args": [
    ]
  },
  "when": {
    "description": [
    ],
    "stdin": [
    ],
    "shell": {
      "name": "bash",
      "options": [
        "-eu",
        "-E"
      ]
    },
    "source": [
    ],
    "environmentVariables": {
      "COMMANDUNIT_BUILTIN_ROOT": "eval:string:${COMMANDUNIT_DEPENDENCIES_ROOT}"
    },
    "cmd": "eval:string:$(error 'missing attribute!')",
    "args": [
    ]
  },
  "then": {
    "description": [
    ],
    "exitCode": [
      "EQUAL",
      0
    ],
    "stdout": {
      "present": [
      ],
      "absent": [
      ]
    },
    "stderr": {
      "present": [
      ],
      "absent": [
      ]
    }
  }
}

As you see, it defines attributes the framework commonly uses and their default values. In this file, type attribute at the top level is defined NORMAL and therefore, this file will be handled by functions defined in json_NORMAL.rc.

The following is an example of the usage of base/normal.json.

Test Case Example
---
"$extends":
  - base/normal.json
when:
  environmentVariables:
    SCRIPTS_DIR: "${COMMANDUNIT_PROJECT_DIR}/src/main/scripts" (1)
  source:
    - ${COMMANDUNIT_BUILTIN_ROOT}/bud/lib/core.rc
    - ${SCRIPTS_DIR}/target_lib.rc                             (1)
  cmd: cat
  args:
    - ${SCRIPTS_DIR}/hello.txt
then:
  exitCode:
    - EQUAL
    - 0
  stdout:
    present:
      - REGEX:Hello world
  stderr:
    absent:
      - REGEX:.+

<1>: environmentVariable array is designed to be evaluated at the beginning of given and when clause. Thus, you can reference the environment variables in the source and args arrays.

Built-in Libraries

commandunit comes with built-in libraries.

Functions listed in this section are guaranteed to be compatible across commandunit versions. They are called "public functions". In case you find any incompatibility in them, please file a bug ticket. However, the libraries are also used by the commandunit itself and some of them are designed for internal use only. Specifications of such internal functions may be changed in future, and it is advised not to use them.

To use the public functions, you first need to source it using an environment variable COMMANDUNIT_BUILTIN_ROOT.

source "${COMMANDUNIT_BUILTIN_ROOT}/bud/lib/core.rc"

public functions are defined one of the following libraries.

  • bud/lib/core.rc:: Core functions such as printing message to stderr.

  • bud/lib/arrays.rc:: Functions to handle array data.

  • bud/lib/json.rc:: Functions to handle/create JSON nodes.

Note that only functions listed in this section are public. The other functions found in the libraries above are not public and designed for internal use of commandunit. Programmatically, you can use such non-public functions, but their compatibilities are not guaranteed and you need to use them at your own risk.

bud/lib/core.rc

message

Prints given arguments to stderr.

assert_that

do eval for the second and the following arguments. If it resulted in non-zero exit code, the first and the value passed to the eval will be printed in stderr and then the process will be aborted.

print_stacktrace

Prints a current stack trace to stderr.

abort

Aborts the current process with exit code 1.

bud/lib/arrays.rc

join_by

Joins the second and the following arguments joining with the first argument. The resulting string will be printed to stdout.

array_contains

Search for the first argument from the second and the following argument. The search is done by using a grep -q -E command. If no element matches the keyword(the first argument), non-0 value will be returned based on grep command’s behavior.

bud/lib/json.rc

to_json_array

Converts given arguments in to a JSON array.

json_value_at

Prints a JSON node found at the path specified by the second argument in the JSON node specified by the first argument.

json_has_key

Checks if a key (second argument) is found at the path (third argument. If omitted, . will be used) in a JSON object given by the first argument. If and only if input is well-formed JSON and the key is found, true will be printed. If the input is well-formed JSON but the key doesn’t exist, false will be printed. If the input is not well-formed, non-0 value will be returned.

json_type_of

Prints a type of JSON node specified by the first argument. The type will be one of: object, array, string, number, and null If non-welformed JSON node is given, the execution will be aborted.

json_object_merge

"Merges" two JSON objects specified by the first and the second arguments. If the same key exists in both the first and second arguments, the one from the second will override the one from the other.

json_array_append

Prints an array created by appending a JSON array specified by the second argument to another specified by the first argument.

Built-in Environment Variables

The se environment variables can be used in your test cases without explicit declaration.

COMMANDUNIT_PROJECT_DIR

The top level directory of you project. The actual value can be different depending on whether you are using commandunit in "native" mode or not.

COMMANDUNIT_BUILTIN_ROOT

The directory under which built-ins are stored. It points a directory under {commandunit-home}/src/main/scripts/lib

top

open this page in a new window

Software Components

commandunit is tested on following platforms.

On Linux

  • Distribution: Ubuntu 22.04

  • bash: GNU bash, version 5.1.16(1)-release (x86_64-pc-linux-gnu)

Dockerized mode

  • git (tested with 2.34.1)

  • dockerd: ???

"native" mode

  • git: (tested with 2.34.1)

  • jq-front: 0.54, Dockerized

  • jq: jq-1.5-1-a5b5cbe or later

  • yaml2json: 0.3.0, installed through npm install yamljs

On macOS (x86_64)

  • bash: GNU bash, version 5.1.12(1)-release (x86_64-apple-darwin20.6.0)

Dockerized mode

  • docker: Docker version 23.0.5, build bc4487a (macOS)

  • dockerd: minikube version: v.1.31.2

"native" mode

$ cat $(which yaml2json)
#!/usr/bin/env bash
yq -o=json "${@}"

On macOS (Apple Silicon)

Dockerized mode

  • git (tested with 2.34.1)

  • docker: Docker version 23.0.5, build bc4487a (macOS)

  • dockerd: Docker Desktop 4.19.0 (106363)

"native" mode

  • git (tested with 2.34.1)

  • jq-front (0.54, Dockerized)

  • jq (jq-1.6)

  • yaml2json (0.3.0, installed through brew, not through npm)

top

open this page in a new window

The commandunit is a pluggable and flexible testing framework (test harness), whose testing description language is based on "JSON" (and "YAML") format.

Introduction

Automated tests can become easily repetitive and not uniformed very often. commandunit addresses this problem for your UNIX-based CLI tools. It allows you to define your test cases in YAML but with the help of jq-front[1]. With this tool, your test will look like this:

test-contains_true.yaml++
---
"$extends":
  - base/normal.json
when:
  description:
    - Run 'array_contains' with the first argument contained by the rest.
  source:
    - /path/to/arrays.rc
  cmd: array_contains
  args:
    - hello
    - hello
    - world
then:
  exitCode:
    - EQUAL
    - 0
  stdout:
    absent:
      - REGEX:.+
  stderr:
    absent:
      - REGEX:.+

Design Concept

Test definitions are repetitive. But any single test is not (should not be) identical to any other one. In generic programming languages have features, we can address it with their features such user define functions, data types, inheritance, and others. Those are not available in popular data notations. commandunit doesn’t try to solve it by itself, but by using jq-front[1], it separates this concern from the test management and execution model.

For more detail, check Design Concept.

Design Detail

To implement the concepts shown in the previous section, commandunit has a pipeline that takes jq-front powered yaml as its input and produces JSON files that defines test cases.

For more detail, check Design Detail.

System Requirements

For the requirements of commandunit for your system (or software components and their version used in testing of commandunit), check System Requirements.

User Manuals

Installation Manual

You can install commandunit easily with its "wrapper" script. Check Installer site. For more details of installation, please check Installation and Quick Start Manual.

Command-line Manual

commandunit comes with a "wrapper" script, which abstracts an invocation step of it so that "native" mode and "dockerized" mode look the same. The wrapper has its own options.

For the detail of commandunit 's options the wrapper’s options, check Command Line Manual.

Programming Manual

Like JUnit and other testing tools, commandunit has its own programming model. Also, it assumes a directory structure, which is designed to be compatible with Maven[2].

You can start using commandunit by cloning, trying and modifying an example project. But for more details, such as syntax of the tool, you can check Programming Manual.

FAQ

The list of frequently asked questions and their answers is available here: FAQ.

References

open this page in a new window