neomake

neomake is a task runner CLI utility that acts as a modern alternative to known utilities like Makefiles.

Project state

neomake is released and stable. It is actively maintained and used in production.

Features

• DAG execution
  Tasks are run in nodes. Nodes can be chained together to create a DAG. Simply specify all the nodes you want executed and it will automagically create the DAG based on the defined dependencies.
• Parallel task execution
  The DAG generations are called stages. Stages are executed in sequence while all tasks inside of the stages are executed in parallel. Workloads are executed in OS threads. The default size of the threadpool is 1 but can be configured.
• Matrix invocations
  Specify n-dimensional matrices that are used to invoke the node many times. You can define dense and sparse matrices. The node will be executed for every element in the cartesion product of the matrix.
• YAML
  No need for any fancy configuration formats or syntax. The entire configuration is done in an easy to understand yaml file, including support for handy features such as YAML anchors (and everything in the YAML 1.2 standard).
• Customizable environment
  You can customize which shell or program (such as bash or python) neomake uses as interpreter for the command. You can also specify arguments that are provided per invocation via the command line, working directories and environment variables on multiple different levels. Generally, values defined in the inner scope will extend and replace the outer scope.
• Plan & execute
  Supporting execution of commands in two stages. First plan and render the entire execution. Then invoke the execution engine with the plan. This way, plans can be stored and reviewed before execution.

Installation

neomake is distributed through cargo.

1. For the latest stable version:
   cargo install neomake
2. For the bleeding edge master branch:
   cargo install --git https://github.com/replicadse/neomake.git

Example

First, initialize an example workflow file with the following command.

neomake workflow init -tpython

Now, execute the count node. Per default, neomake will only use exactly one worker thread and execute the endless embedded python program.

neomake plan -ccount | neomake x

In order to work on all 4 desired executions (defined as 2x2 matrix), call neomake with the number of worker threads desired. Now you will see that the 4 programs will be executed in parallel.

neomake plan -ccount | neomake x -w4

Graph execution

Execute nodes as follows.

neomake plan -f ./test/.neomake.yaml -c bravo -c charlie -oron | neomake execute -fron

Nodes can define an array of dependenies (other nodes) that need to be executed beforehand. All node executions are deduplicated so that every node is only executed exactly once if requested for invocation or as a prerequisite on any level to any node that is to be executed. Alongside the ability to specify multiple node to be executed per command line call, this feature allows for complex workflows to be executed.
Let's assume the following graph of nodes and their dependencies:

neomake ls

---
nodes:
  - name: A
  - name: B
  - name: C
    pre:
      - A
  - name: D
    pre:
      - B
  - name: E
    pre:
      - A
      - D

In words, A and B are nodes without any prerequisites whereas C depends on A and D depends on B. Notably, E depends on both A and D. This means that E also transiently depends on any dependencies of A ({}) and D ({B}).

It is also possible to get a simple description of the workflow to be executed.

neomake describe -cC -cE

---
stages:
  - - A
    - B
  - - D
  - - E

Stages need to run sequentially due to their nodes dependency on nodes executed in a previous stage. Tasks inside a stage are run in parallel (in an OS thread pool of the size given to the worker argument). neomake is also able to identify and prevent recursions in the execution graph and will fail if the execution of such a sub graph is attempted.

Why

Why would someone build a task runner if there's many alternatives out there? A few of the most well known task running utilities / frameworks are (non exhaustive):

• make (Makefile) - the original as well as many different implementations
• Earthly (Earthfile) - executing tasks inside of containers
• pyinvoke (tasks.py) - executing tasks from within python scripts

I built this utility because all of the alternatives I have tried, including the ones listed above were lacking some features. I was basically looking for a subset of the functionality which the GitLab pipelines provide incl. features such as matrix builds and more. Especially things like invoking commands in many locations, parallelizing tasks, easy parameterization and a few more.

Example configuration

version: "0.5"

env:
  capture: "^(CAPTURE)$"
  vars:
    DEFAULT_ENV_VAR: default var
    OVERRIDE_ENV_VAR_0: old e0
    OVERRIDE_ENV_VAR_1: old e1

.anchor: &anchor |
  printf "test anchor"

nodes:
  python:
    description: This is an example of using multiple execution environments (shell and python).
    shell:
      program: bash
      args:
        - -c
    matrix:
      dense:
        dimensions:
          - - env:
                PRINT_VAL: value 0
            - env:
                PRINT_VAL: value 1
    tasks:
      - shell:
          program: python
          args:
            - -c
        script: print('yada')
      - script: printf "$PRINT_VAL"
      - script: *anchor

  a:
    matrix:
      dense:
        drop: "^(0,0,0)$"
        dimensions:
          - - env:
                VA: A0
            - env:
                VA: A1
          - - env:
                VB: B0
            - env:
                VB: B1
            - env:
                VB: B2
          - - env:
                VC: C0
            - env:
                VC: C1
    tasks:
      - script: echo "$VA $VB $VC"
  b:
    pre:
      - a
    tasks:
      - script: echo "b"
  c:
    matrix:
      sparse:
        keep: "^(1,1)$"
        dimensions:
          - - env:
                VA: A0
            - env:
                VA: A1
          - - env:
                VB: B0
            - env:
                VB: B1
    pre:
      - b
    tasks:
      - script: |
          echo "$VA $VB"

  minimal:
    tasks:
      - script: echo "minimal"

  error:
    tasks:
      - script: exit 1

  graph:
    pre:
      - minimal
      - a
      - b
    tasks: []

  test:
    matrix:
      dense:
        dimensions:
          - - env:
                OVERRIDE_ENV_VAR_0: new e0
    tasks:
      - env:
          OVERRIDE_ENV_VAR_1: new e1
        script: |
          set -e

          echo "$DEFAULT_ENV_VAR"
          sleep 1
          echo "$OVERRIDE_ENV_VAR_0"
          sleep 1
          echo "$OVERRIDE_ENV_VAR_1"
          sleep 1
          echo "A"
          sleep 1
          echo "B"
          sleep 1
          echo "C"
          sleep 1
          echo "D"
          sleep 1
          echo "{{ args.test }}" # this will require an argument to be passed via '-a args.test="some-argument"'
          sleep 1
          unknown-command
          echo "too far!"

For more examples, call neomake workflow init --help or look at the schema with neomake workflow schema.

Schema

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "title": "Workflow",
  "description": "The entire workflow definition.",
  "type": "object",
  "required": [
    "nodes",
    "version"
  ],
  "properties": {
    "env": {
      "description": "Env vars.",
      "anyOf": [
        {
          "$ref": "#/definitions/Env"
        },
        {
          "type": "null"
        }
      ]
    },
    "nodes": {
      "description": "All nodes.",
      "type": "object",
      "additionalProperties": {
        "$ref": "#/definitions/Node"
      }
    },
    "version": {
      "description": "The version of this workflow file (major.minor).",
      "type": "string"
    }
  },
  "definitions": {
    "Env": {
      "description": "Environment variables definitions.",
      "type": "object",
      "properties": {
        "capture": {
          "description": "Regex for capturing and storing env vars during compile time.",
          "type": [
            "string",
            "null"
          ]
        },
        "vars": {
          "description": "Explicitly set env vars.",
          "type": [
            "object",
            "null"
          ],
          "additionalProperties": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false
    },
    "Matrix": {
      "description": "An entry in the n-dimensional matrix for the node execution.",
      "oneOf": [
        {
          "type": "object",
          "required": [
            "dense"
          ],
          "properties": {
            "dense": {
              "type": "object",
              "required": [
                "dimensions"
              ],
              "properties": {
                "dimensions": {
                  "type": "array",
                  "items": {
                    "type": "array",
                    "items": {
                      "$ref": "#/definitions/MatrixCell"
                    }
                  }
                },
                "drop": {
                  "type": [
                    "string",
                    "null"
                  ]
                }
              },
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        },
        {
          "type": "object",
          "required": [
            "sparse"
          ],
          "properties": {
            "sparse": {
              "type": "object",
              "required": [
                "dimensions"
              ],
              "properties": {
                "dimensions": {
                  "type": "array",
                  "items": {
                    "type": "array",
                    "items": {
                      "$ref": "#/definitions/MatrixCell"
                    }
                  }
                },
                "keep": {
                  "type": [
                    "string",
                    "null"
                  ]
                }
              },
              "additionalProperties": false
            }
          },
          "additionalProperties": false
        }
      ]
    },
    "MatrixCell": {
      "description": "An entry in the n-dimensional matrix for the node execution.",
      "type": "object",
      "properties": {
        "env": {
          "type": [
            "object",
            "null"
          ],
          "additionalProperties": {
            "type": "string"
          }
        }
      },
      "additionalProperties": false
    },
    "Node": {
      "description": "An individual node for executing a task batch.",
      "type": "object",
      "required": [
        "tasks"
      ],
      "properties": {
        "description": {
          "description": "A description of this node.",
          "type": [
            "string",
            "null"
          ]
        },
        "env": {
          "description": "Env vars.",
          "anyOf": [
            {
              "$ref": "#/definitions/Env"
            },
            {
              "type": "null"
            }
          ]
        },
        "matrix": {
          "description": "An n-dimensional matrix that is executed for every item in its cartesian product.",
          "anyOf": [
            {
              "$ref": "#/definitions/Matrix"
            },
            {
              "type": "null"
            }
          ]
        },
        "pre": {
          "description": "Reference nodes that need to be executed prior to this one.",
          "type": [
            "array",
            "null"
          ],
          "items": {
            "type": "string"
          }
        },
        "shell": {
          "description": "Custom program to execute the scripts.",
          "anyOf": [
            {
              "$ref": "#/definitions/Shell"
            },
            {
              "type": "null"
            }
          ]
        },
        "tasks": {
          "description": "The tasks to be executed.",
          "type": "array",
          "items": {
            "$ref": "#/definitions/Task"
          }
        },
        "workdir": {
          "description": "Custom workdir.",
          "type": [
            "string",
            "null"
          ]
        }
      },
      "additionalProperties": false
    },
    "Shell": {
      "description": "A task execution environment.",
      "type": "object",
      "required": [
        "args",
        "program"
      ],
      "properties": {
        "args": {
          "description": "Custom args (like \\[\"-c\"\\]).",
          "type": "array",
          "items": {
            "type": "string"
          }
        },
        "program": {
          "description": "The program (like \"/bin/bash\").",
          "type": "string"
        }
      },
      "additionalProperties": false
    },
    "Task": {
      "description": "An individual task.",
      "type": "object",
      "required": [
        "script"
      ],
      "properties": {
        "env": {
          "description": "Explicitly set env vars.",
          "type": [
            "object",
            "null"
          ],
          "additionalProperties": {
            "type": "string"
          }
        },
        "script": {
          "description": "The script content to execute. Can contain handlebars placeholders.",
          "type": "string"
        },
        "shell": {
          "description": "Custom program to execute the scripts.",
          "anyOf": [
            {
              "$ref": "#/definitions/Shell"
            },
            {
              "type": "null"
            }
          ]
        },
        "workdir": {
          "description": "Custom workdir.",
          "type": [
            "string",
            "null"
          ]
        }
      },
      "additionalProperties": false
    }
  }
}

Command-Line Help for neomake

This document contains the help content for the neomake command-line program.

Command Overview:

• neomake↴
• neomake man↴
• neomake autocomplete↴
• neomake workflow↴
• neomake workflow init↴
• neomake workflow schema↴
• neomake plan↴
• neomake execute↴
• neomake describe↴
• neomake list↴

neomake

A makefile alternative / task runner.

Usage: neomake [OPTIONS] <COMMAND>

Subcommands:

• man — Renders the manual.
• autocomplete — Renders shell completion scripts.
• workflow — Workflow related subcommands.
• plan — Creates an execution plan.
• execute — Executes an execution plan.
• describe — Describes which nodes are executed in which stages.
• list — Lists all available nodes.

Options:

• -e, --experimental <EXPERIMENTAL> — Enables experimental features.

neomake man

Renders the manual.

Usage: neomake man --out <out> --format <format>

Options:

• -o, --out <OUT>

• -f, --format <FORMAT>

  Possible values: manpages, markdown

neomake autocomplete

Renders shell completion scripts.

Usage: neomake autocomplete --out <out> --shell <shell>

Options:

• -o, --out <OUT>

• -s, --shell <SHELL>

  Possible values: bash, zsh, fish, elvish, powershell

neomake workflow

Workflow related subcommands.

Usage: neomake workflow [COMMAND]

Subcommands:

• init — Initializes a new template workflow.
• schema — Renders the workflow schema to STDOUT.

neomake workflow init

Initializes a new template workflow.

Usage: neomake workflow init [OPTIONS]

Options:

• -t, --template <TEMPLATE> — The template to init with.

  Default value: min

  Possible values: min, max, python

• -o, --output <OUTPUT> — The file to render the output to. "-" renders to STDOUT.

  Default value: ./.neomake.yaml

neomake workflow schema

Renders the workflow schema to STDOUT.

Usage: neomake workflow schema

neomake plan

Creates an execution plan.

Usage: neomake plan [OPTIONS]

Options:

• --workflow <WORKFLOW> — The workflow file to use.

  Default value: ./.neomake.yaml

• -n, --node <NODE> — Adding a node to the plan.

• -r, --regex <REGEX> — Adding a node to the plan.

• -a, --arg <ARG> — Specifies a value for handlebars placeholders.

• -o, --output <OUTPUT> — Specifies the output format.

  Default value: yaml

  Possible values: yaml

neomake execute

Executes an execution plan.

Usage: neomake execute [OPTIONS]

Options:

• -f, --format <FORMAT> — The format of the execution plan.

  Default value: yaml

  Possible values: yaml

• -w, --workers <WORKERS> — Defines how many worker threads are created in the OS thread pool.

  Default value: 1

• --no-stdout <NO-STDOUT> — Disables any output to STDOUT. Useful for preventing leakage of secrets and keeping the logs clean.

• --no-stderr <NO-STDERR> — Disables any output to STDERR. Useful for preventing leakage of secrets and keeping the logs clean.

neomake describe

Describes which nodes are executed in which stages.

Usage: neomake describe [OPTIONS]

Options:

• --workflow <WORKFLOW> — The workflow file to use.

  Default value: ./.neomake.yaml

• -n, --node <NODE> — Adding a node.

• -r, --regex <REGEX> — Adding a node to the plan.

• -o, --output <OUTPUT> — The output format.

  Default value: yaml

  Possible values: yaml

neomake list

Lists all available nodes.

Usage: neomake list [OPTIONS]

Options:

• --workflow <WORKFLOW> — The workflow file to use.

  Default value: ./.neomake.yaml

• -o, --output <OUTPUT> — The output format.

  Default value: yaml

  Possible values: yaml

This document was generated automatically by clap-markdown.

Adrs

1: Usage of ADRs

This project uses ADRs. ADRs are documented in this repository and format. In doubt, the documents in this repository and documentation are to be considered as the single source of truth.

2: Usage of keywords

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document and in all documents in it's subtree as well as all documents that are directly related to this document are to be interpreted as described in https://tools.ietf.org/html/bcp14[BCP 14], https://tools.ietf.org/html/rfc2119[RFC2119] and https://tools.ietf.org/html/rfc8174[RFC8174] when, and only when, they appear in all capitals, as shown here.

3: Versioning

This project makes use of https://semver.org[the semver v2 versioning scheme] for all parts of the official public API. The public stable API is a subset of all available features (see link:/neomake/docs/adrs/4-experimental-flag[this page]).

4: Experimental flag

There is a application level argument (flag) experimental (-e | --experimental) that indicates that experimental features can now be accessed. This flag explicitly marks features that are NOT part of the official public API and therefore NOT considered when applying the versioning scheme (see link:/neomake/docs/adrs/3-versioning[ADR 3]). + This flag is designed to be used with and therefore CAN be used with link:/neomake/docs/adrs/5-feature-flags[feature Flags as specified in ADR 5].

5: Feature flags

This project makes use of cargo feature flags. Feature flags count as part of the public API and are therefore to be considered when applying the version rules IF NOT marked as experimental (see link:/neomake/docs/adrs/4-experimental-flag[ADR 4]). + All feature flags MUST be documented in an appropriate manner in the documentation.