# Straw Boss CI

* Pipeline: series of named steps.
* Step: named set of operations (make, script, something extensible).
  Optionally names an artifact that is the result of the step.
* Artifact: resulting output from a step. strawboss may skip step execution if
  it already has an artifact cached from that step.
* Configuration is two-part. Pipeline, step, and artifact definition are part
  of the project configuration (.strawboss.json? yaml?). Environment
  configuration lives on the strawboss server (supplies DB info, etc.).
* REST API?
* Step execution happens within the root directory of a fresh copy of the repo.
  Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
  the fresh copy is checked out at that reference.


## Projects

StrawBoss must know about the projects it builds. For each project, StrawBoss
must be configured with a unique project name and a URL or repo-specification
that can be passed to `git clone` to clone the project.

For more details see the
[Service Project Definition](#service-project-definition) section.

## Artifacts Repo

The artifacts repository is a directory on the filesystem used to store
build artifacts and information about build status. The top-level artifacts
repo directory contains project build directories: one directory for every
project that StrawBoss has built. Each project directory contains multiple
sub-directories. Each 

## Configuration

### StrawBoss configuration file

StrawBoss expects to find `strawboss.config.json` in the working directory of
the `strawboss` executable. This is the configuration file for StrawBoss
itself. The contents are expected to be a valid JSON object. The top level keys
are:

* `artifactsRepo`: A string denoting the path to the artifacts repository
  directory.

* `authSecret`: Secret key used to sign JWT session tokens.

* `users`: the array of user definition objects. Each user object is required
  to have `username` and `hashedPwd` keys, both string.

* `projects`: an array of project definitions (detailed below).

All are required.

#### Service Project Definition

Project definitions are JSON objects with the following keys:

* `name` *(required)*: A unique name for the project. Names are case sensitive.
  There are no hard limits on names but the recommended convention is
  alpha-numeric using dashes or underscores for word separation (e.g.
  `foo-confabulator` or `web_refractulator`, etc.).

* `repo` *(required)*: A valid `git` URL or repo-specification that can be
  passed to `git clone` to clone the project.

* `defaultBranch` *(optional)*: If a run request is submitted without
  specifying a branch or commit reference, use this branch *(defaults to
  `master`)*.

* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file
  (see the [Project Configuration](#project-configuration) section). If this is
  a relative path it is resolved relative to the project root directory.
  *(defaults to `strawboss.json`)*.

* `envVars` *(optional)*: an object whose key-value pairs are expected to be
  strings. Each pair will be added as an environment variable to the runtime
  environment of builds for this project. *(defaults to `{}`)*.

### Project Configuration

The build configuration for each project lives in the project itself. In the
root directory of the project repo StrawBoss expects to find a project
configuration file named `strawboss.json` (this location can be configured, see
the [Service Project Definition](#service-project-definition) section). The
project configuration file tells StrawBoss what steps of a project are
available to be built, and how to initiate the build for each step. Steps can
be dependant on each other, as long as they do not define any circular
references.

The contents of a project configurations files is expected to be a valid JSON
object. The top level keys are:

* `name` *(required)*: the unique name for the project. This should be the same
  name as configured in the StrawBoss service configuration file.

* `steps` *(required)*: an object of step definitions. Each of this object's
  keys is the name of a step. The corresponding value is the step definition.
  (see the [Step Definition](#step-definition) section for details).

* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible)
  that is expected to print the current version of the project on `stdout`.
  *(defaults to `git describe --tags --always`)*.

####  Step Definition

Step definitions are JSON objects with the following keys:

* `workingDir` *(optional)*: the working directory (relative to the project
  root folder) for all commands executed as part of this step. *(defaults to
  `'.'`, the project root directory)*.

* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
  `sh`)*

* `cmdInput` *(optional)*: an array of string that will be concatenated with
  newlines separating each string and piped as input to the command for this
  step. It is possible to include newlines in the strings, but not required.
  Environment variables included in this input are resolved *before* it is
  passed to `stdin` as `stepCmd` does not have to be a shell and is not
  expected to be able to do this interpolation itself. *(defaults to `[]`)*.

* `artifacts` *(optional)*: an array of strings representing artifacts that
  will be generated by this step and should be saved for future use. Each
  string should represent a path to a file (relative to the project root
  directory). StrawBoss will gather all of these files and store them in the
  artifacts repo for this step and build. These paths may contain environment
  variable references. *(defaults to `[]`)*.

*  `depends` *(optional)*: an array of strings representing step names upon
   which this step depends.  *(defaults to `[]`)*.

* `expectedEnv` *(optional)*: an array of strings representing environment
  variable keys that should be present in the environment when executing this
  step. If there is an environment variable that is listed here but not present
  in the build environment it will cause StrawBoss to fail this step.
  *(defaults to `[]`)*.

* `dontSkip` *(optional)*: boolean. StrawBoss remembers the steps it has run
  and caches the artifacts generated by them. Future builds for the same
  version that request this step normally will not cause StrawBoss to re-run
  the step. If `dontSkip` is set to `true`, the output of this step will always
  be run when it is referenced, regardless of previous cached results.

## Build Process

When performing a build, StrawBoss:

1.  creates a temporary workspace for this build
2.  clones the repo into the workspace
3.  checkout the revision or branch requested for this run
4.  load the project's StrawBoss configuration file.
5.  merge environment variables defined in the project configuration
6.  run `versionCmd` to get the current project version. The result is stored
    in the `VERSION` environment variable.
7.  check the environment variables against `expectedEnv`
8.  check that all the steps named in `depends` have already been run and run
    them if they have not. For each step named in `depends` an environment
    variable is added named `<step-name>_DIR` that contains the absolute path to
    the artifacts repo for that step at this version. This is intended to be
    used to reference artifacts from other steps, e.g.
    `${build_DIR}/site-contents.zip`.
9.  `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
    are resolved and the resulting string are fed line-by-line into the process
    as `stdin`.
10. the files named in `artifacts` are copied into the artifacts repo for this
    step and version.

## Architecture

The following describes the internal architecture of StrawBoss. This section is
written primarily for those wanting to work on StrawBoss itself and is not
nessecary for the setup and use of StrawBoss.

### Artifacts repo

In the top-level of the artifacts repo are project directories.

#### Project Directories

Project directories contain step directories and cached project configuration
files.

##### Cached configuration files.

The cached project configuration files follow this naming convention:
`configuration.<version>.json`. These cached versions of the project
configration are only intended to be used in cases where StrawBoss is not
building anything and doesn't check out a copy of the repo. For example, when a
client queries the REST API for the list of steps in a project, StrawBoss will
consult the most recently modified cached copy of the project configuration
rather than cloning the entire repo just to answer this question.  Whenever
StrawBoss has a copy of the repo, it should look for the actual configuration
file in that version of the repo instead of consulting the cached configuration
files. When determining the "most recent" cached copy, StrawBoss uses the
modification time of the files, again to avoid cloning the repo. API access to
project configurations in this manner is intended as a convenience. The actual
project configuration in the project repository should be considered the source
of truth.

##### Step and Version Directories

A step directory contains a list of version directories using the version
string as the directory names and a list of status JSON files also named after
the version (`<version>.status.json`).

A version directory contains the artifacts from this step of the build,
a `stdout.log` file, and a `stderr.log` file containing the outputs of the
build process.

### Worker Processes

When in server mode StrawBoss has a multi-process model. There is a
supervisory process that serves the REST API, receives build requests, and
spawns worker processes. The worker processes are responsible for actually
running the builds. The supervisory process registers a handler for SIGCHLD,
using the handler to update the supervisor's knowledge of the build results and
`wait` for the process to be cleaned up.

When launched in single-build mode there is no supervisory process. The main
process directly executes the requested build steps.

### Building StrawBoss

To build StrawBoss locally, checkout the repository and in the repo root run:

    nimble build

### Testing

StrawBoss has two test suites, a set of unit tests and a set of functional
tests. All the test code and assets live under the `src/test` subdirectory.

Each test suite has a runner file that serves as an entry for the test process,
named `run_unit_tests.nim` and `run_functional_tests.nim`.

#### Unit Tests

The unit test soruce files live in the `nim/unit` subdirectory and have a
one-to-one correspondence with the StrawBoss source files following this
naming convention: `t<module>.nim`. The unit tests are intended to be run any
time the code is recompiled.

To run the unit tests, use the `unittest` nimble task:

    nimble unittest

#### Functional Tests

The functional test source files live in the `nim/functional` subdirectory.
There is a test project that is used to excercise StrawBoss functionality. To
avoid external coupling it is stored within the StrawBoss repository as a test
asset. To avoid `git` complications it is stored as a Gzipped TAR file and
unpacked to a temporary directory as part of the functional test process.

As the functional tests are more time-consuming and intensive, they are
expected to bu run when performing a build.

To run the functional tests, use the `functest` nimble task:

    nimble functest