Straw Boss CI

• Pipeline: series of named steps.
• Step: named set of operations (make, script, something extensible). Optionally names an artifact that is the result of the step.
• Artifact: resulting output from a step. strawboss may skip step execution if it already has an artifact cached from that step.
• Configuration is two-part. Pipeline, step, and artifact definition are part of the project configuration (.strawboss.json? yaml?). Environment configuration lives on the strawboss server (supplies DB info, etc.).
• Step execution happens within the root directory of a fresh copy of the repo. Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and the fresh copy is checked out at that reference.

Projects

StrawBoss must know about the projects it builds. For each project, StrawBoss must be configured with a unique project name and a URL or repo-specification that can be passed to git clone to clone the project.

For more details see the Service Project Definition section.

Artifacts Repo

The artifacts repository is a directory on the filesystem used to store build artifacts and information about build status. The top-level artifacts repo directory contains project build directories: one directory for every project that StrawBoss has built. Each project directory contains multiple sub-directories. Each

Configuration

There are two points of configuration when working with StrawBoss, the StrawBoss configuration file, and the individual project configurations.

The StrawBoss configuration file is used to configure the StrawBoss instance itself and stores server-side information such as the list of projects known to StrawBoss. If you are setting up StrawBoss on a server you will need to work with this configuration file.

The project configurations are used to configure the build process and options for each proejct and are stored with the projects themselves. If you are working on a project that you wish to build with StrawBoss you will be working with this configuration file.

StrawBoss configuration file

StrawBoss expects to find strawboss.config.json in the working directory of the strawboss executable. This is the configuration file for StrawBoss itself. The contents are expected to be a valid JSON object. The top level keys are:

• buildDataDir: (optional) A string denoting the path to the directory where StrawBoss keeps metadata about builds it has performed and the artifacts resulting from the builds. (defaults to build-data)

• authSecret: (required) Secret key used to sign JWT session tokens.

• users: (required) the array of user definition objects. Each user object is required to have username and hashedPwd keys, both string.

• projects: (required) an array of project definitions (detailed below).

• pwdCost: (required) parameter to the user password hashing algorithm determining the computational cost of the hash.

• maintenancePeriod: (optional) how often, in milliseconds, should the StrawBoss server perform maintenance (clear finished workers, etc). (defaults to 10000, every 10 seconds).

• debug: boolean, should debug behavior be enabled. This is primarily intended for testing during StrawBoss development. (defaults to false)

All are required.

Service Project Definition

Project definitions are JSON objects with the following keys:

• name (required): A unique name for the project. Names are case sensitive. There are no hard limits on names but the recommended convention is alpha-numeric using dashes or underscores for word separation (e.g. foo-confabulator or web_refractulator, etc.).

• repo (required): A valid git URL or repo-specification that can be passed to git clone to clone the project.

• defaultBranch (optional): If a run request is submitted without specifying a branch or commit reference, use this branch (defaults to master).

• cfgFilePath (optional): path to the StrawBoss project configuration file (see the Project Configuration section). If this is a relative path it is resolved relative to the project root directory. (defaults to strawboss.json).

• envVars (optional): an object whose key-value pairs are expected to be strings. Each pair will be added as an environment variable to the runtime environment of builds for this project. (defaults to {}).

Project Configuration

The build configuration for each project lives in the project itself. In the root directory of the project repo StrawBoss expects to find a project configuration file named strawboss.json (this location can be configured, see the Service Project Definition section). The project configuration file tells StrawBoss what steps of a project are available to be built, and how to initiate the build for each step. Steps can be dependant on each other, as long as they do not define any circular references.

The contents of a project configurations files is expected to be a valid JSON object. The top level keys are:

• name (required): the unique name for the project. This should be the same name as configured in the StrawBoss service configuration file.

• steps (required): an object of step definitions. Each of this object's keys is the name of a step. The corresponding value is the step definition. (see the Step Definition section for details).

• versionCmd (optional): a command to be run in a shell (sh-compatible) that is expected to print the current version of the project on stdout. (defaults to git describe --tags --always).

Step Definition

Step definitions are JSON objects with the following keys:

• workingDir (optional): the working directory (relative to the project root folder) for all commands executed as part of this step. (defaults to '.', the project root directory).

• stepCmd (optional): the command to execute for this step. (defaults to true unless cmdInput is given, in which case it defaults to sh)

• cmdInput (optional): an array of string that will be concatenated with newlines separating each string and piped as input to the command for this step. It is possible to include newlines in the strings, but not required. Environment variables included in this input are resolved before it is passed to stdin as stepCmd does not have to be a shell and is not expected to be able to do this interpolation itself. (defaults to []).

• artifacts (optional): an array of strings representing artifacts that will be generated by this step and should be saved for future use. Each string should represent a path to a file (relative to the project root directory). StrawBoss will gather all of these files and store them in the artifacts repo for this step and build. These paths may contain environment variable references. (defaults to []).

• depends (optional): an array of strings representing step names upon which this step depends. (defaults to []).

• expectedEnv (optional): an array of strings representing environment variable keys that should be present in the environment when executing this step. If there is an environment variable that is listed here but not present in the build environment it will cause StrawBoss to fail this step. (defaults to []).

• dontSkip (optional): boolean. StrawBoss remembers the steps it has run and caches the artifacts generated by them. Future builds for the same version that request this step normally will not cause StrawBoss to re-run the step. If dontSkip is set to true, the output of this step will always be run when it is referenced, regardless of previous cached results.

Build Process

When performing a build, StrawBoss:

1. creates a temporary workspace for this build
2. clones the repo into the workspace
3. checkout the revision or branch requested for this run
4. load the project's StrawBoss configuration file.
5. merge environment variables defined in the project configuration
6. run versionCmd to get the current project version. The result is stored in the VERSION environment variable.
7. check the environment variables against expectedEnv
8. check that all the steps named in depends have already been run and run them if they have not. For each step named in depends an environment variable is added named <step-name>_DIR that contains the absolute path to the artifacts repo for that step at this version. This is intended to be used to reference artifacts from other steps, e.g. ${build_DIR}/site-contents.zip.
9. stepCmd is executed in workingDir. Environment variables in cmdInput are resolved and the resulting string are fed line-by-line into the process as stdin.
10. the files named in artifacts are copied into the artifacts repo for this step and version.

Architecture

The following describes the internal architecture of StrawBoss. This section is written primarily for those wanting to work on StrawBoss itself and is not nessecary for the setup and use of StrawBoss.

Artifacts repo

In the top-level of the artifacts repo are project directories.

Project Directories

Project directories contain step directories and cached project configuration files.

Cached configuration files.

The cached project configuration files follow this naming convention: configuration.<version>.json. These cached versions of the project configration are only intended to be used in cases where StrawBoss is not building anything and doesn't check out a copy of the repo. For example, when a client queries the REST API for the list of steps in a project, StrawBoss will consult the most recently modified cached copy of the project configuration rather than cloning the entire repo just to answer this question. Whenever StrawBoss has a copy of the repo, it should look for the actual configuration file in that version of the repo instead of consulting the cached configuration files. When determining the "most recent" cached copy, StrawBoss uses the modification time of the files, again to avoid cloning the repo. API access to project configurations in this manner is intended as a convenience. The actual project configuration in the project repository should be considered the source of truth.

Step and Version Directories

A step directory contains a list of version directories using the version string as the directory names and a list of status JSON files also named after the version (<version>.status.json).

A version directory contains the artifacts from this step of the build, a stdout.log file, and a stderr.log file containing the outputs of the build process.

Worker Processes

When in server mode StrawBoss has a multi-process model. There is a supervisory process that serves the REST API, receives build requests, and spawns worker processes. The worker processes are responsible for actually running the builds. The supervisory process registers a handler for SIGCHLD, using the handler to update the supervisor's knowledge of the build results and wait for the process to be cleaned up.

When launched in single-build mode there is no supervisory process. The main process directly executes the requested build steps.

Building StrawBoss

To build StrawBoss locally, checkout the repository and in the repo root run:

nimble build

Testing

StrawBoss has two test suites, a set of unit tests and a set of functional tests. All the test code and assets live under the src/test subdirectory.

Each test suite has a runner file that serves as an entry for the test process, named run_unit_tests.nim and run_functional_tests.nim.

Unit Tests

The unit test soruce files live in the nim/unit subdirectory and have a one-to-one correspondence with the StrawBoss source files following this naming convention: t<module>.nim. The unit tests are intended to be run any time the code is recompiled.

To run the unit tests, use the unittest nimble task:

nimble unittest

Functional Tests

The functional test source files live in the nim/functional subdirectory. There is a test project that is used to excercise StrawBoss functionality. To avoid external coupling it is stored within the StrawBoss repository as a test asset. To avoid git complications it is stored as a Gzipped TAR file and unpacked to a temporary directory as part of the functional test process.

As the functional tests are more time-consuming and intensive, they are expected to be run when performing a build.

To run the functional tests, use the functest nimble task:

nimble functest