e000b37c35
* Rename artifactsRepo -> buildDataDir to be more explicit about the fact that it holds more than just the artifacts. * Revert removal of run ids. * Move Worker definition into core as part of making the core responsible for accepting run requests. * Make the core module more responsible for internal details of data structure and storage. External callers should not need to construct paths to artifacts, versions, etc. but should be able to call method in the core module to do this work for them. * The working directory no longer contains anything but the checked-out code. All StrawBoss-specific data is stored by StrawBoss elsewhere. * Add a regular maintenance cycle to the server module.
276 lines
12 KiB
Markdown
276 lines
12 KiB
Markdown
# Straw Boss CI
|
|
|
|
* Pipeline: series of named steps.
|
|
* Step: named set of operations (make, script, something extensible).
|
|
Optionally names an artifact that is the result of the step.
|
|
* Artifact: resulting output from a step. strawboss may skip step execution if
|
|
it already has an artifact cached from that step.
|
|
* Configuration is two-part. Pipeline, step, and artifact definition are part
|
|
of the project configuration (.strawboss.json? yaml?). Environment
|
|
configuration lives on the strawboss server (supplies DB info, etc.).
|
|
* Step execution happens within the root directory of a fresh copy of the repo.
|
|
Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
|
|
the fresh copy is checked out at that reference.
|
|
|
|
|
|
## Projects
|
|
|
|
StrawBoss must know about the projects it builds. For each project, StrawBoss
|
|
must be configured with a unique project name and a URL or repo-specification
|
|
that can be passed to `git clone` to clone the project.
|
|
|
|
For more details see the
|
|
[Service Project Definition](#service-project-definition) section.
|
|
|
|
## Artifacts Repo
|
|
|
|
The artifacts repository is a directory on the filesystem used to store
|
|
build artifacts and information about build status. The top-level artifacts
|
|
repo directory contains project build directories: one directory for every
|
|
project that StrawBoss has built. Each project directory contains multiple
|
|
sub-directories. Each
|
|
|
|
## Configuration
|
|
|
|
There are two points of configuration when working with StrawBoss, the
|
|
[StrawBoss configuration file](#strawboss-configuration-file), and the
|
|
individual [project configurations](#project-configuration).
|
|
|
|
The [StrawBoss configuration file](#strawboss-configuration-file) is used to
|
|
configure the StrawBoss instance itself and stores server-side information such
|
|
as the list of projects known to StrawBoss. If you are setting up StrawBoss on
|
|
a server you will need to work with this configuration file.
|
|
|
|
The [project configurations](#project-configuration) are used to configure the
|
|
build process and options for each proejct and are stored with the projects
|
|
themselves. If you are working on a project that you wish to build with
|
|
StrawBoss you will be working with this configuration file.
|
|
|
|
### StrawBoss configuration file
|
|
|
|
StrawBoss expects to find `strawboss.config.json` in the working directory of
|
|
the `strawboss` executable. This is the configuration file for StrawBoss
|
|
itself. The contents are expected to be a valid JSON object. The top level keys
|
|
are:
|
|
|
|
* `buildDataDir`: A string denoting the path to the directory where StrawBoss
|
|
keeps metadata about builds it has performed and the artifacts resulting from
|
|
the builds.
|
|
|
|
* `authSecret`: Secret key used to sign JWT session tokens.
|
|
|
|
* `users`: the array of user definition objects. Each user object is required
|
|
to have `username` and `hashedPwd` keys, both string.
|
|
|
|
* `projects`: an array of project definitions (detailed below).
|
|
|
|
All are required.
|
|
|
|
#### Service Project Definition
|
|
|
|
Project definitions are JSON objects with the following keys:
|
|
|
|
* `name` *(required)*: A unique name for the project. Names are case sensitive.
|
|
There are no hard limits on names but the recommended convention is
|
|
alpha-numeric using dashes or underscores for word separation (e.g.
|
|
`foo-confabulator` or `web_refractulator`, etc.).
|
|
|
|
* `repo` *(required)*: A valid `git` URL or repo-specification that can be
|
|
passed to `git clone` to clone the project.
|
|
|
|
* `defaultBranch` *(optional)*: If a run request is submitted without
|
|
specifying a branch or commit reference, use this branch *(defaults to
|
|
`master`)*.
|
|
|
|
* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file
|
|
(see the [Project Configuration](#project-configuration) section). If this is
|
|
a relative path it is resolved relative to the project root directory.
|
|
*(defaults to `strawboss.json`)*.
|
|
|
|
* `envVars` *(optional)*: an object whose key-value pairs are expected to be
|
|
strings. Each pair will be added as an environment variable to the runtime
|
|
environment of builds for this project. *(defaults to `{}`)*.
|
|
|
|
### Project Configuration
|
|
|
|
The build configuration for each project lives in the project itself. In the
|
|
root directory of the project repo StrawBoss expects to find a project
|
|
configuration file named `strawboss.json` (this location can be configured, see
|
|
the [Service Project Definition](#service-project-definition) section). The
|
|
project configuration file tells StrawBoss what steps of a project are
|
|
available to be built, and how to initiate the build for each step. Steps can
|
|
be dependant on each other, as long as they do not define any circular
|
|
references.
|
|
|
|
The contents of a project configurations files is expected to be a valid JSON
|
|
object. The top level keys are:
|
|
|
|
* `name` *(required)*: the unique name for the project. This should be the same
|
|
name as configured in the StrawBoss service configuration file.
|
|
|
|
* `steps` *(required)*: an object of step definitions. Each of this object's
|
|
keys is the name of a step. The corresponding value is the step definition.
|
|
(see the [Step Definition](#step-definition) section for details).
|
|
|
|
* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible)
|
|
that is expected to print the current version of the project on `stdout`.
|
|
*(defaults to `git describe --tags --always`)*.
|
|
|
|
#### Step Definition
|
|
|
|
Step definitions are JSON objects with the following keys:
|
|
|
|
* `workingDir` *(optional)*: the working directory (relative to the project
|
|
root folder) for all commands executed as part of this step. *(defaults to
|
|
`'.'`, the project root directory)*.
|
|
|
|
* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
|
|
`true` unless `cmdInput` is given, in which case it defaults to `sh`)*
|
|
|
|
* `cmdInput` *(optional)*: an array of string that will be concatenated with
|
|
newlines separating each string and piped as input to the command for this
|
|
step. It is possible to include newlines in the strings, but not required.
|
|
Environment variables included in this input are resolved *before* it is
|
|
passed to `stdin` as `stepCmd` does not have to be a shell and is not
|
|
expected to be able to do this interpolation itself. *(defaults to `[]`)*.
|
|
|
|
* `artifacts` *(optional)*: an array of strings representing artifacts that
|
|
will be generated by this step and should be saved for future use. Each
|
|
string should represent a path to a file (relative to the project root
|
|
directory). StrawBoss will gather all of these files and store them in the
|
|
artifacts repo for this step and build. These paths may contain environment
|
|
variable references. *(defaults to `[]`)*.
|
|
|
|
* `depends` *(optional)*: an array of strings representing step names upon
|
|
which this step depends. *(defaults to `[]`)*.
|
|
|
|
* `expectedEnv` *(optional)*: an array of strings representing environment
|
|
variable keys that should be present in the environment when executing this
|
|
step. If there is an environment variable that is listed here but not present
|
|
in the build environment it will cause StrawBoss to fail this step.
|
|
*(defaults to `[]`)*.
|
|
|
|
* `dontSkip` *(optional)*: boolean. StrawBoss remembers the steps it has run
|
|
and caches the artifacts generated by them. Future builds for the same
|
|
version that request this step normally will not cause StrawBoss to re-run
|
|
the step. If `dontSkip` is set to `true`, the output of this step will always
|
|
be run when it is referenced, regardless of previous cached results.
|
|
|
|
## Build Process
|
|
|
|
When performing a build, StrawBoss:
|
|
|
|
1. creates a temporary workspace for this build
|
|
2. clones the repo into the workspace
|
|
3. checkout the revision or branch requested for this run
|
|
4. load the project's StrawBoss configuration file.
|
|
5. merge environment variables defined in the project configuration
|
|
6. run `versionCmd` to get the current project version. The result is stored
|
|
in the `VERSION` environment variable.
|
|
7. check the environment variables against `expectedEnv`
|
|
8. check that all the steps named in `depends` have already been run and run
|
|
them if they have not. For each step named in `depends` an environment
|
|
variable is added named `<step-name>_DIR` that contains the absolute path to
|
|
the artifacts repo for that step at this version. This is intended to be
|
|
used to reference artifacts from other steps, e.g.
|
|
`${build_DIR}/site-contents.zip`.
|
|
9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
|
|
are resolved and the resulting string are fed line-by-line into the process
|
|
as `stdin`.
|
|
10. the files named in `artifacts` are copied into the artifacts repo for this
|
|
step and version.
|
|
|
|
## Architecture
|
|
|
|
The following describes the internal architecture of StrawBoss. This section is
|
|
written primarily for those wanting to work on StrawBoss itself and is not
|
|
nessecary for the setup and use of StrawBoss.
|
|
|
|
### Artifacts repo
|
|
|
|
In the top-level of the artifacts repo are project directories.
|
|
|
|
#### Project Directories
|
|
|
|
Project directories contain step directories and cached project configuration
|
|
files.
|
|
|
|
##### Cached configuration files.
|
|
|
|
The cached project configuration files follow this naming convention:
|
|
`configuration.<version>.json`. These cached versions of the project
|
|
configration are only intended to be used in cases where StrawBoss is not
|
|
building anything and doesn't check out a copy of the repo. For example, when a
|
|
client queries the REST API for the list of steps in a project, StrawBoss will
|
|
consult the most recently modified cached copy of the project configuration
|
|
rather than cloning the entire repo just to answer this question. Whenever
|
|
StrawBoss has a copy of the repo, it should look for the actual configuration
|
|
file in that version of the repo instead of consulting the cached configuration
|
|
files. When determining the "most recent" cached copy, StrawBoss uses the
|
|
modification time of the files, again to avoid cloning the repo. API access to
|
|
project configurations in this manner is intended as a convenience. The actual
|
|
project configuration in the project repository should be considered the source
|
|
of truth.
|
|
|
|
##### Step and Version Directories
|
|
|
|
A step directory contains a list of version directories using the version
|
|
string as the directory names and a list of status JSON files also named after
|
|
the version (`<version>.status.json`).
|
|
|
|
A version directory contains the artifacts from this step of the build,
|
|
a `stdout.log` file, and a `stderr.log` file containing the outputs of the
|
|
build process.
|
|
|
|
### Worker Processes
|
|
|
|
When in server mode StrawBoss has a multi-process model. There is a
|
|
supervisory process that serves the REST API, receives build requests, and
|
|
spawns worker processes. The worker processes are responsible for actually
|
|
running the builds. The supervisory process registers a handler for SIGCHLD,
|
|
using the handler to update the supervisor's knowledge of the build results and
|
|
`wait` for the process to be cleaned up.
|
|
|
|
When launched in single-build mode there is no supervisory process. The main
|
|
process directly executes the requested build steps.
|
|
|
|
### Building StrawBoss
|
|
|
|
To build StrawBoss locally, checkout the repository and in the repo root run:
|
|
|
|
nimble build
|
|
|
|
### Testing
|
|
|
|
StrawBoss has two test suites, a set of unit tests and a set of functional
|
|
tests. All the test code and assets live under the `src/test` subdirectory.
|
|
|
|
Each test suite has a runner file that serves as an entry for the test process,
|
|
named `run_unit_tests.nim` and `run_functional_tests.nim`.
|
|
|
|
#### Unit Tests
|
|
|
|
The unit test soruce files live in the `nim/unit` subdirectory and have a
|
|
one-to-one correspondence with the StrawBoss source files following this
|
|
naming convention: `t<module>.nim`. The unit tests are intended to be run any
|
|
time the code is recompiled.
|
|
|
|
To run the unit tests, use the `unittest` nimble task:
|
|
|
|
nimble unittest
|
|
|
|
#### Functional Tests
|
|
|
|
The functional test source files live in the `nim/functional` subdirectory.
|
|
There is a test project that is used to excercise StrawBoss functionality. To
|
|
avoid external coupling it is stored within the StrawBoss repository as a test
|
|
asset. To avoid `git` complications it is stored as a Gzipped TAR file and
|
|
unpacked to a temporary directory as part of the functional test process.
|
|
|
|
As the functional tests are more time-consuming and intensive, they are
|
|
expected to be run when performing a build.
|
|
|
|
To run the functional tests, use the `functest` nimble task:
|
|
|
|
nimble functest
|