82a7b301ea
* Implemented periodic maintenance window. * Moved worker creation into the core module. * Worker processes no longer create run requests, but read queued requests from the file system. * Build status and logs have been moved into the StrawBoss data directory. * An initial build status is recorded when the job is queued. * Build status is recorded for build references as well as actual versions. So there will be a build status for "master", for example, that is overwritten whenever "master" is built for that step. * RunRequests now include a timestamp. * Added a Run object to contain both a RunRequest and the corresponding BuildStatus for that run. * API endpoints that talk about runs now return Run objects instead of RunRequests. * Moved all data layer operations into the core module so that the "database API" only lives in one place.
286 lines
13 KiB
Markdown
286 lines
13 KiB
Markdown
# Straw Boss CI
|
|
|
|
* Pipeline: series of named steps.
|
|
* Step: named set of operations (make, script, something extensible).
|
|
Optionally names an artifact that is the result of the step.
|
|
* Artifact: resulting output from a step. strawboss may skip step execution if
|
|
it already has an artifact cached from that step.
|
|
* Configuration is two-part. Pipeline, step, and artifact definition are part
|
|
of the project configuration (.strawboss.json? yaml?). Environment
|
|
configuration lives on the strawboss server (supplies DB info, etc.).
|
|
* Step execution happens within the root directory of a fresh copy of the repo.
|
|
Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
|
|
the fresh copy is checked out at that reference.
|
|
|
|
|
|
## Projects
|
|
|
|
StrawBoss must know about the projects it builds. For each project, StrawBoss
|
|
must be configured with a unique project name and a URL or repo-specification
|
|
that can be passed to `git clone` to clone the project.
|
|
|
|
For more details see the
|
|
[Service Project Definition](#service-project-definition) section.
|
|
|
|
## Artifacts Repo
|
|
|
|
The artifacts repository is a directory on the filesystem used to store
|
|
build artifacts and information about build status. The top-level artifacts
|
|
repo directory contains project build directories: one directory for every
|
|
project that StrawBoss has built. Each project directory contains multiple
|
|
sub-directories. Each
|
|
|
|
## Configuration
|
|
|
|
There are two points of configuration when working with StrawBoss, the
|
|
[StrawBoss configuration file](#strawboss-configuration-file), and the
|
|
individual [project configurations](#project-configuration).
|
|
|
|
The [StrawBoss configuration file](#strawboss-configuration-file) is used to
|
|
configure the StrawBoss instance itself and stores server-side information such
|
|
as the list of projects known to StrawBoss. If you are setting up StrawBoss on
|
|
a server you will need to work with this configuration file.
|
|
|
|
The [project configurations](#project-configuration) are used to configure the
|
|
build process and options for each proejct and are stored with the projects
|
|
themselves. If you are working on a project that you wish to build with
|
|
StrawBoss you will be working with this configuration file.
|
|
|
|
### StrawBoss configuration file
|
|
|
|
StrawBoss expects to find `strawboss.config.json` in the working directory of
|
|
the `strawboss` executable. This is the configuration file for StrawBoss
|
|
itself. The contents are expected to be a valid JSON object. The top level keys
|
|
are:
|
|
|
|
* `buildDataDir`: *(optional)* A string denoting the path to the directory
|
|
where StrawBoss keeps metadata about builds it has performed and the
|
|
artifacts resulting from the builds. *(defaults to `build-data`)*
|
|
|
|
* `authSecret`: *(required)* Secret key used to sign JWT session tokens.
|
|
|
|
* `users`: *(required)* the array of user definition objects. Each user object is required
|
|
to have `username` and `hashedPwd` keys, both string.
|
|
|
|
* `projects`: *(required)* an array of project definitions (detailed below).
|
|
|
|
* `pwdCost`: *(required)* parameter to the user password hashing algorithm determining the
|
|
computational cost of the hash.
|
|
|
|
* `maintenancePeriod`: *(optional)* how often, in milliseconds, should the
|
|
StrawBoss server perform maintenance (clear finished workers, etc).
|
|
*(defaults to `10000`, every 10 seconds)*.
|
|
|
|
* `debug`: boolean, should debug behavior be enabled. This is primarily
|
|
intended for testing during StrawBoss development. *(defaults to `false`)*
|
|
|
|
All are required.
|
|
|
|
#### Service Project Definition
|
|
|
|
Project definitions are JSON objects with the following keys:
|
|
|
|
* `name` *(required)*: A unique name for the project. Names are case sensitive.
|
|
There are no hard limits on names but the recommended convention is
|
|
alpha-numeric using dashes or underscores for word separation (e.g.
|
|
`foo-confabulator` or `web_refractulator`, etc.).
|
|
|
|
* `repo` *(required)*: A valid `git` URL or repo-specification that can be
|
|
passed to `git clone` to clone the project.
|
|
|
|
* `defaultBranch` *(optional)*: If a run request is submitted without
|
|
specifying a branch or commit reference, use this branch *(defaults to
|
|
`master`)*.
|
|
|
|
* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file
|
|
(see the [Project Configuration](#project-configuration) section). If this is
|
|
a relative path it is resolved relative to the project root directory.
|
|
*(defaults to `strawboss.json`)*.
|
|
|
|
* `envVars` *(optional)*: an object whose key-value pairs are expected to be
|
|
strings. Each pair will be added as an environment variable to the runtime
|
|
environment of builds for this project. *(defaults to `{}`)*.
|
|
|
|
### Project Configuration
|
|
|
|
The build configuration for each project lives in the project itself. In the
|
|
root directory of the project repo StrawBoss expects to find a project
|
|
configuration file named `strawboss.json` (this location can be configured, see
|
|
the [Service Project Definition](#service-project-definition) section). The
|
|
project configuration file tells StrawBoss what steps of a project are
|
|
available to be built, and how to initiate the build for each step. Steps can
|
|
be dependant on each other, as long as they do not define any circular
|
|
references.
|
|
|
|
The contents of a project configurations files is expected to be a valid JSON
|
|
object. The top level keys are:
|
|
|
|
* `name` *(required)*: the unique name for the project. This should be the same
|
|
name as configured in the StrawBoss service configuration file.
|
|
|
|
* `steps` *(required)*: an object of step definitions. Each of this object's
|
|
keys is the name of a step. The corresponding value is the step definition.
|
|
(see the [Step Definition](#step-definition) section for details).
|
|
|
|
* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible)
|
|
that is expected to print the current version of the project on `stdout`.
|
|
*(defaults to `git describe --tags --always`)*.
|
|
|
|
#### Step Definition
|
|
|
|
Step definitions are JSON objects with the following keys:
|
|
|
|
* `workingDir` *(optional)*: the working directory (relative to the project
|
|
root folder) for all commands executed as part of this step. *(defaults to
|
|
`'.'`, the project root directory)*.
|
|
|
|
* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
|
|
`true` unless `cmdInput` is given, in which case it defaults to `sh`)*
|
|
|
|
* `cmdInput` *(optional)*: an array of string that will be concatenated with
|
|
newlines separating each string and piped as input to the command for this
|
|
step. It is possible to include newlines in the strings, but not required.
|
|
Environment variables included in this input are resolved *before* it is
|
|
passed to `stdin` as `stepCmd` does not have to be a shell and is not
|
|
expected to be able to do this interpolation itself. *(defaults to `[]`)*.
|
|
|
|
* `artifacts` *(optional)*: an array of strings representing artifacts that
|
|
will be generated by this step and should be saved for future use. Each
|
|
string should represent a path to a file (relative to the project root
|
|
directory). StrawBoss will gather all of these files and store them in the
|
|
artifacts repo for this step and build. These paths may contain environment
|
|
variable references. *(defaults to `[]`)*.
|
|
|
|
* `depends` *(optional)*: an array of strings representing step names upon
|
|
which this step depends. *(defaults to `[]`)*.
|
|
|
|
* `expectedEnv` *(optional)*: an array of strings representing environment
|
|
variable keys that should be present in the environment when executing this
|
|
step. If there is an environment variable that is listed here but not present
|
|
in the build environment it will cause StrawBoss to fail this step.
|
|
*(defaults to `[]`)*.
|
|
|
|
* `dontSkip` *(optional)*: boolean. StrawBoss remembers the steps it has run
|
|
and caches the artifacts generated by them. Future builds for the same
|
|
version that request this step normally will not cause StrawBoss to re-run
|
|
the step. If `dontSkip` is set to `true`, the output of this step will always
|
|
be run when it is referenced, regardless of previous cached results.
|
|
|
|
## Build Process
|
|
|
|
When performing a build, StrawBoss:
|
|
|
|
1. creates a temporary workspace for this build
|
|
2. clones the repo into the workspace
|
|
3. checkout the revision or branch requested for this run
|
|
4. load the project's StrawBoss configuration file.
|
|
5. merge environment variables defined in the project configuration
|
|
6. run `versionCmd` to get the current project version. The result is stored
|
|
in the `VERSION` environment variable.
|
|
7. check the environment variables against `expectedEnv`
|
|
8. check that all the steps named in `depends` have already been run and run
|
|
them if they have not. For each step named in `depends` an environment
|
|
variable is added named `<step-name>_DIR` that contains the absolute path to
|
|
the artifacts repo for that step at this version. This is intended to be
|
|
used to reference artifacts from other steps, e.g.
|
|
`${build_DIR}/site-contents.zip`.
|
|
9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
|
|
are resolved and the resulting string are fed line-by-line into the process
|
|
as `stdin`.
|
|
10. the files named in `artifacts` are copied into the artifacts repo for this
|
|
step and version.
|
|
|
|
## Architecture
|
|
|
|
The following describes the internal architecture of StrawBoss. This section is
|
|
written primarily for those wanting to work on StrawBoss itself and is not
|
|
nessecary for the setup and use of StrawBoss.
|
|
|
|
### Artifacts repo
|
|
|
|
In the top-level of the artifacts repo are project directories.
|
|
|
|
#### Project Directories
|
|
|
|
Project directories contain step directories and cached project configuration
|
|
files.
|
|
|
|
##### Cached configuration files.
|
|
|
|
The cached project configuration files follow this naming convention:
|
|
`configuration.<version>.json`. These cached versions of the project
|
|
configration are only intended to be used in cases where StrawBoss is not
|
|
building anything and doesn't check out a copy of the repo. For example, when a
|
|
client queries the REST API for the list of steps in a project, StrawBoss will
|
|
consult the most recently modified cached copy of the project configuration
|
|
rather than cloning the entire repo just to answer this question. Whenever
|
|
StrawBoss has a copy of the repo, it should look for the actual configuration
|
|
file in that version of the repo instead of consulting the cached configuration
|
|
files. When determining the "most recent" cached copy, StrawBoss uses the
|
|
modification time of the files, again to avoid cloning the repo. API access to
|
|
project configurations in this manner is intended as a convenience. The actual
|
|
project configuration in the project repository should be considered the source
|
|
of truth.
|
|
|
|
##### Step and Version Directories
|
|
|
|
A step directory contains a list of version directories using the version
|
|
string as the directory names and a list of status JSON files also named after
|
|
the version (`<version>.status.json`).
|
|
|
|
A version directory contains the artifacts from this step of the build,
|
|
a `stdout.log` file, and a `stderr.log` file containing the outputs of the
|
|
build process.
|
|
|
|
### Worker Processes
|
|
|
|
When in server mode StrawBoss has a multi-process model. There is a
|
|
supervisory process that serves the REST API, receives build requests, and
|
|
spawns worker processes. The worker processes are responsible for actually
|
|
running the builds. The supervisory process registers a handler for SIGCHLD,
|
|
using the handler to update the supervisor's knowledge of the build results and
|
|
`wait` for the process to be cleaned up.
|
|
|
|
When launched in single-build mode there is no supervisory process. The main
|
|
process directly executes the requested build steps.
|
|
|
|
### Building StrawBoss
|
|
|
|
To build StrawBoss locally, checkout the repository and in the repo root run:
|
|
|
|
nimble build
|
|
|
|
### Testing
|
|
|
|
StrawBoss has two test suites, a set of unit tests and a set of functional
|
|
tests. All the test code and assets live under the `src/test` subdirectory.
|
|
|
|
Each test suite has a runner file that serves as an entry for the test process,
|
|
named `run_unit_tests.nim` and `run_functional_tests.nim`.
|
|
|
|
#### Unit Tests
|
|
|
|
The unit test soruce files live in the `nim/unit` subdirectory and have a
|
|
one-to-one correspondence with the StrawBoss source files following this
|
|
naming convention: `t<module>.nim`. The unit tests are intended to be run any
|
|
time the code is recompiled.
|
|
|
|
To run the unit tests, use the `unittest` nimble task:
|
|
|
|
nimble unittest
|
|
|
|
#### Functional Tests
|
|
|
|
The functional test source files live in the `nim/functional` subdirectory.
|
|
There is a test project that is used to excercise StrawBoss functionality. To
|
|
avoid external coupling it is stored within the StrawBoss repository as a test
|
|
asset. To avoid `git` complications it is stored as a Gzipped TAR file and
|
|
unpacked to a temporary directory as part of the functional test process.
|
|
|
|
As the functional tests are more time-consuming and intensive, they are
|
|
expected to be run when performing a build.
|
|
|
|
To run the functional tests, use the `functest` nimble task:
|
|
|
|
nimble functest
|