diff --git a/README.md b/README.md index 09d2983..c25167c 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,4 @@ -Straw Boss CI -============= +# Straw Boss CI * Pipeline: series of named steps. * Step: named set of operations (make, script, something extensible). @@ -13,3 +12,205 @@ Straw Boss CI * Step execution happens within the root directory of a fresh copy of the repo. Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and the fresh copy is checked out at that reference. + + +## Projects + +StrawBoss must know about the projects it builds. For each project, StrawBoss +must be configured with a unique project name and a URL or repo-specification +that can be passed to `git clone` to clone the project. + +For more details see the Service Project Definitions section. + +## Artifacts Repo + +The artifacts repository is a directory on the filesystem used to store +build artifacts and information about build status. The top-level artifacts +repo directory contains project build directories: one directory for every +project that StrawBoss has built. Each project directory contains multiple +sub-directories. Each + +## Configuration + +### StrawBoss configuration file + +StrawBoss expects to find `strawboss.config.json` in the working directory of +the `strawboss` executable. This is the configuration file for StrawBoss +itself. The contents are expected to be a valid JSON object. The top level keys +are: + +* `artifactsRepo`: A string denoting the path to the artifacts repository + directory. + +* `users`: the array of user definition objects. Each user object is required + to have `username` and `hashedPwd` keys, both string. + +* `tokens`: an array of string, each representing a valid auth token that has + been issued to a client. + +* `projects`: an array of project definitions (detailed below). + +All are required. + +#### Service Project Definitions + +Project definitions are JSON objects with the following keys: + +* `name` *(required)*: A unique name for the project. Names are case sensitive. + There are no hard limits on names but the recommended convention is + alpha-numeric using dashes or underscores for word separation (e.g. + `foo-confabulator` or `web_refractulator`, etc.). + +* `repo` *(required)*: A valid `git` URL or repo-specification that can be + passed to `git clone` to clone the project. + +* `defaultBranch` *(optional)*: If a run request is submitted without + specifying a branch or commit reference, use this branch *(defaults to + `master`)*. + +* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file + (see the Project Configuration section). If this is a relative path it is + resolved relative to the project root directory. *(defaults to + `strawboss.json`)*. + +* `envVars` *(optional)*: an object whose key-value pairs are expected to be + strings. Each pair will be added as an environment variable to the runtime + environment of builds for this project. *(defaults to `{}`)*. + +### Project Configuration + +The build configuration for each project lives in the project itself. In the +root directory of the project repo StrawBoss expects to find a project +configuration file named `strawboss.json` (this location can be configured, see +the Service Project Definitions section). The project configuration file +tells StrawBoss what steps of a project are available to be built, and how to +initiate the build for each step. Steps can be dependant on each other, as long +as they do not define any circular references. + +The contents of a project configurations files is expected to be a valid JSON +object. The top level keys are: + +* `name` *(required)*: the unique name for the project. This should be the same + name as configured in the StrawBoss service configuration file. + +* `steps` *(required)*: an object of step definitions. Each of this object's + keys is the name of a step. The corresponding value is the step definition. + (see the Step Definitions section for details). + +* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible) + that is expected to print the current version of the project on `stdout`. + *(defaults to `git describe --tags --always`)*. + +## Build Process + +When performing a build, StrawBoss: + +1. creates a temporary workspace for this build +2. clones the repo into the workspace +3. checkout the revision or branch requested for this run +4. load the project's StrawBoss configuration file. +5. merge environment variables defined in the project configuration +6. run `versionCmd` to get the current project version +7. check the environment variables against `expectedEnv` +8. check that all the steps named in `depends` have already been run and run + them if they have not. For each step named in `depends` an environment + variable is added named `_DIR` that contains the absolute path to + the artifacts repo for that step at this version. This is intended to be + used to reference artifacts from other steps, e.g. + `${build_DIR}/site-contents.zip`. +9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput` + are resolved and the resulting string are fed line-by-line into the process + as `stdin`. +10. the files named in `artifacts` are copied into the artifacts repo for this + step and version. + +#### Step Definitions + +Step definitions are JSON objects with the following keys: + +* `workingDir` *(optional)*: the working directory (relative to the project + root folder) for all commands executed as part of this step. *(defaults to + `'.'`, the project root directory)*. + +* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to + `sh`)* + +* `cmdInput` *(optional)*: an array of string that will be concatenated with + newlines separating each string and piped as input to the command for this + step. It is possible to include newlines in the strings, but not required. + Environment variables included in this input are resolved *before* it is + passed to `stdin` as `stepCmd` does not have to be a shell and is not + expected to be able to do this interpolation itself. *(defaults to `[]`)*. + +* `artifacts` *(optional)*: an array of strings representing artifacts that + will be generated by this step and should be saved for future use. Each + string should represent a path to a file (relative to the project root + directory). StrawBoss will gather all of these files and store them in the + artifacts repo for this step and build. These paths may contain environment + variable references. *(defaults to `[]`)*. + +* `depends` *(optional)*: an array of strings representing step names upon + which this step depends. *(defaults to `[]`)*. + +* `expectedEnv` *(optional)*: an array of strings representing environment + variable keys that should be present in the environment when executing this + step. If there is an environment variable that is listed here but not present + in the build environment it will cause StrawBoss to fail this step. + *(defaults to `[]`)*. + +* `dontCache` *(optional)*: boolean. StrawBoss remembers the steps it has run + and caches the artifacts generated by them. Future builds for the same + version that request this step normally will not cause StrawBoss to re-run + the step. If `dontCache` is set to `true`, the output of this step will not + be cached, causing this step to always be run when it is referenced. + +## Architecture + +The following describes the internal architecture of StrawBoss. This section is +written primarily for those wanting to work on StrawBoss itself and is not +nessecary for the setup and use of StrawBoss. + +### Artifacts repo + +In the top-level of the artifacts repo are project directories. + +#### Project Directories + +Project directories contain step directories and cached project configuration +files. + +##### Cached configuration files. + +The cached project configuration files follow this naming convention: +`configuration..json`. StrawBoss uses the file modification time to +determine which configuration file is the most recent. These cached versions of +the project configration are only intended to be used in cases where StrawBoss +is not building anything and doesn't check out a copy of the repo. For example, +when a client queries the REST API for the list of steps in a project, +StrawBoss will consult the most recently modified cached copy of the project +configuration rather than cloning the entire repo just to answer this question. +Whenever StrawBoss has a copy of the repo, it should look for the actual +configuration file in that version of the repo instead of consulting the cached +configuration files. + +##### Step and Version Directories + +A step directory contains a list of version directories using the version +string as the directory names and a list of status JSON files also named after +the version (`.status.json`). + +A version directory contains the artifacts from this step of the build, +a `stdout.log` file, and a `stderr.log` file containing the outputs of the +build process. + +### Worker Processes + +When in server mode StrawBoss has a multi-process model. There is a +supervisory process that serves the REST API, receives build requests, and +spawns worker processes. The worker processes are responsible for actually +running the builds. The supervisory process registers a handler for SIGCHLD, +using the handler to update the supervisor's knowledge of the build results and +`wait` for the process to be cleaned up. + +When launched in single-build mode there is no supervisory process. The main +process directly executes the requested build steps. diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..de864ea --- /dev/null +++ b/TODO.md @@ -0,0 +1,6 @@ +* Write a tool to convert JSON Schema into a human-readable format suitable for + documentation. Should use the description, title, and other fields from the + JSON spec. Use this for writing the JSON schema docs instead of duplicating + the description of configuration files between JSON schema and the + documentation. In other words, use the schemas as the single source of truth + and generate everything else from that.