From a5b4d7d3ebc6c14fac4b02f28dcf1705ea9d1a5b Mon Sep 17 00:00:00 2001
From: Jonathan Bernard <jdbernard@gmail.com>
Date: Sun, 5 Mar 2017 23:01:06 -0600
Subject: [PATCH] Added documentation for configuration and planned out a
redesigned architecture.
---
README.md | 205 +++++++++++++++++++++++++++++++++++++++++++++++++++++-
TODO.md | 6 ++
2 files changed, 209 insertions(+), 2 deletions(-)
create mode 100644 TODO.md
diff --git a/README.md b/README.md
index 09d2983..c25167c 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,4 @@
-Straw Boss CI
-=============
+# Straw Boss CI
* Pipeline: series of named steps.
* Step: named set of operations (make, script, something extensible).
@@ -13,3 +12,205 @@ Straw Boss CI
* Step execution happens within the root directory of a fresh copy of the repo.
Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
the fresh copy is checked out at that reference.
+
+
+## Projects
+
+StrawBoss must know about the projects it builds. For each project, StrawBoss
+must be configured with a unique project name and a URL or repo-specification
+that can be passed to `git clone` to clone the project.
+
+For more details see the Service Project Definitions section.
+
+## Artifacts Repo
+
+The artifacts repository is a directory on the filesystem used to store
+build artifacts and information about build status. The top-level artifacts
+repo directory contains project build directories: one directory for every
+project that StrawBoss has built. Each project directory contains multiple
+sub-directories. Each
+
+## Configuration
+
+### StrawBoss configuration file
+
+StrawBoss expects to find `strawboss.config.json` in the working directory of
+the `strawboss` executable. This is the configuration file for StrawBoss
+itself. The contents are expected to be a valid JSON object. The top level keys
+are:
+
+* `artifactsRepo`: A string denoting the path to the artifacts repository
+ directory.
+
+* `users`: the array of user definition objects. Each user object is required
+ to have `username` and `hashedPwd` keys, both string.
+
+* `tokens`: an array of string, each representing a valid auth token that has
+ been issued to a client.
+
+* `projects`: an array of project definitions (detailed below).
+
+All are required.
+
+#### Service Project Definitions
+
+Project definitions are JSON objects with the following keys:
+
+* `name` *(required)*: A unique name for the project. Names are case sensitive.
+ There are no hard limits on names but the recommended convention is
+ alpha-numeric using dashes or underscores for word separation (e.g.
+ `foo-confabulator` or `web_refractulator`, etc.).
+
+* `repo` *(required)*: A valid `git` URL or repo-specification that can be
+ passed to `git clone` to clone the project.
+
+* `defaultBranch` *(optional)*: If a run request is submitted without
+ specifying a branch or commit reference, use this branch *(defaults to
+ `master`)*.
+
+* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file
+ (see the Project Configuration section). If this is a relative path it is
+ resolved relative to the project root directory. *(defaults to
+ `strawboss.json`)*.
+
+* `envVars` *(optional)*: an object whose key-value pairs are expected to be
+ strings. Each pair will be added as an environment variable to the runtime
+ environment of builds for this project. *(defaults to `{}`)*.
+
+### Project Configuration
+
+The build configuration for each project lives in the project itself. In the
+root directory of the project repo StrawBoss expects to find a project
+configuration file named `strawboss.json` (this location can be configured, see
+the Service Project Definitions section). The project configuration file
+tells StrawBoss what steps of a project are available to be built, and how to
+initiate the build for each step. Steps can be dependant on each other, as long
+as they do not define any circular references.
+
+The contents of a project configurations files is expected to be a valid JSON
+object. The top level keys are:
+
+* `name` *(required)*: the unique name for the project. This should be the same
+ name as configured in the StrawBoss service configuration file.
+
+* `steps` *(required)*: an object of step definitions. Each of this object's
+ keys is the name of a step. The corresponding value is the step definition.
+ (see the Step Definitions section for details).
+
+* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible)
+ that is expected to print the current version of the project on `stdout`.
+ *(defaults to `git describe --tags --always`)*.
+
+## Build Process
+
+When performing a build, StrawBoss:
+
+1. creates a temporary workspace for this build
+2. clones the repo into the workspace
+3. checkout the revision or branch requested for this run
+4. load the project's StrawBoss configuration file.
+5. merge environment variables defined in the project configuration
+6. run `versionCmd` to get the current project version
+7. check the environment variables against `expectedEnv`
+8. check that all the steps named in `depends` have already been run and run
+ them if they have not. For each step named in `depends` an environment
+ variable is added named `<step-name>_DIR` that contains the absolute path to
+ the artifacts repo for that step at this version. This is intended to be
+ used to reference artifacts from other steps, e.g.
+ `${build_DIR}/site-contents.zip`.
+9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
+ are resolved and the resulting string are fed line-by-line into the process
+ as `stdin`.
+10. the files named in `artifacts` are copied into the artifacts repo for this
+ step and version.
+
+#### Step Definitions
+
+Step definitions are JSON objects with the following keys:
+
+* `workingDir` *(optional)*: the working directory (relative to the project
+ root folder) for all commands executed as part of this step. *(defaults to
+ `'.'`, the project root directory)*.
+
+* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
+ `sh`)*
+
+* `cmdInput` *(optional)*: an array of string that will be concatenated with
+ newlines separating each string and piped as input to the command for this
+ step. It is possible to include newlines in the strings, but not required.
+ Environment variables included in this input are resolved *before* it is
+ passed to `stdin` as `stepCmd` does not have to be a shell and is not
+ expected to be able to do this interpolation itself. *(defaults to `[]`)*.
+
+* `artifacts` *(optional)*: an array of strings representing artifacts that
+ will be generated by this step and should be saved for future use. Each
+ string should represent a path to a file (relative to the project root
+ directory). StrawBoss will gather all of these files and store them in the
+ artifacts repo for this step and build. These paths may contain environment
+ variable references. *(defaults to `[]`)*.
+
+* `depends` *(optional)*: an array of strings representing step names upon
+ which this step depends. *(defaults to `[]`)*.
+
+* `expectedEnv` *(optional)*: an array of strings representing environment
+ variable keys that should be present in the environment when executing this
+ step. If there is an environment variable that is listed here but not present
+ in the build environment it will cause StrawBoss to fail this step.
+ *(defaults to `[]`)*.
+
+* `dontCache` *(optional)*: boolean. StrawBoss remembers the steps it has run
+ and caches the artifacts generated by them. Future builds for the same
+ version that request this step normally will not cause StrawBoss to re-run
+ the step. If `dontCache` is set to `true`, the output of this step will not
+ be cached, causing this step to always be run when it is referenced.
+
+## Architecture
+
+The following describes the internal architecture of StrawBoss. This section is
+written primarily for those wanting to work on StrawBoss itself and is not
+nessecary for the setup and use of StrawBoss.
+
+### Artifacts repo
+
+In the top-level of the artifacts repo are project directories.
+
+#### Project Directories
+
+Project directories contain step directories and cached project configuration
+files.
+
+##### Cached configuration files.
+
+The cached project configuration files follow this naming convention:
+`configuration.<version>.json`. StrawBoss uses the file modification time to
+determine which configuration file is the most recent. These cached versions of
+the project configration are only intended to be used in cases where StrawBoss
+is not building anything and doesn't check out a copy of the repo. For example,
+when a client queries the REST API for the list of steps in a project,
+StrawBoss will consult the most recently modified cached copy of the project
+configuration rather than cloning the entire repo just to answer this question.
+Whenever StrawBoss has a copy of the repo, it should look for the actual
+configuration file in that version of the repo instead of consulting the cached
+configuration files.
+
+##### Step and Version Directories
+
+A step directory contains a list of version directories using the version
+string as the directory names and a list of status JSON files also named after
+the version (`<version>.status.json`).
+
+A version directory contains the artifacts from this step of the build,
+a `stdout.log` file, and a `stderr.log` file containing the outputs of the
+build process.
+
+### Worker Processes
+
+When in server mode StrawBoss has a multi-process model. There is a
+supervisory process that serves the REST API, receives build requests, and
+spawns worker processes. The worker processes are responsible for actually
+running the builds. The supervisory process registers a handler for SIGCHLD,
+using the handler to update the supervisor's knowledge of the build results and
+`wait` for the process to be cleaned up.
+
+When launched in single-build mode there is no supervisory process. The main
+process directly executes the requested build steps.
diff --git a/TODO.md b/TODO.md
new file mode 100644
index 0000000..de864ea
--- /dev/null
+++ b/TODO.md
@@ -0,0 +1,6 @@
+* Write a tool to convert JSON Schema into a human-readable format suitable for
+ documentation. Should use the description, title, and other fields from the
+ JSON spec. Use this for writing the JSON schema docs instead of duplicating
+ the description of configuration files between JSON schema and the
+ documentation. In other words, use the schemas as the single source of truth
+ and generate everything else from that.