Added documentation for configuration and planned out a redesigned architecture.

README.md

@@ -1,5 +1,4 @@
-Straw Boss CI
-=============
+# Straw Boss CI
 
 * Pipeline: series of named steps.
 * Step: named set of operations (make, script, something extensible).
@@ -13,3 +12,205 @@ Straw Boss CI
 * Step execution happens within the root directory of a fresh copy of the repo.
   Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
   the fresh copy is checked out at that reference.
+
+
+## Projects
+
+StrawBoss must know about the projects it builds. For each project, StrawBoss
+must be configured with a unique project name and a URL or repo-specification
+that can be passed to `git clone` to clone the project.
+
+For more details see the Service Project Definitions section.
+
+## Artifacts Repo
+
+The artifacts repository is a directory on the filesystem used to store
+build artifacts and information about build status. The top-level artifacts
+repo directory contains project build directories: one directory for every
+project that StrawBoss has built. Each project directory contains multiple
+sub-directories. Each 
+
+## Configuration
+
+### StrawBoss configuration file
+
+StrawBoss expects to find `strawboss.config.json` in the working directory of
+the `strawboss` executable. This is the configuration file for StrawBoss
+itself. The contents are expected to be a valid JSON object. The top level keys
+are:
+
+* `artifactsRepo`: A string denoting the path to the artifacts repository
+  directory.
+
+* `users`: the array of user definition objects. Each user object is required
+  to have `username` and `hashedPwd` keys, both string.
+
+* `tokens`: an array of string, each representing a valid auth token that has
+  been issued to a client.
+
+* `projects`: an array of project definitions (detailed below).
+
+All are required.
+
+#### Service Project Definitions
+
+Project definitions are JSON objects with the following keys:
+
+* `name` *(required)*: A unique name for the project. Names are case sensitive.
+  There are no hard limits on names but the recommended convention is
+  alpha-numeric using dashes or underscores for word separation (e.g.
+  `foo-confabulator` or `web_refractulator`, etc.).
+
+* `repo` *(required)*: A valid `git` URL or repo-specification that can be
+  passed to `git clone` to clone the project.
+
+* `defaultBranch` *(optional)*: If a run request is submitted without
+  specifying a branch or commit reference, use this branch *(defaults to
+  `master`)*.
+
+* `cfgFilePath` *(optional)*: path to the StrawBoss project configuration file
+  (see the Project Configuration section). If this is a relative path it is
+  resolved relative to the project root directory. *(defaults to
+  `strawboss.json`)*.
+
+* `envVars` *(optional)*: an object whose key-value pairs are expected to be
+  strings. Each pair will be added as an environment variable to the runtime
+  environment of builds for this project. *(defaults to `{}`)*.
+
+### Project Configuration
+
+The build configuration for each project lives in the project itself. In the
+root directory of the project repo StrawBoss expects to find a project
+configuration file named `strawboss.json` (this location can be configured, see
+the Service Project Definitions section). The project configuration file
+tells StrawBoss what steps of a project are available to be built, and how to
+initiate the build for each step. Steps can be dependant on each other, as long
+as they do not define any circular references.
+
+The contents of a project configurations files is expected to be a valid JSON
+object. The top level keys are:
+
+* `name` *(required)*: the unique name for the project. This should be the same
+  name as configured in the StrawBoss service configuration file.
+
+* `steps` *(required)*: an object of step definitions. Each of this object's
+  keys is the name of a step. The corresponding value is the step definition.
+  (see the Step Definitions section for details).
+
+* `versionCmd` *(optional)*: a command to be run in a shell (`sh`-compatible)
+  that is expected to print the current version of the project on `stdout`.
+  *(defaults to `git describe --tags --always`)*.
+
+## Build Process
+
+When performing a build, StrawBoss:
+
+1.  creates a temporary workspace for this build
+2.  clones the repo into the workspace
+3.  checkout the revision or branch requested for this run
+4.  load the project's StrawBoss configuration file.
+5.  merge environment variables defined in the project configuration
+6.  run `versionCmd` to get the current project version
+7.  check the environment variables against `expectedEnv`
+8.  check that all the steps named in `depends` have already been run and run
+    them if they have not. For each step named in `depends` an environment
+    variable is added named `<step-name>_DIR` that contains the absolute path to
+    the artifacts repo for that step at this version. This is intended to be
+    used to reference artifacts from other steps, e.g.
+    `${build_DIR}/site-contents.zip`.
+9.  `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
+    are resolved and the resulting string are fed line-by-line into the process
+    as `stdin`.
+10. the files named in `artifacts` are copied into the artifacts repo for this
+    step and version.
+
+####  Step Definitions
+
+Step definitions are JSON objects with the following keys:
+
+* `workingDir` *(optional)*: the working directory (relative to the project
+  root folder) for all commands executed as part of this step. *(defaults to
+  `'.'`, the project root directory)*.
+
+* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
+  `sh`)*
+
+* `cmdInput` *(optional)*: an array of string that will be concatenated with
+  newlines separating each string and piped as input to the command for this
+  step. It is possible to include newlines in the strings, but not required.
+  Environment variables included in this input are resolved *before* it is
+  passed to `stdin` as `stepCmd` does not have to be a shell and is not
+  expected to be able to do this interpolation itself. *(defaults to `[]`)*.
+
+* `artifacts` *(optional)*: an array of strings representing artifacts that
+  will be generated by this step and should be saved for future use. Each
+  string should represent a path to a file (relative to the project root
+  directory). StrawBoss will gather all of these files and store them in the
+  artifacts repo for this step and build. These paths may contain environment
+  variable references. *(defaults to `[]`)*.
+
+*  `depends` *(optional)*: an array of strings representing step names upon
+   which this step depends.  *(defaults to `[]`)*.
+
+* `expectedEnv` *(optional)*: an array of strings representing environment
+  variable keys that should be present in the environment when executing this
+  step. If there is an environment variable that is listed here but not present
+  in the build environment it will cause StrawBoss to fail this step.
+  *(defaults to `[]`)*.
+
+* `dontCache` *(optional)*: boolean. StrawBoss remembers the steps it has run
+  and caches the artifacts generated by them.  Future builds for the same
+  version that request this step normally will not cause StrawBoss to re-run
+  the step. If `dontCache` is set to `true`, the output of this step will not
+  be cached, causing this step to always be run when it is referenced.
+
+## Architecture
+
+The following describes the internal architecture of StrawBoss. This section is
+written primarily for those wanting to work on StrawBoss itself and is not
+nessecary for the setup and use of StrawBoss.
+
+### Artifacts repo
+
+In the top-level of the artifacts repo are project directories.
+
+#### Project Directories
+
+Project directories contain step directories and cached project configuration
+files.
+
+##### Cached configuration files.
+
+The cached project configuration files follow this naming convention:
+`configuration.<version>.json`. StrawBoss uses the file modification time to
+determine which configuration file is the most recent. These cached versions of
+the project configration are only intended to be used in cases where StrawBoss
+is not building anything and doesn't check out a copy of the repo. For example,
+when a client queries the REST API for the list of steps in a project,
+StrawBoss will consult the most recently modified cached copy of the project
+configuration rather than cloning the entire repo just to answer this question.
+Whenever StrawBoss has a copy of the repo, it should look for the actual
+configuration file in that version of the repo instead of consulting the cached
+configuration files.
+
+##### Step and Version Directories
+
+A step directory contains a list of version directories using the version
+string as the directory names and a list of status JSON files also named after
+the version (`<version>.status.json`).
+
+A version directory contains the artifacts from this step of the build,
+a `stdout.log` file, and a `stderr.log` file containing the outputs of the
+build process.
+
+### Worker Processes
+
+When in server mode StrawBoss has a multi-process model. There is a
+supervisory process that serves the REST API, receives build requests, and
+spawns worker processes. The worker processes are responsible for actually
+running the builds. The supervisory process registers a handler for SIGCHLD,
+using the handler to update the supervisor's knowledge of the build results and
+`wait` for the process to be cleaned up.
+
+When launched in single-build mode there is no supervisory process. The main
+process directly executes the requested build steps.

TODO.md

@@ -0,0 +1,6 @@
+* Write a tool to convert JSON Schema into a human-readable format suitable for
+  documentation. Should use the description, title, and other fields from the
+  JSON spec. Use this for writing the JSON schema docs instead of duplicating
+  the description of configuration files between JSON schema and the
+  documentation. In other words, use the schemas as the single source of truth
+  and generate everything else from that.