Go to file

Jonathan Bernard c6be698572 Artifact and log lookups. Bugfixes around failure scenarios.

* Added endopint to return the logs from a run.
* Refactored the `pathVar & "/" & pathvar` pattern into `pathVar / pathVar`
  using the `ospaths` module. Cleaner code, more resistant to extra `/` bugs.
* Added endpoints and core methods to list artifacts for a build, as well as to
  retrieve specific artifacts.
* Fixed a problem with the `complete` status being overloaded. The problem was
  that in the case of a multi-step build all of the prerequisite steps will
  return a state of `complete` which will get recorded in the status file for
  the run. The run will continue, but anyone watching the run state file (via
  the API for example) had no definitive way to tell the difference between a
  sub-step completing and the requested (last) step completing. This was caught
  in the functional tests (race condition based on when it polls for the run
  status). The fix was to introduce a new build state: `stepComplete`. The
  inner `doRun` procedure uses this instead of `complete`. Now the only place
  that `complete` is set is at the end of the original call to `run`, right
  before the worker terminates. It checks the last result (from the originally
  requested step) and if this result is `stepComplete` it "finishes" the build
  by setting the state to `complete`. Because this is the only place where
  `complete` is set, an observer is now guaranteed not to see `complete` until
  all steps have run successfully.
* Fixed a long-standing bug with the request handling logic in error cases
  (like requested resources not being available). Issue has something to do
  with the way that `except` blocks become special when in an async context.
  The main jester routes block invokes the handlers in an async context. The
  effect is that one `except` block is fine, but adding more than one (to catch
  different exception types, for example) causes the return type of the route
  handler to change and not match what the outer block is expecting (a Future).
  The fix here is to wrap any exception discrimination within a single outer
  except block, re-raise the exception, and catch it inside this new
  synchronous context. Ex:

  ```nim
    try: someCall(mayFail)
    except:
      try: raise getCurrentException()
      except ExceptionType1:
        # do whatever...
      except ExceptionType2:
        # do whatever
      except:
        # general catch-all
    return true
  ```

  The return at the end is also part of the story. Jester's match handler
  allows a route to defer making a decision about whether it matches. If you
  return true from a route block Jester accepts the result as a matched route.
  If you return false, Jester discards the result and looks for another
  matching route. Normally this is taken care of by the `resp` templates
  provided by Jester, but invoking those templates within the except blocks
  also causes problems, so we manually setup the response and `return true` to
  tell Jester that, yes this route matched, use this response.
* Moved the `/service/debug/ping` endpoint back to `/ping` and removed the
  debug-only fence. I envision this as being useful as a simple healthcheck URL.

2017-12-01 07:09:35 -06:00

src

Artifact and log lookups. Bugfixes around failure scenarios.

2017-12-01 07:09:35 -06:00

.editorconfig

Finished refactor to base the build process around explicit run instances.

2017-11-23 07:30:48 -06:00

.gitignore

Split testing into unit and functional tests.

2017-05-10 11:44:46 -05:00

api.rst

WIP Moving back towards using named runs.

2017-11-22 10:47:04 -06:00

example.json

Initial version.

2017-02-17 17:54:30 -06:00

file-structure.txt

Finished refactor to base the build process around explicit run instances.

2017-11-23 07:30:48 -06:00

README.md

Finished refactor to base the build process around explicit run instances.

2017-11-23 07:30:48 -06:00

strawboss.config.json

Added support for long-lived API keys.

2017-11-25 19:38:18 -06:00

strawboss.nimble

Hacky dependency pinning to get passing builds.

2017-12-01 02:12:22 -06:00

strawboss.projectdef.json

Fix test targets in build definition.

2017-12-01 02:38:36 -06:00

TODO.md

Artifact and log lookups. Bugfixes around failure scenarios.

2017-12-01 07:09:35 -06:00

README.md

Straw Boss CI

Pipeline: series of named steps.
Step: named set of operations (make, script, something extensible). Optionally names an artifact that is the result of the step.
Artifact: resulting output from a step. strawboss may skip step execution if it already has an artifact cached from that step.
Configuration is two-part. Pipeline, step, and artifact definition are part of the project configuration (.strawboss.json? yaml?). Environment configuration lives on the strawboss server (supplies DB info, etc.).
Step execution happens within the root directory of a fresh copy of the repo. Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and the fresh copy is checked out at that reference.

Projects

StrawBoss must know about the projects it builds. For each project, StrawBoss must be configured with a unique project name and a URL or repo-specification that can be passed to git clone to clone the project.

For more details see the Service Project Definition section.

Artifacts Repo

The artifacts repository is a directory on the filesystem used to store build artifacts and information about build status. The top-level artifacts repo directory contains project build directories: one directory for every project that StrawBoss has built. Each project directory contains multiple sub-directories. Each

Configuration

There are two points of configuration when working with StrawBoss, the StrawBoss configuration file, and the individual project configurations.

The StrawBoss configuration file is used to configure the StrawBoss instance itself and stores server-side information such as the list of projects known to StrawBoss. If you are setting up StrawBoss on a server you will need to work with this configuration file.

The project configurations are used to configure the build process and options for each proejct and are stored with the projects themselves. If you are working on a project that you wish to build with StrawBoss you will be working with this configuration file.

StrawBoss configuration file

StrawBoss expects to find strawboss.config.json in the working directory of the strawboss executable. This is the configuration file for StrawBoss itself. The contents are expected to be a valid JSON object. The top level keys are:

buildDataDir: (optional) A string denoting the path to the directory where StrawBoss keeps metadata about builds it has performed and the artifacts resulting from the builds. (defaults to build-data)
authSecret: (required) Secret key used to sign JWT session tokens.
users: (required) the array of user definition objects. Each user object is required to have username and hashedPwd keys, both string.
projects: (required) an array of project definitions (detailed below).
pwdCost: (required) parameter to the user password hashing algorithm determining the computational cost of the hash.
maintenancePeriod: (optional) how often, in milliseconds, should the StrawBoss server perform maintenance (clear finished workers, etc). (defaults to 10000, every 10 seconds).
debug: boolean, should debug behavior be enabled. This is primarily intended for testing during StrawBoss development. (defaults to false)

All are required.

Service Project Definition

Project definitions are JSON objects with the following keys:

name (required): A unique name for the project. Names are case sensitive. There are no hard limits on names but the recommended convention is alpha-numeric using dashes or underscores for word separation (e.g. foo-confabulator or web_refractulator, etc.).
repo (required): A valid git URL or repo-specification that can be passed to git clone to clone the project.
defaultBranch (optional): If a run request is submitted without specifying a branch or commit reference, use this branch (defaults to master).
cfgFilePath (optional): path to the StrawBoss project configuration file (see the Project Configuration section). If this is a relative path it is resolved relative to the project root directory. (defaults to strawboss.json).
envVars (optional): an object whose key-value pairs are expected to be strings. Each pair will be added as an environment variable to the runtime environment of builds for this project. (defaults to {}).

Project Configuration

The build configuration for each project lives in the project itself. In the root directory of the project repo StrawBoss expects to find a project configuration file named strawboss.json (this location can be configured, see the Service Project Definition section). The project configuration file tells StrawBoss what steps of a project are available to be built, and how to initiate the build for each step. Steps can be dependant on each other, as long as they do not define any circular references.

The contents of a project configurations files is expected to be a valid JSON object. The top level keys are:

name (required): the unique name for the project. This should be the same name as configured in the StrawBoss service configuration file.
steps (required): an object of step definitions. Each of this object's keys is the name of a step. The corresponding value is the step definition. (see the Step Definition section for details).
versionCmd (optional): a command to be run in a shell (sh-compatible) that is expected to print the current version of the project on stdout. (defaults to git describe --tags --always).

Step Definition

Step definitions are JSON objects with the following keys:

workingDir (optional): the working directory (relative to the project root folder) for all commands executed as part of this step. (defaults to '.', the project root directory).
stepCmd (optional): the command to execute for this step. (defaults to true unless cmdInput is given, in which case it defaults to sh)
cmdInput (optional): an array of string that will be concatenated with newlines separating each string and piped as input to the command for this step. It is possible to include newlines in the strings, but not required. Environment variables included in this input are resolved before it is passed to stdin as stepCmd does not have to be a shell and is not expected to be able to do this interpolation itself. (defaults to []).
artifacts (optional): an array of strings representing artifacts that will be generated by this step and should be saved for future use. Each string should represent a path to a file (relative to the project root directory). StrawBoss will gather all of these files and store them in the artifacts repo for this step and build. These paths may contain environment variable references. (defaults to []).
depends (optional): an array of strings representing step names upon which this step depends. (defaults to []).
expectedEnv (optional): an array of strings representing environment variable keys that should be present in the environment when executing this step. If there is an environment variable that is listed here but not present in the build environment it will cause StrawBoss to fail this step. (defaults to []).
dontSkip (optional): boolean. StrawBoss remembers the steps it has run and caches the artifacts generated by them. Future builds for the same version that request this step normally will not cause StrawBoss to re-run the step. If dontSkip is set to true, the output of this step will always be run when it is referenced, regardless of previous cached results.

Build Process

When performing a build, StrawBoss:

creates a temporary workspace for this build
clones the repo into the workspace
checkout the revision or branch requested for this run
load the project's StrawBoss configuration file.
merge environment variables defined in the project configuration
run versionCmd to get the current project version. The result is stored in the VERSION environment variable.
check the environment variables against expectedEnv
check that all the steps named in depends have already been run and run them if they have not. For each step named in depends an environment variable is added named <step-name>_DIR that contains the absolute path to the artifacts repo for that step at this version. This is intended to be used to reference artifacts from other steps, e.g. ${build_DIR}/site-contents.zip.
stepCmd is executed in workingDir. Environment variables in cmdInput are resolved and the resulting string are fed line-by-line into the process as stdin.
the files named in artifacts are copied into the artifacts repo for this step and version.

Architecture

The following describes the internal architecture of StrawBoss. This section is written primarily for those wanting to work on StrawBoss itself and is not nessecary for the setup and use of StrawBoss.

Artifacts repo

In the top-level of the artifacts repo are project directories.

Project Directories

Project directories contain step directories and cached project configuration files.

Cached configuration files.

The cached project configuration files follow this naming convention: configuration.<version>.json. These cached versions of the project configration are only intended to be used in cases where StrawBoss is not building anything and doesn't check out a copy of the repo. For example, when a client queries the REST API for the list of steps in a project, StrawBoss will consult the most recently modified cached copy of the project configuration rather than cloning the entire repo just to answer this question. Whenever StrawBoss has a copy of the repo, it should look for the actual configuration file in that version of the repo instead of consulting the cached configuration files. When determining the "most recent" cached copy, StrawBoss uses the modification time of the files, again to avoid cloning the repo. API access to project configurations in this manner is intended as a convenience. The actual project configuration in the project repository should be considered the source of truth.

Step and Version Directories

A step directory contains a list of version directories using the version string as the directory names and a list of status JSON files also named after the version (<version>.status.json).

A version directory contains the artifacts from this step of the build, a stdout.log file, and a stderr.log file containing the outputs of the build process.

Worker Processes

When in server mode StrawBoss has a multi-process model. There is a supervisory process that serves the REST API, receives build requests, and spawns worker processes. The worker processes are responsible for actually running the builds. The supervisory process registers a handler for SIGCHLD, using the handler to update the supervisor's knowledge of the build results and wait for the process to be cleaned up.

When launched in single-build mode there is no supervisory process. The main process directly executes the requested build steps.

Building StrawBoss

To build StrawBoss locally, checkout the repository and in the repo root run:

nimble build

Testing

StrawBoss has two test suites, a set of unit tests and a set of functional tests. All the test code and assets live under the src/test subdirectory.

Each test suite has a runner file that serves as an entry for the test process, named run_unit_tests.nim and run_functional_tests.nim.

Unit Tests

The unit test soruce files live in the nim/unit subdirectory and have a one-to-one correspondence with the StrawBoss source files following this naming convention: t<module>.nim. The unit tests are intended to be run any time the code is recompiled.

To run the unit tests, use the unittest nimble task:

nimble unittest

Functional Tests

The functional test source files live in the nim/functional subdirectory. There is a test project that is used to excercise StrawBoss functionality. To avoid external coupling it is stored within the StrawBoss repository as a test asset. To avoid git complications it is stored as a Gzipped TAR file and unpacked to a temporary directory as part of the functional test process.

As the functional tests are more time-consuming and intensive, they are expected to be run when performing a build.

To run the functional tests, use the functest nimble task:

nimble functest