Compare commits
No commits in common. "master" and "0.2.0" have entirely different histories.
@ -1,10 +0,0 @@
|
||||
[*]
|
||||
charset=utf-8
|
||||
end_of_line=lf
|
||||
indent_style=space
|
||||
indent_size=2
|
||||
max_line_length=79
|
||||
|
||||
[{.babelrc,.stylelintrc,jest.config,.eslintrc,*.bowerrc,*.jsb3,*.jsb2,*.json,*.js}]
|
||||
indent_style=space
|
||||
indent_size=2
|
||||
2
.gitignore
vendored
2
.gitignore
vendored
@ -1,5 +1,3 @@
|
||||
*.sw?
|
||||
nimcache/
|
||||
/strawboss
|
||||
src/test/nim/runtests
|
||||
src/test/nim/run_*_tests
|
||||
|
||||
150
README.md
150
README.md
@ -8,6 +8,7 @@
|
||||
* Configuration is two-part. Pipeline, step, and artifact definition are part
|
||||
of the project configuration (.strawboss.json? yaml?). Environment
|
||||
configuration lives on the strawboss server (supplies DB info, etc.).
|
||||
* REST API?
|
||||
* Step execution happens within the root directory of a fresh copy of the repo.
|
||||
Commit identifiers (hash/ref/etc.) are supplied when a build is triggered and
|
||||
the fresh copy is checked out at that reference.
|
||||
@ -32,20 +33,6 @@ sub-directories. Each
|
||||
|
||||
## Configuration
|
||||
|
||||
There are two points of configuration when working with StrawBoss, the
|
||||
[StrawBoss configuration file](#strawboss-configuration-file), and the
|
||||
individual [project configurations](#project-configuration).
|
||||
|
||||
The [StrawBoss configuration file](#strawboss-configuration-file) is used to
|
||||
configure the StrawBoss instance itself and stores server-side information such
|
||||
as the list of projects known to StrawBoss. If you are setting up StrawBoss on
|
||||
a server you will need to work with this configuration file.
|
||||
|
||||
The [project configurations](#project-configuration) are used to configure the
|
||||
build process and options for each proejct and are stored with the projects
|
||||
themselves. If you are working on a project that you wish to build with
|
||||
StrawBoss you will be working with this configuration file.
|
||||
|
||||
### StrawBoss configuration file
|
||||
|
||||
StrawBoss expects to find `strawboss.config.json` in the working directory of
|
||||
@ -53,26 +40,16 @@ the `strawboss` executable. This is the configuration file for StrawBoss
|
||||
itself. The contents are expected to be a valid JSON object. The top level keys
|
||||
are:
|
||||
|
||||
* `buildDataDir`: *(optional)* A string denoting the path to the directory
|
||||
where StrawBoss keeps metadata about builds it has performed and the
|
||||
artifacts resulting from the builds. *(defaults to `build-data`)*
|
||||
* `artifactsRepo`: A string denoting the path to the artifacts repository
|
||||
directory.
|
||||
|
||||
* `authSecret`: *(required)* Secret key used to sign JWT session tokens.
|
||||
|
||||
* `users`: *(required)* the array of user definition objects. Each user object is required
|
||||
* `users`: the array of user definition objects. Each user object is required
|
||||
to have `username` and `hashedPwd` keys, both string.
|
||||
|
||||
* `projects`: *(required)* an array of project definitions (detailed below).
|
||||
* `tokens`: an array of string, each representing a valid auth token that has
|
||||
been issued to a client.
|
||||
|
||||
* `pwdCost`: *(required)* parameter to the user password hashing algorithm determining the
|
||||
computational cost of the hash.
|
||||
|
||||
* `maintenancePeriod`: *(optional)* how often, in milliseconds, should the
|
||||
StrawBoss server perform maintenance (clear finished workers, etc).
|
||||
*(defaults to `10000`, every 10 seconds)*.
|
||||
|
||||
* `debug`: boolean, should debug behavior be enabled. This is primarily
|
||||
intended for testing during StrawBoss development. *(defaults to `false`)*
|
||||
* `projects`: an array of project definitions (detailed below).
|
||||
|
||||
All are required.
|
||||
|
||||
@ -126,6 +103,30 @@ object. The top level keys are:
|
||||
that is expected to print the current version of the project on `stdout`.
|
||||
*(defaults to `git describe --tags --always`)*.
|
||||
|
||||
## Build Process
|
||||
|
||||
When performing a build, StrawBoss:
|
||||
|
||||
1. creates a temporary workspace for this build
|
||||
2. clones the repo into the workspace
|
||||
3. checkout the revision or branch requested for this run
|
||||
4. load the project's StrawBoss configuration file.
|
||||
5. merge environment variables defined in the project configuration
|
||||
6. run `versionCmd` to get the current project version. The result is stored
|
||||
in the `VERSION` environment variable.
|
||||
7. check the environment variables against `expectedEnv`
|
||||
8. check that all the steps named in `depends` have already been run and run
|
||||
them if they have not. For each step named in `depends` an environment
|
||||
variable is added named `<step-name>_DIR` that contains the absolute path to
|
||||
the artifacts repo for that step at this version. This is intended to be
|
||||
used to reference artifacts from other steps, e.g.
|
||||
`${build_DIR}/site-contents.zip`.
|
||||
9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
|
||||
are resolved and the resulting string are fed line-by-line into the process
|
||||
as `stdin`.
|
||||
10. the files named in `artifacts` are copied into the artifacts repo for this
|
||||
step and version.
|
||||
|
||||
#### Step Definition
|
||||
|
||||
Step definitions are JSON objects with the following keys:
|
||||
@ -135,7 +136,7 @@ Step definitions are JSON objects with the following keys:
|
||||
`'.'`, the project root directory)*.
|
||||
|
||||
* `stepCmd` *(optional)*: the command to execute for this step. *(defaults to
|
||||
`true` unless `cmdInput` is given, in which case it defaults to `sh`)*
|
||||
`sh`)*
|
||||
|
||||
* `cmdInput` *(optional)*: an array of string that will be concatenated with
|
||||
newlines separating each string and piped as input to the command for this
|
||||
@ -166,30 +167,6 @@ Step definitions are JSON objects with the following keys:
|
||||
the step. If `dontSkip` is set to `true`, the output of this step will always
|
||||
be run when it is referenced, regardless of previous cached results.
|
||||
|
||||
## Build Process
|
||||
|
||||
When performing a build, StrawBoss:
|
||||
|
||||
1. creates a temporary workspace for this build
|
||||
2. clones the repo into the workspace
|
||||
3. checkout the revision or branch requested for this run
|
||||
4. load the project's StrawBoss configuration file.
|
||||
5. merge environment variables defined in the project configuration
|
||||
6. run `versionCmd` to get the current project version. The result is stored
|
||||
in the `VERSION` environment variable.
|
||||
7. check the environment variables against `expectedEnv`
|
||||
8. check that all the steps named in `depends` have already been run and run
|
||||
them if they have not. For each step named in `depends` an environment
|
||||
variable is added named `<step-name>_DIR` that contains the absolute path to
|
||||
the artifacts repo for that step at this version. This is intended to be
|
||||
used to reference artifacts from other steps, e.g.
|
||||
`${build_DIR}/site-contents.zip`.
|
||||
9. `stepCmd` is executed in `workingDir`. Environment variables in `cmdInput`
|
||||
are resolved and the resulting string are fed line-by-line into the process
|
||||
as `stdin`.
|
||||
10. the files named in `artifacts` are copied into the artifacts repo for this
|
||||
step and version.
|
||||
|
||||
## Architecture
|
||||
|
||||
The following describes the internal architecture of StrawBoss. This section is
|
||||
@ -208,19 +185,16 @@ files.
|
||||
##### Cached configuration files.
|
||||
|
||||
The cached project configuration files follow this naming convention:
|
||||
`configuration.<version>.json`. These cached versions of the project
|
||||
configration are only intended to be used in cases where StrawBoss is not
|
||||
building anything and doesn't check out a copy of the repo. For example, when a
|
||||
client queries the REST API for the list of steps in a project, StrawBoss will
|
||||
consult the most recently modified cached copy of the project configuration
|
||||
rather than cloning the entire repo just to answer this question. Whenever
|
||||
StrawBoss has a copy of the repo, it should look for the actual configuration
|
||||
file in that version of the repo instead of consulting the cached configuration
|
||||
files. When determining the "most recent" cached copy, StrawBoss uses the
|
||||
modification time of the files, again to avoid cloning the repo. API access to
|
||||
project configurations in this manner is intended as a convenience. The actual
|
||||
project configuration in the project repository should be considered the source
|
||||
of truth.
|
||||
`configuration.<version>.json`. StrawBoss uses the file modification time to
|
||||
determine which configuration file is the most recent. These cached versions of
|
||||
the project configration are only intended to be used in cases where StrawBoss
|
||||
is not building anything and doesn't check out a copy of the repo. For example,
|
||||
when a client queries the REST API for the list of steps in a project,
|
||||
StrawBoss will consult the most recently modified cached copy of the project
|
||||
configuration rather than cloning the entire repo just to answer this question.
|
||||
Whenever StrawBoss has a copy of the repo, it should look for the actual
|
||||
configuration file in that version of the repo instead of consulting the cached
|
||||
configuration files.
|
||||
|
||||
##### Step and Version Directories
|
||||
|
||||
@ -243,43 +217,3 @@ using the handler to update the supervisor's knowledge of the build results and
|
||||
|
||||
When launched in single-build mode there is no supervisory process. The main
|
||||
process directly executes the requested build steps.
|
||||
|
||||
### Building StrawBoss
|
||||
|
||||
To build StrawBoss locally, checkout the repository and in the repo root run:
|
||||
|
||||
nimble build
|
||||
|
||||
### Testing
|
||||
|
||||
StrawBoss has two test suites, a set of unit tests and a set of functional
|
||||
tests. All the test code and assets live under the `src/test` subdirectory.
|
||||
|
||||
Each test suite has a runner file that serves as an entry for the test process,
|
||||
named `run_unit_tests.nim` and `run_functional_tests.nim`.
|
||||
|
||||
#### Unit Tests
|
||||
|
||||
The unit test soruce files live in the `nim/unit` subdirectory and have a
|
||||
one-to-one correspondence with the StrawBoss source files following this
|
||||
naming convention: `t<module>.nim`. The unit tests are intended to be run any
|
||||
time the code is recompiled.
|
||||
|
||||
To run the unit tests, use the `unittest` nimble task:
|
||||
|
||||
nimble unittest
|
||||
|
||||
#### Functional Tests
|
||||
|
||||
The functional test source files live in the `nim/functional` subdirectory.
|
||||
There is a test project that is used to excercise StrawBoss functionality. To
|
||||
avoid external coupling it is stored within the StrawBoss repository as a test
|
||||
asset. To avoid `git` complications it is stored as a Gzipped TAR file and
|
||||
unpacked to a temporary directory as part of the functional test process.
|
||||
|
||||
As the functional tests are more time-consuming and intensive, they are
|
||||
expected to be run when performing a build.
|
||||
|
||||
To run the functional tests, use the `functest` nimble task:
|
||||
|
||||
nimble functest
|
||||
|
||||
17
TODO.md
17
TODO.md
@ -1,11 +1,6 @@
|
||||
TODO
|
||||
|
||||
* Orchestration of docker containers for running builds.
|
||||
* Write API docs.
|
||||
|
||||
NICE TO HAVE
|
||||
|
||||
* Use/create some json-schema -> nim code generator to auto-generate json
|
||||
handling code from schemas.
|
||||
* Use some json-schema -> docs generator to document the API.
|
||||
* Support unique UUID prefixes in URLs.
|
||||
* Write a tool to convert JSON Schema into a human-readable format suitable for
|
||||
documentation. Should use the description, title, and other fields from the
|
||||
JSON spec. Use this for writing the JSON schema docs instead of duplicating
|
||||
the description of configuration files between JSON schema and the
|
||||
documentation. In other words, use the schemas as the single source of truth
|
||||
and generate everything else from that.
|
||||
|
||||
27
api.rst
27
api.rst
@ -1,20 +1,9 @@
|
||||
✓ GET /api/ping -- returns "pong"
|
||||
✓ POST /api/auth-token -- create an return an auth token given {"username": "...", "password": "..."}
|
||||
✓ GET /api/verify-auth -- returns 200 or 401 depend on validity of the provided auth (auth ping)
|
||||
✓ GET /api/projects -- return project summaries
|
||||
- POST /api/projects -- create a new project
|
||||
- GET /api/project/<proj-id> -- TODO
|
||||
* GET /api/project/<proj-id>/runs -- list summary information for all runs
|
||||
* GET /api/project/<proj-id>/runs/active -- list summary information about all currently active runs
|
||||
- GET /api/project/<proj-id>/runs/<run-id> -- list detailed information about a specific run
|
||||
✓ GET /api/project/<proj-id>/versions -- list the versions of this project that have been built
|
||||
* GET /api/project/<proj-id>/version/<ref> -- return detailed project definition (include steps) at a specific version
|
||||
- GET /api/project/<proj-id>/step/<step-id> -- return detailed step information (include runs)
|
||||
* POST /api/project/<proj-id>/step/<step-id>/run/<ref> -- kick off a run
|
||||
GET /api/ping
|
||||
POST /api/auth-token
|
||||
GET /api/projects -- return project summaries
|
||||
POST /api/projects -- create a new project
|
||||
GET /api/project/<proj-id> -- return detailed project record (include steps)
|
||||
GET /api/project/<proj-id>/<step-id> -- return detailed step information (include runs)
|
||||
POST /api/project/<proj-id>/<step-id>/run/<ref> -- kick off a run
|
||||
GET /api/project/<proj-id>/<step-id>/run/<ref> -- return detailed run information
|
||||
|
||||
|
||||
Legend:
|
||||
✓ implemented with passing tests
|
||||
* implemented, needs testing
|
||||
- not implemented
|
||||
M missing (not even stubbed out)
|
||||
|
||||
@ -1,18 +0,0 @@
|
||||
build-data/
|
||||
<project-name>/
|
||||
configurations/
|
||||
<version>.json
|
||||
runs/
|
||||
<id>.request.json
|
||||
<id>.stdout.log
|
||||
<id>.stderr.log
|
||||
<id>.status.json
|
||||
status/
|
||||
<step-name>/
|
||||
<version>.json
|
||||
artifacts/
|
||||
<step-name>/
|
||||
<version>/
|
||||
<artifact-file>
|
||||
|
||||
workspace/
|
||||
@ -3,9 +3,6 @@
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"artifactsRepo": { "type": "string" },
|
||||
"authSecret": { "type": "string" },
|
||||
"debug": { "type": "bool" },
|
||||
"pwdCost": { "type": "integer" },
|
||||
"projects": {
|
||||
"title": "ProjectsList",
|
||||
"type": "array",
|
||||
@ -33,14 +30,19 @@
|
||||
"title": "UserDefinition",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": { "type": "string" },
|
||||
"username": { "type": "string" },
|
||||
"hashedPwd": { "type": "string" }
|
||||
},
|
||||
"required": ["name", "hashedPwd"],
|
||||
"required": ["username", "hashedPwd"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
},
|
||||
"tokens": {
|
||||
"title": "TokensList",
|
||||
"type": "array",
|
||||
"items": { "type": "string" }
|
||||
}
|
||||
},
|
||||
"required": ["artifactsRepo", "authSecret", "pwdCost", "projects", "users"],
|
||||
"required": ["artifactsRepo", "projects", "users", "tokens"],
|
||||
"additionalProperties": false
|
||||
}
|
||||
|
||||
@ -1,58 +1,58 @@
|
||||
import cliutils, docopt, os, sequtils, strutils, tempfile, uuids
|
||||
import docopt, os, sequtils, tempfile
|
||||
|
||||
import strawbosspkg/configuration
|
||||
import strawbosspkg/core
|
||||
import strawbosspkg/server
|
||||
import strawbosspkg/version
|
||||
import strawboss/private/util
|
||||
import strawboss/configuration
|
||||
import strawboss/core
|
||||
import strawboss/server
|
||||
|
||||
let SB_VER = "0.2.0"
|
||||
|
||||
proc logProcOutput*(outMsg, errMsg: TaintedString, cmd: string) =
|
||||
let prefix = if cmd.len > 0: cmd & ": " else: ""
|
||||
if outMsg.len > 0: stdout.writeLine prefix & outMsg
|
||||
if errMsg.len > 0: stderr.writeLine prefix & errMsg
|
||||
let prefix = if cmd != nil: cmd else: ""
|
||||
if outMsg != nil: echo prefix & "(stdout): " & outMsg
|
||||
if errMsg != nil: echo prefix & "(stderr): " & errMsg
|
||||
|
||||
when isMainModule:
|
||||
|
||||
var cfg = loadStrawBossConfig("strawboss.config.json")
|
||||
if not existsDir(cfg.artifactsRepo):
|
||||
echo "Artifacts repo (" & cfg.artifactsRepo & ") does not exist. Creating..."
|
||||
createDir(cfg.artifactsRepo)
|
||||
|
||||
cfg.artifactsRepo = expandFilename(cfg.artifactsRepo)
|
||||
|
||||
let doc = """
|
||||
Usage:
|
||||
strawboss serve [options]
|
||||
strawboss run <requestFile> [options]
|
||||
strawboss hashpwd <pwd>
|
||||
strawboss api-key <username>
|
||||
|
||||
strawboss serve
|
||||
strawboss run <project> <step> [options]
|
||||
|
||||
Options
|
||||
|
||||
-f --force-rebuild Force a build step to re-run even we have cached
|
||||
results from building that step before for this
|
||||
version of the project.
|
||||
|
||||
-r --reference <ref> Build the project at this commit reference.
|
||||
|
||||
-w --workspace <workspace> Use the given directory as the build workspace.
|
||||
|
||||
-c --config-file <cfgFile> Use this config file instead of the default
|
||||
(strawboss.config.json).
|
||||
"""
|
||||
|
||||
let args = docopt(doc, version = "strawboss v" & SB_VERSION)
|
||||
|
||||
let cfgFile = if args["--config-file"]: $args["--config-file"]
|
||||
else: "strawboss.config.json"
|
||||
|
||||
var cfg = loadStrawBossConfig(cfgFile)
|
||||
cfg.pathToExe = paramStr(0)
|
||||
if not existsDir(cfg.buildDataDir):
|
||||
echo "Build data directory (" & cfg.buildDataDir & ") does not exist. Creating..."
|
||||
createDir(cfg.buildDataDir)
|
||||
|
||||
cfg.buildDataDir = expandFilename(cfg.buildDataDir)
|
||||
|
||||
let args = docopt(doc, version = "strawboss v" & SB_VER)
|
||||
|
||||
echo $args
|
||||
if args["run"]:
|
||||
|
||||
var req: RunRequest
|
||||
try: req = loadRunRequest($args["<requestFile>"])
|
||||
except:
|
||||
echo "strawboss: unable to parse run request (" & $args["<requestFile>"] & ")"
|
||||
quit(QuitFailure)
|
||||
let req = RunRequest(
|
||||
projectName: $args["<project>"],
|
||||
stepName: $args["<step>"],
|
||||
buildRef: if args["--reference"]: $args["--reference"] else: nil,
|
||||
forceRebuild: args["--force-rebuild"],
|
||||
workspaceDir: if args["--workspace"]: $args["<workspace>"] else: mkdtemp())
|
||||
|
||||
try:
|
||||
|
||||
if req.workspaceDir.len == 0: req.workspaceDir = mkdtemp()
|
||||
|
||||
let status = core.run(cfg, req, logProcOutput)
|
||||
if status.state == BuildState.failed: raiseEx status.details
|
||||
let status = core.runStep(cfg, req, logProcOutput)
|
||||
if status.state == "failed": raiseEx status.details
|
||||
echo "strawboss: build passed."
|
||||
except:
|
||||
echo "strawboss: build FAILED: " & getCurrentExceptionMsg() & "."
|
||||
@ -62,12 +62,3 @@ Options
|
||||
|
||||
elif args["serve"]: server.start(cfg)
|
||||
|
||||
elif args["hashpwd"]:
|
||||
echo $cfg.pwdCost
|
||||
let pwd = server.hashPwd($args["<pwd>"], cfg.pwdCost)
|
||||
echo pwd
|
||||
echo pwd[0..28]
|
||||
|
||||
elif args["api-key"]:
|
||||
let sessionToken = server.makeApiKey(cfg, $args["<username>"])
|
||||
echo sessionToken
|
||||
|
||||
133
src/main/nim/strawboss/configuration.nim
Normal file
133
src/main/nim/strawboss/configuration.nim
Normal file
@ -0,0 +1,133 @@
|
||||
import logging, json, os, nre, sequtils, strtabs, tables
|
||||
import private/util
|
||||
|
||||
# Types
|
||||
#
|
||||
type
|
||||
BuildStatus* = object
|
||||
state*, details*: string
|
||||
|
||||
Step* = object
|
||||
name*, stepCmd*, workingDir*: string
|
||||
artifacts*, cmdInput*, depends*, expectedEnv*: seq[string]
|
||||
dontSkip*: bool
|
||||
|
||||
ProjectCfg* = object
|
||||
name*: string
|
||||
versionCmd*: string
|
||||
steps*: Table[string, Step]
|
||||
|
||||
ProjectDef* = object
|
||||
cfgFilePath*, defaultBranch*, name*, repo*: string
|
||||
envVars*: StringTableRef
|
||||
|
||||
StrawBossConfig* = object
|
||||
artifactsRepo*: string
|
||||
projects*: seq[ProjectDef]
|
||||
|
||||
RunRequest* = object
|
||||
projectName*, stepName*, buildRef*, workspaceDir*: string
|
||||
forceRebuild*: bool
|
||||
|
||||
# internal utils
|
||||
|
||||
let nullNode = newJNull()
|
||||
proc getIfExists(n: JsonNode, key: string): JsonNode =
|
||||
result = if n.hasKey(key): n[key]
|
||||
else: nullNode
|
||||
|
||||
proc getOrFail(n: JsonNode, key: string, objName: string = ""): JsonNode =
|
||||
if not n.hasKey(key): raiseEx objName & " missing key " & key
|
||||
return n[key]
|
||||
|
||||
# Configuration parsing code
|
||||
|
||||
proc loadStrawBossConfig*(cfgFile: string): StrawBossConfig =
|
||||
if not existsFile(cfgFile):
|
||||
raiseEx "strawboss config file not found: " & cfgFile
|
||||
|
||||
let jsonCfg = parseFile(cfgFile)
|
||||
|
||||
var projectDefs: seq[ProjectDef] = @[]
|
||||
|
||||
for pJson in jsonCfg.getIfExists("projects").getElems:
|
||||
var envVars = newStringTable(modeCaseSensitive)
|
||||
for k, v in pJson.getIfExists("envVars").getFields: envVars[k] = v.getStr("")
|
||||
|
||||
projectDefs.add(
|
||||
ProjectDef(
|
||||
cfgFilePath: pJson.getIfExists("cfgFilePath").getStr("strawboss.json"),
|
||||
defaultBranch: pJson.getIfExists("defaultBranch").getStr("master"),
|
||||
name: pJson.getOrFail("name", "project definition").getStr,
|
||||
envVars: envVars,
|
||||
repo: pJson.getOrFail("repo", "project definition").getStr))
|
||||
|
||||
result = StrawBossConfig(
|
||||
artifactsRepo: jsonCfg.getIfExists("artifactsRepo").getStr("artifacts"),
|
||||
projects: projectDefs)
|
||||
|
||||
proc loadProjectConfig*(cfgFile: string): ProjectCfg =
|
||||
if not existsFile(cfgFile):
|
||||
raiseEx "project config file not found: " & cfgFile
|
||||
|
||||
let jsonCfg = parseFile(cfgFile)
|
||||
|
||||
if not jsonCfg.hasKey("steps"):
|
||||
raiseEx "project configuration is missing steps definition"
|
||||
|
||||
var steps = initTable[string, Step]()
|
||||
for sName, pJson in jsonCfg.getOrFail("steps", "project configuration").getFields:
|
||||
steps[sName] = Step(
|
||||
name: sName,
|
||||
workingDir: pJson.getIfExists("workingDir").getStr("."),
|
||||
stepCmd: pJson.getIfExists("stepCmd").getStr("sh"),
|
||||
depends: pJson.getIfExists("depends").getElems.mapIt(it.getStr),
|
||||
artifacts: pJson.getIfExists("artifacts").getElems.mapIt(it.getStr),
|
||||
cmdInput: pJson.getIfExists("cmdInput").getElems.mapIt(it.getStr),
|
||||
expectedEnv: pJson.getIfExists("expectedEnv").getElems.mapIt(it.getStr),
|
||||
dontSkip: pJson.getIfExists("dontSkip").getStr("false") != "false")
|
||||
|
||||
if steps[sName].stepCmd == "sh" and steps[sName].cmdInput.len == 0:
|
||||
warn "Step " & sName & " uses 'sh' as its command but has no cmdInput."
|
||||
|
||||
result = ProjectCfg(
|
||||
name: jsonCfg.getOrFail("name", "project configuration").getStr,
|
||||
versionCmd: jsonCfg.getIfExists("versionCmd").getStr("git describe --tags --always"),
|
||||
steps: steps)
|
||||
|
||||
proc loadBuildStatus*(statusFile: string): BuildStatus =
|
||||
if not existsFile(statusFile): raiseEx "status file not found: " & statusFile
|
||||
let jsonObj = parseFile(statusFile)
|
||||
|
||||
result = BuildStatus(
|
||||
state: jsonObj.getOrFail("state", "build status").getStr,
|
||||
details: jsonObj.getIfExists("details").getStr("") )
|
||||
|
||||
proc parseRunRequest*(reqStr: string): RunRequest =
|
||||
let reqJson = parseJson(reqStr)
|
||||
|
||||
result = RunRequest(
|
||||
projectName: reqJson.getOrFail("projectName", "RunRequest").getStr,
|
||||
stepName: reqJson.getOrFail("stepName", "RunRequest").getStr,
|
||||
buildRef: reqJson.getOrFail("buildRef", "RunRequest").getStr,
|
||||
workspaceDir: reqJson.getOrFail("workspaceDir", "RunRequest").getStr,
|
||||
forceRebuild: reqJson.getOrFail("forceRebuild", "RunRequest").getBVal)
|
||||
|
||||
# TODO: can we use the marshal module for this?
|
||||
proc `%`*(s: BuildStatus): JsonNode =
|
||||
result = %* {
|
||||
"state": s.state,
|
||||
"details": s.details
|
||||
}
|
||||
|
||||
proc `%`*(req: RunRequest): JsonNode =
|
||||
result = %* {
|
||||
"projectName": req.projectName,
|
||||
"stepName": req.stepName,
|
||||
"buildRef": req.buildRef,
|
||||
"workspaceDir": req.workspaceDir,
|
||||
"forceRebuild": req.forceRebuild
|
||||
}
|
||||
|
||||
proc `$`*(s: BuildStatus): string = result = pretty(%s)
|
||||
proc `$`*(req: RunRequest): string = result = pretty(%req)
|
||||
270
src/main/nim/strawboss/core.nim
Normal file
270
src/main/nim/strawboss/core.nim
Normal file
@ -0,0 +1,270 @@
|
||||
import logging, nre, os, osproc, sequtils, streams, strtabs, strutils, tables, tempfile
|
||||
|
||||
import private/util
|
||||
import configuration
|
||||
from posix import link
|
||||
|
||||
type
|
||||
Workspace = ref object ## Data needed by internal build process
|
||||
artifactsDir*: string ## absolute path to the directory for this version
|
||||
artifactsRepo*: string ## absolute path to the global artifacts repo
|
||||
buildRef*: string ## git-style commit reference to the revision we are building
|
||||
dir*: string ## absolute path to the working directory
|
||||
env*: StringTableRef ## environment variables for all build processes
|
||||
openedFiles*: seq[File] ## all files that we have opened that need to be closed
|
||||
outputHandler*: HandleProcMsgCB ## handler for process output
|
||||
project*: ProjectCfg ## the project configuration
|
||||
projectDef*: ProjectDef ## the StrawBoss project definition
|
||||
status*: BuildStatus ## the current status of the build
|
||||
statusFile*: string ## absolute path to the build status file
|
||||
step*: Step ## the step we're building
|
||||
version*: string ## project version as returned by versionCmd
|
||||
|
||||
proc resolveEnvVars(line: string, env: StringTableRef): string =
|
||||
result = line
|
||||
for found in line.findAll(re"\$\w+|\$\{[^}]+\}"):
|
||||
let key = if found[1] == '{': found[2..^2] else: found[1..^1]
|
||||
if env.hasKey(key): result = result.replace(found, env[key])
|
||||
|
||||
proc emitStatus(status: BuildStatus, statusFilePath: string,
|
||||
outputHandler: HandleProcMsgCB): BuildStatus =
|
||||
if statusFilePath != nil: writeFile(statusFilePath, $status)
|
||||
if outputHandler != nil:
|
||||
outputHandler.sendMsg(status.state & ": " & status.details)
|
||||
result = status
|
||||
|
||||
proc publishStatus(wksp: Workspace, state, details: string) =
|
||||
let status = BuildStatus(state: state, details: details)
|
||||
wksp.status = emitStatus(status, wksp.statusFile, wksp.outputHandler)
|
||||
|
||||
proc setupProject(wksp: Workspace) =
|
||||
|
||||
# Clone the project into the $temp/repo directory
|
||||
let cloneResult = exec("git", wksp.dir,
|
||||
["clone", wksp.projectDef.repo, "repo"],
|
||||
wksp.env, {poUsePath}, wksp.outputHandler)
|
||||
|
||||
if cloneResult.exitCode != 0:
|
||||
raiseEx "unable to clone repo for '" & wksp.projectDef.name & "'"
|
||||
|
||||
# Checkout the requested ref
|
||||
let checkoutResult = exec("git", wksp.dir & "/repo",
|
||||
["checkout", wksp.buildRef],
|
||||
wksp.env, {poUsePath}, wksp.outputHandler)
|
||||
|
||||
if checkoutResult.exitCode != 0:
|
||||
raiseEx "unable to checkout ref " & wksp.buildRef &
|
||||
" for '" & wksp.projectDef.name & "'"
|
||||
|
||||
# Find the strawboss project configuration
|
||||
let projCfgFile = wksp.dir & "/repo/" & wksp.projectDef.cfgFilePath
|
||||
if not existsFile(projCfgFile):
|
||||
raiseEx "Cannot find strawboss project configuration in the project " &
|
||||
"repo (expected at '" & wksp.projectDef.cfgFilePath & "')."
|
||||
|
||||
wksp.project = loadProjectConfig(projCfgFile)
|
||||
|
||||
# Merge in the project-defined env vars
|
||||
for k, v in wksp.projectDef.envVars: wksp.env[k] = v
|
||||
|
||||
# Get the build version
|
||||
let versionProc = startProcess(
|
||||
wksp.project.versionCmd, # command
|
||||
wksp.dir & "/repo", # working dir
|
||||
[], # args
|
||||
wksp.env, # environment
|
||||
{poUsePath, poEvalCommand}) # options
|
||||
|
||||
let versionResult = waitForWithOutput(versionProc, wksp.outputHandler,
|
||||
wksp.project.versionCmd)
|
||||
|
||||
if versionResult.exitCode != 0:
|
||||
raiseEx "Version command (" & wksp.project.versionCmd &
|
||||
") returned non-zero exit code."
|
||||
|
||||
wksp.outputHandler.sendMsg "Building version " & versionResult.output.strip
|
||||
wksp.version = versionResult.output.strip
|
||||
wksp.env["VERSION"] = wksp.version
|
||||
|
||||
proc runStep*(wksp: Workspace, step: Step) =
|
||||
|
||||
let SB_EXPECTED_VARS = ["VERSION"]
|
||||
|
||||
wksp.publishStatus("running",
|
||||
"running '" & step.name & "' for version " & wksp.version &
|
||||
" from " & wksp.buildRef)
|
||||
|
||||
# Ensure all expected environment variables are present.
|
||||
for k in (step.expectedEnv & @SB_EXPECTED_VARS):
|
||||
if not wksp.env.hasKey(k):
|
||||
raiseEx "step " & step.name & " failed: missing required env variable: " & k
|
||||
|
||||
# Ensure that artifacts in steps we depend on are present
|
||||
# TODO: detect circular-references in dependency trees.
|
||||
for dep in step.depends:
|
||||
if not wksp.project.steps.hasKey(dep):
|
||||
raiseEx step.name & " depends on " & dep &
|
||||
" but there is no step named " & dep
|
||||
let depStep = wksp.project.steps[dep]
|
||||
|
||||
# Run that step (may get skipped)
|
||||
runStep(wksp, depStep)
|
||||
|
||||
# Add the artifacts directory for the dependent step to our env so that
|
||||
# further steps can reference it via $<stepname>_DIR
|
||||
wksp.env[depStep.name & "_DIR"] = wksp.artifactsRepo & "/" &
|
||||
wksp.project.name & "/" & dep & "/" & wksp.version
|
||||
|
||||
# Run the step command, piping in cmdInput
|
||||
wksp.outputHandler.sendMsg step.name & ": starting stepCmd: " & step.stepCmd
|
||||
let cmdProc = startProcess(step.stepCmd,
|
||||
wksp.dir & "/repo/" & step.workingDir,
|
||||
[], wksp.env, {poUsePath, poEvalCommand})
|
||||
|
||||
let cmdInStream = inputStream(cmdProc)
|
||||
|
||||
# Replace env variables in step cmdInput as we pipe it in
|
||||
for line in step.cmdInput: cmdInStream.writeLine(line.resolveEnvVars(wksp.env))
|
||||
cmdInStream.flush()
|
||||
cmdInStream.close()
|
||||
|
||||
let cmdResult = waitForWithOutput(cmdProc, wksp.outputHandler, step.stepCmd)
|
||||
|
||||
if cmdResult.exitCode != 0:
|
||||
raiseEx "step " & step.name & " failed: step command returned non-zero exit code"
|
||||
|
||||
# Gather the output artifacts (if we have any)
|
||||
wksp.outputHandler.sendMsg "artifacts: " & $step.artifacts
|
||||
if step.artifacts.len > 0:
|
||||
for a in step.artifacts:
|
||||
let artifactPath = a.resolveEnvVars(wksp.env)
|
||||
let artifactName = artifactPath[(artifactPath.rfind("/")+1)..^1]
|
||||
try:
|
||||
wksp.outputHandler.sendMsg "copy " & wksp.dir & "/repo/" & step.workingDir & "/" & artifactPath & " -> " & wksp.artifactsDir & "/" & artifactName
|
||||
copyFile(wksp.dir & "/repo/" & step.workingDir & "/" & artifactPath,
|
||||
wksp.artifactsDir & "/" & artifactName)
|
||||
except:
|
||||
raiseEx "step " & step.name & " failed: unable to copy artifact " &
|
||||
artifactPath & ":\n" & getCurrentExceptionMsg()
|
||||
|
||||
wksp.publishStatus("complete", "")
|
||||
|
||||
proc runStep*(cfg: StrawBossConfig, req: RunRequest,
|
||||
outputHandler: HandleProcMsgCB = nil): BuildStatus =
|
||||
|
||||
result = BuildStatus(
|
||||
state: "setup",
|
||||
details: "initializing build workspace")
|
||||
discard emitStatus(result, nil, outputHandler)
|
||||
|
||||
var wksp: Workspace
|
||||
|
||||
try:
|
||||
assert req.workspaceDir.isAbsolute
|
||||
if not existsDir(req.workspaceDir): createDir(req.workspaceDir)
|
||||
|
||||
# Find the project definition
|
||||
let matching = cfg.projects.filterIt(it.name == req.projectName)
|
||||
if matching.len == 0: raiseEx "no such project: " & req.projectName
|
||||
elif matching.len > 1: raiseEx "more than one project named : " & req.projectName
|
||||
|
||||
# Read in the existing system environment
|
||||
var env = loadEnv()
|
||||
env["GIT_DIR"] = ".git"
|
||||
|
||||
# Setup our STDOUT and STDERR files
|
||||
let stdoutFile = open(req.workspaceDir & "/stdout.log", fmWrite)
|
||||
let stderrFile = open(req.workspaceDir & "/stderr.log", fmWrite)
|
||||
|
||||
let logFilesOH = makeProcMsgHandler(stdoutFile, stderrFile)
|
||||
|
||||
wksp = Workspace(
|
||||
artifactsDir: nil,
|
||||
artifactsRepo: cfg.artifactsRepo,
|
||||
buildRef:
|
||||
if req.buildRef != nil and req.buildRef.len > 0: req.buildRef
|
||||
else: matching[0].defaultBranch,
|
||||
dir: req.workspaceDir,
|
||||
env: env,
|
||||
openedFiles: @[stdoutFile, stderrFile],
|
||||
outputHandler: combineProcMsgHandlers(outputHandler, logFilesOH),
|
||||
project: ProjectCfg(),
|
||||
projectDef: matching[0],
|
||||
status: result,
|
||||
statusFile: req.workspaceDir & "/" & "status.json",
|
||||
step: Step(),
|
||||
version: nil)
|
||||
|
||||
except:
|
||||
result = BuildStatus(state: "failed",
|
||||
details: getCurrentExceptionMsg())
|
||||
try: discard emitStatus(result, nil, outputHandler)
|
||||
except: discard ""
|
||||
|
||||
try:
|
||||
# Clone the repo and setup the working environment
|
||||
wksp.publishStatus("setup",
|
||||
"cloning project repo and preparing to run '" & req.stepName & "'")
|
||||
wksp.setupProject()
|
||||
|
||||
# Find the requested step
|
||||
if not wksp.project.steps.hasKey(req.stepName):
|
||||
raiseEx "no step name '" & req.stepName & "' for " & req.projectName
|
||||
|
||||
var step = wksp.project.steps[req.stepName]
|
||||
|
||||
# Enfore forceRebuild
|
||||
if req.forceRebuild: step.dontSkip = true
|
||||
|
||||
# Compose the path to the artifacts directory for this step and version
|
||||
wksp.artifactsDir = wksp.artifactsRepo & "/" & wksp.project.name & "/" &
|
||||
step.name & "/" & wksp.version
|
||||
|
||||
# Have we tried to build this before and are we caching the results?
|
||||
if existsFile(wksp.artifactsDir & "/status.json") and not step.dontSkip:
|
||||
let prevStatus = loadBuildStatus(wksp.artifactsDir & "/status.json")
|
||||
|
||||
# If we succeeded last time, no need to rebuild
|
||||
if prevStatus.state == "complete":
|
||||
wksp.outputHandler.sendMsg(
|
||||
"Skipping step '" & step.name & "' for version '" &
|
||||
wksp.version & "': already completed.")
|
||||
return prevStatus
|
||||
else:
|
||||
wksp.outputHandler.sendMsg(
|
||||
"Rebuilding failed step '" & step.name & "' for version '" &
|
||||
wksp.version & "'.")
|
||||
|
||||
# Make the artifacts directory if it doesn't already exist
|
||||
if not existsDir(wksp.artifactsDir): createDir(wksp.artifactsDir)
|
||||
|
||||
# Link status file and output logs to the artifacts dir
|
||||
for fn in @["status.json", "stdout.log", "stderr.log"]:
|
||||
# TODO: roll old files instead of delete them?
|
||||
if existsFile(wksp.artifactsDir & "/" & fn):
|
||||
removeFile(wksp.artifactsDir & "/" & fn)
|
||||
|
||||
if link(wksp.dir & "/" & fn, wksp.artifactsDir & "/" & fn) != 0:
|
||||
wksp.outputHandler.sendMsg(nil,
|
||||
"WARN: could not link " & fn & " to artifacts dir.")
|
||||
|
||||
runStep(wksp, step)
|
||||
|
||||
result = wksp.status
|
||||
|
||||
except:
|
||||
let msg = getCurrentExceptionMsg()
|
||||
try:
|
||||
wksp.publishStatus("failed", msg)
|
||||
result = wksp.status
|
||||
except:
|
||||
result = BuildStatus(state: "failed", details: msg)
|
||||
try: discard emitStatus(result, nil, outputHandler)
|
||||
except: discard ""
|
||||
|
||||
finally:
|
||||
if wksp != nil:
|
||||
for f in wksp.openedFiles:
|
||||
try: close(f)
|
||||
except: discard ""
|
||||
|
||||
78
src/main/nim/strawboss/private/util.nim
Normal file
78
src/main/nim/strawboss/private/util.nim
Normal file
@ -0,0 +1,78 @@
|
||||
import os, osproc, streams, strtabs
|
||||
|
||||
from posix import kill
|
||||
|
||||
type HandleProcMsgCB* = proc (outMsg: TaintedString, errMsg: TaintedString, cmd: string): void
|
||||
|
||||
proc sendMsg*(h: HandleProcMsgCB, outMsg: TaintedString, errMsg: TaintedString = nil, cmd: string = "strawboss"): void =
|
||||
if h != nil: h(outMsg, errMsg, cmd)
|
||||
|
||||
proc raiseEx*(reason: string): void =
|
||||
raise newException(Exception, reason)
|
||||
|
||||
proc envToTable*(): StringTableRef =
|
||||
result = newStringTable()
|
||||
|
||||
for k, v in envPairs():
|
||||
result[k] = v
|
||||
|
||||
proc waitForWithOutput*(p: Process, msgCB: HandleProcMsgCB,
|
||||
procCmd: string = ""):
|
||||
tuple[output: TaintedString, error: TaintedString, exitCode: int] =
|
||||
|
||||
var pout = outputStream(p)
|
||||
var perr = errorStream(p)
|
||||
|
||||
result = (TaintedString"", TaintedString"", -1)
|
||||
var line = newStringOfCap(120).TaintedString
|
||||
while true:
|
||||
if pout.readLine(line):
|
||||
msgCB.sendMsg(line, nil, procCmd)
|
||||
result[0].string.add(line.string)
|
||||
result[0].string.add("\n")
|
||||
elif perr.readLine(line):
|
||||
msgCB.sendMsg(nil, line, procCmd)
|
||||
result[1].string.add(line.string)
|
||||
result[1].string.add("\n")
|
||||
else:
|
||||
result[2] = peekExitCode(p)
|
||||
if result[2] != -1: break
|
||||
close(p)
|
||||
|
||||
proc exec*(command: string, workingDir: string = "",
|
||||
args: openArray[string] = [], env: StringTableRef = nil,
|
||||
options: set[ProcessOption] = {poUsePath},
|
||||
msgCB: HandleProcMsgCB = nil):
|
||||
tuple[output: TaintedString, error: TaintedString, exitCode: int]
|
||||
{.tags: [ExecIOEffect, ReadIOEffect], gcsafe.} =
|
||||
|
||||
var p = startProcess(command, workingDir, args, env, options)
|
||||
result = waitForWithOutput(p, msgCb, command)
|
||||
|
||||
proc loadEnv*(): StringTableRef =
|
||||
result = newStringTable()
|
||||
|
||||
for k, v in envPairs():
|
||||
result[k] = v
|
||||
|
||||
proc makeProcMsgHandler*(outSink, errSink: File): HandleProcMsgCB =
|
||||
result = proc(outMsg, errMsg: TaintedString, cmd: string): void {.closure.} =
|
||||
let prefix = if cmd != nil: cmd & ": " else: ""
|
||||
if outMsg != nil: outSink.writeLine(prefix & outMsg)
|
||||
if errMsg != nil: errSink.writeLine(prefix & errMsg)
|
||||
|
||||
proc makeProcMsgHandler*(outSink, errSink: Stream): HandleProcMsgCB =
|
||||
result = proc(outMsg, errMsg: TaintedString, cmd: string): void {.closure.} =
|
||||
let prefix = if cmd != nil: cmd & ": " else: ""
|
||||
if outMsg != nil: outSink.writeLine(prefix & outMsg)
|
||||
if errMsg != nil: errSink.writeLine(prefix & errMsg)
|
||||
|
||||
proc combineProcMsgHandlers*(a, b: HandleProcMsgCB): HandleProcMsgCB =
|
||||
if a == nil: result = b
|
||||
elif b == nil: result = a
|
||||
else:
|
||||
result = proc(cmd: string, outMsg, errMsg: TaintedString): void =
|
||||
a(cmd, outMsg, errMsg)
|
||||
b(cmd, outMsg, errMsg)
|
||||
|
||||
|
||||
38
src/main/nim/strawboss/server.nim
Normal file
38
src/main/nim/strawboss/server.nim
Normal file
@ -0,0 +1,38 @@
|
||||
import asyncdispatch, jester, json, osproc, tempfile
|
||||
|
||||
import ./configuration, ./core, private/util
|
||||
|
||||
settings:
|
||||
port = Port(8180)
|
||||
|
||||
type Worker = object
|
||||
process*: Process
|
||||
workingDir*: string
|
||||
|
||||
proc spawnWorker(req: RunRequest): Worker =
|
||||
let dir = mkdtemp()
|
||||
var args = @["run", req.projectName, req.stepName, "-r", req.buildRef, "-w", dir]
|
||||
if req.forceRebuild: args.add("-f")
|
||||
result = Worker(
|
||||
process: startProcess("strawboss", ".", args, loadEnv(), {poUsePath}),
|
||||
workingDir: dir)
|
||||
|
||||
proc start*(givenCfg: StrawBossConfig): void =
|
||||
|
||||
var workers: seq[Worker] = @[]
|
||||
|
||||
routes:
|
||||
get "/api/ping":
|
||||
resp $(%*"pong"), "application/json"
|
||||
|
||||
get "/api/projects":
|
||||
resp $(%*[]), "application/json"
|
||||
|
||||
post "/api/project/@projectName/@stepName/run/@buildRef?":
|
||||
workers.add(spawnWorker(RunRequest(
|
||||
projectName: @"projectName",
|
||||
stepName: @"stepName",
|
||||
buildRef: if @"buildRef" != "": @"buildRef" else: nil,
|
||||
forceRebuild: false))) # TODO support this with optional query params
|
||||
|
||||
runForever()
|
||||
@ -1,326 +0,0 @@
|
||||
import cliutils, logging, json, os, sequtils, strtabs, strutils, tables, times,
|
||||
unicode, uuids
|
||||
|
||||
from langutils import sameContents
|
||||
from typeinfo import toAny
|
||||
from strutils import parseEnum
|
||||
|
||||
const ISO_TIME_FORMAT = "yyyy-MM-dd'T'HH:mm:sszzz"
|
||||
|
||||
# Types
|
||||
|
||||
type
|
||||
BuildState* {.pure.} = enum
|
||||
complete, failed, queued, rejected, running, setup, stepComplete
|
||||
|
||||
BuildStatus* = object
|
||||
runId*, details*, version*: string
|
||||
state*: BuildState
|
||||
|
||||
Step* = object
|
||||
containerImage*, name*, stepCmd*, workingDir*: string
|
||||
artifacts*, cmdInput*, depends*, expectedEnv*: seq[string]
|
||||
dontSkip*: bool
|
||||
|
||||
ProjectConfig* = object
|
||||
containerImage*, name*, versionCmd*: string
|
||||
steps*: Table[string, Step]
|
||||
|
||||
ProjectDef* = object
|
||||
cfgFilePath*, defaultBranch*, name*, repo*: string
|
||||
envVars*: StringTableRef
|
||||
|
||||
RunRequest* = object
|
||||
runId*: UUID
|
||||
projectName*, stepName*, buildRef*, workspaceDir*: string
|
||||
timestamp*: DateTime
|
||||
forceRebuild*: bool
|
||||
|
||||
Run* = object
|
||||
id*: UUID
|
||||
request*: RunRequest
|
||||
status*: BuildStatus
|
||||
|
||||
RunLogs* = object
|
||||
runId*: UUID
|
||||
stdout*, stderr*: seq[string]
|
||||
|
||||
User* = object
|
||||
name*: string
|
||||
hashedPwd*: string
|
||||
|
||||
UserRef* = ref User
|
||||
|
||||
StrawBossConfig* = object
|
||||
buildDataDir*: string
|
||||
authSecret*: string
|
||||
filePath*: string
|
||||
debug*: bool
|
||||
logLevel*: Level
|
||||
pathToExe*: string
|
||||
port*: int
|
||||
projects*: seq[ProjectDef]
|
||||
pwdCost*: int8
|
||||
users*: seq[UserRef]
|
||||
maintenancePeriod*: int
|
||||
|
||||
# Equality on custom types
|
||||
proc `==`*(a, b: UserRef): bool = result = a.name == b.name
|
||||
|
||||
proc `==`*(a, b: ProjectDef): bool =
|
||||
if a.envVars.len != b.envVars.len: return false
|
||||
|
||||
for k, v in a.envVars:
|
||||
if not b.envVars.hasKey(k) or a.envVars[k] != b.envVars[k]: return false
|
||||
|
||||
return
|
||||
a.name == b.name and
|
||||
a.cfgFilePath == b.cfgFilePath and
|
||||
a.defaultBranch == b.defaultBranch and
|
||||
a.repo == b.repo
|
||||
|
||||
proc `==`*(a, b: StrawBossConfig): bool =
|
||||
result =
|
||||
a.buildDataDir == b.buildDataDir and
|
||||
a.authSecret == b.authSecret and
|
||||
a.pwdCost == b.pwdCost and
|
||||
a.port == b.port and
|
||||
a.maintenancePeriod == b.maintenancePeriod and
|
||||
a.logLevel == b.logLevel and
|
||||
sameContents(a.users, b.users) and
|
||||
sameContents(a.projects, b.projects)
|
||||
|
||||
proc `==`*(a, b: RunRequest): bool =
|
||||
result =
|
||||
a.runId == b.runId and
|
||||
a.projectName == b.projectName and
|
||||
a.stepName == b.stepName and
|
||||
a.buildRef == b.buildRef and
|
||||
a.timestamp == b.timestamp and
|
||||
a.workspaceDir == b.workspaceDir and
|
||||
a.forceRebuild == b.forceRebuild
|
||||
|
||||
# Useful utilities
|
||||
proc filesMatching*(pat: string): seq[string] = toSeq(walkFiles(pat))
|
||||
|
||||
proc raiseEx*(reason: string): void =
|
||||
raise newException(Exception, reason)
|
||||
|
||||
# internal utils
|
||||
proc getIfExists(n: JsonNode, key: string): JsonNode =
|
||||
## convenience method to get a key from a JObject or return null
|
||||
result = if n.hasKey(key): n[key]
|
||||
else: newJNull()
|
||||
|
||||
proc getOrFail(n: JsonNode, key: string, objName: string = ""): JsonNode =
|
||||
## convenience method to get a key from a JObject or raise an exception
|
||||
if not n.hasKey(key): raiseEx objName & " missing key '" & key & "'"
|
||||
return n[key]
|
||||
|
||||
# Configuration parsing code
|
||||
|
||||
proc parseLogLevel*(level: string): Level =
|
||||
let lvlStr = "lvl" & toUpperAscii(level[0]) & level[1..^1]
|
||||
result = parseEnum[Level](lvlStr)
|
||||
|
||||
proc parseProjectDef*(pJson: JsonNode): ProjectDef =
|
||||
var envVars = newStringTable(modeCaseSensitive)
|
||||
for k, v in pJson.getIfExists("envVars").getFields: envVars[k] = v.getStr("")
|
||||
|
||||
result = ProjectDef(
|
||||
cfgFilePath: pJson.getIfExists("cfgFilePath").getStr("strawboss.json"),
|
||||
defaultBranch: pJson.getIfExists("defaultBranch").getStr("master"),
|
||||
name: pJson.getOrFail("name", "project definition").getStr,
|
||||
envVars: envVars,
|
||||
repo: pJson.getOrFail("repo", "project definition").getStr)
|
||||
|
||||
proc parseStrawBossConfig*(jsonCfg: JsonNode): StrawBossConfig =
|
||||
var users: seq[UserRef] = @[]
|
||||
|
||||
for uJson in jsonCfg.getIfExists("users").getElems:
|
||||
users.add(UserRef(
|
||||
name: uJson.getOrFail("name", "user record").getStr,
|
||||
hashedPwd: uJson.getOrFail("hashedPwd", "user record").getStr))
|
||||
|
||||
result = StrawBossConfig(
|
||||
buildDataDir: jsonCfg.getIfExists("buildDataDir").getStr("build-data"),
|
||||
authSecret: jsonCfg.getOrFail("authSecret", "strawboss config").getStr,
|
||||
debug: jsonCfg.getIfExists("debug").getBool(false),
|
||||
port: int(jsonCfg.getIfExists("port").getInt(8180)),
|
||||
pwdCost: int8(jsonCfg.getOrFail("pwdCost", "strawboss config").getInt),
|
||||
projects: jsonCfg.getIfExists("projects").getElems.mapIt(parseProjectDef(it)),
|
||||
maintenancePeriod: int(jsonCfg.getIfExists("maintenancePeriod").getInt(10000)),
|
||||
logLevel: parseLogLevel(jsonCfg.getIfExists("logLevel").getStr("info")),
|
||||
users: users)
|
||||
|
||||
|
||||
proc loadStrawBossConfig*(cfgFile: string): StrawBossConfig =
|
||||
if not existsFile(cfgFile):
|
||||
raiseEx "strawboss config file not found: " & cfgFile
|
||||
|
||||
result = parseStrawBossConfig(parseFile(cfgFile))
|
||||
result.filePath = cfgFile
|
||||
|
||||
proc loadProjectConfig*(cfgFile: string): ProjectConfig =
|
||||
if not existsFile(cfgFile):
|
||||
raiseEx "project config file not found: " & cfgFile
|
||||
|
||||
let jsonCfg = parseFile(cfgFile)
|
||||
|
||||
if not jsonCfg.hasKey("steps"):
|
||||
raiseEx "project configuration is missing steps definition"
|
||||
|
||||
var steps = initTable[string, Step]()
|
||||
for sName, pJson in jsonCfg.getOrFail("steps", "project configuration").getFields:
|
||||
steps[sName] = Step(
|
||||
name: sName,
|
||||
workingDir: pJson.getIfExists("workingDir").getStr("."),
|
||||
stepCmd: pJson.getIfExists("stepCmd").getStr("NOT GIVEN"),
|
||||
depends: pJson.getIfExists("depends").getElems.mapIt(it.getStr),
|
||||
artifacts: pJson.getIfExists("artifacts").getElems.mapIt(it.getStr),
|
||||
cmdInput: pJson.getIfExists("cmdInput").getElems.mapIt(it.getStr),
|
||||
expectedEnv: pJson.getIfExists("expectedEnv").getElems.mapIt(it.getStr),
|
||||
containerImage: pJson.getIfExists("containerImage").getStr(""),
|
||||
dontSkip: pJson.getIfExists("dontSkip").getBool(false))
|
||||
|
||||
# cmdInput and stepCmd are related, so we have a conditional defaulting.
|
||||
# Four possibilities:
|
||||
|
||||
if steps[sName].stepCmd == "NOT GIVEN" and steps[sName].cmdInput.len == 0:
|
||||
# 1. Neither given: default to no-op
|
||||
steps[sName].stepCmd = "true"
|
||||
|
||||
if steps[sName].stepCmd == "NOT GIVEN" and steps[sName].cmdInput.len > 0:
|
||||
# 2. cmdInput given but not stepCmd: default stepCmd to "sh"
|
||||
steps[sName].stepCmd = "sh"
|
||||
|
||||
# 3. stepCmd given but not cmdInput & 4. both given: use them as-is
|
||||
|
||||
result = ProjectConfig(
|
||||
name: jsonCfg.getOrFail("name", "project configuration").getStr,
|
||||
containerImage: jsonCfg.getIfExists("containerImage").getStr(""),
|
||||
versionCmd: jsonCfg.getIfExists("versionCmd").getStr("git describe --tags --always"),
|
||||
steps: steps)
|
||||
|
||||
proc parseBuildStatus*(statusJson: JsonNode): BuildStatus =
|
||||
result = BuildStatus(
|
||||
runId: statusJson.getOrFail("runId", "run ID").getStr,
|
||||
state: parseEnum[BuildState](statusJson.getOrFail("state", "build status").getStr),
|
||||
details: statusJson.getIfExists("details").getStr("") )
|
||||
|
||||
proc loadBuildStatus*(statusFile: string): BuildStatus =
|
||||
if not existsFile(statusFile): raiseEx "status file not found: " & statusFile
|
||||
let jsonObj = parseFile(statusFile)
|
||||
|
||||
result = parseBuildStatus(jsonObj)
|
||||
|
||||
proc parseRunRequest*(reqJson: JsonNode): RunRequest =
|
||||
result = RunRequest(
|
||||
runId: parseUUID(reqJson.getOrFail("runId", "RunRequest").getStr),
|
||||
projectName: reqJson.getOrFail("projectName", "RunRequest").getStr,
|
||||
stepName: reqJson.getOrFail("stepName", "RunRequest").getStr,
|
||||
buildRef: reqJson.getOrFail("buildRef", "RunRequest").getStr,
|
||||
workspaceDir: reqJson.getOrFail("workspaceDir", "RunRequest").getStr,
|
||||
timestamp: times.parse(reqJson.getOrFail("timestamp", "RunRequest").getStr, ISO_TIME_FORMAT),
|
||||
forceRebuild: reqJson.getOrFail("forceRebuild", "RunRequest").getBool)
|
||||
|
||||
proc loadRunRequest*(reqFilePath: string): RunRequest =
|
||||
if not existsFile(reqFilePath):
|
||||
raiseEx "request file not found: " & reqFilePath
|
||||
|
||||
parseRunRequest(parseFile(reqFilePath))
|
||||
|
||||
proc parseRun*(runJson: JsonNode): Run =
|
||||
result = Run(
|
||||
id: parseUUID(runJson.getOrFail("id", "Run").getStr),
|
||||
request: parseRunRequest(runJson.getOrFail("request", "Run")),
|
||||
status: parseBuildStatus(runJson.getOrFail("status", "Run")))
|
||||
|
||||
# TODO: can we use the marshal module for this?
|
||||
proc `%`*(s: BuildStatus): JsonNode =
|
||||
result = %* {
|
||||
"runId": s.runId,
|
||||
"state": $s.state,
|
||||
"details": s.details }
|
||||
|
||||
proc `%`*(p: ProjectDef): JsonNode =
|
||||
result = %* {
|
||||
"name": p.name,
|
||||
"cfgFilePath": p.cfgFilePath,
|
||||
"defaultBranch": p.defaultBranch,
|
||||
"repo": p.repo }
|
||||
|
||||
result["envVars"] = newJObject()
|
||||
for k, v in p.envVars: result["envVars"][k] = %v
|
||||
|
||||
proc `%`*(s: Step): JsonNode =
|
||||
result = %* {
|
||||
"name": s.name,
|
||||
"stepCmd": s.stepCmd,
|
||||
"workingDir": s.workingDir,
|
||||
"artifacts": s.artifacts,
|
||||
"cmdInput": s.cmdInput,
|
||||
"depends": s.depends,
|
||||
"expectedEnv": s.expectedEnv,
|
||||
"dontSkip": s.dontSkip }
|
||||
|
||||
if s.containerImage.len > 0:
|
||||
result["containerImage"] = %s.containerImage
|
||||
|
||||
proc `%`*(p: ProjectConfig): JsonNode =
|
||||
result = %* {
|
||||
"name": p.name,
|
||||
"versionCmd": p.versionCmd }
|
||||
|
||||
result["steps"] = newJObject()
|
||||
for name, step in p.steps:
|
||||
result["steps"][name] = %step
|
||||
|
||||
if p.containerImage.len > 0:
|
||||
result["containerImage"] = %p.containerImage
|
||||
|
||||
proc `%`*(req: RunRequest): JsonNode =
|
||||
result = %* {
|
||||
"runId": $(req.runId),
|
||||
"projectName": req.projectName,
|
||||
"stepName": req.stepName,
|
||||
"buildRef": req.buildRef,
|
||||
"workspaceDir": req.workspaceDir,
|
||||
"forceRebuild": req.forceRebuild,
|
||||
"timestamp": req.timestamp.format(ISO_TIME_FORMAT) }
|
||||
|
||||
proc `%`*(user: User): JsonNode =
|
||||
result = %* {
|
||||
"name": user.name,
|
||||
"hashedPwd": user.hashedPwd }
|
||||
|
||||
proc `%`*(cfg: StrawBossConfig): JsonNode =
|
||||
result = %* {
|
||||
"buildDataDir": cfg.buildDataDir,
|
||||
"authSecret": cfg.authSecret,
|
||||
"debug": cfg.debug,
|
||||
"port": cfg.port,
|
||||
"projects": %cfg.projects,
|
||||
"pwdCost": cfg.pwdCost,
|
||||
"maintenancePeriod": cfg.maintenancePeriod,
|
||||
"logLevel": toLowerAscii(($cfg.logLevel)[3]) & ($cfg.logLevel)[4..^1],
|
||||
"users": %cfg.users }
|
||||
|
||||
proc `%`*(run: Run): JsonNode =
|
||||
result = %* {
|
||||
"id": $run.id,
|
||||
"request": %run.request,
|
||||
"status": %run.status }
|
||||
|
||||
proc `%`*(logs: RunLogs): JsonNode =
|
||||
result = %* {
|
||||
"runId": $logs.runId,
|
||||
"stdout": %logs.stdout,
|
||||
"stderr": %logs.stderr }
|
||||
|
||||
proc `$`*(s: BuildStatus): string = result = pretty(%s)
|
||||
proc `$`*(req: RunRequest): string = result = pretty(%req)
|
||||
proc `$`*(pd: ProjectDef): string = result = pretty(%pd)
|
||||
proc `$`*(cfg: StrawBossConfig): string = result = pretty(%cfg)
|
||||
proc `$`*(run: Run): string = result = pretty(%run)
|
||||
proc `$`*(logs: RunLogs): string = result = pretty(%logs)
|
||||
@ -1,618 +0,0 @@
|
||||
import cliutils, logging, json, os, ospaths, osproc, sequtils, streams,
|
||||
strtabs, strutils, tables, tempfile, times, uuids
|
||||
|
||||
import ./configuration
|
||||
import nre except toSeq
|
||||
from posix import link, realpath
|
||||
from algorithm import sorted
|
||||
|
||||
type
|
||||
Workspace = ref object ## Data needed by internal build process
|
||||
buildDataDir*: string ## absolute path to the global build data directory for this project
|
||||
buildRef*: string ## git-style commit reference to the revision we are building
|
||||
dir*: string ## absolute path to the working directory
|
||||
env*: StringTableRef ## environment variables for all build processes
|
||||
logLevel*: Level ## log level for output messages
|
||||
openedFiles*: seq[File] ## all files that we have opened that need to be closed
|
||||
outputHandler*: HandleProcMsgCB ## handler for process output
|
||||
project*: ProjectConfig ## the project configuration
|
||||
projectDef*: ProjectDef ## the StrawBoss project definition
|
||||
runRequest*: RunRequest ## the RunRequest that initated the current build
|
||||
status*: BuildStatus ## the current status of the build
|
||||
step*: Step ## the step we're building
|
||||
version*: string ## project version as returned by versionCmd
|
||||
|
||||
Worker* = object
|
||||
runId*: UUID
|
||||
projectName*: string
|
||||
process*: Process
|
||||
|
||||
NotFoundException* = object of Exception
|
||||
|
||||
proc newCopy(w: Workspace): Workspace =
|
||||
var newEnv: StringTableRef = newStringTable()
|
||||
newEnv[] = w.env[]
|
||||
|
||||
result = Workspace(
|
||||
buildDataDir: w.buildDataDir,
|
||||
buildRef: w.buildRef,
|
||||
dir: w.dir,
|
||||
env: newEnv,
|
||||
logLevel: w.logLevel,
|
||||
# workspaces are only responsible for files they have actually openend
|
||||
openedFiles: @[],
|
||||
outputHandler: w.outputHandler,
|
||||
project: w.project,
|
||||
projectDef: w.projectDef,
|
||||
runRequest: w.runRequest,
|
||||
status: w.status,
|
||||
step: w.step,
|
||||
version: w.version)
|
||||
|
||||
const WKSP_ROOT = "/strawboss/wksp"
|
||||
const ARTIFACTS_ROOT = "/strawboss/artifacts"
|
||||
|
||||
proc execWithOutput(wksp: Workspace, cmd, workingDir: string,
|
||||
args: openarray[string], env: StringTableRef,
|
||||
options: set[ProcessOption] = {poUsePath},
|
||||
msgCB: HandleProcMsgCB = nil):
|
||||
tuple[output: TaintedString, error: TaintedString, exitCode: int]
|
||||
{.tags: [ExecIOEffect, ReadIOEffect, RootEffect] .} =
|
||||
|
||||
# Look for a container image to use
|
||||
let containerImage =
|
||||
if wksp.step.containerImage.len > 0: wksp.step.containerImage
|
||||
else: wksp.project.containerImage
|
||||
|
||||
if containerImage.len == 0:
|
||||
return execWithOutput(cmd, workingDir, args, env, options, msgCB)
|
||||
|
||||
var fullEnv = newStringTable(modeCaseSensitive)
|
||||
for k,v in env: fullEnv[k] = v
|
||||
|
||||
var fullArgs = @["run", "-w", WKSP_ROOT, "-v", wksp.dir & ":" & WKSP_ROOT ]
|
||||
|
||||
if wksp.step.name.len == 0:
|
||||
for depStep in wksp.step.depends:
|
||||
fullArgs.add(["-v", ARTIFACTS_ROOT / depStep])
|
||||
fullEnv[depStep & "_DIR"] = ARTIFACTS_ROOT / depStep
|
||||
|
||||
let envFile = mkstemp().name
|
||||
writeFile(envFile, toSeq(fullEnv.pairs()).mapIt(it[0] & "=" & it[1]).join("\n"))
|
||||
|
||||
fullArgs.add(["--env-file", envFile])
|
||||
fullArgs.add(containerImage)
|
||||
fullArgs.add(cmd)
|
||||
|
||||
echo "Executing docker command: \n\t" & "docker " & $(fullArgs & @args)
|
||||
return execWithOutput("docker", wksp.dir, fullArgs & @args, fullEnv, options, msgCB)
|
||||
|
||||
proc exec(w: Workspace, cmd, workingDir: string, args: openarray[string],
|
||||
env: StringTableRef, options: set[ProcessOption] = {poUsePath},
|
||||
msgCB: HandleProcMsgCB = nil): int
|
||||
{.tags: [ExecIOEffect, ReadIOEffect, RootEffect] .} =
|
||||
|
||||
return execWithOutput(w, cmd, workingDir, args, env, options, msgCB)[2]
|
||||
|
||||
# Utility methods for Workspace activities
|
||||
proc sendStatusMsg(oh: HandleProcMsgCB, status: BuildStatus): void =
|
||||
if not oh.isNil:
|
||||
oh.sendMsg($status.state & ": " & status.details, "", "strawboss")
|
||||
|
||||
proc sendMsg(w: Workspace, msg: TaintedString): void =
|
||||
w.outputHandler.sendMsg(msg, "", "strawboss")
|
||||
|
||||
proc sendMsg(w: Workspace, l: Level, msg: TaintedString): void =
|
||||
if l >= w.logLevel: w.sendMsg(msg)
|
||||
|
||||
proc sendErrMsg(w: Workspace, msg: TaintedString): void =
|
||||
w.outputHandler.sendMsg("", msg, "strawboss")
|
||||
|
||||
proc sendErrMsg(w: Workspace, l: Level, msg: TaintedString): void =
|
||||
if l >= w.logLevel: w.sendErrMsg(msg)
|
||||
|
||||
proc resolveEnvVars(wksp: Workspace, line: string): string =
|
||||
result = line
|
||||
for found in line.findAll(re"\$\w+|\$\{[^}]+\}"):
|
||||
let key = if found[1] == '{': found[2..^2] else: found[1..^1]
|
||||
if wksp.env.hasKey(key): result = result.replace(found, wksp.env[key])
|
||||
wksp.sendMsg(lvlDebug, "Variable substitution: \n\t" & line &
|
||||
"\n\t" & result)
|
||||
|
||||
proc publishStatus(wksp: Workspace, state: BuildState, details: string): void =
|
||||
## Update the status for a Workspace and publish this status to the
|
||||
## Workspace's status file and any output message handlers.
|
||||
wksp.status = BuildStatus(
|
||||
runId: $wksp.runRequest.runId,
|
||||
state: state,
|
||||
details: details,
|
||||
version: wksp.version)
|
||||
|
||||
# Write to our run directory, and to our version status
|
||||
writeFile(wksp.buildDataDir / "runs" /
|
||||
$wksp.runRequest.runId & ".status.json", $wksp.status)
|
||||
|
||||
# If we have our step we can save status to the step status
|
||||
if wksp.step.name.len > 0:
|
||||
let stepStatusDir = wksp.buildDataDir / "status" / wksp.step.name
|
||||
if not existsDir(stepStatusDir): createDir(stepStatusDir)
|
||||
writeFile(stepStatusDir / wksp.version & ".json", $wksp.status)
|
||||
|
||||
# If we were asked to build a ref that is not the version directly (like
|
||||
# "master" or something), then let's also save our status under that name.
|
||||
# We're probably overwriting a prior status, but that's OK.
|
||||
if wksp.runRequest.buildRef != wksp.version:
|
||||
writeFile(stepStatusDir / wksp.runRequest.buildRef & ".json",
|
||||
$wksp.status)
|
||||
|
||||
wksp.outputHandler.sendStatusMsg(wksp.status)
|
||||
|
||||
proc ensureProjectDirsExist(cfg: StrawBossConfig, p: ProjectDef): void =
|
||||
for subdir in ["configurations", "runs", "status", "artifacts"]:
|
||||
let fullPath = cfg.buildDataDir / p.name / subdir
|
||||
if not existsDir(fullPath):
|
||||
createDir(fullPath)
|
||||
|
||||
# Data and configuration access
|
||||
|
||||
proc getProject*(cfg: StrawBossConfig, projectName: string): ProjectDef =
|
||||
## Get a project definition by name from the service configuration
|
||||
let candidates = cfg.projects.filterIt(it.name == projectName)
|
||||
if candidates.len == 0:
|
||||
raise newException(NotFoundException, "no project named " & projectName)
|
||||
elif candidates.len > 1:
|
||||
raise newException(NotFoundException, "multiple projects named " & projectName)
|
||||
else: result = candidates[0]
|
||||
|
||||
proc setProject*(cfg: var StrawBossConfig, projectName: string, newDef: ProjectDef): void =
|
||||
## Add a project definition to the service configuration
|
||||
var found = false
|
||||
for idx in 0..<cfg.projects.len:
|
||||
if cfg.projects[idx].name == projectName:
|
||||
cfg.projects[idx] = newDef
|
||||
found = true
|
||||
break
|
||||
|
||||
if not found: cfg.projects.add(newDef)
|
||||
|
||||
proc listVersions*(cfg: StrawBossConfig, projectName: string): seq[string] =
|
||||
## List the versions that have been built for a project.
|
||||
|
||||
let project = cfg.getProject(projectName)
|
||||
|
||||
ensureProjectDirsExist(cfg, project)
|
||||
|
||||
let versionFiles = filesMatching(
|
||||
cfg.buildDataDir / project.name / "configurations/*.json")
|
||||
|
||||
result = versionFiles.map(proc(s: string): string =
|
||||
let slashIdx = s.rfind('/')
|
||||
result = s[(slashIdx + 1)..^6])
|
||||
|
||||
proc getBuildStatus*(cfg: StrawBossConfig,
|
||||
projectName, stepName, buildRef: string): BuildStatus =
|
||||
|
||||
let project = cfg.getProject(projectName)
|
||||
|
||||
let statusFile = cfg.buildDataDir / project.name / "status" /
|
||||
stepName / buildRef & ".json"
|
||||
|
||||
if not existsFile(statusFile):
|
||||
raise newException(NotFoundException,
|
||||
stepName & " has never been built for " & projectName & "@" & buildRef)
|
||||
|
||||
result = loadBuildStatus(statusFile)
|
||||
|
||||
|
||||
proc listArtifacts*(cfg: StrawBossConfig,
|
||||
projectName, stepName, version: string): seq[string] =
|
||||
## List the artifacts that have been built for a step.
|
||||
|
||||
let project = cfg.getProject(projectName)
|
||||
|
||||
ensureProjectDirsExist(cfg, project)
|
||||
|
||||
let buildStatus = cfg.getBuildStatus(projectName, stepName, version)
|
||||
|
||||
if buildStatus.state != BuildState.complete:
|
||||
raise newException(NotFoundException, "step " & stepName &
|
||||
" has never been successfully built for " & projectName & "@" & version)
|
||||
|
||||
result = filesMatching(
|
||||
cfg.buildDataDir / project.name / "artifacts" / stepName / version / "*")
|
||||
.mapIt(it.extractFilename)
|
||||
|
||||
proc getArtifactPath*(cfg: StrawBossConfig,
|
||||
projectName, stepName, version, artifactName: string): string =
|
||||
|
||||
let artifacts = cfg.listArtifacts(projectName, stepName, version)
|
||||
if not artifacts.contains(artifactName):
|
||||
raise newException(NotFoundException, "no artifact named " &
|
||||
artifactName & " exists for step " & stepName & " in project " &
|
||||
projectName & "@" & version)
|
||||
|
||||
result = cfg.buildDataDir / projectName / "artifacts" / stepName / version / artifactName
|
||||
|
||||
proc existsRun*(cfg: StrawBossConfig, projectName, runId: string): bool =
|
||||
existsFile(cfg.buildDataDir / projectName / "runs" / runId & ".request.json")
|
||||
|
||||
proc getRun*(cfg: StrawBossConfig, projectName, runId: string): Run =
|
||||
let project = cfg.getProject(projectName)
|
||||
let runsPath = cfg.buildDataDir / project.name / "runs"
|
||||
|
||||
try: result = Run(
|
||||
id: parseUUID(runId),
|
||||
request: loadRunRequest(runsPath / runId & ".request.json"),
|
||||
status: loadBuildStatus(runsPath / runId & ".status.json"))
|
||||
except: raiseEx "unable to load run information for id " & runId
|
||||
|
||||
proc listRuns*(cfg: StrawBossConfig, projectName: string): seq[Run] =
|
||||
## List the runs that have been performed for a project.
|
||||
let project = cfg.getProject(projectName)
|
||||
ensureProjectDirsExist(cfg, project)
|
||||
|
||||
let runsPath = cfg.buildDataDir / project.name / "runs"
|
||||
let reqPaths = filesMatching(runsPath / "*.request.json")
|
||||
|
||||
result = reqPaths.map(proc(reqPath: string): Run =
|
||||
let runId = reqPath[(runsPath.len + 1)..^14]
|
||||
result = Run(
|
||||
id: parseUUID(runId),
|
||||
request: loadRunRequest(reqPath),
|
||||
status: loadBuildStatus(runsPath / runId & ".status.json")))
|
||||
|
||||
proc getLogs*(cfg: StrawBossConfig, projectname, runId: string): RunLogs =
|
||||
let project = cfg.getProject(projectName)
|
||||
let runsPath = cfg.buildDataDir / project.name / "runs"
|
||||
|
||||
try: result = RunLogs(
|
||||
runId: parseUUID(runId),
|
||||
stdout: toSeq(lines(runsPath / runId & ".stdout.log")),
|
||||
stderr: toSeq(lines(runsPath / runId & ".stderr.log")))
|
||||
except: raiseEx "unable to load logs for run " & runId
|
||||
|
||||
proc getProjectConfig*(cfg: StrawBossConfig,
|
||||
projectName, version: string): ProjectConfig =
|
||||
|
||||
let project = cfg.getProject(projectName)
|
||||
ensureProjectDirsExist(cfg, project)
|
||||
|
||||
# If they didn't give us a version, let try to figure out what is the latest one.
|
||||
var confFilePath: string
|
||||
|
||||
if version.len == 0:
|
||||
|
||||
let candidatePaths = filesMatching(
|
||||
cfg.buildDataDir / project.name / "configurations/*.json")
|
||||
|
||||
if candidatePaths.len == 0:
|
||||
raise newException(NotFoundException,
|
||||
"no versions of this project have been built")
|
||||
|
||||
let modTimes = candidatePaths.mapIt(it.getLastModificationTime)
|
||||
confFilePath = sorted(zip(candidatePaths, modTimes),
|
||||
proc(a, b: tuple): int = cmp(a.b, b.b))[0].a
|
||||
|
||||
#cachedFilePath = sorted(zip(confFilePaths, modTimes),
|
||||
# proc (a, b: tuple): int = cmp(a.b, b.b))[0].a
|
||||
|
||||
# If they did, let's try to load that
|
||||
else:
|
||||
confFilePath =
|
||||
cfg.buildDataDir / project.name / "configurations" / version & ".json"
|
||||
|
||||
if not existsFile(confFilePath):
|
||||
raise newException(NotFoundException,
|
||||
projectName & " version " & version & " has never been built")
|
||||
|
||||
result = loadProjectConfig(confFilePath)
|
||||
|
||||
|
||||
# Internal working methods.
|
||||
proc setupProject(wksp: Workspace) =
|
||||
|
||||
wksp.sendMsg(lvlDebug, "Setting up project.")
|
||||
|
||||
# Clone the project into the $temp directory
|
||||
let cloneArgs = @["clone", wksp.projectDef.repo, wksp.dir]
|
||||
wksp.sendMsg(lvlDebug, "git " & $cloneArgs)
|
||||
|
||||
let cloneResult = exec("git", ".", cloneArgs, wksp.env, {poUsePath},
|
||||
wksp.outputHandler)
|
||||
|
||||
if cloneResult != 0:
|
||||
raiseEx "unable to clone repo for '" & wksp.projectDef.name & "'"
|
||||
|
||||
# Checkout the requested ref
|
||||
let checkoutArgs = @["checkout", wksp.buildRef]
|
||||
wksp.sendMsg(lvlDebug, "git " & $checkoutArgs)
|
||||
|
||||
let checkoutResult = exec("git", wksp.dir, checkoutArgs,
|
||||
wksp.env, {poUsePath}, wksp.outputHandler)
|
||||
|
||||
if checkoutResult != 0:
|
||||
raiseEx "unable to checkout ref " & wksp.buildRef &
|
||||
" for '" & wksp.projectDef.name & "'"
|
||||
|
||||
# Find the strawboss project configuration
|
||||
let projCfgFile = wksp.dir / wksp.projectDef.cfgFilePath
|
||||
wksp.sendMsg(lvlDebug, "Looking for project configuration at '" & projCfgFile & "'")
|
||||
if not existsFile(projCfgFile):
|
||||
raiseEx "Cannot find strawboss project configuration in the project " &
|
||||
"repo (expected at '" & wksp.projectDef.cfgFilePath & "')."
|
||||
|
||||
wksp.project = loadProjectConfig(projCfgFile)
|
||||
|
||||
# Merge in the project-defined env vars
|
||||
for k, v in wksp.projectDef.envVars: wksp.env[k] = v
|
||||
|
||||
# Get the build version
|
||||
let versionResult = execWithOutput(
|
||||
wksp.project.versionCmd, # command
|
||||
wksp.dir, # working dir
|
||||
[], # args
|
||||
wksp.env, # environment
|
||||
{poUsePath, poEvalCommand}) # options
|
||||
|
||||
if versionResult.exitCode != 0:
|
||||
raiseEx "Version command (" & wksp.project.versionCmd &
|
||||
") returned non-zero exit code."
|
||||
|
||||
wksp.version = versionResult.output.strip
|
||||
wksp.env["VERSION"] = wksp.version
|
||||
|
||||
proc doStep*(wksp: Workspace, step: Step): BuildStatus =
|
||||
|
||||
## Lower-level method to execute a given step within the context of a project
|
||||
## workspace that is setup and configured. May be called recursively to
|
||||
## satisfy step dependencies.
|
||||
|
||||
wksp.step = step
|
||||
|
||||
let artifactsDir = wksp.buildDataDir / "artifacts" / step.name / wksp.version
|
||||
|
||||
if not existsDir(artifactsDir): createDir(artifactsDir)
|
||||
|
||||
# Have we tried to build this before and are we caching the results?
|
||||
let statusFilePath = wksp.buildDataDir / "status" / step.name /
|
||||
wksp.version & ".json"
|
||||
|
||||
if existsFile(statusFilePath) and not step.dontSkip:
|
||||
let prevStatus = loadBuildStatus(statusFilePath)
|
||||
|
||||
# If we succeeded last time, no need to rebuild
|
||||
if prevStatus.state == BuildState.complete:
|
||||
wksp.publishStatus(BuildState.stepComplete,
|
||||
"Skipping step '" & step.name & "' for version '" & wksp.version &
|
||||
"': already completed.")
|
||||
return wksp.status
|
||||
else:
|
||||
wksp.sendMsg(
|
||||
"Rebuilding failed step '" & step.name & "' for version '" &
|
||||
wksp.version & "'.")
|
||||
|
||||
let SB_EXPECTED_VARS = ["VERSION"]
|
||||
|
||||
wksp.publishStatus(BuildState.running,
|
||||
"running '" & step.name & "' for version " & wksp.version &
|
||||
" from " & wksp.buildRef)
|
||||
|
||||
# Ensure all expected environment variables are present.
|
||||
for k in (step.expectedEnv & @SB_EXPECTED_VARS):
|
||||
if not wksp.env.hasKey(k):
|
||||
raiseEx "step " & step.name & " failed: missing required env variable: " & k
|
||||
|
||||
# Ensure that artifacts in steps we depend on are present
|
||||
# TODO: detect circular-references in dependency trees.
|
||||
for dep in step.depends:
|
||||
if not wksp.project.steps.hasKey(dep):
|
||||
raiseEx step.name & " depends on " & dep &
|
||||
" but there is no step named " & dep
|
||||
let depStep = wksp.project.steps[dep]
|
||||
|
||||
# Run that step (may get skipped)
|
||||
let runStatus = doStep(core.newCopy(wksp), depStep)
|
||||
|
||||
if not (runStatus.state == BuildState.stepComplete):
|
||||
raiseEx "dependent step failed: " & depStep.name
|
||||
|
||||
wksp.sendMsg(lvlDebug, "dependent step '" & depStep.name &
|
||||
"'completed, resuming '" & wksp.step.name & "'")
|
||||
|
||||
# Add the artifacts directory for the dependent step to our env so that
|
||||
# further steps can reference it via $<stepname>_DIR
|
||||
wksp.env[depStep.name & "_DIR"] = wksp.buildDataDir / "artifacts" /
|
||||
dep / wksp.version
|
||||
|
||||
# Run the step command, piping in cmdInput
|
||||
let stepCmd = wksp.resolveEnvVars(step.stepCmd)
|
||||
let cmdName = if stepCmd.rfind("/") >= 0: stepCmd[(stepCmd.rfind("/") + 1)..^1]
|
||||
else: stepCmd
|
||||
wksp.sendMsg step.name & ": starting stepCmd: " & stepCmd
|
||||
let cmdProc = startProcess(stepCmd,
|
||||
wksp.dir / step.workingDir, [], wksp.env, {poUsePath, poEvalCommand})
|
||||
|
||||
let cmdInStream = inputStream(cmdProc)
|
||||
|
||||
# Replace env variables in step cmdInput as we pipe it in
|
||||
for line in step.cmdInput: cmdInStream.writeLine(wksp.resolveEnvVars(line))
|
||||
cmdInStream.flush()
|
||||
cmdInStream.close()
|
||||
|
||||
let cmdResult = waitFor(cmdProc, wksp.outputHandler, cmdName)
|
||||
|
||||
if cmdResult != 0:
|
||||
raiseEx "step " & step.name & " failed: step command returned non-zero exit code"
|
||||
|
||||
# Gather the output artifacts (if we have any)
|
||||
wksp.sendMsg "artifacts: " & $step.artifacts
|
||||
if step.artifacts.len > 0:
|
||||
for a in step.artifacts:
|
||||
let artifactPath = wksp.resolveEnvVars(a)
|
||||
let artifactName = artifactPath[(artifactPath.rfind("/")+1)..^1]
|
||||
try:
|
||||
wksp.sendMsg "copy " &
|
||||
wksp.dir / step.workingDir / artifactPath & " -> " &
|
||||
artifactsDir / artifactName
|
||||
|
||||
copyFileWithPermissions(wksp.dir / step.workingDir / artifactPath,
|
||||
artifactsDir / artifactName)
|
||||
except:
|
||||
raiseEx "step " & step.name & " failed: unable to copy artifact " &
|
||||
artifactPath & ":\n" & getCurrentExceptionMsg()
|
||||
|
||||
wksp.publishStatus(BuildState.stepComplete, "step " & step.name & " complete")
|
||||
result = wksp.status
|
||||
|
||||
proc run*(cfg: StrawBossConfig, req: RunRequest,
|
||||
outputHandler: HandleProcMsgCB = nil): BuildStatus =
|
||||
|
||||
## Execute a RunReuest given the StrawBoss configuration. This is the main
|
||||
## entrypoint to running a build step.
|
||||
|
||||
result = BuildStatus(
|
||||
runId: $req.runId,
|
||||
state: BuildState.setup,
|
||||
details: "initializing build workspace",
|
||||
version: "")
|
||||
outputHandler.sendStatusMsg(result)
|
||||
|
||||
var wksp: Workspace
|
||||
|
||||
try:
|
||||
# Find the project definition
|
||||
let projectDef = cfg.getProject(req.projectName)
|
||||
|
||||
# Make sure the build data directories for this project exist.
|
||||
ensureProjectDirsExist(cfg, projectDef)
|
||||
|
||||
# Update our run status
|
||||
let runDir = cfg.buildDataDir / projectDef.name / "runs"
|
||||
writeFile(runDir / $req.runId & ".status.json", $result)
|
||||
|
||||
# Read in the existing system environment
|
||||
var env = loadEnv()
|
||||
env["GIT_DIR"] = ".git"
|
||||
|
||||
# Make sure we have a workspace directory
|
||||
assert req.workspaceDir.isAbsolute
|
||||
if not existsDir(req.workspaceDir): createDir(req.workspaceDir)
|
||||
|
||||
# Setup our STDOUT and STDERR files
|
||||
let stdoutFile = open(runDir / $req.runId & ".stdout.log", fmWrite)
|
||||
let stderrFile = open(runDir / $req.runId & ".stderr.log", fmWrite)
|
||||
|
||||
let logFilesOH = makeProcMsgHandler(stdoutFile, stderrFile)
|
||||
|
||||
wksp = Workspace(
|
||||
buildDataDir: cfg.buildDataDir / projectDef.name,
|
||||
buildRef:
|
||||
if req.buildRef.len > 0: req.buildRef
|
||||
else: projectDef.defaultBranch,
|
||||
dir: req.workspaceDir,
|
||||
env: env,
|
||||
logLevel: cfg.logLevel,
|
||||
openedFiles: @[stdoutFile, stderrFile],
|
||||
outputHandler: combineProcMsgHandlers(outputHandler, logFilesOH),
|
||||
project: ProjectConfig(),
|
||||
projectDef: projectDef,
|
||||
runRequest: req,
|
||||
status: result,
|
||||
step: Step(),
|
||||
version: "")
|
||||
|
||||
except:
|
||||
when not defined(release): echo getCurrentException().getStackTrace()
|
||||
result = BuildStatus(runId: $req.runId, state: BuildState.failed,
|
||||
details: getCurrentExceptionMsg(), version: "")
|
||||
try: outputHandler.sendStatusMsg(result)
|
||||
except: discard ""
|
||||
return
|
||||
|
||||
try:
|
||||
# Clone the repo and setup the working environment
|
||||
wksp.publishStatus(BuildState.setup,
|
||||
"cloning project repo and preparing to run '" & req.stepName & "'")
|
||||
wksp.setupProject()
|
||||
|
||||
# Update our cache of project configurations.
|
||||
# TODO: what happens if this fails?
|
||||
copyFileWithPermissions(
|
||||
wksp.dir / wksp.projectDef.cfgFilePath,
|
||||
wksp.buildDataDir / "configurations" / wksp.version & ".json")
|
||||
|
||||
# Find the requested step
|
||||
if not wksp.project.steps.hasKey(req.stepName):
|
||||
raiseEx "no step name '" & req.stepName & "' for " & req.projectName
|
||||
var step = wksp.project.steps[req.stepName]
|
||||
|
||||
if req.forceRebuild: step.dontSkip = true
|
||||
|
||||
var buildStatus = doStep(wksp, step)
|
||||
if buildStatus.state == BuildState.stepComplete:
|
||||
buildStatus.state = BuildState.complete
|
||||
wksp.publishStatus(buildStatus.state, "all steps complete")
|
||||
|
||||
result = wksp.status
|
||||
|
||||
except:
|
||||
when not defined(release): echo getCurrentException().getStackTrace()
|
||||
let msg = getCurrentExceptionMsg()
|
||||
try:
|
||||
wksp.publishStatus(BuildState.failed, msg)
|
||||
result = wksp.status
|
||||
except:
|
||||
result = BuildStatus(runId: $req.runId, state: BuildState.failed,
|
||||
details: msg, version: "")
|
||||
try: outputHandler.sendStatusMsg(result)
|
||||
except: discard ""
|
||||
|
||||
finally:
|
||||
if wksp != nil:
|
||||
# Close open files
|
||||
for f in wksp.openedFiles:
|
||||
try: close(f)
|
||||
except: discard ""
|
||||
|
||||
proc spawnWorker*(cfg: StrawBossConfig, req: RunRequest):
|
||||
tuple[status: BuildStatus, worker: Worker] =
|
||||
|
||||
# Find the project definition (will throw appropriate exceptions)
|
||||
let projectDef = cfg.getProject(req.projectName)
|
||||
let runDir = cfg.buildDataDir / projectDef.name / "runs"
|
||||
let reqFile = runDir / $req.runId & ".request.json"
|
||||
let statusFile = runDir / $req.runId & ".status.json"
|
||||
|
||||
try:
|
||||
# Make sure the build data directories for this project exist.
|
||||
ensureProjectDirsExist(cfg, projectDef)
|
||||
|
||||
# Save the run request
|
||||
writeFile(reqFile, $req)
|
||||
|
||||
# Write the initial build status (queued).
|
||||
let queuedStatus = BuildStatus(
|
||||
runId: $req.runId,
|
||||
state: BuildState.queued,
|
||||
details: "request queued for execution",
|
||||
version: "")
|
||||
writeFile(statusFile, $queuedStatus)
|
||||
|
||||
var args = @["run", reqFile, "-c", cfg.filePath]
|
||||
debug "Launching worker: " & cfg.pathToExe & " " & args.join(" ")
|
||||
|
||||
let worker = Worker(
|
||||
runId: req.runId,
|
||||
projectName: projectDef.name,
|
||||
process: startProcess(cfg.pathToExe, ".", args, loadEnv(), {poUsePath}))
|
||||
|
||||
result = (queuedStatus, worker)
|
||||
|
||||
except:
|
||||
let exMsg = "run request rejected: " & getCurrentExceptionMsg()
|
||||
try:
|
||||
writeFile(statusFile,
|
||||
$(BuildStatus(runId: $req.runId, state: BuildState.rejected,
|
||||
details: exMsg, version: "")))
|
||||
except: discard ""
|
||||
raiseEx exMsg
|
||||
@ -1,469 +0,0 @@
|
||||
import asyncdispatch, bcrypt, cliutils, jester, json, jwt, logging, md5,
|
||||
options, os, osproc, sequtils, strutils, tempfile, times, unittest, uuids
|
||||
|
||||
from mimetypes import getMimeType
|
||||
from asyncfile import openAsync, readToStream, close
|
||||
from asyncnet import send
|
||||
from re import re, find
|
||||
from timeutils import trimNanoSec
|
||||
|
||||
import ./configuration, ./core, ./version
|
||||
|
||||
type
|
||||
Session = object
|
||||
user*: UserRef
|
||||
issuedAt*, expires*: Time
|
||||
|
||||
#const ISO_TIME_FORMAT = "yyyy-MM-dd'T'HH:mm:ss"
|
||||
const JSON = "application/json"
|
||||
|
||||
proc newSession*(user: UserRef): Session =
|
||||
result = Session(
|
||||
user: user,
|
||||
issuedAt: getTime().local.trimNanoSec.toTime,
|
||||
expires: daysForward(7).trimNanoSec.toTime)
|
||||
|
||||
template halt(code: HttpCode,
|
||||
headers: RawHeaders,
|
||||
content: string): typed =
|
||||
## Immediately replies with the specified request. This means any further
|
||||
## code will not be executed after calling this template in the current
|
||||
## route.
|
||||
bind TCActionSend, newHttpHeaders
|
||||
result[0] = CallbackAction.TCActionSend
|
||||
result[1] = code
|
||||
result[2] = some(headers)
|
||||
result[3] = content
|
||||
result.matched = true
|
||||
break allRoutes
|
||||
|
||||
template jsonResp(code: HttpCode, details: string = "", headers: RawHeaders = @{:} ) =
|
||||
halt(
|
||||
code,
|
||||
headers & @{"Content-Type": JSON},
|
||||
$(%* {
|
||||
"statusCode": code.int,
|
||||
"status": $code,
|
||||
"details": details
|
||||
})
|
||||
)
|
||||
|
||||
template json500Resp(ex: ref Exception, details: string = ""): void =
|
||||
when not defined(release): debug ex.getStackTrace()
|
||||
error details & ":\n" & ex.msg
|
||||
jsonResp(Http500)
|
||||
|
||||
proc toJWT*(cfg: StrawBossConfig, session: Session): string =
|
||||
## Make a JST token for this session.
|
||||
var jwt = JWT(
|
||||
header: JOSEHeader(alg: HS256, typ: "jwt"),
|
||||
claims: toClaims(%*{
|
||||
"sub": session.user.name,
|
||||
"iat": session.issuedAt.toUnix.int,
|
||||
"exp": session.expires.toUnix.int }))
|
||||
|
||||
jwt.sign(cfg.authSecret)
|
||||
result = $jwt
|
||||
|
||||
proc fromJWT*(cfg: StrawBossConfig, strTok: string): Session =
|
||||
## Validate a given JWT and extract the session data.
|
||||
let jwt = toJWT(strTok)
|
||||
var secret = cfg.authSecret
|
||||
if not jwt.verify(secret): raiseEx "Unable to verify auth token."
|
||||
jwt.verifyTimeClaims()
|
||||
|
||||
# Find the user record (if authenticated)
|
||||
let username = jwt.claims["sub"].node.str
|
||||
let users = cfg.users.filterIt(it.name == username)
|
||||
if users.len != 1: raiseEx "Could not find session user."
|
||||
|
||||
result = Session(
|
||||
user: users[0],
|
||||
issuedAt: fromUnix(jwt.claims["iat"].node.num),
|
||||
expires: fromUnix(jwt.claims["exp"].node.num))
|
||||
|
||||
proc extractSession(cfg: StrawBossConfig, request: Request): Session =
|
||||
## Helper to extract a session from a reqest.
|
||||
|
||||
# Find the auth header
|
||||
if not request.headers.hasKey("Authorization"):
|
||||
raiseEx "No auth token."
|
||||
|
||||
# Read and verify the JWT token
|
||||
let headerVal = request.headers["Authorization"]
|
||||
if not headerVal.startsWith("Bearer "):
|
||||
raiseEx "Invalid Authentication type (only 'Bearer' is supported)."
|
||||
|
||||
result = fromJWT(cfg, headerVal[7..^1])
|
||||
|
||||
proc hashPwd*(pwd: string, cost: int8): string =
|
||||
let salt = genSalt(cost)
|
||||
result = hash(pwd, salt)
|
||||
|
||||
proc validatePwd*(u: UserRef, givenPwd: string): bool =
|
||||
let salt = u.hashedPwd[0..28] # TODO: magic numbers
|
||||
result = compare(u.hashedPwd, hash(givenPwd, salt))
|
||||
|
||||
proc makeAuthToken*(cfg: StrawBossConfig, uname, pwd: string): string =
|
||||
## Given a username and pwd, validate the combination and generate a JWT
|
||||
## token string.
|
||||
|
||||
if uname.len == 0 or pwd.len == 0:
|
||||
raiseEx "fields 'username' and 'password' required"
|
||||
|
||||
# find the user record
|
||||
let users = cfg.users.filterIt(it.name == uname)
|
||||
if users.len != 1: raiseEx "invalid username or password"
|
||||
|
||||
let user = users[0]
|
||||
|
||||
if not validatePwd(user, pwd): raiseEx "invalid username or password"
|
||||
|
||||
let session = newSession(user)
|
||||
|
||||
result = toJWT(cfg, session)
|
||||
|
||||
proc makeApiKey*(cfg: StrawBossConfig, uname: string): string =
|
||||
## Given a username, make an API token (JWT token string that does not
|
||||
## expire). Note that this does not validate the username/pwd combination. It
|
||||
## is not intended to be exposed publicly via the API, but serve as a utility
|
||||
## function for an administrator to setup a unsupervised account (git access
|
||||
## for example).
|
||||
|
||||
if uname.len == 0: raiseEx "no username given"
|
||||
|
||||
# find the user record
|
||||
let users = cfg.users.filterIt(it.name == uname)
|
||||
if users.len != 1: raiseEx "invalid username"
|
||||
|
||||
let session = Session(
|
||||
user: users[0],
|
||||
issuedAt: getTime(),
|
||||
expires: daysForward(365 * 1000).toTime())
|
||||
|
||||
result = toJWT(cfg, session);
|
||||
|
||||
template checkAuth() =
|
||||
## Check this request for authentication and authorization information.
|
||||
## Injects the session into the running context. If the request is not
|
||||
## authorized, this template returns an appropriate 401 response.
|
||||
|
||||
var session {.inject.}: Session
|
||||
|
||||
try: session = extractSession(cfg, request)
|
||||
except:
|
||||
debug "Auth failed: " & getCurrentExceptionMsg()
|
||||
jsonResp(Http401, "Unauthorized", @{"WWW-Authenticate": "Bearer"})
|
||||
|
||||
proc start*(cfg: StrawBossConfig): void =
|
||||
|
||||
var stopFuture = newFuture[void]()
|
||||
var workers: seq[Worker] = @[]
|
||||
|
||||
settings:
|
||||
port = Port(cfg.port)
|
||||
appName = "/api"
|
||||
|
||||
routes:
|
||||
|
||||
get "/version":
|
||||
resp($(%("strawboss v" & SB_VERSION)), JSON)
|
||||
|
||||
post "/auth-token":
|
||||
var uname, pwd: string
|
||||
try:
|
||||
let jsonBody = parseJson(request.body)
|
||||
uname = jsonBody["username"].getStr
|
||||
pwd = jsonBody["password"].getStr
|
||||
except: jsonResp(Http400)
|
||||
|
||||
try:
|
||||
let authToken = makeAuthToken(cfg, uname, pwd)
|
||||
resp($(%authToken), JSON)
|
||||
except:
|
||||
jsonResp(Http401, getCurrentExceptionMsg())
|
||||
if ctx.cfg.debug: echo getStackTrace()
|
||||
|
||||
get "/verify-auth":
|
||||
checkAuth()
|
||||
|
||||
resp(Http200, $(%*{ "username": session.user.name }), JSON)
|
||||
|
||||
get "/projects":
|
||||
## List project summaries (ProjectDefs only)
|
||||
|
||||
checkAuth()
|
||||
|
||||
resp($(%cfg.projects), JSON)
|
||||
|
||||
post "/projects":
|
||||
## Create a new project definition
|
||||
|
||||
checkAuth()
|
||||
|
||||
# TODO
|
||||
jsonResp(Http501)
|
||||
|
||||
get "/project/@projectName":
|
||||
## Return a project's configuration, as well as it's versions.
|
||||
|
||||
checkAuth()
|
||||
|
||||
# Make sure we know about that project
|
||||
var projDef: ProjectDef
|
||||
try: projDef = cfg.getProject(@"projectName")
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
let msg = "unable to load project definition for project " & @"projectName"
|
||||
json500Resp(getCurrentException(), msg)
|
||||
|
||||
var projConf: ProjectConfig
|
||||
try: projConf = getProjectConfig(cfg, @"projectName", "")
|
||||
except: discard ""
|
||||
|
||||
let respJson = newJObject()
|
||||
respJson["definition"] = %projDef
|
||||
respJson["versions"] = %listVersions(cfg, @"projectName")
|
||||
if projConf.name.len > 0:
|
||||
respJson["latestConfig"] = %projConf
|
||||
|
||||
resp(pretty(respJson), JSON)
|
||||
|
||||
get "/project/@projectName/versions":
|
||||
## Get a list of all versions that we have built
|
||||
|
||||
checkAuth()
|
||||
|
||||
try: resp($(%listVersions(cfg, @"projectName")), JSON)
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
let msg = "unable to list versions for project " & @"projectName"
|
||||
json500Resp(getCurrentException(), msg)
|
||||
|
||||
get "/project/@projectName/version/@version?":
|
||||
## Get a detailed project record including step definitions (ProjectConfig).
|
||||
|
||||
checkAuth()
|
||||
|
||||
# Make sure we know about that project
|
||||
try: resp($(%getProjectConfig(cfg, @"projectName", @"version")), JSON)
|
||||
except: jsonResp(Http404, getCurrentExceptionMsg())
|
||||
|
||||
get "/project/@projectName/runs":
|
||||
## List all runs
|
||||
|
||||
checkAuth()
|
||||
|
||||
try: resp($(%listRuns(cfg, @"projectName")), JSON)
|
||||
except: jsonResp(Http404, getCurrentExceptionMsg())
|
||||
|
||||
get "/project/@projectName/runs/active":
|
||||
## List all currently active runs
|
||||
|
||||
checkAuth()
|
||||
|
||||
var details = ""
|
||||
try:
|
||||
let activeRuns = workers
|
||||
.filterIt(it.process.running and it.projectName == @"projectName")
|
||||
.mapIt(cfg.getRun(@"projectName", $it.runId));
|
||||
resp($(%activeRuns), JSON)
|
||||
except NotFoundException:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
json500Resp(getCurrentException(), "problem loading active runs")
|
||||
|
||||
get "/project/@projectName/run/@runId":
|
||||
## Details for a specific run
|
||||
|
||||
checkAuth()
|
||||
|
||||
# Make sure we know about that project
|
||||
try: discard cfg.getProject(@"projectName")
|
||||
except: jsonResp(Http404, getCurrentExceptionMsg())
|
||||
|
||||
if not existsRun(cfg, @"projectName", @"runId"):
|
||||
jsonResp(Http404, "no such run for project")
|
||||
|
||||
try: resp($getRun(cfg, @"projectName", @"runId"), JSON)
|
||||
except:
|
||||
json500Resp(getCurrentException(),
|
||||
"unable to load run details for project " & @"projectName" &
|
||||
" run " & @"runId")
|
||||
|
||||
get "/project/@projectName/run/@runId/logs":
|
||||
## Get logs from a specific run
|
||||
|
||||
checkAuth()
|
||||
|
||||
try: discard cfg.getProject(@"projectName")
|
||||
except:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
|
||||
if not existsRun(cfg, @"projectName", @"runId"):
|
||||
jsonResp(Http404, "no such run for project")
|
||||
|
||||
try: resp($getLogs(cfg, @"projectName", @"runId"))
|
||||
except:
|
||||
json500Resp(getCurrentException(),
|
||||
"unable to load run logs for " & @"projectName" & " run " & @"runId")
|
||||
|
||||
get "/project/@projectName/step/@stepName/artifacts/@version":
|
||||
## Get the list of artifacts that were built for
|
||||
|
||||
checkAuth()
|
||||
|
||||
debug "Matched artifacts list request: " & $(%*{
|
||||
"project": @"projectName",
|
||||
"step": @"stepName",
|
||||
"version": @"version"
|
||||
})
|
||||
|
||||
try: resp($(%listArtifacts(cfg, @"projectName", @"stepName", @"version")), JSON)
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
json500Resp(getCurrentException(), "unable to list artifacts for " &
|
||||
@"projectName" & ":" & @"stepName" & "@" & @"buildRef")
|
||||
|
||||
get "/project/@projectName/step/@stepName/artifact/@version/@artifactName":
|
||||
## Get a specific artifact that was built.
|
||||
|
||||
checkAuth()
|
||||
|
||||
var artifactPath: string
|
||||
try: artifactPath = getArtifactPath(cfg,
|
||||
@"projectName", @"stepName", @"version", @"artifactName")
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException:
|
||||
jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
json500Resp(getCurrentException(), "unable to check artifact path for " &
|
||||
@"projectName" & ":" & @"stepName" & "@" & @"version")
|
||||
|
||||
enableRawMode
|
||||
|
||||
debug "Preparing: " & artifactPath
|
||||
let fileSize = getFileSize(artifactPath)
|
||||
let mimetype = request.settings.mimes.getMimetype(artifactPath.splitFile.ext[1 .. ^1])
|
||||
if fileSize < 10_000_000: # 10 mb
|
||||
var file = readFile(artifactPath)
|
||||
|
||||
var hashed = getMD5(file)
|
||||
|
||||
# If the user has a cached version of this file and it matches our
|
||||
# version, let them use it
|
||||
if request.headers.hasKey("If-None-Match") and request.headers["If-None-Match"] == hashed:
|
||||
resp(Http304)
|
||||
else:
|
||||
resp(Http200, [
|
||||
("Content-Disposition", "; filename=\"" & @"artifactName" & "\""),
|
||||
("Content-Type", mimetype),
|
||||
("ETag", hashed )], file)
|
||||
else:
|
||||
let headers = @{
|
||||
"Content-Disposition": "; filename=\"" & @"artifactName" & "\"",
|
||||
"Content-Type": mimetype,
|
||||
"Content-Length": $fileSize
|
||||
}
|
||||
request.sendHeaders(Http200, headers)
|
||||
|
||||
var fileStream = newFutureStream[string]("sendStaticIfExists")
|
||||
var file = openAsync(artifactPath, fmRead)
|
||||
# Let `readToStream` write file data into fileStream in the
|
||||
# background.
|
||||
asyncCheck file.readToStream(fileStream)
|
||||
# The `writeFromStream` proc will complete once all the data in the
|
||||
# `bodyStream` has been written to the file.
|
||||
while true:
|
||||
let (hasValue, value) = await fileStream.read()
|
||||
if hasValue: request.send(value)
|
||||
else: break
|
||||
file.close()
|
||||
|
||||
get "/project/@projectName/step/@stepName/status/@buildRef":
|
||||
## Get detailed information about the status of a step (assuming it has been built)
|
||||
|
||||
checkAuth()
|
||||
|
||||
try: resp($cfg.getBuildStatus(@"projectName", @"stepName", @"buildRef"), JSON)
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException: jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except:
|
||||
json500Resp(getCurrentException(), "unable to load the build state for " &
|
||||
@"projectName" & ":" & @"stepName" & "@" & @"buildRef")
|
||||
|
||||
#get "/project/@projectName/step/@stepName/status/@buildRef.svg":
|
||||
## Get an image representing the status of a build
|
||||
|
||||
## TODO: how do we want to handle auth for this? Unlike
|
||||
#checkAuth(): if not authed: return true
|
||||
|
||||
post "/project/@projectName/step/@stepName/run/@buildRef?":
|
||||
# Kick off a run
|
||||
|
||||
checkAuth()
|
||||
|
||||
let runRequest = RunRequest(
|
||||
runId: genUUID(),
|
||||
projectName: @"projectName",
|
||||
stepName: @"stepName",
|
||||
buildRef: if @"buildRef" != "": @"buildRef" else: "",
|
||||
timestamp: getTime().local,
|
||||
forceRebuild: false) # TODO support this with optional query params
|
||||
|
||||
# TODO: instead of immediately spawning a worker, add the request to a
|
||||
# queue to be picked up by a worker. Allows capping the number of worker
|
||||
# prcesses, distributing, etc.
|
||||
try:
|
||||
let (status, worker) = spawnWorker(cfg, runRequest)
|
||||
workers.add(worker)
|
||||
|
||||
resp($Run(
|
||||
id: runRequest.runId,
|
||||
request: runRequest,
|
||||
status: status), JSON)
|
||||
except:
|
||||
try: raise getCurrentException()
|
||||
except NotFoundException: jsonResp(Http404, getCurrentExceptionMsg())
|
||||
except: jsonResp(Http400, getCurrentExceptionMsg())
|
||||
|
||||
post "/service/debug/stop":
|
||||
if not cfg.debug: jsonResp(Http404)
|
||||
else:
|
||||
let shutdownFut = sleepAsync(100)
|
||||
shutdownFut.callback = proc(): void = complete(stopFuture)
|
||||
resp($(%"shutting down"), JSON)
|
||||
|
||||
|
||||
get re".*":
|
||||
jsonResp(Http404, "URL [" & request.path & "] is not present on this server.")
|
||||
|
||||
post re".*":
|
||||
jsonResp(Http404)
|
||||
|
||||
proc performMaintenance(cfg: StrawBossConfig): void =
|
||||
# Prune workers
|
||||
workers = workers.filterIt(it.process.running())
|
||||
debug "Performing maintanance: " & $len(workers) & " active workers after pruning."
|
||||
|
||||
let fut = sleepAsync(cfg.maintenancePeriod)
|
||||
fut.callback =
|
||||
proc(): void =
|
||||
callSoon(proc(): void = performMaintenance(cfg))
|
||||
|
||||
|
||||
info "StrawBoss is bossing people around."
|
||||
|
||||
callSoon(proc(): void = performMaintenance(cfg))
|
||||
waitFor(stopFuture)
|
||||
@ -1,2 +0,0 @@
|
||||
const SB_VERSION* = "0.5.1"
|
||||
|
||||
@ -1,9 +0,0 @@
|
||||
[Unit]
|
||||
Description=StrawBoss build server.
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=strawboss
|
||||
WorkingDirectory=/home/strawboss
|
||||
ExecStart=/home/strawboss/strawboss
|
||||
Restart=on-failure
|
||||
@ -1,18 +0,0 @@
|
||||
{
|
||||
"name": "dummy-project",
|
||||
"versionCmd": "git describe --all --always",
|
||||
"containerImage": "ubuntu",
|
||||
"steps": {
|
||||
"build": {
|
||||
"containerImage": "alpine",
|
||||
"depends": ["test"],
|
||||
"workingDir": "dir1",
|
||||
"stepCmd": "cust-build",
|
||||
"artifacts": ["bin1", "doc1"],
|
||||
"expectedEnv": ["VAR1"],
|
||||
"dontSkip": true,
|
||||
"cmdInput": ["test", "this"]
|
||||
},
|
||||
"test": { }
|
||||
}
|
||||
}
|
||||
@ -1,20 +0,0 @@
|
||||
{
|
||||
"artifactsRepo": "artifacts",
|
||||
"authSecret": "change me",
|
||||
"debug": true,
|
||||
"users": [
|
||||
{ "name": "bob@builder.com", "hashedPwd": "$2a$11$lVZ9U4optQMhzPh0E9A7Yu6XndXblUF3gCa.zmEvJy4F.4C4718b." },
|
||||
{ "name": "sam@sousa.com", "hashedPwd": "testvalue" }
|
||||
],
|
||||
"port": 8180,
|
||||
"pwdCost": 11,
|
||||
"projects": [
|
||||
{ "name": "dummy-project",
|
||||
"repo": "/non-existent/dir",
|
||||
"cfgFilePath": "strawhat.json",
|
||||
"defaultBranch": "deploy",
|
||||
"envVars": { "VAR1": "value" }
|
||||
},
|
||||
{ "name": "test-project",
|
||||
"repo": "" } ]
|
||||
}
|
||||
@ -1,5 +0,0 @@
|
||||
{
|
||||
"runId": "90843e0c-6113-4462-af33-a89ff9731031",
|
||||
"state": "failed",
|
||||
"details": "some very good reason"
|
||||
}
|
||||
@ -1,19 +0,0 @@
|
||||
import tempfile, times, unittest, untar
|
||||
|
||||
from langutils import sameContents
|
||||
|
||||
import ../testutil
|
||||
import ../../../main/nim/strawbosspkg/configuration
|
||||
|
||||
let cfgFilePath = "src/test/json/strawboss.config.json"
|
||||
let cfg = loadStrawBossConfig(cfgFilePath)
|
||||
let TIMEOUT = 2.minutes
|
||||
|
||||
suite "strawboss core":
|
||||
|
||||
# Suite setup: extract test project
|
||||
let testProjTempDir = mkdtemp()
|
||||
let testProjTarFile = newTarFile("src/test/test-project.tar.gz")
|
||||
let testProjName = "test-project"
|
||||
testProjTarFile.extract(testProjTempDir)
|
||||
|
||||
@ -1,250 +0,0 @@
|
||||
import cliutils, httpclient, json, os, osproc, sequtils, strutils, tempfile,
|
||||
times, unittest, untar, uuids
|
||||
|
||||
from langutils import sameContents
|
||||
from algorithm import sorted
|
||||
|
||||
import ../testutil
|
||||
import ../../../main/nim/strawbosspkg/configuration
|
||||
import ../../../main/nim/strawbosspkg/core
|
||||
|
||||
let apiBase = "http://localhost:8180/api"
|
||||
let cfgFilePath = "src/test/json/strawboss.config.json"
|
||||
let cfg = loadStrawBossConfig(cfgFilePath)
|
||||
let TIMEOUT = 2.minutes
|
||||
|
||||
# Util template intended for use to manually review test case.
|
||||
# Inserting into a test case will prevent the test case from cleaning up it's
|
||||
# working files and echo the command to start StrawBoss using that test's
|
||||
# configuration and working files.
|
||||
template keepEnv(): untyped =
|
||||
preserveEnv = true
|
||||
echo "artifacts dir: " & tempBuildDataDir
|
||||
echo "strawboss serve -c " & tempCfgPath
|
||||
|
||||
suite "strawboss server":
|
||||
|
||||
# Suite setup: extract test project
|
||||
let testProjTempDir = mkdtemp()
|
||||
let testProjTarFile = newTarFile("src/test/test-project.tar.gz")
|
||||
let testProjName = "test-project"
|
||||
testProjTarFile.extract(testProjTempDir)
|
||||
|
||||
# per-test setup: spin up a fresh strawboss instance
|
||||
setup:
|
||||
let tempBuildDataDir = mkdtemp()
|
||||
let (_, tempCfgPath) = mkstemp()
|
||||
var preserveEnv = false
|
||||
|
||||
# copy our test config
|
||||
var newCfg = cfg
|
||||
newCfg.buildDataDir = tempBuildDataDir
|
||||
|
||||
# update the repo string for the extracted test project
|
||||
var testProjDef = newCfg.getProject(testProjName)
|
||||
testProjDef.repo = testProjTempDir
|
||||
newCfg.setProject(testProjName, testProjDef)
|
||||
|
||||
# save the updated config and start the strawboss instance using it
|
||||
writeFile(tempCfgPath, $newCfg)
|
||||
let serverProcess = startProcess("./strawboss", ".",
|
||||
@["serve", "-c", tempCfgPath], loadEnv(), {poUsePath})
|
||||
|
||||
# give the server time to spin up
|
||||
sleep(200)
|
||||
|
||||
teardown:
|
||||
discard newAsyncHttpClient().post(apiBase & "/service/debug/stop")
|
||||
|
||||
if not preserveEnv:
|
||||
removeDir(tempBuildDataDir)
|
||||
removeFile(tempCfgPath)
|
||||
|
||||
# give the server time to spin down but kill it after that
|
||||
sleep(200)
|
||||
if serverProcess.running: kill(serverProcess)
|
||||
|
||||
test "handle missing project configuration":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
let resp = http.get(apiBase & "/projects/" & cfg.projects[0].name)
|
||||
check resp.status.startsWith("404")
|
||||
|
||||
test "gives 404 when no versions built":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
let resp = http.get(apiBase & "/projects/" & testProjName & "/versions")
|
||||
check resp.status.startsWith("404")
|
||||
|
||||
test "GET /api/project/@projectName/versions":
|
||||
let cachedConfsDir = tempBuildDataDir & "/" & testProjName & "/configurations"
|
||||
let expectedVersions = @["alpha", "beta", "1.0.0", "1.0.1"]
|
||||
|
||||
# Touch configuration files
|
||||
createDir(cachedConfsDir)
|
||||
for v in expectedVersions:
|
||||
var f: File
|
||||
check open(f, cachedConfsDir & "/" & v & ".json", fmWrite)
|
||||
close(f)
|
||||
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
let resp = http.get(apiBase & "/project/" & testProjName & "/versions")
|
||||
let returnedVersions = parseJson(resp.body).getElems.mapIt(it.getStr)
|
||||
check sameContents(expectedVersions, returnedVersions)
|
||||
|
||||
test "run a successful build with artifacts":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
let resp = http.post(apiBase & "/project/" & testProjName & "/step/build/run/0.2.1")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
# Check that the run was queued
|
||||
let queuedRun = parseRun(parseJson(resp.body))
|
||||
check queuedRun.status.state == BuildState.queued
|
||||
|
||||
# Wait for the build to complete
|
||||
let completedRun = http.waitForBuild(apiBase, testProjname, $queuedRun.id)
|
||||
|
||||
# check that the run directory, run request, status, and output logs exist
|
||||
let runsDir = tempBuildDataDir & "/" & testProjName & "/runs"
|
||||
let runId = $completedRun.id
|
||||
check existsDir(runsDir)
|
||||
for suffix in [".request.json", ".status.json", ".stdout.log", ".stderr.log"]:
|
||||
check existsFile(runsDir & "/" & runId & suffix)
|
||||
|
||||
# check that the project directory has been created in the artifacts repo
|
||||
let runArtifactsDir = tempBuildDataDir & "/" & testProjName & "/artifacts/build/0.2.1"
|
||||
check existsDir(runArtifactsDir)
|
||||
|
||||
# check that the build step status file has been created
|
||||
let statusFile = tempBuildDataDir & "/" & testProjName & "/status/build/0.2.1.json"
|
||||
check fileExists(statusFile)
|
||||
|
||||
# check that the status is complete
|
||||
var status = loadBuildStatus(statusFile)
|
||||
check status.state == BuildState.complete
|
||||
|
||||
# check that the artifacts we expect are present
|
||||
let binFile = runArtifactsDir & "/test_project"
|
||||
check existsFile(binFile)
|
||||
|
||||
test "run a multi-step build":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
|
||||
# Run the "test" step (depends on "build")
|
||||
var resp = http.post(apiBase & "/project/" & testProjname & "/step/test/run/0.2.1")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
let queuedRun = parseRun(parseJson(resp.body))
|
||||
let completedRun = http.waitForBuild(apiBase, testProjName, $queuedRun.id)
|
||||
|
||||
# there should be successful status files for both the build and test steps
|
||||
for step in [("build", BuildState.stepComplete), ("test", BuildState.complete)]:
|
||||
let statusFile = tempBuildDataDir & "/" & testProjName & "/status/" & step[0] & "/0.2.1.json"
|
||||
check fileExists(statusFile)
|
||||
|
||||
let status = loadBuildStatus(statusFile)
|
||||
check status.state == step[1]
|
||||
|
||||
test "run a build in docker":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
|
||||
# Run the "build-docker" step
|
||||
var resp = http.post(apiBase & "/project/" & testProjName & "/step/build-docker/run/0.3.0")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
let queuedRun = parseRun(parseJson(resp.body))
|
||||
check queuedRun.status.state == BuildState.queued
|
||||
|
||||
# Wait for the build to complete
|
||||
let completedRun = http.waitForBuild(apiBase, testProjName, $queuedRun.id)
|
||||
|
||||
# check that the run directory, run request, status, and output logs exist
|
||||
let runsDir = tempBuildDataDir & "/" & testProjName & "/runs"
|
||||
let runId = $completedRun.id
|
||||
check existsDir(runsDir)
|
||||
for suffix in [".request.json", ".status.json", ".stdout.log", ".stderr.log"]:
|
||||
check existsFile(runsDir & "/" & runId & suffix)
|
||||
|
||||
# check that the project directory has been created in the artifacts repo
|
||||
let runArtifactsDir = tempBuildDataDir & "/" & testProjName & "/artifacts/build-docker/0.3.0"
|
||||
check existsDir(runArtifactsDir)
|
||||
|
||||
# check that the build step status file has been created
|
||||
let statusFile = tempBuildDataDir & "/" & testProjName & "/status/build-docker/0.3.0.json"
|
||||
check fileExists(statusFile)
|
||||
|
||||
# check that the status is complete
|
||||
var status = loadBuildStatus(statusFile)
|
||||
check status.state == BuildState.complete
|
||||
|
||||
# check that the artifacts we expect are present
|
||||
let binFile = runArtifactsDir & "/test_project"
|
||||
check existsFile(binFile)
|
||||
|
||||
test "run a multi-step docker-based build":
|
||||
let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
|
||||
# Run the "test" step (depends on "build")
|
||||
var resp = http.post(apiBase & "/project/" & testProjname & "/step/test-docker/run/0.3.0")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
let queuedRun = parseRun(parseJson(resp.body))
|
||||
let completedRun = http.waitForBuild(apiBase, testProjName, $queuedRun.id)
|
||||
|
||||
# there should be successful status files for both the build and test steps
|
||||
for step in [("build-docker", BuildState.stepComplete), ("test-docker", BuildState.complete)]:
|
||||
let statusFile = tempBuildDataDir & "/" & testProjName & "/status/" & step[0] & "/0.3.0.json"
|
||||
check fileExists(statusFile)
|
||||
|
||||
let status = loadBuildStatus(statusFile)
|
||||
check status.state == step[1]
|
||||
|
||||
# TODO
|
||||
#test "already completed steps should not be rebuilt":
|
||||
# let http = newAuthenticatedHttpClient(apibase, "bob@builder.com", "password")
|
||||
# let runArtifactsDir = tempBuildDataDir & "/" & testProjName & "/artifacts/build/0.2.1"
|
||||
# let exeModTime = getLastModificationTime(runArtifactsDir & "/test_project")
|
||||
|
||||
# Run the "build" step
|
||||
# Kick off a build that depends on "build" (which was run in the last test)
|
||||
|
||||
test "kick off multiple runs and check the list of active runs via the API":
|
||||
let http = newAuthenticatedHttpClient(apiBase, "bob@builder.com", "password")
|
||||
|
||||
# Kick off multiple runs of the "long-running" job
|
||||
let queuedRuns = toSeq((1..3)).map(proc (idx: int): Run =
|
||||
let resp = http.post(apiBase & "/project/" & testProjName & "/step/long-running/run/0.3.1")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
return parseRun(parseJson(resp.body)))
|
||||
|
||||
# Collect run ids.
|
||||
let runIds = queuedRuns.mapIt($(it.id)).sorted(cmpIgnoreCase)
|
||||
|
||||
# Check on the runs
|
||||
let getActiveResp = http.get(apiBase & "/project/" & testProjName & "/runs/active")
|
||||
check getActiveResp.status.startsWith("200")
|
||||
|
||||
let activeRuns = parseJson(getActiveResp.body).getElems().mapIt(parseRun(it))
|
||||
let activeRunIds = activeRuns.mapIt($(it.id)).sorted(cmpIgnoreCase)
|
||||
|
||||
# Make sure we see all runs in the active state.
|
||||
check runIds == activeRunIds
|
||||
|
||||
let completedRuns = runIds.map(proc (runId: string): Run =
|
||||
return http.waitForBuild(apiBase, testProjName, runId))
|
||||
|
||||
# Make sure all are completed and all are accounted for
|
||||
check completedRuns.allIt(it.status.state == BuildState.complete)
|
||||
check completedRuns.mapIt($(it.id)).sorted(cmpIgnoreCase) == runIds;
|
||||
|
||||
# Check that there are no more active runs
|
||||
let getActiveResp2 = http.get(apiBase & "/project/" & testProjName & "/runs/active")
|
||||
let remainingActiveRuns = parseJson(getActiveResp2.body).getElems().mapIt(parseRun(it))
|
||||
check remainingActiveRuns.len == 0
|
||||
|
||||
# Last-chance catch to kill the server in case some test err'ed and didn't
|
||||
# reach it's teardown handler
|
||||
discard newAsyncHttpClient().post(apiBase & "/service/debug/stop")
|
||||
|
||||
# Also, delete the extracted test project "source" repo
|
||||
removeDir(testProjTempDir)
|
||||
|
||||
@ -1,3 +0,0 @@
|
||||
import unittest
|
||||
|
||||
import ./functional/tserver
|
||||
@ -1,4 +0,0 @@
|
||||
import unittest
|
||||
|
||||
import ./unit/tserver
|
||||
import ./unit/tconfiguration
|
||||
@ -1,46 +0,0 @@
|
||||
import httpclient, json, os, strutils, times
|
||||
|
||||
import ../../main/nim/strawbosspkg/core
|
||||
import ../../main/nim/strawbosspkg/configuration
|
||||
|
||||
proc newAuthenticatedHttpClient*(apiBase, uname, pwd: string): HttpClient =
|
||||
result = newHttpClient()
|
||||
let authResp = result.post(apiBase & "/auth-token", $(%*{"username": uname, "password": pwd}))
|
||||
assert authResp.status.startsWith("200")
|
||||
result.headers = newHttpHeaders({"Authorization": "Bearer " & parseJson(authResp.body).getStr})
|
||||
|
||||
proc waitForBuild*(client: HttpClient, apiBase, projectName, runId: string,
|
||||
expectedState = BuildState.complete,
|
||||
failedState = BuildState.failed,
|
||||
timeout = 10): Run =
|
||||
|
||||
let startTime = epochTime()
|
||||
var run: Run
|
||||
|
||||
#echo "Waiting for '" & $expectedState & "' from run:\n\t" &
|
||||
# apiBase & "/project/" & projectName & "/run/" & runId
|
||||
|
||||
while true:
|
||||
var curElapsed = epochTime() - startTime
|
||||
|
||||
#echo "Checking (" & $curElapsed & " has passed)."
|
||||
|
||||
if curElapsed > toFloat(timeout):
|
||||
raise newException(Exception, "Timeout exceeded waiting for build.")
|
||||
|
||||
let resp = client.get(apiBase & "/project/" & projectName & "/run/" & runId)
|
||||
|
||||
#echo "Received resp:\n\n" & $resp.status & "\n\n" & $resp.body
|
||||
|
||||
if not resp.status.startsWith("200"):
|
||||
raise newException(IOError, "Unable to retrieve status. Received response: " & resp.body)
|
||||
|
||||
run = parseRun(parseJson(resp.body))
|
||||
|
||||
if run.status.state == failedState:
|
||||
raise newException(IOError, "Run transitioned to failed state '" & $failedState & "'")
|
||||
|
||||
if run.status.state == expectedState:
|
||||
return run
|
||||
|
||||
sleep(200)
|
||||
@ -1,152 +0,0 @@
|
||||
import json, strtabs, times, tables, unittest, uuids
|
||||
|
||||
from langutils import sameContents
|
||||
from timeutils import trimNanoSec
|
||||
import ../../../main/nim/strawbosspkg/configuration
|
||||
|
||||
suite "load and save configuration objects":
|
||||
|
||||
# suite setup & common data
|
||||
let testProjDefStr = """{ "name": "dummy-project", "repo":
|
||||
"/non-existent/dir",
|
||||
"cfgFilePath": "strawhat.json",
|
||||
"defaultBranch": "deploy",
|
||||
"envVars": { "VAR1": "value" } }"""
|
||||
|
||||
let testProjDef = ProjectDef(
|
||||
name: "dummy-project",
|
||||
repo: "/non-existent/dir",
|
||||
cfgFilePath: "strawhat.json",
|
||||
defaultBranch: "deploy",
|
||||
envVars: newStringTable("VAR1", "value", modeCaseInsensitive))
|
||||
|
||||
test "parseRunRequest":
|
||||
let rr1 = RunRequest(
|
||||
runId: genUUID(),
|
||||
projectName: testProjDef.name,
|
||||
stepName: "build",
|
||||
buildRef: "master",
|
||||
workspaceDir: "/no-real/dir",
|
||||
timestamp: getTime().local.trimNanoSec,
|
||||
forceRebuild: true)
|
||||
|
||||
let rrStr = $rr1
|
||||
let rr2 = parseRunRequest(parseJson(rrStr))
|
||||
check rr1 == rr2
|
||||
|
||||
test "parseProjectDef":
|
||||
let pd = parseProjectDef(parseJson(testProjDefStr))
|
||||
|
||||
check:
|
||||
pd.name == "dummy-project"
|
||||
pd.repo == "/non-existent/dir"
|
||||
pd.cfgFilePath == "strawhat.json"
|
||||
pd.defaultBranch == "deploy"
|
||||
pd.envVars.len == 1
|
||||
pd.envVars.hasKey("VAR1")
|
||||
pd.envVars["VAR1"] == "value"
|
||||
|
||||
test "ProjectDef ==":
|
||||
let pd1 = parseProjectDef(parseJson(testProjDefStr))
|
||||
|
||||
check pd1 == testProjDef
|
||||
|
||||
test "ProjectDef != (name)":
|
||||
var pd1 = testProjDef
|
||||
pd1.name = "different"
|
||||
check pd1 != testProjDef
|
||||
|
||||
test "ProjectDef != (repo)":
|
||||
var pd1 = testProjDef
|
||||
pd1.repo = "different"
|
||||
check pd1 != testProjDef
|
||||
|
||||
test "ProjectDef != (cfgFilePath)":
|
||||
var pd1 = testProjDef
|
||||
pd1.cfgFilePath = "different"
|
||||
check pd1 != testProjDef
|
||||
|
||||
test "ProjectDef != (defaultBranch)":
|
||||
var pd1 = testProjDef
|
||||
pd1.defaultBranch = "different"
|
||||
check pd1 != testProjDef
|
||||
|
||||
test "loadStrawBossConfig":
|
||||
let cfg = loadStrawBossConfig("src/test/json/strawboss.config.json")
|
||||
let expectedUsers = @[UserRef(name: "bob@builder.com", hashedPwd: "testvalue"),
|
||||
UserRef(name: "sam@sousa.com", hashedPwd: "testvalue")]
|
||||
let expectedProjects = @[
|
||||
ProjectDef(name: "dummy-project",
|
||||
repo: "/non-existent/dir",
|
||||
defaultBranch: "deploy",
|
||||
cfgFilePath: "strawhat.json",
|
||||
envVars: newStringTable("VAR1", "value", modeCaseSensitive)),
|
||||
ProjectDef(name: "test-project",
|
||||
repo: "",
|
||||
defaultBranch: "master",
|
||||
cfgFilePath: "strawboss.json",
|
||||
envVars: newStringTable(modeCaseSensitive))]
|
||||
|
||||
check:
|
||||
cfg.buildDataDir == "build-data"
|
||||
cfg.authSecret == "change me"
|
||||
cfg.pwdCost == 11
|
||||
sameContents(expectedUsers, cfg.users)
|
||||
sameContents(expectedProjects, cfg.projects)
|
||||
|
||||
test "loadProjectConfig":
|
||||
let pc = loadProjectConfig("src/test/json/dummy-project.config.json")
|
||||
|
||||
check:
|
||||
pc.name == "dummy-project"
|
||||
pc.versionCmd == "git describe --all --always"
|
||||
pc.containerImage == "ubuntu"
|
||||
pc.steps.len == 2
|
||||
|
||||
# Explicitly set properties
|
||||
pc.steps["build"].name == "build"
|
||||
pc.steps["build"].dontSkip == true
|
||||
pc.steps["build"].stepCmd == "cust-build"
|
||||
pc.steps["build"].workingDir == "dir1"
|
||||
pc.steps["build"].containerImage == "alpine"
|
||||
sameContents(pc.steps["build"].artifacts, @["bin1", "doc1"])
|
||||
sameContents(pc.steps["build"].depends, @["test"])
|
||||
sameContents(pc.steps["build"].expectedEnv, @["VAR1"])
|
||||
sameContents(pc.steps["build"].cmdInput, @["test", "this"])
|
||||
|
||||
# Step with defaulted properties
|
||||
pc.steps["test"].name == "test"
|
||||
pc.steps["test"].dontSkip == false
|
||||
pc.steps["test"].stepCmd == "true"
|
||||
pc.steps["test"].workingDir == "."
|
||||
pc.steps["test"].containerImage.len == 0
|
||||
sameContents(pc.steps["test"].artifacts, @[])
|
||||
sameContents(pc.steps["test"].depends, @[])
|
||||
sameContents(pc.steps["test"].expectedEnv, @[])
|
||||
sameContents(pc.steps["test"].cmdInput, @[])
|
||||
|
||||
test "serialze StrawBossConfig to/from string":
|
||||
let cfg = loadStrawBossConfig("src/test/json/strawboss.config.json")
|
||||
let cfgStr = $cfg
|
||||
check cfg == parseStrawBossConfig(parseJson(cfgStr))
|
||||
|
||||
test "%step":
|
||||
let step = Step(
|
||||
name: "build", stepCmd: "true", workingDir: "dirA",
|
||||
artifacts: @[], depends: @["compile"], cmdInput: @[],
|
||||
expectedEnv: @["CWD", "TERM"], dontSkip: true)
|
||||
|
||||
let stepJS = %step
|
||||
|
||||
for k in @["name", "stepCmd", "workingDir", "artifacts", "cmdInput",
|
||||
"depends", "expectedEnv", "dontSkip"]:
|
||||
|
||||
check stepJS.hasKey(k)
|
||||
|
||||
test "loadBuildStatus":
|
||||
let st = loadBuildStatus("src/test/json/test-status.json")
|
||||
|
||||
check:
|
||||
st.runId == "90843e0c-6113-4462-af33-a89ff9731031"
|
||||
st.state == BuildState.failed
|
||||
st.details == "some very good reason"
|
||||
@ -1,94 +0,0 @@
|
||||
import asyncdispatch, cliutils, httpclient, json, os, osproc, sequtils,
|
||||
strutils, times, unittest
|
||||
|
||||
from langutils import sameContents
|
||||
|
||||
import ../testutil
|
||||
import ../../../main/nim/strawbosspkg/configuration
|
||||
import ../../../main/nim/strawbosspkg/server
|
||||
import ../../../main/nim/strawbosspkg/version
|
||||
|
||||
let apiBase = "http://localhost:8180/api"
|
||||
let cfgFilePath = "src/test/json/strawboss.config.json"
|
||||
let cfg = loadStrawBossConfig(cfgFilePath)
|
||||
|
||||
let testuser = UserRef( # note: needs to correspond to an actual user
|
||||
name: "bob@builder.com",
|
||||
hashedPwd: "$2a$11$lVZ9U4optQMhzPh0E9A7Yu6XndXblUF3gCa.zmEvJy4F.4C4718b.")
|
||||
|
||||
suite "strawboss server":
|
||||
|
||||
# suite setup code
|
||||
let serverProcess = startProcess("./strawboss", ".",
|
||||
@["serve", "-c", cfgFilePath], loadEnv(), {poUsePath})
|
||||
|
||||
let http = newHttpClient()
|
||||
|
||||
# give the server time to spin up
|
||||
sleep(100)
|
||||
|
||||
## UNIT TESTS
|
||||
|
||||
test "validate hashed pwd":
|
||||
|
||||
check validatePwd(testuser, "password")
|
||||
|
||||
test "detect invalid pwds":
|
||||
check(not validatePwd(testuser, "Password"))
|
||||
|
||||
test "make and extract a JWT token from a session":
|
||||
let session = newSession(testuser)
|
||||
let tok = toJWT(cfg, session)
|
||||
check fromJWT(cfg, tok) == session
|
||||
|
||||
test "version":
|
||||
let resp = http.get(apiBase & "/version")
|
||||
check:
|
||||
resp.status.startsWith("200")
|
||||
resp.body == "\"strawboss v" & SB_VERSION & "\""
|
||||
|
||||
test "fail auth":
|
||||
let resp = http.post(apiBase & "/auth-token",
|
||||
$(%*{"username": "bob@builder.com", "password": "notpassword"}))
|
||||
check resp.status.startsWith("401")
|
||||
|
||||
test "auth":
|
||||
let resp = http.post(apiBase & "/auth-token",
|
||||
$(%*{"username": "bob@builder.com", "password": "password"}))
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
test "verify valid auth token":
|
||||
let authHttp = newAuthenticatedHttpClient(apiBase, "bob@builder.com", "password")
|
||||
let resp = authHttp.get(apiBase & "/verify-auth")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
test "verify fails when no auth token is given":
|
||||
let resp = http.get(apiBase & "/verify-auth")
|
||||
check resp.status.startsWith("401")
|
||||
|
||||
test "verify fails when invalid auth token is given":
|
||||
let http1 = newHttpClient()
|
||||
http1.headers = newHttpHeaders({"Authorization": "Bearer nope"})
|
||||
let resp = http1.get(apiBase & "/verify-auth")
|
||||
check resp.status.startsWith("401")
|
||||
|
||||
test "fail to get projects when not authenticated":
|
||||
let resp = http.get(apiBase & "/projects")
|
||||
check resp.status.startsWith("401")
|
||||
|
||||
test "get projects":
|
||||
let authHttp = newAuthenticatedHttpClient(apiBase, "bob@builder.com", "password")
|
||||
let resp = authHttp.get(apiBase & "/projects")
|
||||
check resp.status.startsWith("200")
|
||||
|
||||
let projects: seq[ProjectDef] = parseJson(resp.body).getElems.mapIt(parseProjectDef(it))
|
||||
|
||||
check sameContents(projects, cfg.projects)
|
||||
|
||||
# suite tear-down
|
||||
|
||||
# give the server time to spin down but kill it after that
|
||||
discard newAsyncHttpClient().post(apiBase & "/service/debug/stop")
|
||||
sleep(100)
|
||||
if serverProcess.running: kill(serverProcess)
|
||||
|
||||
@ -1 +0,0 @@
|
||||
Subproject commit ab883bd9602a1373347a23c8bee4ed28dd475aec
|
||||
Binary file not shown.
@ -1,20 +0,0 @@
|
||||
#!/bin/bash
|
||||
|
||||
host="${STRAWBOSS_HOST:-localhost:8180}"
|
||||
if [ $# -eq 1 ]; then
|
||||
url="$1"
|
||||
method="GET"
|
||||
data=""
|
||||
elif [ $# -eq 2 ]; then
|
||||
method="$1"
|
||||
url="$2"
|
||||
data=""
|
||||
else
|
||||
method="$1"
|
||||
url="$2"
|
||||
data="$3"
|
||||
fi
|
||||
|
||||
curl -X "$method" -H "Authorization: Bearer $(cat token.txt)" "http://${host}/api/$url" -d "$data"
|
||||
echo ""
|
||||
#echo "curl -X \"$method\" -H \"Authorization: Bearer $(cat token.txt)\" \"localhost:8180/api/$url\" | jq . "
|
||||
@ -1,11 +1,7 @@
|
||||
{
|
||||
"buildDataDir": "build-data",
|
||||
"debug": true,
|
||||
"artifactsRepo": "artifacts",
|
||||
"users": [],
|
||||
"authSecret": "change me",
|
||||
"pwdCost": 11,
|
||||
"maintenancePeriod": 5000,
|
||||
"logLevel": "info",
|
||||
"tokens": [],
|
||||
"projects": [
|
||||
{ "name": "new-life-intro-band",
|
||||
"repo": "/home/jdb/projects/new-life-introductory-band" },
|
||||
|
||||
@ -1,7 +1,7 @@
|
||||
# Package
|
||||
|
||||
bin = @["strawboss"]
|
||||
version = "0.5.1"
|
||||
version = "0.2.0"
|
||||
author = "Jonathan Bernard"
|
||||
description = "My personal continious integration worker."
|
||||
license = "MIT"
|
||||
@ -9,43 +9,5 @@ srcDir = "src/main/nim"
|
||||
|
||||
# Dependencies
|
||||
|
||||
requires @["nim >= 0.19.0", "docopt >= 0.6.8", "isaac >= 0.1.3", "tempfile", "jester >= 0.4.1", "bcrypt",
|
||||
"untar", "uuids >= 0.1.10", "jwt"]
|
||||
requires @["nim >= 0.16.1", "docopt >= 0.1.0", "tempfile", "jester"]
|
||||
|
||||
# Hacky to point to a specific hash. But there is some bug building in the
|
||||
# docker image we use to build the project with the next version. It adds an
|
||||
# ifdef branch to support libssl 1.1 but for some reason that ifdef is set
|
||||
# wrong and it tries to build against the 1.1 API even though the image only
|
||||
# has the 1.0 API. I'm crossing my fingers and hoping that our base image
|
||||
# supports libssl 1.1 before I need to update this library.
|
||||
#requires "https://github.com/yglukhov/nim-jwt#549aa1eb13b8ddc0c6861d15cc2cc5b52bcbef01"
|
||||
|
||||
requires "https://git.jdb-labs.com/jdb/nim-lang-utils.git >= 0.4.0"
|
||||
requires "https://git.jdb-labs.com/jdb/nim-cli-utils.git >= 0.6.0"
|
||||
requires "https://git.jdb-labs.com/jdb/nim-time-utils.git >= 0.4.0"
|
||||
|
||||
# Tasks
|
||||
task functest, "Runs the functional test suite.":
|
||||
exec "nimble build"
|
||||
exec "nim c -r src/test/nim/run_functional_tests.nim"
|
||||
|
||||
task unittest, "Runs the unit test suite.":
|
||||
exec "nimble build"
|
||||
exec "nim c -r src/test/nim/run_unit_tests.nim"
|
||||
|
||||
task test, "Runs both the unit and functional test suites.":
|
||||
exec "nimble build"
|
||||
echo "Building test suites..."
|
||||
exec "nim c src/test/nim/run_unit_tests.nim"
|
||||
exec "nim c src/test/nim/run_functional_tests.nim"
|
||||
echo "\nRunning unit tests."
|
||||
echo "-------------------"
|
||||
exec "src/test/nim/run_unit_tests"
|
||||
echo "\nRunning functional tests."
|
||||
echo "-------------------------"
|
||||
exec "src/test/nim/run_functional_tests"
|
||||
|
||||
task dist, "Creates distributable package.":
|
||||
exec "nimble build"
|
||||
mkdir "dist"
|
||||
exec "cp strawboss strawboss.config.json example.json dist/."
|
||||
|
||||
@ -1,37 +0,0 @@
|
||||
{
|
||||
"name": "strawboss",
|
||||
"containerImage": "nimlang/nim:0.19.0",
|
||||
"steps": {
|
||||
"compile": {
|
||||
"artifacts": ["strawboss"],
|
||||
"stepCmd": "nimble build"
|
||||
},
|
||||
"unittest": {
|
||||
"depends": ["compile"],
|
||||
"stepCmd": "/bin/bash",
|
||||
"cmdInput": [
|
||||
"cp $compile_DIR/strawboss .",
|
||||
"nimble install --depsOnly",
|
||||
"nim c -r src/test/nim/run_unit_tests"
|
||||
]
|
||||
},
|
||||
"functest": {
|
||||
"depends": ["compile"],
|
||||
"stepCmd": "/bin/bash",
|
||||
"cmdInput": [
|
||||
"cp $compile_DIR/strawboss .",
|
||||
"nimble install --depsOnly",
|
||||
"nim c -r src/test/nim/run_functional_tests"
|
||||
]
|
||||
},
|
||||
"build": {
|
||||
"artifacts": ["strawboss-$VERSION.zip"],
|
||||
"depends": ["compile", "unittest", "functest"],
|
||||
"stepCmd": "/bin/bash",
|
||||
"cmdInput": [
|
||||
"cp $compile_DIR/strawboss .",
|
||||
"zip strawboss-$VERSION.zip strawboss strawboss.config.json example.json src/main/systemd/strawboss.service"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@ -1,11 +0,0 @@
|
||||
Run a build. Look for:
|
||||
- Run request archived
|
||||
- Output logs archived with the run request
|
||||
- Artifacts archived in the build-data directory.
|
||||
- Configuration for that version archived in configurations directory.
|
||||
- Status for that version archived in the status directory
|
||||
|
||||
Run the build again for the same project and build ref:
|
||||
- Build should be skipped.
|
||||
- Run request should be archived.
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user