jdb/fiber-orm-nim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: jdb:3.2.0
Branches Tags
jdb:main
jdb:4.2.0 jdb:4.1.0 jdb:4.0.0 jdb:3.2.0 jdb:3.1.1 jdb:3.1.0 jdb:3.0.0 jdb:2.2.0 jdb:2.1.1 jdb:2.1.0 jdb:2.0.0 jdb:1.0.4 jdb:1.0.3 jdb:1.0.2 jdb:1.0.1 jdb:1.0.0 jdb:0.3.6 jdb:0.3.5 jdb:0.3.4 jdb:0.3.3 jdb:0.3.2 jdb:0.3.1 jdb:0.3.0 jdb:0.2.0 jdb:0.1.2 jdb:0.1.1 jdb:0.1.0
...
compare: jdb:4.2.0
Branches Tags
jdb:main
jdb:4.2.0 jdb:4.1.0 jdb:4.0.0 jdb:3.2.0 jdb:3.1.1 jdb:3.1.0 jdb:3.0.0 jdb:2.2.0 jdb:2.1.1 jdb:2.1.0 jdb:2.0.0 jdb:1.0.4 jdb:1.0.3 jdb:1.0.2 jdb:1.0.1 jdb:1.0.0 jdb:0.3.6 jdb:0.3.5 jdb:0.3.4 jdb:0.3.3 jdb:0.3.2 jdb:0.3.1 jdb:0.3.0 jdb:0.2.0 jdb:0.1.2 jdb:0.1.1 jdb:0.1.0

6 Commits
3.2.0 ... 4.2.0

Author SHA1 Message Date
Jonathan Bernard
2301da8143 Add PostgreSQL FOR UPDATE getters 
Add a PostgreSQL-specific getRecordForUpdate helper that appends FOR
UPDATE to the generated SELECT statement so callers can lock a row
inside an explicit transaction.

generateProcsForModels now always emits a direct-connection
get<RecordName>ForUpdate proc that accepts db_postgres.DbConn. There is
intentionally no dbType overload for this API, because reacquiring a
connection via withConnection would defeat the lock's transactional
scope.

The source docs and README now document the new helper and show the
intended usage pattern inside inTransaction:

  db.inTransaction:
    var item = conn.getTodoItemForUpdate(todoId)
    item.priority += 1
    discard conn.updateTodoItem(item)
 2026-03-24 22:04:49 -05:00
Jonathan Bernard
71cb5a7cff Update documentation for new signature changes, bump version. 2026-03-24 21:48:51 -05:00
Jonathan Bernard
1a9314fe4f Add connection overloads for generated ORM procs 
The generated ORM helpers currently only accept the database wrapper
type as their first argument. That works well for the common case, but
it becomes misleading inside inTransaction blocks because the generated
proc will call withConnection again and may therefore use a different
connection than the one that is participating in the transaction.

Add DbConnType-constrained overloads for the generated model CRUD/query
procs, generated lookups, and generated join-table helpers. This lets
callers explicitly use the transaction connection while keeping the
existing dbType-based API intact for non-transactional call sites.

This makes the intended transactional usage straightforward:

  db.inTransaction:
    var userRecord = conn.getUser("userId1")
    userRecord.visitCount += 1
    discard conn.updateUser(userRecord)

AI-Assisted: yes
AI-Tool: OpenAI Codes / gpt-5.4 xhigh
 2026-03-24 21:39:25 -05:00
Jonathan Bernard
bb36bba864 Support distinct versions of types we know how to convert. 2025-09-02 00:40:00 -05:00
Jonathan Bernard
f54bf6e974 Add tryGet<RecordName> versions of get<Record> calls 
`tryGet<RecordName>`  returns Option types rather than raise exceptions.

For example:

    generateProcsForModels(MyDb, [ User ])

will now create both:

    proc getUser*(db: MyDb, id: string): User
    proc tryGetUser*(db: MyDb, id: string): Option[User]
 2025-09-02 00:36:11 -05:00
Jonathan Bernard
e1fa2480d0 Major update to provide thread-safe, robust connection pooling. 
Taking inspiration from the waterpark library, the connection pooling
mechanism has been refactored to be thread-safe. Additionally, the
pooling logic detects and handles stale connections in the pool. When a
connection is requested from the pool, the pool first validates that it
is healthy and replaces it with a fresh connection if necessary. This is
transparent to the requester.

Additionally we refactored the internal logging implementation to make
it more conventient to access logging infrastructure and log from
various sub-scopes within fiber_orm (query, pool, etc.)
 2025-07-27 17:47:07 -05:00
6 changed files with 491 additions and 198 deletions
Expand all files Collapse all files

84
README.rst
View File

@@ -83,7 +83,7 @@ Using Fiber ORM we can generate a data access layer with:
.. code-block:: Nim .. code-block:: Nim
# db.nim # db.nim
import std/[options] import std/[options]
import db_connectors/db_postgres import db_connector/db_postgres
import fiber_orm import fiber_orm
import ./models.nim import ./models.nim
@@ -100,32 +100,90 @@ Using Fiber ORM we can generate a data access layer with:
generateLookup(TodoDB, TimeEntry, @["todoItemId"]) generateLookup(TodoDB, TimeEntry, @["todoItemId"])
This will generate the following procedures: This will generate procedures like the following in two flavors:
* a `dbType` flavor that acquires a connection via `withConnection`
* a connection flavor that operates directly on an existing
`conn: D` where `D: DbConnType`
.. code-block:: Nim .. code-block:: Nim
proc getTodoItem*(db: TodoDB, id: UUID): TodoItem; proc getTodoItem*(db: TodoDB, id: UUID): TodoItem;
proc getTodoItem*[D: DbConnType](conn: D, id: UUID): TodoItem;
proc getTodoItemForUpdate*(conn: db_postgres.DbConn, id: UUID): TodoItem;
proc tryGetTodoItem*(db: TodoDB, id: UUID): Option[TodoItem];
proc tryGetTodoItem*[D: DbConnType](conn: D, id: UUID): Option[TodoItem];
proc getTodoItemIfItExists*(db: TodoDB, id: UUID): Option[TodoItem]; proc getTodoItemIfItExists*(db: TodoDB, id: UUID): Option[TodoItem];
proc getAllTodoItems*(db: TodoDB): seq[TodoItem]; proc getTodoItemIfItExists*[D: DbConnType](
conn: D, id: UUID): Option[TodoItem];
proc getAllTodoItems*(db: TodoDB,
pagination = none[PaginationParams]()): PagedRecords[TodoItem];
proc getAllTodoItems*[D: DbConnType](conn: D,
pagination = none[PaginationParams]()): PagedRecords[TodoItem];
proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem; proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
proc createTodoItem*[D: DbConnType](conn: D, rec: TodoItem): TodoItem;
proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool; proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool;
proc updateTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
proc createOrUpdateTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
proc createOrUpdateTodoItem*[D: DbConnType](
conn: D, rec: TodoItem): TodoItem;
proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool; proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool;
proc deleteTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
proc deleteTodoItem*(db: TodoDB, id: UUID): bool; proc deleteTodoItem*(db: TodoDB, id: UUID): bool;
proc deleteTodoItem*[D: DbConnType](conn: D, id: UUID): bool;
proc findTodoItemsWhere*(db: TodoDB, whereClause: string, proc findTodoItemsWhere*(db: TodoDB, whereClause: string,
values: varargs[string, dbFormat]): seq[TodoItem]; values: varargs[string, dbFormat],
pagination = none[PaginationParams]()): PagedRecords[TodoItem];
proc findTodoItemsWhere*[D: DbConnType](conn: D, whereClause: string,
values: varargs[string, dbFormat],
pagination = none[PaginationParams]()): PagedRecords[TodoItem];
proc getTimeEntry*(db: TodoDB, id: UUID): TimeEntry; proc getTimeEntry*(db: TodoDB, id: UUID): TimeEntry;
proc getTimeEntry*[D: DbConnType](conn: D, id: UUID): TimeEntry;
proc getTimeEntryIfItExists*(db: TodoDB, id: UUID): Option[TimeEntry]; proc getTimeEntryIfItExists*(db: TodoDB, id: UUID): Option[TimeEntry];
proc getAllTimeEntries*(db: TodoDB): seq[TimeEntry]; proc getTimeEntryIfItExists*[D: DbConnType](
conn: D, id: UUID): Option[TimeEntry];
proc getAllTimeEntries*(db: TodoDB,
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
proc getAllTimeEntries*[D: DbConnType](conn: D,
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
proc createTimeEntry*(db: TodoDB, rec: TimeEntry): TimeEntry; proc createTimeEntry*(db: TodoDB, rec: TimeEntry): TimeEntry;
proc createTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): TimeEntry;
proc updateTimeEntry*(db: TodoDB, rec: TimeEntry): bool; proc updateTimeEntry*(db: TodoDB, rec: TimeEntry): bool;
proc updateTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): bool;
proc deleteTimeEntry*(db: TodoDB, rec: TimeEntry): bool; proc deleteTimeEntry*(db: TodoDB, rec: TimeEntry): bool;
proc deleteTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): bool;
proc deleteTimeEntry*(db: TodoDB, id: UUID): bool; proc deleteTimeEntry*(db: TodoDB, id: UUID): bool;
proc deleteTimeEntry*[D: DbConnType](conn: D, id: UUID): bool;
proc findTimeEntriesWhere*(db: TodoDB, whereClause: string, proc findTimeEntriesWhere*(db: TodoDB, whereClause: string,
values: varargs[string, dbFormat]): seq[TimeEntry]; values: varargs[string, dbFormat],
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
proc findTimeEntriesWhere*[D: DbConnType](conn: D, whereClause: string,
values: varargs[string, dbFormat],
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
proc findTimeEntriesByTodoItemId(db: TodoDB, todoItemId: UUID): seq[TimeEntry]; proc findTimeEntriesByTodoItemId*(db: TodoDB, todoItemId: UUID,
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
proc findTimeEntriesByTodoItemId*[D: DbConnType](
conn: D, todoItemId: UUID,
pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
Use the `dbType` flavor when the caller does not already have a connection.
Use the connection flavor inside `withConnection` or `inTransaction`.
The generated `get<RecordName>ForUpdate` helper is PostgreSQL-specific and
is only available for direct PostgreSQL connections.
Warning: do not call the `dbType` flavor from inside `inTransaction`.
Those overloads call `withConnection` and may acquire a different
connection, causing the statements to execute outside the active
transaction.
.. code-block:: Nim
db.inTransaction:
var item = conn.getTodoItemForUpdate(todoId)
item.priority += 1
discard conn.updateTodoItem(item)
Object-Relational Modeling Object-Relational Modeling
========================== ==========================
@@ -133,11 +191,11 @@ Object-Relational Modeling
Model Class Model Class
----------- -----------
Fiber ORM uses simple Nim `object`s and `ref object`s as model classes. Fiber ORM uses simple Nim objects and ref objects as model classes.
Fiber ORM expects there to be one table for each model class. Fiber ORM expects there to be one table for each model class.
Name Mapping Name Mapping
```````````` ^^^^^^^^^^^^
Fiber ORM uses `snake_case` for database identifiers (column names, table Fiber ORM uses `snake_case` for database identifiers (column names, table
names, etc.) and `camelCase` for Nim identifiers. We automatically convert names, etc.) and `camelCase` for Nim identifiers. We automatically convert
model names to and from table names (`TodoItem` <-> `todo_items`), as well model names to and from table names (`TodoItem` <-> `todo_items`), as well
@@ -168,7 +226,7 @@ procedures in the `fiber_orm/util`_ module for details.
.. _util: fiber_orm/util.html .. _util: fiber_orm/util.html
ID Field ID Field
```````` ^^^^^^^^
Fiber ORM expects every model class to have a field named `id`, with a Fiber ORM expects every model class to have a field named `id`, with a
corresponding `id` column in the model table. This field must be either a corresponding `id` column in the model table. This field must be either a
@@ -257,8 +315,10 @@ Many of the Fiber ORM macros expect a database object type to be passed.
In the example above the `pool.DbConnPool`_ object is used as database In the example above the `pool.DbConnPool`_ object is used as database
object type (aliased as `TodoDB`). This is the intended usage pattern, but object type (aliased as `TodoDB`). This is the intended usage pattern, but
anything can be passed as the database object type so long as there is a anything can be passed as the database object type so long as there is a
defined `withConn` template that provides an injected `conn: DbConn` object defined `withConnection` template that provides an injected `conn: DbConn` object
to the provided statement body. to the provided statement body.
The generated connection-flavor procedures are intended to work directly
with that `conn` value.
For example, a valid database object implementation that opens a new For example, a valid database object implementation that opens a new
connection for every request might look like this: connection for every request might look like this:
@@ -269,7 +329,7 @@ connection for every request might look like this:
type TodoDB* = object type TodoDB* = object
connString: string connString: string
template withConn*(db: TodoDB, stmt: untyped): untyped = template withConnection*(db: TodoDB, stmt: untyped): untyped =
let conn {.inject.} = open("", "", "", db.connString) let conn {.inject.} = open("", "", "", db.connString)
try: stmt try: stmt
finally: close(conn) finally: close(conn)

2
fiber_orm.nimble
View File

@@ -1,6 +1,6 @@
# Package # Package
version = "3.2.0" version = "4.2.0"
author = "Jonathan Bernard" author = "Jonathan Bernard"
description = "Lightweight Postgres ORM for Nim." description = "Lightweight Postgres ORM for Nim."
license = "GPL-3.0" license = "GPL-3.0"

415
src/fiber_orm.nim
View File

@@ -100,39 +100,89 @@
## ##
## generateLookup(TodoDB, TimeEntry, @["todoItemId"]) ## generateLookup(TodoDB, TimeEntry, @["todoItemId"])
## ##
## This will generate the following procedures: ## This will generate procedures like the following in two flavors:
##
## * a `dbType` flavor that acquires a connection via `withConnection`
## * a connection flavor that operates directly on an existing
## `conn: D` where `D: DbConnType`
## ##
## .. code-block:: Nim ## .. code-block:: Nim
## proc getTodoItem*(db: TodoDB, id: UUID): TodoItem; ## proc getTodoItem*(db: TodoDB, id: UUID): TodoItem;
## ## proc getTodoItem*[D: DbConnType](conn: D, id: UUID): TodoItem;
## proc getTodoItemForUpdate*(conn: db_postgres.DbConn, id: UUID): TodoItem;
## proc tryGetTodoItem*(db: TodoDB, id: UUID): Option[TodoItem];
## proc tryGetTodoItem*[D: DbConnType](conn: D, id: UUID): Option[TodoItem];
## proc getTodoItemIfItExists*(db: TodoDB, id: UUID): Option[TodoItem];
## proc getTodoItemIfItExists*[D: DbConnType](
## conn: D, id: UUID): Option[TodoItem];
## proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem; ## proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
## proc createTodoItem*[D: DbConnType](conn: D, rec: TodoItem): TodoItem;
## proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool;
## proc createOrUpdateTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc updateTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
## proc createOrUpdateTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
## proc createOrUpdateTodoItem*[D: DbConnType](
## conn: D, rec: TodoItem): TodoItem;
## proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool;
## proc deleteTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
## proc deleteTodoItem*(db: TodoDB, id: UUID): bool; ## proc deleteTodoItem*(db: TodoDB, id: UUID): bool;
## proc deleteTodoItem*[D: DbConnType](conn: D, id: UUID): bool;
## ##
## proc getAllTodoItems*(db: TodoDB, ## proc getAllTodoItems*(db: TodoDB,
## pagination = none[PaginationParams]()): seq[TodoItem]; ## pagination = none[PaginationParams]()): PagedRecords[TodoItem];
## proc getAllTodoItems*[D: DbConnType](conn: D,
## pagination = none[PaginationParams]()): PagedRecords[TodoItem];
## ##
## proc findTodoItemsWhere*(db: TodoDB, whereClause: string, ## proc findTodoItemsWhere*(db: TodoDB, whereClause: string,
## values: varargs[string, dbFormat], pagination = none[PaginationParams]() ## values: varargs[string, dbFormat], pagination = none[PaginationParams]()
## ): seq[TodoItem]; ## ): PagedRecords[TodoItem];
## proc findTodoItemsWhere*[D: DbConnType](conn: D, whereClause: string,
## values: varargs[string, dbFormat], pagination = none[PaginationParams]()
## ): PagedRecords[TodoItem];
## ##
## proc getTimeEntry*(db: TodoDB, id: UUID): TimeEntry; ## proc getTimeEntry*(db: TodoDB, id: UUID): TimeEntry;
## proc getTimeEntry*[D: DbConnType](conn: D, id: UUID): TimeEntry;
## proc createTimeEntry*(db: TodoDB, rec: TimeEntry): TimeEntry; ## proc createTimeEntry*(db: TodoDB, rec: TimeEntry): TimeEntry;
## proc createTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): TimeEntry;
## proc updateTimeEntry*(db: TodoDB, rec: TimeEntry): bool; ## proc updateTimeEntry*(db: TodoDB, rec: TimeEntry): bool;
## proc updateTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): bool;
## proc deleteTimeEntry*(db: TodoDB, rec: TimeEntry): bool; ## proc deleteTimeEntry*(db: TodoDB, rec: TimeEntry): bool;
## proc deleteTimeEntry*[D: DbConnType](conn: D, rec: TimeEntry): bool;
## proc deleteTimeEntry*(db: TodoDB, id: UUID): bool; ## proc deleteTimeEntry*(db: TodoDB, id: UUID): bool;
## proc deleteTimeEntry*[D: DbConnType](conn: D, id: UUID): bool;
## ##
## proc getAllTimeEntries*(db: TodoDB, ## proc getAllTimeEntries*(db: TodoDB,
## pagination = none[PaginationParams]()): seq[TimeEntry]; ## pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
## proc getAllTimeEntries*[D: DbConnType](conn: D,
## pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
## ##
## proc findTimeEntriesWhere*(db: TodoDB, whereClause: string, ## proc findTimeEntriesWhere*(db: TodoDB, whereClause: string,
## values: varargs[string, dbFormat], pagination = none[PaginationParams]() ## values: varargs[string, dbFormat], pagination = none[PaginationParams]()
## ): seq[TimeEntry]; ## ): PagedRecords[TimeEntry];
## proc findTimeEntriesWhere*[D: DbConnType](conn: D, whereClause: string,
## values: varargs[string, dbFormat], pagination = none[PaginationParams]()
## ): PagedRecords[TimeEntry];
## ##
## proc findTimeEntriesByTodoItemId(db: TodoDB, todoItemId: UUID, ## proc findTimeEntriesByTodoItemId*(db: TodoDB, todoItemId: UUID,
## pagination = none[PaginationParams]()): seq[TimeEntry]; ## pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
## proc findTimeEntriesByTodoItemId*[D: DbConnType](
## conn: D, todoItemId: UUID,
## pagination = none[PaginationParams]()): PagedRecords[TimeEntry];
##
## Use the `dbType` flavor when the caller does not already have a connection.
## Use the connection flavor inside `withConnection` or `inTransaction`.
## The generated `get<RecordName>ForUpdate` helper is PostgreSQL-specific and
## is only available for direct PostgreSQL connections.
##
## Warning: do not call the `dbType` flavor from inside `inTransaction`.
## Those overloads call `withConnection` and may acquire a different
## connection, causing the statements to execute outside the active
## transaction.
##
## .. code-block:: Nim
## db.inTransaction:
## var item = conn.getTodoItemForUpdate(todoId)
## item.priority += 1
## discard conn.updateTodoItem(item)
## ##
## Object-Relational Modeling ## Object-Relational Modeling
## ========================== ## ==========================
@@ -140,11 +190,11 @@
## Model Class ## Model Class
## ----------- ## -----------
## ##
## Fiber ORM uses simple Nim `object`s and `ref object`s as model classes. ## Fiber ORM uses simple Nim objects and ref objects as model classes.
## Fiber ORM expects there to be one table for each model class. ## Fiber ORM expects there to be one table for each model class.
## ##
## Name Mapping ## Name Mapping
## ```````````` ## ^^^^^^^^^^^^
## Fiber ORM uses `snake_case` for database identifiers (column names, table ## Fiber ORM uses `snake_case` for database identifiers (column names, table
## names, etc.) and `camelCase` for Nim identifiers. We automatically convert ## names, etc.) and `camelCase` for Nim identifiers. We automatically convert
## model names to and from table names (`TodoItem` <-> `todo_items`), as well ## model names to and from table names (`TodoItem` <-> `todo_items`), as well
@@ -175,7 +225,7 @@
## .. _util: fiber_orm/util.html ## .. _util: fiber_orm/util.html
## ##
## ID Field ## ID Field
## ```````` ## ^^^^^^^^
## ##
## Fiber ORM expects every model class to have a field named `id`, with a ## Fiber ORM expects every model class to have a field named `id`, with a
## corresponding `id` column in the model table. This field must be either a ## corresponding `id` column in the model table. This field must be either a
@@ -266,6 +316,8 @@
## anything can be passed as the database object type so long as there is a ## anything can be passed as the database object type so long as there is a
## defined `withConnection` template that provides a `conn: DbConn` object ## defined `withConnection` template that provides a `conn: DbConn` object
## to the provided statement body. ## to the provided statement body.
## The generated connection-flavor procedures are intended to work directly
## with that `conn` value.
## ##
## For example, a valid database object implementation that opens a new ## For example, a valid database object implementation that opens a new
## connection for every request might look like this: ## connection for every request might look like this:
@@ -285,16 +337,17 @@
## .. _pool.DbConnPool: fiber_orm/pool.html#DbConnPool ## .. _pool.DbConnPool: fiber_orm/pool.html#DbConnPool
## ##
import std/[json, macros, options, sequtils, strutils] import std/[json, macros, options, sequtils, strutils]
import db_connector/db_common import db_connector/[db_common, db_postgres]
import namespaced_logging, uuids import uuids
from std/unicode import capitalize from std/unicode import capitalize
import ./fiber_orm/db_common as fiber_db_common import ./fiber_orm/db_common as fiber_db_common
import ./fiber_orm/pool import ./fiber_orm/[pool, util]
import ./fiber_orm/util import ./fiber_orm/private/logging
export pool, util export pool, util
export logging.enableDbLogging
type type
PagedRecords*[T] = object PagedRecords*[T] = object
@@ -308,26 +361,6 @@ type
NotFoundError* = object of CatchableError NotFoundError* = object of CatchableError
## Error type raised when no record matches a given ID ## Error type raised when no record matches a given ID
var logService {.threadvar.}: Option[ThreadLocalLogService]
var logger {.threadvar.}: Option[Logger]
proc makeQueryLogEntry(
m: string,
sql: string,
args: openArray[(string, string)] = []): JsonNode =
result = %*{ "method": m, "sql": sql }
for (k, v) in args: result[k] = %v
proc logQuery*(methodName: string, sqlStmt: string, args: openArray[(string, string)] = []) =
# namespaced_logging would do this check for us, but we don't want to even
# build the log object if we're not actually logging
if logService.isNone: return
if logger.isNone: logger = logService.getLogger("fiber_orm/query")
logger.debug(makeQueryLogEntry(methodName, sqlStmt, args))
proc enableDbLogging*(svc: ThreadLocalLogService) =
logService = some(svc)
proc newMutateClauses(): MutateClauses = proc newMutateClauses(): MutateClauses =
return MutateClauses( return MutateClauses(
columns: @[], columns: @[],
@@ -353,7 +386,6 @@ proc createRecord*[D: DbConnType, T](db: D, rec: T): T =
" RETURNING " & columnNamesForModel(rec).join(",") " RETURNING " & columnNamesForModel(rec).join(",")
logQuery("createRecord", sqlStmt) logQuery("createRecord", sqlStmt)
debug(logService.getLogger("fiber_orm/query"), %*{ "values": mc.values })
let newRow = db.getRow(sql(sqlStmt), mc.values) let newRow = db.getRow(sql(sqlStmt), mc.values)
@@ -422,6 +454,36 @@ template getRecord*[D: DbConnType](db: D, modelType: type, id: typed): untyped =
rowToModel(modelType, row) rowToModel(modelType, row)
template getRecordForUpdate*(db: db_postgres.DbConn, modelType: type, id: typed): untyped =
## Fetch a record by id and lock it with `FOR UPDATE`.
##
## This is PostgreSQL-specific and should only be used inside a transaction.
let sqlStmt =
"SELECT " & columnNamesForModel(modelType).join(",") &
" FROM " & tableName(modelType) &
" WHERE id = ? FOR UPDATE"
logQuery("getRecordForUpdate", sqlStmt, [("id", $id)])
let row = db.getRow(sql(sqlStmt), @[$id])
if allIt(row, it.len == 0):
raise newException(NotFoundError, "no " & modelName(modelType) & " record for id " & $id)
rowToModel(modelType, row)
template tryGetRecord*[D: DbConnType](db: D, modelType: type, id: typed): untyped =
## Fetch a record by id.
let sqlStmt =
"SELECT " & columnNamesForModel(modelType).join(",") &
" FROM " & tableName(modelType) &
" WHERE id = ?"
logQuery("tryGetRecord", sqlStmt, [("id", $id)])
let row = db.getRow(sql(sqlStmt), @[$id])
if allIt(row, it.len == 0): none[modelType]()
else: some(rowToModel(modelType, row))
template findRecordsWhere*[D: DbConnType]( template findRecordsWhere*[D: DbConnType](
db: D, db: D,
modelType: type, modelType: type,
@@ -594,23 +656,50 @@ template findViaJoinTable*[D: DbConnType](
macro generateProcsForModels*(dbType: type, modelTypes: openarray[type]): untyped = macro generateProcsForModels*(dbType: type, modelTypes: openarray[type]): untyped =
## Generate all standard access procedures for the given model types. For a ## Generate all standard access procedures for the given model types. For a
## `model class`_ named `TodoItem`, this will generate the following ## `model class`_ named `TodoItem`, this will generate `dbType` and
## procedures: ## connection overloads for procedures like the following:
## ##
## .. code-block:: Nim ## .. code-block:: Nim
## proc getTodoItem*(db: TodoDB, id: idType): TodoItem; ## proc getTodoItem*(db: TodoDB, id: idType): TodoItem;
## proc getAllTodoItems*(db: TodoDB): TodoItem; ## proc getTodoItem*[D: DbConnType](conn: D, id: idType): TodoItem;
## proc getTodoItemForUpdate*(conn: db_postgres.DbConn, id: idType): TodoItem;
## proc tryGetTodoItem*(db: TodoDB, id: idType): Option[TodoItem];
## proc tryGetTodoItem*[D: DbConnType](conn: D, id: idType): Option[TodoItem];
## proc getTodoItemIfItExists*(db: TodoDB, id: idType): Option[TodoItem];
## proc getTodoItemIfItExists*[D: DbConnType](
## conn: D, id: idType): Option[TodoItem];
## proc getAllTodoItems*(db: TodoDB): PagedRecords[TodoItem];
## proc getAllTodoItems*[D: DbConnType](conn: D): PagedRecords[TodoItem];
## proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem; ## proc createTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
## proc createTodoItem*[D: DbConnType](conn: D, rec: TodoItem): TodoItem;
## proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc deleteTodoItem*(db: TodoDB, rec: TodoItem): bool;
## proc deleteTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
## proc deleteTodoItem*(db: TodoDB, id: idType): bool; ## proc deleteTodoItem*(db: TodoDB, id: idType): bool;
## proc deleteTodoItem*[D: DbConnType](conn: D, id: idType): bool;
## proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc updateTodoItem*(db: TodoDB, rec: TodoItem): bool;
## proc createOrUpdateTodoItem*(db: TodoDB, rec: TodoItem): bool; ## proc updateTodoItem*[D: DbConnType](conn: D, rec: TodoItem): bool;
## proc createOrUpdateTodoItem*(db: TodoDB, rec: TodoItem): TodoItem;
## proc createOrUpdateTodoItem*[D: DbConnType](
## conn: D, rec: TodoItem): TodoItem;
## ##
## proc findTodoItemsWhere*( ## proc findTodoItemsWhere*(
## db: TodoDB, whereClause: string, values: varargs[string]): TodoItem; ## db: TodoDB,
## whereClause: string,
## values: varargs[string, dbFormat],
## pagination = none[PaginationParams]()): PagedRecords[TodoItem];
## proc findTodoItemsWhere*[D: DbConnType](
## conn: D,
## whereClause: string,
## values: varargs[string, dbFormat],
## pagination = none[PaginationParams]()): PagedRecords[TodoItem];
## ##
## `dbType` is expected to be some type that has a defined `withConnection` ## `dbType` is expected to be some type that has a defined `withConnection`
## procedure (see `Database Object`_ for details). ## procedure (see `Database Object`_ for details).
## The `dbType` overloads are convenience wrappers around `withConnection`.
## Inside `inTransaction`, prefer the overloads that take `conn: D` where
## `D: DbConnType` so all operations use the transaction connection.
## The generated `get<RecordName>ForUpdate` helper is PostgreSQL-specific and
## is only available for direct PostgreSQL connections.
## ##
## .. _Database Object: #database-object ## .. _Database Object: #database-object
result = newStmtList() result = newStmtList()
@@ -622,6 +711,8 @@ macro generateProcsForModels*(dbType: type, modelTypes: openarray[type]): untype
let modelName = $(t.getType[1]) let modelName = $(t.getType[1])
let getName = ident("get" & modelName) let getName = ident("get" & modelName)
let getForUpdateName = ident("get" & modelName & "ForUpdate")
let tryGetName = ident("tryGet" & modelName)
let getIfExistsName = ident("get" & modelName & "IfItExists") let getIfExistsName = ident("get" & modelName & "IfItExists")
let getAllName = ident("getAll" & pluralize(modelName)) let getAllName = ident("getAll" & pluralize(modelName))
let findWhereName = ident("find" & pluralize(modelName) & "Where") let findWhereName = ident("find" & pluralize(modelName) & "Where")
@@ -634,14 +725,35 @@ macro generateProcsForModels*(dbType: type, modelTypes: openarray[type]): untype
proc `getName`*(db: `dbType`, id: `idType`): `t` = proc `getName`*(db: `dbType`, id: `idType`): `t` =
db.withConnection conn: result = getRecord(conn, `t`, id) db.withConnection conn: result = getRecord(conn, `t`, id)
proc `getName`*[D: DbConnType](conn: D, id: `idType`): `t` =
result = getRecord(conn, `t`, id)
proc `getForUpdateName`*(conn: db_postgres.DbConn, id: `idType`): `t` =
result = getRecordForUpdate(conn, `t`, id)
proc `tryGetName`*(db: `dbType`, id: `idType`): Option[`t`] =
db.withConnection conn: result = tryGetRecord(conn, `t`, id)
proc `tryGetName`*[D: DbConnType](conn: D, id: `idType`): Option[`t`] =
result = tryGetRecord(conn, `t`, id)
proc `getIfExistsName`*(db: `dbType`, id: `idType`): Option[`t`] = proc `getIfExistsName`*(db: `dbType`, id: `idType`): Option[`t`] =
db.withConnection conn: db.withConnection conn:
try: result = some(getRecord(conn, `t`, id)) try: result = some(getRecord(conn, `t`, id))
except NotFoundError: result = none[`t`]() except NotFoundError: result = none[`t`]()
proc `getIfExistsName`*[D: DbConnType](conn: D, id: `idType`): Option[`t`] =
try: result = some(getRecord(conn, `t`, id))
except NotFoundError: result = none[`t`]()
proc `getAllName`*(db: `dbType`, pagination = none[PaginationParams]()): PagedRecords[`t`] = proc `getAllName`*(db: `dbType`, pagination = none[PaginationParams]()): PagedRecords[`t`] =
db.withConnection conn: result = getAllRecords(conn, `t`, pagination) db.withConnection conn: result = getAllRecords(conn, `t`, pagination)
proc `getAllName`*[D: DbConnType](
conn: D,
pagination = none[PaginationParams]()): PagedRecords[`t`] =
result = getAllRecords(conn, `t`, pagination)
proc `findWhereName`*( proc `findWhereName`*(
db: `dbType`, db: `dbType`,
whereClause: string, whereClause: string,
@@ -650,21 +762,43 @@ macro generateProcsForModels*(dbType: type, modelTypes: openarray[type]): untype
db.withConnection conn: db.withConnection conn:
result = findRecordsWhere(conn, `t`, whereClause, values, pagination) result = findRecordsWhere(conn, `t`, whereClause, values, pagination)
proc `findWhereName`*[D: DbConnType](
conn: D,
whereClause: string,
values: varargs[string, dbFormat],
pagination = none[PaginationParams]()): PagedRecords[`t`] =
result = findRecordsWhere(conn, `t`, whereClause, values, pagination)
proc `createName`*(db: `dbType`, rec: `t`): `t` = proc `createName`*(db: `dbType`, rec: `t`): `t` =
db.withConnection conn: result = createRecord(conn, rec) db.withConnection conn: result = createRecord(conn, rec)
proc `createName`*[D: DbConnType](conn: D, rec: `t`): `t` =
result = createRecord(conn, rec)
proc `updateName`*(db: `dbType`, rec: `t`): bool = proc `updateName`*(db: `dbType`, rec: `t`): bool =
db.withConnection conn: result = updateRecord(conn, rec) db.withConnection conn: result = updateRecord(conn, rec)
proc `updateName`*[D: DbConnType](conn: D, rec: `t`): bool =
result = updateRecord(conn, rec)
proc `createOrUpdateName`*(db: `dbType`, rec: `t`): `t` = proc `createOrUpdateName`*(db: `dbType`, rec: `t`): `t` =
db.inTransaction: result = createOrUpdateRecord(conn, rec) db.inTransaction: result = createOrUpdateRecord(conn, rec)
proc `createOrUpdateName`*[D: DbConnType](conn: D, rec: `t`): `t` =
result = createOrUpdateRecord(conn, rec)
proc `deleteName`*(db: `dbType`, rec: `t`): bool = proc `deleteName`*(db: `dbType`, rec: `t`): bool =
db.withConnection conn: result = deleteRecord(conn, rec) db.withConnection conn: result = deleteRecord(conn, rec)
proc `deleteName`*[D: DbConnType](conn: D, rec: `t`): bool =
result = deleteRecord(conn, rec)
proc `deleteName`*(db: `dbType`, id: `idType`): bool = proc `deleteName`*(db: `dbType`, id: `idType`): bool =
db.withConnection conn: result = deleteRecord(conn, `t`, id) db.withConnection conn: result = deleteRecord(conn, `t`, id)
proc `deleteName`*[D: DbConnType](conn: D, id: `idType`): bool =
result = deleteRecord(conn, `t`, id)
macro generateLookup*(dbType: type, modelType: type, fields: seq[string]): untyped = macro generateLookup*(dbType: type, modelType: type, fields: seq[string]): untyped =
## Create a lookup procedure for a given set of field names. For example, ## Create a lookup procedure for a given set of field names. For example,
## given the TODO database demostrated above, ## given the TODO database demostrated above,
@@ -676,42 +810,49 @@ macro generateLookup*(dbType: type, modelType: type, fields: seq[string]): untyp
## ##
## .. code-block:: Nim ## .. code-block:: Nim
## proc findTodoItemsByOwnerAndPriority*(db: SampleDB, ## proc findTodoItemsByOwnerAndPriority*(db: SampleDB,
## owner: string, priority: int): seq[TodoItem] ## owner: string, priority: int,
## pagination = none[PaginationParams]()): PagedRecords[TodoItem]
## proc findTodoItemsByOwnerAndPriority*[D: DbConnType](conn: D,
## owner: string, priority: int,
## pagination = none[PaginationParams]()): PagedRecords[TodoItem]
##
## Use the `db` overload for standalone calls and the `conn` overload inside
## `withConnection` or `inTransaction`.
let fieldNames = fields[1].mapIt($it) let fieldNames = fields[1].mapIt($it)
let procName = ident("find" & pluralize($modelType.getType[1]) & "By" & fieldNames.mapIt(it.capitalize).join("And")) let procName = ident("find" & pluralize($modelType.getType[1]) & "By" & fieldNames.mapIt(it.capitalize).join("And"))
# Create proc skeleton
result = quote do:
proc `procName`*(db: `dbType`): PagedRecords[`modelType`] =
db.withConnection conn: result = findRecordsBy(conn, `modelType`)
var callParams = quote do: @[] var callParams = quote do: @[]
# Add dynamic parameters for the proc definition and inner proc call # Add dynamic parameters for the generated proc and inner proc call.
for n in fieldNames: for n in fieldNames:
let paramTuple = newNimNode(nnkPar) let paramTuple = newNimNode(nnkPar)
paramTuple.add(newColonExpr(ident("field"), newLit(identNameToDb(n)))) paramTuple.add(newColonExpr(ident("field"), newLit(identNameToDb(n))))
paramTuple.add(newColonExpr(ident("value"), ident(n))) paramTuple.add(newColonExpr(ident("value"), ident(n)))
# Add the parameter to the outer call (the generated proc)
# result[3] is ProcDef -> [3]: FormalParams
result[3].add(newIdentDefs(ident(n), ident("string")))
# Build up the AST for the inner procedure call
callParams[1].add(paramTuple) callParams[1].add(paramTuple)
# Add the optional pagination parameters to the generated proc definition let dbProcDefAST = quote do:
result[3].add(newIdentDefs( proc `procName`*(db: `dbType`): PagedRecords[`modelType`] =
db.withConnection conn:
result = findRecordsBy(conn, `modelType`, `callParams`, pagination)
let connProcDefAST = quote do:
proc `procName`*[D: DbConnType](conn: D): PagedRecords[`modelType`] =
result = findRecordsBy(conn, `modelType`, `callParams.copyNimTree`, pagination)
for n in fieldNames:
dbProcDefAST[3].add(newIdentDefs(ident(n), ident("string")))
connProcDefAST[3].add(newIdentDefs(ident(n), ident("string")))
dbProcDefAST[3].add(newIdentDefs(
ident("pagination"), newEmptyNode(), ident("pagination"), newEmptyNode(),
quote do: none[PaginationParams]())) quote do: none[PaginationParams]()))
# Add the call params to the inner procedure call connProcDefAST[3].add(newIdentDefs(
# result[6][0][1][0][1] is ident("pagination"), newEmptyNode(),
# ProcDef -> [6]: StmtList (body) -> [0]: Command -> quote do: none[PaginationParams]()))
# [2]: StmtList (withConnection body) -> [0]: Asgn (result =) ->
# [1]: Call (inner findRecords invocation) result = newStmtList()
result[6][0][2][0][1].add(callParams) result.add dbProcDefAST
result[6][0][2][0][1].add(quote do: pagination) result.add connProcDefAST
macro generateProcsForFieldLookups*(dbType: type, modelsAndFields: openarray[tuple[t: type, fields: seq[string]]]): untyped = macro generateProcsForFieldLookups*(dbType: type, modelsAndFields: openarray[tuple[t: type, fields: seq[string]]]): untyped =
result = newStmtList() result = newStmtList()
@@ -721,32 +862,38 @@ macro generateProcsForFieldLookups*(dbType: type, modelsAndFields: openarray[tup
let fieldNames = i[1][1][1].mapIt($it) let fieldNames = i[1][1][1].mapIt($it)
let procName = ident("find" & $modelType & "sBy" & fieldNames.mapIt(it.capitalize).join("And")) let procName = ident("find" & $modelType & "sBy" & fieldNames.mapIt(it.capitalize).join("And"))
# Create proc skeleton
let procDefAST = quote do:
proc `procName`*(db: `dbType`): PagedRecords[`modelType`] =
db.withConnection conn: result = findRecordsBy(conn, `modelType`)
var callParams = quote do: @[] var callParams = quote do: @[]
# Add dynamic parameters for the proc definition and inner proc call # Add dynamic parameters for the generated proc and inner proc call.
for n in fieldNames: for n in fieldNames:
let paramTuple = newNimNode(nnkPar) let paramTuple = newNimNode(nnkPar)
paramTuple.add(newColonExpr(ident("field"), newLit(identNameToDb(n)))) paramTuple.add(newColonExpr(ident("field"), newLit(identNameToDb(n))))
paramTuple.add(newColonExpr(ident("value"), ident(n))) paramTuple.add(newColonExpr(ident("value"), ident(n)))
procDefAST[3].add(newIdentDefs(ident(n), ident("string")))
callParams[1].add(paramTuple) callParams[1].add(paramTuple)
# Add the optional pagination parameters to the generated proc definition let dbProcDefAST = quote do:
procDefAST[3].add(newIdentDefs( proc `procName`*(db: `dbType`): PagedRecords[`modelType`] =
db.withConnection conn:
result = findRecordsBy(conn, `modelType`, `callParams`, pagination)
let connProcDefAST = quote do:
proc `procName`*[D: DbConnType](conn: D): PagedRecords[`modelType`] =
result = findRecordsBy(conn, `modelType`, `callParams.copyNimTree`, pagination)
for n in fieldNames:
dbProcDefAST[3].add(newIdentDefs(ident(n), ident("string")))
connProcDefAST[3].add(newIdentDefs(ident(n), ident("string")))
dbProcDefAST[3].add(newIdentDefs(
ident("pagination"), newEmptyNode(), ident("pagination"), newEmptyNode(),
quote do: none[PaginationParams]())) quote do: none[PaginationParams]()))
procDefAST[6][0][1][0][1].add(callParams) connProcDefAST[3].add(newIdentDefs(
procDefAST[6][0][1][0][1].add(quote do: pagination) ident("pagination"), newEmptyNode(),
quote do: none[PaginationParams]()))
result.add procDefAST result.add dbProcDefAST
result.add connProcDefAST
macro generateJoinTableProcs*( macro generateJoinTableProcs*(
dbType, model1Type, model2Type: type, dbType, model1Type, model2Type: type,
@@ -758,11 +905,19 @@ macro generateJoinTableProcs*(
## This macro will generate the following procedures: ## This macro will generate the following procedures:
## ##
## .. code-block:: Nim ## .. code-block:: Nim
## proc findTodoItemsByTimeEntry*(db: SampleDB, timeEntry: TimeEntry): seq[TodoItem] ## proc getTodoItemsByTimeEntry*(db: SampleDB, timeEntry: TimeEntry,
## proc findTimeEntriesByTodoItem*(db: SampleDB, todoItem: TodoItem): seq[TimeEntry] ## pagination = none[PaginationParams]()): PagedRecords[TodoItem]
## proc getTodoItemsByTimeEntry*[D: DbConnType](conn: D, timeEntry: TimeEntry,
## pagination = none[PaginationParams]()): PagedRecords[TodoItem]
## proc getTimeEntriesByTodoItem*(db: SampleDB, todoItem: TodoItem,
## pagination = none[PaginationParams]()): PagedRecords[TimeEntry]
## proc getTimeEntriesByTodoItem*[D: DbConnType](conn: D, todoItem: TodoItem,
## pagination = none[PaginationParams]()): PagedRecords[TimeEntry]
## ##
## `dbType` is expected to be some type that has a defined `withConnection` ## `dbType` is expected to be some type that has a defined `withConnection`
## procedure (see `Database Object`_ for details). ## procedure (see `Database Object`_ for details).
## As with the other generated helpers, use the connection overloads when
## you are already inside `withConnection` or `inTransaction`.
## ##
## .. _Database Object: #database-object ## .. _Database Object: #database-object
result = newStmtList() result = newStmtList()
@@ -794,6 +949,18 @@ macro generateJoinTableProcs*(
id, id,
pagination) pagination)
proc `getModel1Name`*[D: DbConnType](
conn: D,
id: `id2Type`,
pagination = none[PaginationParams]()): PagedRecords[`model1Type`] =
result = findViaJoinTable(
conn,
`joinTableNameNode`,
`model1Type`,
`model2Type`,
id,
pagination)
proc `getModel1Name`*( proc `getModel1Name`*(
db: `dbType`, db: `dbType`,
rec: `model2Type`, rec: `model2Type`,
@@ -806,10 +973,21 @@ macro generateJoinTableProcs*(
rec, rec,
pagination) pagination)
proc `getModel1Name`*[D: DbConnType](
conn: D,
rec: `model2Type`,
pagination = none[PaginationParams]()): PagedRecords[`model1Type`] =
result = findViaJoinTable(
conn,
`joinTableNameNode`,
`model1Type`,
rec,
pagination)
proc `getModel2Name`*( proc `getModel2Name`*(
db: `dbType`, db: `dbType`,
id: `id1Type`, id: `id1Type`,
pagination = none[PaginationParams]()): Pagedrecords[`model2Type`] = pagination = none[PaginationParams]()): PagedRecords[`model2Type`] =
db.withConnection conn: db.withConnection conn:
result = findViaJoinTable( result = findViaJoinTable(
conn, conn,
@@ -819,10 +997,22 @@ macro generateJoinTableProcs*(
id, id,
pagination) pagination)
proc `getModel2Name`*[D: DbConnType](
conn: D,
id: `id1Type`,
pagination = none[PaginationParams]()): PagedRecords[`model2Type`] =
result = findViaJoinTable(
conn,
`joinTableNameNode`,
`model2Type`,
`model1Type`,
id,
pagination)
proc `getModel2Name`*( proc `getModel2Name`*(
db: `dbType`, db: `dbType`,
rec: `model1Type`, rec: `model1Type`,
pagination = none[PaginationParams]()): Pagedrecords[`model2Type`] = pagination = none[PaginationParams]()): PagedRecords[`model2Type`] =
db.withConnection conn: db.withConnection conn:
result = findViaJoinTable( result = findViaJoinTable(
conn, conn,
@@ -831,6 +1021,17 @@ macro generateJoinTableProcs*(
rec, rec,
pagination) pagination)
proc `getModel2Name`*[D: DbConnType](
conn: D,
rec: `model1Type`,
pagination = none[PaginationParams]()): PagedRecords[`model2Type`] =
result = findViaJoinTable(
conn,
`joinTableNameNode`,
`model2Type`,
rec,
pagination)
proc associate*( proc associate*(
db: `dbType`, db: `dbType`,
rec1: `model1Type`, rec1: `model1Type`,
@@ -838,6 +1039,12 @@ macro generateJoinTableProcs*(
db.withConnection conn: db.withConnection conn:
associate(conn, `joinTableNameNode`, rec1, rec2) associate(conn, `joinTableNameNode`, rec1, rec2)
proc associate*[D: DbConnType](
conn: D,
rec1: `model1Type`,
rec2: `model2Type`): void =
associate(conn, `joinTableNameNode`, rec1, rec2)
proc associate*( proc associate*(
db: `dbType`, db: `dbType`,
rec2: `model2Type`, rec2: `model2Type`,
@@ -845,36 +1052,22 @@ macro generateJoinTableProcs*(
db.withConnection conn: db.withConnection conn:
associate(conn, `joinTableNameNode`, rec1, rec2) associate(conn, `joinTableNameNode`, rec1, rec2)
proc initPool*[D: DbConnType]( proc associate*[D: DbConnType](
connect: proc(): D, conn: D,
poolSize = 10, rec2: `model2Type`,
hardCap = false, rec1: `model1Type`): void =
healthCheckQuery = "SELECT 'true' AS alive"): DbConnPool[D] = associate(conn, `joinTableNameNode`, rec1, rec2)
## Initialize a new DbConnPool. See the `initDb` procedure in the `Example
## Fiber ORM Usage`_ for an example
##
## * `connect` must be a factory which creates a new `DbConn`.
## * `poolSize` sets the desired capacity of the connection pool.
## * `hardCap` defaults to `false`.
## When `false`, the pool can grow beyond the configured capacity, but will
## release connections down to the its capacity (no less than `poolSize`).
##
## When `true` the pool will not create more than its configured capacity.
## It a connection is requested, none are free, and the pool is at
## capacity, this will result in an Error being raised.
## * `healthCheckQuery` should be a simple and fast SQL query that the pool
## can use to test the liveliness of pooled connections.
##
## .. _Example Fiber ORM Usage: #basic-usage-example-fiber-orm-usage
initDbConnPool(DbConnPoolConfig[D](
connect: connect,
poolSize: poolSize,
hardCap: hardCap,
healthCheckQuery: healthCheckQuery))
template inTransaction*(db, body: untyped) = template inTransaction*(db, body: untyped) =
## Execute `body` inside a transaction using a single connection bound to
## `conn`.
##
## When calling generated Fiber ORM helpers inside this block, use the
## overloads that take `conn: D` where `D: DbConnType`. Do not call the
## overloads that take the outer database object, because those call
## `withConnection` again and may acquire a different connection.
## If you need to lock a PostgreSQL row before modifying it, use the
## generated `get<RecordName>ForUpdate` helper.
db.withConnection conn: db.withConnection conn:
conn.exec(sql"BEGIN TRANSACTION") conn.exec(sql"BEGIN TRANSACTION")
try: try:

148
src/fiber_orm/pool.nim
View File

@@ -4,85 +4,77 @@
## Simple database connection pooling implementation compatible with Fiber ORM. ## Simple database connection pooling implementation compatible with Fiber ORM.
import std/[sequtils, sugar] when (NimMajor, NimMinor, NimPatch) < (2, 0, 0):
when not defined(gcArc) and not defined (gcOrc):
{.error: "fiber_orm requires either --mm:arc or --mm:orc.".}
import std/[deques, locks, sequtils, sugar]
import db_connector/db_common import db_connector/db_common
from db_connector/db_sqlite import getRow, close from db_connector/db_sqlite import getRow, close
from db_connector/db_postgres import getRow, close from db_connector/db_postgres import getRow, close
import ./db_common as fiber_db_common import ./db_common as fiber_db_common
import ./private/logging
type type
DbConnPoolConfig*[D: DbConnType] = object DbConnPool*[D: DbConnType] = ptr DbConnPoolObj[D]
connect*: () -> D ## Factory procedure to create a new DBConn
poolSize*: int ## The pool capacity.
hardCap*: bool ## Is the pool capacity a hard cap? DbConnPoolObj[D: DbConnType] = object
##
## When `false`, the pool can grow beyond the
## configured capacity, but will release connections
## down to the its capacity (no less than `poolSize`).
##
## When `true` the pool will not create more than its
## configured capacity. It a connection is requested,
## none are free, and the pool is at capacity, this
## will result in an Error being raised.
healthCheckQuery*: string ## Should be a simple and fast SQL query that the
## pool can use to test the liveliness of pooled
## connections.
PooledDbConn[D: DbConnType] = ref object
conn: D
id: int
free: bool
DbConnPool*[D: DbConnType] = ref object
## Database connection pool ## Database connection pool
conns: seq[PooledDbConn[D]] connect: proc (): D {.raises: [DbError].}
cfg: DbConnPoolConfig[D] healthCheckQuery: SqlQuery
lastId: int entries: Deque[D]
cond: Cond
lock: Lock
proc initDbConnPool*[D: DbConnType](cfg: DbConnPoolConfig[D]): DbConnPool[D] =
result = DbConnPool[D](
conns: @[],
cfg: cfg)
proc newConn[D: DbConnType](pool: DbConnPool[D]): PooledDbConn[D] = proc close*[D: DbConnType](pool: DbConnPool[D]) =
pool.lastId += 1 ## Safely close all connections and release resources for the given pool.
{.gcsafe.}: getLogger("pool").debug("closing connection pool")
let conn = pool.cfg.connect() withLock(pool.lock):
result = PooledDbConn[D]( while pool.entries.len > 0: close(pool.entries.popFirst())
conn: conn,
id: pool.lastId,
free: true)
pool.conns.add(result)
proc maintain[D: DbConnType](pool: DbConnPool[D]): void = deinitLock(pool.lock)
pool.conns.keepIf(proc (pc: PooledDbConn[D]): bool = deinitCond(pool.cond)
if not pc.free: return true `=destroy`(pool[])
deallocShared(pool)
proc newDbConnPool*[D: DbConnType](
poolSize: int,
connectFunc: proc(): D {.raises: [DbError].},
healthCheckQuery = "SELECT 1;"): DbConnPool[D] =
## Initialize a new DbConnPool. See the `initDb` procedure in the `Example
## Fiber ORM Usage`_ for an example
##
## * `connect` must be a factory which creates a new `DbConn`.
## * `poolSize` sets the desired capacity of the connection pool.
## * `healthCheckQuery` should be a simple and fast SQL query that the pool
## can use to test the liveliness of pooled connections. By default it uses
## `SELECT 1;`
##
## .. _Example Fiber ORM Usage: ../fiber_orm.html#basic-usage-example-fiber-orm-usage
result = cast[DbConnPool[D]](allocShared0(sizeof(DbConnPoolObj[D])))
initCond(result.cond)
initLock(result.lock)
result.entries = initDeque[D](poolSize)
result.connect = connectFunc
result.healthCheckQuery = sql(healthCheckQuery)
try: try:
discard getRow(pc.conn, sql(pool.cfg.healthCheckQuery), []) for _ in 0 ..< poolSize: result.entries.addLast(connectFunc())
return true except DbError as ex:
except: try: result.close()
try: pc.conn.close() # try to close the connection except: discard
except: discard "" getLogger("pool").error(
return false msg = "unable to initialize connection pool",
) err = ex)
raise ex
let freeConns = pool.conns.filterIt(it.free)
if pool.conns.len > pool.cfg.poolSize and freeConns.len > 0:
let numToCull = min(freeConns.len, pool.conns.len - pool.cfg.poolSize)
if numToCull > 0: proc take*[D: DbConnType](pool: DbConnPool[D]): D {.raises: [DbError], gcsafe.} =
let toCull = freeConns[0..numToCull]
pool.conns.keepIf((pc) => toCull.allIt(it.id != pc.id))
for culled in toCull:
try: culled.conn.close()
except: discard ""
proc take*[D: DbConnType](pool: DbConnPool[D]): tuple[id: int, conn: D] =
## Request a connection from the pool. Returns a DbConn if the pool has free ## Request a connection from the pool. Returns a DbConn if the pool has free
## connections, or if it has the capacity to create a new connection. If the ## connections, or if it has the capacity to create a new connection. If the
## pool is configured with a hard capacity limit and is out of free ## pool is configured with a hard capacity limit and is out of free
@@ -90,25 +82,33 @@ proc take*[D: DbConnType](pool: DbConnPool[D]): tuple[id: int, conn: D] =
## ##
## Connections taken must be returned via `release` when the caller is ## Connections taken must be returned via `release` when the caller is
## finished using them in order for them to be released back to the pool. ## finished using them in order for them to be released back to the pool.
pool.maintain withLock(pool.lock):
let freeConns = pool.conns.filterIt(it.free) while pool.entries.len == 0: wait(pool.cond, pool.lock)
result = pool.entries.popFirst()
let reserved = # check that the connection is healthy
if freeConns.len > 0: freeConns[0] try: discard getRow(result, pool.healthCheckQuery, [])
else: pool.newConn() except DbError:
{.gcsafe.}:
# if it's not, let's try to close it and create a new connection
try:
getLogger("pool").info(
"pooled connection failed health check, opening a new connection")
close(result)
except: discard
result = pool.connect()
reserved.free = false
return (id: reserved.id, conn: reserved.conn)
proc release*[D: DbConnType](pool: DbConnPool[D], connId: int): void = proc release*[D: DbConnType](pool: DbConnPool[D], conn: D) {.raises: [], gcsafe.} =
## Release a connection back to the pool. ## Release a connection back to the pool.
let foundConn = pool.conns.filterIt(it.id == connId) withLock(pool.lock):
if foundConn.len > 0: foundConn[0].free = true pool.entries.addLast(conn)
signal(pool.cond)
template withConnection*[D: DbConnType](pool: DbConnPool[D], conn, stmt: untyped): untyped = template withConnection*[D: DbConnType](pool: DbConnPool[D], conn, stmt: untyped): untyped =
## Convenience template to provide a connection from the pool for use in a ## Convenience template to provide a connection from the pool for use in a
## statement block, automatically releasing that connnection when done. ## statement block, automatically releasing that connnection when done.
block: block:
let (connId, conn) = take(pool) let conn = take(pool)
try: stmt try: stmt
finally: release(pool, connId) finally: release(pool, conn)

34
src/fiber_orm/private/logging.nim Normal file
View File

@@ -0,0 +1,34 @@
import std/[json, options]
import namespaced_logging
export namespaced_logging.log
export namespaced_logging.debug
export namespaced_logging.info
export namespaced_logging.notice
export namespaced_logging.warn
export namespaced_logging.error
export namespaced_logging.fatal
var logService {.threadvar.}: Option[ThreadLocalLogService]
var logger {.threadvar.}: Option[Logger]
proc makeQueryLogEntry(
m: string,
sql: string,
args: openArray[(string, string)] = []): JsonNode =
result = %*{ "method": m, "sql": sql }
for (k, v) in args: result[k] = %v
proc logQuery*(methodName: string, sqlStmt: string, args: openArray[(string, string)] = []) =
# namespaced_logging would do this check for us, but we don't want to even
# build the log object if we're not actually logging
if logService.isNone: return
if logger.isNone: logger = logService.getLogger("fiber_orm/query")
logger.debug(makeQueryLogEntry(methodName, sqlStmt, args))
proc enableDbLogging*(svc: ThreadLocalLogService) =
logService = some(svc)
proc getLogger*(scope: string): Option[Logger] =
logService.getLogger("fiber_orm/" & scope)

6
src/fiber_orm/util.nim
View File

@@ -256,6 +256,12 @@ func createParseStmt*(t, value: NimNode): NimNode =
else: error "Cannot parse column with unknown generic instance type: " & $t.getTypeInst else: error "Cannot parse column with unknown generic instance type: " & $t.getTypeInst
elif t.typeKind == ntyDistinct:
result = quote do:
block:
let tmp: `t` = `value`
tmp
elif t.typeKind == ntyRef: elif t.typeKind == ntyRef:
if $t.getTypeInst == "JsonNode": if $t.getTypeInst == "JsonNode":