Define explicit ownership for custom appenders and formatters

Add SharedLogState for synchronized value data and factories for
independently owned appender state and captured formatter closures.
Reject unsafe configuration state, update the test collector to value
snapshots, and document the breaking APIs in version 3.0.0.

Expand ORC ownership and sanitizer coverage and reject stale test
executables.
This commit is contained in:
jdb committed 2026-09-28 14:28:24 -05:00
1 parent 851caef706
commit b2d0e9dade
11 files changed
+808 -110

No files matched your search

+104 -38
View File
@@ -104,16 +104,44 @@ loggers upwards in the scope naming heirarchy. This pattern is explained in
detail in the [logback documentation][effective logging level] and applies in
the same manner to loggers in this library.
### LogMessageFormater
### LogFormatter
Both the [ConsoleLogAppender](#ConsoleLogAppender) and
[FileLogAppender](#FileLogAppender) can be given a *LogMessageFormatter* to
determine how a log message is formatted before being written.
Console, file, and std/logging appenders accept a *LogFormatter*. Noncapturing
functions such as *formatSimpleTextLog* and *formatJsonStructuredLog* are
converted automatically:
```nim
type LogMessageFormatter* = proc (msg: LogMessage): string {.gcsafe.}
let appender = initConsoleLogAppender(formatter = formatJsonStructuredLog)
```
For a formatter that captures state, pass value configuration and a
noncapturing factory to *initLogFormatter*. The factory creates a separate
closure in each thread on first use:
```nim
proc makeFormatter(prefix: string): StatefulLogFormatterFunc =
var count = 0
result = proc(msg: LogMessage): string {.gcsafe.} =
inc count
prefix & $count & ": " & msg.msg
let appender = initConsoleLogAppender(
formatter = initLogFormatter("request ", makeFormatter))
```
The factory's input must contain value data, such as strings, sequences of
values, or *SharedLogState* handles. Ordinary `ref` values and closures are
rejected, including when nested inside other types. The returned closure may
capture independently created references and mutable state belonging to its
thread. It must not return a closure shared with another thread.
Each appender clone starts fresh formatter state; configuration refresh can
reate new clones. For counters or other data that must persist across
refreshes or be shared between threads, pass a *SharedLogState* handle as
factory input. To invoke a formatter directly, use
`formatter.format(message)`.
## Available Appenders
### ConsoleLogAppender
@@ -122,7 +150,7 @@ Used for writing logs to stdout or stderr.
```nim
proc initConsoleLogAppender*(
formatter = formatSimpleTextLog,
formatter: LogFormatter = nil, # defaults to formatSimpleTextLog
## formatJsonStructuredLog is another useful formatter provided
## or you can write your own
useStderr = false, ## stdout is used by default
@@ -144,7 +172,7 @@ Used for writing logs to files.
```nim
proc initFileLogAppender*(
filePath: string,
formatter = formatSimpleTextLog,
formatter: LogFormatter = nil, # defaults to formatSimpleTextLog
## formatJsonStructuredLog is another useful formatter provided
## or you can write your own
namespace = "",
@@ -178,41 +206,75 @@ messages regardless of whether namespaced\_logging has other appenders by
setting `fallbackOnly = false`.
```nim
func initStdLoggingAppender*(
proc initStdLoggingAppender*(
fallbackOnly = true,
formatter = formatForwardedLog,
formatter: LogFormatter = nil, # defaults to formatForwardedLog
namespace = "",
threshold = lvlAll): StdLoggingAppender {.gcsafe.}
```
### CustomLogAppender
Provides an extension point for custom logging implementations.
Provides an extension point for custom logging implementation.
Use *initCustomLogAppender* for state that can be copied by value. Its callback
receives `state: var T` and a *LogMessage*. Each clone owns a separate state
value. Ordinary `ref` values, closures, and embedded locks or atomics are
rejected, including inside containers and objects.
For mutable state shared by all workers, put value data in a *SharedLogState*:
```nim
func initCustomLogAppender*[T](
state: T, # arbitrary state needed for the appender
doLogMessage: CustomLogAppenderFunc[T],
# custom log appender implementation
namespace = "",
threshold = lvlAll): CustomLogAppender[T] {.gcsafe.} =
let count = initSharedLogState(0)
proc countMessage(state: var SharedLogState[int], msg: LogMessage) {.gcsafe, noconv.} =
state.withState(proc(value: var int) = inc value)
var service = initLogService()
service.addAppender(initCustomLogAppender(count, countMessage))
# Pass service copies to workers as described below.
# count.snapshot() reads the current count safely.
```
The `state` field allows you to explicitly pass in any data that is required
for the custom functionality.
Copies of the handle share the same data and keep it alive. *withState* runs
its callback synchronously with exclusive access; *snapshot* returns a value
copy made under the same lock. Neither operation allows unsynchronized access
to the stored payload. Do not retain a pointer to the callback's argument or
call back into the same handle from inside *withState*, since its lock is not
recursive.
> [!CAUTION]
> Custom state is currently copied by assignment when an appender is cloned.
> Ordinary ORC `ref` values, including references nested in value types, must
> not be shared this way across threads. Keeping an original reference alive
> does not protect reference counts. A safe custom-state and captured-formatter
> ownership API is still pending; use independently owned values or an explicit
> shared-state abstraction with a synchronized lifetime and access protocol.
The payload must contain value data without ordinary references, closures, or
embedded synchronization primitives. Strings and sequences of supported values
are accepted. Raw pointers and types with custom copy/destruction hooks remain
an explicit caller responsibility: their lifetime and synchronization must be
safe across threads. *SharedLogState* does not make a raw pointer's target
safe.
See [testutil][] and the unit tests in [namespaced\_logging][nsl-unit-tests]
for a single-threaded example; the test collector's lock does not make its
reference-counted state safe to share across threads.
The [testutil][] collector demonstrates shared state using value snapshots.
*LoggedMessages.get()* returns `seq[RecordedLogMessage]`: timestamps are
*Time* values, additional JSON data is serialized text, and exceptions retain
name, message, and stacktrace text. Returned snapshots can be inspected and
modified independently without sharing the original message's references.
### FactoryCustomLogAppender
Provides an extension point for custom logging with state created separately
in each logging thread. Use this when state needs ordinary references or
closures owned by that thread.
Call *initFactoryCustomLogAppender(seed, factory, doLogMessage)* during logging
setup. The noncapturing factory receives a copyable value seed and creates
independently owned state on the first message handled by each clone, in the
thread doing the logging. The logging callback receives `state: var T` and a
*LogMessage*.
The seed follows the same ownership restrictions as *CustomLogAppender* state;
it may include *SharedLogState* handles. The state returned by the factory may
contain independently created references and closures. State lasts for that
clone's lifetime; configuration refresh may replace it. A factory that raises
is retried on the next message, and errors go to the configured error handler.
Do not retain managed references from an incoming *LogMessage* for use in
another thread.
## Notes on Use in Multi-Threaded Applications
@@ -247,12 +309,13 @@ subsequent logging calls in every thread using that service, without recreating
loggers or configuring each worker separately.
> [!NOTE]
> Thread-safety work is still in progress. For now, use the built-in appenders
> with default or stateless formatters when sharing a service across threads.
> Custom appenders with shared reference-based state and formatters with
> captured state are not yet generally safe. Runtime configuration refresh
> and delivery of queued background output at application exit also have known
> failures. See [the regression tests](tests/README.md) for the remaining work.
> Thread-safety work is still in progress. Custom state and captured
> formatters must follow the ownership rules above. Runtime configuration
> refresh, concurrent local-time timestamp construction, and delivery of queued
> background output at application exit still have known failures. Nim 2.2.10
> also produces sanitizer reports in its default allocator during some
> cross-thread allocation patterns. See
> [the regression tests](tests/README.md) for the remaining work.
## Architectural Design
@@ -335,10 +398,13 @@ configuration. This internal type is not exposed to library users but manages:
detection
Console and file writers remain process-wide. The shared control block owns
appender prototypes; refresh borrows them under the configuration lock and calls
`clone()` to create local instances. Clones must not retain ordinary managed
references shared across threads. That custom-state and formatter contract is
the next part of the ownership work.
appender prototypes; refresh borrows them under the configuration lock and
calls `clone()` to create local instances. Clones must not retain ordinary
managed references shared across threads. Built-in custom appenders copy
checked value state or shared-state handles. Factory appenders and formatters
copy value seeds and create their managed state lazily within the thread that
uses each clone. The state of an already-used clone is never copied into
another clone.
### Thread Safety Model