Define explicit ownership for custom appenders and formatters
Add SharedLogState for synchronized value data and factories for independently owned appender state and captured formatter closures. Reject unsafe configuration state, update the test collector to value snapshots, and document the breaking APIs in version 3.0.0. Expand ORC ownership and sanitizer coverage and reject stale test executables.
This commit is contained in:
1 parent
851caef706
commit
b2d0e9dade
11 files changed
+808
-110
No files matched your search
@@ -104,16 +104,44 @@ loggers upwards in the scope naming heirarchy. This pattern is explained in
|
||||
detail in the [logback documentation][effective logging level] and applies in
|
||||
the same manner to loggers in this library.
|
||||
|
||||
### LogMessageFormater
|
||||
### LogFormatter
|
||||
|
||||
Both the [ConsoleLogAppender](#ConsoleLogAppender) and
|
||||
[FileLogAppender](#FileLogAppender) can be given a *LogMessageFormatter* to
|
||||
determine how a log message is formatted before being written.
|
||||
Console, file, and std/logging appenders accept a *LogFormatter*. Noncapturing
|
||||
functions such as *formatSimpleTextLog* and *formatJsonStructuredLog* are
|
||||
converted automatically:
|
||||
|
||||
```nim
|
||||
type LogMessageFormatter* = proc (msg: LogMessage): string {.gcsafe.}
|
||||
let appender = initConsoleLogAppender(formatter = formatJsonStructuredLog)
|
||||
```
|
||||
|
||||
For a formatter that captures state, pass value configuration and a
|
||||
noncapturing factory to *initLogFormatter*. The factory creates a separate
|
||||
closure in each thread on first use:
|
||||
|
||||
```nim
|
||||
proc makeFormatter(prefix: string): StatefulLogFormatterFunc =
|
||||
var count = 0
|
||||
result = proc(msg: LogMessage): string {.gcsafe.} =
|
||||
inc count
|
||||
prefix & $count & ": " & msg.msg
|
||||
|
||||
let appender = initConsoleLogAppender(
|
||||
formatter = initLogFormatter("request ", makeFormatter))
|
||||
```
|
||||
|
||||
The factory's input must contain value data, such as strings, sequences of
|
||||
values, or *SharedLogState* handles. Ordinary `ref` values and closures are
|
||||
rejected, including when nested inside other types. The returned closure may
|
||||
capture independently created references and mutable state belonging to its
|
||||
thread. It must not return a closure shared with another thread.
|
||||
|
||||
Each appender clone starts fresh formatter state; configuration refresh can
|
||||
reate new clones. For counters or other data that must persist across
|
||||
refreshes or be shared between threads, pass a *SharedLogState* handle as
|
||||
factory input. To invoke a formatter directly, use
|
||||
`formatter.format(message)`.
|
||||
|
||||
|
||||
## Available Appenders
|
||||
|
||||
### ConsoleLogAppender
|
||||
@@ -122,7 +150,7 @@ Used for writing logs to stdout or stderr.
|
||||
|
||||
```nim
|
||||
proc initConsoleLogAppender*(
|
||||
formatter = formatSimpleTextLog,
|
||||
formatter: LogFormatter = nil, # defaults to formatSimpleTextLog
|
||||
## formatJsonStructuredLog is another useful formatter provided
|
||||
## or you can write your own
|
||||
useStderr = false, ## stdout is used by default
|
||||
@@ -144,7 +172,7 @@ Used for writing logs to files.
|
||||
```nim
|
||||
proc initFileLogAppender*(
|
||||
filePath: string,
|
||||
formatter = formatSimpleTextLog,
|
||||
formatter: LogFormatter = nil, # defaults to formatSimpleTextLog
|
||||
## formatJsonStructuredLog is another useful formatter provided
|
||||
## or you can write your own
|
||||
namespace = "",
|
||||
@@ -178,41 +206,75 @@ messages regardless of whether namespaced\_logging has other appenders by
|
||||
setting `fallbackOnly = false`.
|
||||
|
||||
```nim
|
||||
func initStdLoggingAppender*(
|
||||
proc initStdLoggingAppender*(
|
||||
fallbackOnly = true,
|
||||
formatter = formatForwardedLog,
|
||||
formatter: LogFormatter = nil, # defaults to formatForwardedLog
|
||||
namespace = "",
|
||||
threshold = lvlAll): StdLoggingAppender {.gcsafe.}
|
||||
```
|
||||
|
||||
### CustomLogAppender
|
||||
|
||||
Provides an extension point for custom logging implementations.
|
||||
Provides an extension point for custom logging implementation.
|
||||
|
||||
Use *initCustomLogAppender* for state that can be copied by value. Its callback
|
||||
receives `state: var T` and a *LogMessage*. Each clone owns a separate state
|
||||
value. Ordinary `ref` values, closures, and embedded locks or atomics are
|
||||
rejected, including inside containers and objects.
|
||||
|
||||
For mutable state shared by all workers, put value data in a *SharedLogState*:
|
||||
|
||||
```nim
|
||||
func initCustomLogAppender*[T](
|
||||
state: T, # arbitrary state needed for the appender
|
||||
doLogMessage: CustomLogAppenderFunc[T],
|
||||
# custom log appender implementation
|
||||
namespace = "",
|
||||
threshold = lvlAll): CustomLogAppender[T] {.gcsafe.} =
|
||||
let count = initSharedLogState(0)
|
||||
|
||||
proc countMessage(state: var SharedLogState[int], msg: LogMessage) {.gcsafe, noconv.} =
|
||||
state.withState(proc(value: var int) = inc value)
|
||||
|
||||
var service = initLogService()
|
||||
service.addAppender(initCustomLogAppender(count, countMessage))
|
||||
# Pass service copies to workers as described below.
|
||||
# count.snapshot() reads the current count safely.
|
||||
```
|
||||
|
||||
The `state` field allows you to explicitly pass in any data that is required
|
||||
for the custom functionality.
|
||||
Copies of the handle share the same data and keep it alive. *withState* runs
|
||||
its callback synchronously with exclusive access; *snapshot* returns a value
|
||||
copy made under the same lock. Neither operation allows unsynchronized access
|
||||
to the stored payload. Do not retain a pointer to the callback's argument or
|
||||
call back into the same handle from inside *withState*, since its lock is not
|
||||
recursive.
|
||||
|
||||
> [!CAUTION]
|
||||
> Custom state is currently copied by assignment when an appender is cloned.
|
||||
> Ordinary ORC `ref` values, including references nested in value types, must
|
||||
> not be shared this way across threads. Keeping an original reference alive
|
||||
> does not protect reference counts. A safe custom-state and captured-formatter
|
||||
> ownership API is still pending; use independently owned values or an explicit
|
||||
> shared-state abstraction with a synchronized lifetime and access protocol.
|
||||
The payload must contain value data without ordinary references, closures, or
|
||||
embedded synchronization primitives. Strings and sequences of supported values
|
||||
are accepted. Raw pointers and types with custom copy/destruction hooks remain
|
||||
an explicit caller responsibility: their lifetime and synchronization must be
|
||||
safe across threads. *SharedLogState* does not make a raw pointer's target
|
||||
safe.
|
||||
|
||||
See [testutil][] and the unit tests in [namespaced\_logging][nsl-unit-tests]
|
||||
for a single-threaded example; the test collector's lock does not make its
|
||||
reference-counted state safe to share across threads.
|
||||
The [testutil][] collector demonstrates shared state using value snapshots.
|
||||
*LoggedMessages.get()* returns `seq[RecordedLogMessage]`: timestamps are
|
||||
*Time* values, additional JSON data is serialized text, and exceptions retain
|
||||
name, message, and stacktrace text. Returned snapshots can be inspected and
|
||||
modified independently without sharing the original message's references.
|
||||
|
||||
### FactoryCustomLogAppender
|
||||
|
||||
Provides an extension point for custom logging with state created separately
|
||||
in each logging thread. Use this when state needs ordinary references or
|
||||
closures owned by that thread.
|
||||
|
||||
Call *initFactoryCustomLogAppender(seed, factory, doLogMessage)* during logging
|
||||
setup. The noncapturing factory receives a copyable value seed and creates
|
||||
independently owned state on the first message handled by each clone, in the
|
||||
thread doing the logging. The logging callback receives `state: var T` and a
|
||||
*LogMessage*.
|
||||
|
||||
The seed follows the same ownership restrictions as *CustomLogAppender* state;
|
||||
it may include *SharedLogState* handles. The state returned by the factory may
|
||||
contain independently created references and closures. State lasts for that
|
||||
clone's lifetime; configuration refresh may replace it. A factory that raises
|
||||
is retried on the next message, and errors go to the configured error handler.
|
||||
Do not retain managed references from an incoming *LogMessage* for use in
|
||||
another thread.
|
||||
|
||||
## Notes on Use in Multi-Threaded Applications
|
||||
|
||||
@@ -247,12 +309,13 @@ subsequent logging calls in every thread using that service, without recreating
|
||||
loggers or configuring each worker separately.
|
||||
|
||||
> [!NOTE]
|
||||
> Thread-safety work is still in progress. For now, use the built-in appenders
|
||||
> with default or stateless formatters when sharing a service across threads.
|
||||
> Custom appenders with shared reference-based state and formatters with
|
||||
> captured state are not yet generally safe. Runtime configuration refresh
|
||||
> and delivery of queued background output at application exit also have known
|
||||
> failures. See [the regression tests](tests/README.md) for the remaining work.
|
||||
> Thread-safety work is still in progress. Custom state and captured
|
||||
> formatters must follow the ownership rules above. Runtime configuration
|
||||
> refresh, concurrent local-time timestamp construction, and delivery of queued
|
||||
> background output at application exit still have known failures. Nim 2.2.10
|
||||
> also produces sanitizer reports in its default allocator during some
|
||||
> cross-thread allocation patterns. See
|
||||
> [the regression tests](tests/README.md) for the remaining work.
|
||||
|
||||
|
||||
## Architectural Design
|
||||
@@ -335,10 +398,13 @@ configuration. This internal type is not exposed to library users but manages:
|
||||
detection
|
||||
|
||||
Console and file writers remain process-wide. The shared control block owns
|
||||
appender prototypes; refresh borrows them under the configuration lock and calls
|
||||
`clone()` to create local instances. Clones must not retain ordinary managed
|
||||
references shared across threads. That custom-state and formatter contract is
|
||||
the next part of the ownership work.
|
||||
appender prototypes; refresh borrows them under the configuration lock and
|
||||
calls `clone()` to create local instances. Clones must not retain ordinary
|
||||
managed references shared across threads. Built-in custom appenders copy
|
||||
checked value state or shared-state handles. Factory appenders and formatters
|
||||
copy value seeds and create their managed state lazily within the thread that
|
||||
uses each clone. The state of an already-used clone is never copied into
|
||||
another clone.
|
||||
|
||||
### Thread Safety Model
|
||||
|
||||
|
||||
Reference in new issue
Block a user