nim-namespaced-logging/README.md

Namespaced Logging for Nim

namespaced_logging is intended to be a high-performance, thread-safe logging framework similar to std/logging with support for namespace-scoped logging similar to log4j or logback for Nim. It has four main motivating features:

• Hierarchical, namespaced logging
• Safe and straightforward to use in multi-threaded applications.
• Native support for structured logging.
• Simple, autoconfigured usage pattern reminiscent of the std/logging interface (not yet implemented)

Getting Started

Install the package via nimble:

# Not yet in official Nim packages. TODO once we've battle-tested it a little
nimble install https://github.com/jdbernard/nim-namespaced-logging

Usage Patterns

Manual Configuration

import namespaced_logging

# Manually creating a LogService. This is an independent logging root fully
# isolated from subsequent LogServices initialized with initLogService
var ls = initLogService()

# Configure logging
ls.addAppender(initConsoleLogAppender())
ls.addAppender(initFileLogAppender("app.log"))
ls.setThreshold("api", lvlWarn)

# Create loggers
let localLogSvc = threadLocalRef(ls)
let apiLogger = localLogSvc.getLogger("api")
let dbLogger = localLogSvc.getLogger("db")

Manual Multithreaded Application

import namespaced_logging

# Main thread setup
var logService = initLogService()
logService.addAppender(initConsoleLogAppender())

var localLogSvc = threadLocalRef(logService) # for use on main thread

# Worker thread function
proc worker(ls: LogService) {.thread.} =
  let localLogSvc = threadLocalRef(ls)
  let logger = localLogSvc.getLogger("worker")

  # Runtime configuration changes
  localLogSvc.setThreshold("worker", lvlDebug)
  logger.debug("Worker configured")

# Safe thread creation
createThread(workerThread, worker, logService)

Dynamic Configuration

# Configuration can change at runtime
proc configureLogging(localLogSvc: ThreadLocalLogService, verbose: bool) =
  if verbose:
    localLogSvc.setRootThreshold(lvlDebug)
    localLogSvc.addAppender(initFileLogAppender("debug.log"))
  else:
    localLogSvc.setRootThreshold(lvlInfo)

# Changes automatically propagate to all threads

Loggers and Appenders

The logging system is composed of two main components: loggers and appenders. Loggers are used to create log events, which are then passed to the appenders. Appenders take log events and write them to some destination, such as the console, a file, or a network socket. Appenders also have a logging level threshold, which determines which log events are acted upon by the appender, and, optionally, a namespace filter, which determines from which loggers the appender accepts log events.

Heirarchical Logging Namespaces

Loggers are organized hierarchically, with the hierarchy defined by the logger scope. A logger with the scope app/service/example is conceptually a child of the logger with the scope app/service. By default, appenders accept log events from all loggers, but this can be restricted by setting a namespace filter on the appender. An appender with a namespace set will accept log events from all loggers with scopes that start with the namespace. For example, an appender with the namespace app will accept log events from the loggers app, app/service, and app/service/example, but not from api/service.

The other impact of the logger heirarchy is in the effective logging level of the logger. An explicit logging level threshold can be set for any scope. Any scope that does not have an explicit inherits its threshold from ancestor loggers upwards in the scope naming heirarchy. This pattern is explained in detail in the logback documentation and applies in the same manner to loggers in this library.

LogMessageFormater

Both the ConsoleLogAppender and FileLogAppender can be given a LogMessageFormatter to determine how a log message is formatted before being written.

type LogMessageFormatter* = proc (msg: LogMessage): string {.gcsafe.}

Available Appenders

ConsoleLogAppender

Used for writing logs to stdout or stderr.

proc initConsoleLogAppender*(
    formatter = formatSimpleTextLog,
      ## formatJsonStructuredLog is another useful formatter provided
      ## or you can write your own
    useStderr = false,  ## stdout is used by default
    namespace = "",     ## appender matches all scopes by default
    threshold = lvlAll  ## and accepts all message levels by default
  ): ConsoleLogAppender {.gcsafe.}

The first time a message is sent to any ConsoleLogAppender, we create a writer thread which writes messages to the specified output in the order they are received, flushing the file handle after each write to enforce an ordering. The ConsoleLogAppender implementation uses a channel to send messages to the writer thread.

FileLogAppender

Used for writing logs to files.

proc initFileLogAppender*(
    filePath: string,
    formatter = formatSimpleTextLog,
      ## formatJsonStructuredLog is another useful formatter provided
      ## or you can write your own
    namespace = "",
    threshold = lvlAll
  ): FileLogAppender {.gcsafe.}

Similar to the ConsoleLogAppender implementation, the first time a message is sent to any FileLogAppender we create a writer thread which writes messages to files associated with the FileLogAppender configured for the current LogService.

namespaced_logging does not currently have built-in logic for file rotation, but it does play nice with external file rotation strategies. We do not hold open file handles. The FileLogAppender attempts to batch messages by destination file, opens the file with mode fmAppend, writes the current batch of log messages, and then closes the file handle. Because of this, it has no problem if another process moves or truncates any of the target log files.

StdLoggingAppender

Provides a fallback to std/logging-based logging. This is primarily intended for use in libraries or other contexts where you want to fall back to std/logging if the application is not using or hasn't configured namespaced_logging.

By default the StdLoggingAppender only logs when no namespaced_logging appenders are configured but it can also be configured to always forward log messages regardless of whether namespaced_logging has other appenders by setting fallbackOnly = false.

func initStdLoggingAppender*(
    fallbackOnly = true,
    formatter = formatForwardedLog,
    namespace = "",
    threshold = lvlAll): StdLoggingAppender {.gcsafe.}

CustomLogAppender

Provides an extension point for custom logging implementations.

func initCustomLogAppender*[T](
    state: T,   # arbitrary state needed for the appender
    doLogMessage: CustomLogAppenderFunc[T],
      # custom log appender implementation
    namespace = "",
    threshold = lvlAll): CustomLogAppender[T] {.gcsafe.} =

The state field allows you to explicitly pass in any data that is required for the custom functionality.

Caution

Custom state is currently copied by assignment when an appender is cloned. Ordinary ORC ref values, including references nested in value types, must not be shared this way across threads. Keeping an original reference alive does not protect reference counts. A safe custom-state and captured-formatter ownership API is still pending; use independently owned values or an explicit shared-state abstraction with a synchronized lifetime and access protocol.

See testutil and the unit tests in namespaced_logging for a single-threaded example; the test collector's lock does not make its reference-counted state safe to share across threads.

Notes on Use in Multi-Threaded Applications

This library is intended to support thread-safe logging in multi-threaded applications, and hopefully to behave more intuitively in a multi-threaded environment than std/logging while presenting a similar API. A key difference is that logging setup can happen separately from thread creation, such as when an HTTP server manages its own worker threads.

The LogService object is the main entry point for the logging system and should be initialized via initLogService during application setup, usually on the main thread. Pass a copy of that LogService to each worker that needs logging. Such copies use the same logging configuration and will be updated when there are changes (e.g. to appenders or logging thresholds). Calling initLogService again creates a separate, independent configuration.

Within each thread, call threadLocalRef to obtain a ThreadLocalLogService, then use its getLogger method to create loggers. Keep those local services and loggers within the thread that created them. If the main thread also logs, create a local service and loggers there too. Appenders configured during setup are available to each thread without repeating that setup in every worker.

Once workers have their own copies of the LogService, the original service can go out of scope and the thread that initialized it can finish. Logging remains available while any service copy, local service, or logger using that service remains alive. Join worker threads before exiting the application so they have an opportunity to finish logging.

You can change logging configuration through a service copy or a local service in any thread. Those changes are intended to take effect automatically on subsequent logging calls in every thread using that service, without recreating loggers or configuring each worker separately.

Note

Thread-safety work is still in progress. For now, use the built-in appenders with default or stateless formatters when sharing a service across threads. Custom appenders with shared reference-based state and formatters with captured state are not yet generally safe. Runtime configuration refresh and delivery of queued background output at application exit also have known failures. See the regression tests for the remaining work.

Architectural Design

Overview

The namespaced logging library attempts to balance performance, safety, and usability in multithreaded environments. The design centers on two key types: LogService and ThreadLocalLogService.

LogService (Value Type)

type LogService* = object
  shared: ptr SharedLogServiceState

The LogService object is intended to support uses cases such as:

• Main thread initialization: a mutable LogService supports all of the configuration functions you would typically need when initializing logging for an application on the main thread.
• Cross-thread communication: Copies of a LogService can be passed between threads while keeping the shared configuration alive. Configuration changes are intended to become visible to each associated ThreadLocalLogService automatically on subsequent logging calls.
• Service composition: independently initialized LogService objects are truly independent and multiple can be created and embedded in larger application contexts.

Tip

The LogService object is the object that is intended to be shared across threads.

ThreadLocalLogService (Reference Type)

type ThreadLocalLogService* = ref object
  serviceHandle: LogService
  configVersion: int
  appenders: seq[LogAppender]
  thresholds: TableRef[string, Level]

threadLocalRef builds a fresh ThreadLocalLogService cache in the calling thread, even when the configuration has not changed. We use reference semantics within the context of a thread so that Logger objects created within the same thread context share the same ThreadLocalLogService reference, avoiding the need to synchronize every Logger individually.

ThreadLocalLogService is no longer an alias for ref LogService. To obtain a shared handle from a local service, use local.service instead of local[]:

let handle = localLogSvc.service
# Pass handle to a worker, which calls handle.threadLocalRef() itself.

Configuration methods remain available on both types. Shared handles contain no cache; ensureFreshness(handle) only validates initialization, and handle.appenders() clones the current shared configuration. Local services retain their per-thread caches and refresh them through ensureFreshness(local).

ThreadLocalLogService is the object that users are expected to interact with during regular operation and support both the configuration functions of LogService and the creation of Logger objects.

Caution

ThreadLocalLogService objects should never be shared outside the context of the thread in which they were initialized.

SharedLogServiceState (Internal)

Under the hood LogService holds a pointer to a manually allocated shared control block that serves as the single source of truth for logging configuration. This internal type is not exposed to library users but manages:

• Shared configuration state: Appenders, thresholds, and root logging level
• Synchronization primitives: Locks and atomic variables for thread coordination
• Ownership: Atomic retain/release and explicit destruction of configuration
• Configuration versioning: Atomic version numbers for efficient change detection

Console and file writers remain process-wide. The shared control block owns appender prototypes; refresh borrows them under the configuration lock and calls clone() to create local instances. Clones must not retain ordinary managed references shared across threads. That custom-state and formatter contract is the next part of the ownership work.

Thread Safety Model

Safe Cross-Thread Pattern

# Main thread setup
var logService = initLogService()
logService.addAppender(initConsoleLogAppender())

# Each copy retains the shared control block atomically.
proc workerThread(ls: LogService) {.thread.} =
  # Convert to thread-local reference for efficient operations
  let tlls = threadLocalRef(ls)
  let logger = tlls.getLogger("worker")
  logger.info("Worker thread started")

createThread(worker, workerThread, logService)

Unsafe Pattern (Avoided by Design)

# DON'T DO THIS - unsafe reference sharing
# ThreadLocalLogService should not be shared across threads
let tlls = threadLocalRef(initLogService())
createThread(worker, someProc, tlls)  # ❌ Potential GC issues

Configuration Synchronization

Atomic Version Checking

The library uses atomic version numbers to efficiently detect configuration changes:

proc ensureFreshness*(ls: ThreadLocalLogService) =
  # Cheap atomic check first
  if ls.configVersion == ls.shared.configVersion.load():
    return  # No changes, return immediately

  # Only acquire lock and copy if versions differ
  withLock ls.shared.lock:
    ls.configVersion = ls.shared.configVersion.load
    # Sync state...

Goals/Motivation:

• Most logging operations skip expensive synchronization so the hot path is fast.
• Propogate changes automatically so all threads see configuration updates.
• Minimize lock contention by only acquiring when configuration changes

Thread-Local Caching

Each thread maintains its own copy of the logging configuration in ThreadLocalLogService:

• Appenders: Thread-local copies created via clone() method
• Thresholds: Complete copy of namespace-to-level mappings
• Version tracking: Local version number for change detection

This caching strategy provides:

• High performance: No locks needed for normal logging operations
• Consistency: All threads eventually see the same configuration
• Isolation: Thread-local state prevents cross-thread interference

Error Handling

Overview

For errors that occur during logging operations, there is a callback-based error handling system designed to attempt to gracefully handle such failures. Since logging is typically a non-critical operation we prioritize application stability over guaranteed log delivery.

Error Handler

The library uses a callback-based error handling pattern where applications can register custom error handlers to be notified when logging operations fail. The error handler receives:

• error: The exception that caused the failure
• msg: A descriptive message providing context about where the error occurred

type ErrorHandlerFunc* = proc(error: ref Exception, msg: string) {.gcsafe, nimcall.}

Default Error Handler

namespaced_logging uses the defaultErrorHandlerFunc if a custom error handler has not been configured. The default handler:

1. Attempts to write to stderr, assuming it is likely to be available and monitored
2. Writes an error message and includes both the exception message and stack trace (not available in release mode).
3. Fails silently if it is unable to write to to stderr.

Configuration

Setting Custom Error Handlers

# During initialization
var logService = initLogService(errorHandler = myCustomErrorHandler)

# Or at runtime on either the LogService...
logService.setErrorHandler(myCustomErrorHandler)

# ... or on a ThreadLocalLogService
var localLogSvc = threadLocalRef(logService)
localLogSvc.setErrorHandler(myCustomErrorHandler)

Disabling Error Reporting

proc silentErrorHandler(err: ref Exception, msg: string) {.gcsafe, nimcall.} =
  discard # Do nothing

logService.setErrorHandler(silentErrorHandler)

Best Practices

Provide Fallbacks

proc robustErrorHandler(err: ref Exception, msg: string) {.gcsafe, nimcall.} =
  # Primary: Send to monitoring system
  if not sendToMonitoring(err, msg):
    # Secondary: Write to dedicated error log
    if not writeToErrorLog(err, msg):
      # Tertiary: Use stderr as last resort
      try:
        stderr.writeLine("LOGGING ERROR [" & msg & "]: " & err.msg)
        stderr.flushFile()
      except: discard

Keep Error Handlers Simple

As much as possible, avoid complex operations that might themselves fail. Don't do heavy operations like database writes, complex network operations, or file system operations that might fail and cause cascading errors.

Testing

Run nimble test for all unit, regression, and ThreadSanitizer ownership tests. It runs every suite even when an earlier suite fails. For focused debugging, use nimble testUnit, nimble testRegression, or nimble testThreadSafety. The regression tests currently expose known defects and are expected to fail until those defects are fixed. See tests/README.md for commands, requirements, and coverage limitations.