nim-namespaced-logging/README.md

# Namespaced Logging for Nim

`namespaced_logging` is intended to be a high-performance, thread-safe logging
framework similar to [std/logging][std-logging] with support for
namespace-scoped logging similar to [log4j][] or [logback][] for Nim. It has
four main motivating features:
- Hierarchical, namespaced logging
- Safe and straightforward to use in multi-threaded applications.
- Native support for structured logging.
- Simple, autoconfigured usage pattern reminiscent of the
  [std/logging][std-logging] interface (*not yet implemented*)

## Getting Started

Install the package via nimble:

```bash
# Not yet in official Nim packages. TODO once we've battle-tested it a little
nimble install https://github.com/jdbernard/nim-namespaced-logging
```

## Usage Patterns

### Manual Configuration
```nim
import namespaced_logging

# Manually creating a LogService. This is an independent logging root fully
# isolated from subsequent LogServices initialized with initLogService
var ls = initLogService()

# Configure logging
ls.addAppender(initConsoleLogAppender())
ls.addAppender(initFileLogAppender("app.log"))
ls.setThreshold("api", lvlWarn)

# Create loggers
let localLogSvc = threadLocalRef(ls)
let apiLogger = localLogSvc.getLogger("api")
let dbLogger = localLogSvc.getLogger("db")
```

### Manual Multithreaded Application
```nim
import namespaced_logging

# Main thread setup
var logService = initLogService()
logService.addAppender(initConsoleLogAppender())

var localLogSvc = threadLocalRef(logService) # for use on main thread

# Worker thread function
proc worker(ls: LogService) {.thread.} =
  let localLogSvc = threadLocalRef(ls)
  let logger = localLogSvc.getLogger("worker")

  # Runtime configuration changes
  localLogSvc.setThreshold("worker", lvlDebug)
  logger.debug("Worker configured")

# Safe thread creation
createThread(workerThread, worker, logService)
```

### Dynamic Configuration
```nim
# Configuration can change at runtime
proc configureLogging(localLogSvc: ThreadLocalLogService, verbose: bool) =
  if verbose:
    localLogSvc.setRootThreshold(lvlDebug)
    localLogSvc.addAppender(initFileLogAppender("debug.log"))
  else:
    localLogSvc.setRootThreshold(lvlInfo)

# Changes automatically propagate to all threads
```

## Loggers and Appenders

The logging system is composed of two main components: loggers and appenders.
Loggers are used to create log events, which are then passed to the appenders.
Appenders take log events and write them to some destination, such as the
console, a file, or a network socket. Appenders also have a logging level
threshold, which determines which log events are acted upon by the appender,
and, optionally, a namespace filter, which determines from which loggers the
appender accepts log events.

### Heirarchical Logging Namespaces

Loggers are organized hierarchically, with the hierarchy defined by the logger
scope. A logger with the scope `app/service/example` is conceptually a child of
the logger with the scope `app/service`. By default, appenders accept log
events from all loggers, but this can be restricted by setting a namespace
filter on the appender. An appender with a namespace set will accept log events
from all loggers with scopes that start with the namespace. For example, an
appender with the namespace `app` will accept log events from the loggers
`app`, `app/service`, and `app/service/example`, but not from `api/service`.

The other impact of the logger heirarchy is in the effective logging level of
the logger. An explicit logging level threshold can be set for any scope. Any
scope that does not have an explicit inherits its threshold from ancestor
loggers upwards in the scope naming heirarchy. This pattern is explained in
detail in the [logback documentation][effective logging level] and applies in
the same manner to loggers in this library.

### LogMessageFormater

Both the [ConsoleLogAppender](#ConsoleLogAppender) and
[FileLogAppender](#FileLogAppender) can be given a *LogMessageFormatter* to
determine how a log message is formatted before being written.

```nim
type LogMessageFormatter* = proc (msg: LogMessage): string {.gcsafe.}
```

## Available Appenders

### ConsoleLogAppender

Used for writing logs to stdout or stderr.

```nim
proc initConsoleLogAppender*(
    formatter = formatSimpleTextLog,
      ## formatJsonStructuredLog is another useful formatter provided
      ## or you can write your own
    useStderr = false,  ## stdout is used by default
    namespace = "",     ## appender matches all scopes by default
    threshold = lvlAll  ## and accepts all message levels by default
  ): ConsoleLogAppender {.gcsafe.}
```

The first time a message is sent to any *ConsoleLogAppender*, we create a
writer thread which writes messages to the specified output in the order they
are received, flushing the file handle after each write to enforce an ordering.
The ConsoleLogAppender implementation uses a channel to send messages to the
writer thread.

### FileLogAppender

Used for writing logs to files.

```nim
proc initFileLogAppender*(
    filePath: string,
    formatter = formatSimpleTextLog,
      ## formatJsonStructuredLog is another useful formatter provided
      ## or you can write your own
    namespace = "",
    threshold = lvlAll
  ): FileLogAppender {.gcsafe.}

```

Similar to the *ConsoleLogAppender* implementation, the first time a message is
sent to any *FileLogAppender* we create a writer thread which writes messages
to files associated with the *FileLogAppender* configured for the current
*LogService*.

`namespaced_logging` does not currently have built-in logic for file
rotation, but it does play nice with external file rotation strategies. We do
not hold open file handles. The *FileLogAppender* attempts to batch messages
by destination file, opens the file with mode `fmAppend`, writes the current
batch of log messages, and then closes the file handle. Because of this, it has
no problem if another process moves or truncates any of the target log files.

### StdLoggingAppender

Provides a fallback to [std/logging][std-logging]-based logging. This is
primarily intended for use in libraries or other contexts where you want to
fall back to std/logging if the application is not using or hasn't configured
namespaced\_logging.

By default the *StdLoggingAppender* only logs when no namespaced\_logging
appenders are configured but it can also be configured to always forward log
messages regardless of whether namespaced\_logging has other appenders by
setting `fallbackOnly = false`.

```nim
func initStdLoggingAppender*(
    fallbackOnly = true,
    formatter = formatForwardedLog,
    namespace = "",
    threshold = lvlAll): StdLoggingAppender {.gcsafe.}
```

### CustomLogAppender

Provides an extension point for custom logging implementations.

```nim
func initCustomLogAppender*[T](
    state: T,   # arbitrary state needed for the appender
    doLogMessage: CustomLogAppenderFunc[T],
      # custom log appender implementation
    namespace = "",
    threshold = lvlAll): CustomLogAppender[T] {.gcsafe.} =
```

The `state` field allows you to explicitly pass in any data that is required
for the custom functionality.

> [!CAUTION]
> Custom state is currently copied by assignment when an appender is cloned.
> Ordinary ORC `ref` values, including references nested in value types, must
> not be shared this way across threads. Keeping an original reference alive
> does not protect reference counts. A safe custom-state and captured-formatter
> ownership API is still pending; use independently owned values or an explicit
> shared-state abstraction with a synchronized lifetime and access protocol.

See [testutil][] and the unit tests in [namespaced\_logging][nsl-unit-tests]
for a single-threaded example; the test collector's lock does not make its
reference-counted state safe to share across threads.


## Notes on Use in Multi-Threaded Applications

This library is intended to support thread-safe logging in multi-threaded
applications, and hopefully to behave more intuitively in a multi-threaded
environment than [std/logging][std-logging] while presenting a similar API. A
key difference is that logging setup can happen separately from thread
creation, such as when an HTTP server manages its own worker threads.

The *LogService* object is the main entry point for the logging system and
should be initialized via *initLogService* during application setup, usually on
the main thread.  Pass a copy of that *LogService* to each worker that needs
logging. Such copies use the same logging configuration and will be updated
when there are changes (e.g. to appenders or logging thresholds). Calling
*initLogService* again creates a separate, independent configuration.

Within each thread, call *threadLocalRef* to obtain a *ThreadLocalLogService*,
then use its *getLogger* method to create loggers. Keep those local services and
loggers within the thread that created them. If the main thread also logs,
create a local service and loggers there too. Appenders configured during setup
are available to each thread without repeating that setup in every worker.

Once workers have their own copies of the *LogService*, the original service
can go out of scope and the thread that initialized it can finish. Logging
remains available while any service copy, local service, or logger using that
service remains alive. Join worker threads before exiting the application so
they have an opportunity to finish logging.

You can change logging configuration through a service copy or a local service
in any thread. Those changes are intended to take effect automatically on
subsequent logging calls in every thread using that service, without recreating
loggers or configuring each worker separately.

> [!NOTE]
> Thread-safety work is still in progress. For now, use the built-in appenders
> with default or stateless formatters when sharing a service across threads.
> Custom appenders with shared reference-based state and formatters with
> captured state are not yet generally safe. Runtime configuration refresh
> and delivery of queued background output at application exit also have known
> failures. See [the regression tests](tests/README.md) for the remaining work.


## Architectural Design

### Overview

The namespaced logging library attempts to balance performance, safety, and
usability in multithreaded environments. The design centers on two key types:
*LogService* and *ThreadLocalLogService*.

#### LogService (Value Type)
```nim
type LogService* = object
  shared: ptr SharedLogServiceState
```

The *LogService* object is intended to support uses cases such as:
- **Main thread initialization**: a mutable *LogService* supports all of the
  configuration functions you would typically need when initializing logging
  for an application on the main thread.
- **Cross-thread communication**: Copies of a *LogService* can be passed
  between threads while keeping the shared configuration alive. Configuration
  changes are intended to become visible to each associated
  *ThreadLocalLogService* automatically on subsequent logging calls.
- **Service composition**: independently initialized *LogService* objects are
  truly independent and multiple can be created and embedded in larger
  application contexts.

> [!TIP]
> The *LogService* object is the object that is intended to be shared across
> threads.

#### ThreadLocalLogService (Reference Type)
```nim
type ThreadLocalLogService* = ref object
  serviceHandle: LogService
  configVersion: int
  appenders: seq[LogAppender]
  thresholds: TableRef[string, Level]
```

*threadLocalRef* builds a fresh *ThreadLocalLogService* cache in the calling
thread, even when the configuration has not changed. We use reference
semantics within the context of a thread so that *Logger* objects created
within the same thread context share the same *ThreadLocalLogService*
reference, avoiding the need to synchronize every *Logger* individually.

*ThreadLocalLogService* is no longer an alias for `ref LogService`. To obtain a
shared handle from a local service, use `local.service` instead of `local[]`:

```nim
let handle = localLogSvc.service
# Pass handle to a worker, which calls handle.threadLocalRef() itself.
```

Configuration methods remain available on both types. Shared handles contain no
cache; `ensureFreshness(handle)` only validates initialization, and
`handle.appenders()` clones the current shared configuration. Local services
retain their per-thread caches and refresh them through `ensureFreshness(local)`.

*ThreadLocalLogService* is the object that users are expected to interact with
during regular operation and support both the configuration functions of
*LogService* and the creation of *Logger* objects.

> [!CAUTION]
> *ThreadLocalLogService* objects should **never** be shared outside the
> context of the thread in which they were initialized.

#### SharedLogServiceState (Internal)

Under the hood *LogService* holds a pointer to a manually allocated shared
control block that serves as the single source of truth for logging
configuration. This internal type is not exposed to library users but manages:

- **Shared configuration state**: Appenders, thresholds, and root logging level
- **Synchronization primitives**: Locks and atomic variables for thread
  coordination
- **Ownership**: Atomic retain/release and explicit destruction of configuration
- **Configuration versioning**: Atomic version numbers for efficient change
  detection

Console and file writers remain process-wide. The shared control block owns
appender prototypes; refresh borrows them under the configuration lock and calls
`clone()` to create local instances. Clones must not retain ordinary managed
references shared across threads. That custom-state and formatter contract is
the next part of the ownership work.

### Thread Safety Model

#### Safe Cross-Thread Pattern
```nim
# Main thread setup
var logService = initLogService()
logService.addAppender(initConsoleLogAppender())

# Each copy retains the shared control block atomically.
proc workerThread(ls: LogService) {.thread.} =
  # Convert to thread-local reference for efficient operations
  let tlls = threadLocalRef(ls)
  let logger = tlls.getLogger("worker")
  logger.info("Worker thread started")

createThread(worker, workerThread, logService)
```

#### Unsafe Pattern (Avoided by Design)
```nim
# DON'T DO THIS - unsafe reference sharing
# ThreadLocalLogService should not be shared across threads
let tlls = threadLocalRef(initLogService())
createThread(worker, someProc, tlls)  # ❌ Potential GC issues
```

### Configuration Synchronization

#### Atomic Version Checking

The library uses atomic version numbers to efficiently detect configuration
changes:

```nim
proc ensureFreshness*(ls: ThreadLocalLogService) =
  # Cheap atomic check first
  if ls.configVersion == ls.shared.configVersion.load():
    return  # No changes, return immediately

  # Only acquire lock and copy if versions differ
  withLock ls.shared.lock:
    ls.configVersion = ls.shared.configVersion.load
    # Sync state...
```

Goals/Motivation:
- Most logging operations skip expensive synchronization so the hot path is
  fast.
- Propogate changes automatically so all threads see configuration updates.
- Minimize lock contention by only acquiring when configuration changes

#### Thread-Local Caching

Each thread maintains its own copy of the logging configuration in
*ThreadLocalLogService*:

- **Appenders**: Thread-local copies created via `clone()` method
- **Thresholds**: Complete copy of namespace-to-level mappings
- **Version tracking**: Local version number for change detection

This caching strategy provides:
- **High performance**: No locks needed for normal logging operations
- **Consistency**: All threads eventually see the same configuration
- **Isolation**: Thread-local state prevents cross-thread interference

## Error Handling

### Overview

For errors that occur during logging operations, there is a callback-based
error handling system designed to attempt to gracefully handle such failures.
Since logging is typically a non-critical operation we prioritize application
stability over guaranteed log delivery.

### Error Handler

The library uses a callback-based error handling pattern where applications can
register custom error handlers to be notified when logging operations fail. The
error handler receives:
- `error`: The exception that caused the failure
- `msg`: A descriptive message providing context about where the error occurred

```nim
type ErrorHandlerFunc* = proc(error: ref Exception, msg: string) {.gcsafe, nimcall.}
```

### Default Error Handler

namespaced\_logging uses the `defaultErrorHandlerFunc` if a custom error
handler has not been configured. The default handler:

1. Attempts to write to stderr, assuming it is likely to be available and monitored
2. Writes an error message and includes both the exception message and stack
   trace (not available in release mode).
3. Fails silently if it is unable to write to to stderr.

### Configuration

#### Setting Custom Error Handlers
```nim
# During initialization
var logService = initLogService(errorHandler = myCustomErrorHandler)

# Or at runtime on either the LogService...
logService.setErrorHandler(myCustomErrorHandler)

# ... or on a ThreadLocalLogService
var localLogSvc = threadLocalRef(logService)
localLogSvc.setErrorHandler(myCustomErrorHandler)
```

#### Disabling Error Reporting
```nim
proc silentErrorHandler(err: ref Exception, msg: string) {.gcsafe, nimcall.} =
  discard # Do nothing

logService.setErrorHandler(silentErrorHandler)
```

### Best Practices

#### Provide Fallbacks

```nim
proc robustErrorHandler(err: ref Exception, msg: string) {.gcsafe, nimcall.} =
  # Primary: Send to monitoring system
  if not sendToMonitoring(err, msg):
    # Secondary: Write to dedicated error log
    if not writeToErrorLog(err, msg):
      # Tertiary: Use stderr as last resort
      try:
        stderr.writeLine("LOGGING ERROR [" & msg & "]: " & err.msg)
        stderr.flushFile()
      except: discard
```

#### Keep Error Handlers Simple

As much as possible, avoid complex operations that might themselves fail.
Don't do heavy operations like database writes, complex network operations, or
file system operations that might fail and cause cascading errors.

## Testing

Run `nimble test` for all unit, regression, and ThreadSanitizer ownership tests.
It runs every suite even when an earlier suite fails. For focused debugging,
use `nimble testUnit`, `nimble testRegression`, or `nimble testThreadSafety`.
The regression tests currently expose known defects and are expected to fail
until those defects are fixed. See [tests/README.md](tests/README.md) for commands,
requirements, and coverage limitations.

[log4j]: https://logging.apache.org/log4j/2.x/
[logback]: https://logback.qos.ch/
[effective logging level]: https://logback.qos.ch/manual/architecture.html#effectiveLevel
[atomic-load]: https://nim-lang.org/docs/atomics.html#load%2CAtomic%5BT%5D%2CMemoryOrder
[std-logging]: https://nim-lang.org/docs/logging.html
[testutil]: /blob/main/src/namespaced_logging/testutil.nim
[nsl-unit-tests]: https://github.com/jdbernard/nim-namespaced-logging/blob/main/src/namespaced_logging.nim#L904