Drain background writers before process exit

Serialize writer submission and shutdown, enqueue explicit stop messages after accepted records, and finish the final file batch before joining. Keep drained process-wide channel storage until exit to avoid the Nim 2.2.10 Channel.close mutex teardown defect.

Cover empty records, multiple destinations, joined producers, and concurrent shutdown. Include writer lifecycle sanitizer tests in nimble test and update usage documentation.

Validation: all 32 unit tests, eight writer exit scenarios, and writer lifecycle ThreadSanitizer tests pass. The full suite retains the known configuration refresh, timestamp, and default allocator failures.

tests/README.md

@@ -8,9 +8,9 @@ Run from the repository root:
 | Command | Coverage |
 | --- | --- |
 | `nimble test` | All unit, regression, and ThreadSanitizer tests under ORC with threads enabled |
-| `nimble testUnit` | Existing embedded unit tests |
+| `nimble testUnit` | Embedded unit tests, including writer submission/shutdown coordination |
 | `nimble testRegression` | Custom-state/formatter ownership, handle lifetime, configuration freshness, refresh failure recovery, appender-state cleanup, and writer shutdown |
-| `nimble testThreadSafety` | Separate ThreadSanitizer processes for service ownership, last-owner teardown, reconfiguration, shared state, factory state, formatter closures, collector snapshots, timestamps, and the Nim allocator, plus an independent-service control |
+| `nimble testThreadSafety` | Separate ThreadSanitizer processes for service ownership, last-owner teardown, reconfiguration, shared state, factory state, formatter closures, collector snapshots, timestamps, the Nim allocator, and writer lifecycle, plus an independent-service control |
 
 `nimble test` attempts every suite even if an earlier suite fails to compile or
 run, summarizes the failing suites, and exits unsuccessfully if any suite fails.
@@ -21,7 +21,9 @@ when a nested Nimble command fails to propagate its exit status.
 
 Executables are generated in ignored `tests/bin/`. The shutdown child is built
 with `-d:multithreaded`; `--threads:on` alone does not enable background writers.
-The other tests do not need background I/O to exercise ownership.
+The ownership tests do not need background I/O. The embedded writer lifecycle
+tests call the internal writer functions directly; their sanitizer build is
+`tests/bin/writer_lifecycle`.
 
 The full run and the sanitizer subset require Linux, Bash, GNU `timeout`, and a
 C compiler and runtime supporting `-fsanitize=thread`. The sanitizer instruments
@@ -33,6 +35,7 @@ one scenario after building:
 
 ```sh
 TSAN_OPTIONS=halt_on_error=1:exitcode=66 timeout 30s tests/bin/thread_safety service
+TSAN_OPTIONS=halt_on_error=1:exitcode=66 timeout 30s tests/bin/writer_lifecycle 'Background writer lifecycle::*'
 ```
 
 The worker start barrier ensures argument setup finishes before the concurrent
@@ -82,20 +85,33 @@ the unsafe forms are tested as compilation failures.
 The in-process regression tests are deterministic. The lifetime suite checks
 copying, self-assignment, replacement, moving, container copies, and exporting a
 handle from a local service. Shutdown tests launch
-a fresh process for each destination and check all 10,000 messages after normal
-exit, using isolated temporary paths and a 15-second deadline. They exercise the
-real exit hook without sleeps to hide queued-message loss. Like the race tests,
-the shutdown failure can depend on scheduling; a passing run is not proof of
-thread safety or reliable draining.
+a fresh process for each scenario and check all 10,000 numbered messages after
+normal exit, using isolated temporary paths and a 15-second deadline. Scenarios
+cover stdout, stderr, files, empty formatted messages, multiple file destinations,
+and four joined producers. They verify message content and order within each
+producer and destination, using the real exit hook without sleeps or explicit
+flushes. File batches must preserve blank records and finish their final batch
+before the writer stops.
 
-This suite does not yet force an active producer/channel-close race or exhaust
-ORC cycle-collector interleavings. It provides targeted regressions for ownership,
-configuration, cleanup, timestamps, allocator behavior, and shutdown.
+The embedded lifecycle tests start producers, release them to submit more
+records while the file writer shuts down, and verify every accepted record.
+They also check repeated shutdown, rejection of sends after shutdown, and
+waking the console writer after an empty message. The sanitizer task runs these
+tests as a separate process even when another scenario fails.
+
+Nim 2.2.10's `Channel.close()` destroys its internal mutex while holding it,
+which the lifecycle sanitizer test exposed after the writers had joined.
+Shutdown therefore drains and joins the writers, rejects further submissions,
+and leaves the process-wide channels allocated until process exit. It does not
+call the faulty runtime teardown or suppress its diagnostics.
+
+The suite provides targeted regressions; it does not exhaust thread scheduling
+or ORC cycle-collector interleavings.
 
 The ownership scenarios, lifetime suite, and appender-state cleanup regression
 exercise the new ownership contracts. `isEnabled` freshness, failed-refresh
-recovery, local-time timestamp construction, allocator reports, and writer
-shutdown remain for subsequent work.
+recovery, local-time timestamp construction, and allocator reports remain for
+subsequent work.
 
 The standalone `ownership.nim` regression suite passed AddressSanitizer and
 LeakSanitizer with `--mm:orc -d:useMalloc`. An additional worker-thread build found

tests/regressions.nim

@@ -134,7 +134,8 @@ suite "ORC ownership and configuration regressions":
     check destroyed == 1
 
 suite "Background writer shutdown regressions":
-  for destination in ["file", "console"]:
+  for destination in ["file", "console", "stderr", "file-empty", "console-empty",
+                      "files", "file-parallel", "console-parallel"]:
     test destination & " writer drains accepted messages at normal process exit":
       let directory = createTempDir("namespaced_logging_shutdown_", "")
       let outputPath = directory / "messages.log"
@@ -150,9 +151,25 @@ suite "Background writer shutdown regressions":
       check exitCode == 0
       require fileExists(outputPath)
 
-      let received = readFile(outputPath).splitLines()
-      # Check every record and its order, not just file existence. splitLines
-      # includes one final empty item because each record ends with a newline.
-      check received.len == 10_001
-      for i in 0 ..< min(10_000, received.len - 1):
-        check received[i] == "message " & $i
+      if destination.endsWith("-parallel"):
+        let received = readFile(outputPath).splitLines()
+        require received.len == 10_001
+        var next = [0, 2_500, 5_000, 7_500]
+        for line in received[0 ..< 10_000]:
+          require line.startsWith("message ")
+          let number = parseInt(line[8 .. ^1])
+          require number in 0 ..< 10_000
+          let worker = number div 2_500
+          check number == next[worker]
+          inc next[worker]
+        check next == [2_500, 5_000, 7_500, 10_000]
+      else:
+        var expected: string
+        for i in 0 ..< 10_000:
+          if destination == "file-empty" and i mod 128 == 0:
+            expected.add("\n")
+          expected.add("message " & $i & "\n")
+        check readFile(outputPath) == expected
+        if destination == "files":
+          require fileExists(outputPath & ".second")
+          check readFile(outputPath & ".second") == expected

tests/run_tsan.sh

@@ -2,6 +2,7 @@
 set -u
 
 binary=${1:?Expected the ThreadSanitizer test executable}
+writer_binary=${2:?Expected the writer lifecycle test executable}
 result=0
 for scenario in independent service service-lifetime service-reconfigure custom-state formatter shared-state-lifetime factory-state collector formatter-lifetime timestamp allocator; do
   echo "ThreadSanitizer scenario: $scenario"
@@ -13,4 +14,13 @@ for scenario in independent service service-lifetime service-reconfigure custom-
     result=1
   fi
 done
+
+echo "ThreadSanitizer scenario: writer-lifecycle"
+if TSAN_OPTIONS=halt_on_error=1:exitcode=66 timeout 30s "$writer_binary" 'Background writer lifecycle::*'; then
+  echo "PASS: writer-lifecycle"
+else
+  code=$?
+  echo "FAIL: writer-lifecycle (exit $code)" >&2
+  result=1
+fi
 exit "$result"

tests/writer_child.nim

@@ -1,4 +1,4 @@
-import std/os
+import std/[os, strutils]
 import ../src/namespaced_logging
 
 when not defined(multithreaded):
@@ -6,24 +6,59 @@ when not defined(multithreaded):
 
 proc formatLine(msg: LogMessage): string {.gcsafe.} = msg.msg & "\n"
 
+proc formatOptionalLine(msg: LogMessage): string {.gcsafe.} =
+  if msg.msg.len > 0: msg.msg & "\n" else: ""
+
+proc writeMessages(args: tuple[service: LogService, worker: int]) {.thread.} =
+  let logger = args.service.threadLocalRef().getLogger("shutdown")
+  for i in 0 ..< 2_500:
+    # Keep the timestamp race out of this writer lifecycle test.
+    logger.log(LogMessage(scope: "shutdown", level: lvlInfo,
+      msg: "message " & $(args.worker * 2_500 + i)))
+
 proc main() =
   doAssert paramCount() == 2
+  let destination = paramStr(1)
   var service = initLogService()
-  case paramStr(1)
-  of "file":
+  case destination
+  of "file", "file-parallel", "file-empty":
     service.addAppender(initFileLogAppender(paramStr(2)))
-  of "console":
+  of "files":
+    service.addAppender(initFileLogAppender(paramStr(2), namespace = "first"))
+    service.addAppender(initFileLogAppender(paramStr(2) & ".second", namespace = "second"))
+  of "console", "console-parallel", "console-empty":
     # Keep stdout open through the library's exit hook. Redirecting it to a
     # file also prevents a full parent/child pipe from blocking the writer.
     doAssert open(stdout, paramStr(2), fmWrite)
-    service.addAppender(initConsoleLogAppender(formatter = formatLine))
+    if destination == "console-empty":
+      service.addAppender(initConsoleLogAppender(formatter = formatOptionalLine))
+    else:
+      service.addAppender(initConsoleLogAppender(formatter = formatLine))
+  of "stderr":
+    doAssert open(stderr, paramStr(2), fmWrite)
+    service.addAppender(initConsoleLogAppender(useStderr = true, formatter = formatLine))
   else:
     raise newException(ValueError, "unknown writer destination")
 
-  let local = service.threadLocalRef()
-  let logger = local.getLogger("shutdown")
-  for i in 0 ..< 10_000:
-    logger.info("message " & $i)
+  if destination.endsWith("-parallel"):
+    var workers: array[4, Thread[tuple[service: LogService, worker: int]]]
+    for i in 0 ..< workers.len:
+      createThread(workers[i], writeMessages, (service, i))
+    joinThreads(workers)
+  else:
+    let local = service.threadLocalRef()
+    if destination == "files":
+      let first = local.getLogger("first")
+      let second = local.getLogger("second")
+      for i in 0 ..< 10_000:
+        first.info("message " & $i)
+        second.info("message " & $i)
+    else:
+      let logger = local.getLogger("shutdown")
+      for i in 0 ..< 10_000:
+        if destination.endsWith("-empty") and i mod 128 == 0:
+          logger.info("")
+        logger.info("message " & $i)
   # No sleep or explicit flush: normal exit must preserve accepted messages.
 
 main()