v0.16.1 doc update

.gitignore

@@ -9,6 +9,7 @@ script/
 build/
 *.log
 *.toml
+!config/*.toml
 build.sh
 catalog.txt
 combined.txt

README.md

@@ -6,7 +6,7 @@
     <td>
       <h1>LogWisp</h1>
       <p>
-        <a href="https://golang.org"><img src="https://img.shields.io/badge/Go-1.25-00ADD8?style=flat&logo=go" alt="Go"></a>
+        <a href="https://golang.org"><img src="https://img.shields.io/badge/Go-1.26-00ADD8?style=flat&logo=go" alt="Go"></a>
         <a href="https://opensource.org/licenses/BSD-3-Clause"><img src="https://img.shields.io/badge/License-BSD_3--Clause-blue.svg" alt="License"></a>
         <a href="doc/"><img src="https://img.shields.io/badge/Docs-Available-green.svg" alt="Documentation"></a>
       </p>
@@ -16,78 +16,127 @@
 
 # LogWisp
 
-A high-performance, pipeline-based log transport and processing system built in Go. LogWisp provides flexible log collection, filtering, formatting, and distribution with enterprise-grade security and reliability features.
+A pipeline-based log transport and processing system written in Go. LogWisp
+collects log entries from files, stdin, or other LogWisp nodes; rate-limits,
+filters, and formats them; and distributes them to files, consoles, live network
+streams, or downstream LogWisp nodes.
 
 ## Features
 
-### Core Capabilities
+### Pipeline
-- **Pipeline Architecture**: Independent processing pipelines with source(s) → filter → format → sink(s) flow
-- **Multiple Input Sources**: Directory monitoring, stdin, HTTP, TCP
-- **Flexible Output Sinks**: Console, file, HTTP SSE, TCP streaming, HTTP/TCP forwarding
-- **Real-time Processing**: Sub-millisecond latency with configurable buffering
-- **Hot Configuration Reload**: Update pipelines without service restart
 
-### Data Processing
+- **Independent pipelines**, each `sources → flow → sinks`, running concurrently
-- **Pattern-based Filtering**: Chainable include/exclude filters with regex support
+  in one process
-- **Multiple Formatters**: Raw, JSON, and template-based text formatting
+- **Fan-in and fan-out**: many sources and many sinks per pipeline
-- **Rate Limiting**: Pipeline rate control
+- **Never blocks**: a stalled sink drops its own events and is counted, rather
+  than stalling the pipeline or its sibling sinks
+- **Hot reload** via `SIGHUP`/`SIGUSR1` or a config file watch, with the new
+  configuration validated before the old service is torn down
 
-### Security & Reliability
+### Inputs
-- **Authentication**: mTLS support for HTTPS
-- **TLS Encryption**: TLS 1.2/1.3 support for HTTP connections
-- **Access Control**: IP whitelisting/blacklisting, connection limits
-- **Automatic Reconnection**: Resilient client connections with exponential backoff
-- **File Rotation**: Size-based rotation with retention policies
 
-### Operational Features
+`file` (directory tail with rotation detection and JSON line parsing),
-- **Status Monitoring**: Real-time statistics and health endpoints
+`console` (stdin), `random` (synthetic generator), `null`, and the chain ingest
-- **Signal Handling**: Graceful shutdown and configuration reload via signals
+listeners `tcp_chain` and `http_chain`.
-- **Background Mode**: Daemon operation with proper signal handling
+
-- **Quiet Mode**: Silent operation for automated deployments
+### Outputs
+
+`console`, `file` (rotating with retention), `http` (Server-Sent Events plus a
+JSON status endpoint), `tcp` (broadcast server), `null`, and the chain
+forwarders `tcp_chain` and `http_chain`.
+
+### Processing
+
+- **Filters**: chainable include/exclude RE2 patterns with `or`/`and` logic
+- **Formatters**: `raw`, `txt`, and `json` with selectable sanitizer policies
+- **Rate limiting**: token bucket with an optional per-entry size cap
+- **Heartbeats**: flow-level keep-alive entries that reach every sink
+
+### Chaining
+
+Multi-node topologies over a versioned protocol. Chain links carry the
+**structured entry**, not the formatted text, so a relay can filter and reformat
+as if the entries were local. Entries keep a `node` label identifying their
+origin across any number of hops. Chain sinks reconnect automatically with
+exponential backoff and jitter.
+
+### Transport security
+
+- TLS 1.2/1.3 on every network source and sink, listener and dialer alike
+- Mutual TLS: listeners can require and verify client certificates; dialers can
+  present a client identity
+
+mTLS is currently a CA-wide membership check — any certificate the configured CA
+issued is accepted, and the peer's Common Name is recorded but not used for
+authorization. See [Security](doc/security.md) for the exact boundary and
+[the mTLS authentication plan](doc/mtls-auth-plan.md) for the proposed work.
+Password, token, and SCRAM authentication were removed during the restructure
+and are not currently available.
 
 ## Documentation
 
-Available in `doc/` directory.
+| Document | Contents |
+|----------|----------|
+| [Installation](doc/installation.md) | Building, installing, running as a service |
+| [Architecture](doc/architecture.md) | Component model, data flow, concurrency, back-pressure |
+| [Configuration](doc/configuration.md) | TOML structure, precedence, environment and CLI overrides |
+| [Sources](doc/sources.md) | Every input plugin and its options |
+| [Sinks](doc/sinks.md) | Every output plugin and its options |
+| [Filters](doc/filters.md) | Pattern-based inclusion and exclusion |
+| [Formatters](doc/formatters.md) | Output shaping and sanitization |
+| [Chaining](doc/chaining.md) | Multi-node topologies and the chain wire protocol |
+| [Networking](doc/networking.md) | Listeners, dialers, timeouts, connection limits |
+| [Security](doc/security.md) | TLS and mTLS configuration, threat model, current limits |
+| [mTLS Authentication Plan](doc/mtls-auth-plan.md) | Design for certificate-based authorization |
+| [CLI](doc/cli.md) | Flags, signals, exit codes |
+| [Operations](doc/operations.md) | Running, monitoring, tuning, troubleshooting |
 
-- [Installation Guide](doc/installation.md) - Platform setup and service configuration
+A fully annotated configuration covering every option ships as
-- [Architecture Overview](doc/architecture.md) - System design and component interaction
+[`config/logwisp.toml`](config/logwisp.toml).
-- [Configuration Reference](doc/configuration.md) - TOML structure and configuration methods
-- [Input Sources](doc/sources.md) - Available source types and configurations
-- [Output Sinks](doc/sinks.md) - Sink types and output options
-- [Filters](doc/filters.md) - Pattern-based log filtering
-- [Formatters](doc/formatters.md) - Log formatting and transformation
-- [Security](doc/security.md) - mTLS configurations and access control
-- [Networking](doc/networking.md) - TLS, rate limiting, and network features
-- [Command Line Interface](doc/cli.md) - CLI flags and subcommands
-- [Operations Guide](doc/operations.md) - Running and maintaining LogWisp
 
 ## Quick Start
 
-Install LogWisp and create a basic configuration:
+```bash
+make
+```
 
 ```toml
+# logwisp.toml
 [[pipelines]]
 name = "default"
 
-[[pipelines.sources]]
+[pipelines.flow.format]
-type = "directory"
+type = "json"
-[pipelines.sources.directory]
+sanitizer_policy = "json"
-path = "./"
+
+[[pipelines.plugin_sources]]
+id = "app_logs"
+type = "file"
+[pipelines.plugin_sources.config]
+directory = "/var/log/myapp"
 pattern = "*.log"
 
-[[pipelines.sinks]]
+[[pipelines.plugin_sinks]]
+id = "stdout"
 type = "console"
-[pipelines.sinks.console]
+[pipelines.plugin_sinks.config]
 target = "stdout"
 ```
 
-Run with: `logwisp -c config.toml`
+```bash
+logwisp -c logwisp.toml
+```
+
+Running with no configuration file starts a self-demonstrating pipeline: a
+synthetic generator writing JSON to stdout.
 
 ## System Requirements
 
-- **Operating Systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
+- **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
 - **Architecture**: amd64
-- **Go Version**: 1.25+ (for building from source)
+- **Go**: 1.26+ to build from source
+
+Network sources and sinks bind and dial over IPv4 only.
 
 ## License
 

config/logwisp.toml

@@ -0,0 +1,305 @@
+###############################################################################
+### LogWisp Configuration
+### Default location: ~/.config/logwisp/logwisp.toml
+### Precedence: CLI flags > Environment > File > Defaults
+###
+### Commented values are the built-in defaults unless marked "example".
+### Uncommenting a default is a no-op.
+###
+### NOTE: authentication (password/token/SCRAM), network access control (ACL),
+### http/tcp ingest sources, and http_client/tcp_client sinks were removed in
+### the restructure and are not available in this version. TLS and mTLS ARE
+### available on every network source and sink; see the [...tls] blocks below.
+###
+### Environment overrides are currently read WITHOUT the LOGWISP_ prefix
+### (QUIET, LOGGING_LEVEL, ...). LOGWISP_CONFIG_FILE and LOGWISP_CONFIG_DIR
+### are the exceptions and do carry it. Array-indexed paths such as
+### pipelines.0.name cannot be set from the CLI or environment at all.
+###############################################################################
+
+###############################################################################
+### Global Settings
+###############################################################################
+
+quiet = false                           # Suppress console output
+status_reporter = true                  # Periodic status logging (30s, DEBUG level)
+auto_reload = false                     # Config auto-reload on file change
+
+###############################################################################
+### Logging (LogWisp's internal operational logging)
+###############################################################################
+
+[logging]
+output = "stdout"                       # file|stdout|stderr|split|all|none
+level = "info"                          # debug|info|warn|error
+# format = "txt"                        # raw|txt|json
+# sanitization = ""                     # raw|json|txt|shell (empty = logger default)
+
+# [logging.file]                        # Used when output is "file" or "all"
+# directory = "./log"
+# name = "logwisp"
+# max_size_mb = 100
+# max_total_size_mb = 1000
+# retention_hours = 168.0               # 7 days
+
+## Validated but NOT applied: the console destination comes from `output` above.
+# [logging.console]
+# target = "stdout"                     # stdout|stderr|split
+
+###############################################################################
+### Pipelines
+### Each pipeline: plugin_sources -> flow (rate_limit|filters|format) -> plugin_sinks
+### Names must be unique. 1+ source and 1+ sink required.
+###############################################################################
+
+[[pipelines]]
+name = "default"
+
+###============================================================================
+### Flow (processing between sources and sinks)
+###============================================================================
+
+## policy="pass" short-circuits the size check too: enforcing a size cap needs
+## rate > 0 AND policy = "drop".
+# [pipelines.flow.rate_limit]
+# rate = 0.0                            # Entries/second (0 = limiter disabled)
+# burst = 0.0                           # Burst capacity (defaults to rate)
+# policy = "pass"                       # pass|drop
+# max_entry_size_bytes = 0              # 0 = unlimited
+
+## Filters: sequential include/exclude chain, matched against "<source> <level> <message>"
+# [[pipelines.flow.filters]]
+# type = "include"                      # include|exclude
+# logic = "or"                          # or|and
+# patterns = [".*ERROR.*", ".*WARN.*"]  # example; RE2 syntax
+
+# [pipelines.flow.format]
+# type = "raw"                          # raw|json|txt
+# flags = 0                             # Formatter flags (0 = defaults per type)
+# timestamp_format = ""                 # Go time layout (formatter default if empty)
+# sanitizer_policy = ""                 # raw|json|txt|shell (defaults per type)
+
+## Flow-level heartbeat (fan-out to all sinks, traverses chain links)
+# [pipelines.flow.heartbeat]
+# enabled = false
+# interval_ms = 1000                    # Minimum 100
+# include_timestamp = false
+# include_stats = false
+# format = "txt"                        # txt|json|raw ("comment" is rejected)
+
+###============================================================================
+### TLS (shared shape; applies to every network source and sink)
+###
+### Listeners  (tcp/http sinks, tcp_chain/http_chain sources):
+###   cert_file + key_file are REQUIRED; client_auth + client_ca_file enable mTLS.
+### Dialers    (tcp_chain/http_chain sinks):
+###   ca_file + server_name verify the server; cert_file + key_file present a
+###   client identity for mTLS.
+###
+### enabled = false      Master switch
+### cert_file = ""       Local certificate
+### key_file = ""        Private key for cert_file (set together)
+### client_auth = false  Listener: require and verify a client certificate
+### client_ca_file = ""  Listener: CA bundle verifying client certs
+### ca_file = ""         Dialer: CA bundle verifying the server (empty = system)
+### server_name = ""     Dialer: SNI / name to verify (empty = configured host)
+### insecure_skip_verify = false   Dialer: disable verification (never in prod)
+### min_version = "1.3"  "1.2" or "1.3". No max_version, no cipher_suites.
+###
+### Any certificate signed by client_ca_file is accepted; the peer CN is
+### recorded but not used for authorization. See doc/security.md.
+###============================================================================
+
+###============================================================================
+### Sources (1+ required)
+###============================================================================
+
+## Null source (testing)
+# [[pipelines.plugin_sources]]
+# id = "null_in"
+# type = "null"
+
+[[pipelines.plugin_sources]]
+id = "default_source"
+type = "file"
+[pipelines.plugin_sources.config]
+directory = "./"                        # Directory to monitor (required, not recursive)
+pattern = "*.log"                       # Glob pattern (* and ? only)
+## Tailing an already-open file polls at a fixed 100ms, regardless of this value.
+check_interval_ms = 100                 # Directory rescan interval (min 10)
+
+## Console source (stdin, single instance per pipeline)
+# [[pipelines.plugin_sources]]
+# id = "console_in"
+# type = "console"
+# [pipelines.plugin_sources.config]
+# buffer_size = 1000
+
+## Random source (testing; special=true exercises sanitizer policies)
+# [[pipelines.plugin_sources]]
+# id = "random_in"
+# type = "random"
+# [pipelines.plugin_sources.config]
+# interval_ms = 500
+# jitter_ms = 0                         # Clamped to interval_ms
+# format = "txt"                        # raw|txt|json
+# length = 20
+# special = false
+
+## TCP chain source (stdlib listener; receives NDJSON from upstream tcp_chain sinks)
+## Topology: jail [file source -> tcp_chain sink] -> host [tcp_chain source -> aggregate sink]
+# [[pipelines.plugin_sources]]
+# id = "chain_in"
+# type = "tcp_chain"
+# [pipelines.plugin_sources.config]
+# host = "0.0.0.0"                      # IPv4 only
+# port = 9440                           # Required
+# buffer_size = 1000
+# max_connections = 0                   # 0 = unlimited
+# read_timeout_ms = 0                   # idle deadline, 0 = none
+# hello_timeout_ms = 10000              # Protocol preamble deadline
+# trust_node = true                     # false: label entries by remote address
+# [pipelines.plugin_sources.config.tls]
+# enabled = true                        # example: mTLS listener
+# cert_file = "/etc/logwisp/tls/server.crt"
+# key_file = "/etc/logwisp/tls/server.key"
+# client_auth = true
+# client_ca_file = "/etc/logwisp/tls/client-ca.crt"
+# min_version = "1.3"
+
+## HTTP chain source (stdlib listener; receives NDJSON batches from upstream http_chain sinks)
+# [[pipelines.plugin_sources]]
+# id = "hchain_in"
+# type = "http_chain"
+# [pipelines.plugin_sources.config]
+# host = "0.0.0.0"
+# port = 9441                           # Required
+# ingest_path = "/ingest"               # Must start with "/"
+# buffer_size = 1000
+# max_body_bytes = 8388608              # per-request cap (8 MiB)
+# read_timeout_ms = 30000
+# trust_node = true                     # false: label entries by remote address
+# [pipelines.plugin_sources.config.tls]
+# enabled = true                        # example: mTLS listener
+# cert_file = "/etc/logwisp/tls/server.crt"
+# key_file = "/etc/logwisp/tls/server.key"
+# client_auth = true
+# client_ca_file = "/etc/logwisp/tls/client-ca.crt"
+
+###============================================================================
+### Sinks (1+ required, fan-out)
+###============================================================================
+
+## Null sink (testing)
+# [[pipelines.plugin_sinks]]
+# id = "null_out"
+# type = "null"
+
+[[pipelines.plugin_sinks]]
+id = "default_sink"
+type = "console"
+[pipelines.plugin_sinks.config]
+target = "stdout"                       # stdout|stderr ("split" NOT supported)
+# buffer_size = 1000
+
+## File sink (rotating)
+# [[pipelines.plugin_sinks]]
+# id = "file_out"
+# type = "file"
+# [pipelines.plugin_sinks.config]
+# directory = "./logs"                  # Required
+# name = "output"                       # Required
+# max_size_mb = 100
+# max_total_size_mb = 1000
+# min_disk_free_mb = 0                  # 0 = no floor (only negatives become 100)
+# retention_hours = 168.0
+# buffer_size = 1000
+# flush_interval_ms = 100
+
+## HTTP sink (SSE streaming server + JSON status endpoint; IPv4 clients only)
+## Both endpoints are UNAUTHENTICATED and the stream sends
+## Access-Control-Allow-Origin: *. Bind to a trusted interface.
+# [[pipelines.plugin_sinks]]
+# id = "http_out"
+# type = "http"
+# [pipelines.plugin_sinks.config]
+# host = "0.0.0.0"
+# port = 8081                           # Required
+# stream_path = "/stream"               # Must start with "/"
+# status_path = "/status"               # Must differ from stream_path
+# buffer_size = 1000                    # Sink input queue
+# client_buffer_size = 256              # Per-client send queue
+# write_timeout_ms = 0                  # Per-event deadline, 0 = none
+# max_connections = 0                   # 0 = unlimited
+# [pipelines.plugin_sinks.config.tls]
+# enabled = true                        # example
+# cert_file = "/etc/logwisp/tls/server.crt"
+# key_file = "/etc/logwisp/tls/server.key"
+# client_auth = false
+# client_ca_file = ""
+
+## TCP sink (streaming server, IPv4 clients only)
+# [[pipelines.plugin_sinks]]
+# id = "tcp_out"
+# type = "tcp"
+# [pipelines.plugin_sinks.config]
+# host = "0.0.0.0"
+# port = 9090                           # Required
+# buffer_size = 1000
+# client_buffer_size = 256
+# write_timeout_ms = 5000               # Missed deadline disconnects the client
+# keep_alive = true
+# keep_alive_period_ms = 30000
+# max_connections = 0
+# [pipelines.plugin_sinks.config.tls]
+# enabled = true                        # example
+# cert_file = "/etc/logwisp/tls/server.crt"
+# key_file = "/etc/logwisp/tls/server.key"
+
+## TCP chain sink (stdlib client; forwards to downstream tcp_chain source)
+## Do NOT point a chain sink at a chain source in the SAME pipeline: entries
+## loop back in and amplify without bound.
+# [[pipelines.plugin_sinks]]
+# id = "chain_out"
+# type = "tcp_chain"
+# [pipelines.plugin_sinks.config]
+# host = "10.0.0.1"                     # Required
+# port = 9440                           # Required
+# node = ""                             # origin label, default hostname; preserved across hops
+# buffer_size = 1000
+# dial_timeout_ms = 5000
+# write_timeout_ms = 5000
+# backoff_min_ms = 500
+# backoff_max_ms = 30000
+# keep_alive = true
+# keep_alive_period_ms = 30000
+# [pipelines.plugin_sinks.config.tls]
+# enabled = true                        # example: mTLS dialer
+# ca_file = "/etc/logwisp/tls/ca.crt"
+# server_name = ""
+# insecure_skip_verify = false
+# cert_file = "/etc/logwisp/tls/client.crt"
+# key_file = "/etc/logwisp/tls/client.key"
+# min_version = "1.3"
+
+## HTTP chain sink (stdlib client; batched NDJSON POST to downstream http_chain source)
+# [[pipelines.plugin_sinks]]
+# id = "hchain_out"
+# type = "http_chain"
+# [pipelines.plugin_sinks.config]
+# host = "10.0.0.1"                     # Required
+# port = 9441                           # Required
+# ingest_path = "/ingest"
+# node = ""                             # origin label, default hostname; preserved across hops
+# buffer_size = 1000
+# max_batch_count = 100
+# max_batch_bytes = 1048576
+# flush_interval_ms = 1000
+# request_timeout_ms = 10000            # Covers dial + write + response
+# backoff_min_ms = 500
+# backoff_max_ms = 30000
+# [pipelines.plugin_sinks.config.tls]
+# enabled = true                        # example: mTLS dialer
+# ca_file = "/etc/logwisp/tls/ca.crt"
+# cert_file = "/etc/logwisp/tls/client.crt"
+# key_file = "/etc/logwisp/tls/client.key"

doc/README.md

@@ -1,73 +1,103 @@
-# LogWisp
+# LogWisp Documentation
 
-A pipeline-based log transport and processing system built in Go. LogWisp provides flexible log collection, filtering, formatting, and distribution with security and reliability features.
+LogWisp is a pipeline-based log transport and processing system written in Go.
+It collects log entries from files, stdin, or other LogWisp nodes; rate-limits,
+filters, and formats them; and distributes them to files, consoles, live network
+streams, or downstream LogWisp nodes.
 
-## Features
+## Documentation Map
 
-### Core Capabilities
+| Document | Contents |
-- **Pipeline Architecture**: Independent processing pipelines with source(s) → filter → format → sink(s) flow
+|----------|----------|
-- **Multiple Input Sources**: File monitoring, console (stdin), random log generation, null
+| [Installation](installation.md) | Building, installing, and running as a service |
-- **Flexible Output Sinks**: Console, file, HTTP SSE, TCP streaming, null
+| [Architecture](architecture.md) | Component model, data flow, concurrency, back-pressure |
-- **Real-time Processing**: Sub-millisecond latency with configurable buffering
+| [Configuration](configuration.md) | TOML structure, precedence, environment and CLI overrides |
-- **Hot Configuration Reload**: Update pipelines without service restart
+| [Sources](sources.md) | Every input plugin and its options |
-- **Session Management**: Built-in session tracking for multiple client connections
+| [Sinks](sinks.md) | Every output plugin and its options |
+| [Filters](filters.md) | Pattern-based inclusion and exclusion |
+| [Formatters](formatters.md) | Output shaping and sanitization |
+| [Chaining](chaining.md) | Multi-node topologies and the chain wire protocol |
+| [Networking](networking.md) | Listeners, dialers, timeouts, connection limits |
+| [Security](security.md) | TLS and mTLS configuration, threat model, current limits |
+| [mTLS Authentication Plan](mtls-auth-plan.md) | Design for certificate-based authorization |
+| [CLI](cli.md) | Flags, signals, exit codes |
+| [Operations](operations.md) | Running, monitoring, tuning, troubleshooting |
 
-### Data Processing
+A fully annotated configuration covering every option lives at
-- **Pattern-based Filtering**: Chainable include/exclude filters with regex support
+[`config/logwisp.toml`](../config/logwisp.toml).
-- **Multiple Formatters**: Raw, JSON, and text formatting with integrated sanitizer policies
-- **Rate Limiting**: Pipeline rate controls
-- **Heartbeat Generation**: Flow-level heartbeat events for keep-alives
 
-### Security & Reliability  
+## Capabilities
-- **File Rotation**: Size-based rotation with retention policies
-- **Status Monitoring**: Real-time statistics and health endpoints
-- **Signal Handling**: Graceful shutdown and configuration reload via signals
-- **Background Mode**: Daemon operation with proper signal handling
-- **Quiet Mode**: Silent operation for automated deployments
 
-## Documentation
+### Pipeline
 
-- [Installation Guide](installation.md) - Platform setup and service configuration
+- Independent named pipelines, each `sources → flow → sinks`
-- [Architecture Overview](architecture.md) - System design and component interaction  
+- Fan-in (many sources per pipeline) and fan-out (many sinks per pipeline)
-- [Configuration Reference](configuration.md) - TOML structure and configuration methods
+- Non-blocking sink dispatch: a stalled sink drops its own events and never
-- [Input Sources](sources.md) - Available source types and configurations
+  stalls the pipeline or its sibling sinks
-- [Output Sinks](sinks.md) - Sink types and output options
+- Hot reload of pipeline configuration via `SIGHUP`/`SIGUSR1` or a file watch
-- [Filters](filters.md) - Pattern-based log filtering
+
-- [Formatters](formatters.md) - Log formatting and transformation
+### Inputs
-- [Networking & Security](networking.md) - Network features (Note: TLS and Auth are currently placeholders in the new architecture)
+
-- [Command Line Interface](cli.md) - CLI flags and subcommands
+`file` (directory tail with rotation detection), `console` (stdin),
-- [Operations Guide](operations.md) - Running and maintaining LogWisp
+`random` (synthetic generator), `null`, and the chain ingest listeners
+`tcp_chain` and `http_chain`.
+
+### Outputs
+
+`console`, `file` (rotating), `http` (Server-Sent Events plus a JSON status
+endpoint), `tcp` (broadcast server), `null`, and the chain forwarders
+`tcp_chain` and `http_chain`.
+
+### Processing
+
+- Token-bucket rate limiting with an optional per-entry size cap
+- Chainable include/exclude regex filters with `or`/`and` logic
+- `raw`, `txt`, and `json` formatting with selectable sanitizer policies
+- Optional flow-level heartbeat entries
+
+### Transport security
+
+- TLS 1.2/1.3 on every network source and sink, listener and dialer alike
+- Mutual TLS: listeners can require and verify client certificates; dialers can
+  present a client identity. See [Security](security.md) for what this does and
+  does not currently give you.
 
 ## Quick Start
 
-Install LogWisp and create a basic configuration:
-
 ```toml
 [[pipelines]]
 name = "default"
 
+[pipelines.flow.format]
+type = "json"
+sanitizer_policy = "json"
+
 [[pipelines.plugin_sources]]
-id = "default_source"
+id = "app_logs"
 type = "file"
 [pipelines.plugin_sources.config]
-directory = "./"
+directory = "/var/log/myapp"
 pattern = "*.log"
 
 [[pipelines.plugin_sinks]]
-id = "default_sink"
+id = "stdout"
 type = "console"
 [pipelines.plugin_sinks.config]
 target = "stdout"
 ```
 
-Run with: `logwisp -c config.toml`
+```bash
+logwisp -c config.toml
+```
 
 ## System Requirements
 
-- **Operating Systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
+- **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
 - **Architecture**: amd64
-- **Go Version**: 1.25+ (for building from source)
+- **Go**: 1.26+ to build from source
+
+Network sources and sinks bind and dial over IPv4 only.
 
 ## License
 
-BSD 3-Clause License
+BSD 3-Clause.

doc/architecture.md

@@ -1,171 +1,195 @@
 # Architecture Overview
 
-LogWisp implements a pipeline-based architecture for flexible log processing and distribution.
+LogWisp moves log entries through independent pipelines. Everything else —
+plugins, sessions, TLS, statistics — hangs off that spine.
 
-## Core Concepts
+## Component Hierarchy
-
-### Pipeline Model
-
-Each pipeline operates independently with a source → filter → format → sink flow. Multiple pipelines can run concurrently within a single LogWisp instance, each processing different log streams with unique configurations.
-
-### Component Hierarchy
 
 ```
-Service (Main Process)
+main
-├── Pipeline 1
+└── Service
-│   ├── Plugin Sources (1 or more)
+    ├── Pipeline "app"
+    │   ├── Registry           instance tracking, single-instance enforcement
+    │   ├── Session Manager    per-pipeline connection/session bookkeeping
+    │   ├── Sources[]          plugin instances, keyed by id
     │   ├── Flow
-│   │   ├── Heartbeat Generator (optional)
+    │   │   ├── Rate Limiter   optional, token bucket
-│   │   ├── Rate Limiter (optional)
+    │   │   ├── Filter Chain   optional, ordered
-│   │   ├── Filter Chain (optional)
+    │   │   ├── Formatter      raw | txt | json, with sanitizer
-│   │   └── Formatter (optional)
+    │   │   └── Heartbeat      optional generator
-│   └── Plugin Sinks (1 or more)
+    │   └── Sinks[]            plugin instances, keyed by id
-├── Pipeline 2
+    ├── Pipeline "audit"
-│   └── [Similar structure]
+    │   └── ...
-└── Status Reporter (optional)
+    └── Status Reporter        optional, 30s interval
 ```
 
+Package map:
+
+| Package | Responsibility |
+|---------|----------------|
+| `cmd/logwisp` | Entry point, help, logger bootstrap, signal loop, status reporter |
+| `internal/config` | Typed config schema, loading, top-level validation |
+| `internal/service` | Owns the pipeline set; start, stop, shutdown, global stats |
+| `internal/pipeline` | Pipeline runtime and per-pipeline plugin registry |
+| `internal/flow` | Rate limiter, filter chain invocation, formatting, heartbeat |
+| `internal/filter` | Regex filter and filter chain |
+| `internal/format` | Adapter over `lixenwraith/log` formatter + sanitizer |
+| `internal/source/*` | Source plugins |
+| `internal/sink/*` | Sink plugins |
+| `internal/plugin` | Global factory registry populated by plugin `init()` |
+| `internal/chain` | Chain wire protocol: hello preamble, entry codec, backoff |
+| `internal/tlsx` | The single seam between `TLSOptions` and `crypto/tls` |
+| `internal/session` | Session manager and per-instance proxy |
+| `internal/core` | Shared types (`LogEntry`, `TransportEvent`), capabilities, constants |
+| `internal/tokenbucket` | Rate limiter primitive |
+| `internal/sanitize` | Standalone hex-escaping helpers |
+
+## Plugin Registration
+
+Every plugin registers itself in an `init()` function, and
+`cmd/logwisp/bootstrap.go` blank-imports each package to trigger those
+`init()`s. Adding a plugin therefore means writing the package, calling
+`plugin.RegisterSource` / `plugin.RegisterSink`, and adding one blank import.
+
+Registration may attach metadata. The `console` source declares
+`MaxInstances: 1`, because a process has only one stdin; the per-pipeline
+registry rejects a second instance of any such type.
+
 ## Data Flow
 
-### Processing Stages
+### Entry lifecycle
 
-1. **Source Stage**: Plugin sources monitor inputs and generate log entries
+1. **Source** produces a `core.LogEntry` and publishes it to every subscriber
-2. **Flow - Rate Limiting**: Optional pipeline-level rate control
+   channel it has handed out. Publication is non-blocking: a full subscriber
-3. **Flow - Filtering**: Pattern-based inclusion/exclusion
+   channel increments the source's `dropped_entries` counter.
-4. **Flow - Formatting**: Transform entries to desired output format with sanitization
+2. **Flow** applies, in order: rate limit → filter chain → formatter. A drop at
-5. **Distribution**: Fan-out to multiple plugin sinks
+   any stage ends the entry's life and increments `flow.total_dropped`.
+3. The formatter output becomes a `core.TransportEvent`, which carries both the
+   formatted `Payload` and the original structured `Entry`.
+4. **Dispatch** sends the event to every sink's input channel with a
+   non-blocking send.
 
-### Entry Lifecycle
+`LogEntry` fields:
 
-Log entries flow through the pipeline as `core.LogEntry` structures containing:
+| Field | Purpose |
-- **Time**: Entry timestamp
+|-------|---------|
-- **Level**: Log level (DEBUG, INFO, WARN, ERROR)
+| `Time` | Entry timestamp |
-- **Source**: Origin identifier
+| `Node` | Origin node label for chained topologies; stamped at the first hop, preserved by relays |
-- **Message**: Log content
+| `Source` | Origin identifier within the node (filename, plugin id, …) |
-- **Fields**: Additional metadata (JSON)
+| `Level` | `DEBUG`/`INFO`/`WARN`/`ERROR`/`TRACE`, when detected |
-- **RawSize**: Original entry size
+| `Message` | Log content |
+| `Fields` | Optional structured metadata as raw JSON |
+| `RawSize` | Original byte size, used by the entry-size cap |
 
-### Buffering Strategy
+Carrying `Entry` alongside `Payload` is what makes chain sinks
+format-independent: a `tcp_chain` or `http_chain` sink re-serializes the
+structured entry rather than shipping whatever text the local formatter chose.
 
-Each component maintains internal buffers to handle burst traffic:
+### Back-pressure and drops
-- Sources: Configurable buffer size (default 1000 entries)
-- Sinks: Independent buffers per sink
-- Network components: Additional TCP/HTTP buffers
 
-*"Sink dispatch uses non-blocking sends. When a sink's input buffer is full, the event is dropped for that sink only and counted in pipeline statistics (`total_dropped_by_sink`). The policy is uniform and not configurable; a slow sink does not stall the pipeline or other sinks."*
+There is exactly one drop policy and it is not configurable: **never block**.
 
-## Component Types
+| Stage | Full-buffer behaviour | Counter |
+|-------|----------------------|---------|
+| Source → subscriber | Drop the entry | source `dropped_entries` |
+| Flow | Drop on rate limit, filter, or format error | `flow.total_dropped` |
+| Pipeline → sink | Drop for that sink only | pipeline `total_dropped_by_sink` |
+| TCP/HTTP sink → client queue | Drop for that client only | sink `dropped_writes` |
 
-### Sources (Input)
+The `tcp_chain` sink is the one deliberate exception. It holds a line across
-
+reconnects until it is written or the process shuts down, so a downstream
-- **File Source**: File system directory monitoring with rotation detection
+outage propagates backwards as a full input buffer and surfaces as
-- **Console Source**: Standard input processing (stdin)
+`total_dropped_by_sink` on the pipeline rather than as silent data loss inside
-- **Random Source**: Generates random log entries for testing
+the sink. The `http_chain` sink retries a batch with backoff, and drops it only
-- **Null Source**: Discards logs, used for testing
+on a non-retryable response or on shutdown (`dropped_batches`).
-
-### Sinks (Output)
-
-- **Console Sink**: stdout/stderr output
-- **File Sink**: Rotating file writer
-- **HTTP Sink**: Server-Sent Events (SSE) streaming
-- **TCP Sink**: TCP server for client connections
-- **Null Sink**: Discards all received events
-
-### Processing Components
-
-- **Rate Limiter**: Token bucket algorithm for flow control
-- **Filter Chain**: Sequential pattern matching
-- **Formatters**: Raw, JSON, or text transformation with sanitizer policies
 
 ## Concurrency Model
 
-### Goroutine Architecture
+- One goroutine per source drains that source's subscription and feeds the flow.
+- The flow's formatter holds a mutex; the underlying formatter reuses an
+  internal buffer and is not goroutine-safe.
+- Each network sink runs one broadcast/broker goroutine plus, per connection, a
+  writer goroutine (and for TCP, a reader goroutine that exists only to detect
+  disconnects and refresh session activity).
+- Chain sinks run a single run-loop goroutine that exclusively owns the
+  connection or the pending batch, so no locking is needed around either.
+- Statistics are atomics; configuration and registries use RW mutexes;
+  shutdown is context cancellation plus wait groups.
 
-- Each source runs in dedicated goroutines for monitoring
+### Shutdown ordering
-- Sinks operate independently with their own processing loops
-- Network listeners use optimized event loops (gnet for TCP)
-- Pipeline processing uses channel-based communication
 
-### Synchronization
+`Pipeline.Stop` is deliberately ordered so in-flight data drains:
 
-- Atomic counters for statistics
+1. Stop all sources concurrently; each closes its subscriber channels.
-- Read-write mutexes for configuration access
+2. Wait for the run loop, which ends when every subscription channel closes.
-- Context-based cancellation for graceful shutdown
+3. Stop all sinks concurrently.
-- Wait groups for coordinated startup/shutdown
 
 ## Network Architecture
 
-### Connection Patterns
+All listeners bind `tcp4` and all dialers dial `tcp4`. IPv6 clients cannot
+connect; this is deliberate, not an oversight.
 
-**Chaining Design**:
+| Plugin | Role | Protocol |
-- Future plan
+|--------|------|----------|
+| `tcp` sink | Listener | Raw broadcast of formatted payloads |
+| `http` sink | Listener | HTTP/1.1 SSE; HTTP/2 negotiated via ALPN when TLS is on |
+| `tcp_chain` source | Listener | Chain protocol, persistent NDJSON stream |
+| `http_chain` source | Listener | Chain protocol, NDJSON batches over POST |
+| `tcp_chain` sink | Dialer | Chain protocol, persistent stream, auto-reconnect |
+| `http_chain` sink | Dialer | Chain protocol, batched POST with retry |
 
-**Monitoring Design**:
+TLS is built in exactly one place, `internal/tlsx`, which exposes
-- TCP Sink: Debugging interface
+`Server(opts)` for listeners and `Client(opts, host)` for dialers. See
-- HTTP Sink: Browser-based live monitoring
+[Security](security.md).
 
-### Protocol Support
+## Sessions
 
-- HTTP/1.1 and HTTP/2 for HTTP connections
+Each pipeline owns a `session.Manager`. Plugins receive a `session.Proxy`
-- Raw TCP connections
+scoped to their instance id, so one plugin cannot see or remove another's
+sessions. A session records the remote address, creation and last-activity
+timestamps, and metadata — including `tls` and `tls_peer_cn` for TLS peers.
+
+Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The
+HTTP sink's broker treats a vanished session as an eviction signal and closes
+the corresponding SSE client.
+
+Session metadata is currently bookkeeping only: nothing in the pipeline makes
+an authorization decision from it. Closing that gap is the subject of the
+[mTLS authentication plan](mtls-auth-plan.md).
+
+## Configuration Reload
+
+Reload (signal or file watch) rebuilds the entire service:
+
+1. Re-read the config through the config manager.
+2. Build a **new** service from it. If construction fails, the old service keeps
+   running untouched.
+3. Shut the old service down, start the new one, and restart the status
+   reporter if it is enabled.
+
+Because this is a full rebuild, listening sockets close and reopen and all
+clients are disconnected. Application logging is configured once at startup and
+is **not** re-applied on reload.
 
 ## Resource Management
 
-### Memory Management
+- Every buffer is bounded; the drop-not-block policy keeps memory flat under
+  load.
+- Network sinks and chain sources accept a `max_connections` cap. Admission is
+  a load-then-check, so a burst can over-admit by roughly one connection.
+- Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an
+  oversized line is a protocol violation and terminates the connection.
+- The `http_chain` source caps each request body at `max_body_bytes`.
+- File sinks rotate on size, cap total rotated size, and honour a retention
+  window.
 
-- Bounded buffers prevent unbounded growth
+## Performance Notes
-- Automatic garbage collection via Go runtime
-- Connection limits prevent resource exhaustion
 
-### File Management
+- In-memory entry processing is sub-millisecond; the formatter mutex is the
-
+  only shared serialization point in the hot path.
-- Automatic rotation based on size thresholds
+- File tailing detects new content within roughly 100 ms (fixed poll), while
-- Retention policies for old log files
+  `check_interval_ms` governs how quickly a *newly created* file is noticed.
-- Minimum disk space checks before writing
+- `http_chain` trades latency for efficiency: entries wait up to
-
+  `flush_interval_ms` (default 1 s) before a batch is sent.
-### Connection Management
+- Scale out with more pipelines per process, more sinks per pipeline, or more
-
+  nodes chained together.
-- HTTP sink silenty drops IPv6 connections (deliberate IPv4-only enforcement)
-
-*Note:* Placeholder; below features are removed in restructuring and will be added in the future release.
-
-- Per-IP connection limits
-- Global connection caps
-- Automatic reconnection with exponential backoff
-- Keep-alive for persistent connections
-
-## Reliability Features
-
-### Fault Tolerance
-
-- Panic recovery in pipeline processing
-- Independent pipeline operation
-- Sink failure isolation
-
-### Data Integrity
-
-- Entry validation at ingestion
-- Size limits for entries and batches
-- Duplicate detection in file monitoring
-- Position tracking for file reads
-
-## Performance Characteristics
-
-### Throughput
-
-- Pipeline rate limiting: Configurable (default 1000 entries/second)
-- Network throughput: Limited by network and sink capacity
-- File monitoring: Sub-second detection (default 100ms interval)
-
-### Latency
-
-- Entry processing: Sub-millisecond in-memory
-- Network forwarding: Depends on batch configuration
-- File detection: Configurable check interval
-
-### Scalability
-
-- Horizontal: Multiple LogWisp instances with different configurations
-- Vertical: Multiple pipelines per instance
-- Fan-out: Multiple sinks per pipeline
-- Fan-in: Multiple sources per pipeline

doc/chaining.md

@@ -0,0 +1,212 @@
+# Chaining
+
+Chaining links LogWisp nodes together. An edge node forwards its entries to a
+relay or collector, which can filter, reformat, fan out, or forward them again.
+Unlike the `tcp` and `http` sinks — which emit *formatted text* for humans and
+generic clients — chain links carry the **structured entry**, so downstream
+nodes can filter and reformat as if the entries were local.
+
+## Topology
+
+```
+  edge-01                       relay                        consumers
+  ┌───────────────┐             ┌────────────────────┐       ┌──────────────┐
+  │ file source   │             │ tcp_chain source   │       │ browser (SSE)│
+  │      ↓        │  TCP/TLS    │        ↓           │  ───► │ nc / telnet  │
+  │ tcp_chain sink├────────────►│      flow          │       │ archive file │
+  └───────────────┘   :15801    │        ↓           │       └──────────────┘
+                                │ http sink, tcp sink│
+  edge-02                       │ file sink          │
+  ┌───────────────┐             │ http_chain sink ───┼──►  upstream collector
+  │ file source   │  HTTP/TLS   │                    │
+  │      ↓        ├────────────►│ http_chain source  │
+  │ http_chain sink│   :15802   └────────────────────┘
+  └───────────────┘
+```
+
+Both chain sources can feed a single pipeline (fan-in) whose sinks then fan the
+merged stream out. `test/chain-test.sh` builds the two-independent-pipelines
+variant; `test/chain-aggregate-test.sh` builds the fan-in variant.
+
+## Node Identity
+
+Chained entries carry a `node` label identifying where they originated.
+
+- A chain **sink** stamps `node` on any entry that does not already have one.
+  The label comes from the `node` option, defaulting to `os.Hostname()`.
+- A chain **source** either honours the sender's label or overrides it,
+  according to `trust_node`:
+
+| `trust_node` | Behaviour |
+|--------------|-----------|
+| `true` (default) | Keep the label the sender declared; fall back to the remote address when absent |
+| `false` | Always overwrite with the sender's remote address |
+
+Relays preserve `node`, so a label survives any number of hops and identifies
+the original producer rather than the last relay.
+
+Formatters render node identity as a syslog-style prefix on the source field:
+`edge-01/app.log`. In JSON output the node therefore appears inside the source
+field, not as a separate top-level key.
+
+> `trust_node = true` means an authenticated peer can claim **any** node label,
+> including one belonging to another host. On an untrusted network use
+> `trust_node = false`, or read the
+> [mTLS authentication plan](mtls-auth-plan.md), which proposes binding the
+> label to the peer's certificate identity.
+
+## Wire Protocol
+
+Protocol version: **1**. Both transports carry the same canonical entry
+encoding, and differ only in how the preamble and framing are expressed.
+
+### TCP transport
+
+A persistent connection carrying newline-delimited JSON.
+
+1. The dialer connects and, under TLS, completes the handshake.
+2. The dialer immediately writes the hello preamble as one JSON line:
+
+   ```json
+   {"logwisp":1,"node":"edge-01"}
+   ```
+
+3. The listener reads that line within `hello_timeout_ms` and rejects the
+   connection if it is malformed or declares a different protocol version.
+4. Every subsequent line is one JSON-encoded `LogEntry`.
+
+Line size is bounded at 1 MiB. An oversized line is a protocol violation and
+terminates the connection, because the scanner cannot resynchronize afterwards.
+
+### HTTP transport
+
+Batches of NDJSON delivered by `POST`, with the preamble expressed as headers.
+
+| Header | Direction | Meaning |
+|--------|-----------|---------|
+| `X-Logwisp-Protocol` | request | Protocol version; must be `1` |
+| `X-Logwisp-Node` | request | Origin node label |
+| `Content-Type` | request | `application/x-ndjson` |
+| `X-Logwisp-Accepted` | response | Number of entries ingested |
+
+Responses: `204` on success, `400` for a bad protocol version or a malformed
+body, `413` when the body cap is exceeded, `405` for a non-`POST` method.
+
+### Entry encoding
+
+```json
+{
+  "time": "2026-01-02T15:04:05.123456789Z",
+  "node": "edge-01",
+  "source": "app.log",
+  "level": "ERROR",
+  "message": "connection refused",
+  "fields": {"attempt": 3}
+}
+```
+
+`node`, `level`, and `fields` are omitted when empty. A missing `time` is filled
+in at ingest.
+
+## Delivery Semantics
+
+| Transport | Guarantee | Failure behaviour |
+|-----------|-----------|-------------------|
+| `tcp_chain` | Per-line, held across reconnects | Retries with exponential backoff plus ±20 % jitter until written or shutdown; back-pressure appears upstream as `total_dropped_by_sink` |
+| `http_chain` | At-least-once per batch | Retries transport errors, `408`, `429`, `5xx`; drops on any other non-2xx (`dropped_batches`) |
+
+`http_chain` batches can be delivered twice when a successful request's response
+is lost. There is no de-duplication downstream; design your consumers to
+tolerate it, or use `tcp_chain` where each line is written once per successful
+write.
+
+Neither transport persists anything to disk. Entries buffered in memory during
+an outage are lost if the process exits.
+
+## Worked Example
+
+**Edge node** — tail files, forward over mTLS:
+
+```toml
+[[pipelines]]
+name = "edge"
+
+[[pipelines.plugin_sources]]
+id = "app"
+type = "file"
+[pipelines.plugin_sources.config]
+directory = "/var/log/myapp"
+pattern = "*.log"
+
+[[pipelines.plugin_sinks]]
+id = "forward"
+type = "tcp_chain"
+[pipelines.plugin_sinks.config]
+host = "relay.internal"
+port = 15801
+node = "edge-01"
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+ca_file   = "/etc/logwisp/tls/ca.crt"
+cert_file = "/etc/logwisp/tls/edge-01.crt"
+key_file  = "/etc/logwisp/tls/edge-01.key"
+```
+
+**Relay** — ingest, keep errors only, archive and stream:
+
+```toml
+[[pipelines]]
+name = "relay"
+
+[[pipelines.flow.filters]]
+type = "include"
+patterns = ["ERROR", "FATAL"]
+
+[pipelines.flow.format]
+type = "json"
+sanitizer_policy = "json"
+
+[[pipelines.plugin_sources]]
+id = "ingest"
+type = "tcp_chain"
+[pipelines.plugin_sources.config]
+host = "0.0.0.0"
+port = 15801
+trust_node = true
+[pipelines.plugin_sources.config.tls]
+enabled        = true
+cert_file      = "/etc/logwisp/tls/relay.crt"
+key_file       = "/etc/logwisp/tls/relay.key"
+client_auth    = true
+client_ca_file = "/etc/logwisp/tls/ca.crt"
+
+[[pipelines.plugin_sinks]]
+id = "archive"
+type = "file"
+[pipelines.plugin_sinks.config]
+directory = "/var/log/logwisp"
+name = "errors"
+
+[[pipelines.plugin_sinks]]
+id = "live"
+type = "http"
+[pipelines.plugin_sinks.config]
+host = "127.0.0.1"
+port = 8080
+```
+
+## Operational Notes
+
+- **Formatting is a relay decision.** Because chain links carry structured
+  entries, the edge node's `flow.format` affects only its own local sinks. Set
+  the output shape on the node that owns the human-facing sink.
+- **Filtering early saves bandwidth.** A filter on the edge drops entries before
+  they cross the network; a filter on the relay is easier to change centrally.
+- **Rate limits are per pipeline.** An edge limit protects the link; a relay
+  limit protects the relay from a noisy edge.
+- **Heartbeats traverse chain links** as ordinary structured entries and keep
+  otherwise-idle links and their sessions warm.
+- **Ports** used by the bundled test scripts: `15801` tcp_chain ingest, `15802`
+  http_chain ingest, `15803` tcp sink, `15804` http sink.
+- **Use `127.0.0.1`, not `localhost`**, when testing locally: all listeners and
+  dialers are IPv4-only, and `localhost` may resolve to `::1`.

doc/cli.md

@@ -1,196 +1,184 @@
 # Command Line Interface
 
-LogWisp CLI reference for commands and options.
-
-## Synopsis
-
-```bash
-logwisp [command] [options]
-logwisp [options]
 ```
-
+logwisp [options]
-## Commands
+logwisp help | -h | --help
-
-### Main Commands
-
-| Command | Description |
-|---------|-------------|
-| `--version` | Display version information |
-| `--help` | Show help information |
-
-### version Command
-
-Display version information.
-
-```bash
-logwisp version
-logwisp -v
 logwisp --version
 ```
 
-Output includes:
+LogWisp has no subcommands. Earlier releases shipped `logwisp auth` and
-- Version number
+`logwisp tls` for credential and certificate generation; both were removed
-- Build date
+during the restructure. Use `openssl` or your PKI tooling instead — see
-- Git commit hash
+[Security](security.md).
-- Go version
 
-## Global Options
+## Options
 
-### Configuration Options
+Any scalar configuration key is settable as a flag using its TOML path:
+
+```
+--<path>=<value>        e.g. --logging.level=debug
+--<path> <value>        e.g. --logging.level debug
+--<path>                bare flag, means true
+```
+
+### Common
 
 | Flag | Description | Default |
 |------|-------------|---------|
-| `-c, --config` | Configuration file path | `./logwisp.toml` |
+| `-c <path>` | Configuration file | `./logwisp.toml` |
-| `-q, --quiet` | Suppress console output | false |
+| `--config=<path>` | Configuration file (equals form only) | `./logwisp.toml` |
-| `--status-reporter` | Status logging | true |
+| `--quiet` | Suppress all application output | `false` |
-| `--auto-reload` | Enable config hot reload | false |
+| `--status_reporter=<bool>` | Periodic status logging | `true` |
+| `--auto_reload=<bool>` | Reload config when the file changes | `false` |
+| `--version` | Print version and exit | — |
+| `-h`, `--help`, `help` | Print usage and exit | — |
 
-### Logging Options
+> `--config <path>` with a space is not recognized. The path resolver
+> understands `-c <path>` and `--config=<path>` only; the space form is treated
+> as an unknown flag, warned about, and ignored, after which LogWisp silently
+> falls back to `./logwisp.toml`.
+>
+> `-c` as the final argument, with no path after it, crashes with an index
+> panic rather than reporting a usage error.
 
-| Flag | Description | Values |
+### Logging
-|------|-------------|--------|
-| `--logging.output` | Log output mode | file, stdout, stderr, split, all, none |
-| `--logging.level` | Log level | debug, info, warn, error |
-| `--logging.file.directory` | Log directory | Path |
-| `--logging.file.name` | Log filename | String |
-| `--logging.file.max_size_mb` | Max file size | Integer |
-| `--logging.file.max_total_size_mb` | Total size limit | Integer |
-| `--logging.file.retention_hours` | Retention period | Float |
-| `--logging.console.target` | Console target | stdout, stderr, split |
-| `--logging.console.format` | Output format | txt, json |
 
-### Pipeline Options
+| Flag | Values |
+|------|--------|
+| `--logging.output` | `file`, `stdout`, `stderr`, `split`, `all`, `none` |
+| `--logging.level` | `debug`, `info`, `warn`, `error` |
+| `--logging.format` | `raw`, `txt`, `json` |
+| `--logging.sanitization` | `raw`, `json`, `txt`, `shell` |
+| `--logging.file.directory` | path |
+| `--logging.file.name` | string |
+| `--logging.file.max_size_mb` | integer |
+| `--logging.file.max_total_size_mb` | integer |
+| `--logging.file.retention_hours` | float |
 
-Configure pipelines via CLI (N = array index, 0-based).
+`--logging.console.target` is accepted but has no effect; the console
+destination is derived from `--logging.output`.
 
-**Pipeline Configuration:**
+### Pipelines
 
-| Flag | Description |
+Pipelines, sources, sinks, and filters **cannot** be configured from the command
-|------|-------------|
+line. Array-indexed paths such as `--pipelines.0.name=app` or
-| `--pipelines.N.name` | Pipeline name |
+`--pipelines.0.plugin_sinks.0.type=null` are reported as unrecognized and
-| `--pipelines.N.plugin_sources.N.type` | Source type |
+ignored:
-| `--pipelines.N.flow.filters.N.type` | Filter type |
-| `--pipelines.N.plugin_sinks.N.type` | Sink type |
 
-## Flag Formats
+```
-
+Warning: unrecognized flags ignored: [pipelines.0.name]
-### Boolean Flags
-
-```bash
-logwisp --quiet
-logwisp --quiet=true
-logwisp --pipelines.0.plugin_sources.0.type=console
 ```
 
-### String Flags
+Use a configuration file. Older documentation described CLI pipeline overrides
-
+that the current loader does not implement.
-```bash
-logwisp --config /etc/logwisp/config.toml
-logwisp -c config.toml
-```
-
-### Nested Configuration
-
-```bash
-logwisp --logging.level=debug
-logwisp --pipelines.0.name=myapp
-logwisp --pipelines.0.sources.0.type=console
-```
-
-### Array Values (JSON)
-
-```bash
-logwisp --pipelines.0.flow.filters.0.patterns='["ERROR","WARN"]'
-```
 
 ## Environment Variables
 
-All flags can be set via environment:
+Configuration paths map to environment variables by replacing `.` with `_` and
+uppercasing:
 
 ```bash
-export LOGWISP_QUIET=true
+export QUIET=true
-export LOGWISP_LOGGING_LEVEL=debug
+export LOGGING_LEVEL=debug
-export LOGWISP_PIPELINES_0_NAME=myapp
+export LOGGING_FILE_DIRECTORY=/var/log/logwisp
 ```
 
-## Configuration Precedence
+> The `LOGWISP_` prefix is **not** currently applied to these — see
+> [Configuration](configuration.md#environment-variables). Bare names like
+> `QUIET` are what LogWisp actually reads, which is worth knowing both to make
+> overrides work and to avoid accidental collisions.
 
-1. Command-line flags (highest)
+The two variables that do carry the prefix are read directly by the path
+resolver:
+
+| Variable | Effect |
+|----------|--------|
+| `LOGWISP_CONFIG_FILE` | Configuration file path; joined onto `LOGWISP_CONFIG_DIR` when both are set |
+| `LOGWISP_CONFIG_DIR` | Configuration directory; alone, implies `<dir>/logwisp.toml` |
+
+As with flags, array elements cannot be set this way.
+
+## Precedence
+
+1. Command-line flags
 2. Environment variables
 3. Configuration file
-4. Built-in defaults (lowest)
+4. Built-in defaults
 
-## Exit Codes
+## Signals
-
-| Code | Description |
-|------|-------------|
-| 0 | Success |
-| 1 | General error |
-| 2 | Configuration file not found |
-| 137 | SIGKILL received |
-
-## Signal Handling
 
 | Signal | Action |
 |--------|--------|
-| SIGINT (Ctrl+C) | Graceful shutdown |
+| `SIGINT` | Graceful shutdown |
-| SIGTERM | Graceful shutdown |
+| `SIGTERM` | Graceful shutdown |
-| SIGHUP | Reload configuration |
+| `SIGHUP` | Reload configuration |
-| SIGUSR1 | Reload configuration |
+| `SIGUSR1` | Reload configuration |
-| SIGKILL | Immediate termination |
+
+`SIGHUP` is ignored during startup, before the signal handler is installed, so
+LogWisp survives a terminal hang-up like `nohup`. Once running, it triggers a
+reload rather than terminating.
+
+Reload rebuilds the whole service. A configuration error leaves the running
+service untouched; see [Configuration](configuration.md#hot-reload).
+
+## Exit Codes
+
+| Code | Meaning |
+|------|---------|
+| `0` | Clean shutdown, or `--version` / `--help` |
+| `1` | General error: config load or validation failure, logger init failure, service bootstrap failure |
+| `2` | Explicitly requested configuration file not found |
+
+Exit code 2 applies only when the file was named explicitly (`-c`,
+`--config=`, or the `LOGWISP_CONFIG_*` variables). A missing discovered default
+is not an error, and LogWisp starts on built-in defaults.
+
+## Built-in Defaults
+
+With no configuration file present, LogWisp runs one pipeline named
+`default_pipeline`: a `random` source with `special = true`, JSON formatting,
+a rate limit of 5 entries/second with a burst of 10 and `policy = "drop"`, and a
+`console` sink on stdout. It is a self-demonstrating idle mode, not a useful
+production configuration.
+
+Note that as soon as your file defines `[[pipelines]]`, that entire default
+pipeline — rate limit included — is replaced rather than merged.
 
 ## Usage Patterns
 
-### Development Mode
+**Development**
 
 ```bash
-# Verbose logging to console
+# verbose, everything to stderr
-logwisp --logging.output=stderr --logging.level=debug
+logwisp -c dev.toml --logging.output=stderr --logging.level=debug
 
-# Quick test with stdin
+# no config at all: synthetic generator to stdout
-logwisp --pipelines.0.plugin_sources.0.type=console --pipelines.0.plugin_sinks.0.type=console
+logwisp
 ```
 
-### Production Deployment
+**Configuration check**
 
 ```bash
-# Background with file logging
+# starts the service; a config error exits non-zero before any pipeline runs
-logwisp --background --config /etc/logwisp/prod.toml --logging.output=file
+logwisp -c /etc/logwisp/logwisp.toml --logging.level=debug
-
-# Systemd service
-ExecStart=/usr/local/bin/logwisp --config /etc/logwisp/config.toml
 ```
 
-### Debugging
+There is no dry-run or validate-only mode. The closest approximation is starting
+with debug logging and stopping once the pipelines report as started.
+
+**Production**
 
 ```bash
-# Check configuration
+logwisp -c /etc/logwisp/logwisp.toml --logging.output=file
-logwisp --config test.toml --logging.level=debug --disable-status-reporter
-
-# Dry run (verify config only)
-logwisp --config test.toml --quiet
 ```
 
-## Help System
+Run under a supervisor (systemd, rc.d) rather than backgrounding it — there is
+no `--background` flag; earlier releases had one and it was removed. See
+[Installation](installation.md).
 
-### General Help
+**Reload**
 
 ```bash
-logwisp --help
+kill -HUP  $(pidof logwisp)
-logwisp -h
+kill -USR1 $(pidof logwisp)
-logwisp help
 ```
-
-## Special Flags
-
-### Internal Flags
-
-These flags are for internal use:
-- `--background-daemon`: Child process indicator
-- `--config-save-on-exit`: Save config on shutdown
-
-### Hidden Behaviors
-
-- SIGHUP ignored ignored during startup (after startup triggers config reload)
-- Automatic panic recovery in pipelines
-- Resource cleanup on shutdown

doc/configuration.md

@@ -1,41 +1,64 @@
 # Configuration Reference
 
-LogWisp configuration uses TOML format with flexible override mechanisms.
+LogWisp is configured with TOML. A complete annotated file listing every option
+and its default ships as [`config/logwisp.toml`](../config/logwisp.toml).
 
 ## Configuration Precedence
 
-Configuration sources are evaluated in order:
+Sources are merged in this order, highest priority first:
-1. **Command-line flags** (highest priority)
+
-2. **Environment variables**
+1. Command-line flags
-3. **Configuration file**
+2. Environment variables
-4. **Built-in defaults** (lowest priority)
+3. Configuration file
+4. Built-in defaults
+
+The `pipelines` array is replaced wholesale, not merged: as soon as your file
+defines `[[pipelines]]`, the built-in default pipeline (and its default rate
+limit and formatter) disappears entirely.
 
 ## File Location
 
-LogWisp searches for configuration in order:
+The path is resolved before any other configuration is read:
-1. Path specified via `--config` flag
+
-2. Path from `LOGWISP_CONFIG_FILE` environment variable
+1. `-c <path>` on the command line
-3. `~/.config/logwisp/logwisp.toml`
+2. `--config=<path>` on the command line
-4. `./logwisp.toml` in current directory
+3. `$LOGWISP_CONFIG_FILE`, joined onto `$LOGWISP_CONFIG_DIR` when both are set
+4. `$LOGWISP_CONFIG_DIR/logwisp.toml`
+5. `~/.config/logwisp/logwisp.toml`, if it exists
+6. `./logwisp.toml`
+
+Missing file behaviour differs by how it was chosen. An explicitly requested
+file that does not exist is a fatal error (exit code 2); a missing discovered
+default is not an error, and LogWisp starts on built-in defaults.
+
+> `--config <path>` with a space is **not** recognized as a config path. It is
+> parsed as an unknown flag, warned about, and ignored — LogWisp then silently
+> falls back to `./logwisp.toml`. Use `-c <path>` or `--config=<path>`.
 
 ## Global Settings
 
-Top-level configuration options:
-
 | Setting | Type | Default | Description |
 |---------|------|---------|-------------|
-| `quiet` | bool | false | Suppress console output |
+| `quiet` | bool | `false` | Disable all application logging and console diagnostics |
-| `status_reporter` | bool | true | Periodic status logging |
+| `status_reporter` | bool | `true` | Emit a periodic status report every 30 s at DEBUG level |
-| `auto_reload` | bool | false | Enable file watch for auto-reload |
+| `auto_reload` | bool | `false` | Watch the config file and reload pipelines on change |
 
-## Logging Configuration
+`--version` prints version information and exits; it is not a persistent
+setting.
 
-LogWisp's internal operational logging:
+Note that `status_reporter` writes at DEBUG level, so it produces nothing unless
+`logging.level = "debug"`.
+
+## Application Logging
+
+This configures LogWisp's own operational log, not the log data it transports.
 
 ```toml
 [logging]
 output = "stdout"      # file | stdout | stderr | split | all | none
 level  = "info"        # debug | info | warn | error
+format = "txt"         # raw | txt | json
+# sanitization = ""    # raw | json | txt | shell
 
 [logging.file]
 directory         = "./log"
@@ -43,161 +66,220 @@ name = "logwisp"
 max_size_mb       = 100
 max_total_size_mb = 1000
 retention_hours   = 168.0
-
-[logging.console]
-target = "stdout"  # stdout|stderr|split
 ```
 
-### Output Modes
+### Output modes
 
-- **file**: Write to log files only
+| Mode | Behaviour |
-- **stdout**: Write to standard output
+|------|-----------|
-- **stderr**: Write to standard error
+| `file` | Files only |
-- **split**: INFO/DEBUG to stdout, WARN/ERROR to stderr
+| `stdout` | Standard output only |
-- **all**: Write to both file and console
+| `stderr` | Standard error only |
-- **none**: Disable all logging
+| `split` | DEBUG/INFO to stdout, WARN/ERROR to stderr |
+| `all` | Files plus split console |
+| `none` | No application logging |
+
+`[logging.file]` applies only to the `file` and `all` modes.
+
+> `[logging.console].target` is accepted and validated (`stdout`, `stderr`,
+> `split`) but **not applied**. The console destination is derived from
+> `logging.output`. The key is retained for compatibility; setting it has no
+> effect.
+
+`quiet = true` overrides every logging setting and disables both file and
+console output.
 
 ## Pipeline Configuration
 
-Each `[[pipelines]]` section defines an independent processing pipeline:
-
 ```toml
 [[pipelines]]
-name = "pipeline-name"
+name = "app"                       # required, unique across pipelines
 
-# Rate limiting (optional)
+# --- flow: everything between sources and sinks ---
 [pipelines.flow.rate_limit]
 rate                 = 1000.0
 burst                = 2000.0
-policy = "drop"  # pass|drop
+policy               = "drop"
-max_entry_size_bytes = 0  # 0=unlimited
+max_entry_size_bytes = 65536
 
-# Format configuration (optional)
-[pipelines.flow.format]
-type = "json"  # raw|json|txt
-sanitizer_policy = "json"
-
-[[pipelines.plugin_sources]]
-id = "my_source"
-type = "file"
-[pipelines.plugin_sources.config]
-# ... source-specific config
-
-# Filters (optional)
 [[pipelines.flow.filters]]
 type     = "include"
 logic    = "or"
 patterns = ["ERROR", "WARN"]
 
-# Sinks (required, 1+)
+[pipelines.flow.format]
+type             = "json"
+sanitizer_policy = "json"
+
+[pipelines.flow.heartbeat]
+enabled     = true
+interval_ms = 30000
+
+# --- sources: one or more ---
+[[pipelines.plugin_sources]]
+id   = "app_logs"                  # unique within the pipeline
+type = "file"
+[pipelines.plugin_sources.config]
+directory = "/var/log/myapp"
+
+# --- sinks: one or more ---
 [[pipelines.plugin_sinks]]
-id = "my_sink"
+id   = "sse"
 type = "http"
 [pipelines.plugin_sinks.config]
-# ... sink-specific config
+port = 8080
 ```
 
+Every source and sink is a plugin instance with three keys:
+
+| Key | Meaning |
+|-----|---------|
+| `id` | Instance identifier, unique within the pipeline; appears in logs and stats |
+| `type` | Registered plugin type |
+| `config` | Plugin-specific table; see [Sources](sources.md) and [Sinks](sinks.md) |
+
+`config_file` is reserved on both structures for a future include mechanism and
+is not implemented.
+
+### Flow stages
+
+| Block | Optional | Reference |
+|-------|----------|-----------|
+| `flow.rate_limit` | yes | below |
+| `flow.filters` | yes | [Filters](filters.md) |
+| `flow.format` | yes (defaults to `raw`) | [Formatters](formatters.md) |
+| `flow.heartbeat` | yes | below |
+
+#### Rate limiting
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `rate` | float | `0` | Entries per second; `<= 0` disables the limiter entirely |
+| `burst` | float | `rate` | Token bucket capacity |
+| `policy` | string | `pass` | `pass` allows everything through, `drop` discards over-limit entries |
+| `max_entry_size_bytes` | int | `0` | Per-entry byte cap; `0` = unlimited |
+
+Two behaviours are easy to trip over:
+
+- The limiter is constructed only when `rate > 0`. With `rate = 0`,
+  `max_entry_size_bytes` is never enforced.
+- `policy = "pass"` short-circuits the whole check, including the size cap.
+  To enforce a size cap you need `rate > 0` **and** `policy = "drop"`.
+
+#### Heartbeat
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | bool | `false` | Enable heartbeat generation |
+| `interval_ms` | int | `1000` | Interval; minimum `100` |
+| `include_timestamp` | bool | `false` | `false` formats with level only, no timestamp |
+| `include_stats` | bool | `false` | Attach `beat_count` and measured `interval_ms` as fields |
+| `format` | string | `txt` | `txt`, `json`, or `raw` |
+
+Heartbeats are ordinary entries with source `heartbeat` and level `INFO`. They
+are generated after the flow's filter and rate-limit stages, so filters do not
+suppress them, and they reach every sink in the pipeline.
+
+> `format = "comment"` (SSE comment framing) appears in older documentation and
+> in a code path in the generator, but the validator rejects it and the pipeline
+> fails to start. Use `txt`, `json`, or `raw`.
+
 ## Environment Variables
 
-All configuration options support environment variable overrides:
+Environment overrides are derived from the TOML path: `.` becomes `_` and the
+result is uppercased.
 
-### Naming Convention
+| TOML path | Environment variable |
-
-- Prefix: `LOGWISP_`
-- Path separator: `_` (underscore)
-- Array indices: Numeric suffix (0-based)
-- Case: UPPERCASE
-
-### Mapping Examples
-
-| TOML Path | Environment Variable |
 |-----------|---------------------|
-| `quiet` | `LOGWISP_QUIET` |
+| `quiet` | `QUIET` |
-| `logging.level` | `LOGWISP_LOGGING_LEVEL` |
+| `status_reporter` | `STATUS_REPORTER` |
-| `pipelines[0].name` | `LOGWISP_PIPELINES_0_NAME` |
+| `logging.level` | `LOGGING_LEVEL` |
-| `pipelines[0].plugin_sources[0].type` | `LOGWISP_PIPELINES_0_PLUGIN_SOURCES_0_TYPE` |
+| `logging.file.directory` | `LOGGING_FILE_DIRECTORY` |
+
+> **The `LOGWISP_` prefix is not currently applied.** The configuration loader
+> requests it, but supplying a custom path-to-variable transform replaces the
+> prefixing step rather than composing with it, so LogWisp reads bare
+> `QUIET`, `LOGGING_LEVEL`, and so on from the environment. Treat this as
+> current behaviour to be aware of — bare names like `QUIET` can collide with
+> unrelated variables — rather than as intended design.
+>
+> The two exceptions are `LOGWISP_CONFIG_FILE` and `LOGWISP_CONFIG_DIR`, which
+> are read directly by the path resolver and **do** carry the prefix.
+
+Only scalar paths that exist in the configuration schema can be set this way.
+Array elements cannot: `PIPELINES_0_NAME` has no effect.
 
 ## Command-Line Overrides
 
-All configuration options can be overridden via CLI flags:
+Any scalar configuration path is settable as a flag using its TOML path:
 
 ```bash
-logwisp --quiet \
+logwisp --logging.level=debug --status_reporter=false
-  --logging.level=debug \
+logwisp --logging.level debug          # space form also works
-  --pipelines.0.name=myapp \
+logwisp --quiet                        # bare flag means true
-  --pipelines.0.plugin_sources.0.type=console
 ```
 
-## Configuration Validation
+Unrecognized flags are reported on stderr before the logger exists and are then
+ignored:
 
-LogWisp validates configuration at startup:
+```
-- Rpipelines non-empty, name non-empty, ≥1 source, ≥1 sink, logging enum values.equired fields presence
+Warning: unrecognized flags ignored: [pipelines.0.name]
+```
 
-Partial check in plugin constructor:
+> Array-indexed paths are **not** settable from the command line.
-- Type correctness
+> `--pipelines.0.name=x`, `--pipelines.0.plugin_sinks.0.type=null`, and similar
-- Port conflicts
+> flags are reported as unrecognized and ignored. Pipelines, sources, sinks, and
-- Path accessibility
+> filters can only be defined in the configuration file. Older documentation
-- Pattern compilation
+> claimed otherwise.
-- Network address formats
+
+## Validation
+
+Startup validation is intentionally split.
+
+`internal/config` validates only global structure:
+
+- at least one pipeline
+- unique, non-empty pipeline names
+- at least one source and one sink per pipeline
+- `logging.output`, `logging.level`, `logging.format`, `logging.sanitization`,
+  and `logging.console.target` enum membership
+
+Everything else is validated by the plugin constructor that owns it — port
+range, required paths, path prefixes, enum values, regex compilation, TLS file
+loading. A failure there aborts pipeline construction with a message naming the
+pipeline, plugin id, and offending key.
+
+There is **no** cross-pipeline port-conflict detection. Two sinks bound to the
+same port fail at listener bind time, when the pipeline starts.
 
 ## Hot Reload
 
-Enable configuration hot reload:
-
 ```toml
 auto_reload = true
 ```
 
-Or via command line:
+or send `SIGHUP` / `SIGUSR1`.
-```bash
-logwisp --auto-reload
-```
 
-Reload triggers:
+Reload rebuilds the whole service: a new service is constructed from the new
-- File modification detection
+configuration first, and only if that succeeds is the old one shut down. A
-- SIGHUP or SIGUSR1 signals
+configuration error therefore leaves the running service untouched.
 
-Reloadable items:
+| Reloaded | Not reloaded |
-- Pipeline configurations
+|----------|--------------|
-- Sources and sinks
+| Pipelines, sources, sinks | `logging.*` (applied once at startup) |
-- Filters and formatters
+| Filters, formatters, rate limits, heartbeats | `quiet` |
-- Rate limits
+| `status_reporter` | `auto_reload` (the watcher is not restarted) |
 
-Non-reloadable (requires restart):
+Because the rebuild is total, listeners close and reopen and every connected
-- Logging configuration
+client is disconnected. Chain sinks reconnect on their own backoff schedule.
-- Global settings
 
-## Default Configuration
+## Type Reference
 
-Minimal working configuration:
+| TOML type | Go type | Command-line / environment form |
-
+|-----------|---------|-------------------------------|
-```toml
+| String | `string` | Plain text |
-[[pipelines]]
+| Integer | `int64` | Decimal string |
-name = "default"
+| Float | `float64` | Decimal string |
-
+| Boolean | `bool` | `true` / `false`, or a bare flag for `true` |
-[[pipelines.plugin_sources]]
+| Array | `[]T` | Not settable outside the file |
-id = "default_source"
+| Table | struct | Nested path with `.` (flags) or `_` (environment) |
-type = "file"
-[pipelines.plugin_sources.config]
-directory = "./"
-pattern = "*.log"
-
-[[pipelines.plugin_sinks]]
-id = "default_sink"
-type = "console"
-[pipelines.plugin_sinks.config]
-target = "stdout"
-```
-
-## Configuration Schema
-
-### Type Reference
-
-| TOML Type | Go Type | Environment Format |
-|-----------|---------|-------------------|
-| String | string | Plain text |
-| Integer | int64 | Numeric string |
-| Float | float64 | Decimal string |
-| Boolean | bool | true/false |
-| Array | []T | JSON array string |
-| Table | struct | Nested with `_` |

doc/filters.md

@@ -1,185 +1,192 @@
 # Filters
 
-LogWisp filters control which log entries pass through the pipeline using pattern matching.
+Filters decide which entries continue through a pipeline. They run in the flow,
-
+after rate limiting and before formatting.
-## Filter Types
-
-### Include Filter
-
-Only entries matching patterns pass through.
 
 ```toml
 [[pipelines.flow.filters]]
 type     = "include"
-logic = "or"  # or|and
+logic    = "or"
-patterns = [
+patterns = ["ERROR", "WARN"]
-    "ERROR",
-    "WARN",
-    "CRITICAL"
-]
 ```
 
-### Exclude Filter
+## Options
-
-Entries matching patterns are dropped.
-
-```toml
-[[pipelines.flow.filters]]
-type = "exclude"
-patterns = [
-    "DEBUG",
-    "TRACE",
-    "health-check"
-]
-```
-
-## Configuration Options
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `type` | string | Required | Filter type (include/exclude) |
+| `type` | string | `include` | `include` (only matches pass) or `exclude` (matches are dropped) |
-| `logic` | string | "or" | Pattern matching logic (or/and) |
+| `logic` | string | `or` | `or` (any pattern matches) or `and` (every pattern matches) |
-| `patterns` | []string | Required | Pattern list |
+| `patterns` | []string | `[]` | Go RE2 regular expressions |
 
-## Pattern Syntax
+A filter with no patterns passes everything. Invalid patterns fail at startup
+with the filter index and the offending pattern in the message.
 
-Patterns support regular expression syntax:
+## What Gets Matched
 
-### Basic Patterns
+Patterns are matched against a single string assembled from the entry:
-- **Literal match**: `"ERROR"` - matches "ERROR" anywhere
-- **Case-insensitive**: `"(?i)error"` - matches "error", "ERROR", "Error"
-- **Word boundary**: `"\\berror\\b"` - matches whole word only
 
-### Advanced Patterns
+```
-- **Alternation**: `"ERROR|WARN|FATAL"`
+"<source> <level> <message>"
-- **Character classes**: `"[0-9]{3}"`
+```
-- **Wildcards**: `".*exception.*"`
-- **Line anchors**: `"^ERROR"` (start), `"ERROR$"` (end)
 
-### Special Characters
+Empty parts are omitted, so an entry with no detected level matches
-Escape special regex characters with backslash:
+`"<source> <message>"`. This means a pattern can target the source name or the
-- `.` → `\\.`
+level as easily as the message body:
-- `*` → `\\*`
-- `[` → `\\[`
-- `(` → `\\(`
 
-## Filter Logic
+| Pattern | Matches |
+|---------|---------|
+| `"^app\\.log "` | Entries whose source is `app.log` |
+| `"ERROR"` | Level `ERROR`, or the word `ERROR` anywhere in the message |
+
+The structured `fields` payload is **not** part of the match text.
+
+For entries that arrived over a chain link, the `source` used here is the bare
+source — the `node/source` prefix is applied later, by the formatter — so
+filtering by originating node requires matching on the message, or filtering on
+the node that produces the entries.
+
+## Filter Types
+
+### include
+
+Only matching entries pass. Everything else is dropped.
+
+```toml
+[[pipelines.flow.filters]]
+type     = "include"
+patterns = ["ERROR", "WARN", "FATAL"]
+```
+
+### exclude
+
+Matching entries are dropped. Everything else passes.
+
+```toml
+[[pipelines.flow.filters]]
+type     = "exclude"
+patterns = ["/healthz", "TRACE"]
+```
+
+## Logic
+
+### or (default)
 
-### OR Logic (default)
-Entry passes if ANY pattern matches:
 ```toml
 logic    = "or"
 patterns = ["ERROR", "WARN"]
-# Passes: "ERROR in module", "WARN: low memory"
+# passes: "ERROR in module"      "WARN: low memory"
-# Blocks: "INFO: started"
+# blocks: "INFO: started"
 ```
 
-### AND Logic
+### and
-Entry passes only if ALL patterns match:
+
 ```toml
 logic    = "and"
 patterns = ["database", "ERROR"]
-# Passes: "ERROR: database connection failed"
+# passes: "ERROR: database connection failed"
-# Blocks: "ERROR: file not found"
+# blocks: "ERROR: file not found"
 ```
 
-## Filter Chain
+With `logic = "and"` on an `exclude` filter, an entry is dropped only when it
+matches *every* pattern.
 
-Multiple filters execute sequentially:
+## Filter Chains
+
+Filters are evaluated in declaration order and an entry must survive all of
+them. The first filter to reject an entry ends its life; later filters never see
+it.
 
 ```toml
-# First filter: Include errors and warnings
+# 1. keep only production traffic
-[[pipelines.flow.filters]]
-type = "include"
-patterns = ["ERROR", "WARN"]
-
-# Second filter: Exclude test environments
-[[pipelines.flow.filters]]
-type = "exclude"
-patterns = ["test-env", "staging"]
-```
-
-Processing order:
-1. Entry arrives from source
-2. Include filter evaluates
-3. If passed, exclude filter evaluates
-4. If passed all filters, entry continues to sink
-
-## Performance Considerations
-
-### Pattern Compilation
-- Patterns compile once at startup
-- Invalid patterns cause startup failure
-- Complex patterns may impact performance
-
-### Optimization Tips
-- Place most selective filters first
-- Use simple patterns when possible
-- Combine related patterns with alternation
-- Avoid excessive wildcards (`.*`)
-
-## Filter Statistics
-
-Filters track:
-- Total entries evaluated
-- Entries passed
-- Entries blocked
-- Processing time per pattern
-
-## Common Use Cases
-
-### Log Level Filtering
-```toml
-[[pipelines.filters]]
-type = "include"
-patterns = ["ERROR", "WARN", "FATAL", "CRITICAL"]
-```
-
-### Application Filtering
-```toml
-[[pipelines.flow.filters]]
-type = "include"
-patterns = ["app1", "app2", "app3"]
-```
-
-### Noise Reduction
-```toml
-[[pipelines.flow.filters]]
-type = "exclude"
-patterns = [
-    "health-check",
-    "ping",
-    "/metrics",
-    "heartbeat"
-]
-```
-
-### Security Filtering
-```toml
-[[pipelines.flow.filters]]
-type = "exclude"
-patterns = [
-    "password",
-    "token",
-    "api[_-]key",
-    "secret"
-]
-```
-
-### Multi-stage Filtering
-```toml
-# Include production logs
 [[pipelines.flow.filters]]
 type     = "include"
 patterns = ["prod-", "production"]
 
-# Include only errors
+# 2. of that, keep only failures
 [[pipelines.flow.filters]]
 type     = "include"
 patterns = ["ERROR", "EXCEPTION", "FATAL"]
 
-# Exclude known issues
+# 3. minus known noise
 [[pipelines.flow.filters]]
 type     = "exclude"
 patterns = ["ECONNRESET", "broken pipe"]
 ```
+
+Order matters for cost, not for correctness: put the most selective filter first
+so later ones evaluate fewer entries.
+
+## Pattern Syntax
+
+Go's RE2 syntax. No backreferences and no lookaround — RE2 guarantees linear
+time, which is exactly what you want in a log hot path.
+
+| Need | Pattern |
+|------|---------|
+| Literal substring | `ERROR` |
+| Case-insensitive | `(?i)error` |
+| Whole word | `\\berror\\b` |
+| Alternation | `ERROR\|WARN\|FATAL` |
+| Character class | `[0-9]{3}` |
+| Anchors | `^ERROR`, `ERROR$` |
+| Any characters | `.*exception.*` |
+
+Remember that TOML basic strings process escapes, so a regex backslash needs
+doubling: `"\\berror\\b"`. TOML literal strings avoid the issue:
+`'\berror\b'`.
+
+Anchors apply to the assembled match text, which begins with the source name —
+so `^ERROR` will not match an entry whose source is non-empty. Use
+`\\bERROR\\b` instead unless you mean to anchor on the source.
+
+## Common Recipes
+
+**Severity floor**
+
+```toml
+[[pipelines.flow.filters]]
+type     = "include"
+patterns = ["ERROR", "FATAL", "CRITICAL"]
+```
+
+**Noise reduction**
+
+```toml
+[[pipelines.flow.filters]]
+type     = "exclude"
+patterns = ["/healthz", "/metrics", "\\bping\\b"]
+```
+
+**Secret suppression** — see [Security](security.md); filters are the only
+redaction mechanism LogWisp currently offers.
+
+```toml
+[[pipelines.flow.filters]]
+type     = "exclude"
+patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret", "token"]
+```
+
+Note this drops the whole entry, it does not redact part of it.
+
+**Per-application routing** — run one pipeline per application, each with its
+own include filter, rather than trying to route inside one pipeline. Sinks fan
+out to *all* sinks in a pipeline; there is no conditional routing.
+
+## Statistics
+
+Each filter reports `type`, `logic`, `pattern_count`, `total_processed`,
+`total_matched`, and `total_dropped`. The chain reports `filter_count`,
+`total_processed`, and `total_passed`; the pipeline derives
+`total_filtered` as the difference.
+
+## Performance
+
+Patterns compile once at startup. Every entry that reaches the filter stage is
+evaluated against every filter until one rejects it, so cost scales with the
+number of patterns and their complexity. Prefer literal substrings and simple
+alternations over broad `.*` wildcards.
+
+Filters log at DEBUG on every entry — pattern text, match results, and the
+final decision. That is invaluable when a filter is not behaving as expected and
+very expensive in production; keep `logging.level` at `info` or higher on a busy
+pipeline.

doc/formatters.md

@@ -1,180 +1,175 @@
 # Formatters
 
-LogWisp formatters transform log entries before output to sinks.
+The formatter is the last flow stage. It turns a `core.LogEntry` into the byte
+payload that sinks write, applying a sanitizer policy on the way.
 
-## Formatter Types
+```toml
+[pipelines.flow.format]
+type             = "json"
+sanitizer_policy = "json"
+flags            = 0
+timestamp_format = ""
+```
 
-### Raw Formatter
+One formatter serves the whole pipeline. Sinks receive an identical payload;
+there is no per-sink formatting. When you need two shapes of the same data, run
+two pipelines, or chain to a node that formats differently.
 
-Outputs the log message as-is with optional newline.
+Omitting `[pipelines.flow.format]` entirely selects `raw`.
+
+## Options
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `type` | string | `raw` | `raw`, `txt` (alias `text`), or `json` |
+| `sanitizer_policy` | string | derived from `type` | `raw`, `txt`, `json`, or `shell` |
+| `flags` | int64 | `0` | Bitmask override; `0` selects a per-type default |
+| `timestamp_format` | string | formatter default | Go reference layout, e.g. `"2006-01-02T15:04:05Z07:00"` |
+
+## Types
+
+### raw
+
+Passthrough. `FlagRaw` bypasses both formatting and sanitization, so the
+message reaches the sink exactly as the source produced it.
 
 ```toml
 [pipelines.flow.format]
 type = "raw"
-sanitizer_policy = "raw"
-flags = 1
 ```
 
-**Configuration Options:**
+Fastest option, and the right one when you are relaying text that is already in
+its final form. Note that it also bypasses sanitization, so control characters
+in the source data reach your sinks intact.
 
-| Option | Type | Default | Description |
+### txt
-|--------|------|---------|-------------|
-| `add_new_line` | bool | true | Append newline to messages |
-| `type` | string | "raw" | raw, json, or txt |
-| `flags` | int64 | 0 | log/formatter flags override |
-| `sanitizer_policy` | string | | Sanitizer policy (e.g. "json", "raw", "txt", "shell") |
 
-### JSON Formatter
+Human-readable line output with a timestamp and level.
-
-Produces structured JSON output.
-
-```toml
-[pipelines.format]
-type = "json"
-
-+[pipelines.flow.format]
- type = "json"
-sanitizer_policy = "json"
-```
-
-**Output Structure:**
-```json
-{
-  "timestamp": "2024-01-01T12:00:00Z",
-  "level": "ERROR",
-  "source": "app",
-  "message": "Connection failed"
-}
-```
-
-### Text Formatter
-
-Template-based text formatting.
 
 ```toml
 [pipelines.flow.format]
 type             = "txt"
 sanitizer_policy = "txt"
-timestamp_format = "2006-01-02T15:04:05.000Z07:00"
+timestamp_format = "2006-01-02 15:04:05"
 ```
 
-**Configuration Options:**
+### json
 
-| Option | Type | Default | Description |
+Structured output, the natural choice for downstream ingestion.
-|--------|------|---------|-------------|
-| `timestamp_format` | string | "" | Time format override |
 
-**Default Template:**
-```
-[{{.Timestamp | FmtTime}}] [{{.Level | ToUpper}}] {{.Source}} - {{.Message}}{{ if .Fields }} {{.Fields}}{{ end }}
-```
-
-## Template Functions
-
-Available functions in text templates:
-
-| Function | Description | Example |
-|----------|-------------|---------|
-| `FmtTime` | Format timestamp | `{{.Timestamp \| FmtTime}}` |
-| `ToUpper` | Convert to uppercase | `{{.Level \| ToUpper}}` |
-| `ToLower` | Convert to lowercase | `{{.Source \| ToLower}}` |
-| `TrimSpace` | Remove whitespace | `{{.Message \| TrimSpace}}` |
-
-## Template Variables
-
-Available variables in templates:
-
-| Variable | Type | Description |
-|----------|------|-------------|
-| `.Timestamp` | time.Time | Entry timestamp |
-| `.Level` | string | Log level |
-| `.Source` | string | Source identifier |
-| `.Message` | string | Log message |
-| `.Fields` | string | Additional fields (JSON) |
-
-## Time Format Strings
-
-Common Go time format patterns:
-
-| Pattern | Example Output |
-|---------|---------------|
-| `2006-01-02T15:04:05Z07:00` | 2024-01-02T15:04:05Z |
-| `2006-01-02 15:04:05` | 2024-01-02 15:04:05 |
-| `Jan 2 15:04:05` | Jan 2 15:04:05 |
-| `15:04:05.000` | 15:04:05.123 |
-| `2006/01/02` | 2024/01/02 |
-
-## Format Selection
-
-### Default Behavior
-
-If no formatter specified:
-- **HTTP/TCP sinks**: JSON format
-- **Console/File sinks**: Raw format
-- **Client sinks**: JSON format
-
-### Per-Pipeline Configuration
-
-Each pipeline can have its own formatter:
-
-```toml
-[[pipelines]]
-name = "json-pipeline"
-[pipelines.flow.format]
-type = "json"
-
-[[pipelines]]
-name = "text-pipeline"
-[pipelines.flow.format]
-type = "txt"
-```
-
-## Message Processing
-
-### JSON Message Handling
-
-When using JSON formatter with JSON log messages:
-1. Attempts to parse message as JSON
-2. Merges fields with LogWisp metadata
-3. LogWisp fields take precedence
-4. Falls back to string if parsing fails
-
-### Field Preservation
-
-LogWisp metadata always includes:
-- Timestamp (from source or current time)
-- Level (detected or default)
-- Source (origin identifier)
-- Message (original content)
-
-## Performance Characteristics
-
-### Formatter Performance
-
-Relative performance (fastest to slowest):
-1. **Raw**: Direct passthrough
-2. **Text**: Template execution
-3. **JSON**: Serialization
-4. **JSON (pretty)**: Formatted serialization
-
-### Optimization Tips
-
-- Use raw format for high throughput
-- Cache template compilation (automatic)
-- Minimize template complexity
-- Avoid pretty JSON in production
-
-## Common Configurations
-
-### Structured Logging
 ```toml
 [pipelines.flow.format]
 type             = "json"
+sanitizer_policy = "json"
 ```
 
-### Human-Readable Logs
+Output has the shape:
-```toml
+
-[pipelines.flow.format]
+```json
-type = "txt"
+{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}
-timestamp_format = "15:04:05"
 ```
+
+The exact key names and structure come from the `lixenwraith/log` formatter, not
+from LogWisp; they are stable for a given dependency version but are not part of
+LogWisp's own configuration surface.
+
+## Flags
+
+`flags` is a bitmask passed to the underlying formatter. Leave it at `0` unless
+you need to override the defaults.
+
+| Value | Name | Effect |
+|-------|------|--------|
+| `1` | Raw | Bypass formatting and sanitization entirely |
+| `2` | ShowTimestamp | Emit the timestamp |
+| `4` | ShowLevel | Emit the level |
+| `8` | StructuredJSON | Render attached fields as a JSON object |
+| `16` | NoTimestamp | Suppress the timestamp |
+| `32` | NoLevel | Suppress the level |
+
+With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
+(timestamp + level) for every other type. `8` is added automatically whenever an
+entry carries parseable `fields`.
+
+Examples: `flags = 4` for level only, no timestamp; `flags = 2` for timestamp
+only, no level.
+
+## Sanitizer Policies
+
+The sanitizer runs before serialization and neutralizes control characters that
+would otherwise break framing or reach a terminal.
+
+| Policy | Behaviour | Use with |
+|--------|-----------|----------|
+| `raw` | No-op passthrough | `type = "raw"` where you control the data |
+| `txt` | Escapes non-printable characters | File and console sinks |
+| `json` | Escapes control characters for safe JSON embedding | `type = "json"`, chain links |
+| `shell` | Strips shell metacharacters, whitespace, and control characters | Data that will be passed to a command |
+
+When `sanitizer_policy` is omitted, the policy is derived from `type`: `json`
+for `json`, `txt` for `txt`/`text`, and `raw` for anything else — so the safe
+pairing is the default.
+
+> `shell` strips dangerous characters but is **not** sufficient to make a string
+> safe for shell construction. Pass arguments through `exec` argv instead of
+> building command lines.
+
+To see a policy working, point a pipeline at the `random` source with
+`special = true`, which injects control bytes and multi-byte Unicode into every
+message.
+
+## Node Identity in Output
+
+Entries that arrived over a chain link carry a `Node` label. The formatter
+renders it as a syslog-style prefix on the source field:
+
+```
+edge-01/app.log
+```
+
+Entries with no node label show the bare source. Node identity therefore appears
+*inside* the source field rather than as a separate output key — worth knowing
+when writing downstream parsers or grep patterns.
+
+## Structured Fields
+
+When an entry carries `Fields` (raw JSON), the formatter parses it and switches
+to structured rendering by adding the `StructuredJSON` flag automatically.
+Fields reach a pipeline in two ways: from the `file` source when a tailed line
+parses as JSON with a `fields` key, and from the heartbeat generator when
+`include_stats = true`.
+
+## Choosing a Configuration
+
+| Goal | Configuration |
+|------|---------------|
+| Maximum throughput, data already formatted | `type = "raw"` |
+| Human reading in a terminal or file | `type = "txt"`, `sanitizer_policy = "txt"` |
+| Downstream ingestion (Loki, Elasticsearch, jq) | `type = "json"`, `sanitizer_policy = "json"` |
+| Compact console output | `type = "txt"`, `flags = 4` |
+| Untrusted log content | never `raw`; pick `txt` or `json` and set the matching policy |
+
+## Formatting and Chain Links
+
+Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload.
+They re-serialize the structured entry into the canonical chain encoding, which
+makes them independent of the local formatter.
+
+The practical consequence: setting `flow.format` on an edge node changes only
+that node's own local sinks. The output shape seen by a human or a downstream
+system is decided on the node that owns the sink they read.
+
+If an event ever reaches a chain sink without a structured entry, the sink wraps
+the formatted payload into a synthetic entry and counts it in `synthesized`.
+A non-zero `synthesized` count means something upstream lost structure.
+
+## Performance
+
+Relative cost, cheapest first: `raw` (passthrough) → `txt` (line assembly) →
+`json` (serialization). Sanitization adds a scan of the message; the `raw`
+policy skips it.
+
+The formatter holds a mutex because the underlying implementation reuses an
+internal buffer and is not goroutine-safe. It is the only shared serialization
+point in the hot path, and the reason a single pipeline formats entries one at
+a time.

doc/installation.md

@@ -1,49 +1,62 @@
 # Installation Guide
 
-LogWisp installation and service configuration for Linux and FreeBSD systems.
+## Requirements
 
-## Installation Methods
+- **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
+- **Architecture**: amd64
+- **Go**: 1.26 or newer, to build from source
 
-### Pre-built Binaries
+## Building from Source
-
-Download the latest release binary for your platform and install to `/usr/local/bin`:
 
 ```bash
-# Linux amd64
+git clone https://github.com/lixenwraith/logwisp.git
-wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-linux-amd64
-chmod +x logwisp-linux-amd64
-sudo mv logwisp-linux-amd64 /usr/local/bin/logwisp
-
-# FreeBSD amd64  
-fetch https://github.com/yourusername/logwisp/releases/latest/download/logwisp-freebsd-amd64
-chmod +x logwisp-freebsd-amd64
-sudo mv logwisp-freebsd-amd64 /usr/local/bin/logwisp
-```
-
-### Building from Source
-
-Requires Go 1.24 or newer:
-
-```bash
-git clone https://github.com/yourusername/logwisp.git
 cd logwisp
-go build -o logwisp ./src/cmd/logwisp
+make
-sudo install -m 755 logwisp /usr/local/bin/
+sudo make install          # installs to $PREFIX/bin, default /usr/local/bin
 ```
 
-### Go Install Method
+The Makefile works with both GNU make and BSD make. Targets:
 
-Install directly using Go (version information will not be embedded):
+| Target | Effect |
+|--------|--------|
+| `make` / `make build` | Build `bin/logwisp` with version metadata |
+| `make dev` | Build with the race detector enabled |
+| `make install` | Install the binary to `$(PREFIX)/bin` (default `/usr/local`) |
+| `make uninstall` | Intended to remove the installed binary — currently broken: it expands to `$(BINDIR)/bin/logwisp` instead of `$(BINDIR)/logwisp`, so it removes nothing. Delete the binary by hand |
+| `make clean` | Remove the built binary |
+| `make version` | Print the version, commit, and build time that would be embedded |
+
+Version, commit hash, and build time are injected via `-ldflags` from `git
+describe` and `git rev-parse`. A plain `go build` produces a working binary that
+reports `dev` for all three:
 
 ```bash
-go install github.com/yourusername/logwisp/cmd/logwisp@latest
+go build -o bin/logwisp ./cmd/logwisp
 ```
 
-## Service Configuration
+`go install github.com/lixenwraith/logwisp/cmd/logwisp@latest` also works, with
+the same loss of version metadata.
+
+## Configuration
+
+Copy the annotated reference configuration and edit it:
+
+```bash
+sudo mkdir -p /etc/logwisp
+sudo cp config/logwisp.toml /etc/logwisp/logwisp.toml
+```
+
+LogWisp searches, in order: `-c <path>`, `--config=<path>`,
+`$LOGWISP_CONFIG_DIR`/`$LOGWISP_CONFIG_FILE`, `~/.config/logwisp/logwisp.toml`,
+`./logwisp.toml`. See [Configuration](configuration.md).
+
+## Running as a Service
+
+LogWisp has no daemon mode; run it in the foreground under a supervisor.
 
 ### Linux (systemd)
 
-Create systemd service file `/etc/systemd/system/logwisp.service`:
+`/etc/systemd/system/logwisp.service`:
 
 ```ini
 [Unit]
@@ -55,30 +68,47 @@ Type=simple
 User=logwisp
 Group=logwisp
 ExecStart=/usr/local/bin/logwisp -c /etc/logwisp/logwisp.toml
+ExecReload=/bin/kill -HUP $MAINPID
 Restart=on-failure
 RestartSec=10
+WorkingDirectory=/var/lib/logwisp
 StandardOutput=journal
 StandardError=journal
-WorkingDirectory=/var/lib/logwisp
+
+# Hardening
+NoNewPrivileges=true
+PrivateTmp=true
+ProtectSystem=strict
+ProtectHome=true
+ReadWritePaths=/var/log/logwisp /var/lib/logwisp
 
 [Install]
 WantedBy=multi-user.target
 ```
 
-Setup service user and directories:
+`ExecReload` gives you `systemctl reload logwisp` for configuration and
+certificate rotation without dropping the process.
+
+If a pipeline binds a port below 1024, add
+`AmbientCapabilities=CAP_NET_BIND_SERVICE` rather than running as root.
+
+Setup:
 
 ```bash
-sudo useradd -r -s /bin/false logwisp
+sudo useradd -r -s /usr/sbin/nologin logwisp
 sudo mkdir -p /etc/logwisp /var/lib/logwisp /var/log/logwisp
 sudo chown logwisp:logwisp /var/lib/logwisp /var/log/logwisp
 sudo systemctl daemon-reload
-sudo systemctl enable logwisp
+sudo systemctl enable --now logwisp
-sudo systemctl start logwisp
 ```
 
+The service account needs **read** access to every directory a `file` source
+watches and **write** access to every directory a `file` sink or
+`logging.file` writes to.
+
 ### FreeBSD (rc.d)
 
-Create rc script `/usr/local/etc/rc.d/logwisp`:
+`/usr/local/etc/rc.d/logwisp`:
 
 ```sh
 #!/bin/sh
@@ -92,8 +122,9 @@ Create rc script `/usr/local/etc/rc.d/logwisp`:
 name="logwisp"
 rcvar="${name}_enable"
 pidfile="/var/run/${name}.pid"
-command="/usr/local/bin/logwisp"
+procname="/usr/local/bin/logwisp"
-command_args="-c /usr/local/etc/logwisp/logwisp.toml"
+command="/usr/sbin/daemon"
+command_args="-p ${pidfile} -f ${procname} -c /usr/local/etc/logwisp/logwisp.toml"
 
 load_rc_config $name
 : ${logwisp_enable:="NO"}
@@ -101,7 +132,7 @@ load_rc_config $name
 run_rc_command "$1"
 ```
 
-Setup service:
+Setup:
 
 ```bash
 sudo chmod +x /usr/local/etc/rc.d/logwisp
@@ -112,45 +143,66 @@ sudo sysrc logwisp_enable="YES"
 sudo service logwisp start
 ```
 
-## Directory Structure
+## Directory Layout
-
-Standard installation directories:
 
 | Purpose | Linux | FreeBSD |
 |---------|-------|---------|
 | Binary | `/usr/local/bin/logwisp` | `/usr/local/bin/logwisp` |
 | Configuration | `/etc/logwisp/` | `/usr/local/etc/logwisp/` |
-| Working Directory | `/var/lib/logwisp/` | `/var/db/logwisp/` |
+| TLS material | `/etc/logwisp/tls/` | `/usr/local/etc/logwisp/tls/` |
-| Log Files | `/var/log/logwisp/` | `/var/log/logwisp/` |
+| Working directory | `/var/lib/logwisp/` | `/var/db/logwisp/` |
-| PID File | `/var/run/logwisp.pid` | `/var/run/logwisp.pid` |
+| Application logs | `/var/log/logwisp/` | `/var/log/logwisp/` |
 
-## Post-Installation Verification
+Key files should be mode `0600` and owned by the service account.
 
-Verify the installation:
+## Verification
 
 ```bash
-# Check version
+logwisp --version
-logwisp version
 
-# Test configuration
+# start in the foreground with debug logging and watch pipelines come up
-logwisp -c /etc/logwisp/logwisp.toml --disable-status-reporter
+logwisp -c /etc/logwisp/logwisp.toml --logging.level=debug --logging.output=stderr
 
-# Check service status (Linux)
+sudo systemctl status logwisp      # Linux
-sudo systemctl status logwisp
+sudo service logwisp status        # FreeBSD
-
-# Check service status (FreeBSD)
-sudo service logwisp status
 ```
 
-## Uninstallation
+Expect `Created source instance`, `Created sink instance`, and
+`Starting pipeline` for each configured pipeline. There is no validate-only
+mode; see [Operations](operations.md#checking-a-configuration).
+
+## Test Scripts
+
+Two end-to-end scripts under `test/` build multi-node chain topologies against a
+local build:
+
+```bash
+make
+./test/chain-test.sh --auto             # two independent relay pipelines
+./test/chain-aggregate-test.sh --auto   # fan-in: both edges into one pipeline
+```
+
+Without `--auto` they run the relay in the foreground for interactive
+inspection. They need bash 5+, coreutils, and curl, and they bind ports
+15801–15804. Generated configuration and logs land in `test/run/`.
+
+> Two of the three `--auto` assertions currently report `FAIL` against a
+> working build. They grep the sink output for `"source":"edge-tcp/` and
+> `"node":"edge-http"`, but the JSON formatter emits the `node/source` label
+> under the key `trace`. The transport itself is healthy — the
+> `total_processed` assertion passes and the streamed entries carry
+> `"trace":"edge-tcp/random_rand"` as expected. Until the assertions are
+> updated, verify the streams by eye with `nc 127.0.0.1 15803` and
+> `curl -sN http://127.0.0.1:15804/stream`.
+
+## Uninstall
 
 ### Linux
 
 ```bash
-sudo systemctl stop logwisp
+sudo systemctl disable --now logwisp
-sudo systemctl disable logwisp
+sudo rm /usr/local/bin/logwisp /etc/systemd/system/logwisp.service
-sudo rm /usr/local/bin/logwisp
+sudo systemctl daemon-reload
-sudo rm /etc/systemd/system/logwisp.service
 sudo rm -rf /etc/logwisp /var/lib/logwisp /var/log/logwisp
 sudo userdel logwisp
 ```
@@ -160,8 +212,7 @@ sudo userdel logwisp
 ```bash
 sudo service logwisp stop
 sudo sysrc -x logwisp_enable
-sudo rm /usr/local/bin/logwisp
+sudo rm /usr/local/bin/logwisp /usr/local/etc/rc.d/logwisp
-sudo rm /usr/local/etc/rc.d/logwisp
 sudo rm -rf /usr/local/etc/logwisp /var/db/logwisp /var/log/logwisp
 sudo pw userdel logwisp
 ```

doc/mtls-auth-plan.md

@@ -0,0 +1,368 @@
+# Implementation Plan: mTLS as Authentication
+
+**Status:** proposal, not implemented.
+**Scope:** turn the existing transport-level mutual TLS into a real
+authentication and authorization mechanism.
+
+## Problem
+
+LogWisp already does mutual TLS at the transport layer. A listener with
+`client_auth = true` refuses any peer that cannot present a certificate
+chaining to `client_ca_file`, and `internal/tlsx` already extracts the peer's
+Common Name and stashes it in session metadata as `tls_peer_cn`.
+
+Nothing reads it back. The result is a CA-wide membership check with no notion
+of *which* peer connected:
+
+1. **No per-identity authorization.** Every certificate the CA issues is
+   equivalent. There is no way to say "only `edge-01` and `edge-02` may write to
+   this ingest port", so one CA cannot serve several trust domains, and
+   withdrawing one peer means rotating the CA bundle for all of them.
+2. **Node labels are unauthenticated.** With `trust_node = true` (the default) a
+   peer declares its own `node` label in the chain hello or the
+   `X-Logwisp-Node` header. Any certificate holder can claim any label,
+   including another host's, and every downstream consumer will attribute those
+   entries accordingly. The only current defence, `trust_node = false`, replaces
+   the label with a remote address — unforgeable, but useless for identifying a
+   host behind NAT or a load balancer.
+3. **The `http` sink has no authentication at all**, even with TLS on. Its
+   stream and status endpoints are readable by anyone who can reach the port.
+
+Password, token, and SCRAM authentication were removed during the plugin/flow
+restructure and the move to standard-library networking. Certificates are the
+one credential the current transport already carries, which makes mTLS the
+cheapest path back to authenticated peers.
+
+## Goals
+
+- Authorize peers by certificate identity, per listener.
+- Bind the chain `node` label to the authenticated identity, so origin
+  attribution is trustworthy.
+- Gate the `http` sink's endpoints on client certificates.
+- Make identity visible in sessions, statistics, and logs.
+- Change nothing for existing configurations that omit the new block.
+
+## Non-Goals
+
+- Reviving password, token, or SCRAM authentication. The reserved hooks
+  (`chain.Hello.Auth`, the `Authorization` header comment in the `http_chain`
+  sink, the "Future: password auth block" comments in the network options
+  structs) stay reserved.
+- IP allow/deny lists and per-peer rate limits. Related, but a separate feature
+  with its own config surface.
+- OCSP. See [Revocation](#revocation) for what is proposed instead.
+- Authorization *within* a stream — the unit of decision is a connection (TCP)
+  or a request (HTTP), never an individual entry.
+
+## Design
+
+### Identity
+
+The authenticated identity is a single string derived from the peer's verified
+leaf certificate. Because `tls.RequireAndVerifyClientCert` has already validated
+the chain, signature, and validity window by the time we look, extraction is
+pure field selection.
+
+| `identity` mode | Source | Notes |
+|-----------------|--------|-------|
+| `cn` (default) | `Subject.CommonName` | Matches the existing `tls_peer_cn` metadata |
+| `san_dns` | first `DNSNames` entry | Preferred for host identities |
+| `san_uri` | first `URIs` entry | SPIFFE-style IDs |
+| `san_email` | first `EmailAddresses` entry | Operator identities |
+
+An empty identity is a rejection, not an empty match: a certificate with no
+usable identity field cannot satisfy any policy.
+
+Identities are not secrets, so ordinary string comparison is fine; there is no
+timing side channel worth defending here.
+
+### Configuration
+
+A new `auth` table sits beside `tls` in every network plugin's `config`. Keeping
+it separate from `tls` matters: TLS answers "is this channel private and is the
+peer chained to a CA", auth answers "may *this* peer do *this*", and a later
+non-certificate method should be able to reuse the block.
+
+```toml
+[pipelines.plugin_sources.config.auth]
+type           = "mtls"                  # none (default) | mtls
+identity       = "cn"                    # cn | san_dns | san_uri | san_email
+allow          = ["edge-01", "edge-02"]  # exact identities
+allow_patterns = ["^edge-\\d{2}$"]       # RE2, anchored by the author
+node_binding   = "force"                 # none | assert | force
+```
+
+| Option | Type | Default | Meaning |
+|--------|------|---------|---------|
+| `type` | string | `none` | `none` preserves today's behaviour exactly; `mtls` enables the policy |
+| `identity` | string | `cn` | Which certificate field is the identity |
+| `allow` | []string | `[]` | Exact identity matches |
+| `allow_patterns` | []string | `[]` | RE2 patterns matched against the identity |
+| `node_binding` | string | `force` when `type = "mtls"` | See below |
+
+Empty `allow` **and** empty `allow_patterns` under `type = "mtls"` means "any
+identity the CA vouches for" — that is, today's behaviour, but with the identity
+now recorded and node binding available. It is a deliberate, documented default
+rather than a silent deny-all, and startup logs say so plainly.
+
+`node_binding` applies only to the chain sources, where a `node` label is
+declared:
+
+| Value | Behaviour |
+|-------|-----------|
+| `none` | `trust_node` governs, as today |
+| `assert` | The declared label must equal the identity; a mismatch is rejected |
+| `force` | The declared label is ignored and the identity is used |
+
+`force` is the default under `type = "mtls"` because it is the only setting
+where a misconfigured or hostile edge cannot mislabel its entries. `assert`
+exists for operators who want the mismatch to be loud rather than silently
+corrected. `node_binding` overrides `trust_node`; when both are set, the
+constructor logs that `trust_node` is being ignored.
+
+For the `tcp` and `http` sinks, which have no node concept, `node_binding` is
+ignored.
+
+Dialer-side plugins (`tcp_chain` and `http_chain` sinks) accept the same block
+to pin the *server's* identity beyond hostname verification. This is deferred to
+phase 4 and does nothing before then.
+
+### Validation
+
+At plugin construction, before anything binds:
+
+- `type = "mtls"` on a listener requires `tls.enabled = true` and
+  `tls.client_auth = true`. Silently accepting an auth policy the transport
+  cannot enforce is the failure mode worth designing out.
+- `identity` must be one of the four modes.
+- Every entry in `allow_patterns` must compile.
+- `node_binding` must be one of the three values.
+
+Errors follow existing style: `auth: type "mtls" requires tls.client_auth`.
+
+### New package: `internal/authz`
+
+```go
+package authz
+
+// Policy is the compiled form of config.AuthOptions.
+type Policy struct { /* mode, identity selector, exact set, patterns, binding */ }
+
+// New compiles a policy. Returns (nil, nil) when auth is disabled, matching
+// the tlsx.Server / tlsx.Client convention so callers can nil-check.
+func New(o *config.AuthOptions) (*Policy, error)
+
+// Identity is the outcome of a successful authorization.
+type Identity struct {
+    Name   string // the selected certificate field
+    Method string // "mtls"
+}
+
+// Authorize extracts and checks the peer identity from a completed handshake.
+func (p *Policy) Authorize(cs *tls.ConnectionState) (Identity, error)
+
+// ResolveNode applies node_binding to a declared label.
+func (p *Policy) ResolveNode(declared string, id Identity) (string, error)
+
+// Stats reports counters for the sink/source stats map.
+func (p *Policy) Stats() map[string]any
+```
+
+This mirrors `internal/tlsx`: one small package that is the single seam between
+declarative config and a cross-cutting concern, with `(nil, nil)` for the
+disabled case so every call site is a nil check rather than a branch on config.
+
+### Enforcement Points
+
+Each plugin already has the right spot, and in two cases the code says so.
+
+**`tcp_chain` source** (`internal/source/tcpchain/tcpchain.go`, `handleConn`)
+
+Today: handshake → read hello → decode → resolve node from `trust_node` →
+create session. Insert authorization between the handshake and the hello read,
+so an unauthorized peer never gets a preamble parsed on its behalf, and replace
+the node resolution with `Policy.ResolveNode`.
+
+```go
+if tlsState != nil && s.authPolicy != nil {
+    id, err := s.authPolicy.Authorize(*tlsState)
+    if err != nil {
+        s.authRejected.Add(1)
+        s.logger.Warn("msg", "Connection rejected by auth policy",
+            "component", "tcp_chain_source", "remote_addr", remote, "error", err)
+        return // deferred cleanup closes conn
+    }
+    ident = id
+}
+// ... read and decode hello ...
+connNode, err = s.authPolicy.ResolveNode(hello.Node, ident)
+```
+
+**`http_chain` source** (`internal/source/httpchain/httpchain.go`, `handleIngest`)
+
+Per request, from `r.TLS`, before the body is read — an unauthorized sender
+should not get to stream 8 MiB into the process. Rejection is `403`, distinct
+from the `400` used for protocol errors, so a sender can tell "you are not
+allowed" from "your batch was malformed". `ResolveNode` then governs the
+`X-Logwisp-Node` header exactly as it governs the TCP hello.
+
+**`tcp` sink** (`internal/sink/tcp/tcp.go`, `handleConn`)
+
+There is already a comment marking the place: *"Password-auth extension point:
+preamble verification runs in handleConn post-handshake, pre-registration."*
+Authorization goes precisely there — after the explicit handshake, before the
+session is created and the client is registered — so an unauthorized peer never
+appears in the client map and never receives a broadcast.
+
+**`http` sink** (`internal/sink/http/http.go`, `Start`)
+
+Also already marked: *"Auth extension point: wrap mux with auth middleware once
+credentials land, e.g. handler = authMiddleware(cfg)(handler)."* A middleware
+around the mux covers both the stream and the status endpoint with one wrapper,
+and keeps the handlers themselves unaware of authorization.
+
+```go
+var handler http.Handler = mux
+if h.authPolicy != nil {
+    handler = authMiddleware(h.authPolicy, h.logger)(handler)
+}
+```
+
+The middleware rejects with `403` and no body detail — the status endpoint
+leaks host, port, and throughput counters, so a rejection should not leak policy
+shape on top of it.
+
+### Capabilities
+
+`core.CapAuth` is currently derived from `tlsConfig.ClientAuth`. It should
+reflect the auth policy instead:
+
+```go
+if s.authPolicy != nil {
+    caps = append(caps, core.CapAuth)
+}
+```
+
+`Pipeline.initSourceCapabilities` and `initSinkCapabilities` treat `CapAuth` as
+a no-op placeholder today. They become the natural place for a cross-cutting
+check: a plugin advertising `CapAuth` without `CapTLS` is a contradiction and
+should fail pipeline construction rather than start.
+
+### Observability
+
+Every authorization decision must be visible, because a silent deny is
+indistinguishable from a network fault at 3am.
+
+- **Session metadata** gains `auth_method` and `auth_identity` alongside the
+  existing `tls` and `tls_peer_cn`.
+- **Statistics** gain `auth_enabled`, `auth_rejected`, and `node_binding` in the
+  `details` map of every affected source and sink, so rejections show up in the
+  status reporter and the `http` sink's status endpoint.
+- **Logs** record a WARN per rejection with the remote address, the extracted
+  identity (or the reason extraction failed), and the policy that rejected it.
+  Accepted connections log the identity at INFO on the chain sources and at
+  DEBUG on the sinks, matching each plugin's existing verbosity.
+
+### Revocation
+
+Certificate revocation is deliberately handled by the allow-list rather than by
+CRL or OCSP:
+
+1. Remove the identity from `allow` / `allow_patterns`.
+2. `kill -HUP`.
+
+The reload path already rebuilds every pipeline, so the policy takes effect on
+the next connection and existing connections are dropped by the rebuild itself.
+This is one moving part instead of three, it needs no network calls on the
+handshake path, and it is exact — no window between revocation and the next CRL
+publication.
+
+A CRL file loaded next to `client_ca_file` and re-read on reload is a reasonable
+phase-4 addition for operators with existing CRL infrastructure. OCSP stapling
+is out of scope: it adds a network dependency to the handshake path for a
+system whose entire design is "never block".
+
+## Phases
+
+### Phase 1 — Identity and policy (no enforcement)
+
+- `internal/config`: add `AuthOptions` plus an `Auth *AuthOptions` field to the
+  four listener option structs.
+- `internal/tlsx`: add `PeerIdentity(cs tls.ConnectionState, mode string) string`
+  beside the existing `PeerCN`.
+- `internal/authz`: new package — `Policy`, `New`, `Authorize`, `ResolveNode`,
+  `Stats`.
+- Unit tests: identity extraction per mode, exact and pattern matching, empty
+  policy, malformed patterns, node binding in all three modes.
+
+Nothing behaves differently yet, which makes this phase safe to merge alone.
+
+### Phase 2 — Chain sources
+
+- Wire `authz` into `tcp_chain` and `http_chain` sources at the points above.
+- Replace the `trust_node` node resolution with `ResolveNode`.
+- Add validation, capabilities, statistics, and session metadata.
+- Add `test/mtls-chain-test.sh`, modelled on `test/chain-test.sh`: generate a
+  CA, a server certificate, and two client certificates with `openssl`; assert
+  that an allowed identity delivers entries, a non-allowed identity is rejected,
+  a peer with no certificate fails the handshake, and a peer declaring another
+  node's label is rejected under `assert` and corrected under `force`.
+
+This phase alone closes the node-spoofing hole, which is the sharpest of the
+three problems.
+
+### Phase 3 — Sinks
+
+- `tcp` sink: authorize in `handleConn` before registration.
+- `http` sink: `authMiddleware` around the mux, covering stream and status.
+- Extend the test script to cover both.
+
+### Phase 4 — Optional hardening
+
+- Dialer-side server identity pinning for the chain sinks.
+- CRL file support alongside `client_ca_file`, re-read on reload.
+- Certificate expiry warnings at startup and on reload — a leaf expiring inside
+  30 days logged at WARN, since nothing warns today.
+- Surface `tls_peer_cn` / `auth_identity` in the `http` sink's status output for
+  connected clients.
+
+## Compatibility
+
+No configuration breaks. Omitting the `auth` block, or setting
+`type = "none"`, reproduces current behaviour byte for byte: `Policy` is nil,
+every call site short-circuits, and `trust_node` continues to govern node
+labels.
+
+The one behavioural note for adopters: turning on `type = "mtls"` defaults
+`node_binding` to `force`, so entries from a peer whose certificate identity
+differs from its configured `node` label will be relabelled. That is the point
+of the feature, but it will move data between labels in a dashboard, so the
+release note should say it in those terms.
+
+## Estimated Cost
+
+| Phase | Files touched | Rough size |
+|-------|--------------|-----------|
+| 1 | `config/config.go`, `config/validate.go`, `tlsx/tlsx.go`, new `authz/` + tests | ~400 lines |
+| 2 | Two chain sources, new test script | ~200 lines |
+| 3 | `sink/tcp`, `sink/http`, test script extension | ~150 lines |
+| 4 | `tlsx`, both chain sinks, `sink/http` status | ~250 lines |
+
+Phases 1–2 are the security-relevant core; phase 3 closes the read-side
+exposure; phase 4 is discretionary.
+
+## Open Questions
+
+1. **Should an empty allow-list deny instead of allow?** Deny-by-default is the
+   safer instinct, but it makes `type = "mtls"` with no list a footgun that
+   silently drops all traffic. The proposal is allow-with-a-loud-startup-log;
+   the alternative is requiring a non-empty list and erroring at construction,
+   which is arguably better and costs one line.
+2. **Should `identity` accept a list of modes** (try `san_uri`, fall back to
+   `cn`)? Simpler as a single mode; heterogeneous PKI is the argument against.
+3. **Per-identity rate limits.** The natural follow-on once identity exists, and
+   the natural home for the per-IP limiting that was also removed. Deliberately
+   out of scope here so this feature stays reviewable.
+4. **Whether `assert` should reject or warn-and-correct.** As proposed it
+   rejects, which is unambiguous but turns a certificate/config mismatch into an
+   outage. `force` is the forgiving option, and it is the default.

doc/networking.md

@@ -1,37 +1,103 @@
 # Networking
 
-*Note: Under redesign*
+Everything LogWisp does over a socket, and the knobs that shape it. For
+certificates and trust see [Security](security.md); for multi-node topologies
+see [Chaining](chaining.md).
 
-## TLS Configuration
+## Address Family
 
-*Note: As of the latest architecture updates, network TLS, mTLS, and Rate Limiting features are undergoing a redesign and are currently acting as placeholders. The documentation below details the structure for future updates.*
+**All listeners bind `tcp4` and all dialers dial `tcp4`.** IPv6 is not
+supported, deliberately. An IPv6 client cannot connect and will simply see a
+connection failure.
 
-## Connection Management
+When testing locally use `127.0.0.1`, not `localhost` — the latter may resolve
+to `::1` and appear as an unexplained connection refusal.
 
-### TCP Keep-Alive
+## Network Plugins
+
+| Plugin | Role | Protocol | Purpose |
+|--------|------|----------|---------|
+| `tcp` sink | Listener | Raw stream | Broadcast formatted payloads to clients |
+| `http` sink | Listener | HTTP SSE | Browser-friendly live stream plus status JSON |
+| `tcp_chain` source | Listener | Chain v1 | Ingest a persistent NDJSON stream |
+| `http_chain` source | Listener | Chain v1 | Ingest NDJSON batches over POST |
+| `tcp_chain` sink | Dialer | Chain v1 | Forward entries over a persistent connection |
+| `http_chain` sink | Dialer | Chain v1 | Forward entries as batched POSTs |
+
+There is no port registry and no default port: `port` is required on every
+network plugin. There is also no cross-pipeline conflict detection — two sinks
+on the same port fail at bind time when the pipeline starts:
+
+```
+ERROR msg="Failed to start sink" error="tcp sink bind 0.0.0.0:9090: listen tcp4 0.0.0.0:9090: bind: address already in use"
+```
+
+## Timeouts
+
+Every network plugin exposes the deadlines relevant to its role. Zero means "no
+deadline" wherever the table says so.
+
+| Plugin | Option | Default | Bounds |
+|--------|--------|---------|--------|
+| `tcp` sink | `write_timeout_ms` | `5000` | One write to one client; a miss disconnects that client |
+| `http` sink | `write_timeout_ms` | `0` (none) | One SSE event write |
+| `tcp_chain` source | `hello_timeout_ms` | `10000` | Reading the protocol preamble |
+| `tcp_chain` source | `read_timeout_ms` | `0` (none) | Idle time between entries |
+| `http_chain` source | `read_timeout_ms` | `30000` | Reading a whole request body |
+| `tcp_chain` sink | `dial_timeout_ms` | `5000` | TCP connect |
+| `tcp_chain` sink | `write_timeout_ms` | `5000` | One line write |
+| `http_chain` sink | `request_timeout_ms` | `10000` | Dial plus write plus response |
+
+Fixed, non-configurable bounds:
+
+| Bound | Value | Applies to |
+|-------|-------|------------|
+| TLS handshake | 10 s | All TLS listeners and dialers |
+| HTTP read-header timeout | 10 s | `http` sink, `http_chain` source |
+| HTTP server shutdown grace | 2 s | `http` sink, `http_chain` source |
+| Max single entry line | 1 MiB | Chain listeners |
+
+The `http` sink deliberately leaves the server's `WriteTimeout` unset, since it
+would terminate long-lived SSE streams; per-event deadlines come from
+`write_timeout_ms` instead.
+
+## Connection Limits
+
+`max_connections` caps concurrent connections on the `tcp` sink, the `http`
+sink, and the `tcp_chain` source. `0` means unlimited.
+
+- Admission is a load-then-check, so a burst can over-admit by roughly one
+  connection. This is accepted, not a bug to work around.
+- On the `tcp` sink and `tcp_chain` source the count is taken at accept, so it
+  bounds concurrent TLS handshakes as well as established sessions.
+- Over-limit connections are closed immediately and counted in `rejected_conns`
+  (TCP) or `rejected_clients` (HTTP, which first answers `503`).
+
+The `http_chain` source has no connection cap; it bounds work with
+`max_body_bytes` and `read_timeout_ms` instead.
+
+There is **no** per-IP limiting and no IP allow/deny list. `flow.rate_limit` is
+a pipeline-wide entry rate limit, not a network-level one — it cannot
+distinguish or throttle an individual peer.
+
+## Keep-Alive
+
+TCP keep-alive is available on the `tcp` sink (for accepted connections) and the
+`tcp_chain` sink (for its outbound connection):
 
 ```toml
-[[pipelines.plugin_sinks]]
-id = "tcp_out"
-type = "tcp"
-[pipelines.plugin_sinks.config]
 keep_alive           = true
-keep_alive_period_ms = 30000  # 30 seconds
+keep_alive_period_ms = 30000
 ```
 
-### Connection Timeouts
+This is kernel-level keep-alive; it detects a dead peer but does not keep an
+application-level stream flowing. For that, use a heartbeat.
 
-```toml
+## Heartbeats
-[[pipelines.plugin_sinks]]
-id = "http_out"
-type = "http"
-[pipelines.plugin_sinks.config]
-write_timeout_ms = 10000  # 10 seconds
-```
 
-## Heartbeat Configuration
+Heartbeats are a **flow-level** feature, not a per-sink one. Enabling one
-
+injects a synthetic entry into the pipeline at a fixed interval; it reaches
-Keep connections alive with periodic heartbeats. Note that Heartbeat is a flow-level feature in the new architecture.
+every sink and traverses chain links as an ordinary structured entry.
 
 ```toml
 [pipelines.flow.heartbeat]
@@ -39,57 +105,81 @@ enabled = true
 interval_ms       = 30000
 include_timestamp = true
 include_stats     = false
-format = "comment"  # comment|event|json
+format            = "txt"      # txt | json | raw
 ```
 
-## Network Protocols
+Use it to keep idle SSE clients, TCP clients, and chain links from being
+reaped by intermediate NAT or proxy timeouts, and to make an idle pipeline
+visibly alive.
 
-### HTTP/HTTPS
+> `format = "comment"` (SSE `:` comment framing) is rejected by validation
+> despite appearing in older documentation and in a still-present code branch.
+> A pipeline configured with it fails to start.
 
-- HTTP/1.1 support
+## Reconnection
-- Persistent connections
-- Server-Sent Events (SSE)
 
-### TCP
+Chain sinks reconnect on their own. Both use exponential backoff between
+`backoff_min_ms` and `backoff_max_ms` with ±20 % jitter, and both are
+interruptible by shutdown.
 
-- Raw TCP sockets
+```toml
-- Newline-delimited protocol
+backoff_min_ms = 500
+backoff_max_ms = 30000
+```
 
-## Port Configuration
+The connection is established lazily, so an edge node starts cleanly even when
+its relay is down and connects as soon as the relay appears. Reconnect counts
+are reported in the sink's `reconnects` statistic.
 
-### Default Ports
+Server-side sinks (`tcp`, `http`) do not reconnect; clients are expected to
+retry. Browsers reconnect SSE streams automatically.
 
-| Service | Default Port | Protocol |
+## Protocol Details
-|---------|--------------|----------|
-| HTTP Sink | 8080 | HTTP |
-| TCP Sink | 9090 | TCP |
 
-### Port Conflict Prevention
+**HTTP sink (SSE)** — HTTP/1.1 in plaintext; HTTP/2 is negotiated via ALPN when
+TLS is enabled. Only `GET` is routed to the stream and status paths. Each event
+is framed as one `data:` line per newline in the payload, so multi-line entries
+stream intact. Response headers set `Cache-Control: no-cache`,
+`X-Accel-Buffering: no`, and `Access-Control-Allow-Origin: *`.
 
-LogWisp validates port usage at startup:
+**TCP sink** — raw payload bytes, no framing added by the sink. Whether entries
-- Detects port conflicts across pipelines
+are newline-delimited depends on the formatter.
-- Prevents duplicate bindings
+
+**Chain transports** — see [Chaining](chaining.md) for the hello preamble,
+headers, and entry encoding.
 
 ## Troubleshooting
 
-### Common Issues
+**Connection refused**
+- Confirm the pipeline started; a bind failure is logged at ERROR.
+- Confirm you are dialing IPv4. `localhost` may resolve to `::1`.
+- Check the port is not already bound by another pipeline in the same process.
 
-**Connection Refused**
+**TLS handshake failure**
-- Check firewall rules
+- `client didn't provide a certificate` — the listener has `client_auth = true`
-- Verify service is running
+  and the dialer has no `cert_file`/`key_file`.
-- Confirm correct port/host
+- `certificate signed by unknown authority` — the dialer's `ca_file` does not
+  contain the issuer of the server certificate, or the listener's
+  `client_ca_file` does not contain the issuer of the client certificate.
+- `certificate is not valid for any names` / SAN mismatch — the dialed `host`
+  is not covered by the server certificate's SANs; set `server_name`.
+- `protocol version not supported` — one side is pinned to `min_version = "1.3"`
+  and the other cannot negotiate it.
+- Handshake failures appear as WARN with the remote address, and increment
+  `tls_handshake_errors`.
 
-**TLS Handshake Failure**
+**Entries not arriving over a chain link**
-- Verify certificate validity
+- Check the sink's `connected` statistic and its `reconnects` count.
-- Check certificate chain
+- Check the source's `parse_errors` — a version skew shows up here.
-- Confirm TLS versions match
+- On `http_chain`, remember entries wait up to `flush_interval_ms` before a
+  batch is sent.
 
-**Rate Limit Exceeded**
+**Clients connect but see nothing**
-- Adjust rate limit parameters
+- The pipeline may be filtering everything out; check `flow.filters` stats.
-- Add IP to whitelist
+- The rate limiter may be dropping everything; check `rate_limiter` stats.
-- Implement client-side throttling
+- Nothing may be arriving from the sources; check source `total_entries`.
 
-**Connection Timeout**
+**Entries missing under load**
-- Increase timeout values
+- Compare `dropped_writes` (per-client queue full — raise
-- Check network latency
+  `client_buffer_size`) against `total_dropped_by_sink` (sink input queue full
-- Verify keep-alive settings
+  — raise `buffer_size` or reduce sink latency).

doc/operations.md

@@ -1,130 +1,161 @@
 # Operations Guide
 
-Running, monitoring, and maintaining LogWisp in production.
+Running, monitoring, and maintaining LogWisp.
 
-*Note: TLS, acccess control under redesign*
+## Starting
-
-## Starting LogWisp
-
-### Manual Start
 
 ```bash
-# Foreground with default config
+# foreground, explicit config
+logwisp -c /etc/logwisp/logwisp.toml
+
+# no config: built-in demo pipeline (random source -> stdout)
 logwisp
-
-# Background mode
-logwisp --background
-
-# With specific configuration
-logwisp --config /etc/logwisp/production.toml
 ```
 
-### Service Management
+There is no built-in daemon mode. Run LogWisp in the foreground under a
+supervisor — systemd, rc.d, or a container runtime — which is where restart,
+log capture, and resource limits belong. See [Installation](installation.md).
+
+**systemd**
 
-**Linux (systemd):**
 ```bash
 sudo systemctl start logwisp
-sudo systemctl stop logwisp
-sudo systemctl restart logwisp
 sudo systemctl status logwisp
+sudo journalctl -u logwisp -f
 ```
 
-**FreeBSD (rc.d):**
+**FreeBSD rc.d**
+
 ```bash
 sudo service logwisp start
-sudo service logwisp stop
-sudo service logwisp restart
 sudo service logwisp status
 ```
 
-## Configuration Management
+## Configuration Changes
 
-### Hot Reload
+### Hot reload
 
-Enable automatic configuration reload:
 ```toml
-config_auto_reload = true
+auto_reload = true
 ```
 
-Or via command line:
+or send a signal:
-```bash
-logwisp --config-auto-reload
-```
 
-Trigger manual reload:
 ```bash
 kill -HUP $(pidof logwisp)
-# or
-kill -USR1 $(pidof logwisp)
 ```
 
-### Configuration Validation
+Reload constructs a new service from the new configuration **before** tearing
+the old one down, so a broken configuration leaves the running service intact
+and logs the failure:
+
+```
+ERROR msg="Failed to bootstrap new service, keeping old service running" error=...
+```
+
+What reload does *not* do:
+
+- Re-apply `logging.*`; application logging is configured once at startup.
+- Preserve connections. Listeners close and reopen, and every SSE, TCP, and
+  chain client is disconnected. Chain sinks reconnect on their own backoff;
+  browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
+- Reload certificates without a reload — certificate files are read at plugin
+  construction, so rotation requires `SIGHUP`.
+
+Plan reloads on a busy relay the way you would plan a restart.
+
+### Checking a configuration
+
+There is no validate-only mode. To check a file, start it with debug logging and
+watch for pipeline startup:
 
-Test configuration without starting:
 ```bash
-logwisp --config test.toml --quiet --status-reporter=false
+logwisp -c candidate.toml --logging.level=debug --logging.output=stderr
 ```
 
-Check for errors:
+Success looks like `Created source instance`, `Created sink instance`, and
-- Port conflicts
+`Starting pipeline` for each pipeline. Failures name the pipeline and the
-- Invalid patterns
+offending key:
-- Missing required fields
+
-- File permissions
+```
+ERROR msg="Failed to create pipeline" pipeline=app error="failed to create sink out: port: must be 1-65535, got 0"
+```
+
+Remember that most validation lives in plugin constructors, so a config only
+proves itself when the pipeline is actually built.
 
 ## Monitoring
 
-### Status Reporter
+### Status reporter
 
-Built-in periodic status logging (30-second intervals):
+Enabled by default, every 30 seconds. It logs at **DEBUG**, so it produces
+nothing unless `logging.level = "debug"` — a common surprise.
 
-```
-[INFO] Status report active_pipelines=2 time=15:04:05
-[INFO] Pipeline status pipeline=app entries_processed=10523
-[INFO] Pipeline status pipeline=system entries_processed=5231
-```
-
-Disable if not needed:
 ```toml
-disable_status_reporter = true
+status_reporter = true
+
+[logging]
+level = "debug"
 ```
 
-### HTTP Status Endpoint
+It emits a service summary and then walks each pipeline, flattening scalar
+statistics into log fields and recursing into flow, rate limiter, filter,
+source, and sink stats.
+
+Disable with `status_reporter = false`.
+
+### HTTP status endpoint
+
+When a pipeline has an `http` sink:
 
-When using HTTP sink:
 ```bash
-curl http://localhost:8080/status | jq .
+curl -s http://127.0.0.1:8080/status | jq .
 ```
 
-Response structure:
 ```json
 {
-  "uptime": "2h15m30s",
+  "service": "LogWisp",
-  "pipelines": {
+  "version": "v0.16.0",
-    "default": {
+  "instance_id": "sse",
-      "sources": 1,
+  "server": {
-      "sinks": 2,
+    "type": "http",
-      "processed": 15234,
+    "host": "0.0.0.0",
-      "filtered": 523,
+    "port": 8080,
-      "dropped": 12
+    "tls": false,
-    }
+    "active_clients": 3,
+    "buffer_size": 1000,
+    "uptime_seconds": 8130
+  },
+  "endpoints": { "stream": "/stream", "status": "/status" },
+  "statistics": {
+    "total_processed": 15234,
+    "dropped_writes": 12,
+    "rejected_clients": 0
   }
 }
 ```
 
-### Metrics Collection
+This endpoint is scoped to one sink, not to the whole process, and it is
+**unauthenticated**. Bind it to a trusted interface.
 
-Track via logs:
+### Metrics worth watching
-- Total entries processed
+
-- Entries filtered
+| Metric | Where | Meaning if rising |
-- Entries dropped
+|--------|-------|-------------------|
-- Active connections
+| `dropped_entries` | source | Downstream cannot keep up with the source |
-- Buffer utilization
+| `total_dropped` | flow | Rate limit or filters are discarding entries (often intended) |
+| `total_dropped_by_sink` | pipeline | A sink's input queue is full |
+| `dropped_writes` | tcp/http sink | A specific client is too slow |
+| `rejected_conns` / `rejected_clients` | tcp/http sink, tcp_chain source | `max_connections` is being hit |
+| `tls_handshake_errors` | tcp sink, tcp_chain source | Certificate or version mismatch, or scanning |
+| `parse_errors` | chain source | Protocol or version skew upstream |
+| `reconnects` | chain sink | Unstable link or a flapping downstream |
+| `dropped_batches` | http_chain sink | Downstream rejecting batches permanently |
+| `synthesized` | chain sink | Events reaching the sink without structure |
 
 ## Log Management
 
-### LogWisp's Operational Logs
+LogWisp's own operational log:
-
-Configuration for LogWisp's own logs:
 
 ```toml
 [logging]
@@ -135,193 +166,140 @@ level = "info"
 directory         = "/var/log/logwisp"
 name              = "logwisp"
 max_size_mb       = 100
-retention_hours = 168
+max_total_size_mb = 1000
+retention_hours   = 168.0
 ```
 
-### Log Rotation
+Rotation is automatic on size, with a total-size cap and a retention window.
+There is no signal to reopen log files, so do not move files out from under
+LogWisp and expect it to reattach — let it rotate, or restart it.
 
-Automatic rotation based on:
+Production level: `info`, or `warn` on a busy relay. Avoid `debug` under load:
-- File size threshold
+the filter stage logs several lines per entry evaluated.
-- Total size limit
-- Retention period
-
-Manual rotation:
-```bash
-# Move current log
-mv /var/log/logwisp/logwisp.log /var/log/logwisp/logwisp.log.1
-# Send signal to reopen
-kill -USR1 $(pidof logwisp)
-```
-
-### Log Levels
-
-Operational log levels:
-- **debug**: Detailed debugging information
-- **info**: General operational messages
-- **warn**: Warning conditions
-- **error**: Error conditions
-
-Production recommendation: `info` or `warn`
 
 ## Performance Tuning
 
-### Buffer Sizing
+### Buffers
 
-Adjust buffers based on load:
+Raise `buffer_size` when `total_dropped_by_sink` is climbing but the sink itself
+is healthy — that is a burst-absorption problem. Raise `client_buffer_size` when
+`dropped_writes` is climbing for network sinks; that is a slow-consumer problem,
+and a bigger buffer only buys time.
 
 ```toml
-[[pipelines.plugin_sources]]
+[pipelines.plugin_sinks.config]
-id = "file_in"
+buffer_size        = 5000
-type = "file"
+client_buffer_size = 1024
-[pipelines.plugin_sources.config]
 ```
 
-### Rate Limiting
+### Rate limiting
-
-Protect against overload:
 
 ```toml
 [pipelines.flow.rate_limit]
-rate = 1000.0        # Entries per second
+rate                 = 1000.0
-burst = 2000.0       # Burst capacity
+burst                = 2000.0
-policy = "drop"      # Drop excess entries
+policy               = "drop"
+max_entry_size_bytes = 65536
 ```
 
+Two behaviours to keep in mind: the limiter does not exist at all when
+`rate <= 0`, and `policy = "pass"` short-circuits the size cap as well as the
+rate check. Enforcing `max_entry_size_bytes` therefore requires `rate > 0` and
+`policy = "drop"`.
+
+### Formatting
+
+`raw` is the cheapest and skips sanitization; `json` costs the most. The
+formatter serializes on a mutex, so it is the one shared bottleneck in a
+pipeline — splitting work across pipelines parallelizes it.
+
+### Chain batching
+
+`http_chain` trades latency for efficiency. Lower `flush_interval_ms` for
+freshness, raise `max_batch_count` and `max_batch_bytes` for throughput. Use
+`tcp_chain` when per-entry latency matters.
+
 ## Troubleshooting
 
-### Common Issues
+**Nothing appears at the sink**
 
-**High Memory Usage**
+Walk the pipeline in order and read the counters: source `total_entries` (is
-- Check buffer sizes
+anything being produced?), flow `total_dropped` (filters or rate limit?),
-- Monitor goroutine count
+pipeline `total_dropped_by_sink` (sink backed up?), sink `total_processed`.
-- Review retention settings
 
-**Dropped Entries**
+**File source reads nothing**
-- Increase buffer sizes
-- Add rate limiting
-- Check sink performance
 
-**Connection Errors**
+- The watcher seeks to end-of-file on start; only content appended afterwards is
-- Verify network connectivity
+  read. Positions are in memory, so a restart re-seeks to end and anything
-- Check firewall rules
+  written during the downtime is lost.
-- Review TLS certificates
+- `pattern` is a filename glob with `*` and `?` only, and matching is not
+  recursive.
+- `check_interval_ms` governs how quickly a *new file* is noticed; tailing an
+  open file polls at a fixed 100 ms.
 
-### Debug Mode
+**High memory use**
 
-Enable detailed logging:
+Buffers are bounded, so unbounded growth almost always means many buffers:
-```bash
+count sinks × `buffer_size`, plus clients × `client_buffer_size`. A `tcp_chain`
-logwisp --logging.level=debug --logging.output=stderr
+sink blocked on an unreachable downstream also holds its full input queue.
-```
 
-### Health Checks
+**Chain link not delivering**
 
-Implement external monitoring:
+Check `connected` and `reconnects` on the sink, `parse_errors` on the source,
-```bash
+and remember `http_chain` waits up to `flush_interval_ms`. For TLS problems see
-#!/bin/bash
+[Networking](networking.md#troubleshooting).
-# Health check script
-if ! curl -sf http://localhost:8080/status > /dev/null; then
-  echo "LogWisp health check failed"
-  exit 1
-fi
-```
 
-## Backup and Recovery
+**Environment variable override has no effect**
 
-### Configuration Backup
+LogWisp currently reads these **without** the `LOGWISP_` prefix — `QUIET`,
-
+`LOGGING_LEVEL`, and so on. Array-indexed paths cannot be set from the
-```bash
+environment or the command line at all.
-# Backup configuration
-cp /etc/logwisp/logwisp.toml /backup/logwisp-$(date +%Y%m%d).toml
-
-# Version control
-git add /etc/logwisp/
-git commit -m "LogWisp config update"
-```
-
-### State Recovery
-
-LogWisp maintains minimal state:
-- File read positions (automatic)
-- Connection state (automatic)
-
-Recovery after crash:
-1. Service automatically restarts (systemd/rc.d)
-2. File sources resume from last position
-3. Network sources accept new connections
-4. Clients reconnect automatically
 
 ## Security Operations
 
-### Certificate Management
+**Certificate rotation**
 
-Monitor certificate expiration:
 ```bash
-openssl x509 -in /path/to/cert.pem -noout -enddate
+openssl x509 -in /etc/logwisp/tls/relay.crt -noout -enddate
 ```
 
-Rotate certificates:
+Certificates load at plugin construction, so rotation is: write the new files,
-1. Generate new certificates
+then `kill -HUP`. Automate the expiry check; nothing in LogWisp warns you.
-2. Update configuration
-3. Reload service (SIGHUP)
 
-### Access Auditing
+**Access review**
 
-Monitor access patterns:
+With mTLS, any certificate signed by the configured `client_ca_file` is
-- Review connection logs
+accepted — there is no per-identity allow-list, so "access review" means
-- Monitor rate limit hits
+reviewing what your CA has issued. Peer Common Names are recorded in session
+metadata but are not surfaced in statistics and are not used for authorization.
+See [Security](security.md) and the
+[mTLS authentication plan](mtls-auth-plan.md).
+
+**Secret leakage**
+
+Filters are the only redaction mechanism, and they drop whole entries rather
+than masking parts of them. See [Filters](filters.md#common-recipes).
 
 ## Maintenance
 
-### Planned Maintenance
+**Upgrades**
 
-1. Notify users of maintenance window
+1. Read the changelog for configuration-schema changes.
-2. Stop accepting new connections
+2. Start the new binary against the current configuration in a scratch
-3. Drain existing connections
+   environment.
-4. Perform maintenance
+3. Stop the old process, install the new binary, start it.
-5. Restart service
+4. Confirm each pipeline started and that counters are advancing.
 
-### Upgrade Process
+**Backup**
 
-1. Download new version
+Configuration files and TLS material are the only durable state worth backing
-2. Test with current configuration
+up. LogWisp keeps no persistent runtime state: file read positions live in
-3. Stop old version
+memory, connections are re-established on restart, and in-flight entries are
-4. Install new version
+lost.
-5. Start service
-6. Verify operation
 
-### Cleanup Tasks
+**Redundancy**
 
-Regular maintenance:
+Because there is no persistence, availability comes from topology, not from
-- Remove old log files
+LogWisp itself. Give each edge two chain sinks pointing at two relays if you
-- Clean temporary files
+need to survive a relay outage, and accept that this duplicates entries
-- Verify disk space
+downstream.
-- Update documentation
-
-## Disaster Recovery
-
-### Backup Strategy
-
-- Configuration files: Daily
-- TLS certificates: After generation
-- Authentication credentials: Secure storage
-
-### Recovery Procedures
-
-Service failure:
-1. Check service status
-2. Review error logs
-3. Verify configuration
-4. Restart service
-
-Data loss:
-1. Restore configuration from backup
-2. Regenerate certificates if needed
-3. Recreate authentication credentials
-4. Restart service
-
-### Business Continuity
-
-- Run multiple instances for redundancy
-- Use load balancer for distribution
-- Implement monitoring alerts
-- Document recovery procedures

doc/security.md

@@ -1,4 +1,242 @@
 # Security
 
-*Note: Security features like mTLS and IP-Based Access Control are currently under redesign and act as placeholders in the new architecture. Future versions will reintroduce full security capabilities.*
+This page covers LogWisp's transport security: what it protects, how to
+configure it, and — equally important — what it does not yet do.
 
+## Current State
+
+| Capability | Status |
+|------------|--------|
+| TLS 1.2 / 1.3 on all network sources and sinks | Implemented |
+| Server certificate verification by dialers | Implemented |
+| Mutual TLS (client certificate required and verified) | Implemented at the transport layer |
+| Peer identity (certificate CN) recorded per session | Implemented |
+| Authorization from peer identity (CN allow-lists, node binding) | **Not implemented** — see [mtls-auth-plan.md](mtls-auth-plan.md) |
+| Password, token, or SCRAM authentication | **Removed**; not currently available |
+| IP allow/deny lists, per-IP connection or request limits | **Not implemented** |
+| Authentication on the `http` sink's stream and status endpoints | **Not implemented** |
+
+Earlier releases carried basic-auth, bearer-token, and SCRAM authentication.
+Those were removed during the move to the plugin/flow architecture and the
+switch to standard-library networking. Only certificate-based transport
+security survived that transition.
+
+## The TLS Block
+
+One option shape serves both roles, so the configuration reads the same
+wherever it appears. Which keys matter depends on whether the plugin listens or
+dials.
+
+```toml
+[pipelines.plugin_sources.config.tls]     # or plugin_sinks.config.tls
+enabled              = false
+cert_file            = ""
+key_file             = ""
+client_auth          = false
+client_ca_file       = ""
+ca_file              = ""
+server_name          = ""
+insecure_skip_verify = false
+min_version          = "1.3"
+```
+
+| Option | Role | Default | Description |
+|--------|------|---------|-------------|
+| `enabled` | both | `false` | Master switch; when false the whole block is ignored |
+| `cert_file` | both | — | Local certificate. **Required** for listeners; optional client identity for dialers |
+| `key_file` | both | — | Private key for `cert_file`. Must be set together with it |
+| `client_auth` | listener | `false` | Require and verify a client certificate (mTLS) |
+| `client_ca_file` | listener | — | CA bundle used to verify client certificates. **Required** when `client_auth` is true |
+| `ca_file` | dialer | system store | CA bundle used to verify the server certificate |
+| `server_name` | dialer | the configured `host` | SNI and certificate name to verify against |
+| `insecure_skip_verify` | dialer | `false` | Disable server verification |
+| `min_version` | both | `"1.3"` | `"1.2"` or `"1.3"` |
+
+**Roles by plugin:**
+
+| Plugin | Role | Keys that apply |
+|--------|------|-----------------|
+| `tcp` sink, `http` sink | Listener | `cert_file`, `key_file`, `client_auth`, `client_ca_file`, `min_version` |
+| `tcp_chain` source, `http_chain` source | Listener | same as above |
+| `tcp_chain` sink, `http_chain` sink | Dialer | `ca_file`, `server_name`, `insecure_skip_verify`, `cert_file`, `key_file`, `min_version` |
+
+> `min_version` takes `"1.2"` or `"1.3"`. The older `"TLS1.2"` spelling from
+> pre-restructure releases is rejected. There is no `max_version` and no
+> `cipher_suites` option; TLS 1.3 suites are not configurable in Go, and the
+> 1.2 defaults are the standard library's.
+
+### Validation
+
+Misconfiguration fails at plugin construction, before the pipeline starts:
+
+- a listener with `enabled = true` and no `cert_file`/`key_file`
+- `client_auth = true` with no `client_ca_file`
+- a dialer with only one of `cert_file` / `key_file`
+- a certificate or key that will not load, or a CA file containing no
+  certificates
+- a `min_version` that is neither `"1.2"` nor `"1.3"`
+
+## Enabling mTLS
+
+### 1. Generate a CA and certificates
+
+LogWisp no longer ships a certificate-generation subcommand; the `logwisp tls`
+command was removed with the rest of the CLI restructure. Use `openssl`,
+`cfssl`, `step-cli`, or your existing PKI.
+
+```bash
+# CA
+openssl req -x509 -newkey rsa:4096 -nodes -days 3650 \
+  -keyout ca.key -out ca.crt -subj "/CN=LogWisp CA"
+
+# Relay (server) certificate — SAN must match how clients address it
+openssl req -newkey rsa:2048 -nodes -keyout relay.key -out relay.csr \
+  -subj "/CN=relay.internal"
+openssl x509 -req -in relay.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
+  -out relay.crt -days 825 \
+  -extfile <(printf "subjectAltName=DNS:relay.internal\nextendedKeyUsage=serverAuth")
+
+# Edge (client) certificate — CN identifies the node
+openssl req -newkey rsa:2048 -nodes -keyout edge-01.key -out edge-01.csr \
+  -subj "/CN=edge-01"
+openssl x509 -req -in edge-01.csr -CA ca.crt -CAkey ca.key -CAcreateserial \
+  -out edge-01.crt -days 825 \
+  -extfile <(printf "extendedKeyUsage=clientAuth")
+```
+
+The server certificate's SAN must cover the address clients dial. Dialers seed
+`ServerName` from the configured `host`, so an IP literal in `host` requires an
+IP SAN, and a DNS name requires a DNS SAN. Override with `server_name` when the
+dialed address and the certificate name legitimately differ.
+
+### 2. Configure the listener
+
+```toml
+[pipelines.plugin_sources.config.tls]
+enabled        = true
+cert_file      = "/etc/logwisp/tls/relay.crt"
+key_file       = "/etc/logwisp/tls/relay.key"
+client_auth    = true
+client_ca_file = "/etc/logwisp/tls/ca.crt"
+min_version    = "1.3"
+```
+
+### 3. Configure the dialer
+
+```toml
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+ca_file   = "/etc/logwisp/tls/ca.crt"
+cert_file = "/etc/logwisp/tls/edge-01.crt"
+key_file  = "/etc/logwisp/tls/edge-01.key"
+min_version = "1.3"
+```
+
+### 4. Verify
+
+Startup logs report both flags:
+
+```
+INFO msg="TCP chain source initialized" ... tls=true mtls=true
+INFO msg="TCP chain sink initialized"   ... tls=true mtls=true
+```
+
+A client that presents no certificate is refused during the handshake:
+
+```
+WARN msg="TLS handshake failed" component=tcp_chain_source
+     remote_addr=127.0.0.1:53840 error="tls: client didn't provide a certificate"
+```
+
+Handshake failures are counted in the `tls_handshake_errors` statistic on the
+`tcp` sink and the `tcp_chain` source.
+
+## What mTLS Currently Buys You
+
+With `client_auth = true`, the transport enforces:
+
+- the peer holds a certificate chaining to `client_ca_file`
+- the certificate is within its validity window and not structurally broken
+- the peer holds the matching private key
+
+That is a real membership check: an attacker without a CA-issued certificate
+cannot connect at all.
+
+## What It Does Not Buy You
+
+**Any** valid certificate from the configured CA is accepted. LogWisp extracts
+the peer's Common Name into session metadata (`tls_peer_cn`) but never consults
+it, so within one CA there is no way to express:
+
+- "only `edge-01` and `edge-02` may connect to this ingest port"
+- "the node label `edge-01` may only be claimed by the holder of the `edge-01`
+  certificate"
+- "this certificate may connect but only at this rate"
+
+Two consequences follow.
+
+1. **A compromised edge can impersonate any other edge.** With
+   `trust_node = true` (the default) a peer declares its own node label. Any
+   certificate holder can claim `edge-99`, or `relay`, and downstream consumers
+   will attribute its entries accordingly. Setting `trust_node = false` replaces
+   the label with the remote address, which is coarse but not forgeable at the
+   application layer.
+2. **Revocation is CA-wide.** With no CRL or OCSP checking and no per-identity
+   allow-list, withdrawing one node's access means re-issuing the CA or rotating
+   the CA bundle for every peer.
+
+Closing both gaps is the subject of the
+[mTLS authentication plan](mtls-auth-plan.md).
+
+## Unauthenticated Surfaces
+
+These endpoints have no access control at all. Bind them to a trusted interface
+or front them with an authenticating proxy.
+
+| Surface | Exposure |
+|---------|----------|
+| `http` sink `stream_path` | Full log stream, with `Access-Control-Allow-Origin: *`, so any browser origin can read it |
+| `http` sink `status_path` | Host, port, TLS flag, uptime, client counts, throughput counters |
+| `tcp` sink | Full log stream to any client that connects |
+
+`max_connections` bounds concurrency on all three but does not distinguish
+callers.
+
+## Operational Guidance
+
+**Certificates**
+
+- Use a dedicated CA for LogWisp so its trust decisions stay independent.
+- Keep leaf lifetimes short (90–825 days) and automate renewal.
+- Key files should be `0600` and owned by the service account.
+- Rotation requires a reload (`SIGHUP`), because certificates are loaded once at
+  plugin construction; there is no on-disk watch for certificate files.
+- Check expiry: `openssl x509 -in relay.crt -noout -enddate`.
+
+**Deployment**
+
+- Prefer `min_version = "1.3"`. Drop to `"1.2"` only for a peer that genuinely
+  cannot do 1.3.
+- Never enable `insecure_skip_verify` outside a lab; it disables server
+  verification entirely and makes the connection trivially interceptable.
+- Bind listeners to specific interfaces rather than `0.0.0.0` where you can.
+- Use `trust_node = false` on any ingest port reachable from a network you do
+  not fully control.
+- Run LogWisp as an unprivileged user with write access only to its own log and
+  configuration directories.
+
+**Log content**
+
+Logs routinely contain secrets that were never meant to leave the host. Filters
+are the available tool:
+
+```toml
+[[pipelines.flow.filters]]
+type = "exclude"
+patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret"]
+```
+
+Choose a sanitizer policy that matches the sink — `json` for JSON output,
+`txt` for files and consoles — so control characters in log data cannot break
+framing or inject terminal escapes downstream. See
+[Formatters](formatters.md).

doc/sinks.md

@@ -1,79 +1,107 @@
 # Output Sinks
 
-LogWisp sinks deliver processed log entries to various destinations.
+Sinks consume `core.TransportEvent` values — a formatted `Payload` plus the
+original structured `Entry` — and deliver them somewhere. Each sink is declared
+as a `[[pipelines.plugin_sinks]]` entry with an `id`, a `type`, and a
+type-specific `config` table.
 
-## Sink Types
+Registered types: `console`, `file`, `http`, `tcp`, `null`, `tcp_chain`,
+`http_chain`.
 
-### Console Sink
+Dispatch into a sink is non-blocking. A sink whose input queue is full drops the
+event *for itself only* and the pipeline counts it in `total_dropped_by_sink`;
+sibling sinks are unaffected.
 
-Output to stdout/stderr.
+---
+
+## console
+
+Writes formatted payloads to stdout or stderr.
 
 ```toml
 [[pipelines.plugin_sinks]]
-id = "console_out"
+id   = "stdout"
 type = "console"
 [pipelines.plugin_sinks.config]
-target = "stdout"  # stdout|stderr|split
+target      = "stdout"
 buffer_size = 1000
 ```
 
-**Configuration Options:**
-
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `target` | string | "stdout" | Output target (stdout/stderr/split) |
+| `target` | string | `stdout` | `stdout` or `stderr` |
-| `buffer_size` | int | 1000 | Internal buffer size |
+| `buffer_size` | int | `1000` | Sink input queue depth |
 
-**Target Modes:**
+> `split` is **not** a valid target for this sink and is rejected at startup.
-- **stdout**: All output to standard output
+> Level-based splitting exists only for LogWisp's own application log
-- **stderr**: All output to standard error
+> (`logging.output = "split"`).
-- **split**: INFO/DEBUG to stdout, WARN/ERROR to stderr
 
-### File Sink
+Payloads are written verbatim; the sink adds no framing. Whether entries are
+newline-terminated is decided by the formatter.
 
-Write logs to rotating files.
+---
+
+## file
+
+Rotating file writer.
 
 ```toml
 [[pipelines.plugin_sinks]]
-id = "file_out"
+id   = "archive"
 type = "file"
 [pipelines.plugin_sinks.config]
-directory = "./logs"
+directory         = "/var/log/logwisp"
 name              = "output"
 max_size_mb       = 100
 max_total_size_mb = 1000
-min_disk_free_mb = 500
+min_disk_free_mb  = 0
 retention_hours   = 168.0
 buffer_size       = 1000
-flush_interval_ms = 1000
+flush_interval_ms = 100
 ```
 
-**Configuration Options:**
-
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `directory` | string | Required | Output directory |
+| `directory` | string | **required** | Output directory |
-| `name` | string | Required | Base filename |
+| `name` | string | **required** | Base filename |
-| `max_size_mb` | int | 100 | Rotation threshold |
+| `max_size_mb` | int | `100` | Rotate when the active file reaches this size |
-| `max_total_size_mb` | int | 1000 | Total size limit |
+| `max_total_size_mb` | int | `1000` | Cap across all rotated files |
-| `min_disk_free_mb` | int | 500 | Minimum free disk space |
+| `min_disk_free_mb` | int | `0` | Free-space floor before writing; `0` = no floor |
-| `retention_hours` | float | 168 | Delete files older than |
+| `retention_hours` | float | `168.0` | Delete rotated files older than this |
-| `buffer_size` | int | 1000 | Internal buffer size |
+| `buffer_size` | int | `1000` | Sink input queue depth |
-| `flush_interval_ms` | int | 1000 | Force flush interval |
+| `flush_interval_ms` | int | `100` | Forced flush interval |
 
-**Features:**
+> `min_disk_free_mb` has an unusual default. The constructor replaces only
-- Automatic rotation on size
+> *negative* values with `100`; leaving the key unset yields `0`, which means no
-- Retention management
+> free-space floor. Set it explicitly if you want one.
-- Disk space monitoring
-- Periodic flushing
 
-### HTTP Sink
+The sink drives an internal writer configured for raw output with timestamps and
+levels disabled, so what lands on disk is exactly the formatted payload.
 
-SSE (Server-Sent Events) streaming server.
+---
+
+## null
+
+Discards everything, counting entries and bytes. Useful for benchmarking a
+source or flow in isolation.
 
 ```toml
 [[pipelines.plugin_sinks]]
-id = "http_out"
+id   = "discard"
+type = "null"
+```
+
+No options. The input queue is fixed at 1000.
+
+---
+
+## http
+
+Server-Sent Events stream plus a JSON status endpoint.
+
+```toml
+[[pipelines.plugin_sinks]]
+id   = "sse"
 type = "http"
 [pipelines.plugin_sinks.config]
 host               = "0.0.0.0"
@@ -84,28 +112,59 @@ buffer_size = 1000
 client_buffer_size = 256
 write_timeout_ms   = 0
 max_connections    = 0
-```
 
-**Configuration Options:**
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+cert_file = "/etc/logwisp/tls/server.crt"
+key_file  = "/etc/logwisp/tls/server.key"
+```
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `host` | string | "0.0.0.0" | Bind address |
+| `host` | string | `0.0.0.0` | Bind address; IPv4 only |
-| `port` | int | Required | Listen port |
+| `port` | int | **required** | Listen port |
-| `stream_path` | string | "/stream" | SSE stream endpoint |
+| `stream_path` | string | `/stream` | SSE endpoint; must start with `/` |
-| `status_path` | string | "/status" | Status endpoint |
+| `status_path` | string | `/status` | Status endpoint; must start with `/` and differ from `stream_path` |
-| `buffer_size` | int | 1000 | Sink input queue size |
+| `buffer_size` | int | `1000` | Sink input queue depth |
-| `client_buffer_size` | int | 256 | Per-client send queue size |
+| `client_buffer_size` | int | `256` | Per-client send queue depth |
-| `write_timeout_ms` | int | 0 | Write deadline per event (0 = none) |
+| `write_timeout_ms` | int | `0` | Per-event write deadline; `0` = none |
-| `max_connections` | int | 0 | Concurrent connection cap (0 = unlimited) |
+| `max_connections` | int | `0` | Concurrent stream cap; `0` = unlimited |
+| `tls` | table | — | Listener TLS; see [Security](security.md) |
 
-### TCP Sink
+**Behaviour**
 
-TCP streaming server for debugging and raw client forwarding.
+- Only `GET` is routed to either path; anything else gets `405`.
+- On connect the client receives an `event: connected` frame carrying its
+  client id, session id, sink instance id, endpoint paths, and buffer size.
+- Payloads are framed per the SSE spec, one `data:` line per newline in the
+  payload, so multi-line entries stream correctly.
+- The server sets no `WriteTimeout` (that would kill long-lived streams);
+  per-write deadlines come from `write_timeout_ms` via `http.ResponseController`.
+- A client whose send queue is full has that event dropped
+  (`dropped_writes`); it is not disconnected.
+- Clients whose session has been idle-expired by the session manager are
+  evicted by the broker.
+- On shutdown, connected clients receive
+  `event: disconnect / data: {"reason":"server_shutdown"}`.
+- HTTP/2 is negotiated via ALPN when TLS is enabled; plaintext is HTTP/1.1.
+
+**Status endpoint** returns service and version identity, host, port, TLS flag,
+active client count, buffer size, uptime, endpoint paths, and the
+`total_processed` / `dropped_writes` / `rejected_clients` counters.
+
+> Both endpoints are unauthenticated, and the stream response carries
+> `Access-Control-Allow-Origin: *`, so any web origin can read it. Bind to a
+> trusted interface, or put an authenticating reverse proxy in front.
+
+---
+
+## tcp
+
+Broadcasts formatted payloads to every connected TCP client.
 
 ```toml
 [[pipelines.plugin_sinks]]
-id = "tcp_out"
+id   = "tap"
 type = "tcp"
 [pipelines.plugin_sinks.config]
 host                 = "0.0.0.0"
@@ -116,39 +175,161 @@ write_timeout_ms = 5000
 keep_alive           = true
 keep_alive_period_ms = 30000
 max_connections      = 0
-```
 
-**Configuration Options:**
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+cert_file = "/etc/logwisp/tls/server.crt"
+key_file  = "/etc/logwisp/tls/server.key"
+```
 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `host` | string | "0.0.0.0" | Bind address |
+| `host` | string | `0.0.0.0` | Bind address; IPv4 only |
-| `port` | int | Required | Listen port |
+| `port` | int | **required** | Listen port |
-| `buffer_size` | int | 1000 | Sink input queue size |
+| `buffer_size` | int | `1000` | Sink input queue depth |
-| `client_buffer_size` | int | 256 | Per-client send queue size |
+| `client_buffer_size` | int | `256` | Per-client send queue depth |
-| `write_timeout_ms` | int | 5000 | Write timeout |
+| `write_timeout_ms` | int | `5000` | Per-write deadline |
-| `keep_alive` | bool | true | Enable TCP keep-alive |
+| `keep_alive` | bool | `true` | Enable TCP keep-alive on accepted connections |
-| `keep_alive_period_ms` | int | 30000 | Keep-alive interval |
+| `keep_alive_period_ms` | int | `30000` | Keep-alive idle period |
-| `max_connections` | int | 0 | Concurrent connection cap (0 = unlimited) |
+| `max_connections` | int | `0` | Concurrent connection cap; `0` = unlimited |
-```
+| `tls` | table | — | Listener TLS |
 
-### Null Sink
+**Behaviour**
+
+- The sink is write-only. Each connection also runs a reader that discards
+  inbound bytes; it exists to detect disconnects and to refresh session
+  activity when a client sends anything.
+- A write that misses its deadline means the kernel buffer stayed full for the
+  whole timeout, so the client is disconnected immediately rather than retried.
+- A client whose send queue is full has that event dropped (`dropped_writes`)
+  and stays connected.
+- With TLS enabled the handshake runs under a 10 s bound *after* the
+  `max_connections` check, so concurrent handshakes are bounded too.
+
+---
+
+## tcp_chain
+
+Forwards structured entries to a downstream LogWisp `tcp_chain` source over one
+persistent connection. See [Chaining](chaining.md).
 
 ```toml
 [[pipelines.plugin_sinks]]
-id = "null_out"
+id   = "to_relay"
-type = "null"
+type = "tcp_chain"
+[pipelines.plugin_sinks.config]
+host                 = "relay.internal"
+port                 = 15801
+node                 = "edge-01"
+buffer_size          = 1000
+dial_timeout_ms      = 5000
+write_timeout_ms     = 5000
+backoff_min_ms       = 500
+backoff_max_ms       = 30000
+keep_alive           = true
+keep_alive_period_ms = 30000
+
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+ca_file   = "/etc/logwisp/tls/ca.crt"
+cert_file = "/etc/logwisp/tls/client.crt"
+key_file  = "/etc/logwisp/tls/client.key"
 ```
 
-## Buffer Management
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `host` | string | **required** | Downstream host |
+| `port` | int | **required** | Downstream port |
+| `node` | string | `os.Hostname()` | Origin label stamped on first-hop entries |
+| `buffer_size` | int | `1000` | Sink input queue depth |
+| `dial_timeout_ms` | int | `5000` | TCP connect timeout |
+| `write_timeout_ms` | int | `5000` | Per-write deadline |
+| `backoff_min_ms` | int | `500` | Reconnect backoff floor |
+| `backoff_max_ms` | int | `30000` | Reconnect backoff ceiling |
+| `keep_alive` | bool | `true` | Enable TCP keep-alive |
+| `keep_alive_period_ms` | int | `30000` | Keep-alive idle period |
+| `tls` | table | — | Dialer TLS; `cert_file`/`key_file` present a client identity |
 
-- Full input buffer: entry dropped for that sink only (counted per pipeline as `total_dropped_by_sink`)"
+**Behaviour**
+
+- The connection is established lazily, so pipeline start does not depend on the
+  downstream being up.
+- Each entry is serialized as one canonical JSON line. Delivery holds the line
+  across reconnects until it is written or the process shuts down, with
+  exponential backoff plus ±20 % jitter between attempts.
+- Because delivery blocks the sink's run loop during an outage, back-pressure
+  surfaces as a full input queue and is counted by the pipeline as
+  `total_dropped_by_sink`.
+- With TLS, dial and handshake are bounded together by
+  `dial_timeout_ms` + 10 s.
+- Events arriving without a structured entry are wrapped from the formatted
+  payload and counted in `synthesized`.
+
+**Statistics**: `target`, `node`, `tls`, `connected`, `reconnects`,
+`write_errors`, `synthesized`.
+
+---
+
+## http_chain
+
+Batches structured entries as NDJSON and POSTs them to a downstream LogWisp
+`http_chain` source.
+
+```toml
+[[pipelines.plugin_sinks]]
+id   = "to_collector"
+type = "http_chain"
+[pipelines.plugin_sinks.config]
+host               = "collector.internal"
+port               = 15802
+ingest_path        = "/ingest"
+node               = "edge-01"
+buffer_size        = 1000
+max_batch_count    = 100
+max_batch_bytes    = 1048576
+flush_interval_ms  = 1000
+request_timeout_ms = 10000
+backoff_min_ms     = 500
+backoff_max_ms     = 30000
+
+[pipelines.plugin_sinks.config.tls]
+enabled   = true
+ca_file   = "/etc/logwisp/tls/ca.crt"
+cert_file = "/etc/logwisp/tls/client.crt"
+key_file  = "/etc/logwisp/tls/client.key"
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `host` | string | **required** | Downstream host |
+| `port` | int | **required** | Downstream port |
+| `ingest_path` | string | `/ingest` | Endpoint path; must start with `/` |
+| `node` | string | `os.Hostname()` | Origin label stamped on first-hop entries |
+| `buffer_size` | int | `1000` | Sink input queue depth |
+| `max_batch_count` | int | `100` | Flush after this many entries |
+| `max_batch_bytes` | int | `1048576` | Flush after this many bytes (1 MiB) |
+| `flush_interval_ms` | int | `1000` | Flush after this long |
+| `request_timeout_ms` | int | `10000` | Covers dial, write, and response |
+| `backoff_min_ms` | int | `500` | Retry backoff floor |
+| `backoff_max_ms` | int | `30000` | Retry backoff ceiling |
+| `tls` | table | — | Dialer TLS; `cert_file`/`key_file` present a client identity |
+
+**Behaviour**
+
+- Delivery is at-least-once per batch: a retried batch can be delivered twice if
+  the first attempt succeeded but the response was lost.
+- Retries apply to transport errors, `408`, `429`, and `5xx`. Any other
+  non-2xx response is treated as permanent, and the batch is dropped and counted
+  in `dropped_batches`.
+- HTTP/2 is off by design; batched NDJSON POSTs gain nothing from it.
+- On shutdown a single best-effort flush of the pending batch is attempted.
+
+**Statistics**: `target`, `node`, `tls`, `batches_sent`, `request_errors`,
+`dropped_batches`, `synthesized`.
+
+---
 
 ## Sink Statistics
 
-All sinks track:
+Every sink reports: `id`, `type`, `total_processed`, `active_connections`,
-- Total entries processed
+`start_time`, `last_processed`, and a type-specific `details` map.
-- Active connections
-- Failed sends
-- Retry attempts
-- Last processed timestamp

doc/sources.md

@@ -1,65 +1,99 @@
 # Input Sources
 
-LogWisp sources monitor various inputs and generate log entries for pipeline processing.
+Sources produce `core.LogEntry` values for a pipeline. Every source is declared
-
+as a `[[pipelines.plugin_sources]]` entry with an `id`, a `type`, and a
-## Source Types
+type-specific `config` table.
-
-### Directory Source
-
-Monitors a directory for log files matching a pattern. (type: `file`)
 
 ```toml
 [[pipelines.plugin_sources]]
-id = "file_in"
+id   = "app_logs"
 type = "file"
 [pipelines.plugin_sources.config]
 directory = "/var/log/myapp"
-pattern = "*.log"          # Glob pattern
-check_interval_ms = 100    # Poll interval
 ```
 
-**Configuration Options:**
+Registered types: `file`, `console`, `random`, `null`, `tcp_chain`,
+`http_chain`.
 
-| Option | Type | Default | Description |
+Publication from any source is non-blocking. When a subscriber channel is full
-|--------|------|---------|-------------|
+the entry is dropped and counted in `dropped_entries`.
-| `directory` | string | Required | Directory to monitor |
-| `pattern` | string | "*" | File pattern (glob) |
-| `check_interval_ms` | int | 100 | File check interval in milliseconds |
 
-**Features:**
+---
-- Automatic rotation detection (inode + size tracking)
-- In-memory position tracking; on restart, monitoring resumes from the current end of each file (offsets are not persisted)
-- Concurrent file monitoring
-- Pattern-based file selection
 
-### Stdin Source
+## file
 
-Reads log entries from standard input.
+Tails every file in a directory whose name matches a glob.
 
 ```toml
 [[pipelines.plugin_sources]]
-id = "console_in"
+id   = "app_logs"
+type = "file"
+[pipelines.plugin_sources.config]
+directory         = "/var/log/myapp"
+pattern           = "*.log"
+check_interval_ms = 100
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `directory` | string | **required** | Directory to scan; not recursive |
+| `pattern` | string | `*` | Glob over filenames; `*` and `?` only |
+| `check_interval_ms` | int | `100` | Directory rescan interval; minimum `10` |
+
+**Behaviour**
+
+- `check_interval_ms` governs how often the directory is rescanned for new or
+  removed files. Tailing an already-open file polls on a **fixed 100 ms**
+  interval that this option does not change.
+- Each matched file gets its own watcher. Watchers for files that disappear are
+  stopped and removed on the next scan.
+- A new watcher seeks to end-of-file. Positions live in memory only, so a
+  restart resumes from the current end of each file and content written while
+  LogWisp was down is not read.
+- Rotation is detected from size decrease, modification-time reset, a position
+  beyond end-of-file, or an inode change. An inode change where the new file is
+  already larger than the recorded position is treated as an atomic save, not a
+  rotation, and the position is preserved.
+- Lines are parsed as JSON when they contain `time`, `level`, `msg`, and
+  `fields` keys; `time` is read as RFC3339Nano. Anything else is kept as plain
+  text with the level inferred from common markers (`[ERROR]`, `WARN:`, and so
+  on).
+- `Source` is set to the file's base name.
+
+**Statistics**: per-watcher size, position, entries read, rotation count, and
+last read time, plus `active_watchers`.
+
+---
+
+## console
+
+Reads newline-delimited entries from standard input.
+
+```toml
+[[pipelines.plugin_sources]]
+id   = "stdin"
 type = "console"
 [pipelines.plugin_sources.config]
 buffer_size = 1000
 ```
 
-**Configuration Options:**
-
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `buffer_size` | int | 1000 | Internal buffer size |
+| `buffer_size` | int | `1000` | Subscriber channel depth |
 
-**Features:**
+At most **one** instance per pipeline: the type is registered with
-- Line-based processing
+`MaxInstances: 1`, and a second instance is rejected at pipeline construction.
-- Automatic level detection
+The level is inferred from the line text, and `Source` is set to `console`.
-- Non-blocking reads
 
-### Random Source
+---
+
+## random
+
+Synthetic entry generator for development, smoke tests, and sanitizer testing.
 
 ```toml
 [[pipelines.plugin_sources]]
-id = "random_in"
+id   = "generator"
 type = "random"
 [pipelines.plugin_sources.config]
 interval_ms = 500
@@ -69,39 +103,147 @@ length = 20
 special     = false
 ```
 
- **Configuration Options:**
- 
 | Option | Type | Default | Description |
 |--------|------|---------|-------------|
-| `interval_ms` | int | 500 | Generation interval |
+| `interval_ms` | int | `500` | Emission period |
-| `jitter_ms` | int | 0 | Random jitter interval |
+| `jitter_ms` | int | `0` | Symmetric jitter; clamped to `interval_ms`, must be non-negative |
-| `format` | string | "txt" | "txt", "json", "raw" |
+| `format` | string | `txt` | `raw` (message only), `txt` (bracketed line), `json` (JSON object as the message) |
-| `length` | int | 20 | Log length |
+| `length` | int | `20` | Message length in characters |
-| `special` | bool | false | Include special characters |
+| `special` | bool | `false` | Inject control and non-ASCII characters |
 
-## Source Statistics
+`special = true` is the intended way to exercise sanitizer policies: it inserts
+control bytes and multi-byte Unicode into otherwise ordinary messages. Levels
+are chosen at random from DEBUG, INFO, WARN, ERROR.
 
-All sources track:
+---
-- Total entries received
-- Dropped entries (buffer full)
-- Invalid entries
-- Last entry timestamp
-- Active connections (network sources)
-- Source-specific metrics
 
-### Null Source
+## null
+
+Produces nothing. Useful as a placeholder so a sink-only pipeline satisfies the
+"at least one source" requirement.
 
 ```toml
 [[pipelines.plugin_sources]]
-id = "null_in"
+id   = "void"
 type = "null"
-[pipelines.plugin_sources.config]
 ```
 
-## Buffer Management
+No options.
 
-Each source maintains internal buffers:
+---
-- Default size: 1000 entries
+
-- Drop policy when full
+## tcp_chain
-- Configurable per source
+
-- Non-blocking writes
+Listens for persistent NDJSON streams from upstream LogWisp `tcp_chain` sinks.
+See [Chaining](chaining.md) for the protocol.
+
+```toml
+[[pipelines.plugin_sources]]
+id   = "ingest_tcp"
+type = "tcp_chain"
+[pipelines.plugin_sources.config]
+host             = "0.0.0.0"
+port             = 15801
+buffer_size      = 1000
+max_connections  = 0
+read_timeout_ms  = 0
+hello_timeout_ms = 10000
+trust_node       = true
+
+[pipelines.plugin_sources.config.tls]
+enabled        = true
+cert_file      = "/etc/logwisp/tls/server.crt"
+key_file       = "/etc/logwisp/tls/server.key"
+client_auth    = true
+client_ca_file = "/etc/logwisp/tls/client-ca.crt"
+min_version    = "1.3"
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `host` | string | `0.0.0.0` | Bind address; IPv4 only |
+| `port` | int | **required** | Listen port, 1–65535 |
+| `buffer_size` | int | `1000` | Subscriber channel depth |
+| `max_connections` | int | `0` | Concurrent connection cap; `0` = unlimited |
+| `read_timeout_ms` | int | `0` | Per-connection idle read deadline; `0` = none |
+| `hello_timeout_ms` | int | `10000` | Deadline for the hello preamble |
+| `trust_node` | bool | `true` | `false` overrides the sender's node label with its remote address |
+| `tls` | table | — | Listener TLS; see [Security](security.md) |
+
+**Behaviour**
+
+- TLS handshakes run explicitly with a 10 s bound before the preamble is read,
+  after the `max_connections` admission check.
+- A connection is rejected if the first line is not a valid hello with a
+  matching protocol version.
+- Each accepted connection gets a session recording the remote address, node
+  label, and — under TLS — `tls` and `tls_peer_cn`.
+- A malformed entry line increments `parse_errors` and is skipped; the
+  connection survives. A line over 1 MiB is a protocol violation and terminates
+  the connection.
+
+**Statistics**: `active_connections`, `rejected_conns`, `parse_errors`,
+`tls_handshake_errors`, `trust_node`.
+
+---
+
+## http_chain
+
+Accepts NDJSON batches POSTed by upstream LogWisp `http_chain` sinks.
+
+```toml
+[[pipelines.plugin_sources]]
+id   = "ingest_http"
+type = "http_chain"
+[pipelines.plugin_sources.config]
+host            = "0.0.0.0"
+port            = 15802
+ingest_path     = "/ingest"
+buffer_size     = 1000
+max_body_bytes  = 8388608
+read_timeout_ms = 30000
+trust_node      = true
+
+[pipelines.plugin_sources.config.tls]
+enabled        = true
+cert_file      = "/etc/logwisp/tls/server.crt"
+key_file       = "/etc/logwisp/tls/server.key"
+client_auth    = true
+client_ca_file = "/etc/logwisp/tls/client-ca.crt"
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `host` | string | `0.0.0.0` | Bind address; IPv4 only |
+| `port` | int | **required** | Listen port |
+| `ingest_path` | string | `/ingest` | Endpoint path; must start with `/` |
+| `buffer_size` | int | `1000` | Subscriber channel depth |
+| `max_body_bytes` | int | `8388608` | Per-request body cap (8 MiB) |
+| `read_timeout_ms` | int | `30000` | Full request read deadline |
+| `trust_node` | bool | `true` | `false` overrides the sender's node label with its remote address |
+| `tls` | table | — | Listener TLS |
+
+**Behaviour**
+
+- Only `POST` to `ingest_path` is routed; other methods get `405` with an
+  `Allow` header, and other paths get `404`.
+- A missing or mismatched `X-Logwisp-Protocol` header is rejected with `400`.
+- Batch acceptance is atomic: entries are published only after the body reads
+  cleanly end to end. A transfer error rejects the whole batch (`400`, or `413`
+  when the body cap is hit) so the sender retries it. A malformed *line* inside
+  an otherwise clean transfer is skipped and counted in `parse_errors`.
+- Success is `204 No Content` with `X-Logwisp-Accepted` set to the number of
+  entries ingested.
+- Sessions are cached per remote host + declared node and recreated after idle
+  expiry.
+
+**Statistics**: `total_requests`, `rejected_requests`, `parse_errors`,
+`cached_sessions`, `trust_node`.
+
+---
+
+## Source Statistics
+
+Every source reports: `id`, `type`, `total_entries`, `dropped_entries`,
+`start_time`, `last_entry_time`, and a type-specific `details` map. These appear
+in the status reporter output and in the `http` sink's status endpoint.

go.mod

@@ -1,10 +1,10 @@
 module logwisp
 
-go 1.26.0
+go 1.26.5
 
 require (
 	github.com/lixenwraith/config v0.1.1-0.20260712172228-ccd280ba6a98
-	github.com/lixenwraith/log v0.1.1-0.20260717175128-82eea9846ccd
+	github.com/lixenwraith/log v0.1.1-0.20260801090951-2c40643523b3
 )
 
 require (

go.sum

@@ -6,12 +6,14 @@ github.com/go-viper/mapstructure/v2 v2.5.0 h1:vM5IJoUAy3d7zRSVtIwQgBj7BiWtMPfmPE
 github.com/go-viper/mapstructure/v2 v2.5.0/go.mod h1:oJDH3BJKyqBA2TXFhDsKDGDTlndYOZ6rGS0BRZIxGhM=
 github.com/lixenwraith/config v0.1.1-0.20260712172228-ccd280ba6a98 h1:VEFo2WsYgM5YYfzAfvXRKigCWr3zEz6h4M59N5TXMpk=
 github.com/lixenwraith/config v0.1.1-0.20260712172228-ccd280ba6a98/go.mod h1:J9ydxY7he4Dz+S59xKs/kDQ2YAv+zOLF05b3Rr2/ogE=
-github.com/lixenwraith/log v0.1.1-0.20260717175128-82eea9846ccd h1:06Rk4DLvJW1kdeclH5HwywizgtMI64PHNE/2sBeuiwA=
+github.com/lixenwraith/log v0.1.1-0.20260724174821-e688c5a07207 h1:EnDMcnFkgTwTGo1ojoJ85/b6aH3yxlBUAg6AefaBcrI=
-github.com/lixenwraith/log v0.1.1-0.20260717175128-82eea9846ccd/go.mod h1:2+qURSVdWcX7REFH+3jUtIbtc+NtsAPecSvXLeGyK6U=
+github.com/lixenwraith/log v0.1.1-0.20260724174821-e688c5a07207/go.mod h1:fvGT3IxlJQLJ1is4OP8Iet8gVzTJTGcuT90FFB2WBtc=
+github.com/lixenwraith/log v0.1.1-0.20260801090951-2c40643523b3 h1:5ojkxyuOKiUBRYhqPqKJS1lrqkjckYqYMNGBB49kdY0=
+github.com/lixenwraith/log v0.1.1-0.20260801090951-2c40643523b3/go.mod h1:fvGT3IxlJQLJ1is4OP8Iet8gVzTJTGcuT90FFB2WBtc=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
+github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
-github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
+github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405 h1:yhCVgyC4o1eVCa2tZl7eS0r+SDo693bJlVdllGtEeKM=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=