v0.36.0 doc sweep update

README.md

@@ -22,8 +22,8 @@ streams, or downstream LogWisp nodes.
 - **A Unix filter out of the box**: with no configuration, `lw` copies stdin
   to stdout line for line and exits at the end of input; flags add filters,
   formats, sources and sinks
-- **Presets**: `tail`, `serve`, `edge` and `aggregator` build the common
-  pipelines from a few keys (`lw --preset tail:path=/var/log/app`);
+- **Presets**: `pipe`, `tail`, `serve`, `edge` and `aggregator` build the
+  common pipelines from a few keys (`lw --preset tail:path=/var/log/app`);
   `lw --dump` turns any command line into a configuration file
 - **Independent pipelines**, each `sources → flow → sinks`, running concurrently
   in one process
@@ -53,11 +53,11 @@ configuration engine the website's configurator runs as WebAssembly:
   and `t` again reruns as a reload, keeping the running pipelines when the new
   ones do not build. Listeners bind, dialers connect and files are written for
   real; a password is asked in a masked dialog
-- **Read any log**: the view parses LogWisp's JSON, lw's own log, pino, zap,
-  slog, docker, journald, ECS, GCP, logfmt and stamped text into time, level,
-  message and fields, filters them by level or pattern, and shows the payload
-  as the sinks receive it. It never slows a pipeline: what it misses is
-  counted
+- **Read any log**: the view parses LogWisp's JSON, lw's own log, vif
+  records, pino, zap, slog, docker, journald, ECS, GCP, logfmt and stamped
+  text into time, level, message and fields, filters them by level or
+  pattern, and shows the payload as the sinks receive it. It never slows a
+  pipeline: what it misses is counted
 - **Any terminal**: `cmd | lw -T` taps piped input, since the screen draws on
   the controlling terminal; colors range from true color to none, and a text
   console gets CP437 glyphs
@@ -105,13 +105,13 @@ exponential backoff and jitter.
 - Authorization by certificate identity: an `auth` block admits named peers
   (exact or RE2) rather than everything the CA issued, gates the `http` sink's
   stream and status endpoints, and lets a dialer pin the server it talks to
-- Password authentication (Argon2id-SCRAM) on the same block: listeners hold
-  verifiers, never passwords, and do no KDF work; every login outside proxy
-  mode is bound to the listener's certificate, so no relay presenting another
-  one can use it; HTTP listeners issue short-lived bearer tokens. `lw auth`
-  manages credentials files and logs viewers in; behind a site's
-  TLS-terminating proxy, browsers log in through a shipped login page and a
-  dependency-free JS client
+- Password authentication (Argon2id-SCRAM) on the same block: no login costs
+  a listener KDF work, and with a credentials file it holds verifiers, never
+  passwords; every login outside proxy mode is bound to the listener's
+  certificate, so no relay presenting another one can use it; HTTP listeners
+  issue short-lived bearer tokens. `lw auth` manages credentials files and
+  logs viewers in; behind a site's TLS-terminating proxy, browsers log in
+  through a shipped login page and a dependency-free JS client
 - Node binding: a chain source can label entries from the sender's certificate
   identity or username instead of from what the sender claims, so origin
   attribution is not forgeable
@@ -127,7 +127,7 @@ See [Security](doc/security.md) for configuration and the exact boundary, and
 ## Documentation
 
 - [Installation](doc/installation.md): building, installing, services,
-  the container image, packaging
+  the container image, packaging, the end-to-end test scripts
 - [Deployment](doc/deployment.md): `deploy/lw-deploy.sh` for edges,
   aggregators, containers, services, jails
 - [Architecture](doc/architecture.md): component model, data flow,
@@ -176,8 +176,9 @@ journalctl -f | ./bin/lw --sink http:host=127.0.0.1,port=8080
 ./bin/lw --source 'file:directory=/var/log/myapp,pattern=*.log' \
     --format json:sanitizer_policy=json > all.json
 
-# the same with a preset; on a terminal the level names are in color
-./bin/lw --preset tail:path=/var/log/myapp
+# the same files with a preset, raw to stdout; on a terminal the level
+# names are in color
+./bin/lw --preset 'tail:path=/var/log/myapp/*.log'
 
 # compose on screen from a preset menu; r runs, t taps, o prints
 ./bin/lw -T
@@ -235,8 +236,8 @@ Debian, Ubuntu and Arch. See [Installation](doc/installation.md).
 ## System Requirements
 
 - **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
-- **Architecture**: amd64
-- **Go**: 1.27.2+ to build from source
+- **Architecture**: amd64; arm64 builds but is untested
+- **To build**: Go 1.27.2+, and GNU make or BSD make
 
 Network sources and sinks use IPv4 or IPv6, following the address family of
 the configured host.

cmd/lw/main.go

@@ -1,3 +1,8 @@
+// Command lw is LogWisp: it runs the pipelines its file, LOGWISP_ variables
+// and flags describe, rebuilt on SIGHUP, SIGUSR1 or an auto_reload, or
+// composes them on screen with --tui. Its auth, tls, preset and config commands
+// manage credentials and certificates, print a preset's pipeline as TOML and
+// write the annotated configuration file.
 package main
 
 import (

config/logwisp.toml

@@ -30,7 +30,7 @@
 # quiet = false                         # No log of lw's own; pipeline output still flows
 # status_reporter = true                # Periodic status logging (30s, DEBUG level); false without a file
 # auto_reload = false                   # Watch file; read at startup (signals always reread it)
-# color = "auto"                        # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR unset
+# color = "auto"                        # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR empty or unset, TERM not dumb
 
 ###############################################################################
 ### Logging (lw's own log)
@@ -61,7 +61,7 @@
 ###############################################################################
 
 # [[pipelines]]
-# name = "default"                      # example
+# name = "default"                      # Required (example)
 
 ###============================================================================
 ### Flow: the stages between sources and sinks, validated whether on or off
@@ -83,7 +83,7 @@
 
 # [pipelines.flow.format]
 # type = "raw"                          # raw|txt|json
-# flags = 0                             # Formatter flags; 0 = by type
+# flags = 0                             # For experts, a sum of bits: 1 messages verbatim, 16 no time, 32 no level; 0 = by type
 # timestamp_format = ""                 # Go time layout; empty = RFC 3339 with nanoseconds
 # sanitizer_policy = ""                 # raw|json|txt|shell; empty = by type
 
@@ -146,7 +146,7 @@
 ###                        certificate to the user (needs tls.client_auth)
 ### allow                  mtls: exact identities. Empty allow AND
 ###                        allow_patterns admits any identity the CA vouches
-###                        for (logged WARN)
+###                        for (a listener logs a WARN)
 ### allow_patterns         mtls: RE2 patterns; anchor them yourself (^...$)
 ### node_binding           chain sources: binds the node label to the identity
 ###                        or username, overriding trust_node; unset = force
@@ -205,7 +205,8 @@
 # pattern = "*"                         # Glob of the whole file name (* and ? only)
 # check_interval_ms = 100               # Directory rescan, at least 10; open files are polled every 100 ms
 ## raw = true never parses a line; with format type "raw" each line is relayed
-## as written, except that empty lines are skipped and CRLF becomes LF.
+## as written, except that empty lines are skipped, CRLF becomes LF and a line
+## over 1 MiB, or one still being written when read, is split.
 # raw = false
 # from = "end"                          # end|start: where a newly found file is read from
 
@@ -300,7 +301,7 @@
 # node_binding = ""
 # credentials_file = ""                 # scram: written by `lw auth add-user`
 # token_lifetime_ms = 0
-# trusted_proxies = []                  # the http sink only
+# trusted_proxies = []                  # refused here: the http sink only
 # username = ""
 # password_file = ""
 # [pipelines.plugin_sources.config.acl]
@@ -380,7 +381,6 @@
 # identity = ""
 # allow = []
 # allow_patterns = []
-# node_binding = ""                     # chain sources only
 # credentials_file = ""                 # scram: log in with `lw auth token`
 # token_lifetime_ms = 0
 # trusted_proxies = []                  # scram: proxies ending the browsers' TLS; tls may then be off
@@ -423,7 +423,6 @@
 # identity = ""
 # allow = []
 # allow_patterns = []
-# node_binding = ""                     # chain sources only
 # credentials_file = ""                 # scram: readers use `lw auth stream`
 # username = ""
 # password_file = ""
@@ -443,7 +442,7 @@
 # [pipelines.plugin_sinks.config]
 # host = "aggregator.example.org"       # Required (example); an IPv6 literal bare, e.g. "2001:db8::1"
 # port = 9440                           # Required (example)
-# node = ""                             # Origin label, kept across hops; empty = the host name
+# node = ""                             # Origin label, kept across hops; empty = this machine's host name
 # buffer_size = 1000
 # dial_timeout_ms = 5000
 # write_timeout_ms = 5000
@@ -476,7 +475,7 @@
 # host = "aggregator.example.org"       # Required (example); an IPv6 literal bare, e.g. "2001:db8::1"
 # port = 9441                           # Required (example)
 # ingest_path = "/ingest"               # Must start with "/"; not "/auth"
-# node = ""                             # Origin label, kept across hops; empty = the host name
+# node = ""                             # Origin label, kept across hops; empty = this machine's host name
 # buffer_size = 1000
 # max_batch_count = 100
 # max_batch_bytes = 1048576

deploy/README.md

@@ -13,3 +13,8 @@ deploy/lw-deploy.sh --yes ...        # no questions: defaults, or fail on a miss
 ```
 
 Roles, runtimes, scenarios and what is not covered: [doc/deployment.md](../doc/deployment.md).
+
+`package/` holds the service files (`logwisp.service`, `.sysusers`,
+`.tmpfiles`, `logwisp.rc`) that `make install` and this script install, the
+shell completions, and the deb, Arch and FreeBSD packaging:
+[doc/installation.md](../doc/installation.md#installing).

deploy/lw-deploy.sh

@@ -105,7 +105,9 @@ Containers (Linux)
                           publishes nothing and listens on --listen
 
 Native services
-  --bin FILE              lw binary to install (default: the installed one, else bin/lw)
+  --bin FILE              lw binary to install (default: the installed one, else
+                          bin/lw); manual: the lw that adds users and checks
+                          (default: lw on PATH, else bin/lw)
   --jail NAME             FreeBSD: install into this running jail through jexec
   --package-dir DIR       service files (default deploy/package)
 

doc/README.md

@@ -4,7 +4,7 @@ The [README](../README.md) has LogWisp's features, a quick start and the
 system requirements.
 
 - [Installation](installation.md): building, installing, services,
-  the container image, packaging
+  the container image, packaging, the end-to-end test scripts
 - [Deployment](deployment.md): `deploy/lw-deploy.sh` for edges,
   aggregators, containers, services, jails
 - [Architecture](architecture.md): component model, data flow,

doc/architecture.md

@@ -45,18 +45,23 @@ Package map:
 - `internal/filter`: regex filter and filter chain.
 - `internal/format`: adapter over the `lixenwraith/log` formatter and
   sanitizer.
-- `internal/source/*`, `internal/sink/*`: the source and sink plugins.
+- `internal/source`, `internal/sink`: the `Source` and `Sink` interfaces and
+  their stats, and the stop helpers network sinks share (`sink.FlushBound`,
+  `FlushContext`, `Drain`); their subpackages are the plugins.
 - `internal/plugin`: global factory registry populated by plugin `init()`.
 - `internal/chain`: chain wire protocol: hello preamble, entry codec, backoff.
-- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`.
+- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`; also
+  the certificates `lw tls` writes and a listener makes at startup, pins and
+  startup warnings.
 - `internal/authz`: the single seam between `AuthOptions` and the network
   plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`; `Authorize`
   refuses under `scram`.
 - `internal/netacl`: a listener's `acl` block: address rules, PROXY headers
   and per-client limits; its `Table` is SCRAM throttling's limiter too.
 - `internal/session`: session manager and per-instance proxy.
-- `internal/core`: shared types (`LogEntry`, `TransportEvent`), capabilities,
-  constants.
+- `internal/core`: shared types (`LogEntry`, `TransportEvent`, the viewer's
+  `Tap`), the level table `core.Levels`, capabilities, constants, and
+  `Network` and `Listen`, which keep a socket to its host's address family.
 - `internal/tokenbucket`: rate limiter primitive.
 - `internal/tui`: `lw --tui`, the diagram and the live view of what `t`
   runs; `cmd/lw/screen.go` runs the pipelines for it.
@@ -65,6 +70,8 @@ Package map:
   terminal acts on, which the console sink shares.
 - `internal/version`: version, commit and build time, stamped at build.
 - `internal/testutil`: fixtures for tests only.
+- `config`: embeds `logwisp.toml`, the annotated default configuration that
+  `lw config init` writes and the packages install.
 
 ## Plugin Registration
 
@@ -98,6 +105,11 @@ instance of a single type fails, naming both pipelines.
 4. **Dispatch** sends the event to every sink's input channel with a
    non-blocking send, except to a sink declaring `CapBackpressure`.
 
+A heartbeat joins at step 4: the flow writes it in its format with a
+structured `Entry`, and the pipeline offers it to the tap and dispatches it,
+so no rate limit or filter drops it and `flow.total_processed` does not count
+it.
+
 `LogEntry` fields:
 
 - `Time`: entry timestamp.
@@ -161,7 +173,10 @@ shutdown (`dropped_batches`).
 `Pipeline.Stop` is ordered so in-flight data drains:
 
 1. Release dispatch waits on backpressure sinks; stop all sources
-   concurrently, each closing its subscriber channels.
+   concurrently, each closing its subscriber channels. From the release on,
+   dispatch picks at random between a backpressure sink's input and the
+   release, so it drops about half of the entries still in flight for one
+   (`total_dropped_by_sink`) even with room.
 2. Wait for the run loop, which ends when every subscription channel closes.
 3. Stop all sinks concurrently: console and file sinks write their queue;
    network sinks deliver theirs within `sink.FlushBound`, their write or
@@ -197,11 +212,13 @@ TLS is built in exactly one place, `internal/tlsx`, which exposes
 ## Sessions
 
 Each pipeline owns a `session.Manager`. Plugins receive a `session.Proxy`
-scoped to their instance id, so one plugin cannot see or remove another's
-sessions. A session records the remote address, creation and last-activity
-timestamps, and metadata — including `tls` and `tls_peer_cn` for TLS peers,
-`auth_method` / `auth_identity` for authorized ones, and `peer_addr`, the proxy,
-for a client a PROXY header names.
+scoped to their id, so a plugin sees and removes only the sessions under its
+id; ids are unique per list, so a source and a sink that share one (the
+built-in pipe's two `console`s) share that scope. A session records the
+remote address, creation and last-activity timestamps, and metadata —
+including `tls` and `tls_peer_cn` for TLS peers, `auth_method` /
+`auth_identity` for authorized ones, and `peer_addr`, the proxy, for a client
+a PROXY header names.
 
 Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The
 HTTP sink's broker treats a vanished session as an eviction signal and closes
@@ -277,7 +294,10 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key",
   it: answer it in `dirs` and call again.
 - Pipelines are a JSON list in the file's shape (`name`, `flow`,
   `plugin_sources`, `plugin_sinks`); a misspelled key is refused. A node is
-  `{role, id, index}`: a source's or sink's id, a filter's index.
+  `{role, id, index}`: `source` or `sink` by its id, `filters` by its index,
+  or the flow stage `rate_limit`, `format` or `heartbeat`, which `set` turns
+  on and `remove` off; `add` takes `source`, `sink`, or `filters` with the
+  type `include` or `exclude`.
 
 ## Resource Management
 
@@ -285,18 +305,25 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key",
   load.
 - `max_connections` caps connections on the `tcp` sink and the `tcp_chain`
   source, and streams on the `http` sink. Admission is a load-then-check, so a
-  burst can over-admit by roughly one.
+  burst can over-admit by the connections that arrive before the count rises.
+- A listener's `acl` block caps each client: `max_connections_per_client`,
+  and on the `http` sink and `http_chain` source
+  `requests_per_second_per_client` ([Security](security.md#per-client-limits)).
 - Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an
   oversized line terminates a `tcp_chain` link and fails an `http_chain`
   batch with `400`.
 - The `http_chain` source caps each request body at `max_body_bytes`.
-- File sinks rotate on size, cap total rotated size, and honour a retention
-  window.
+- File sinks rotate on size, cap total rotated size, honour a retention
+  window, and with `min_disk_free_mb` delete the oldest files to keep that
+  much disk free.
 
 ## Performance Notes
 
-- In-memory entry processing is sub-millisecond; the formatter mutex is the
-  only shared serialization point in the hot path.
+- In-memory entry processing is sub-millisecond. Besides the formatter's
+  mutex, an entry can take the rate limiter's bucket lock (`policy = "drop"`)
+  and the pipeline's session-manager lock, which the network plugins and the
+  console and random sources take as entries pass, to refresh session
+  activity.
 - File tailing detects new content within roughly 100 ms (fixed poll), while
   `check_interval_ms` governs how quickly a *newly created* file is noticed.
 - `http_chain` trades latency for efficiency: entries wait up to

doc/chaining.md

@@ -44,7 +44,7 @@ Chained entries carry a `node` label identifying where they originated.
 
 Relays preserve `node`, so a label survives any number of hops and identifies
 the original producer, unless a hop's source overwrites it: `trust_node =
-false`, or `node_binding = "force"` (the default with an `auth` block).
+false`, or `node_binding = "force"` (the default under `mtls` or `scram`).
 Formatters write the label as a prefix on the source
 ([Formatters](formatters.md#node-identity-in-output)).
 
@@ -55,9 +55,10 @@ under `scram`. `auth.node_binding`:
 - `none`: `trust_node` governs the connection label and the per-entry `node`
   field.
 - `assert`: the connection label must equal the identity, or the peer is
-  rejected; `trust_node` governs the per-entry `node` field.
-- `force` (default with an `auth` block): the identity is the connection label
-  and overwrites every per-entry `node` field.
+  rejected; `trust_node` governs the per-entry `node` field, the label standing
+  in wherever it would put the remote address.
+- `force` (default under `mtls` or `scram`): the identity is the connection
+  label and overwrites every per-entry `node` field.
 
 Pick `force` at an ingest boundary you do not trust — it is the only setting
 where a compromised edge cannot mislabel its entries, including through the
@@ -87,17 +88,23 @@ A persistent connection carrying newline-delimited JSON.
    {"logwisp":1,"node":"edge-01"}
    ```
 
-3. The listener reads that line within `hello_timeout_ms` and rejects the
-   connection if it is malformed or declares a different protocol version.
+3. The listener reads that line, at most 4 KiB, within `hello_timeout_ms` and
+   rejects the connection if it is malformed or declares a different protocol
+   version.
 4. Every subsequent line is one JSON-encoded `LogEntry`.
 
 Line size is bounded at 1 MiB. An oversized line is a protocol violation and
 terminates the connection, because the scanner cannot resynchronize afterwards.
 The cap applies to the encoded line, envelope and JSON escapes included, so a
 message near 1 MiB exceeds it (the console and file sources cut lines at
-1 MiB). On `tcp_chain` such an entry ends the link and is lost, together with
-any line the sink writes before it notices the close. On `http_chain` it fails
-its whole batch with `400`, which the sink drops.
+1 MiB). On `tcp_chain` such an entry ends the link. If the socket took the
+whole line, the entry is lost, together with any line the sink writes before it
+notices the close. If the line overruns the cap by more than the socket buffers
+hold, as a message of control characters can (most escape to six bytes), the
+write fails and the sink resends the line on every reconnect: the link stalls
+until a stop or reload, and later entries count in `total_dropped_by_sink`. The
+sink does not yet drop a line the source must refuse. On `http_chain` such an
+entry fails its whole batch with `400`, which the sink drops.
 
 Under `auth.type = "scram"` the hello also carries the SCRAM client-first
 message, and three more lines precede the entries, each one JSON object of at
@@ -119,18 +126,20 @@ binary ignores it and the dialer gives up after 10 s.
 
 ### HTTP transport
 
-Batches of NDJSON delivered by `POST`, with the preamble expressed as headers.
+Batches of NDJSON delivered by `POST` to `ingest_path` (default `/ingest`), with
+the preamble expressed as headers.
 
 - Request headers
   - `X-Logwisp-Protocol`: the protocol version; must be `1`.
-  - `X-Logwisp-Node`: the origin node label.
+  - `X-Logwisp-Node`: the sending sink's `node`, the connection label, as in the
+    TCP hello.
   - `Content-Type`: `application/x-ndjson`.
   - `Authorization`: `Bearer <token>`, under `scram`.
 - Response header
   - `X-Logwisp-Accepted`: the number of entries ingested.
 
-Responses: `204` on success; `400` for a bad protocol version or a malformed
-body, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for
+Responses: `204` on success; `400` for a bad protocol version or a transfer
+error, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for
 another path; `405` for a non-`POST` method; `429` (with `Retry-After`) past
 `acl.requests_per_second_per_client`; `401` for a missing, invalid or expired
 token; `403` for a refused certificate or node label.
@@ -157,7 +166,8 @@ oversized body.
 ```
 
 `node`, `level`, and `fields` are omitted when empty. A missing `time` is filled
-in at ingest.
+in at ingest. A line that does not decode is skipped and counted in
+`parse_errors`; the connection or batch stands.
 
 ## Delivery Semantics
 

doc/cli.md

@@ -74,23 +74,24 @@ pipelines have [their own flags](#pipelines):
 - `--schema`: print every setting, pipeline flag, flow stage, plugin option,
   preset and output form as one JSON document, and exit; tools build their
   option lists from it:
-  - a key: its kind, default, rules and help; a table (`tls`, `auth`, `acl`)
-    holds its own keys, those of the plugin's side (listener or dialer),
-    HTTP's only on a plugin that speaks HTTP. Every listener lists
-    `auth.node_binding`, which only chain sources use, and every HTTP
-    listener `auth.trusted_proxies`, which only the `http` sink admits
+  - a key: its kind, default, rules and help, and a setting's `variable`; a
+    table (`tls`, `auth`, `acl`) holds its own keys, those of the plugin's
+    side (listener, chain source or dialer), HTTP's only on a plugin that
+    speaks HTTP: `auth.node_binding` only on the chain sources, and
+    `auth.trusted_proxies` on both HTTP listeners, though only the `http`
+    sink admits it
   - a flow stage: its keys, and `on`, the values that turn it on
   - a preset key: its default and help, and the choices (`enum`) and `hint`
     of the option it sets
   - `forms`: `command`, `environment` and `file`, each with its hint
 - `--color [WHEN]`, WHEN `auto`, `always` or `never`: level names in color on
-  console sinks, default `auto` (a terminal, `NO_COLOR` unset, `TERM` not
-  `dumb`); bare, `always`. A console sink's own `color` wins
+  console sinks, default `auto` (a terminal, `NO_COLOR` empty or unset,
+  `TERM` not `dumb`); bare, `always`. A console sink's own `color` wins
   ([console sink](sinks.md#console))
 - `-q`, `--quiet`: silence lw's own log and notices, default `false`;
   pipeline output still flows
-- `--status_reporter=BOOL`: periodic status logging, default `true` with a
-  configuration file, `false` without
+- `--status_reporter=BOOL`: pipeline statistics every 30 s, logged at
+  `debug`, default `true` with a configuration file, `false` without
 - `--auto_reload=BOOL`: reload when the configuration file changes, default
   `false`
 - `-V`, `--version`: print the version and exit
@@ -105,12 +106,15 @@ The last `-c` wins; a missing or empty path is an error.
   default `stderr`, so stdout carries only data
 - `--logging.level`: `debug`, `info`, `warn` or `error`, default `info` with a
   configuration file, `warn` without
-- `--logging.format`: `raw`, `txt` or `json`
-- `--logging.sanitization`: `raw`, `json`, `txt` or `shell`
-- `--logging.file.directory`: a path
-- `--logging.file.name`: a string
-- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers
-- `--logging.file.retention_hours`: a float
+- `--logging.format`: `raw`, `txt` or `json`, default `txt`
+- `--logging.sanitization`: `raw`, `json`, `txt` or `shell`, default `raw`
+- `--logging.file.directory`: a path, default `./log`; the limits below
+  delete any `.log` file in it
+  ([Application Logging](configuration.md#application-logging))
+- `--logging.file.name`: a string, default `logwisp`
+- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers,
+  default `100` and `1000`
+- `--logging.file.retention_hours`: a float, default `168`; `0` keeps files
 
 `--logging.console.target` is accepted but has no effect; the console
 destination is derived from `--logging.output`.
@@ -308,8 +312,8 @@ Any pipeline flag on the command line makes LogWisp ignore all of them.
 
 A container needs no configuration file. Without one lw logs less
 ([Built-in Defaults](#built-in-defaults)), so pass `LOGWISP_LOGGING_LEVEL=info`
-and `LOGWISP_STATUS_REPORTER=true`; the image sets neither, as a variable would
-override a mounted file's value:
+(`debug`, with `LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the
+image sets neither, as a variable would override a mounted file's value:
 
 ```bash
 docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
@@ -318,7 +322,7 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
   -e LOGWISP_FILTER='exclude:patterns=DEBUG' \
   -e LOGWISP_SINK='http:host=0.0.0.0,port=8080' \
   -e LOGWISP_SINK_1=console \
-  -e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true \
+  -e LOGWISP_LOGGING_LEVEL=info \
   logwisp:dev
 ```
 
@@ -335,8 +339,9 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
 
 ## Exit Codes
 
-- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version` /
-  `--help`, `--dump`, or a valid `--check`
+- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version`,
+  `--help`, `--schema`, `--dump`, a valid `--check`, or a `--tui` screen
+  closed without `r`
 - `1`: general error: a configuration load or validation failure (`--check`
   included, and a bad value on the command line), a logger init failure, a
   service bootstrap failure
@@ -360,7 +365,7 @@ lw < app.log > copy.log
 - Each line of stdin is one entry and one line of stdout.
   - The terminator (`\n` or `\r\n`) is written back as `\n`; an unterminated
     last line is kept and terminated.
-  - Blank lines are skipped.
+  - Empty lines are skipped; a line of spaces is an entry.
   - A line over 1 MiB continues in the next entry; no byte is lost.
 - Nothing is dropped: a slow reader slows the reading of stdin.
 - On a terminal, control characters are written as `<hex>` (`ESC` becomes
@@ -504,16 +509,16 @@ WebAssembly; `t` runs them under the screen and shows what flows.
   - The bar counts the shown pipeline's entries in, filtered by the flow,
     dropped by sinks, and missed by the screen: the view never slows a
     pipeline.
-- Live view: from 40 rows it sits under the diagram; shorter, it takes the
-  screen while it has the keys. `t` or Esc gives them back to the diagram
-  (Esc first closes an open detail).
-  - The canvas cursor picks its point, so the view opens on the first
-    source: a source shows its entries, a flow stage (or the filter the
-    inspector is on) what reaches it, the heartbeat its own, the sinks (or
-    no part) what they got. `v` switches between all (what the flow dropped
-    dimmed), what passed and what was dropped there, with no restart; the
-    title says which. lw's own log shows from warn up at every point; `s`
-    also offers it alone, debug too.
+- Live view: from 40 rows it sits under the diagram, unless the inspector is
+  open below the canvas; otherwise it takes the screen while it has the keys.
+  `t` or Esc gives them back to the diagram (Esc first closes an open detail).
+  - The canvas cursor picks its point, the first source until it moves: a
+    source shows its entries, a flow stage (or the filter the inspector is
+    on) what reaches it, the heartbeat its own, the sinks (or no part) what
+    the flow offered them (a full sink drops its copy alone). `v` switches
+    between all (what the flow dropped dimmed), what passed and what was
+    dropped there, with no restart; the title says which. lw's own log shows
+    from warn up at every point; `s` also offers it alone, debug too.
   - A row is the entry's time, level, tag, message and fields, read from any
     line shape: LogWisp's json, lw's own log, vif records, pino, zap, slog,
     docker, journald, ECS, GCP, logfmt, or text after an RFC3339 stamp and

doc/configuration.md

@@ -1,7 +1,8 @@
 # Configuration Reference
 
 LogWisp is configured with TOML. A complete annotated file listing every option
-and its default ships as [`config/logwisp.toml`](../config/logwisp.toml).
+and its default ships as [`config/logwisp.toml`](../config/logwisp.toml);
+`lw config init` writes it ([CLI](cli.md#lw-config)).
 
 ## Configuration Precedence
 
@@ -45,8 +46,8 @@ Top-level keys, each as type and default:
 - `auto_reload` (bool, `false`): watch the config file and reload pipelines on
   change.
 - `color` (string, `"auto"`): level names in color on console sinks, which may
-  set their own: `"auto"` (a terminal, `NO_COLOR` unset, `TERM` not `dumb`),
-  `"always"` or `"never"`; see the [console sink](sinks.md#console).
+  set their own: `"auto"` (a terminal, `NO_COLOR` empty or unset, `TERM` not
+  `dumb`), `"always"` or `"never"`; see the [console sink](sinks.md#console).
 
 `--version`, `--check`, `--dump`, `--schema` and `--tui` are switches, not
 settings; the matching variables are ignored.
@@ -80,8 +81,9 @@ retention_hours   = 168.0
 - `all`: files plus split console.
 - `none`: no application logging.
 
-`[logging.file]` applies only to the `file` and `all` modes. Its limits delete
-any `.log` file in `directory`, lw's or not: give lw a directory of its own.
+`[logging.file]` applies only to the `file` and `all` modes. Its limits, and a
+floor of 10 MB free on the disk that no key changes, delete any `.log` file in
+`directory`, lw's or not: give lw a directory of its own.
 `retention_hours = 0` keeps files. The `stdout`, `split` and `all` modes mix
 the log into a console sink's stdout. `[logging.console].target` is validated
 (`stdout`, `stderr`, `split`) but not applied: `logging.output` chooses the
@@ -220,11 +222,12 @@ pipelines[0].plugin_sinks[http].config: unknown key "tls.enabeld"
 - `auth` enums (`identity`, `node_binding`) are checked whatever `type` is; a
   block naming peers or credentials with `type` `none` or unset is refused
   ([Security](security.md#validation-1)).
-- Once the rest is valid, the file itself: the first key the schema does not
-  declare outside a plugin's `config` is refused (`config file "…": unknown
-  key "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path
-  would drop the whole table it heads. A top-level `config_file`, `version`,
-  `check` or `dump` is ignored.
+- The file itself, after the rest at start and by `lw --check` but before it
+  at a reload: the first key the schema does not declare outside a plugin's
+  `config` is refused (`config file "…": unknown key
+  "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path would
+  drop the whole table it heads. A top-level `config_file`, `version`, `check`
+  or `dump` is ignored.
 - Files a plugin names (certificates, keys, CA files, credentials, password
   files) are read when it is built: at start, at reload and by `lw --check`.
   Those failures name the pipeline and plugin.
@@ -234,14 +237,15 @@ exits 1 ([Networking](networking.md#network-plugins)).
 
 ## Hot Reload
 
-`auto_reload = true` watches the configuration file. `SIGHUP` or `SIGUSR1`
-rereads it even when watching is off and always rebuilds, so certificates and
-credentials rotate without a TOML edit. CLI and environment overrides,
-pipelines included, are captured at startup and keep their precedence. A
-discovered default file that does not exist is not an error, so an instance
-configured by flags or variables alone still rebuilds; once read, though, its
-removal fails the reload and the running service stays as it is; restart to
-run without the file.
+`auto_reload = true` watches the configuration file: lw reads it every second
+and applies a change once two reads in a row agree, so within about 2 s.
+`SIGHUP` or `SIGUSR1` rereads it even when watching is off and always
+rebuilds, so certificates and credentials rotate without a TOML edit. CLI and
+environment overrides, pipelines included, are captured at startup and keep
+their precedence. A discovered default file that does not exist is not an
+error, so an instance configured by flags or variables alone still rebuilds;
+once read, though, its removal fails the reload and the running service stays
+as it is; restart to run without the file.
 
 Reload builds a new service from a detached snapshot of the configuration,
 validated again, and shuts the old one down only if that succeeds: parse,
@@ -271,6 +275,13 @@ Each TOML type, its Go type, and its command-line / environment form:
 - Boolean (`bool`): `true` / `false`, or a bare flag for `true`.
 - Array (`[]T`): only in [pipeline specs](cli.md#pipelines), as a repeated key.
 - Table (struct): nested path with `.` (flags) or `_` (environment).
+- Path (`string`: a plugin's `directory` and `*_file` keys,
+  `logging.file.directory`): `~` alone or a leading `~/` is the home directory
+  where lw opens it (`--dump` prints it as typed); `~user` stays as typed; a
+  relative path starts at lw's working directory. The configuration file's own
+  path (`-c`, `LOGWISP_CONFIG_*`) is read as typed.
 
 Integer keys refuse fractions and out-of-range values; number keys refuse inf
-and NaN. Configuration files are TOML only.
+and NaN. A suffix names a key's unit: `_ms` milliseconds, `_mb` megabytes of
+1,000,000 bytes, `_bytes` bytes, `_hours` hours. Configuration files are TOML
+only.

doc/deployment.md

@@ -10,7 +10,9 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
 ## Roles
 
 - `edge`
-  - tails each `--log-dir` (not recursive) for `--log-pattern` (default `*.log`)
+  - tails each `--log-dir` (not recursive) for `--log-pattern` (default
+    `*.log`), a newly found file from its end, or with `--from start` from its
+    first line
   - forwards to `--aggregator` over `tcp_chain` (port 9440) or, with
     `--chain http`, `http_chain` (port 9441)
   - labels its entries `--node`: the short host name, or the jail name
@@ -27,15 +29,16 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
 - `docker`, `podman` (Linux)
   - the container runs `--read-only --cap-drop ALL --security-opt
     no-new-privileges -u 65532:65532 --restart unless-stopped`
-  - `--config-dir` (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`,
-    and the credentials files, owned by 65532 and mounted read-only at
-    `/etc/logwisp`
+  - `--name` (default `logwisp-ROLE`) names the container; `--config-dir`
+    (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`, and the
+    credentials files, owned by 65532 and mounted read-only at `/etc/logwisp`
   - log directories mount read-only and the output directory writable, each at
     its host path; `--log-group` adds that group's id to the container
   - listener ports publish on `--listen`, an IP address; the process inside
     binds `0.0.0.0`
   - `--network NET` joins a user network, created when missing, where edges
-    reach an aggregator by container name; `host` publishes nothing
+    reach an aggregator by container name; `host` publishes nothing, and the
+    process binds `--listen` itself
   - `--image` defaults to `logwisp:dev`, the `make image` tag; `--build` builds
     it from the repository. Behind an HTTPS-intercepting proxy add
     `--build-ca FILE` and, for a proxy on loopback,
@@ -47,10 +50,11 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
     script does not relabel, since `:z` on a shared directory such as
     `/var/log` would relabel it for every other service too
 - `native`, Linux
-  - needs systemd; installs `lw` to `/usr/local/bin` unless one is installed
-    (`--bin FILE`, or `--build` for `make build`), and
-    `deploy/package/logwisp.service`, `logwisp.sysusers` and
-    `logwisp.tmpfiles` under `/etc`
+  - needs systemd; installs `deploy/package/logwisp.service`,
+    `logwisp.sysusers` and `logwisp.tmpfiles` under `/etc`, and `--bin FILE`
+    or the repository's `bin/lw` (`--build` runs `make build` first) to
+    `/usr/local/bin`, unless neither flag is given and an `lw` is in
+    `/usr/local/bin` or `/usr/bin`
   - a package's unit (`/usr/lib/systemd/system/logwisp.service`) is used as it
     is, with the `lw` its `ExecStart` runs; `--bin` and `--build` then stop the
     run, since the service would never run that binary
@@ -58,14 +62,17 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
   - refuses log and output directories under `/tmp` and `/var/tmp`: the unit's
     `PrivateTmp=yes` gives the service its own, empty ones
   - adds `logwisp.service.d/deploy.conf` when the node needs it:
-    `ReadWritePaths` for an output outside `/var/log/logwisp`,
-    `ProtectHome=read-only` for logs or output under `/home`,
+    `ReadWritePaths` for an output other than `/var/log/logwisp` or
+    `/var/lib/logwisp`, `ProtectHome=read-only` for logs or output under
+    `/home`, `/root` or `/run/user`,
     `SupplementaryGroups` for `--log-group`, `CAP_NET_BIND_SERVICE` for a port
     below 1024
 - `native`, FreeBSD
-  - installs `deploy/package/logwisp.rc` as `/usr/local/etc/rc.d/logwisp`,
-    creates the `logwisp` user with `pw`, enables it with `sysrc`; LogWisp logs
-    to syslog under the tag `logwisp`
+  - installs `lw` to `/usr/local/bin` unless one is there (`--bin` and
+    `--build` as on Linux), and `deploy/package/logwisp.rc` as
+    `/usr/local/etc/rc.d/logwisp`; creates the `logwisp` user with `pw` and
+    adds it to `--log-group`, enables the service with `sysrc`; LogWisp logs to
+    syslog under the tag `logwisp`
   - writes `/usr/local/etc/logwisp/logwisp.toml`
   - `--jail NAME` writes the files and runs the commands through `jexec`, so
     the jail's own symlinks cannot redirect a write onto the host; the jail
@@ -98,9 +105,9 @@ needs `--force`.
     certificate must name the address dialed (an IP SAN for an IP), or the
     name given as `--server-name`
 - `--auth` on the chain link
-  - `mtls`: edges present `--cert-file`; the aggregator admits the `--allow`
-    CNs (none: any certificate the CA issued); an edge's `--allow` pins the
-    aggregator's CN
+  - `mtls`: edges present `--cert-file`; the aggregator verifies them with
+    `--ca-file` and admits the `--allow` CNs (none: any certificate the CA
+    issued); an edge's `--allow` pins the aggregator's CN
   - `scram`: the aggregator creates each `--add-user`, its password generated
     into `--secrets-dir`, an absolute path (default `~/logwisp-secrets`), and
     handed to `lw auth add-user`, natively or through the image; copy each
@@ -108,8 +115,9 @@ needs `--force`.
     `.pass` is the default `--username`
   - either binds the node label to the authenticated identity
 - `--sink-auth` on the `http` and `tcp` outputs
-  - `mtls` with `--sink-allow`, or `scram` with `--add-viewer` (a separate
-    `viewers.toml`); viewers log in with `lw auth token` and `lw auth stream`
+  - `mtls` with `--ca-file` and `--sink-allow`, or `scram` with `--add-viewer`
+    (a separate `viewers.toml`); viewers log in with `lw auth token` and
+    `lw auth stream`
   - without it the outputs admit anyone who reaches them: bind `--listen` to a
     trusted address
 - without `--auth` or `--sink-auth`, the auth flags imply their mode:
@@ -156,10 +164,11 @@ Variants:
 - the aggregator on the host itself: drop `--jail logs`
 - no daemon per jail: one `standalone` or `edge` on the host tails each jail's
   logs through the host path (`--log-dir JAIL_PATH/var/log`); the entries then
-  carry the host's label and the bare file name, so files of the same name in
-  two jails cannot be told apart
-- a Linux aggregator: run step 2 with `--runtime docker` on that host and point
-  the edges at it; the chain protocol is the same
+  carry the bare file name and, from an edge, the host's label, so files of
+  the same name in two jails cannot be told apart
+- a Linux aggregator: run step 2 on that host with `--runtime docker` in place
+  of `--runtime native --jail logs`, and point the edges at it; the chain
+  protocol is the same
 
 ## Linux host with containers
 

doc/filters.md

@@ -129,9 +129,10 @@ time, which is exactly what you want in a log hot path.
 - Anchors: `^app\.log ` (the text starts with the source), `ERROR$`
 - Any characters: `.*exception.*`
 
-Remember that TOML basic strings process escapes, so a regex backslash needs
-doubling: `"\\berror\\b"`. TOML literal strings avoid the issue:
-`'\berror\b'`.
+TOML strings process escapes, so a regex backslash needs doubling:
+`"\\berror\\b"`. LogWisp's TOML reader has no literal (single-quoted)
+strings, so `'\berror\b'` does not load, and a single `\b` is TOML's
+backspace: `"\berror\b"` loads and never matches.
 
 Anchors apply to the assembled match text, which begins with the source name,
 so `^ERROR` never matches an entry that has a source; use `\bERROR\b`.
@@ -176,10 +177,11 @@ Each filter reports `type`, `logic`, `pattern_count`, `total_processed`,
 
 ## Performance
 
-Patterns compile once at startup. Every entry that reaches the filter stage is
-evaluated against every filter until one rejects it, so cost scales with the
-number of patterns and their complexity. Prefer literal substrings and simple
-alternations over broad `.*` wildcards.
+Patterns compile when a pipeline is built, at startup and on each reload.
+Every entry that reaches the filter stage is evaluated against every filter
+until one rejects it, and each filter runs all of its patterns whatever its
+`logic`, so cost scales with the number of patterns and their complexity.
+Prefer literal substrings and simple alternations over broad `.*` wildcards.
 
 Filters log every pattern match and decision at DEBUG: useful when a filter
 misbehaves, expensive on a busy pipeline, where `logging.level` stays at

doc/formatters.md

@@ -11,8 +11,11 @@ flags            = 0
 timestamp_format = ""
 ```
 
-One formatter serves the whole pipeline. Sinks receive an identical payload,
-one record ending in exactly one newline; there is no per-sink formatting.
+One formatter serves the whole pipeline. Sinks receive an identical payload
+ending in a newline; under `raw` a message's own newlines stay, so a record
+can span lines. Apart from the `http` sink's SSE framing, only the console
+sink alters what it writes, escaping control characters on a terminal and
+painting level names ([console](sinks.md#console)).
 When you need two shapes of the same data, run two pipelines, or chain to a
 node that formats differently.
 
@@ -38,7 +41,10 @@ bypasses formatting and sanitization: the message reaches the sink as the
 source produced it, control characters included, with no timestamp, level or
 source prefix added, and a newline appended unless the message ends in one.
 An entry that also carries `fields` gets the fields JSON appended verbatim
-after a single space; `raw` never drops data and never re-encodes it.
+after a single space, so `FlagRaw` (`1`) never drops data or re-encodes it.
+Heartbeats format without it, so in `raw` their `include_stats` fields print
+in Go's map syntax (`heartbeat map[beat_count:0 type:heartbeat]`), as any
+entry's fields do under a non-zero `flags` without `1`.
 
 ```toml
 [pipelines.flow.format]
@@ -48,14 +54,22 @@ type = "raw"
 The closest to verbatim transport is the `console` source, or the `file`
 source with `raw = true`. The whole line, terminator removed, becomes the
 message and `fields` stays empty; `raw` writes it back with one `\n`. It is not
-byte-exact: blank lines are skipped, CRLF becomes LF, and a line over 1 MiB
-continues in the next entry. The `file` source's JSON branch splits a line
-into message and fields, so `raw` writes `<msg> <fields>` rather than the
-original object.
+byte-exact: empty lines are skipped, CRLF becomes LF, and a line over 1 MiB,
+or one the `file` source reads while it is still being written, continues in
+the next entry. The `file` source's JSON branch splits a line into message
+and fields, so `raw` writes `<msg> <fields>` rather than the original object.
 
 ### txt
 
-Human-readable line output with a timestamp and level.
+Human-readable lines, `<time> <level> <source> <message>`, any fields after
+the message. The message and the fields are each double-quoted when empty or
+holding whitespace, a non-printable character, a quote, or a shell or `=`
+character (`$`, `;`, `<`, `*`, ...), with `"` and `\` backslash-escaped
+inside; the time and source never are:
+
+```
+2026-01-02T15:04:05.123Z ERROR edge-01/app.log "connection refused"
+```
 
 ```toml
 [pipelines.flow.format]
@@ -103,8 +117,8 @@ you need to override the defaults.
 
 With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
 (timestamp + level) for every other type. `8` is added automatically whenever
-an entry carries parseable `fields` and `1` is not set (it shapes `json` only);
-`1` always wins.
+an entry's `fields` is a non-empty JSON object and `1` is not set (it shapes
+`json` only); `1` always wins.
 
 `txt` and `json` show timestamp and level unless a suppress bit says otherwise
 (suppress wins over show), so `2` and `4` alone change nothing. Examples:
@@ -123,15 +137,19 @@ Each policy, then what to use it with:
 - `txt`: writes `<` and non-printable characters as `<hex>` of their bytes
   (`<b>` becomes `<3c>b>`).
   - File and console sinks.
-- `json`: escapes control characters for safe JSON embedding.
+- `json`: writes control characters as JSON escapes (`\t`, `\u001b`) into
+  the text, which `type = "json"` quotes again: a reader that decodes the
+  record gets the escape, never the character.
   - `type = "json"`.
 - `shell`: strips shell metacharacters, whitespace, and control characters.
   - Data that will be passed to a command.
 
-Every policy but `raw` replaces invalid UTF-8 with U+FFFD. With
-`type = "raw"` and `flags = 0` no policy applies (`1` bypasses it). When
-`sanitizer_policy` is omitted, it is the policy of the same name as `type`, so
-the safe pairing is the default.
+Every policy but `raw` replaces invalid UTF-8 with U+FFFD. `type = "json"` is
+valid JSON under any policy, since its own quoting escapes control characters
+and replaces invalid UTF-8; with policy `raw` the decoded message keeps its
+control characters. With `type = "raw"` and `flags = 0` no policy applies (`1`
+bypasses it). When `sanitizer_policy` is omitted, it is the policy of the same
+name as `type`, so the safe pairing is the default.
 
 > `shell` strips dangerous characters but is **not** sufficient to make a string
 > safe for shell construction. Pass arguments through `exec` argv instead of
@@ -150,13 +168,15 @@ downstream parsers and grep patterns need to know.
 
 ## Structured Fields
 
-When an entry carries `Fields` (raw JSON) and `FlagRaw` is not set, the
-formatter parses it and adds the `StructuredJSON` flag. Fields reach a pipeline
-from the `file` source when a tailed line parses as JSON with a `fields` key,
-from the `tcp_chain` and `http_chain` sources when the upstream entry carried
-them, and from the heartbeat when `include_stats = true`. In `json` output
-such an entry is `{"time":…,"level":…,"trace":…,"message":…,"fields":{…}}`;
-without fields the message sits in `"fields":[…]`.
+When an entry's `fields` is a non-empty JSON object and `1` is not set, the
+formatter parses it and adds `8`; without `1`, any other `fields` value
+(`[1,2]`, `{}`, `"s"`) is dropped. Fields reach a pipeline from the `file`
+source when a tailed line parses as JSON with a `fields` key, from the
+`tcp_chain` and `http_chain` sources when the upstream entry carried them, and
+from the heartbeat when `include_stats = true`. In `json` output such an entry
+is `{"time":…,"level":…,"trace":…,"message":…,"fields":{…}}`; without fields
+the message sits in `"fields":[…]`. In `txt` the fields follow the message in
+Go's map syntax, keys sorted: `"disk full" "map[dev:sda pct:99]"`.
 
 ## Choosing a Configuration
 

doc/installation.md

@@ -24,7 +24,7 @@ make build              # bin/lw; plain `make` only lists the targets
 The Makefile works with GNU make and BSD make alike. Targets:
 
 - Build
-  - `make build`: `bin/lw`, with version, commit and build time from git
+  - `make build`: `bin/lw`, with version and commit from git, and the build time
   - `make release`: the same, static (`CGO_ENABLED=0`), `-trimpath`, stripped
   - `make dev`: built with the race detector
   - `make wasm`: `bin/lwconf/`, the configuration engine a website drops in:
@@ -36,8 +36,8 @@ The Makefile works with GNU make and BSD make alike. Targets:
     [Architecture](architecture.md#configuration-engine)
   - `make version`: the metadata a build would embed
   - `make completion`: regenerate the bash, zsh and fish completion in
-    `deploy/package/completion/` from `cmd/lw`'s `shorts` and `commands`
-    tables; `go test ./cmd/lw` fails while they differ
+    `deploy/package/completion/` from lw's own tables (its flags, commands
+    and presets); `go test ./cmd/lw` fails while they differ
   - `make clean`: remove `bin/`
 - Check
   - `make test`: the Go tests
@@ -79,8 +79,9 @@ checkout.
 first, or in one command: `make release install`. It honours `DESTDIR`,
 `PREFIX`, `SYSCONFDIR`, `BINDIR` (default `$PREFIX/bin`, the path the systemd
 unit and the rc.d script run `lw` from), `MANDIR`, `DOCDIR` and `LICENSEDIR`
-(under `$PREFIX/share` by default) and `SERVICE`; `make help` prints the
-values in effect:
+(under `$PREFIX/share` by default) and `SERVICE`; `make help` prints
+`PREFIX`, `SYSCONFDIR`, `DESTDIR`, `SERVICE` and `INSTALL_OS` as they are in
+effect:
 
 - Linux defaults: `PREFIX=/usr`, `SYSCONFDIR=/etc`. Outside a package
   manager, prefer `PREFIX=/usr/local`; systemd reads units, sysusers and
@@ -190,9 +191,11 @@ rc.conf variables:
 
 `make deb` builds a release and packages what `make install` stages, in
 Debian's layout, as `bin/logwisp_VERSION_ARCH.deb`, the architecture taken
-from the binary. A release tag gives the version; an untagged or modified
-tree's sorts below every release and by build time, so each build upgrades
-the last. `DEB_MAINTAINER` names the maintainer.
+from the binary. A release tag gives the version, and a commit past one
+sorts between it and the next release. A modified tree, or one with no tag
+in its history, adds the build time, so each rebuild upgrades the last; with
+no tag, the version sorts below every release. `DEB_MAINTAINER` names the
+maintainer.
 
 ```bash
 make deb
@@ -213,8 +216,8 @@ sudo systemctl enable --now logwisp
 
 `make arch` builds the PKGBUILD in `deploy/package/arch/` from the working
 tree, committed or not, into `bin/arch/`, then prints the `pacman -U` line
-that installs the package. It needs `base-devel` and `go`, and runs as a
-user: `makepkg` refuses root.
+that installs the package. It needs a git checkout, `base-devel`, `git` and
+`go`, and runs as a user: `makepkg` refuses root.
 
 ```bash
 make arch
@@ -343,10 +346,10 @@ docker run -d --name logwisp \
   ```
 
   Without a file lw logs less ([CLI](cli.md#built-in-defaults)): add
-  `-e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true`; the image
-  sets neither, as a variable would override a mounted file's value. Without
-  `-i`, stdin is `/dev/null`, so a container needs a source other than
-  `console`.
+  `-e LOGWISP_LOGGING_LEVEL=info` (`debug`, with
+  `-e LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the image sets
+  neither, as a variable would override a mounted file's value. Without `-i`,
+  stdin is `/dev/null`, so a container needs a source other than `console`.
 
 - Secrets: only as mounted files, at `/run/secrets` (Docker or Compose
   secrets, a Kubernetes secret volume), named by `tls.key_file`,
@@ -354,8 +357,8 @@ docker run -d --name logwisp \
   the environment: `docker inspect` and `/proc` show it. Make each file
   readable by 65532 only (`chown 65532` and mode `0400`).
 - Privileges: keep the image's non-root user, a read-only root filesystem,
-  `--cap-drop ALL` and `no-new-privileges`. Without capabilities a listener
-  needs a port of 1024 or above; publish it on any host port.
+  `--cap-drop ALL` and `no-new-privileges`. Without capabilities, a port of
+  1024 or above binds on every runtime; publish it on any host port.
 - Listeners bind `0.0.0.0` inside the container; `127.0.0.1` would be
   unreachable through `-p`.
 - Writable data: a `file` sink or `logging.file` needs a volume writable by
@@ -413,18 +416,22 @@ starting it, run `lw --check -c FILE`; see
 
 ## Test Scripts
 
-The end-to-end scripts in `test/` run against `bin/lw` and share
-`test/lib.sh`. `make e2e` runs each with `--auto` and reports it as PASS, FAIL
-or SKIP (exit 77, e.g. no IPv6 loopback); only a failure fails the target.
+The end-to-end scripts in `test/` run against `bin/lw` (`LOGWISP_BIN` names
+another) and share `test/lib.sh`. `make e2e` runs each with `--auto` and
+reports it as PASS, FAIL or SKIP (exit 77, e.g. no IPv6 loopback); only a
+failure fails the target.
 
 - Modes:
   - without flags: start the daemons, print a guide of the ports and one-line
-    commands (run `. test/run-X/env` first for `$LW`, `$CA` and the token
-    helper), and wait for Ctrl-C
+    commands (where the guide says so, run `. test/run-X/env` first for `$LW`,
+    `$CA` and the token helper), and wait for Ctrl-C. `acl-test.sh`,
+    `passthrough-test.sh`, `pipe-test.sh` and `preset-test.sh` are one-shot:
+    either mode runs the checks and exits.
   - `--auto`: run the checks and tear down; `--keep` leaves the daemons up
     after a pass
   - `--follow`: show only WARN, ERROR and key events from the daemons
-- Daemons log only to `test/run*/log/NAME.out`, never to the terminal.
+- Daemons log only to `log/NAME.out` in the run directory, never to the
+  terminal.
 - Colour only on a terminal; `NO_COLOR` turns it off, `FORCE_COLOR=1` on.
 - Exit status: 0 all passed, 1 a failure, 77 skipped.
 - Each script owns a port range and a gitignored run directory:

doc/lw.1

@@ -1,4 +1,4 @@
-.Dd October 11, 2026
+.Dd October 14, 2026
 .Dt LW 1
 .Os
 .Sh NAME
@@ -53,7 +53,7 @@ joins a console source to a console sink in the raw format, without a filter
 or rate limit; a configuration file that
 defines no pipelines runs it too.
 Each line is written with one newline, whether it ended in a newline, a
-carriage return and newline, or nothing; blank lines are skipped, and a line
+carriage return and newline, or nothing; empty lines are skipped, and a line
 over 1 MiB continues in the next one.
 When standard input is a terminal,
 .Nm
@@ -74,7 +74,8 @@ Under a supervisor, standard input is usually
 .Pa /dev/null :
 a console source there ends at once.
 .Pp
-The console source and sink never drop a line.
+While their pipeline runs, the console source and sink never drop a line;
+a stop or reload can drop lines still in flight.
 A busy pipeline slows the reading of standard input; a console sink whose
 output is slow makes its pipeline wait, other sinks included.
 On a terminal the console sink writes control characters, escape sequences
@@ -133,7 +134,10 @@ The options are as follows:
 .It Fl c Ar file , Fl \-config Ar file , Fl \-config= Ns Ar file
 Read the configuration from
 .Ar file ,
-which must exist.
+which must exist; a relative path that climbs above the current directory
+with
+.Ql ..
+is refused, so give such a path absolute.
 The last one given wins.
 .It Fl t , Fl \-check
 Build every pipeline and plugin as a start would, loading options, TLS files
@@ -197,7 +201,7 @@ and the terminal reminder; console sinks still write.
 .It Fl \-color Op Ar when
 Color the level names console sinks write:
 .Cm auto
-.Pq the default: on a terminal, with Ev NO_COLOR unset and Ev TERM not Cm dumb ,
+.Pq the default: on a terminal, with Ev NO_COLOR empty or unset and Ev TERM not Cm dumb ,
 .Cm always
 .Pq what a bare Fl \-color No means
 or
@@ -562,7 +566,9 @@ and
 is required; both may be given.
 .El
 .Pp
-A listener needs no files:
+A listener with
+.Cm tls.enabled=true
+needs no files:
 .Cm tls.self_signed=true ,
 or
 .Cm tls.issuer_cert_file
@@ -601,8 +607,8 @@ when it exists; otherwise one is generated and written there, or printed
 once to standard output.
 .Fl \-generate
 replaces an existing password without reading one.
-A missing or empty credentials file is created with mode 0600; an existing
-one keeps its owner and mode.
+A missing credentials file is created with mode 0600; an existing one keeps
+its owner and mode, including an empty one made beforehand to choose them.
 .It Cm remove-user Fl \-credentials Ar file Fl u Ar name
 Remove a user.
 The last user cannot be removed.
@@ -694,16 +700,28 @@ counting from 1, adds more, as in
 .It Ev LOGWISP_FORMAT , LOGWISP_RATE_LIMIT , LOGWISP_HEARTBEAT
 The pipeline's format, rate limit and heartbeat specifications.
 .It Ev NO_COLOR , TERM
-Set to anything,
+Set to anything but the empty string,
 .Ev NO_COLOR
 turns
 .Cm auto
 color off, as does
-.Ev TERM=dumb .
+.Ev TERM=dumb ,
+which also keeps the
+.Fl \-tui
+screen from starting.
+.It Ev LC_ALL , LC_CTYPE , LANG
+The first one set names the locale; one other than UTF-8 makes the
+.Fl \-tui
+screen draw in ASCII.
 .It Ev HOME
 Where
 .Pa ~/.config/logwisp/logwisp.toml
-is looked for.
+is looked for, and what a leading
+.Ql ~
+names in a plugin's directory and file options, a preset's
+.Cm path
+and
+.Fl \-logging.file.directory .
 .El
 .Pp
 Secrets never go in the environment: TLS keys, credentials files and
@@ -761,8 +779,12 @@ a full one is renamed with its time, as
 .Nm
 exits 0 at the end of input, after a clean shutdown, and for
 .Fl \-version ,
+.Fl \-schema ,
 help,
-.Fl \-dump
+.Fl \-dump ,
+a
+.Fl \-tui
+screen left by printing or quitting,
 and a configuration
 .Fl \-check
 accepts; 1 on a failure, such as a configuration that does not load, or a
@@ -796,7 +818,9 @@ Shut down gracefully.
 .It Dv SIGHUP , SIGUSR1
 Reread the configuration file, certificates and credentials, and rebuild
 every pipeline.
-An invalid configuration leaves the running service untouched.
+An invalid configuration leaves the running service untouched; one that builds
+but fails to start, as on a port another process holds, leaves nothing running
+until a later reload succeeds.
 Before the service runs,
 .Dv SIGHUP
 is ignored, as under
@@ -831,9 +855,9 @@ $ tail \-F app.log | lw \-\-format txt
 Serve standard input as a live stream on the loopback address only, since
 the
 .Cm http
-sink listens on every address by default, and read it from another terminal
-as server-sent events; a client gets the lines that arrive while it is
-connected:
+sink listens on every IPv4 address by default, and read it from another
+terminal as server-sent events; a client gets the lines that arrive while it
+is connected:
 .Bd -literal -offset indent
 $ journalctl \-f | lw \-\-sink http:host=127.0.0.1,port=8080
 $ curl \-N http://127.0.0.1:8080/stream
@@ -924,7 +948,7 @@ generate the password into a file, then reload the server:
 .Xr daemon 8
 .Pp
 The guides to configuration, plugins, security and deployment, installed in
-.Pa share/doc/logwisp
+.Pa share/doc/logwisp/doc
 under the installation prefix and kept at
 .Lk https://github.com/lixenwraith/logwisp/tree/main/doc
 .Sh SECURITY CONSIDERATIONS

doc/networking.md

@@ -8,8 +8,8 @@ see [Chaining](chaining.md).
 
 Every listener and dialer keeps strictly to the family of its `host`:
 
-- IPv4 literal (`127.0.0.1`, the default `0.0.0.0`): IPv4 only (`tcp4`). An
-  empty `host` is the IPv4 wildcard, as `0.0.0.0`.
+- IPv4 literal (`127.0.0.1`, the default `0.0.0.0`): IPv4 only (`tcp4`). A
+  listener's empty `host` is the IPv4 wildcard, as `0.0.0.0`.
 - IPv6 literal (`::1`, `::`, `fe80::1%eth0`): IPv6 only (`tcp6`).
   - `::` takes no IPv4 connections, on Linux and FreeBSD alike, whatever the
     system's `bindv6only` default.
@@ -38,8 +38,9 @@ Every listener and dialer keeps strictly to the family of its `host`:
   - SCRAM throttling counts an IPv6 client by its /64, so the hosts on a link
     share one link-local budget per interface (zone): each can pick any
     `fe80::/64` address.
-  - A peer with a zone never matches `auth.trusted_proxies`: put the proxy on
-    loopback or a routed address.
+  - A peer with a zone never matches `auth.trusted_proxies` or
+    `acl.proxy_from`: put the proxy on loopback or a routed address.
+    `acl.allow` and `deny` match it without its zone.
 
 ## Network Plugins
 
@@ -79,7 +80,9 @@ deadline where the default is `0` (none); elsewhere it selects the default.
     under `scram` the whole login.
   - `read_timeout_ms`, default `0` (none): idle time between entries.
 - `http_chain` source
-  - `read_timeout_ms`, default `30000`: reading a whole request body.
+  - `read_timeout_ms`, default `30000`: reading a whole request, headers and
+    body; also how long an idle keep-alive connection waits for its next
+    request.
 - `tcp_chain` sink
   - `dial_timeout_ms`, default `5000`: TCP connect.
   - `write_timeout_ms`, default `5000`: one line write.
@@ -89,14 +92,16 @@ deadline where the default is `0` (none); elsewhere it selects the default.
 
 Fixed, non-configurable bounds:
 
-- TLS handshake: 10 s, on all TLS listeners and dialers.
+- TLS handshake: 10 s, on all TLS listeners and dialers; on the `http_chain`
+  source, `read_timeout_ms` when shorter.
 - HTTP read-header timeout: 10 s, on the `http` sink and `http_chain` source.
 - HTTP server shutdown grace: 2 s on the `http_chain` source and the `http`
   sink; on the `http` sink, `write_timeout_ms` instead when set below 2 s.
 - Single entry line: at most 1 MiB, on chain listeners.
 - SCRAM login: 10 s, on the `tcp` sink, chain sinks, `lw auth`, and each
   `/auth` request.
-- SCRAM line or `/auth` body: at most 4 KiB, on `scram` listeners and dialers.
+- Hello and SCRAM lines, `/auth` bodies: at most 4 KiB, on `tcp_chain` sources
+  and on `scram` listeners and dialers.
 
 The `http` sink deliberately leaves the server's `WriteTimeout` unset, since it
 would terminate long-lived SSE streams; per-event deadlines come from
@@ -108,9 +113,10 @@ would terminate long-lived SSE streams; per-event deadlines come from
 `tcp_chain` source, and concurrent streams on the `http` sink (other requests
 are not counted); `0` means unlimited.
 
-- Admission is a load-then-check, so a burst can over-admit by roughly one.
-- On the `tcp` sink and `tcp_chain` source the count is taken at accept, so it
-  bounds concurrent TLS handshakes as well as established sessions.
+- A connection or stream counts only after the check that admits it, so a
+  burst can over-admit by several, more the larger the burst.
+- On the `tcp` sink and `tcp_chain` source a connection counts before its TLS
+  handshake, so the cap bounds handshakes in progress as well as sessions.
 - Over the limit a TCP connection is closed and counted in `rejected_conns`;
   an `http` stream request is answered `503` and counted in
   `rejected_clients`, its connection kept.
@@ -155,10 +161,11 @@ backoff_min_ms = 500
 backoff_max_ms = 30000
 ```
 
-The connection is established lazily, so an edge node starts cleanly even when
-its relay is down and connects as soon as the relay appears. The `tcp_chain`
-sink reports `connected` and `reconnects`; the `http_chain` sink reports
-`request_errors` and `dropped_batches`.
+The connection is established lazily, when the sink first has something to
+send, so an edge node starts cleanly even when its relay is down, and connects
+at its next attempt once the relay is up. The `tcp_chain` sink reports
+`connected` and `reconnects`; the `http_chain` sink reports `request_errors`
+and `dropped_batches`.
 
 Server-side sinks (`tcp`, `http`) do not reconnect; clients are expected to
 retry. Browsers reconnect SSE streams automatically.
@@ -196,7 +203,7 @@ preamble, headers, and entry encoding.
 - The listener's `acl` refused it: `acl_denied` or `acl_limited` rises, and
   its log has `Connection refused by acl` at WARN with a `reason`, once a
   minute at most. The dialer sees `EOF` or `connection reset by peer` (curl:
-  exit 52 or 56).
+  exit 52 or 56; over TLS, 35).
   - `address rules` — `allow` or `deny` excludes the client: the one a PROXY
     header names (`remote_addr`, the proxy as `peer_addr`), else the socket
     peer. Behind the `http` sink's `auth.trusted_proxies` that is the proxy:
@@ -337,7 +344,7 @@ preamble, headers, and entry encoding.
   batch is sent.
 
 **Entries arriving under an unexpected node label**
-- `auth.node_binding` defaults to `force` with any `auth` block, which
+- `auth.node_binding` defaults to `force` under `mtls` or `scram`, which
   relabels every entry with the sender's certificate identity or username. If
   a dashboard suddenly shows a different label, that is why. Use
   `node_binding = "assert"` to keep upstream origin labels on relay-to-relay

doc/operations.md

@@ -25,7 +25,8 @@ A supervisor's stdin is `/dev/null` (systemd's default, `docker run` without
 `-i`), where a console source ends at once. So a service defines its
 pipelines: a file without `[[pipelines]]` runs the built-in stdin-to-stdout
 pipeline and exits 0 at once, which `Restart=on-failure` does not restart, and
-a container configured through variables needs a `LOGWISP_SOURCE`.
+a container configured through variables needs a `LOGWISP_SOURCE`, or a
+`LOGWISP_PRESET` with a source of its own (`aggregator`, or one given a `path`).
 
 **systemd**
 
@@ -50,10 +51,12 @@ sudo service logwisp status
 auto_reload = true
 ```
 
-or send a signal:
+or send `SIGHUP` or `SIGUSR1`; the packaged unit and rc.d script send `SIGHUP`
+on reload:
 
 ```bash
 kill -HUP $(pidof lw)
+sudo systemctl reload logwisp     # rc.d: sudo service logwisp reload
 ```
 
 Signals reread the selected TOML file even when `auto_reload` is disabled.
@@ -67,10 +70,13 @@ running`.
 
 What reload does *not* do:
 
-- Re-apply `logging.*`; application logging is configured once at startup.
+- Re-apply `logging.*`, `quiet` or `auto_reload`; they take effect at startup
+  only.
 - Preserve connections. Listeners close and reopen, and every SSE, TCP, and
   chain client is disconnected. Chain sinks reconnect on their own backoff;
   browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
+- Keep bearer tokens. Each reload draws a new token key, so every token is
+  revoked: run `lw auth token` again, and browser viewers log in again.
 - Notice rotated certificates on its own: certificate files are read at plugin
   construction and `auto_reload` watches only the configuration file, so
   rotation requires `SIGHUP`.
@@ -179,6 +185,11 @@ Each counter, where it is reported, and what a rise means:
 - `reconnects` (`tcp_chain` sink): an unstable link or a flapping downstream
 - `dropped_batches` (`http_chain` sink): downstream is rejecting batches
   permanently
+- `request_errors` (`http_chain` sink): failed posts, each retried under
+  backoff unless refused for good
+- `rejected_requests` (`http_chain` source): batches refused for auth,
+  protocol version or node binding, or a body over `max_body_bytes`, holding
+  a line over 1 MiB, or cut short
 - `synthesized` (chain sinks): events reach the sink without structure
 - `acl_denied`, `acl_limited` (a listener whose `acl` block sets a rule or
   limit; the `http` status always): connections or requests the address
@@ -189,10 +200,10 @@ Each counter, where it is reported, and what a rise means:
 
 LogWisp's own log goes to stderr by default. Without a configuration file lw
 logs less ([CLI](cli.md#built-in-defaults)), so a container configured only
-through variables passes `LOGWISP_LOGGING_LEVEL=info` and
-`LOGWISP_STATUS_REPORTER=true`; the image sets neither, as a variable would
-override a mounted file's value. `lw-deploy.sh` always passes `-c`, so its
-nodes log at `info`. To files:
+through variables passes `LOGWISP_LOGGING_LEVEL=info` (`debug`, with
+`LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the image sets
+neither, as a variable would override a mounted file's value. `lw-deploy.sh`
+always passes `-c`, so its nodes log at `info`. To files:
 
 ```toml
 [logging]
@@ -268,33 +279,42 @@ pipeline `total_dropped_by_sink` (sink backed up?), sink `total_processed`.
 
 **A pipeline stalls behind a console sink**
 
-A console sink never drops: when its output is slow (a paused terminal, a full
-pipe, a stalled log collector) its pipeline waits, and the pipeline's other
-sinks with it. Stdin is read slower; other sources queue, then drop and count
-`dropped_entries`. Where a service's stdout may stall, give it a `file` sink
-instead. A closed stdout pipe ends lw with `SIGPIPE`.
+A console sink never drops while its pipeline runs: when its output is slow
+(a paused terminal, a full pipe, a stalled log collector) its pipeline waits,
+and the pipeline's other sinks with it. Stdin is read slower; other sources
+queue, then drop and count `dropped_entries`. Where a service's stdout may
+stall, give it a `file` sink instead. A closed stdout pipe ends lw with
+`SIGPIPE`.
 
 **File source reads nothing**
 
-- The watcher seeks to end-of-file on start; only content appended afterwards is
-  read. Positions are in memory, so a restart re-seeks to end and anything
-  written during the downtime is lost. `from = "start"` reads each file whole
-  instead, and replays it on every restart.
+- A watcher seeks to end-of-file when its file is first found, at start or by
+  a later scan, so a new file's lines written before that scan are not read.
+  Positions are in memory, so a restart or a reload re-seeks to end, and
+  anything written meanwhile is lost. `from = "start"` reads each file whole
+  instead, and replays it on every restart and reload.
 - `pattern` is a filename glob with `*` and `?` only, and matching is not
   recursive.
 - `check_interval_ms` governs how quickly a *new file* is noticed; tailing an
   open file polls at a fixed 100 ms.
+- lw's log says why: INFO `Waiting for directory` while the directory is
+  missing, WARN `Cannot read directory` or `Cannot read file` when permissions
+  or a service sandbox refuse it, and WARN `Not following lw's own output` for
+  the file lw's stdout or stderr goes to.
 
 **High memory use**
 
 Buffers are bounded, so unbounded growth almost always means many buffers:
-count sinks × `buffer_size`, plus clients × `client_buffer_size`. A `tcp_chain`
-sink blocked on an unreachable downstream also holds its full input queue.
+count each source's and sink's `buffer_size` (a fixed 1000 on the `file`,
+`random` and `null` sources and the `null` sink), clients ×
+`client_buffer_size`, and each `http` sink's `replay_lines`. A `tcp_chain` sink
+blocked on an unreachable downstream also holds its full input queue.
 
 **Chain link not delivering**
 
-Check `connected` and `reconnects` on a `tcp_chain` sink, `parse_errors` on the
-source, and remember `http_chain` waits up to `flush_interval_ms`. For TLS
+Check `connected` and `reconnects` on a `tcp_chain` sink, `request_errors` on
+an `http_chain` sink, `rejected_requests` (`http_chain`) and `parse_errors` on
+the source, and remember `http_chain` waits up to `flush_interval_ms`. For TLS
 problems see [Networking](networking.md#troubleshooting).
 
 **Environment variable override has no effect**
@@ -342,7 +362,9 @@ than masking parts of them. See [Filters](filters.md#common-recipes).
 
 1. Run the new binary's `lw --check -c FILE` against the current
    configuration: unknown or renamed keys fail there.
-2. Stop the old process, install the new binary, start it.
+2. Stop the old process, install the new binary, start it. A `self_signed`
+   listener makes a new key at every start: give its dialers the new
+   `pin_sha256` from its WARN line.
 3. Confirm each pipeline started and that counters are advancing.
 
 **Backup**

doc/security.md

@@ -26,10 +26,10 @@ configure it, and — equally important — what it does not yet do.
   [Browsers behind a TLS-terminating proxy](#browsers-behind-a-tls-terminating-proxy).
 - Server pinning by dialers: certificate identity (`mtls`), bound certificate
   (`scram`).
-- Address rules and per-client connection and request limits on every
-  listener, applied before TLS to the client a PROXY header (v1 or v2) from a
-  listed L4 proxy names, and per request to the client an L7 proxy forwards:
-  see [The ACL Block](#the-acl-block).
+- Address rules and per-client limits (connections on every listener, a
+  request rate on HTTP listeners), applied before TLS to the client a PROXY
+  header (v1 or v2) from a listed L4 proxy names, and per request to the
+  client an L7 proxy forwards: see [The ACL Block](#the-acl-block).
 - Startup warnings for expiring certificates and risky settings: see
   [Startup Warnings](#startup-warnings).
 - Unknown configuration keys rejected: a typo in `tls`, `auth`, `acl` or a
@@ -40,8 +40,8 @@ configure it, and — equally important — what it does not yet do.
     comments;
   - a string in the file is one list entry, so a filter pattern holding a
     comma is not split into patterns that match nothing;
-  - the file is opened without blocking, so a FIFO swapped in for it cannot
-    hang the watcher or a `SIGHUP` reload;
+  - the file must be a regular file and is opened without blocking, so a
+    FIFO in its place fails a reload rather than hanging it;
   - a permission change on the file is reported once and does not stop
     auto-reload;
   - parse and conversion errors name the key, never the value, so a secret
@@ -60,12 +60,15 @@ configure it, and — equally important — what it does not yet do.
   and `auth_rejected` only.
 - More than one identity mode per listener: `auth.identity` takes one of
   `cn`, `san_dns`, `san_uri`, `san_email`.
+- One read of the configuration file: lw's unknown-key check opens it again,
+  blocking, so a FIFO swapped in between the two reads hangs that load.
 
 Two credentials are supported. Certificates (`mtls`) are the one the transport
 already carries: the `tls` block establishes that a peer chains to your CA, and
 the `auth` block decides which of those peers may do what. Passwords (`scram`)
-need no client PKI: the listener holds verifiers, never passwords, and every
-login is bound to the listener's certificate.
+need no client PKI: a credentials file holds verifiers, never passwords, and a
+login is bound to the listener's certificate unless the listener sits behind
+`trusted_proxies`.
 
 ## The TLS Block
 
@@ -139,7 +142,8 @@ built:
   `ca_file`, `server_name`, `insecure_skip_verify` or `pin_sha256` on a
   listener
 - `client_auth = true` with no `client_ca_file`
-- only one of `cert_file` / `key_file`
+- only one of `cert_file` / `key_file`, or of `issuer_cert_file` /
+  `issuer_key_file`
 - a `min_version` that is neither `"1.2"` nor `"1.3"`
 - a key the plugin does not know, at any depth: `unknown key "tls.enabeld"`. A
   misspelled option must not silently leave a protection switched off
@@ -212,8 +216,8 @@ Options by where they apply, each with its default:
 
 Under `mtls` the identity is one string pulled from the peer's verified leaf
 certificate; under `scram` it is the username. The handshake has already
-checked the chain, signature, and validity window, so this is pure field
-selection.
+checked the chain, signature, and validity window (on a dialer with
+`pin_sha256`, only the key), so this is pure field selection.
 
 Modes, each with the field it reads and its typical use:
 
@@ -229,17 +233,17 @@ identity is a refusal, not an empty match.
 ### The allow list
 
 `allow` is an exact-match set; `allow_patterns` holds RE2 patterns. An identity
-passes if it appears in either. Under `scram` the credentials file is the allow
-list, and both keys are refused.
+passes if it appears in either. Under `scram` the listener's users are the
+allow list, and both keys are refused.
 
 Leaving **both** empty under `type = "mtls"` admits any identity the CA vouches
 for. That is deliberate — it is how you enable node binding without enumerating
 a whole fleet — but it is announced rather than silent:
 
 ```
-WARN msg="Auth policy admits any identity the configured CA vouches for"
-     component=tcp_chain_source instance_id=in_tcp
-     hint="set auth.allow or auth.allow_patterns to authorize named peers"
+WARN msg "Auth policy admits any identity the configured CA vouches for"
+     component tcp_chain_source instance_id in_tcp
+     hint "set auth.allow or auth.allow_patterns to authorize named peers"
 ```
 
 Anchor your patterns. `allow_patterns = ["edge-\\d{2}"]` matches
@@ -275,8 +279,9 @@ attribution survives.
 When binding is active the source says so at startup:
 
 ```
-INFO msg="Node labels bound to peer identity; trust_node is ignored"
-     component=tcp_chain_source node_binding=force trust_node=true
+INFO msg "Node labels bound to peer identity; trust_node is ignored"
+     component tcp_chain_source instance_id in_tcp
+     node_binding force trust_node true
 ```
 
 ### Dialer-side pinning
@@ -284,7 +289,8 @@ INFO msg="Node labels bound to peer identity; trust_node is ignored"
 Under `mtls`, the block on a chain sink pins the *server's* identity. Hostname
 verification already proves the server holds a certificate valid for the address
 you dialed; pinning additionally requires that certificate to name an identity
-you listed.
+you listed. With `pin_sha256` the key's holder writes that certificate, so the
+pin is the whole check.
 
 ```toml
 [pipelines.plugin_sinks.config.auth]
@@ -319,8 +325,9 @@ files (`credentials_file`, `password_file`) when the plugin is built:
   `identity` but no `tls.client_auth`
 - a `scram` dialer without `username` and `password_file`, with
   `credentials_file`, `token_lifetime_ms`, `trusted_proxies` or `identity`
-  (server pinning is `mtls` only), or with a password file that is empty or
-  over 1024 bytes
+  (server pinning is `mtls` only)
+- when the plugin is built: a `password_file` holding no password or over
+  1024 bytes, or under 8 bytes for a listener's one user
 - `trusted_proxies` on anything but the `http` sink, with `identity`, or with
   an entry that is neither an address nor a CIDR; `login_page` or
   `viewer_page` without `trusted_proxies`, `viewer_page` without `login_page`
@@ -576,10 +583,10 @@ allow = ["relay.example.org"]
 Startup logs report the transport flags and the compiled policy:
 
 ```
-INFO msg="TCP chain source initialized" ... tls=true mtls=true
-     auth="mtls identity=cn allow=[2 exact, 0 pattern(s)] node_binding=force"
-INFO msg="TCP chain sink initialized"   ... tls=true mtls=true
-     auth="mtls identity=cn allow=[1 exact, 0 pattern(s)] node_binding=none"
+INFO msg "TCP chain source initialized" ... tls true mtls true
+     auth "mtls identity=cn allow=[2 exact, 0 pattern(s)] node_binding=force"
+INFO msg "TCP chain sink initialized"   ... tls true mtls true
+     auth "mtls identity=cn allow=[1 exact, 0 pattern(s)] node_binding=none"
 ```
 
 A client that presents no certificate is refused during the handshake
@@ -588,8 +595,8 @@ A client whose certificate is valid but whose identity is not authorized gets
 past the handshake and is refused by the policy:
 
 ```
-WARN msg="Connection rejected by auth policy" component=tcp_chain_source
-     remote_addr=<addr> error="auth: refused: identity \"edge-99\" is not allowed"
+WARN msg "Connection rejected by auth policy" component tcp_chain_source ...
+     remote_addr <addr> error "auth: refused: identity \"edge-99\" is not allowed"
 ```
 
 Policy rejections are counted in the plugin's `auth_rejected` statistic; the
@@ -627,8 +634,8 @@ username      = "edge-01"
 password_file = "/etc/logwisp/edge-01.pass"
 ```
 
-Startup reports `auth="scram users=2 certificate=independent node_binding=force"`
-on such a chain source, and `auth="scram user=edge-01"` on the dialer.
+Startup reports `auth "scram users=2 certificate=independent node_binding=force"`
+on such a chain source, and `auth "scram user=edge-01"` on the dialer.
 
 ### Mechanism
 
@@ -670,7 +677,8 @@ requires `tls.client_auth`) binds the two: the certificate's identity field must
 equal the username, so every peer needs its own certificate *and* its own
 password, like `clientcert=verify-full`. A mismatch fails the login; an HTTP
 request whose certificate does not match its token's user gets `403`. Startup
-shows the mode as `certificate=independent` or `certificate=cn=username`.
+shows the mode as `certificate=independent`, or as the bound field, such as
+`certificate=cn=username`.
 
 ### Tokens
 
@@ -717,12 +725,14 @@ stored_key = "<base64>"
 username = "edge-01"
 ```
 
-Write it with [`lw auth add-user`](cli.md#lw-auth) rather than by
-hand. The whole file is validated when the plugin is built: a `decoy_key` of at
-least 32 bytes, at least one user, unique names, no unknown keys, and one Argon2
-profile and salt length for every user, since mixed profiles would tell a prober
-which users exist. `decoy_key` keeps unknown-user challenges stable across
-restarts and edits; the CLI creates it once and preserves it. A single probe
+Write it with [`lw auth add-user`](cli.md#lw-auth) rather than by hand. A
+username, in the file or in `auth.username`, is 1 to 256 bytes of UTF-8
+without `,`, `=` or a control character. The whole file is validated when the
+plugin is built: a `decoy_key` of at least 32 bytes, at least one user, unique
+names, no unknown keys, and one Argon2 profile and salt length for every user,
+since mixed profiles would tell a prober which users exist. `decoy_key` keeps
+unknown-user challenges stable across restarts and edits; the CLI creates it
+once and preserves it. A single probe
 cannot tell a real user from an unknown one, but a prober polling a name sees
 its salt change when the user is added, rotated or removed. The file is read
 only when the plugin is built and `auto_reload` does not watch it: send
@@ -756,7 +766,7 @@ edge. `password_file` is read by what it names, on either side:
     on that terminal, never in the log or on stderr: logs persist, are
     forwarded, and stderr is often a file.
   - With no terminal (systemd, cron) the plugin fails. `lw ... &` stops on
-    `SIGTTIN` at the prompt.
+    `SIGTTOU` at the prompt.
   - Under `lw --tui`'s `t`, the screen asks in a masked dialog instead; a
     generated password stays shown until a key closes it.
 
@@ -865,9 +875,9 @@ location /logs/ {
   - The token lives in a page variable only, never in storage or the URL. It
     lasts as long as the page and at most the token lifetime: a reload, another
     tab, or a reconnect after expiry asks again; sign out revokes it.
-  - The login page says that cookies are unavailable and links to the viewer.
-    A page not served over HTTPS keeps no `Secure` cookie either: both pages
-    name HTTPS instead and offer no sign-in.
+  - The login page says that cookies are unavailable and, when the sink serves
+    the viewer, links to it. A page not served over HTTPS keeps no `Secure`
+    cookie either: both pages name HTTPS instead and offer no sign-in.
 - **Integrating `scram.js`.** A site with its own CSP can copy `scram.js` from
   `internal/sink/http/web/` into its bundle: one dependency-free ES module in
   plain JS (no WebAssembly, so no `'wasm-unsafe-eval'`), where `base` is the
@@ -1007,10 +1017,11 @@ layer.
 identities you list may connect, so one CA can serve several trust domains and
 a single peer can be withdrawn without touching the others.
 
-**`auth` with `type = "scram"`** — a password check, per listener: only the
-users in its credentials file may connect, the login cannot be relayed through
-another certificate (proxy mode, where TLS ends at the site's proxy, aside),
-and with `identity` each user is tied to its own certificate.
+**`auth` with `type = "scram"`** — a password check, per listener: only its
+users (a credentials file's, or its one `username`) may connect, the login
+cannot be relayed through another certificate (proxy mode, where TLS ends at
+the site's proxy, aside), and with `identity` each user is tied to its own
+certificate.
 
 Under either method the chain `node` label can be bound to the identity, so a
 compromised edge cannot attribute its entries to another host, and the `http`
@@ -1093,12 +1104,13 @@ where TLS ends at the trusted proxy; `mtls` also requires `client_auth`.
 **Log content**
 
 Logs routinely contain secrets that were never meant to leave the host. Filters
-are the available tool:
+are the available tool; patterns are case-sensitive, so `(?i)` makes one catch
+`Authorization: Bearer` too:
 
 ```toml
 [[pipelines.flow.filters]]
 type = "exclude"
-patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret"]
+patterns = ["(?i)password|api[_-]?key|authorization|bearer |secret"]
 ```
 
 Choose a sanitizer policy that matches the sink — `json` for JSON output,

doc/sinks.md

@@ -18,7 +18,8 @@ whose input queue is full drops the event for itself only
 ## console
 
 Writes formatted payloads to stdout or stderr. It is the sink of the built-in
-`pipe` pipeline, and of every spec pipeline given no `--sink`.
+`pipe` pipeline, and of every pipeline from flags or variables that gets no
+sink from `--sink`, `LOGWISP_SINK` or its preset.
 
 ```toml
 [[pipelines.plugin_sinks]]
@@ -40,8 +41,8 @@ Options, each as type and default:
   bytes, `ESC` as `<1b>`: `auto` when the output is a terminal, `always`, or
   `never`.
 - `color` (string, the top-level `color`, itself `auto`): paint level names:
-  `auto` on a terminal with `NO_COLOR` unset and `TERM` not `dumb`, `always`,
-  or `never`.
+  `auto` on a terminal with `NO_COLOR` empty or unset and `TERM` not `dumb`,
+  `always`, or `never`.
 
 Payloads are written as the formatter made them, one record per line.
 
@@ -60,10 +61,12 @@ Payloads are written as the formatter made them, one record per line.
   sink's own, written after escaping. A line whose source found no level
   stays plain. The codes go into whatever the format makes, JSON included:
   for output another program parses keep `auto` or `never`, not `always`.
-- Backpressure: the sink never drops. When its output is slow (a slow reader,
-  a paused terminal, a stalled log collector) the pipeline waits for it, with
-  its other sinks. A service whose stdout may stall writes to a `file` sink
-  instead. A closed stdout pipe ends lw with `SIGPIPE`, as it ends `cat`.
+- Backpressure: the sink never drops while its pipeline runs; a stop or
+  reload can drop entries still in flight (`total_dropped_by_sink`). When its
+  output is slow (a slow reader, a paused terminal, a stalled log collector)
+  the pipeline waits for it, with its other sinks. A service whose stdout may
+  stall writes to a `file` sink instead. A closed stdout pipe ends lw with
+  `SIGPIPE`, as it ends `cat`.
 - On stop it writes what is queued.
 
 ---
@@ -90,9 +93,9 @@ flush_interval_ms = 100
 Options, each as type and default:
 
 - `directory` (string, **required**): output directory. The sink writes
-  `NAME.log` and rotates it to `NAME_YYMMDD_HHMMSS.log`. The limits below
-  delete *any* `.log` file in the directory, so give each file sink a
-  directory of its own.
+  `NAME.log` and rotates it to `NAME_YYMMDD_HHMMSS.log`, adding
+  `_1`, `_2`, ... when that name is taken. The limits below delete *any*
+  `.log` file in the directory, so give each file sink a directory of its own.
 - `name` (string, **required**): file name base.
 - `max_size_mb` (int, `100`): rotate when the active file reaches this size.
 - `max_total_size_mb` (int, `1000`): above this total, the oldest `.log` files
@@ -103,7 +106,8 @@ Options, each as type and default:
 - `retention_hours` (float, `168.0`): `.log` files there older than this are
   deleted; `0` keeps them.
 - `buffer_size` (int, `1000`): sink input queue depth, and the depth of the
-  internal writer's queue, which drops and counts when full.
+  internal writer's queue, which drops when full. That drop is reported
+  nowhere, and `total_processed` still counts the entry. At most `1048576`.
 - `flush_interval_ms` (int, `100`): forced flush interval.
 
 The sink drives an internal writer configured for raw output with timestamps and
@@ -115,8 +119,8 @@ it hands the writer what is queued before closing the file.
 ## null
 
 Discards; holds a sink's place until one is chosen, so a pipeline can be set
-up before its sink is. It counts entries and bytes, so it also isolates a
-source or flow when measuring it.
+up before its sink is. It counts entries and bytes (`total_processed`,
+`total_bytes`), so it also isolates a source or flow when measuring it.
 
 ```toml
 [[pipelines.plugin_sinks]]
@@ -172,7 +176,8 @@ Options, each as type and default:
 - `buffer_size` (int, `1000`): sink input queue depth.
 - `client_buffer_size` (int, `256`): per-client send queue depth.
 - `write_timeout_ms` (int, `0`): per-event write deadline; `0` = none.
-- `max_connections` (int, `0`): concurrent stream cap; `0` = unlimited.
+- `max_connections` (int, `0`): concurrent stream cap; `0` = unlimited. A
+  stream past it gets `503` and counts in `rejected_clients`.
 - `replay_lines` (int, `0`; `1000` from the `serve` preset): the last entries
   a new stream gets after its `connected` event, before the live ones, so a
   browser opened late sees a backlog; `0` = none.
@@ -231,7 +236,8 @@ Options, each as type and default:
 - Stream responses carry `Content-Type: text/event-stream`,
   `Cache-Control: no-cache` and `X-Accel-Buffering: no`. On connect the client
   receives an `event: connected` frame carrying its client id, session id,
-  sink instance id, endpoint paths, and buffer size.
+  sink instance id, endpoint paths, and `client_buffer_size` (as
+  `buffer_size`).
 - Payloads are framed per the SSE spec, one `data:` line per line break in the
   payload (CRLF, LF or a lone CR), so multi-line entries stream correctly and
   no entry can inject an `event:`, `id:` or `retry:` field.
@@ -312,6 +318,7 @@ Options, each as type and default:
   `keep_alive_period_ms` idle; `false` sends none.
 - `keep_alive_period_ms` (int, `30000`): keep-alive idle period.
 - `max_connections` (int, `0`): concurrent connection cap; `0` = unlimited.
+  A connection past it is closed at once and counts in `rejected_conns`.
 - `tls`, `auth`: as on `http`.
 - `acl` (table): as on `http`, without `requests_per_second_per_client`.
 

doc/sources.md

@@ -6,7 +6,8 @@ type-specific `config` table. Registered types: `file`, `console`, `random`,
 `null`, `tcp_chain`, `http_chain`.
 
 Publication is non-blocking for every source but `console`: a full subscriber
-channel drops the entry and counts it in `dropped_entries`. The `console`
+channel drops the entry and counts it in `dropped_entries`. That channel holds
+`buffer_size` entries, or 1000 on a source without that option. The `console`
 source waits instead (see [console](#console)).
 
 ---
@@ -31,7 +32,7 @@ Options, each as type and default:
 
 - `directory` (string, required): directory to scan; not recursive. A leading
   `~/`, here as in every file or directory option, is the home directory; a
-  relative path starts at lw's working directory; a missing one is waited for.
+  relative path starts at lw's working directory.
 - `pattern` (string, `*`): glob over filenames; `*` and `?` only.
 - `check_interval_ms` (int, `100`): directory rescan interval; minimum `10`.
   An open file is polled every 100 ms whatever this is.
@@ -44,30 +45,37 @@ Options, each as type and default:
 - Each matched file gets its own watcher, and its entries the file's base
   name as `Source`. Watchers for files that disappear are stopped and removed
   on the next scan.
+- A missing directory is waited for: INFO `Waiting for directory`, then
+  `Directory readable` once a scan reads it. A directory or file that cannot be
+  read is retried at each scan or poll, with WARN `Cannot read directory` or
+  `Cannot read file` when the failure starts or changes, not at each retry.
 - The file lw's stdout or stderr is redirected to is never followed, with a
   warning: `lw > out.log` in the directory would read its own output back.
 - A line longer than 1 MiB continues in the next entry, as on the console
-  source.
+  source; so does the rest of a line a poll reads while it is still being
+  written.
 - A new watcher seeks to end-of-file. Positions live in memory only, so a
   restart or a reload resumes from the current end of each file, and content
   written meanwhile is not read. `from = "start"` instead reads each file whole
   when its watcher is created, which a process writing beside LogWisp needs, at
   the cost of replaying files already on disk at every restart and reload.
-- Rotation is detected from size decrease, modification-time reset, a position
-  beyond end-of-file, or an inode change. An inode change where the new file is
-  at least as large as the recorded position is treated as an atomic save, not
-  a rotation, and the position is preserved. Each rotation also publishes an
-  INFO entry `Log rotation detected (#N): REASON` with the file's name as
-  `Source`; it reaches every sink, so exclude it with a filter
-  (`Log rotation detected`) where unwanted.
+- Rotation is detected from size decrease, a modification time that went back
+  without growth, a position more than 1 KiB beyond end-of-file, or an inode
+  change. An inode change where the new file is non-empty and at least as large
+  as the recorded position is treated as an atomic save, not a rotation, and
+  the position is preserved. Each rotation also publishes an INFO entry
+  `Log rotation detected (#N): REASON` with the file's name as `Source`; it
+  reaches every sink, so exclude it with a filter (`Log rotation detected`)
+  where unwanted.
 - A rotation that renames in place — what a size-capped writer does — puts the
   same inode back under a name `pattern` also matches. Its watcher resumes at
   the position the original reached, so `from = "start"` reads the tail an
   unfinished read left behind rather than the whole archive a second time.
-- A line is parsed as JSON only when it is an object whose top-level keys are
-  all drawn from `time`, `level`, `msg` and `fields` — the four an entry can
-  carry. `time` is read as RFC3339Nano. Any other key, and any non-object line,
-  is kept whole as text, because parsing it would drop the rest. `level` is
+- A line is parsed as JSON only when it is a non-empty object whose top-level
+  keys are all drawn from `time`, `level`, `msg` and `fields` (the four an
+  entry can carry), with any `level` or `msg` a string. `time` is read as
+  RFC3339Nano; the read time stands in when it is missing or does not parse.
+  Any other line is kept whole as text, so nothing in it is dropped. `level` is
   kept as written (a filter sees `Error` or `fatal` as is), and a line without
   `level` has none; only text lines get the inference below.
 - A text line's level is its first whole word, in any case, naming one:
@@ -76,20 +84,23 @@ Options, each as type and default:
   console sink paints that word and the http sink's viewer filters on it.
 - `raw = true` skips the JSON branch. The line, with its terminator (`\n` or
   `\r\n`) removed, becomes the message. `fields` stays empty, the time is the
-  read time, and the level is inferred from the text. Blank lines are skipped.
+  read time, and the level is inferred from the text. Empty lines are skipped.
   With `format.type = "raw"` each line is written back with one `\n`, which
   carries records the envelope cannot hold but is not byte-exact; see
   [Formatters](formatters.md#raw).
 
-**Statistics**: per-watcher size, position, entries read, rotation count, and
-last read time, plus `active_watchers`.
+**Statistics**: `active_watchers`, and `watchers`, one map per followed file:
+`directory` (the file's path), `size`, `position`, `entries_read`,
+`rotations`, `last_read`. The status report shows that list only as
+`watchers_count`.
 
 ---
 
 ## console
 
 Reads newline-delimited entries from standard input. It is the source of the
-built-in `pipe` pipeline, and of every spec pipeline given no `--source`.
+built-in `pipe` pipeline, and of every pipeline from flags or variables left
+without one: no `--source`, and no preset that brings one.
 
 ```toml
 [[pipelines.plugin_sources]]
@@ -106,21 +117,24 @@ Option, as type and default:
 Lines:
 - Each line is one entry, its terminator (`\n` or `\r\n`) removed; the
   formatter writes one back. An unterminated last line is kept.
-- Blank lines are skipped.
+- Empty lines are skipped; a line of spaces is an entry.
 - A line over 1 MiB continues in the next entry; no byte is lost.
 - The level is inferred from the line text, and `Source` is set to `console`.
 
 Delivery and lifetime:
-- Nothing is dropped: the source waits for its pipeline, so a slow pipeline
-  slows the reading of stdin.
+- The source drops nothing while its pipeline runs: it waits, so a slow
+  pipeline slows the reading of stdin.
 - One reader serves the process. A reload's new source continues where the old
-  one stopped; no line is read twice. Hence at most **one** console source in
-  the whole configuration, checked at validation.
+  one stopped, and no line is read twice, but lines the old pipeline still
+  holds at the reload can be lost. Hence at most **one** console source in the
+  whole configuration, checked at validation.
 - At the end of input the source ends, and with it a pipeline whose sources
   have all ended; lw exits once every pipeline has
   ([CLI](cli.md#built-in-defaults)). Under a supervisor whose stdin is
   `/dev/null` that is at once.
 
+**Statistics**: `ended`, true once standard input has ended.
+
 ---
 
 ## random
@@ -151,10 +165,13 @@ Options, each as type and default:
   longer message.
 - `special` (bool, `false`): with `length` ≥ 3, mix in up to one control byte
   (`\n` and `\r` among them) and a few non-ASCII bytes, often invalid UTF-8.
-  `raw` and `txt` carry them as they are; `json` escapes them inside its
-  message.
+  `raw` and `txt` carry them as they are; `json` escapes the control byte and
+  replaces each invalid byte with U+FFFD.
 
 Levels (`txt` and `json` only) are random among DEBUG, INFO, WARN, ERROR.
+`Source` is `random_ID`, ID being the source's `id`.
+
+**Statistics**: `format`, `interval_ms`, `jitter_ms`, `length`, `special`.
 
 ---
 
@@ -295,7 +312,9 @@ node_binding = "force"
 
 Options, each as type and default:
 
-- `host`, `port`, `trust_node`, `tls`, `auth`: as on [tcp_chain](#tcp_chain).
+- `host`, `port`, `trust_node`, `tls`: as on [tcp_chain](#tcp_chain).
+- `auth` (table): as on `tcp_chain`, plus `token_lifetime_ms`, the `scram`
+  bearer token lifetime: 10 s to 24 h, `0` = 15 minutes.
 - `ingest_path` (string, `/ingest`): endpoint path; must start with `/`, and
   `/auth` is reserved for the login.
 - `buffer_size` (int, `1000`): subscriber channel depth.

doc/todo.md

@@ -20,7 +20,7 @@ release tag, and both skeletons download the tagged source.
    git tag -a vX.Y.Z -m vX.Y.Z      # git tag -s signs it with your key
    git push origin vX.Y.Z
    ```
-   - `make` then stamps `vX.Y.Z` (from `git describe`), and
+   - The Makefile's builds then stamp `vX.Y.Z` (from `git describe`), and
      `go install github.com/lixenwraith/logwisp/cmd/lw@vX.Y.Z` works.
    - A GitHub release for the tag is optional; the tag alone serves
      `archive/vX.Y.Z.tar.gz`, which the PKGBUILD downloads.
@@ -33,10 +33,12 @@ release tag, and both skeletons download the tagged source.
    - Push `PKGBUILD` and `.SRCINFO` to
      `ssh://aur@aur.archlinux.org/logwisp.git`; optionally a `logwisp-git`
      package built from the main branch.
-3. FreeBSD, `deploy/package/freebsd/` (`sysutils/logwisp`):
+3. FreeBSD, `deploy/package/freebsd/` (`sysutils/logwisp`), copied into a
+   ports tree with `cp -RL` (`files/logwisp.in` links to
+   `deploy/package/logwisp.rc`):
    - `DISTVERSION=X.Y.Z` and `MAINTAINER=` your address.
-   - `make makesum` writes `distinfo`: the module zip that the Go proxy
-     serves for `GO_MODULE` at the tag.
+   - `make makesum` writes `distinfo`, the checksums of the `.mod` and `.zip`
+     that the Go proxy serves for `GO_MODULE` at the tag.
    - The `logwisp` user needs an ID registered in the ports tree. Take a
      number below 1000 that is free in both `/usr/ports/UIDs` and
      `/usr/ports/GIDs`, and add to the same patch:
@@ -57,7 +59,7 @@ release tag, and both skeletons download the tagged source.
      (lw is static by design) and `maintainer-script-calls-systemctl` (dh's
      scripts use `deb-systemd-helper` instead).
 5. With a tag, the packaging workflow's FreeBSD job can build the port itself
-   from a ports tree (`make stage check-plist` in a copy of
+   from a ports tree (`make stage check-plist` in a `cp -RL` copy of
    `deploy/package/freebsd/`), rather than BSD make on the checkout.
 
 ## 2. What `lw --tui` and its engine leave for later
@@ -71,16 +73,16 @@ in place. Not yet:
 - Per-sink taps: what each sink took, where the tap at the flow's exit shows
   what dispatch offered (a full sink input drops for that sink only).
 - Live filter patching: a filter edited while `t` runs takes effect through
-  `Filter.UpdatePatterns` (unused), not a rerun that rebinds every listener.
+  `Filter.UpdatePatterns` (unused, and racing `Apply`, which reads the
+  patterns without `f.mu`), not a rerun that rebinds every listener.
 - The wide-rune table `internal/tui/view.go` copies from
   `lixenwraith/terminal`'s one-cell rule goes once terminal exports that rule.
 - A scripted session on a pseudo-terminal for Run, each output and `t` (keys,
   the password dialog, `SIGHUP`), on a text console and over ssh.
-- The schema lists `auth.node_binding` on every listener and
-  `auth.trusted_proxies` on both HTTP listeners, though `AuthOptions.Check`
-  refuses a binding off the chain sources (under `mtls` or `scram`) and
-  `trusted_proxies` off the `http` sink; side tags for chain sources and the
-  `http` sink would filter them as `http` filters HTTP's keys.
+- The schema lists `auth.trusted_proxies` on both HTTP listeners, though
+  `AuthOptions.Check` refuses it off the `http` sink; a side tag for the
+  `http` sink would filter it as `chain_listener` keeps `node_binding` to the
+  chain sources.
 - Optional, in `lixenwraith/color`: export the 16 xterm reference shades
   (`RGBTo16` maps the site's hues badly, so the 16-color tier is chosen by
   hand).

internal/authz/authz.go

@@ -573,7 +573,8 @@ func (p *Policy) Rejected() uint64 {
 // Stats reports policy state for a plugin's stats details map. Merge it in
 // with maps.Copy so rejections surface in the status reporter and in the
 // http sink's status endpoint. allowed counts admissions (certificate checks,
-// SCRAM logins); rejected counts every refusal.
+// SCRAM logins), rejected refusals; a throttled or busy login counts in
+// auth_throttled or auth_busy, a forwarded client the acl refuses in acl_denied.
 func (p *Policy) Stats() map[string]any {
 	if p == nil {
 		return map[string]any{"auth": MethodNone}

internal/authz/scram.go

@@ -196,7 +196,8 @@ type asked struct {
 	tty bool
 }
 
-// startupOver is set once lw's first service is built
+// startupOver is set once lw builds the service it runs, after any run on the
+// --tui screen
 var startupOver atomic.Bool
 
 // EndStartup stops reading descriptors, pipes, FIFOs and terminals: at a

internal/chain/proto.go

@@ -1,3 +1,7 @@
+// Package chain is the protocol between logwisp nodes: the hello a dialer
+// opens with (ProtocolVersion, node, SCRAM), the NDJSON entries after it, the
+// headers and /auth path of its HTTP mapping, and the reconnect backoff the
+// chain sinks share.
 package chain
 
 import (

internal/compose/compose.go

@@ -1,7 +1,8 @@
 // Package compose is the configuration engine lw --tui and the website share:
-// pipelines started empty, from a preset, a configuration or a command line,
-// edited, validated, and written as a command line, environment or file. It
-// reads no file and links no network or terminal code, so it builds as WebAssembly.
+// pipelines started empty, from a preset, a configuration or a pasted form,
+// edited, validated, and written as a command line, environment or file that
+// reads back to the same pipelines; Schema is lw --schema. It reads no file and
+// links no TLS, HTTP or terminal code, so it builds as WebAssembly.
 package compose
 
 import (

internal/config/config.go

@@ -1,3 +1,8 @@
+// Package config is lw's configuration: the settings and pipeline tables, the
+// plugin catalogue whose option structs' tags are the schema, the pipeline
+// flags, LOGWISP_ variables and presets that build pipelines, and the rules
+// Decode and ValidateConfig apply. Load's Manager merges the file, environment
+// and command line, and rereads the file on reload.
 package config
 
 // --- LogWisp Configuration Options ---
@@ -31,7 +36,7 @@ type LogConfig struct {
 	// File output settings (when Output includes "file" or "all")
 	File *LogFileConfig `toml:"file"`
 
-	// Console output settings
+	// Accepted for older files: logging.output chooses the console
 	Console *LogConsoleConfig `toml:"console"`
 }
 
@@ -44,7 +49,8 @@ type LogFileConfig struct {
 	RetentionHours float64 `toml:"retention_hours" help:"hours a .log file is kept; 0 keeps them"`
 }
 
-// LogConsoleConfig defines settings for console-based application logging
+// LogConsoleConfig is logging.console, whose target has no effect:
+// logging.output chooses the console stream
 type LogConsoleConfig struct {
 	Target string `toml:"target" lw:"enum=stdout|stderr|split" help:"accepted without effect: logging.output chooses the console"`
 }

internal/config/schema.go

@@ -231,9 +231,9 @@ func settingKeys(v reflect.Value, prefix string) []Key {
 }
 
 // Decode reads a plugin's config map into its options: unknown keys, kinds,
-// the defaults, the tag rules, the tls, auth and acl rules of its side, then the
-// options' own Check; a leading ~ in a path is the home directory. It reads no
-// file.
+// the defaults, the tag rules and the options' own Check, then the host, tls,
+// acl and auth rules of its side; a leading ~ in a path is the home directory.
+// It reads no file.
 func Decode[T any](role, typ string, m map[string]any) (*T, error) {
 	v, err := decode(role, typ, m)
 	if err != nil {
@@ -272,7 +272,8 @@ func decode(role, typ string, m map[string]any) (any, error) {
 }
 
 // Settle gives a decoded table its defaults, applies the tag rules, then its
-// own Check: Decode's last steps, for the flow stages a file decodes typed
+// own Check, as Decode does before its side's rules, for the flow stages a
+// file decodes typed
 func Settle(v any) error {
 	Fill(v)
 	if err := checkTags(reflect.ValueOf(v).Elem(), ""); err != nil {

internal/config/spec.go

@@ -238,7 +238,8 @@ func SpecPipelines(specs []Spec, isDir IsDir) ([]PipelineConfig, error) {
 }
 
 // buildPipelines turns specs, in order, into pipelines; specs before the first
-// --pipeline go to one named "cli". Every call returns fresh maps.
+// --pipeline go to one named after the preset that starts it, or else "cli".
+// Every call returns fresh maps.
 func buildPipelines(specs []pipelineSpec, isDir IsDir) ([]PipelineConfig, error) {
 	var pipelines []PipelineConfig
 	var seen map[*specKind]bool

internal/config/validate.go

@@ -25,7 +25,8 @@ import (
 // ValidateConfig checks a configuration without reading a file it names:
 // the settings, then each pipeline's flow stages, and its sources and sinks
 // decoded in full. Its error joins every problem, one a part, each a
-// *KeyError naming the path of the refused key. lw --check opens the files.
+// *KeyError naming the refused key's path, save "no pipelines configured".
+// lw --check opens the files.
 func ValidateConfig(cfg *Config) error {
 	if cfg == nil {
 		return fmt.Errorf("config is nil")
@@ -521,9 +522,11 @@ func (c *FilterConfig) Check() error {
 	return nil
 }
 
-// Scan decodes a flow stage's command-line map into target, rejecting keys
-// target does not declare: a misspelled key must fail, not be dropped. Its
-// values take their kinds as a plugin's do, a wrong one refused at its key.
+// Scan decodes a table into target: a flow stage from a spec or an edit, a
+// file's root or the website's pipelines. It refuses a key target does not
+// declare, nested tables' too: a misspelled key must fail, not be dropped.
+// Values outside a list of tables take their kinds as a plugin's do, a wrong
+// one refused at its key.
 func Scan(configMap map[string]any, target any) error {
 	if err := checkKeys(configMap, reflect.TypeOf(target), ""); err != nil {
 		return err
@@ -580,7 +583,7 @@ func checkKeys(m map[string]any, t reflect.Type, prefix string) error {
 
 // checkFileKeys rejects keys the configuration file declares that Config does
 // not: a misspelled table path such as plugin_sinks.confg.tls would otherwise
-// drop the whole table. Plugin config maps are checked by Scan instead.
+// drop the whole table. Plugin config maps are checked by Decode instead.
 func checkFileKeys(path string) error {
 	data, err := os.ReadFile(path)
 	if errors.Is(err, fs.ErrNotExist) {

internal/core/flow.go

@@ -1,3 +1,7 @@
+// Package core holds what the parts of a pipeline share: the LogEntry and the
+// TransportEvent sinks get, the Tap a viewer reads a flow's decisions from,
+// the level names, plugin capabilities, size limits and timings, and Network
+// and Listen, which keep a listener or dialer to its host's address family.
 package core
 
 import (
@@ -29,7 +33,8 @@ type TransportEvent struct {
 
 // Tapped is what a pipeline's flow decided about one entry: Stage names the
 // stage that dropped Event.Entry (rate_limit, filters at Index, format), or is
-// "" when it passed and Event.Payload is what the sinks got.
+// "" when it passed and Event.Payload is what dispatch offered the sinks; a
+// full sink input drops it for that sink only.
 type Tapped struct {
 	Pipeline string
 	Source   string // the source's id; "" for a heartbeat

internal/filter/filter.go

@@ -37,7 +37,6 @@ func NewFilter(cfg config.FilterConfig, logger *log.Logger) (*Filter, error) {
 		logger:   logger,
 	}
 
-	// Compile patterns
 	for i, pattern := range cfg.Patterns {
 		re, err := regexp.Compile(pattern)
 		if err != nil {
@@ -82,8 +81,8 @@ func (f *Filter) Apply(entry core.LogEntry) bool {
 		"logic", f.config.Logic,
 		"entry_level", entry.Level,
 		"entry_source", entry.Source,
-		"entry_message", entry.Message[:min(100, len(entry.Message))], // First 100 chars
-		"text_to_match", text[:min(150, len(text))], // First 150 chars
+		"entry_message", entry.Message[:min(100, len(entry.Message))],
+		"text_to_match", text[:min(150, len(text))],
 		"patterns", f.config.Patterns)
 
 	for i, pattern := range f.config.Patterns {
@@ -103,7 +102,6 @@ func (f *Filter) Apply(entry core.LogEntry) bool {
 		"component", "filter",
 		"matched", matched)
 
-	// Determine if we should pass or drop
 	shouldPass := false
 	switch f.config.Type {
 	case config.FilterTypeInclude:
@@ -137,11 +135,12 @@ func (f *Filter) GetStats() map[string]any {
 	}
 }
 
-// UpdatePatterns allows for dynamic, thread-safe updates to the filter's regex patterns
+// UpdatePatterns replaces the patterns under f.mu, which Apply does not take:
+// it is safe only before entries flow
 func (f *Filter) UpdatePatterns(patterns []string) error {
 	compiled := make([]*regexp.Regexp, 0, len(patterns))
 
-	// Compile all patterns first
+	// Compiled before the swap, so a bad pattern leaves the filter as it was
 	for i, pattern := range patterns {
 		re, err := regexp.Compile(pattern)
 		if err != nil {
@@ -150,7 +149,6 @@ func (f *Filter) UpdatePatterns(patterns []string) error {
 		compiled = append(compiled, re)
 	}
 
-	// Update atomically
 	f.mu.Lock()
 	f.patterns = compiled
 	f.config.Patterns = patterns
@@ -166,7 +164,6 @@ func (f *Filter) UpdatePatterns(patterns []string) error {
 func (f *Filter) matches(text string) bool {
 	switch f.config.Logic {
 	case config.FilterLogicOr:
-		// Match any pattern
 		for _, re := range f.patterns {
 			if re.MatchString(text) {
 				return true
@@ -175,7 +172,6 @@ func (f *Filter) matches(text string) bool {
 		return false
 
 	case config.FilterLogicAnd:
-		// Must match all patterns
 		for _, re := range f.patterns {
 			if !re.MatchString(text) {
 				return false

internal/flow/heartbeat.go

@@ -83,7 +83,6 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
 		Message: "heartbeat",
 	}
 
-	// Add stats if configured
 	if hg.config.IncludeStats {
 		fields := map[string]any{
 			"type":       "heartbeat",
@@ -98,13 +97,11 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
 		entry.Fields = fieldsJSON
 	}
 
-	// Use formatter to generate payload
 	var payload []byte
 	var err error
 
-	// Check if we need special formatting for heartbeat
+	// Unreachable: heartbeat.format's enum (txt, json, raw) refuses "comment"
 	if hg.config.Format == "comment" {
-		// SSE comment format - bypass formatter for this special case
 		if hg.config.IncludeStats {
 			beatNum := hg.beatCount.Load()
 			payload = []byte(": heartbeat " + t.Format(time.RFC3339) +
@@ -115,7 +112,6 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
 	} else {
 		// Use flow's formatter for consistent formatting
 		if adapter, ok := hg.formatter.(*format.FormatterAdapter); ok {
-			// Customize flags for heartbeat if needed
 			// The formatter shows a timestamp unless told otherwise
 			customFlags := formatter.FlagDefault
 			if !hg.config.IncludeTimestamp {
@@ -123,14 +119,12 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
 			}
 			payload, err = adapter.FormatWithFlags(entry, customFlags)
 		} else {
-			// Fallback to standard format
 			payload, err = hg.formatter.Format(entry)
 		}
 
 		if err != nil {
 			hg.logger.Error("msg", "Failed to format heartbeat",
 				"error", err)
-			// Fallback to simple text
 			payload = []byte("heartbeat: " + t.Format(time.RFC3339) + "\n")
 		}
 	}

internal/netacl/netacl.go

@@ -1,8 +1,8 @@
 // Package netacl is the address seam of the network listeners, beside tlsx
-// and authz: a Policy reads the PROXY header of the proxies it trusts, then
-// refuses a peer by its client's address, or past its client's limits, before
-// TLS or a byte of the protocol is read. New returns nil when nothing is set,
-// and every method tolerates a nil receiver.
+// and authz: a Policy reads its trusted proxies' PROXY headers, refuses a peer
+// by address or connection count before TLS, and an HTTP request past its rate
+// or from a forwarded client the rules refuse. Its Table also throttles SCRAM.
+// New returns nil when nothing is set; every method tolerates a nil receiver.
 package netacl
 
 import (

internal/pipeline/pipeline.go

@@ -1,3 +1,8 @@
+// Package pipeline runs one pipeline: its sources through the flow to its
+// sinks, each plugin made by the factory its type registered. Sinks start
+// before the sources and stop after them, so entries in flight arrive; a full
+// sink drops its event unless it declares core.CapBackpressure. A tap gives a
+// viewer a copy of what the flow decides.
 package pipeline
 
 import (
@@ -39,7 +44,7 @@ type Pipeline struct {
 	cancel       context.CancelFunc
 	wg           sync.WaitGroup
 	running      atomic.Bool
-	halt         chan struct{} // closed by Stop: backpressure sinks drop again
+	halt         chan struct{} // closed by Stop: backpressure events may then drop, room or not
 	finished     chan struct{} // closed when every source has ended
 	finishedOnce sync.Once
 }

internal/pipeline/pipeline_test.go

@@ -143,8 +143,8 @@ func TestStopReleasesStalledBackpressureSink(t *testing.T) {
 }
 
 // The tap gets each entry's verdict under its source's id: the stage that
-// dropped it, or what the sinks got. Drops come only when All is set, and
-// heartbeats come as passed, from no source.
+// dropped it, or what dispatch offered the sinks. Drops come only when All is
+// set, and heartbeats come as passed, from no source.
 func TestTapSeesEachVerdict(t *testing.T) {
 	const n = 1000
 	verdict := func(msg string) (string, int) {

internal/plugin/factory.go

@@ -1,3 +1,6 @@
+// Package plugin maps each source and sink type to the constructor its
+// package registers in init; a type without a row in config's catalogue is
+// refused.
 package plugin
 
 import (

internal/service/service.go

@@ -1,3 +1,6 @@
+// Package service holds the pipelines of one configuration: it builds all of
+// them or none, starts and stops them together, gives a viewer's tap to each,
+// and reports when every one has finished.
 package service
 
 import (

internal/session/session.go

@@ -1,3 +1,7 @@
+// Package session tracks the sessions of a pipeline's plugins: a client
+// connection, or a plugin's own input or output. A Manager per pipeline sweeps
+// the idle ones; each plugin reaches it through a Proxy that sees only its
+// instance's sessions.
 package session
 
 import (
@@ -11,7 +15,7 @@ import (
 	"github.com/lixenwraith/logwisp/internal/core"
 )
 
-// Session represents a connection session
+// Session is a client connection, or a plugin's own input or output
 type Session struct {
 	InstanceID   string         // Plugin instance identifier
 	ID           string         // Unique session identifier
@@ -20,8 +24,7 @@ type Session struct {
 	LastActivity time.Time      // Last activity timestamp
 	Metadata     map[string]any // Optional metadata (e.g., TLS info)
 
-	// Connection context
-	Source string // Source type: "tcp_source", "http_source", "tcp_sink", etc.
+	Source string // the owning plugin's instance ID, which Proxy sets
 }
 
 // Manager handles the lifecycle of sessions
@@ -34,7 +37,7 @@ type Manager struct {
 	cleanupTicker *time.Ticker
 	done          chan struct{}
 
-	// Expiry callbacks by source type
+	// Expiry callbacks by plugin instance
 	expiryCallbacks map[string]func(sessionID, remoteAddr string)
 	callbacksMu     sync.RWMutex
 }
@@ -52,7 +55,6 @@ func NewManager(maxIdleTime time.Duration) *Manager {
 		expiryCallbacks: make(map[string]func(sessionID, remoteAddr string)),
 	}
 
-	// Start cleanup routine
 	m.startCleanup()
 
 	return m
@@ -145,7 +147,7 @@ func (m *Manager) GetSessionCount() int {
 	return len(m.sessions)
 }
 
-// GetSessionsBySource returns snapshots of sessions matching a specific source type.
+// GetSessionsBySource returns snapshots of sessions matching a specific plugin instance.
 func (m *Manager) GetSessionsBySource(source string) []*Session {
 	m.mu.RLock()
 	defer m.mu.RUnlock()
@@ -159,7 +161,7 @@ func (m *Manager) GetSessionsBySource(source string) []*Session {
 	return sessions
 }
 
-// GetActiveSessionsBySource returns snapshots of active sessions for a given source.
+// GetActiveSessionsBySource returns snapshots of active sessions for a given plugin instance.
 func (m *Manager) GetActiveSessionsBySource(source string) []*Session {
 	m.mu.RLock()
 	defer m.mu.RUnlock()
@@ -240,7 +242,7 @@ func (m *Manager) RegisterExpiryCallback(source string, callback func(sessionID,
 	m.expiryCallbacks[source] = callback
 }
 
-// UnregisterExpiryCallback removes an expiry callback for a given source type
+// UnregisterExpiryCallback removes an expiry callback for a given plugin instance
 func (m *Manager) UnregisterExpiryCallback(source string) {
 	m.callbacksMu.Lock()
 	defer m.callbacksMu.Unlock()
@@ -290,7 +292,6 @@ func (m *Manager) cleanupIdleSessions() {
 
 		for _, session := range expiredSessions {
 			if callback, exists := callbacks[session.Source]; exists {
-				// Call callback to notify owner
 				go callback(session.ID, session.RemoteAddr)
 			}
 		}

internal/sink/console/console.go

@@ -82,8 +82,8 @@ func NewConsoleSinkPlugin(
 
 	output := Stream(opts.Target)
 
-	// inline decides auto: a terminal, NO_COLOR unset, TERM not dumb. Paint
-	// returns its text unchanged exactly when color is off.
+	// inline decides auto: a terminal, NO_COLOR empty or unset, TERM not
+	// dumb. Paint returns its text unchanged exactly when color is off.
 	printer := inline.New(output)
 	if opts.Color != "auto" {
 		printer.SetColor(opts.Color == "always")

internal/sink/file/file.go

@@ -56,7 +56,6 @@ func NewFileSinkPlugin(
 	logger *log.Logger,
 	proxy *session.Proxy,
 ) (sink.Sink, error) {
-	// Create empty config struct
 	opts, err := config.Decode[config.FileSinkOptions]("sink", "file", configMap)
 	if err != nil {
 		return nil, err
@@ -71,7 +70,7 @@ func NewFileSinkPlugin(
 	writerConfig.RetentionPeriodHrs = opts.RetentionHours
 	writerConfig.BufferSize = opts.BufferSize
 	writerConfig.FlushIntervalMs = opts.FlushIntervalMs
-	// Sink logic
+	// The payload is already formatted: the writer adds nothing and writes the file only
 	writerConfig.EnableConsole = false
 	writerConfig.EnableFile = true
 	writerConfig.ShowTimestamp = false
@@ -97,7 +96,6 @@ func NewFileSinkPlugin(
 	}
 	fs.lastProcessed.Store(time.Time{})
 
-	// Create session for file output
 	fs.session = proxy.CreateSession(
 		fmt.Sprintf("file:///%s/%s", opts.Directory, opts.Name),
 		map[string]any{
@@ -173,12 +171,10 @@ func (fs *FileSink) Stop() {
 		<-fs.exited
 	}
 
-	// Remove session
 	if fs.session != nil {
 		fs.proxy.RemoveSession(fs.session.ID)
 	}
 
-	// Shutdown the writer with timeout
 	if err := fs.writer.Shutdown(core.LoggerShutdownTimeout); err != nil {
 		fs.logger.Error("msg", "Error shutting down file writer",
 			"component", "file_sink",

internal/sink/http/http.go

@@ -452,7 +452,6 @@ func (h *HTTPSink) handleStream(w http.ResponseWriter, r *http.Request) {
 	w.Header().Set("X-Accel-Buffering", "no")
 	w.WriteHeader(http.StatusOK)
 
-	// Connected event with metadata, parity with fasthttp sink
 	info, _ := json.Marshal(map[string]any{
 		"client_id":   strconv.FormatUint(id, 10),
 		"session_id":  sess.ID,
@@ -621,10 +620,11 @@ func (h *HTTPSink) GetStats() sink.SinkStats {
 // handlers; absent when auth is disabled
 type identityKey struct{}
 
-// authMiddleware gates every endpoint on the policy: a client certificate
-// or a bearer token. The rejection carries no detail: the status endpoint
-// already exposes host, port, and throughput counters, so a refusal should
-// not add the shape of the policy on top of that.
+// authMiddleware gates stream and status on the policy: a client
+// certificate, a bearer token or, behind a proxy, the session cookie. The
+// rejection carries no detail: the status endpoint already exposes host,
+// port, and throughput counters, so a refusal should not add the shape of
+// the policy on top of that.
 func (h *HTTPSink) authMiddleware(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		ident, status, err := h.auth.AuthorizeRequest(r)

internal/sink/http/http_test.go

@@ -373,7 +373,7 @@ func TestProxiedStreamSessionKeepsTheProxy(t *testing.T) {
 
 // The constructor refuses what it cannot serve: the login pages outside
 // scram's proxy mode, a viewer without the login page, a path under /auth, and
-// a path ServeMux or a URL would not take literally, which panicked at Start.
+// a path ServeMux or a URL would not take literally.
 func TestUnservableOptionsAreRefused(t *testing.T) {
 	scram := map[string]any{"type": "scram", "credentials_file": scramCredentials(t)}
 	pki := testutil.NewPKI(t, "viewer-01")

internal/sink/httpchain/httpchain.go

@@ -53,7 +53,7 @@ type HTTPChainSink struct {
 	tlsEnabled bool
 	mtls       bool
 
-	// Authorization: pins the downstream server's identity
+	// Authorization: mtls pins the downstream server's identity; scram logs this sink in
 	auth *authz.Policy
 
 	client *http.Client
@@ -189,7 +189,7 @@ func (t *HTTPChainSink) Capabilities() []core.Capability {
 		caps = append(caps, core.CapTLS)
 	}
 	if t.auth.Enabled() {
-		caps = append(caps, core.CapAuth) // pins the server identity
+		caps = append(caps, core.CapAuth) // pins the server, or logs in under scram
 	}
 	return caps
 }

internal/sink/sink.go

@@ -9,7 +9,7 @@ import (
 
 // Sink represents an output data stream.
 type Sink interface {
-	// Capabilities returns a slice of supported Source capabilities
+	// Capabilities returns a slice of supported Sink capabilities
 	Capabilities() []core.Capability
 
 	// Input returns the channel for sending transport events to this sink.

internal/sink/tcp/tcp.go

@@ -221,8 +221,8 @@ func (t *TCPSink) Start(ctx context.Context) error {
 	go t.acceptLoop()
 	go t.broadcastLoop()
 
-	// Pipeline context cancellation mirrors gnet engine stop: cease accepting
-	// and tear down existing connections
+	// Pipeline context cancellation stops accepting and tears down the
+	// connections, as Stop does
 	go func() {
 		select {
 		case <-ctx.Done():
@@ -299,7 +299,8 @@ func (t *TCPSink) acceptLoop() {
 		}
 
 		if t.config.MaxConnections > 0 && t.activeConns.Load() >= t.config.MaxConnections {
-			// Load/admit race can over-admit by a conn under burst; acceptable
+			// The count rises only in handleConn, so a burst can over-admit by
+			// the conns accepted before their handleConn runs; acceptable
 			t.rejectedConns.Add(1)
 			conn.Close()
 			continue
@@ -328,7 +329,7 @@ func (t *TCPSink) handleConn(conn net.Conn) {
 	defer t.wg.Done()
 	remote := conn.RemoteAddr().String()
 
-	// Counted from accept: max_connections bounds concurrent handshakes too
+	// Counted before the handshake: max_connections bounds concurrent handshakes too
 	count := t.activeConns.Add(1)
 	defer func() {
 		conn.Close()

internal/sink/tcpchain/tcpchain.go

@@ -44,7 +44,7 @@ type TCPChainSink struct {
 	network   string
 	tlsConfig *tls.Config
 
-	// Authorization: pins the downstream server's identity
+	// Authorization: mtls pins the downstream server's identity; scram logs this sink in
 	auth *authz.Policy
 
 	input  chan core.TransportEvent
@@ -158,7 +158,7 @@ func (t *TCPChainSink) Capabilities() []core.Capability {
 		caps = append(caps, core.CapTLS)
 	}
 	if t.auth.Enabled() {
-		caps = append(caps, core.CapAuth) // pins the server identity
+		caps = append(caps, core.CapAuth) // pins the server, or logs in under scram
 	}
 	return caps
 }

internal/source/file/file_test.go

@@ -158,7 +158,7 @@ func TestRemoveWatcherPreservesReplacement(t *testing.T) {
 }
 
 // A line longer than core.MaxLogEntryBytes continues in the next entry, and the
-// lines around it are read once: the watcher used to stop at it for good
+// lines around it are read once
 func TestLongLineContinuesInTheNextEntry(t *testing.T) {
 	path := filepath.Join(t.TempDir(), "app.log")
 	long := strings.Repeat("x", core.MaxLogEntryBytes+10)

internal/source/source.go

@@ -20,7 +20,7 @@ type Source interface {
 	// Stop gracefully shuts down the source
 	Stop()
 
-	// SourceStats contains statistics about a source
+	// GetStats returns the source's statistics
 	GetStats() SourceStats
 }
 

internal/tlsx/tlsx.go

@@ -1,6 +1,8 @@
-// Package tlsx builds crypto/tls configurations from config.TLSOptions.
-// It is the single seam between declarative TLS config and the stdlib;
-// each network plugin calls exactly one constructor.
+// Package tlsx is the TLS seam of the network plugins, beside authz and
+// netacl: Server and Client build a plugin's crypto/tls configuration from
+// config.TLSOptions, a listener without files gets a certificate issued on the
+// process key, lw tls makes its CA and certificates here, and LogWarnings
+// reports settings that work but are likely mistakes.
 package tlsx
 
 import (

internal/tui/tui.go

@@ -1,7 +1,8 @@
 // Package tui is lw --tui: pipelines composed on a full-screen canvas drawn as
-// the website draws them, then run or written as a command line, environment
-// or file. The pipelines are a compose.Composition; this package only draws
-// them and turns keys into the engine's edits.
+// the website draws them, run in place with a live view of what their stages
+// pass and drop, then left to run or written as a command line, environment
+// or file. The pipelines are a compose.Composition; this package draws them,
+// turns keys into the engine's edits and runs them through a Runner from cmd/lw.
 package tui
 
 import (

internal/tui/view_test.go

@@ -187,8 +187,8 @@ func TestARowCountsAllItHolds(t *testing.T) {
 
 // The canvas cursor picks the point and v the verdicts at it: a source's
 // entries, what reaches a stage or a filter, passes it or it drops, the
-// heartbeats, what the sinks got and what the flow dropped on the way; lw's
-// own log shows from warn up at every point, and whole at its own
+// heartbeats, what the flow offered the sinks and what it dropped on the way;
+// lw's own log shows from warn up at every point, and whole at its own
 func TestThePointAndModePickTheRows(t *testing.T) {
 	c := pipeline(t, []string{"null", "null"}, []string{"null"})
 	for _, typ := range []string{"include", "exclude"} {