v0.23.0 color, presets, certificates made at startup, cli.go, sink flush

doc/README.md

@@ -31,8 +31,7 @@ streams, or downstream LogWisp nodes.
   manual, [`lw.1`](lw.1)
 - [Operations](operations.md): running, monitoring, tuning, troubleshooting
 - [To Do](todo.md): planned work in priority order: network access control
-  and the PROXY protocol, a hardening review of the config libraries,
-  packaging
+  and the PROXY protocol, packaging
 
 A fully annotated configuration covering every option lives at
 [`config/logwisp.toml`](../config/logwisp.toml).

doc/chaining.md

@@ -26,7 +26,9 @@ nodes can filter and reformat as if the entries were local.
 
 Both chain sources can feed a single pipeline (fan-in) whose sinks then fan the
 merged stream out. `test/chain-test.sh` builds the two-independent-pipelines
-variant; `test/chain-aggregate-test.sh` builds the fan-in variant.
+variant; `test/chain-aggregate-test.sh` builds the fan-in variant. The `edge`
+and `aggregator` [presets](cli.md#presets) build an authenticated TLS link of
+either transport from a few keys.
 
 ## Node Identity
 

doc/cli.md

@@ -4,7 +4,11 @@
 lw [options] [< input]
 lw -c <file> [options]
 lw --check [options]
+lw --dump [options]
+lw --preset NAME[,key=value...] [options]
 lw auth <command> [flags]
+lw tls <command> [flags]
+lw preset <name> [flags]
 lw help | -h | --help
 lw --version
 ```
@@ -14,9 +18,10 @@ stdout, line for line, until the end of input ([Built-in
 Defaults](#built-in-defaults)). With a file it runs the file's pipelines as a
 service.
 
-`lw auth` manages SCRAM credentials and logs in to `scram` listeners; see
-[below](#lw-auth). There is no certificate-generation command: use
-`openssl` or your PKI tooling — see [Security](security.md#enabling-mtls).
+A [preset](#presets) builds a common pipeline from a few keys. `lw auth`
+manages SCRAM credentials and logs in to `scram` listeners
+([below](#lw-auth)); `lw tls` makes a certificate authority and certificates
+([below](#lw-tls)). A command is recognized only as the first argument.
 
 ## Options
 
@@ -36,6 +41,11 @@ pipelines have [their own flags](#pipelines):
   `./logwisp.toml` ([Configuration](configuration.md#file-location))
 - `--check`: build every pipeline and plugin, bind nothing, report and exit;
   see [Usage Patterns](#usage-patterns)
+- `--dump`: print the effective configuration, flags, environment and presets
+  resolved into it, as TOML that `-c` reads back, and exit
+- `--color [auto|always|never]`: level names in color on console sinks,
+  default `auto` (a terminal, `NO_COLOR` unset, `TERM` not `dumb`); bare,
+  `always`. A console sink's own `color` wins ([console sink](sinks.md#console))
 - `--quiet`: silence lw's own log and notices, default `false`; pipeline
   output still flows
 - `--status_reporter=<bool>`: periodic status logging, default `true` with a
@@ -70,7 +80,10 @@ Pipeline flags define whole pipelines. They replace the configuration file's
 pipelines and the built-in default; every other key keeps its precedence.
 
 - `--pipeline NAME` starts a pipeline; specs before the first one go to a
-  pipeline named `cli`
+  pipeline named `cli`, or after the preset that starts them
+- `--preset SPEC` starts a pipeline with a [preset](#presets); later flags add
+  sources, sinks and filters to it, or replace its format, rate limit and
+  heartbeat
 - `--source SPEC`, `--sink SPEC` and `--filter SPEC` add a stage, repeatable
 - `--format SPEC`, `--rate-limit SPEC` and `--heartbeat SPEC`, once per pipeline
 - both `--flag SPEC` and `--flag=SPEC` work; `--` ends options
@@ -129,6 +142,61 @@ lw --pipeline app --source file,directory=/var/log/app \
 
 Pipelines over stdin and stdout are under [Usage Patterns](#usage-patterns).
 
+## Presets
+
+`--preset NAME,key=value,...` takes the [SPEC](#pipelines) syntax. A list
+key (`hosts`, `proxy`) takes several values, repeated (`hosts=a,hosts=b`) or
+`,`-separated; any other key takes one. `lw preset NAME [-key value ...]`
+prints the pipeline a preset expands to, ready for a configuration file;
+`lw preset NAME -h` lists its keys and defaults, `lw preset` the presets. An
+unknown key fails and lists the valid ones. A `path` is a file, a directory
+(its files) or a glob; a preset that takes one reads stdin without it, and
+`from` (`end`, or `start`) is where reading a file starts.
+
+- `pipe`: stdin to stdout, the built-in default
+  - `format` (`raw`)
+- `tail`: follow files to stdout, like `tail -F`
+  - `path` (required), `from`, `format` (`raw`)
+- `serve`: a live HTTP stream (SSE), an [http sink](sinks.md#http)
+  - `path`, `from`, `format` (`json`), `listen` (`127.0.0.1:8080`)
+  - `tls`: `off` (default), `self` (a self-signed certificate made at
+    startup), `issuer` (one signed by `issuer_cert` and `issuer_key`), or
+    `files` (`cert` and `key`); `hosts` adds names to a made certificate
+  - `users`: a credentials file; readers then log in with SCRAM
+  - `proxy`, `viewer=true`: the built-in login and viewer pages, which work
+    only behind the TLS-terminating proxies `proxy` names (a browser cannot
+    bind its login to the TLS channel); there is no standalone viewer
+- `edge`: forward to an aggregator over TLS (`tcp_chain`, or `http_chain`
+  with `transport=http`)
+  - `to` (required), `path`, `from`, `transport` (`tcp`), `node`
+  - verify the aggregator by `ca` (a CA file; default: system roots) or `pin`
+    (its `tls.pin_sha256`), and `server_name`
+  - authenticate by `user` and `password_file` (SCRAM), or a client `cert` and
+    `key` (mTLS): without either the preset refuses to run
+- `aggregator`: receive from edges over TLS, to stdout or files
+  - `listen` (`0.0.0.0:9000`), `transport` (`tcp`), `format` (`json`)
+  - `tls`: `self` (default), `issuer` or `files`, as for `serve`; never `off`
+  - authenticate by `users` (SCRAM) or `client_ca` (mTLS): one is required
+  - `out`: a directory for `aggregate*.log` files; default stdout
+
+```bash
+# follow a directory, level names in color on a terminal
+lw --preset tail,path=/var/log/app
+
+# serve it over HTTPS, self-signed, to SCRAM users; keep the result as a file
+lw --preset serve,path=/var/log/app,tls=self,users=/etc/logwisp/users.toml --dump > serve.toml
+
+# an aggregator with a certificate made at startup, and an edge pinning it
+lw --preset aggregator,users=/etc/logwisp/users.toml,out=/var/log/edges
+lw --preset edge,path=/var/log/app,to=agg.example.org:9000,pin=sha256//BASE64,\
+user=edge-01,password_file=/etc/logwisp/edge-01.pass
+```
+
+A self-signed aggregator logs its pin at startup (`pin_sha256`, a warning, so
+it shows without a configuration file too). The pin holds across reloads and
+changes when lw restarts; a fleet signs with an issuer instead, and edges set
+`ca`. See [Security](security.md#certificates-made-at-startup).
+
 ## Environment Variables
 
 Configuration paths map to environment variables by replacing `.` with `_`,
@@ -156,6 +224,7 @@ One pipeline can come from the environment, with the [SPEC](#pipelines) syntax.
 Any pipeline flag on the command line makes LogWisp ignore all of them.
 
 - `LOGWISP_PIPELINE`: the pipeline's name, default `cli`
+- `LOGWISP_PRESET`: the [preset](#presets) that starts it
 - `LOGWISP_SOURCE`, `LOGWISP_SINK`, `LOGWISP_FILTER`: one stage each
   - `LOGWISP_SOURCE_1` .. `LOGWISP_SOURCE_N` add more, after the unnumbered
     one in numeric order; likewise `LOGWISP_SINK_N` and `LOGWISP_FILTER_N`
@@ -211,7 +280,7 @@ service untouched; see [Configuration](configuration.md#hot-reload).
 ## Exit Codes
 
 - `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version` /
-  `--help`, or a valid `--check`
+  `--help`, `--dump`, or a valid `--check`
 - `1`: general error: a configuration load or validation failure (`--check`
   included), a logger init failure, a service bootstrap failure
 - `2`: an explicitly requested configuration file is not found
@@ -239,8 +308,8 @@ lw < app.log > copy.log
   - A line over 1 MiB continues in the next entry; no byte is lost.
 - Nothing is dropped: a slow reader slows the reading of stdin.
 - On a terminal, control characters are written as `<hex>` (`ESC` becomes
-  `<1b>`); pipes and files get the bytes unchanged
-  ([console sink](sinks.md#console)).
+  `<1b>`) and level names are in color; pipes and files get the bytes
+  unchanged ([console sink](sinks.md#console)).
 - When a console source reads a terminal, lw says so once on stderr, unless
   `--quiet`: `lw: reading standard input; Ctrl-D ends it (lw --help for usage)`.
 
@@ -255,10 +324,9 @@ values from the file, the environment or flags win.
 finishes when all its sources have ended, and once every pipeline has
 finished lw shuts down gracefully and exits 0. Only a console source ends on
 its own: file and network sources run until a signal, and so does a pipeline
-holding one. Console and file sinks write what is queued before the
-exit. The `http` and `tcp` sinks do not yet flush their clients' queues, so a
-connected client can miss the last entries of a finite input
-([To Do](todo.md)).
+holding one. Every sink writes what is queued before the exit; the `http`
+and `tcp` sinks give each connected client `write_timeout_ms`, at most 2 s,
+to take it.
 
 ## Usage Patterns
 
@@ -363,8 +431,9 @@ Neither touches a running LogWisp: send `SIGHUP`, since `auto_reload` does not
 watch the credentials file.
 
 **`token`** and **`stream`** build the same TLS and SCRAM client as a chain
-sink. TLS flags: `-ca-file` (default: system roots), `-server-name` (default:
-the host), and `-cert-file` / `-key-file` for a listener with `tls.client_auth`.
+sink. TLS flags: `-ca-file` (default: system roots) or `-pin-sha256` (the
+`tls.pin_sha256` a self-signed listener logs), `-server-name` (default: the
+host), and `-cert-file` / `-key-file` for a listener with `tls.client_auth`.
 There is no flag to skip verification: an unverified server could relay the
 login. Redirects are not followed. `-unbound` logs in to an `http` sink behind
 a TLS-terminating proxy (`auth.trusted_proxies`), still pinning the proxy's
@@ -403,3 +472,32 @@ curl --cacert ca.crt -H @<(printf 'Authorization: Bearer %s\n' \
 lw auth stream -addr HOST:PORT -user viewer -password-file viewer.pass \
   -ca-file ca.crt
 ```
+
+## `lw tls`
+
+Makes ECDSA P-256 certificates for `tls` blocks. It never replaces a file
+(remove one to replace it) and writes keys `0600`, certificates `0644`.
+
+```
+lw tls ca   -dir DIR [-name NAME] [-days 3650]
+lw tls cert -ca-dir DIR -name NAME [-host NAME,...] [-server] [-client]
+            [-days 397] [-out DIR]
+```
+
+- `ca`: `DIR/ca.crt` and `DIR/ca.key`, a CA that signs leaves only (path
+  length 0). `ca.crt` is what dialers set as `tls.ca_file` and listeners as
+  `tls.client_ca_file`; with `ca.key` it is a listener's
+  `tls.issuer_cert_file` and `tls.issuer_key_file`.
+- `cert`: `NAME.crt` and `NAME.key`, in `-out` (default: `-ca-dir`).
+  - `-server` for a listener: `-host` lists the DNS names and IP addresses it
+    carries, default `NAME`; `-client` for a dialer presenting a certificate;
+    both may be given.
+  - `NAME` is the subject CN, the identity `auth.type = "mtls"` matches.
+  - Valid 397 days (the longest browsers accept), never past the CA. It prints
+    the certificate's `pin_sha256`.
+
+```bash
+lw tls ca -dir /etc/logwisp/pki
+lw tls cert -ca-dir /etc/logwisp/pki -name agg.example.org -server
+lw tls cert -ca-dir /etc/logwisp/pki -name edge-01 -client
+```

doc/configuration.md

@@ -46,9 +46,12 @@ Top-level keys, each as type and default:
   emit a status report every 30 s at DEBUG level.
 - `auto_reload` (bool, `false`): watch the config file and reload pipelines on
   change.
+- `color` (string, `"auto"`): level names in color on console sinks, which may
+  set their own: `"auto"` (a terminal, `NO_COLOR` unset, `TERM` not `dumb`),
+  `"always"` or `"never"`; see the [console sink](sinks.md#console).
 
-`--version` prints version information and exits; it is not a persistent
-setting.
+`--version`, `--check` and `--dump` print and exit; they are not persistent
+settings.
 
 Note that `status_reporter` writes at DEBUG level, so it produces nothing unless
 `logging.level = "debug"`.
@@ -311,6 +314,7 @@ What a reload applies:
   - pipelines, sources, sinks
   - filters, formatters, rate limits, heartbeats
   - `status_reporter`
+  - `color`
 - Not reloaded:
   - `logging.*`: applied once at startup
   - `quiet`

doc/lw.1

@@ -1,4 +1,4 @@
-.Dd October 3, 2026
+.Dd October 4, 2026
 .Dt LW 1
 .Os
 .Sh NAME
@@ -6,10 +6,12 @@
 .Nd LogWisp log transport
 .Sh SYNOPSIS
 .Nm
-.Op Fl \-check
+.Op Fl \-check | Fl \-dump
 .Op Fl c Ar file
 .Op Fl \-quiet
+.Op Fl \-color Op Ar when
 .Op Fl \- Ns Ar key Ns = Ns Ar value ...
+.Op Fl \-preset Ar spec
 .Op Fl \-pipeline Ar name
 .Op Fl \-source Ar spec ...
 .Op Fl \-sink Ar spec ...
@@ -18,7 +20,7 @@
 .Op Fl \-rate\-limit Ar spec
 .Op Fl \-heartbeat Ar spec
 .Nm
-.Cm auth
+.Cm auth | tls | preset
 .Ar command
 .Op Ar flags
 .Nm
@@ -63,11 +65,13 @@ source runs until shutdown.
 When every pipeline has finished,
 .Nm
 shuts down gracefully and exits 0.
-Console and file sinks first write what they hold; the
+Every sink first writes what it holds; the
 .Cm http
 and
 .Cm tcp
-sinks do not yet, so a client can miss the last lines of a finite input.
+sinks give each client
+.Cm write_timeout_ms ,
+at most 2 seconds, to take it.
 Under a supervisor, standard input is usually
 .Pa /dev/null :
 a console source there ends at once.
@@ -88,6 +92,13 @@ option sets this:
 .Cm always
 or
 .Cm never .
+Its
+.Cm color
+option, by default the top-level
+.Cm color ,
+paints the level name in each line: debug cyan, info green, warn yellow,
+error bold red, trace grey, in the 16 colors every terminal and text console
+renders, with the terminal theme choosing the shades.
 .Pp
 .Nm
 stays in the foreground and logs to standard error; run a configuration as a
@@ -109,6 +120,22 @@ Set any scalar configuration key by its TOML path, such as
 A bare
 .Fl \- Ns Ar key
 sets a boolean to true.
+.It Fl \-color Op Ar when
+Color the level names console sinks write:
+.Cm auto
+.Pq the default: on a terminal, with Ev NO_COLOR unset and Ev TERM not Cm dumb ,
+.Cm always
+.Pq what a bare Fl \-color No means
+or
+.Cm never .
+A console sink's own
+.Cm color
+option takes precedence.
+The codes go into whatever the format makes, JSON included, so output another
+program parses keeps
+.Cm auto
+or
+.Cm never .
 .It Fl \-quiet
 Turn off the log of
 .Nm
@@ -151,6 +178,11 @@ or opening, reading or creating a log file; print
 and exit 0, or the error and exit 1, or 2 when a named configuration file is
 missing
 .Pq see Sx EXIT STATUS .
+.It Fl \-dump
+Print the effective configuration, with the options, environment and
+presets resolved into it, as a file
+.Fl c
+reads back, and exit.
 .It Fl \-version
 Print the version, commit and build time, and exit.
 .It Fl h , Fl \-help , Cm help
@@ -176,7 +208,12 @@ Start a new pipeline.
 Specifications before the first
 .Fl \-pipeline
 belong to a pipeline named
-.Dq cli .
+.Dq cli ,
+or after the preset that starts them.
+.It Fl \-preset Ar spec
+Start the pipeline with a preset, described in
+.Sx Presets ;
+later options add sources, sinks and filters to it, or replace its format.
 .It Fl \-source Ar spec , Fl \-sink Ar spec , Fl \-filter Ar spec
 Add a source, sink or filter; each is repeatable.
 .It Fl \-format Ar spec , Fl \-rate\-limit Ar spec , Fl \-heartbeat Ar spec
@@ -218,6 +255,125 @@ Any pipeline flag on the command line makes
 .Nm
 ignore the pipeline variables in
 .Sx ENVIRONMENT .
+.Ss Presets
+A preset builds a common pipeline from a few keys:
+.Fl \-preset Ar name Ns Op , Ns Ar key Ns = Ns Ar value ... ,
+in the syntax of the other specifications;
+.Nm
+.Cm preset Ar name Op Fl Ns Ar key Ar value ...
+prints the pipeline it expands to, and
+.Fl h
+there lists its keys and defaults.
+A
+.Ar path
+is a file, a directory, whose files are followed, or a glob; without one, a
+preset that takes it reads standard input.
+.Bl -tag -width Ds
+.It Cm pipe Op Cm format
+Standard input to standard output, the pipeline
+.Nm
+runs without a configuration.
+.It Cm tail path Op Cm from , format
+Follow files to standard output, like
+.Ql tail \-F .
+.It Cm serve Oo Cm path , listen , format , tls , users , proxy , viewer Oc
+Serve a live HTTP stream on
+.Cm listen
+.Pq default Li 127.0.0.1:8080 .
+.Cm tls
+is
+.Cm off ,
+.Cm self
+.Pq a self-signed certificate made at startup ,
+.Cm issuer
+.Pq one signed by Cm issuer_cert No and Cm issuer_key
+or
+.Cm files
+.Pq Cm cert No and Cm key ;
+.Cm users
+names a credentials file, so readers log in with SCRAM.
+The built-in login and viewer pages,
+.Cm viewer=true ,
+work only behind a TLS-terminating proxy named by
+.Cm proxy ,
+since a browser cannot bind its login to the TLS channel.
+.It Cm edge to Oo Cm path , transport , ca | pin , user , password_file , cert , key , node Oc
+Forward to an aggregator at
+.Cm to
+over TLS, by
+.Cm tcp_chain
+or, with
+.Cm transport=http ,
+.Cm http_chain .
+It refuses to run without credentials:
+.Cm user
+and
+.Cm password_file ,
+or a client
+.Cm cert
+and
+.Cm key .
+.It Cm aggregator Oo Cm listen , transport , tls , users , client_ca , out , format Oc
+Receive from edges on
+.Cm listen
+.Pq default Li 0.0.0.0:9000
+over TLS, never off
+.Pq default Cm tls=self ,
+authenticated by the
+.Cm users
+credentials file or the client certificates
+.Cm client_ca
+vouches for, and write to standard output or to files in
+.Cm out .
+.El
+.Ss Certificate commands
+.Nm
+.Cm tls
+makes ECDSA P-256 certificates; it never replaces a file, and writes keys
+with mode 0600.
+.Bl -tag -width Ds
+.It Cm ca Fl dir Ar dir Oo Fl name Ar name Oc Op Fl days Ar n
+Create a certificate authority,
+.Pa dir/ca.crt
+and
+.Pa dir/ca.key ,
+valid 3650 days.
+.It Cm cert Fl ca-dir Ar dir Fl name Ar name Oo Fl host Ar names Oc Oo Fl server Oc Oo Fl client Oc Oo Fl days Ar n Oc Op Fl out Ar dir
+Issue
+.Pa name.crt
+and
+.Pa name.key
+from that authority, valid 397 days and never past it.
+.Fl host
+lists the DNS names and addresses a server certificate carries
+.Pq default: Ar name ;
+.Ar name
+is also the subject, the identity
+.Cm auth.type=mtls
+checks.
+.El
+.Pp
+A listener needs no files:
+.Cm tls.self_signed=true ,
+or
+.Cm tls.issuer_cert_file
+with
+.Cm tls.issuer_key_file ,
+make its certificate at startup, for
+.Cm tls.hosts ,
+its
+.Cm host ,
+this machine's name, localhost and the loopback addresses.
+Its key lasts until
+.Nm
+exits, across reloads.
+A self-signed listener logs the
+.Cm tls.pin_sha256
+that dialers set in place of
+.Cm tls.ca_file ;
+it changes when
+.Nm
+restarts, so a fleet uses an issuer instead.
 .Ss Authentication commands
 .Nm
 .Cm auth
@@ -258,7 +414,9 @@ The
 .Ar tls-flags
 are
 .Fl ca-file Ar file
-.Pq default: the system roots ,
+.Pq default: the system roots
+or
+.Fl pin-sha256 Ar pin ,
 .Fl server-name Ar name ,
 and
 .Fl cert-file Ar file
@@ -290,6 +448,8 @@ The directory holding
 .Pa logwisp.toml .
 .It Ev LOGWISP_PIPELINE
 The name of the one pipeline the variables below define.
+.It Ev LOGWISP_PRESET
+The preset that starts it.
 All of these are ignored when a pipeline option is on the command line.
 .It Ev LOGWISP_SOURCE , LOGWISP_SINK , LOGWISP_FILTER
 Source, sink and filter specifications, in the syntax of the matching
@@ -299,6 +459,13 @@ counting from 1, adds more, as in
 .Ev LOGWISP_SINK_1 .
 .It Ev LOGWISP_FORMAT , LOGWISP_RATE_LIMIT , LOGWISP_HEARTBEAT
 The pipeline's format, rate limit and heartbeat specifications.
+.It Ev NO_COLOR , TERM
+Set to anything,
+.Ev NO_COLOR
+turns
+.Cm auto
+color off, as does
+.Ev TERM=dumb .
 .El
 .Pp
 Secrets never go in the environment: TLS keys, credentials files and
@@ -357,8 +524,11 @@ as it ends
 the shell reports status 141.
 .Pp
 .Nm
-.Cm auth
-exits 0 on success or help, 1 on failure, and 2 on a usage error.
+.Cm auth ,
+.Cm tls
+and
+.Cm preset
+exit 0 on success or help, 1 on failure, and 2 on a usage error.
 .Sh SIGNALS
 .Bl -tag -width "SIGINT, SIGTERM"
 .It Dv SIGINT , SIGTERM
@@ -405,6 +575,33 @@ $ lw \-\-source 'file,directory=/var/log/app,pattern=*.log,raw=true' \e
     > all.log
 .Ed
 .Pp
+Follow every file in a directory, with the level names in color on a
+terminal:
+.Bd -literal -offset indent
+$ lw \-\-preset tail,path=/var/log/app
+.Ed
+.Pp
+Print the configuration a set of options makes, to keep as a file:
+.Bd -literal -offset indent
+$ lw \-\-preset serve,path=/var/log/app,tls=self \-\-dump > serve.toml
+.Ed
+.Pp
+Ship a directory from an edge to an aggregator: on the aggregator, create
+the authority, its certificate and a user for the edge; copy
+.Pa ca.crt
+and the password file to the edge; then run both:
+.Bd -literal -offset indent
+# lw tls ca \-dir /etc/logwisp/pki
+# lw tls cert \-ca-dir /etc/logwisp/pki \-name agg.example.org \-server
+# lw auth add-user \-credentials /etc/logwisp/users.toml \e
+    \-user edge-01 \-password-file /root/edge-01.pass
+agg# lw \-\-preset aggregator,tls=files,users=/etc/logwisp/users.toml,\e
+cert=/etc/logwisp/pki/agg.example.org.crt,\e
+key=/etc/logwisp/pki/agg.example.org.key,out=/var/log/edges
+edge# lw \-\-preset edge,path=/var/log/app,to=agg.example.org:9000,\e
+ca=/etc/logwisp/pki/ca.crt,user=edge-01,password_file=/root/edge-01.pass
+.Ed
+.Pp
 Check a configuration without binding a port or touching a log file:
 .Bd -literal -offset indent
 $ lw \-\-check \-c /etc/logwisp/logwisp.toml

doc/networking.md

@@ -213,6 +213,9 @@ headers, and entry encoding.
   `client_ca_file` does not contain the issuer of the client certificate.
 - `certificate is not valid for any names` / SAN mismatch — the dialed `host`
   is not covered by the server certificate's SANs; set `server_name`.
+- `server key sha256//... matches no tls.pin_sha256` — the listener is not the
+  one pinned, or a self-signed listener restarted and made a new key: take the
+  pin from its startup warning, or move it to an issuer and `ca_file`.
 - `protocol version not supported` — one side is pinned to `min_version = "1.3"`
   and the other cannot negotiate it.
 - Listeners log handshake failures at WARN with the remote address (the `tcp`

doc/operations.md

@@ -17,8 +17,10 @@ lw exits on its own only at the end of input: once every pipeline has
 finished, a pipeline finishing when all its sources have ended. Only a console
 source ends, when stdin does; file and network sources run until a signal, and
 lw shuts down gracefully, exit 0, either way.
-Console and file sinks write what is queued before the exit; the `http` and
-`tcp` sinks do not yet flush their clients' queues ([To Do](todo.md)). See
+Every sink writes what is queued before the exit; the `http` and `tcp` sinks
+give each client `write_timeout_ms`, at most 2 s, to take it. A reload stops
+the old sinks the same way, so a client that stopped reading delays it by
+that much. See
 [CLI](cli.md#built-in-defaults).
 
 There is no built-in daemon mode. Run LogWisp in the foreground under a

doc/security.md

@@ -8,7 +8,11 @@ configure it, and — equally important — what it does not yet do.
 **Implemented:**
 
 - TLS 1.2 / 1.3 on all network sources and sinks.
-- Server certificate verification by dialers.
+- Server certificate verification by dialers, by a CA or by the pin of the
+  server's key (`pin_sha256`).
+- Certificates without files: listeners make theirs at startup, self-signed
+  or from an issuer CA, and `lw tls` makes a CA and certificates: see
+  [Certificates made at startup](#certificates-made-at-startup).
 - Mutual TLS (client certificate required and verified), at the transport
   layer.
 - Peer identity recorded per session.
@@ -26,16 +30,20 @@ configure it, and — equally important — what it does not yet do.
   [Startup Warnings](#startup-warnings).
 - Unknown configuration keys rejected: a typo in `tls`, `auth` or a table path
   fails startup.
-- Configuration input hardened (lixenwraith/config v0.2.2, reviewed with
-  adversarial tests and fuzzing like `auth`):
+- Configuration input hardened (lixenwraith/config v0.2.2 and
+  lixenwraith/toml v0.1.3, reviewed with adversarial tests and fuzzing like
+  `auth`):
+  - a document creates at most 65536 tables, so a small file cannot exhaust
+    memory, and unescaped control characters are refused in strings and
+    comments;
   - a string in the file is one list entry, so a filter pattern holding a
     comma is not split into patterns that match nothing;
   - the file is opened without blocking, so a FIFO swapped in for it cannot
     hang the watcher or a `SIGHUP` reload;
   - a permission change on the file is reported once and does not stop
     auto-reload;
-  - conversion errors name the key, never the value, so a secret placed in
-    the wrong key does not reach the log.
+  - parse and conversion errors name the key, never the value, so a secret
+    placed in the wrong key does not reach the log.
 
 **Not implemented:**
 
@@ -69,6 +77,11 @@ client_ca_file       = ""
 ca_file              = ""
 server_name          = ""
 insecure_skip_verify = false
+pin_sha256           = ""
+self_signed          = false
+issuer_cert_file     = ""
+issuer_key_file      = ""
+hosts                = []
 min_version          = "1.3"
 ```
 
@@ -76,11 +89,16 @@ Options by role, each with its default:
 
 - Both roles
   - `enabled` (`false`): master switch; when false the whole block is ignored.
-  - `cert_file`: local certificate; **required** for listeners, an optional
-    client identity for dialers.
+  - `cert_file`: local certificate; a listener needs it, `self_signed` or the
+    issuer files, a dialer may present it as its client identity.
   - `key_file`: private key for `cert_file`; set the two together.
   - `min_version` (`"1.3"`): `"1.2"` or `"1.3"`.
 - Listeners
+  - `self_signed` (`false`): make a self-signed certificate at startup.
+  - `issuer_cert_file`, `issuer_key_file`: make one at startup, signed by this
+    CA (`lw tls ca`).
+  - `hosts` (`[]`): names and addresses a made certificate carries, beyond the
+    listener's `host`, `os.Hostname()`, `localhost`, `127.0.0.1` and `::1`.
   - `client_auth` (`false`): require and verify a client certificate (mTLS).
   - `client_ca_file`: CA bundle that verifies client certificates; **required**
     when `client_auth` is true.
@@ -89,6 +107,9 @@ Options by role, each with its default:
   - `server_name` (the configured `host`): SNI and certificate name to verify
     against.
   - `insecure_skip_verify` (`false`): disable server verification.
+  - `pin_sha256`: `sha256//BASE64` of the server's public key, curl's
+    `--pinnedpubkey` form, `;` between several; it replaces the CA and name
+    checks.
 
 Listeners are the `tcp` and `http` sinks and the `tcp_chain` and `http_chain`
 sources; dialers are the `tcp_chain` and `http_chain` sinks.
@@ -102,7 +123,12 @@ sources; dialers are the `tcp_chain` and `http_chain` sinks.
 
 Misconfiguration fails at plugin construction, before the pipeline starts:
 
-- a listener with `enabled = true` and no `cert_file`/`key_file`
+- a listener with `enabled = true` and none of `cert_file`/`key_file`,
+  `self_signed`, `issuer_cert_file`/`issuer_key_file`, or more than one; an
+  issuer that is no CA (`lw tls ca` makes one)
+- `pin_sha256` with `ca_file` or `insecure_skip_verify`, or not of the
+  `sha256//BASE64` form; a listener key on a dialer, or `pin_sha256` on a
+  listener
 - `client_auth = true` with no `client_ca_file`
 - a dialer with only one of `cert_file` / `key_file`
 - a certificate or key that will not load, or a CA file containing no
@@ -298,20 +324,53 @@ Errors read like `auth: type "mtls" requires tls.client_auth`.
 Some settings work but are usually mistakes. Each plugin reports them at WARN
 when it is constructed, so startup and every reload repeat them:
 
-- a certificate in `cert_file`, `ca_file` or `client_ca_file` that has expired,
-  is not yet valid, or expires within 30 days
+- a certificate in `cert_file`, `ca_file`, `client_ca_file` or
+  `issuer_cert_file` that has expired, is not yet valid, or expires within 30
+  days
+- a self-signed listener, with the `pin_sha256` its dialers need
 - `insecure_skip_verify` on a dialer
 - an `allow_patterns` entry not anchored at both ends of every alternative
   (`^a|b$` admits `a…` and `…b`)
-- a `key_file`, `credentials_file` or `password_file` every local user can read
+- a `key_file`, `issuer_key_file`, `credentials_file` or `password_file` every
+  local user can read
   (once per path per process)
 
+## Certificates made at startup
+
+A listener needs no certificate files. With `self_signed = true`, or with
+`issuer_cert_file` and `issuer_key_file`, it makes an ECDSA P-256 key once per
+process and, at startup and every reload, a certificate for it valid 397 days
+(never past the issuer), for `hosts`, its `host` (unless a wildcard),
+`os.Hostname()`, `localhost`, `127.0.0.1` and `::1`.
+
+- **Self-signed.** No CA can vouch for it, so dialers pin its key:
+  `pin_sha256`, which the listener logs at WARN on every start. A reload
+  keeps the key and so the pin; a restart changes both, and every dialer must
+  be given the new pin. Suits one aggregator and a few edges, a test, or a
+  stream read by `curl --pinnedpubkey`.
+- **Issuer.** `lw tls ca` makes a CA once; the listener signs its own
+  certificate with it, and dialers verify it with `ca_file = ca.crt`, by name,
+  across restarts. The CA key then lives on the listener's host: give each
+  listener group its own CA, or issue certificates elsewhere with
+  `lw tls cert` and use `cert_file`.
+- **Pins.** A pin is the SHA-256 of the server's public key, not of its
+  certificate, so it survives reissue. `pin_sha256` accepts several, `;`
+  between them, for a rotation. It replaces chain, name and validity checks:
+  whoever holds the key is the server. `scram` dialers still bind every login
+  to that certificate.
+
+```bash
+lw tls ca -dir /etc/logwisp/pki                       # ca.crt, ca.key (0600)
+lw tls cert -ca-dir /etc/logwisp/pki -name agg.example.org -server
+lw tls cert -ca-dir /etc/logwisp/pki -name edge-01 -client
+```
+
 ## Enabling mTLS
 
 ### 1. Generate a CA and certificates
 
-LogWisp does not generate certificates; use `openssl`, `cfssl`, `step-cli`, or
-your existing PKI.
+`lw tls` makes them ([above](#certificates-made-at-startup)); with `openssl`,
+`cfssl`, `step-cli` or an existing PKI:
 
 ```bash
 # CA

doc/sinks.md

@@ -27,6 +27,7 @@ type = "console"
 target      = "stdout"
 buffer_size = 1000
 escape      = "auto"
+color       = "auto"
 ```
 
 Options, each as type and default:
@@ -36,6 +37,9 @@ Options, each as type and default:
 - `escape` (string, `auto`): write control characters as `<hex>` of their
   bytes, `ESC` as `<1b>`: `auto` when the output is a terminal, `always`, or
   `never`.
+- `color` (string, the top-level `color`, itself `auto`): paint level names:
+  `auto` on a terminal with `NO_COLOR` unset and `TERM` not `dumb`, `always`,
+  or `never`.
 
 > `split` is **not** a valid target for this sink and is rejected at startup.
 > Level-based splitting exists only for LogWisp's own application log
@@ -50,6 +54,14 @@ Payloads are written as the formatter made them, one record per line.
   characters, such as the joiners emoji and Indic scripts need, are kept.
   `never` keeps an application's own ANSI colours; under `auto`, pipes and
   files get the bytes unchanged.
+- Color: the first whole word in a line, in any case, that names the entry's
+  level is painted: `DEBUG`/`DBG` cyan (blue is hard to read on black),
+  `INFO`/`INF` green, `WARNING`/`WARN` yellow, `ERROR`/`ERR`/`FATAL` bold red,
+  `TRACE` grey. The colors are ANSI 16, so the terminal's theme picks the
+  shades and a Linux or BSD text console renders them; the codes are the
+  sink's own, written after escaping. A line whose source found no level
+  stays plain. The codes go into whatever the format makes, JSON included:
+  for output another program parses keep `auto` or `never`, not `always`.
 - Backpressure: the sink never drops. When its output is slow (a slow reader,
   a paused terminal, a stalled log collector) the pipeline waits for it, with
   its other sinks. A service whose stdout may stall writes to a `file` sink
@@ -205,10 +217,11 @@ Options, each as type and default:
 - Clients whose session has been idle-expired by the session manager are
   evicted by the broker. With the idle comment above, that reaches only a peer
   that has stopped accepting bytes on a sink configured `write_timeout_ms = 0`.
-- On shutdown, connected clients receive
-  `event: disconnect / data: {"reason":"server_shutdown"}`. Entries still
-  queued for them are not flushed first, so a client can miss the last ones
-  of a finite input ([To Do](todo.md)).
+- On shutdown (a reload too), each stream writes its queue and what remains
+  of the sink's input, then
+  `event: disconnect / data: {"reason":"server_shutdown"}`, on its own and
+  within `write_timeout_ms`, at most 2 s; a client still behind is cut then,
+  and one that stopped reading costs the others nothing.
 - HTTP/2 is negotiated via ALPN when TLS is enabled; plaintext is HTTP/1.1.
 
 **Status endpoint** returns service and version identity, host, port, TLS flag,
@@ -287,8 +300,8 @@ Options, each as type and default:
   whole timeout, so the client is disconnected immediately rather than retried.
 - A client whose send queue is full has that event dropped (`dropped_writes`)
   and stays connected.
-- On shutdown, entries still queued for clients are not flushed first, as
-  with the `http` sink.
+- On shutdown, as with the `http` sink, each client receives what is queued,
+  within `write_timeout_ms`, at most 2 s, before it is disconnected.
 - With TLS enabled the handshake runs under a 10 s bound *after* the
   `max_connections` check, so concurrent handshakes are bounded too.
 - With an `auth` block, authorization runs after that handshake and *before*

doc/todo.md

@@ -122,23 +122,7 @@ connection level, below TLS, so a refused peer costs no handshake.
   - Denied and allowed clients; per-client SCRAM budgets behind one proxy
     address.
 
-## 2. Flush network sinks at shutdown
-
-At the end of a finite input lw shuts down, and the `http` and `tcp` sinks
-disconnect their clients without writing the entries still queued for them, so
-a client can miss the last ones. On `Stop`, each sink's broker should first move
-its input queue into the client queues, then let every client writer drain its
-queue within a bound (`write_timeout_ms`) before the disconnect frame. Verify
-with a finite stdin into each sink and a connected client counting lines.
-
-## 3. Bump `lixenwraith/toml`
-
-Its hardening commits (a table budget, TOML 1.1 escapes, control characters
-refused outside strings, bounded errors that never print a value) wait to merge
-in that repository. Then bump it here and drop this item; security.md's library
-note already covers it.
-
-## 4. Packaging: AUR, FreeBSD ports, Debian
+## 2. Packaging: AUR, FreeBSD ports, Debian
 
 The foundation exists:
 - The `lw` binary name, free in Arch (official repositories and AUR), Ubuntu