v0.22.1 cli flag and behavior, and test output improvement

doc/README.md

@@ -41,21 +41,24 @@ A fully annotated configuration covering every option lives at
 
 ### Pipeline
 
+- A Unix filter without configuration: stdin to stdout, line for line
 - Independent named pipelines, each `sources → flow → sinks`
 - Fan-in (many sources per pipeline) and fan-out (many sinks per pipeline)
-- Non-blocking sink dispatch: a stalled sink drops its own events and never
-  stalls the pipeline or its sibling sinks
+- Isolated sinks: a stalled sink drops its own events rather than stall the
+  pipeline; only the console sink waits, so a filter loses no line
 - Hot reload of pipeline configuration via `SIGHUP`/`SIGUSR1` or a file watch
 
 ### Inputs
 
-`file` (directory tail with rotation detection), `console` (stdin),
+`file` (directory tail with rotation detection), `console` (stdin, one
+reader per process),
 `random` (synthetic generator), `null`, and the chain ingest listeners
 `tcp_chain` and `http_chain`.
 
 ### Outputs
 
-`console`, `file` (rotating), `http` (Server-Sent Events plus a JSON status
+`console` (control characters escaped on a terminal), `file` (rotating),
+`http` (Server-Sent Events plus a JSON status
 endpoint), `tcp` (broadcast server), `null`, and the chain forwarders
 `tcp_chain` and `http_chain`.
 
@@ -83,9 +86,20 @@ See [Security](security.md) for what each layer does and does not give you.
 
 ## Quick Start
 
+Without a configuration file `lw` is a filter, exiting 0 at the end of input;
+a pipeline without `--source` reads stdin and one without `--sink` writes
+stdout:
+
+```bash
+lw < app.log > copy.log
+tail -F app.log | lw --filter include,patterns=ERROR,patterns=WARN
+```
+
+As a service, a file holds the pipelines:
+
 ```toml
 [[pipelines]]
-name = "default"
+name = "app"
 
 [pipelines.flow.format]
 type = "json"

doc/architecture.md

@@ -64,14 +64,16 @@ registry rejects a second instance of any such type.
 ### Entry lifecycle
 
 1. **Source** produces a `core.LogEntry` and publishes it to every subscriber
-   channel it has handed out. Publication is non-blocking: a full subscriber
-   channel increments the source's `dropped_entries` counter.
+   channel it has handed out; the pipeline subscribes before the source
+   starts. Publication is non-blocking: a full subscriber channel increments
+   the source's `dropped_entries` counter. The console source alone waits,
+   since stdin can simply be read later.
 2. **Flow** applies, in order: rate limit → filter chain → formatter. A drop at
    any stage ends the entry's life and increments `flow.total_dropped`.
 3. The formatter output becomes a `core.TransportEvent`, which carries both the
    formatted `Payload` and the original structured `Entry`.
 4. **Dispatch** sends the event to every sink's input channel with a
-   non-blocking send.
+   non-blocking send, except to a sink declaring `CapBackpressure`.
 
 `LogEntry` fields:
 
@@ -90,15 +92,17 @@ structured entry rather than shipping whatever text the local formatter chose.
 
 ### Back-pressure and drops
 
-There is exactly one drop policy and it is not configurable: **never block**.
-Each stage drops and counts instead of waiting:
+The drop policy is not configurable: **never block**, except where waiting
+loses nothing. Each stage drops and counts instead of waiting:
 
 - Source → subscriber: a full channel drops the entry; source
-  `dropped_entries`.
+  `dropped_entries`. The console source waits instead and reads stdin later.
 - Flow: the rate limiter, a filter rejection or a format error drops it;
   `flow.total_dropped`.
 - Pipeline → sink: a full sink input drops it for that sink only; pipeline
-  `total_dropped_by_sink`.
+  `total_dropped_by_sink`. A sink declaring `core.CapBackpressure`, the console
+  sink, makes dispatch wait instead, so a filter loses no line to a slow
+  reader and its pipeline runs at the reader's pace; `Stop` ends the wait.
 - TCP/HTTP sink → client queue: a full queue drops it for that client only;
   sink `dropped_writes`.
 
@@ -126,9 +130,16 @@ on a non-retryable response or on shutdown (`dropped_batches`).
 
 `Pipeline.Stop` is deliberately ordered so in-flight data drains:
 
-1. Stop all sources concurrently; each closes its subscriber channels.
+1. Release dispatch waits on backpressure sinks; stop all sources
+   concurrently, each closing its subscriber channels.
 2. Wait for the run loop, which ends when every subscription channel closes.
-3. Stop all sinks concurrently.
+3. Stop all sinks concurrently; console and file sinks write their queue
+   first. Then cancel the pipeline context.
+
+A finite source ends on its own: the console source at the end of stdin. The
+run loop then returns and the pipeline's `Finished` channel closes; once every
+pipeline has finished, the service's `Done` closes and `main` shuts down as on
+`SIGTERM`, exiting 0.
 
 ## Network Architecture
 

doc/cli.md

@@ -1,13 +1,19 @@
 # Command Line Interface
 
 ```
-lw [options]
+lw [options] [< input]
+lw -c <file> [options]
 lw --check [options]
 lw auth <command> [flags]
 lw help | -h | --help
 lw --version
 ```
 
+Without a configuration file or pipeline options, lw is a filter: stdin to
+stdout, line for line, until the end of input ([Built-in
+Defaults](#built-in-defaults)). With a file it runs the file's pipelines as a
+service.
+
 `lw auth` manages SCRAM credentials and logs in to `scram` listeners; see
 [below](#lw-auth). There is no certificate-generation command: use
 `openssl` or your PKI tooling — see [Security](security.md#enabling-mtls).
@@ -25,12 +31,15 @@ pipelines have [their own flags](#pipelines):
 
 ### Common
 
-- `-c <path>`, `--config <path>`, `--config=<path>`: the configuration file,
-  default `./logwisp.toml`
+- `-c <path>`, `--config <path>`, `--config=<path>`: the configuration file;
+  unnamed, `~/.config/logwisp/logwisp.toml` if it exists, else
+  `./logwisp.toml` ([Configuration](configuration.md#file-location))
 - `--check`: build every pipeline and plugin, bind nothing, report and exit;
   see [Usage Patterns](#usage-patterns)
-- `--quiet`: suppress all application output, default `false`
-- `--status_reporter=<bool>`: periodic status logging, default `true`
+- `--quiet`: silence lw's own log and notices, default `false`; pipeline
+  output still flows
+- `--status_reporter=<bool>`: periodic status logging, default `true` with a
+  configuration file, `false` without
 - `--auto_reload=<bool>`: reload when the configuration file changes, default
   `false`
 - `--version`: print the version and exit
@@ -41,8 +50,10 @@ path returns an error, and `--` ends option parsing.
 
 ### Logging
 
-- `--logging.output`: `file`, `stdout`, `stderr`, `split`, `all` or `none`
-- `--logging.level`: `debug`, `info`, `warn` or `error`
+- `--logging.output`: `file`, `stdout`, `stderr`, `split`, `all` or `none`,
+  default `stderr`, so stdout carries only data
+- `--logging.level`: `debug`, `info`, `warn` or `error`, default `info` with a
+  configuration file, `warn` without
 - `--logging.format`: `raw`, `txt` or `json`
 - `--logging.sanitization`: `raw`, `json`, `txt` or `shell`
 - `--logging.file.directory`: a path
@@ -63,6 +74,10 @@ pipelines and the built-in default; every other key keeps its precedence.
 - `--source SPEC`, `--sink SPEC` and `--filter SPEC` add a stage, repeatable
 - `--format SPEC`, `--rate-limit SPEC` and `--heartbeat SPEC`, once per pipeline
 - both `--flag SPEC` and `--flag=SPEC` work; `--` ends options
+- a pipeline without `--source` reads stdin (console source `stdin`), one
+  without `--sink` writes stdout (console sink `stdout`)
+- stdin has one reader, so the whole configuration holds at most one console
+  source: `pipeline "b": a console source already reads stdin in pipeline "a"`
 
 A SPEC is a comma-separated list:
 
@@ -96,7 +111,7 @@ A misspelled plugin key fails when the plugin is built, before any listener
 opens: `failed to create sink http: ... unknown key "tls.enabeld"`.
 
 ```bash
-# tail a directory, serve it over SSE
+# tail a directory, serve it over SSE on every interface
 lw --source 'file,directory=/var/log/app,pattern=*.log' \
    --sink http,host=0.0.0.0,port=8080
 
@@ -106,12 +121,14 @@ lw --source file,directory=/var/log/app \
    --sink "http,port=8443,auth.type=scram,auth.credentials_file=/etc/logwisp/users.toml,\
 tls.enabled=true,tls.cert_file=/etc/logwisp/server.crt,tls.key_file=/etc/logwisp/server.key"
 
-# two pipelines
-lw --pipeline app --source file,directory=/var/log/app --sink console \
+# two pipelines: app writes stdout, relay a file
+lw --pipeline app --source file,directory=/var/log/app \
    --pipeline relay --source tcp_chain,port=9000 \
    --sink file,directory=/var/log/relay,name=relay
 ```
 
+Pipelines over stdin and stdout are under [Usage Patterns](#usage-patterns).
+
 ## Environment Variables
 
 Configuration paths map to environment variables by replacing `.` with `_`,
@@ -154,21 +171,28 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
   -e LOGWISP_FILTER='exclude,patterns=DEBUG' \
   -e LOGWISP_SINK='http,host=0.0.0.0,port=8080' \
   -e LOGWISP_SINK_1=console \
+  -e LOGWISP_LOGGING_LEVEL=info \
   logwisp
 ```
 
+Without a file, lw logs at `warn` with no status reporter, so a container
+configured through the environment passes `LOGWISP_LOGGING_LEVEL=info` (and
+`LOGWISP_STATUS_REPORTER=true`) to keep the service defaults. The image does
+not set them: environment variables would override a mounted file's values.
+
 ## Precedence
 
 - Pipelines
   - pipeline flags, if any are given
   - else pipeline variables, if any are set
   - else the configuration file's `[[pipelines]]`
-  - else the built-in default
+  - else the built-in `pipe` ([Built-in Defaults](#built-in-defaults))
 - Every other key
   1. command-line flags
   2. environment variables
   3. configuration file
-  4. built-in defaults
+  4. built-in defaults; `logging.level` and `status_reporter` depend on
+     whether a file was found
 
 A reload rereads the file and keeps the command-line or environment pipelines.
 
@@ -186,10 +210,13 @@ service untouched; see [Configuration](configuration.md#hot-reload).
 
 ## Exit Codes
 
-- `0`: clean shutdown, `--version` / `--help`, or a valid `--check`
+- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version` /
+  `--help`, or a valid `--check`
 - `1`: general error: a configuration load or validation failure (`--check`
   included), a logger init failure, a service bootstrap failure
 - `2`: an explicitly requested configuration file is not found
+- killed by `SIGPIPE` (shell status `141`): the reader of stdout went away, as
+  in `lw | head`; `cat` ends the same way
 
 Exit code 2 applies only when the file was named explicitly (`-c`,
 `--config=`, or the `LOGWISP_CONFIG_*` variables). A missing discovered default
@@ -197,35 +224,77 @@ is not an error, and LogWisp starts on built-in defaults.
 
 ## Built-in Defaults
 
-With no configuration file present, LogWisp runs one pipeline named
-`default_pipeline`: a `random` source with `special = true`, JSON formatting,
-a rate limit of 5 entries/second with a burst of 10 and `policy = "drop"`, and a
-`console` sink on stdout. It is a self-demonstrating idle mode, not a useful
-production configuration.
+Without a configuration file and without pipeline flags or variables, lw runs
+one built-in pipeline, `pipe`: console source `stdin`, `raw` format, console
+sink `stdout`, no filter, no rate limit. lw is then a Unix filter, like `cat`:
 
-As soon as the file defines `[[pipelines]]`, or pipeline flags or variables are
-given, that entire default pipeline — rate limit included — is replaced rather
-than merged.
+```bash
+lw < app.log > copy.log
+```
+
+- Each line of stdin is one entry and one line of stdout.
+  - The terminator (`\n` or `\r\n`) is written back as `\n`; an unterminated
+    last line is kept and terminated.
+  - Blank lines are skipped.
+  - A line over 1 MiB continues in the next entry; no byte is lost.
+- Nothing is dropped: a slow reader slows the reading of stdin.
+- On a terminal, control characters are written as `<hex>` (`ESC` becomes
+  `<1b>`); pipes and files get the bytes unchanged
+  ([console sink](sinks.md#console)).
+- When a console source reads a terminal, lw says so once on stderr, unless
+  `--quiet`: `lw: reading standard input; Ctrl-D ends it (lw --help for usage)`.
+
+A file that defines no `[[pipelines]]` still runs `pipe`. The file's
+`[[pipelines]]`, or pipeline flags or variables, replace it whole.
+
+Without a file lw also logs less: `logging.level` defaults to `warn` and
+`status_reporter` to `false`, against `info` and `true` with one. Explicit
+values from the file, the environment or flags win.
+
+**End of input.** When stdin ends, the console source ends. A pipeline
+finishes when all its sources have ended, and once every pipeline has
+finished lw shuts down gracefully and exits 0. Only a console source ends on
+its own: file and network sources run until a signal, and so does a pipeline
+holding one. Console and file sinks write what is queued before the
+exit. The `http` and `tcp` sinks do not yet flush their clients' queues, so a
+connected client can miss the last entries of a finite input
+([To Do](todo.md)).
 
 ## Usage Patterns
 
+**Filter**
+
+```bash
+# copy, line for line; exits 0 at the end of input
+lw < app.log > copy.log
+
+# errors and warnings only
+tail -F app.log | lw --filter include,patterns=ERROR,patterns=WARN
+
+# stdin as a live SSE stream; the http and tcp sinks bind 0.0.0.0 by default
+journalctl -f | lw --sink http,host=127.0.0.1,port=8080
+
+# a directory's files to one file, through stdout
+lw --source file,directory=/var/log/app,pattern='*.log' > all.log
+```
+
+A console sink never drops: when its reader is slow, the pipeline waits, and
+so does the reading of stdin.
+
 **Development**
 
 ```bash
-# verbose, everything to stderr
-lw -c dev.toml --logging.output=stderr --logging.level=debug
-
-# no config at all: synthetic generator to stdout
-lw
+# a file's pipelines, verbose
+lw -c dev.toml --logging.level=debug
 ```
 
 **Configuration check**
 
 `lw --check` validates without running: it builds every pipeline and plugin as
 a start would (options, TLS files, credentials files, startup warnings), but
-binds no port and opens, reads or creates no log file; it prints
-`configuration ok` and exits 0, or the error and exits 1 (2 when a named file
-is missing, see [Exit Codes](#exit-codes)):
+binds no port, reads no stdin, and opens, reads or creates no log file; it
+prints `configuration ok: N pipeline(s)` and exits 0, or the error and exits 1
+(2 when a named file is missing, see [Exit Codes](#exit-codes)):
 
 ```bash
 lw --check -c /etc/logwisp/logwisp.toml
@@ -238,8 +307,10 @@ lw --check --source file,directory=/var/log/app --sink http,port=8080
 lw -c /etc/logwisp/logwisp.toml --logging.output=file
 ```
 
-Run under a supervisor (systemd, rc.d) rather than backgrounding it — there is
-no `--background` flag; earlier releases had one and it was removed. See
+Run it in the foreground under a supervisor (systemd, rc.d, a container
+runtime); there is no `--background` flag. A supervisor's stdin is usually
+`/dev/null`, where a console source ends at once, so a service's file defines
+its pipelines. See [Operations](operations.md#starting) and
 [Installation](installation.md).
 
 **Reload**

doc/configuration.md

@@ -41,9 +41,9 @@ runtime metadata and cannot be redirected by a `config_file` key inside the file
 Top-level keys, each as type and default:
 
 - `quiet` (bool, `false`): disable all application logging and console
-  diagnostics.
-- `status_reporter` (bool, `true`): emit a status report every 30 s at DEBUG
-  level.
+  diagnostics; pipeline output still flows.
+- `status_reporter` (bool, `true` with a configuration file, `false` without):
+  emit a status report every 30 s at DEBUG level.
 - `auto_reload` (bool, `false`): watch the config file and reload pipelines on
   change.
 
@@ -56,11 +56,12 @@ Note that `status_reporter` writes at DEBUG level, so it produces nothing unless
 ## Application Logging
 
 This configures LogWisp's own operational log, not the log data it transports.
+It goes to stderr by default, so stdout carries only a console sink's data.
 
 ```toml
 [logging]
-output = "stdout"      # file | stdout | stderr | split | all | none
-level  = "info"        # debug | info | warn | error
+output = "stderr"      # file | stdout | stderr | split | all | none
+level  = "info"        # debug | info | warn | error; warn without a file
 format = "txt"         # raw | txt | json
 # sanitization = ""    # raw | json | txt | shell
 
@@ -81,7 +82,15 @@ retention_hours   = 168.0
 - `all`: files plus split console.
 - `none`: no application logging.
 
-`[logging.file]` applies only to the `file` and `all` modes.
+`[logging.file]` applies only to the `file` and `all` modes. The `stdout`,
+`split` and `all` modes mix the log into a console sink's stdout.
+
+Without a configuration file (none named, none found) lw is a command-line
+tool: `level` defaults to `warn` and `status_reporter` to `false`. Explicit
+values from the environment or flags still win. A container configured only
+through variables or flags passes `LOGWISP_LOGGING_LEVEL=info` to log as a
+service; the image does not set it, since a variable would override a mounted
+file's value.
 
 > `[logging.console].target` is accepted and validated (`stdout`, `stderr`,
 > `split`) but **not applied**. The console destination is derived from

doc/formatters.md

@@ -11,8 +11,8 @@ flags            = 0
 timestamp_format = ""
 ```
 
-One formatter serves the whole pipeline. Sinks receive an identical payload;
-there is no per-sink formatting. When you need two shapes of the same data, run
+One formatter serves the whole pipeline. Sinks receive an identical payload,
+one record ending in exactly one newline; there is no per-sink formatting. When you need two shapes of the same data, run
 two pipelines, or chain to a node that formats differently.
 
 Omitting `[pipelines.flow.format]` entirely selects `raw`.
@@ -33,9 +33,10 @@ Each as type and default:
 ### raw
 
 Passthrough. `FlagRaw` bypasses formatting and sanitization: the message reaches
-the sink exactly as the source produced it, with no timestamp, level or source
-prefix added. An entry that also carries `fields` gets the fields JSON appended
-verbatim after a single space — `raw` never drops data and never re-encodes it.
+the sink as the source produced it, with no timestamp, level or source prefix
+added, and a newline appended unless the message ends in one. An entry that also
+carries `fields` gets the fields JSON appended verbatim after a single space —
+`raw` never drops data and never re-encodes it.
 
 ```toml
 [pipelines.flow.format]
@@ -47,8 +48,9 @@ its final form. Note that it also bypasses sanitization, so control characters
 in the source data reach your sinks intact.
 
 Byte-exact transport needs a source that does not split the line: the `console`
-source, or the `file` source with `raw = true`. Both put the whole line —
-newline included — in the message and leave `fields` empty. The `file` source's
+source, or the `file` source with `raw = true`. Both put the whole line, its
+terminator removed, in the message and leave `fields` empty; `raw` writes it
+back as one `\n`. The `file` source's
 JSON branch splits a line into message and fields, so `raw` reassembles it as
 `<msg> <fields>` rather than reproducing the original object.
 

doc/installation.md

@@ -52,9 +52,11 @@ Variables:
 - `SOURCE_DATE_EPOCH`: when set, the embedded build time comes from it, so
   package builds are reproducible.
 
-A plain `go build -o bin/lw ./cmd/lw` works too, and reports `dev` for the
-version metadata. `go install github.com/lixenwraith/logwisp/cmd/lw@latest`
-does not work until the module is renamed to its canonical path.
+A plain `go build -o bin/lw ./cmd/lw` works too, as does
+`go install github.com/lixenwraith/logwisp/cmd/lw@latest` (or `@vX.Y.Z`).
+Without the Makefile's `-ldflags`, `lw --version` reports what Go stamps into
+the binary: the module version, or a pseudo-version with the commit in a
+checkout.
 
 ## Installing
 
@@ -201,6 +203,12 @@ docker run -d --name logwisp \
   -e LOGWISP_SINK=console
   ```
 
+  Without a file lw logs warnings only and runs no status reporter; add
+  `-e LOGWISP_LOGGING_LEVEL=info` (and `-e LOGWISP_STATUS_REPORTER=true`) for
+  the service defaults. The image cannot set them: a variable would override
+  a mounted file's value. Without `-i`, stdin is `/dev/null`, so a container
+  needs a source other than `console`.
+
 - Secrets: only as mounted files, at `/run/secrets` (Docker or Compose
   secrets, a Kubernetes secret volume), named by `tls.key_file`,
   `auth.credentials_file` and `auth.password_file`. Never put a password in
@@ -237,10 +245,9 @@ The foundation for distribution packages is in place:
 - skeletons: `deploy/package/arch/PKGBUILD` and a `sysutils/logwisp` port in
   `deploy/package/freebsd/`
 
-Still missing: tagged, signed release tarballs; the module rename to
-`github.com/lixenwraith/logwisp`, which `go install` and FreeBSD's
-`go:modules` need; finishing and submitting the AUR package and the port;
-Debian packaging. [To Do](todo.md) has the steps.
+Still missing: release tags, which both skeletons download; finishing and
+submitting the AUR package and the port; Debian packaging.
+[To Do](todo.md) has the steps.
 
 ## Verification
 

doc/lw.1

@@ -1,4 +1,4 @@
-.Dd October 2, 2026
+.Dd October 3, 2026
 .Dt LW 1
 .Os
 .Sh NAME
@@ -36,14 +36,65 @@ forward to other nodes.
 Every network source and sink can use TLS, mutual TLS and
 Argon2id-SCRAM password authentication.
 .Pp
+With no configuration file and no pipeline options,
 .Nm
-stays in the foreground and logs to standard output; run it under a
-supervisor such as
+is a filter like
+.Xr cat 1 :
+it copies standard input to standard output line by line and exits 0 at the
+end of input.
+This built-in pipeline,
+.Dq pipe ,
+joins the console source
+.Dq stdin
+to the console sink
+.Dq stdout
+in the raw format, without a filter or rate limit; a configuration file that
+defines no pipelines runs it too.
+Each line is written with one newline, whether it ended in a newline, a
+carriage return and newline, or nothing; blank lines are skipped, and a line
+over 1 MiB continues in the next one.
+When standard input is a terminal,
+.Nm
+prints a reminder on standard error that Control-D ends the input.
+.Pp
+A pipeline finishes when all its sources have ended.
+Only a console source ends, at the end of standard input; every other
+source runs until shutdown.
+When every pipeline has finished,
+.Nm
+shuts down gracefully and exits 0.
+Console and file sinks first write what they hold; the
+.Cm http
+and
+.Cm tcp
+sinks do not yet, so a client can miss the last lines of a finite input.
+Under a supervisor, standard input is usually
+.Pa /dev/null :
+a console source there ends at once.
+.Pp
+The console source and sink never drop a line.
+A busy pipeline slows the reading of standard input; a console sink whose
+output is slow makes its pipeline wait, other sinks included.
+On a terminal the console sink writes control characters, escape sequences
+included, as the hex of their bytes in angle brackets, such as
+.Ql <1b> ,
+so that a log line cannot drive the terminal; pipes and files get the bytes
+unchanged.
+Its
+.Cm escape
+option sets this:
+.Cm auto
+.Pq the default ,
+.Cm always
+or
+.Cm never .
+.Pp
+.Nm
+stays in the foreground and logs to standard error; run a configuration as a
+service under a supervisor such as
 .Xr systemd 1
 or
 .Xr daemon 8 .
-With no configuration file it runs a demonstration pipeline: a synthetic
-generator writing JSON to standard output.
 .Pp
 The options are as follows:
 .Bl -tag -width Ds
@@ -59,23 +110,39 @@ A bare
 .Fl \- Ns Ar key
 sets a boolean to true.
 .It Fl \-quiet
-Suppress console output.
+Turn off the log of
+.Nm
+and the terminal reminder; console sinks still write.
 .It Fl \-status_reporter= Ns Ar bool
 Log pipeline statistics every 30 seconds at the debug level
-.Pq default true .
+.Pq default true with a configuration file, false without .
 .It Fl \-auto_reload= Ns Ar bool
 Reload when the configuration file changes
 .Pq default false .
 .It Fl \-logging.output= Ns Ar mode
 One of
-.Cm file , stdout , stderr , split , all
+.Cm stderr
+.Pq the default ,
+.Cm stdout ,
+.Cm split
+.Pq warnings and errors to standard error, the rest to standard output ,
+.Cm file ,
+.Cm all
+.Pq split and file
 or
 .Cm none .
+The modes that use standard output mix the log into what a console sink
+writes there.
 .It Fl \-logging.level= Ns Ar level
 One of
 .Cm debug , info , warn
 or
-.Cm error .
+.Cm error ;
+the default is
+.Cm info
+with a configuration file and
+.Cm warn
+without.
 .It Fl \-check
 Build every pipeline and plugin as a start would, loading options, TLS files
 and credentials files and printing startup warnings, without binding a port
@@ -97,6 +164,12 @@ the configuration file.
 They then replace the file's pipelines and the built-in default; every other
 key keeps its usual precedence.
 A reload rereads the file but keeps these pipelines.
+A pipeline without a source reads standard input with the console source
+.Dq stdin ;
+one without a sink writes standard output with the console sink
+.Dq stdout .
+Standard input has one reader: a second console source anywhere in the
+configuration fails validation.
 .Bl -tag -width Ds
 .It Fl \-pipeline Ar name
 Start a new pipeline.
@@ -265,7 +338,7 @@ The service's working directory on Linux and on
 .El
 .Sh EXIT STATUS
 .Nm
-exits 0 after a clean shutdown,
+exits 0 at the end of input, after a clean shutdown,
 .Fl \-version
 or help; 1 on a configuration, logger or startup error; and 2 when a
 configuration file named by
@@ -273,6 +346,15 @@ configuration file named by
 or
 .Ev LOGWISP_CONFIG_*
 does not exist.
+A reader of standard output that goes away, as in
+.Ql lw | head ,
+ends
+.Nm
+with
+.Dv SIGPIPE ,
+as it ends
+.Xr cat 1 ;
+the shell reports status 141.
 .Pp
 .Nm
 .Cm auth
@@ -287,23 +369,59 @@ every pipeline.
 An invalid configuration leaves the running service untouched.
 .El
 .Sh EXAMPLES
+Copy a file line by line:
+.Bd -literal -offset indent
+$ lw < app.log > copy.log
+.Ed
+.Pp
+Follow a log and keep only errors and warnings:
+.Bd -literal -offset indent
+$ tail \-F app.log | lw \-\-filter include,patterns=ERROR,patterns=WARN
+.Ed
+.Pp
+Prefix each line with the time, the level found in it and the source:
+.Bd -literal -offset indent
+$ tail \-F app.log | lw \-\-format txt
+.Ed
+.Pp
+Serve standard input as a live stream on the loopback address only, since
+the
+.Cm http
+sink listens on every address by default, and read it from another terminal
+as server-sent events; a client gets the lines that arrive while it is
+connected:
+.Bd -literal -offset indent
+$ journalctl \-f | lw \-\-sink http,host=127.0.0.1,port=8080
+$ curl \-N http://127.0.0.1:8080/stream
+.Ed
+.Pp
+Follow every
+.Pa .log
+file in a directory, as
+.Ql tail \-F
+does, and write the new lines unparsed to one file:
+.Bd -literal -offset indent
+$ lw \-\-source 'file,directory=/var/log/app,pattern=*.log,raw=true' \e
+    > all.log
+.Ed
+.Pp
+Check a configuration without binding a port or touching a log file:
+.Bd -literal -offset indent
+$ lw \-\-check \-c /etc/logwisp/logwisp.toml
+.Ed
+.Pp
 Run a configuration in the foreground with debug logging:
 .Bd -literal -offset indent
 $ lw \-c /etc/logwisp/logwisp.toml \-\-logging.level=debug
 .Ed
 .Pp
-Tail a directory and serve it as a live stream, without a file:
-.Bd -literal -offset indent
-$ lw \-\-source 'file,directory=/var/log/app,pattern=*.log' \e
-    \-\-sink http,host=127.0.0.1,port=8080
-.Ed
-.Pp
 Create a credentials file the service account can read, add a user,
 letting
 .Nm
 generate the password into a file, then reload the server:
 .Bd -literal -offset indent
-# install \-m 0600 \-o logwisp \-g logwisp /dev/null /etc/logwisp/users.toml
+# install \-m 0600 \-o logwisp \-g logwisp \e
+    /dev/null /etc/logwisp/users.toml
 # lw auth add-user \-credentials /etc/logwisp/users.toml \e
     \-user alice \-password-file /root/alice.pass
 # systemctl reload logwisp

doc/operations.md

@@ -5,17 +5,32 @@ Running, monitoring, and maintaining LogWisp.
 ## Starting
 
 ```bash
-# foreground, explicit config
-lw -c /etc/logwisp/logwisp.toml
+# a filter: stdin to stdout, exits 0 at the end of input
+lw < app.log > copy.log
+tail -F app.log | lw --filter include,patterns=ERROR
 
-# no config: built-in demo pipeline (random source -> stdout)
-lw
+# a service: the file's pipelines, until SIGINT or SIGTERM
+lw -c /etc/logwisp/logwisp.toml
 ```
 
+lw exits on its own only at the end of input: once every pipeline has
+finished, a pipeline finishing when all its sources have ended. Only a console
+source ends, when stdin does; file and network sources run until a signal, and
+lw shuts down gracefully, exit 0, either way.
+Console and file sinks write what is queued before the exit; the `http` and
+`tcp` sinks do not yet flush their clients' queues ([To Do](todo.md)). See
+[CLI](cli.md#built-in-defaults).
+
 There is no built-in daemon mode. Run LogWisp in the foreground under a
 supervisor — systemd, rc.d, or a container runtime — which is where restart,
 log capture, and resource limits belong. See [Installation](installation.md).
 
+A supervisor's stdin is `/dev/null` (systemd's default, `docker run` without
+`-i`), where a console source ends at once. So a service defines its
+pipelines: a file without `[[pipelines]]` runs the built-in stdin-to-stdout
+pipeline and exits 0 at once, which `Restart=on-failure` does not restart, and
+a container configured through variables needs a `LOGWISP_SOURCE`.
+
 **systemd**
 
 ```bash
@@ -64,6 +79,9 @@ What reload does *not* do:
   construction and `auto_reload` watches only the configuration file, so
   rotation requires `SIGHUP`.
 
+Standard input survives a reload: one reader serves the process, so the new
+console source continues where the old one stopped, and no line is read twice.
+
 Plan reloads on a busy relay the way you would plan a restart.
 Listener bind/start failures happen after the old service stops; these can leave
 the application without working pipelines until a corrected configuration reloads.
@@ -94,8 +112,9 @@ proves itself when the pipeline is actually built.
 
 ### Status reporter
 
-Enabled by default, every 30 seconds. It logs at **DEBUG**, so it produces
-nothing unless `logging.level = "debug"` — a common surprise.
+Enabled by default with a configuration file (off without one), every 30
+seconds. It logs at **DEBUG**, so it produces nothing unless
+`logging.level = "debug"` — a common surprise.
 
 ```toml
 status_reporter = true
@@ -153,10 +172,12 @@ query it with a client certificate or a token from `lw auth token`.
 
 Each counter, where it is reported, and what a rise means:
 
-- `dropped_entries` (source): downstream cannot keep up with the source
+- `dropped_entries` (source): downstream cannot keep up with the source; the
+  console source never drops, it reads stdin slower
 - `total_dropped` (flow): the rate limit or filters are discarding entries,
   often by intent
-- `total_dropped_by_sink` (pipeline): a sink's input queue is full
+- `total_dropped_by_sink` (pipeline): a sink's input queue is full; a console
+  sink waits instead, so its entries count only when shutdown cuts the wait
 - `dropped_writes` (`tcp` and `http` sinks): a client's queue overflowed,
   because it is too slow or one burst exceeded `client_buffer_size`
 - `rejected_conns` (`tcp` sink, `tcp_chain` source), `rejected_clients`
@@ -172,7 +193,11 @@ Each counter, where it is reported, and what a rise means:
 
 ## Log Management
 
-LogWisp's own operational log:
+LogWisp's own operational log goes to stderr by default, at `info` with a
+configuration file and at `warn` without one; explicit values win. A container
+configured only through variables or flags therefore logs at `warn`: pass
+`-e LOGWISP_LOGGING_LEVEL=info`. `lw-deploy.sh` always passes `-c`, so its
+nodes log at `info`. To files:
 
 ```toml
 [logging]
@@ -253,6 +278,14 @@ Walk the pipeline in order and read the counters: source `total_entries` (is
 anything being produced?), flow `total_dropped` (filters or rate limit?),
 pipeline `total_dropped_by_sink` (sink backed up?), sink `total_processed`.
 
+**A pipeline stalls behind a console sink**
+
+A console sink never drops: when its output is slow (a paused terminal, a full
+pipe, a stalled log collector) its pipeline waits, and the pipeline's other
+sinks with it. Stdin is read slower; other sources queue, then drop and count
+`dropped_entries`. Where a service's stdout may stall, give it a `file` sink
+instead. A closed stdout pipe ends lw with `SIGPIPE`.
+
 **File source reads nothing**
 
 - The watcher seeks to end-of-file on start; only content appended afterwards is

doc/security.md

@@ -853,3 +853,9 @@ Choose a sanitizer policy that matches the sink — `json` for JSON output,
 `txt` for files and consoles — so control characters in log data cannot break
 framing or inject terminal escapes downstream. See
 [Formatters](formatters.md).
+
+The `raw` format passes control characters through. A console sink writing to
+a terminal escapes them itself (`escape = "auto"`), so a log line cannot move
+the cursor, retitle the window, write the clipboard (OSC 52) or reorder text
+with bidi controls; pipes and files get the bytes unchanged. See
+[Sinks](sinks.md#console).

doc/sinks.md

@@ -16,7 +16,8 @@ sibling sinks are unaffected.
 
 ## console
 
-Writes formatted payloads to stdout or stderr.
+Writes formatted payloads to stdout or stderr. It is the sink of the built-in
+`pipe` pipeline, and of every spec pipeline given no `--sink`.
 
 ```toml
 [[pipelines.plugin_sinks]]
@@ -25,19 +26,35 @@ type = "console"
 [pipelines.plugin_sinks.config]
 target      = "stdout"
 buffer_size = 1000
+escape      = "auto"
 ```
 
 Options, each as type and default:
 
 - `target` (string, `stdout`): `stdout` or `stderr`.
 - `buffer_size` (int, `1000`): sink input queue depth.
+- `escape` (string, `auto`): write control characters as `<hex>` of their
+  bytes, `ESC` as `<1b>`: `auto` when the output is a terminal, `always`, or
+  `never`.
 
 > `split` is **not** a valid target for this sink and is rejected at startup.
 > Level-based splitting exists only for LogWisp's own application log
 > (`logging.output = "split"`).
 
-Payloads are written verbatim; the sink adds no framing. Whether entries are
-newline-terminated is decided by the formatter.
+Payloads are written as the formatter made them, one record per line.
+
+- Escaping: a log line must not drive the terminal that shows it (cursor and
+  screen control, title changes, OSC 52 clipboard writes, bidi spoofing). C0
+  and C1 controls other than tab, DEL, the bidi controls, the line and
+  paragraph separators and invalid UTF-8 are escaped; other invisible
+  characters, such as the joiners emoji and Indic scripts need, are kept.
+  `never` keeps an application's own ANSI colours; under `auto`, pipes and
+  files get the bytes unchanged.
+- Backpressure: the sink never drops. When its output is slow (a slow reader,
+  a paused terminal, a stalled log collector) the pipeline waits for it, with
+  its other sinks. A service whose stdout may stall writes to a `file` sink
+  instead. A closed stdout pipe ends lw with `SIGPIPE`, as it ends `cat`.
+- On stop it writes what is queued.
 
 ---
 
@@ -68,7 +85,8 @@ Options, each as type and default:
 - `max_total_size_mb` (int, `1000`): cap across all rotated files.
 - `min_disk_free_mb` (int, `0`): free-space floor before writing; `0` = none.
 - `retention_hours` (float, `168.0`): delete rotated files older than this.
-- `buffer_size` (int, `1000`): sink input queue depth.
+- `buffer_size` (int, `1000`): sink input queue depth, and the depth of the
+  internal writer's queue, which drops and counts when full.
 - `flush_interval_ms` (int, `100`): forced flush interval.
 
 > `min_disk_free_mb` has an unusual default. The constructor replaces only
@@ -76,7 +94,8 @@ Options, each as type and default:
 > free-space floor. Set it explicitly if you want one.
 
 The sink drives an internal writer configured for raw output with timestamps and
-levels disabled, so what lands on disk is exactly the formatted payload.
+levels disabled, so what lands on disk is exactly the formatted payload. On stop
+it hands the writer what is queued before closing the file.
 
 ---
 
@@ -187,7 +206,9 @@ Options, each as type and default:
   evicted by the broker. With the idle comment above, that reaches only a peer
   that has stopped accepting bytes on a sink configured `write_timeout_ms = 0`.
 - On shutdown, connected clients receive
-  `event: disconnect / data: {"reason":"server_shutdown"}`.
+  `event: disconnect / data: {"reason":"server_shutdown"}`. Entries still
+  queued for them are not flushed first, so a client can miss the last ones
+  of a finite input ([To Do](todo.md)).
 - HTTP/2 is negotiated via ALPN when TLS is enabled; plaintext is HTTP/1.1.
 
 **Status endpoint** returns service and version identity, host, port, TLS flag,
@@ -266,6 +287,8 @@ Options, each as type and default:
   whole timeout, so the client is disconnected immediately rather than retried.
 - A client whose send queue is full has that event dropped (`dropped_writes`)
   and stays connected.
+- On shutdown, entries still queued for clients are not flushed first, as
+  with the `http` sink.
 - With TLS enabled the handshake runs under a 10 s bound *after* the
   `max_connections` check, so concurrent handshakes are bounded too.
 - With an `auth` block, authorization runs after that handshake and *before*

doc/sources.md

@@ -84,7 +84,8 @@ last read time, plus `active_watchers`.
 
 ## console
 
-Reads newline-delimited entries from standard input.
+Reads newline-delimited entries from standard input. It is the source of the
+built-in `pipe` pipeline, and of every spec pipeline given no `--source`.
 
 ```toml
 [[pipelines.plugin_sources]]
@@ -98,9 +99,23 @@ Option, as type and default:
 
 - `buffer_size` (int, `1000`): subscriber channel depth.
 
-At most **one** instance per pipeline: the type is registered with
-`MaxInstances: 1`, and a second instance is rejected at pipeline construction.
-The level is inferred from the line text, and `Source` is set to `console`.
+Lines:
+- Each line is one entry, its terminator (`\n` or `\r\n`) removed; the
+  formatter writes one back. An unterminated last line is kept.
+- Blank lines are skipped.
+- A line over 1 MiB continues in the next entry; no byte is lost.
+- The level is inferred from the line text, and `Source` is set to `console`.
+
+Delivery and lifetime:
+- Nothing is dropped: the source waits for its pipeline, so a slow pipeline
+  slows the reading of stdin.
+- One reader serves the process. A reload's new source continues where the old
+  one stopped; no line is read twice. Hence at most **one** console source in
+  the whole configuration, checked at validation.
+- At the end of input the source ends, and with it a pipeline whose sources
+  have all ended; lw exits once every pipeline has
+  ([CLI](cli.md#built-in-defaults)). Under a supervisor whose stdin is
+  `/dev/null` that is at once.
 
 ---
 

doc/todo.md

@@ -122,7 +122,16 @@ connection level, below TLS, so a refused peer costs no handshake.
   - Denied and allowed clients; per-client SCRAM budgets behind one proxy
     address.
 
-## 2. Hardening review of `lixenwraith/config` and `lixenwraith/toml`
+## 2. Flush network sinks at shutdown
+
+At the end of a finite input lw shuts down, and the `http` and `tcp` sinks
+disconnect their clients without writing the entries still queued for them, so
+a client can miss the last ones. On `Stop`, each sink's broker should first move
+its input queue into the client queues, then let every client writer drain its
+queue within a bound (`write_timeout_ms`) before the disconnect frame. Verify
+with a finite stdin into each sink and a connected client counting lines.
+
+## 3. Hardening review of `lixenwraith/config` and `lixenwraith/toml`
 
 Both parse untrusted-shaped input (config files, environment, command line)
 and were not reviewed with the auth work. Review them as `auth` was reviewed:
@@ -151,52 +160,59 @@ the library repository, then a dependency bump here.
   - Fuzz targets committed in the library repositories.
   - A note in this repository's security.md once done.
 
-## 3. Packaging: AUR, FreeBSD ports, Debian
+## 4. Packaging: AUR, FreeBSD ports, Debian
 
 The foundation exists:
-- The `lw` binary name.
+- The `lw` binary name, free in Arch (official repositories and AUR), Ubuntu
+  24.04 and 26.04, and the FreeBSD 15.1 ports tree.
+- The canonical module path `github.com/lixenwraith/logwisp`, which
+  `go install`, FreeBSD's `USES=go:modules` and Debian's dh-golang expect.
 - `make install` with `DESTDIR`, `PREFIX` and `SYSCONFDIR`.
 - The `doc/lw.1` manual.
 - Service files in `deploy/package/`: systemd unit, sysusers, tmpfiles and the
   FreeBSD rc.d script.
 - Skeletons in `deploy/package/arch/` and `deploy/package/freebsd/`.
 
-What remains:
+What remains. Steps 1 to 3 are the maintainer's: they need push rights and an
+identity, and both skeletons download the tagged source.
 
-- Releases:
-  - The repository has no tags, so versions come from `git describe` and fall
-    back to a commit hash. Tag releases `vX.Y.Z`.
-  - Publish source tarballs with checksums, and sign them.
-- Module path:
-  - The module is `logwisp`, so `go install github.com/lixenwraith/logwisp/cmd/lw@latest`
-    does not work.
-  - FreeBSD's `USES=go:modules` and Debian's dh-golang expect the canonical
-    path.
-  - Rename the module to `github.com/lixenwraith/logwisp`: a mechanical
-    import rewrite in one commit.
-- Name check before submission:
-  - Arch: `pkgfile -s lw` against the official repositories, and the AUR
-    package list for `lw` and `logwisp`.
-  - Already checked: Ubuntu 26.04 and 24.04 contents and the FreeBSD 15.1
-    ports tree ship no `bin/lw`.
-- Arch (AUR): `deploy/package/arch/PKGBUILD` already has the source URL, the
-  Arch Go flags (`-trimpath -buildmode=pie -mod=readonly -modcacherw`),
-  `backup` for the config and `check()`. Remaining:
-  - the checksum of a tagged tarball in place of `SKIP`, and a maintainer;
-  - namcap, `.SRCINFO`, and publishing `logwisp` (optionally `logwisp-git`).
-- FreeBSD: `deploy/package/freebsd/` already has `USES=go:1.27,modules`,
-  `USE_RC_SUBR=logwisp`, `USERS`/`GROUPS`, and a pkg-plist with the `.sample`
-  config and the manual. Remaining:
-  - `distinfo` (or `GH_TUPLE`), which needs the module rename and a tag;
-  - a real `MAINTAINER` and a `UIDs`/`GIDs` entry for `logwisp`;
-  - poudriere on 14.x and 15.x, then submission through Bugzilla
-    (`sysutils/logwisp`).
-- Debian:
-  - Debian policy wants every Go dependency packaged; the four `lixenwraith`
-    libraries are not.
-  - Either package them too (dh-golang), or start with an `.deb` built by the
-    Makefile `install` target (nfpm or `dpkg-deb`) and a PPA, and move to the
-    archive later.
-- Shell completion for bash, zsh and fish, installed by `make install`.
-- A packaging CI job: build the AUR package in an Arch container and the port
-  in a FreeBSD VM, run `lw --version` and `make image-check`.
+1. Tag the release on the merged main commit, `vX.Y.Z`:
+   ```
+   git tag -a vX.Y.Z -m vX.Y.Z      # git tag -s signs it with your key
+   git push origin vX.Y.Z
+   ```
+   - `make` then stamps `vX.Y.Z` (from `git describe`), and
+     `go install github.com/lixenwraith/logwisp/cmd/lw@vX.Y.Z` works.
+   - A GitHub release for the tag is optional; the tag alone serves
+     `archive/vX.Y.Z.tar.gz`, which the PKGBUILD downloads.
+2. Arch (AUR), `deploy/package/arch/PKGBUILD`:
+   - `pkgver=X.Y.Z` and the `# Maintainer: Name <email>` line.
+   - `updpkgsums` (pacman-contrib) replaces `sha256sums=('SKIP')` with the
+     tarball's checksum.
+   - `namcap PKGBUILD`, then `makepkg -si` (builds, runs `check()`,
+     installs), then `makepkg --printsrcinfo > .SRCINFO`.
+   - Push `PKGBUILD` and `.SRCINFO` to
+     `ssh://aur@aur.archlinux.org/logwisp.git`; optionally a `logwisp-git`
+     package built from the main branch.
+3. FreeBSD, `deploy/package/freebsd/` (`sysutils/logwisp`):
+   - `DISTVERSION=X.Y.Z` and `MAINTAINER=` your address.
+   - `make makesum` writes `distinfo`: the module zip that the Go proxy
+     serves for `GO_MODULE` at the tag.
+   - The `logwisp` user needs an ID registered in the ports tree. Take a
+     number below 1000 that is free in both `/usr/ports/UIDs` and
+     `/usr/ports/GIDs`, and add to the same patch:
+     ```
+     UIDs: logwisp:*:NNN:NNN::0:0:LogWisp daemon:/nonexistent:/usr/sbin/nologin
+     GIDs: logwisp:*:NNN:
+     ```
+   - `portlint -AC`, `poudriere testport` on 14.x and 15.x jails, then a
+     Bugzilla report with the port directory and the UIDs/GIDs diff.
+4. Debian:
+   - Debian policy wants every Go dependency packaged; the four `lixenwraith`
+     libraries are not.
+   - Either package them too (dh-golang), or start with an `.deb` built by the
+     Makefile `install` target (nfpm or `dpkg-deb`) and a PPA, and move to the
+     archive later.
+5. Shell completion for bash, zsh and fish, installed by `make install`.
+6. A packaging CI job: build the AUR package in an Arch container and the port
+   in a FreeBSD VM, run `lw --version` and `make image-check`.