diff --git a/README.md b/README.md index 558bc1c..854dbab 100644 --- a/README.md +++ b/README.md @@ -22,8 +22,8 @@ streams, or downstream LogWisp nodes. - **A Unix filter out of the box**: with no configuration, `lw` copies stdin to stdout line for line and exits at the end of input; flags add filters, formats, sources and sinks -- **Presets**: `tail`, `serve`, `edge` and `aggregator` build the common - pipelines from a few keys (`lw --preset tail:path=/var/log/app`); +- **Presets**: `pipe`, `tail`, `serve`, `edge` and `aggregator` build the + common pipelines from a few keys (`lw --preset tail:path=/var/log/app`); `lw --dump` turns any command line into a configuration file - **Independent pipelines**, each `sources → flow → sinks`, running concurrently in one process @@ -53,11 +53,11 @@ configuration engine the website's configurator runs as WebAssembly: and `t` again reruns as a reload, keeping the running pipelines when the new ones do not build. Listeners bind, dialers connect and files are written for real; a password is asked in a masked dialog -- **Read any log**: the view parses LogWisp's JSON, lw's own log, pino, zap, - slog, docker, journald, ECS, GCP, logfmt and stamped text into time, level, - message and fields, filters them by level or pattern, and shows the payload - as the sinks receive it. It never slows a pipeline: what it misses is - counted +- **Read any log**: the view parses LogWisp's JSON, lw's own log, vif + records, pino, zap, slog, docker, journald, ECS, GCP, logfmt and stamped + text into time, level, message and fields, filters them by level or + pattern, and shows the payload as the sinks receive it. It never slows a + pipeline: what it misses is counted - **Any terminal**: `cmd | lw -T` taps piped input, since the screen draws on the controlling terminal; colors range from true color to none, and a text console gets CP437 glyphs @@ -105,13 +105,13 @@ exponential backoff and jitter. - Authorization by certificate identity: an `auth` block admits named peers (exact or RE2) rather than everything the CA issued, gates the `http` sink's stream and status endpoints, and lets a dialer pin the server it talks to -- Password authentication (Argon2id-SCRAM) on the same block: listeners hold - verifiers, never passwords, and do no KDF work; every login outside proxy - mode is bound to the listener's certificate, so no relay presenting another - one can use it; HTTP listeners issue short-lived bearer tokens. `lw auth` - manages credentials files and logs viewers in; behind a site's - TLS-terminating proxy, browsers log in through a shipped login page and a - dependency-free JS client +- Password authentication (Argon2id-SCRAM) on the same block: no login costs + a listener KDF work, and with a credentials file it holds verifiers, never + passwords; every login outside proxy mode is bound to the listener's + certificate, so no relay presenting another one can use it; HTTP listeners + issue short-lived bearer tokens. `lw auth` manages credentials files and + logs viewers in; behind a site's TLS-terminating proxy, browsers log in + through a shipped login page and a dependency-free JS client - Node binding: a chain source can label entries from the sender's certificate identity or username instead of from what the sender claims, so origin attribution is not forgeable @@ -127,7 +127,7 @@ See [Security](doc/security.md) for configuration and the exact boundary, and ## Documentation - [Installation](doc/installation.md): building, installing, services, - the container image, packaging + the container image, packaging, the end-to-end test scripts - [Deployment](doc/deployment.md): `deploy/lw-deploy.sh` for edges, aggregators, containers, services, jails - [Architecture](doc/architecture.md): component model, data flow, @@ -176,8 +176,9 @@ journalctl -f | ./bin/lw --sink http:host=127.0.0.1,port=8080 ./bin/lw --source 'file:directory=/var/log/myapp,pattern=*.log' \ --format json:sanitizer_policy=json > all.json -# the same with a preset; on a terminal the level names are in color -./bin/lw --preset tail:path=/var/log/myapp +# the same files with a preset, raw to stdout; on a terminal the level +# names are in color +./bin/lw --preset 'tail:path=/var/log/myapp/*.log' # compose on screen from a preset menu; r runs, t taps, o prints ./bin/lw -T @@ -235,8 +236,8 @@ Debian, Ubuntu and Arch. See [Installation](doc/installation.md). ## System Requirements - **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+) -- **Architecture**: amd64 -- **Go**: 1.27.2+ to build from source +- **Architecture**: amd64; arm64 builds but is untested +- **To build**: Go 1.27.2+, and GNU make or BSD make Network sources and sinks use IPv4 or IPv6, following the address family of the configured host. diff --git a/cmd/lw/main.go b/cmd/lw/main.go index 2d77626..751862d 100644 --- a/cmd/lw/main.go +++ b/cmd/lw/main.go @@ -1,3 +1,8 @@ +// Command lw is LogWisp: it runs the pipelines its file, LOGWISP_ variables +// and flags describe, rebuilt on SIGHUP, SIGUSR1 or an auto_reload, or +// composes them on screen with --tui. Its auth, tls, preset and config commands +// manage credentials and certificates, print a preset's pipeline as TOML and +// write the annotated configuration file. package main import ( diff --git a/config/logwisp.toml b/config/logwisp.toml index 6fc2e7a..b6101d9 100644 --- a/config/logwisp.toml +++ b/config/logwisp.toml @@ -30,7 +30,7 @@ # quiet = false # No log of lw's own; pipeline output still flows # status_reporter = true # Periodic status logging (30s, DEBUG level); false without a file # auto_reload = false # Watch file; read at startup (signals always reread it) -# color = "auto" # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR unset +# color = "auto" # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR empty or unset, TERM not dumb ############################################################################### ### Logging (lw's own log) @@ -61,7 +61,7 @@ ############################################################################### # [[pipelines]] -# name = "default" # example +# name = "default" # Required (example) ###============================================================================ ### Flow: the stages between sources and sinks, validated whether on or off @@ -83,7 +83,7 @@ # [pipelines.flow.format] # type = "raw" # raw|txt|json -# flags = 0 # Formatter flags; 0 = by type +# flags = 0 # For experts, a sum of bits: 1 messages verbatim, 16 no time, 32 no level; 0 = by type # timestamp_format = "" # Go time layout; empty = RFC 3339 with nanoseconds # sanitizer_policy = "" # raw|json|txt|shell; empty = by type @@ -146,7 +146,7 @@ ### certificate to the user (needs tls.client_auth) ### allow mtls: exact identities. Empty allow AND ### allow_patterns admits any identity the CA vouches -### for (logged WARN) +### for (a listener logs a WARN) ### allow_patterns mtls: RE2 patterns; anchor them yourself (^...$) ### node_binding chain sources: binds the node label to the identity ### or username, overriding trust_node; unset = force @@ -205,7 +205,8 @@ # pattern = "*" # Glob of the whole file name (* and ? only) # check_interval_ms = 100 # Directory rescan, at least 10; open files are polled every 100 ms ## raw = true never parses a line; with format type "raw" each line is relayed -## as written, except that empty lines are skipped and CRLF becomes LF. +## as written, except that empty lines are skipped, CRLF becomes LF and a line +## over 1 MiB, or one still being written when read, is split. # raw = false # from = "end" # end|start: where a newly found file is read from @@ -300,7 +301,7 @@ # node_binding = "" # credentials_file = "" # scram: written by `lw auth add-user` # token_lifetime_ms = 0 -# trusted_proxies = [] # the http sink only +# trusted_proxies = [] # refused here: the http sink only # username = "" # password_file = "" # [pipelines.plugin_sources.config.acl] @@ -380,7 +381,6 @@ # identity = "" # allow = [] # allow_patterns = [] -# node_binding = "" # chain sources only # credentials_file = "" # scram: log in with `lw auth token` # token_lifetime_ms = 0 # trusted_proxies = [] # scram: proxies ending the browsers' TLS; tls may then be off @@ -423,7 +423,6 @@ # identity = "" # allow = [] # allow_patterns = [] -# node_binding = "" # chain sources only # credentials_file = "" # scram: readers use `lw auth stream` # username = "" # password_file = "" @@ -443,7 +442,7 @@ # [pipelines.plugin_sinks.config] # host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1" # port = 9440 # Required (example) -# node = "" # Origin label, kept across hops; empty = the host name +# node = "" # Origin label, kept across hops; empty = this machine's host name # buffer_size = 1000 # dial_timeout_ms = 5000 # write_timeout_ms = 5000 @@ -476,7 +475,7 @@ # host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1" # port = 9441 # Required (example) # ingest_path = "/ingest" # Must start with "/"; not "/auth" -# node = "" # Origin label, kept across hops; empty = the host name +# node = "" # Origin label, kept across hops; empty = this machine's host name # buffer_size = 1000 # max_batch_count = 100 # max_batch_bytes = 1048576 diff --git a/deploy/README.md b/deploy/README.md index 11865e1..9388c1d 100644 --- a/deploy/README.md +++ b/deploy/README.md @@ -13,3 +13,8 @@ deploy/lw-deploy.sh --yes ... # no questions: defaults, or fail on a miss ``` Roles, runtimes, scenarios and what is not covered: [doc/deployment.md](../doc/deployment.md). + +`package/` holds the service files (`logwisp.service`, `.sysusers`, +`.tmpfiles`, `logwisp.rc`) that `make install` and this script install, the +shell completions, and the deb, Arch and FreeBSD packaging: +[doc/installation.md](../doc/installation.md#installing). diff --git a/deploy/lw-deploy.sh b/deploy/lw-deploy.sh index 03b9c08..7a2e925 100755 --- a/deploy/lw-deploy.sh +++ b/deploy/lw-deploy.sh @@ -105,7 +105,9 @@ Containers (Linux) publishes nothing and listens on --listen Native services - --bin FILE lw binary to install (default: the installed one, else bin/lw) + --bin FILE lw binary to install (default: the installed one, else + bin/lw); manual: the lw that adds users and checks + (default: lw on PATH, else bin/lw) --jail NAME FreeBSD: install into this running jail through jexec --package-dir DIR service files (default deploy/package) diff --git a/doc/README.md b/doc/README.md index 72ea60c..a98669e 100644 --- a/doc/README.md +++ b/doc/README.md @@ -4,7 +4,7 @@ The [README](../README.md) has LogWisp's features, a quick start and the system requirements. - [Installation](installation.md): building, installing, services, - the container image, packaging + the container image, packaging, the end-to-end test scripts - [Deployment](deployment.md): `deploy/lw-deploy.sh` for edges, aggregators, containers, services, jails - [Architecture](architecture.md): component model, data flow, diff --git a/doc/architecture.md b/doc/architecture.md index 0f9a1e4..6dc9cc0 100644 --- a/doc/architecture.md +++ b/doc/architecture.md @@ -45,18 +45,23 @@ Package map: - `internal/filter`: regex filter and filter chain. - `internal/format`: adapter over the `lixenwraith/log` formatter and sanitizer. -- `internal/source/*`, `internal/sink/*`: the source and sink plugins. +- `internal/source`, `internal/sink`: the `Source` and `Sink` interfaces and + their stats, and the stop helpers network sinks share (`sink.FlushBound`, + `FlushContext`, `Drain`); their subpackages are the plugins. - `internal/plugin`: global factory registry populated by plugin `init()`. - `internal/chain`: chain wire protocol: hello preamble, entry codec, backoff. -- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`. +- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`; also + the certificates `lw tls` writes and a listener makes at startup, pins and + startup warnings. - `internal/authz`: the single seam between `AuthOptions` and the network plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`; `Authorize` refuses under `scram`. - `internal/netacl`: a listener's `acl` block: address rules, PROXY headers and per-client limits; its `Table` is SCRAM throttling's limiter too. - `internal/session`: session manager and per-instance proxy. -- `internal/core`: shared types (`LogEntry`, `TransportEvent`), capabilities, - constants. +- `internal/core`: shared types (`LogEntry`, `TransportEvent`, the viewer's + `Tap`), the level table `core.Levels`, capabilities, constants, and + `Network` and `Listen`, which keep a socket to its host's address family. - `internal/tokenbucket`: rate limiter primitive. - `internal/tui`: `lw --tui`, the diagram and the live view of what `t` runs; `cmd/lw/screen.go` runs the pipelines for it. @@ -65,6 +70,8 @@ Package map: terminal acts on, which the console sink shares. - `internal/version`: version, commit and build time, stamped at build. - `internal/testutil`: fixtures for tests only. +- `config`: embeds `logwisp.toml`, the annotated default configuration that + `lw config init` writes and the packages install. ## Plugin Registration @@ -98,6 +105,11 @@ instance of a single type fails, naming both pipelines. 4. **Dispatch** sends the event to every sink's input channel with a non-blocking send, except to a sink declaring `CapBackpressure`. +A heartbeat joins at step 4: the flow writes it in its format with a +structured `Entry`, and the pipeline offers it to the tap and dispatches it, +so no rate limit or filter drops it and `flow.total_processed` does not count +it. + `LogEntry` fields: - `Time`: entry timestamp. @@ -161,7 +173,10 @@ shutdown (`dropped_batches`). `Pipeline.Stop` is ordered so in-flight data drains: 1. Release dispatch waits on backpressure sinks; stop all sources - concurrently, each closing its subscriber channels. + concurrently, each closing its subscriber channels. From the release on, + dispatch picks at random between a backpressure sink's input and the + release, so it drops about half of the entries still in flight for one + (`total_dropped_by_sink`) even with room. 2. Wait for the run loop, which ends when every subscription channel closes. 3. Stop all sinks concurrently: console and file sinks write their queue; network sinks deliver theirs within `sink.FlushBound`, their write or @@ -197,11 +212,13 @@ TLS is built in exactly one place, `internal/tlsx`, which exposes ## Sessions Each pipeline owns a `session.Manager`. Plugins receive a `session.Proxy` -scoped to their instance id, so one plugin cannot see or remove another's -sessions. A session records the remote address, creation and last-activity -timestamps, and metadata — including `tls` and `tls_peer_cn` for TLS peers, -`auth_method` / `auth_identity` for authorized ones, and `peer_addr`, the proxy, -for a client a PROXY header names. +scoped to their id, so a plugin sees and removes only the sessions under its +id; ids are unique per list, so a source and a sink that share one (the +built-in pipe's two `console`s) share that scope. A session records the +remote address, creation and last-activity timestamps, and metadata — +including `tls` and `tls_peer_cn` for TLS peers, `auth_method` / +`auth_identity` for authorized ones, and `peer_addr`, the proxy, for a client +a PROXY header names. Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The HTTP sink's broker treats a vanished session as an eviction signal and closes @@ -277,7 +294,10 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key", it: answer it in `dirs` and call again. - Pipelines are a JSON list in the file's shape (`name`, `flow`, `plugin_sources`, `plugin_sinks`); a misspelled key is refused. A node is - `{role, id, index}`: a source's or sink's id, a filter's index. + `{role, id, index}`: `source` or `sink` by its id, `filters` by its index, + or the flow stage `rate_limit`, `format` or `heartbeat`, which `set` turns + on and `remove` off; `add` takes `source`, `sink`, or `filters` with the + type `include` or `exclude`. ## Resource Management @@ -285,18 +305,25 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key", load. - `max_connections` caps connections on the `tcp` sink and the `tcp_chain` source, and streams on the `http` sink. Admission is a load-then-check, so a - burst can over-admit by roughly one. + burst can over-admit by the connections that arrive before the count rises. +- A listener's `acl` block caps each client: `max_connections_per_client`, + and on the `http` sink and `http_chain` source + `requests_per_second_per_client` ([Security](security.md#per-client-limits)). - Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an oversized line terminates a `tcp_chain` link and fails an `http_chain` batch with `400`. - The `http_chain` source caps each request body at `max_body_bytes`. -- File sinks rotate on size, cap total rotated size, and honour a retention - window. +- File sinks rotate on size, cap total rotated size, honour a retention + window, and with `min_disk_free_mb` delete the oldest files to keep that + much disk free. ## Performance Notes -- In-memory entry processing is sub-millisecond; the formatter mutex is the - only shared serialization point in the hot path. +- In-memory entry processing is sub-millisecond. Besides the formatter's + mutex, an entry can take the rate limiter's bucket lock (`policy = "drop"`) + and the pipeline's session-manager lock, which the network plugins and the + console and random sources take as entries pass, to refresh session + activity. - File tailing detects new content within roughly 100 ms (fixed poll), while `check_interval_ms` governs how quickly a *newly created* file is noticed. - `http_chain` trades latency for efficiency: entries wait up to diff --git a/doc/chaining.md b/doc/chaining.md index 1cef5e4..a5684a7 100644 --- a/doc/chaining.md +++ b/doc/chaining.md @@ -20,7 +20,7 @@ nodes can filter and reformat as if the entries were local. ┌───────────────┐ │ http_chain sink ───┼──► upstream collector │ file source │ HTTP/TLS │ │ │ ↓ ├────────────►│ http_chain source │ - │ http_chain sink│ :15802 └────────────────────┘ + │http_chain sink│ :15802 └────────────────────┘ └───────────────┘ ``` @@ -44,7 +44,7 @@ Chained entries carry a `node` label identifying where they originated. Relays preserve `node`, so a label survives any number of hops and identifies the original producer, unless a hop's source overwrites it: `trust_node = -false`, or `node_binding = "force"` (the default with an `auth` block). +false`, or `node_binding = "force"` (the default under `mtls` or `scram`). Formatters write the label as a prefix on the source ([Formatters](formatters.md#node-identity-in-output)). @@ -55,9 +55,10 @@ under `scram`. `auth.node_binding`: - `none`: `trust_node` governs the connection label and the per-entry `node` field. - `assert`: the connection label must equal the identity, or the peer is - rejected; `trust_node` governs the per-entry `node` field. -- `force` (default with an `auth` block): the identity is the connection label - and overwrites every per-entry `node` field. + rejected; `trust_node` governs the per-entry `node` field, the label standing + in wherever it would put the remote address. +- `force` (default under `mtls` or `scram`): the identity is the connection + label and overwrites every per-entry `node` field. Pick `force` at an ingest boundary you do not trust — it is the only setting where a compromised edge cannot mislabel its entries, including through the @@ -87,17 +88,23 @@ A persistent connection carrying newline-delimited JSON. {"logwisp":1,"node":"edge-01"} ``` -3. The listener reads that line within `hello_timeout_ms` and rejects the - connection if it is malformed or declares a different protocol version. +3. The listener reads that line, at most 4 KiB, within `hello_timeout_ms` and + rejects the connection if it is malformed or declares a different protocol + version. 4. Every subsequent line is one JSON-encoded `LogEntry`. Line size is bounded at 1 MiB. An oversized line is a protocol violation and terminates the connection, because the scanner cannot resynchronize afterwards. The cap applies to the encoded line, envelope and JSON escapes included, so a message near 1 MiB exceeds it (the console and file sources cut lines at -1 MiB). On `tcp_chain` such an entry ends the link and is lost, together with -any line the sink writes before it notices the close. On `http_chain` it fails -its whole batch with `400`, which the sink drops. +1 MiB). On `tcp_chain` such an entry ends the link. If the socket took the +whole line, the entry is lost, together with any line the sink writes before it +notices the close. If the line overruns the cap by more than the socket buffers +hold, as a message of control characters can (most escape to six bytes), the +write fails and the sink resends the line on every reconnect: the link stalls +until a stop or reload, and later entries count in `total_dropped_by_sink`. The +sink does not yet drop a line the source must refuse. On `http_chain` such an +entry fails its whole batch with `400`, which the sink drops. Under `auth.type = "scram"` the hello also carries the SCRAM client-first message, and three more lines precede the entries, each one JSON object of at @@ -119,18 +126,20 @@ binary ignores it and the dialer gives up after 10 s. ### HTTP transport -Batches of NDJSON delivered by `POST`, with the preamble expressed as headers. +Batches of NDJSON delivered by `POST` to `ingest_path` (default `/ingest`), with +the preamble expressed as headers. - Request headers - `X-Logwisp-Protocol`: the protocol version; must be `1`. - - `X-Logwisp-Node`: the origin node label. + - `X-Logwisp-Node`: the sending sink's `node`, the connection label, as in the + TCP hello. - `Content-Type`: `application/x-ndjson`. - `Authorization`: `Bearer `, under `scram`. - Response header - `X-Logwisp-Accepted`: the number of entries ingested. -Responses: `204` on success; `400` for a bad protocol version or a malformed -body, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for +Responses: `204` on success; `400` for a bad protocol version or a transfer +error, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for another path; `405` for a non-`POST` method; `429` (with `Retry-After`) past `acl.requests_per_second_per_client`; `401` for a missing, invalid or expired token; `403` for a refused certificate or node label. @@ -157,7 +166,8 @@ oversized body. ``` `node`, `level`, and `fields` are omitted when empty. A missing `time` is filled -in at ingest. +in at ingest. A line that does not decode is skipped and counted in +`parse_errors`; the connection or batch stands. ## Delivery Semantics diff --git a/doc/cli.md b/doc/cli.md index 9fd22c3..76e5cbe 100644 --- a/doc/cli.md +++ b/doc/cli.md @@ -74,23 +74,24 @@ pipelines have [their own flags](#pipelines): - `--schema`: print every setting, pipeline flag, flow stage, plugin option, preset and output form as one JSON document, and exit; tools build their option lists from it: - - a key: its kind, default, rules and help; a table (`tls`, `auth`, `acl`) - holds its own keys, those of the plugin's side (listener or dialer), - HTTP's only on a plugin that speaks HTTP. Every listener lists - `auth.node_binding`, which only chain sources use, and every HTTP - listener `auth.trusted_proxies`, which only the `http` sink admits + - a key: its kind, default, rules and help, and a setting's `variable`; a + table (`tls`, `auth`, `acl`) holds its own keys, those of the plugin's + side (listener, chain source or dialer), HTTP's only on a plugin that + speaks HTTP: `auth.node_binding` only on the chain sources, and + `auth.trusted_proxies` on both HTTP listeners, though only the `http` + sink admits it - a flow stage: its keys, and `on`, the values that turn it on - a preset key: its default and help, and the choices (`enum`) and `hint` of the option it sets - `forms`: `command`, `environment` and `file`, each with its hint - `--color [WHEN]`, WHEN `auto`, `always` or `never`: level names in color on - console sinks, default `auto` (a terminal, `NO_COLOR` unset, `TERM` not - `dumb`); bare, `always`. A console sink's own `color` wins + console sinks, default `auto` (a terminal, `NO_COLOR` empty or unset, + `TERM` not `dumb`); bare, `always`. A console sink's own `color` wins ([console sink](sinks.md#console)) - `-q`, `--quiet`: silence lw's own log and notices, default `false`; pipeline output still flows -- `--status_reporter=BOOL`: periodic status logging, default `true` with a - configuration file, `false` without +- `--status_reporter=BOOL`: pipeline statistics every 30 s, logged at + `debug`, default `true` with a configuration file, `false` without - `--auto_reload=BOOL`: reload when the configuration file changes, default `false` - `-V`, `--version`: print the version and exit @@ -105,12 +106,15 @@ The last `-c` wins; a missing or empty path is an error. default `stderr`, so stdout carries only data - `--logging.level`: `debug`, `info`, `warn` or `error`, default `info` with a configuration file, `warn` without -- `--logging.format`: `raw`, `txt` or `json` -- `--logging.sanitization`: `raw`, `json`, `txt` or `shell` -- `--logging.file.directory`: a path -- `--logging.file.name`: a string -- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers -- `--logging.file.retention_hours`: a float +- `--logging.format`: `raw`, `txt` or `json`, default `txt` +- `--logging.sanitization`: `raw`, `json`, `txt` or `shell`, default `raw` +- `--logging.file.directory`: a path, default `./log`; the limits below + delete any `.log` file in it + ([Application Logging](configuration.md#application-logging)) +- `--logging.file.name`: a string, default `logwisp` +- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers, + default `100` and `1000` +- `--logging.file.retention_hours`: a float, default `168`; `0` keeps files `--logging.console.target` is accepted but has no effect; the console destination is derived from `--logging.output`. @@ -308,8 +312,8 @@ Any pipeline flag on the command line makes LogWisp ignore all of them. A container needs no configuration file. Without one lw logs less ([Built-in Defaults](#built-in-defaults)), so pass `LOGWISP_LOGGING_LEVEL=info` -and `LOGWISP_STATUS_REPORTER=true`; the image sets neither, as a variable would -override a mounted file's value: +(`debug`, with `LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the +image sets neither, as a variable would override a mounted file's value: ```bash docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \ @@ -318,7 +322,7 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \ -e LOGWISP_FILTER='exclude:patterns=DEBUG' \ -e LOGWISP_SINK='http:host=0.0.0.0,port=8080' \ -e LOGWISP_SINK_1=console \ - -e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true \ + -e LOGWISP_LOGGING_LEVEL=info \ logwisp:dev ``` @@ -335,8 +339,9 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \ ## Exit Codes -- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version` / - `--help`, `--dump`, or a valid `--check` +- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version`, + `--help`, `--schema`, `--dump`, a valid `--check`, or a `--tui` screen + closed without `r` - `1`: general error: a configuration load or validation failure (`--check` included, and a bad value on the command line), a logger init failure, a service bootstrap failure @@ -360,7 +365,7 @@ lw < app.log > copy.log - Each line of stdin is one entry and one line of stdout. - The terminator (`\n` or `\r\n`) is written back as `\n`; an unterminated last line is kept and terminated. - - Blank lines are skipped. + - Empty lines are skipped; a line of spaces is an entry. - A line over 1 MiB continues in the next entry; no byte is lost. - Nothing is dropped: a slow reader slows the reading of stdin. - On a terminal, control characters are written as `` (`ESC` becomes @@ -504,16 +509,16 @@ WebAssembly; `t` runs them under the screen and shows what flows. - The bar counts the shown pipeline's entries in, filtered by the flow, dropped by sinks, and missed by the screen: the view never slows a pipeline. -- Live view: from 40 rows it sits under the diagram; shorter, it takes the - screen while it has the keys. `t` or Esc gives them back to the diagram - (Esc first closes an open detail). - - The canvas cursor picks its point, so the view opens on the first - source: a source shows its entries, a flow stage (or the filter the - inspector is on) what reaches it, the heartbeat its own, the sinks (or - no part) what they got. `v` switches between all (what the flow dropped - dimmed), what passed and what was dropped there, with no restart; the - title says which. lw's own log shows from warn up at every point; `s` - also offers it alone, debug too. +- Live view: from 40 rows it sits under the diagram, unless the inspector is + open below the canvas; otherwise it takes the screen while it has the keys. + `t` or Esc gives them back to the diagram (Esc first closes an open detail). + - The canvas cursor picks its point, the first source until it moves: a + source shows its entries, a flow stage (or the filter the inspector is + on) what reaches it, the heartbeat its own, the sinks (or no part) what + the flow offered them (a full sink drops its copy alone). `v` switches + between all (what the flow dropped dimmed), what passed and what was + dropped there, with no restart; the title says which. lw's own log shows + from warn up at every point; `s` also offers it alone, debug too. - A row is the entry's time, level, tag, message and fields, read from any line shape: LogWisp's json, lw's own log, vif records, pino, zap, slog, docker, journald, ECS, GCP, logfmt, or text after an RFC3339 stamp and diff --git a/doc/configuration.md b/doc/configuration.md index ae00ef2..3c2564b 100644 --- a/doc/configuration.md +++ b/doc/configuration.md @@ -1,7 +1,8 @@ # Configuration Reference LogWisp is configured with TOML. A complete annotated file listing every option -and its default ships as [`config/logwisp.toml`](../config/logwisp.toml). +and its default ships as [`config/logwisp.toml`](../config/logwisp.toml); +`lw config init` writes it ([CLI](cli.md#lw-config)). ## Configuration Precedence @@ -45,8 +46,8 @@ Top-level keys, each as type and default: - `auto_reload` (bool, `false`): watch the config file and reload pipelines on change. - `color` (string, `"auto"`): level names in color on console sinks, which may - set their own: `"auto"` (a terminal, `NO_COLOR` unset, `TERM` not `dumb`), - `"always"` or `"never"`; see the [console sink](sinks.md#console). + set their own: `"auto"` (a terminal, `NO_COLOR` empty or unset, `TERM` not + `dumb`), `"always"` or `"never"`; see the [console sink](sinks.md#console). `--version`, `--check`, `--dump`, `--schema` and `--tui` are switches, not settings; the matching variables are ignored. @@ -80,8 +81,9 @@ retention_hours = 168.0 - `all`: files plus split console. - `none`: no application logging. -`[logging.file]` applies only to the `file` and `all` modes. Its limits delete -any `.log` file in `directory`, lw's or not: give lw a directory of its own. +`[logging.file]` applies only to the `file` and `all` modes. Its limits, and a +floor of 10 MB free on the disk that no key changes, delete any `.log` file in +`directory`, lw's or not: give lw a directory of its own. `retention_hours = 0` keeps files. The `stdout`, `split` and `all` modes mix the log into a console sink's stdout. `[logging.console].target` is validated (`stdout`, `stderr`, `split`) but not applied: `logging.output` chooses the @@ -220,11 +222,12 @@ pipelines[0].plugin_sinks[http].config: unknown key "tls.enabeld" - `auth` enums (`identity`, `node_binding`) are checked whatever `type` is; a block naming peers or credentials with `type` `none` or unset is refused ([Security](security.md#validation-1)). -- Once the rest is valid, the file itself: the first key the schema does not - declare outside a plugin's `config` is refused (`config file "…": unknown - key "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path - would drop the whole table it heads. A top-level `config_file`, `version`, - `check` or `dump` is ignored. +- The file itself, after the rest at start and by `lw --check` but before it + at a reload: the first key the schema does not declare outside a plugin's + `config` is refused (`config file "…": unknown key + "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path would + drop the whole table it heads. A top-level `config_file`, `version`, `check` + or `dump` is ignored. - Files a plugin names (certificates, keys, CA files, credentials, password files) are read when it is built: at start, at reload and by `lw --check`. Those failures name the pipeline and plugin. @@ -234,14 +237,15 @@ exits 1 ([Networking](networking.md#network-plugins)). ## Hot Reload -`auto_reload = true` watches the configuration file. `SIGHUP` or `SIGUSR1` -rereads it even when watching is off and always rebuilds, so certificates and -credentials rotate without a TOML edit. CLI and environment overrides, -pipelines included, are captured at startup and keep their precedence. A -discovered default file that does not exist is not an error, so an instance -configured by flags or variables alone still rebuilds; once read, though, its -removal fails the reload and the running service stays as it is; restart to -run without the file. +`auto_reload = true` watches the configuration file: lw reads it every second +and applies a change once two reads in a row agree, so within about 2 s. +`SIGHUP` or `SIGUSR1` rereads it even when watching is off and always +rebuilds, so certificates and credentials rotate without a TOML edit. CLI and +environment overrides, pipelines included, are captured at startup and keep +their precedence. A discovered default file that does not exist is not an +error, so an instance configured by flags or variables alone still rebuilds; +once read, though, its removal fails the reload and the running service stays +as it is; restart to run without the file. Reload builds a new service from a detached snapshot of the configuration, validated again, and shuts the old one down only if that succeeds: parse, @@ -271,6 +275,13 @@ Each TOML type, its Go type, and its command-line / environment form: - Boolean (`bool`): `true` / `false`, or a bare flag for `true`. - Array (`[]T`): only in [pipeline specs](cli.md#pipelines), as a repeated key. - Table (struct): nested path with `.` (flags) or `_` (environment). +- Path (`string`: a plugin's `directory` and `*_file` keys, + `logging.file.directory`): `~` alone or a leading `~/` is the home directory + where lw opens it (`--dump` prints it as typed); `~user` stays as typed; a + relative path starts at lw's working directory. The configuration file's own + path (`-c`, `LOGWISP_CONFIG_*`) is read as typed. Integer keys refuse fractions and out-of-range values; number keys refuse inf -and NaN. Configuration files are TOML only. +and NaN. A suffix names a key's unit: `_ms` milliseconds, `_mb` megabytes of +1,000,000 bytes, `_bytes` bytes, `_hours` hours. Configuration files are TOML +only. diff --git a/doc/deployment.md b/doc/deployment.md index 5085bda..a94f33f 100644 --- a/doc/deployment.md +++ b/doc/deployment.md @@ -10,7 +10,9 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)). ## Roles - `edge` - - tails each `--log-dir` (not recursive) for `--log-pattern` (default `*.log`) + - tails each `--log-dir` (not recursive) for `--log-pattern` (default + `*.log`), a newly found file from its end, or with `--from start` from its + first line - forwards to `--aggregator` over `tcp_chain` (port 9440) or, with `--chain http`, `http_chain` (port 9441) - labels its entries `--node`: the short host name, or the jail name @@ -27,15 +29,16 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)). - `docker`, `podman` (Linux) - the container runs `--read-only --cap-drop ALL --security-opt no-new-privileges -u 65532:65532 --restart unless-stopped` - - `--config-dir` (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`, - and the credentials files, owned by 65532 and mounted read-only at - `/etc/logwisp` + - `--name` (default `logwisp-ROLE`) names the container; `--config-dir` + (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`, and the + credentials files, owned by 65532 and mounted read-only at `/etc/logwisp` - log directories mount read-only and the output directory writable, each at its host path; `--log-group` adds that group's id to the container - listener ports publish on `--listen`, an IP address; the process inside binds `0.0.0.0` - `--network NET` joins a user network, created when missing, where edges - reach an aggregator by container name; `host` publishes nothing + reach an aggregator by container name; `host` publishes nothing, and the + process binds `--listen` itself - `--image` defaults to `logwisp:dev`, the `make image` tag; `--build` builds it from the repository. Behind an HTTPS-intercepting proxy add `--build-ca FILE` and, for a proxy on loopback, @@ -47,10 +50,11 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)). script does not relabel, since `:z` on a shared directory such as `/var/log` would relabel it for every other service too - `native`, Linux - - needs systemd; installs `lw` to `/usr/local/bin` unless one is installed - (`--bin FILE`, or `--build` for `make build`), and - `deploy/package/logwisp.service`, `logwisp.sysusers` and - `logwisp.tmpfiles` under `/etc` + - needs systemd; installs `deploy/package/logwisp.service`, + `logwisp.sysusers` and `logwisp.tmpfiles` under `/etc`, and `--bin FILE` + or the repository's `bin/lw` (`--build` runs `make build` first) to + `/usr/local/bin`, unless neither flag is given and an `lw` is in + `/usr/local/bin` or `/usr/bin` - a package's unit (`/usr/lib/systemd/system/logwisp.service`) is used as it is, with the `lw` its `ExecStart` runs; `--bin` and `--build` then stop the run, since the service would never run that binary @@ -58,14 +62,17 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)). - refuses log and output directories under `/tmp` and `/var/tmp`: the unit's `PrivateTmp=yes` gives the service its own, empty ones - adds `logwisp.service.d/deploy.conf` when the node needs it: - `ReadWritePaths` for an output outside `/var/log/logwisp`, - `ProtectHome=read-only` for logs or output under `/home`, + `ReadWritePaths` for an output other than `/var/log/logwisp` or + `/var/lib/logwisp`, `ProtectHome=read-only` for logs or output under + `/home`, `/root` or `/run/user`, `SupplementaryGroups` for `--log-group`, `CAP_NET_BIND_SERVICE` for a port below 1024 - `native`, FreeBSD - - installs `deploy/package/logwisp.rc` as `/usr/local/etc/rc.d/logwisp`, - creates the `logwisp` user with `pw`, enables it with `sysrc`; LogWisp logs - to syslog under the tag `logwisp` + - installs `lw` to `/usr/local/bin` unless one is there (`--bin` and + `--build` as on Linux), and `deploy/package/logwisp.rc` as + `/usr/local/etc/rc.d/logwisp`; creates the `logwisp` user with `pw` and + adds it to `--log-group`, enables the service with `sysrc`; LogWisp logs to + syslog under the tag `logwisp` - writes `/usr/local/etc/logwisp/logwisp.toml` - `--jail NAME` writes the files and runs the commands through `jexec`, so the jail's own symlinks cannot redirect a write onto the host; the jail @@ -98,9 +105,9 @@ needs `--force`. certificate must name the address dialed (an IP SAN for an IP), or the name given as `--server-name` - `--auth` on the chain link - - `mtls`: edges present `--cert-file`; the aggregator admits the `--allow` - CNs (none: any certificate the CA issued); an edge's `--allow` pins the - aggregator's CN + - `mtls`: edges present `--cert-file`; the aggregator verifies them with + `--ca-file` and admits the `--allow` CNs (none: any certificate the CA + issued); an edge's `--allow` pins the aggregator's CN - `scram`: the aggregator creates each `--add-user`, its password generated into `--secrets-dir`, an absolute path (default `~/logwisp-secrets`), and handed to `lw auth add-user`, natively or through the image; copy each @@ -108,8 +115,9 @@ needs `--force`. `.pass` is the default `--username` - either binds the node label to the authenticated identity - `--sink-auth` on the `http` and `tcp` outputs - - `mtls` with `--sink-allow`, or `scram` with `--add-viewer` (a separate - `viewers.toml`); viewers log in with `lw auth token` and `lw auth stream` + - `mtls` with `--ca-file` and `--sink-allow`, or `scram` with `--add-viewer` + (a separate `viewers.toml`); viewers log in with `lw auth token` and + `lw auth stream` - without it the outputs admit anyone who reaches them: bind `--listen` to a trusted address - without `--auth` or `--sink-auth`, the auth flags imply their mode: @@ -156,10 +164,11 @@ Variants: - the aggregator on the host itself: drop `--jail logs` - no daemon per jail: one `standalone` or `edge` on the host tails each jail's logs through the host path (`--log-dir JAIL_PATH/var/log`); the entries then - carry the host's label and the bare file name, so files of the same name in - two jails cannot be told apart -- a Linux aggregator: run step 2 with `--runtime docker` on that host and point - the edges at it; the chain protocol is the same + carry the bare file name and, from an edge, the host's label, so files of + the same name in two jails cannot be told apart +- a Linux aggregator: run step 2 on that host with `--runtime docker` in place + of `--runtime native --jail logs`, and point the edges at it; the chain + protocol is the same ## Linux host with containers diff --git a/doc/filters.md b/doc/filters.md index 2ceeef7..7210bcc 100644 --- a/doc/filters.md +++ b/doc/filters.md @@ -129,9 +129,10 @@ time, which is exactly what you want in a log hot path. - Anchors: `^app\.log ` (the text starts with the source), `ERROR$` - Any characters: `.*exception.*` -Remember that TOML basic strings process escapes, so a regex backslash needs -doubling: `"\\berror\\b"`. TOML literal strings avoid the issue: -`'\berror\b'`. +TOML strings process escapes, so a regex backslash needs doubling: +`"\\berror\\b"`. LogWisp's TOML reader has no literal (single-quoted) +strings, so `'\berror\b'` does not load, and a single `\b` is TOML's +backspace: `"\berror\b"` loads and never matches. Anchors apply to the assembled match text, which begins with the source name, so `^ERROR` never matches an entry that has a source; use `\bERROR\b`. @@ -176,10 +177,11 @@ Each filter reports `type`, `logic`, `pattern_count`, `total_processed`, ## Performance -Patterns compile once at startup. Every entry that reaches the filter stage is -evaluated against every filter until one rejects it, so cost scales with the -number of patterns and their complexity. Prefer literal substrings and simple -alternations over broad `.*` wildcards. +Patterns compile when a pipeline is built, at startup and on each reload. +Every entry that reaches the filter stage is evaluated against every filter +until one rejects it, and each filter runs all of its patterns whatever its +`logic`, so cost scales with the number of patterns and their complexity. +Prefer literal substrings and simple alternations over broad `.*` wildcards. Filters log every pattern match and decision at DEBUG: useful when a filter misbehaves, expensive on a busy pipeline, where `logging.level` stays at diff --git a/doc/formatters.md b/doc/formatters.md index 4df8a25..eab24d7 100644 --- a/doc/formatters.md +++ b/doc/formatters.md @@ -11,8 +11,11 @@ flags = 0 timestamp_format = "" ``` -One formatter serves the whole pipeline. Sinks receive an identical payload, -one record ending in exactly one newline; there is no per-sink formatting. +One formatter serves the whole pipeline. Sinks receive an identical payload +ending in a newline; under `raw` a message's own newlines stay, so a record +can span lines. Apart from the `http` sink's SSE framing, only the console +sink alters what it writes, escaping control characters on a terminal and +painting level names ([console](sinks.md#console)). When you need two shapes of the same data, run two pipelines, or chain to a node that formats differently. @@ -38,7 +41,10 @@ bypasses formatting and sanitization: the message reaches the sink as the source produced it, control characters included, with no timestamp, level or source prefix added, and a newline appended unless the message ends in one. An entry that also carries `fields` gets the fields JSON appended verbatim -after a single space; `raw` never drops data and never re-encodes it. +after a single space, so `FlagRaw` (`1`) never drops data or re-encodes it. +Heartbeats format without it, so in `raw` their `include_stats` fields print +in Go's map syntax (`heartbeat map[beat_count:0 type:heartbeat]`), as any +entry's fields do under a non-zero `flags` without `1`. ```toml [pipelines.flow.format] @@ -48,14 +54,22 @@ type = "raw" The closest to verbatim transport is the `console` source, or the `file` source with `raw = true`. The whole line, terminator removed, becomes the message and `fields` stays empty; `raw` writes it back with one `\n`. It is not -byte-exact: blank lines are skipped, CRLF becomes LF, and a line over 1 MiB -continues in the next entry. The `file` source's JSON branch splits a line -into message and fields, so `raw` writes ` ` rather than the -original object. +byte-exact: empty lines are skipped, CRLF becomes LF, and a line over 1 MiB, +or one the `file` source reads while it is still being written, continues in +the next entry. The `file` source's JSON branch splits a line into message +and fields, so `raw` writes ` ` rather than the original object. ### txt -Human-readable line output with a timestamp and level. +Human-readable lines, `