v0.36.0 doc sweep update
Package / arch (push) Canceled after 0s
Package / user (push) Canceled after 0s
Package / deb (push) Canceled after 0s
Package / freebsd (push) Canceled after 0s
Package / image (push) Canceled after 0s
Test / test (push) Canceled after 0s
Test / e2e (push) Canceled after 0s

This commit is contained in:
lixen committed 2026-10-14 11:33:04 -04:00
1 parent de87b9a008
commit a63b512733
51 files changed
+641 -420

No files matched your search

+20 -19
View File
@@ -22,8 +22,8 @@ streams, or downstream LogWisp nodes.
- **A Unix filter out of the box**: with no configuration, `lw` copies stdin - **A Unix filter out of the box**: with no configuration, `lw` copies stdin
to stdout line for line and exits at the end of input; flags add filters, to stdout line for line and exits at the end of input; flags add filters,
formats, sources and sinks formats, sources and sinks
- **Presets**: `tail`, `serve`, `edge` and `aggregator` build the common - **Presets**: `pipe`, `tail`, `serve`, `edge` and `aggregator` build the
pipelines from a few keys (`lw --preset tail:path=/var/log/app`); common pipelines from a few keys (`lw --preset tail:path=/var/log/app`);
`lw --dump` turns any command line into a configuration file `lw --dump` turns any command line into a configuration file
- **Independent pipelines**, each `sources → flow → sinks`, running concurrently - **Independent pipelines**, each `sources → flow → sinks`, running concurrently
in one process in one process
@@ -53,11 +53,11 @@ configuration engine the website's configurator runs as WebAssembly:
and `t` again reruns as a reload, keeping the running pipelines when the new and `t` again reruns as a reload, keeping the running pipelines when the new
ones do not build. Listeners bind, dialers connect and files are written for ones do not build. Listeners bind, dialers connect and files are written for
real; a password is asked in a masked dialog real; a password is asked in a masked dialog
- **Read any log**: the view parses LogWisp's JSON, lw's own log, pino, zap, - **Read any log**: the view parses LogWisp's JSON, lw's own log, vif
slog, docker, journald, ECS, GCP, logfmt and stamped text into time, level, records, pino, zap, slog, docker, journald, ECS, GCP, logfmt and stamped
message and fields, filters them by level or pattern, and shows the payload text into time, level, message and fields, filters them by level or
as the sinks receive it. It never slows a pipeline: what it misses is pattern, and shows the payload as the sinks receive it. It never slows a
counted pipeline: what it misses is counted
- **Any terminal**: `cmd | lw -T` taps piped input, since the screen draws on - **Any terminal**: `cmd | lw -T` taps piped input, since the screen draws on
the controlling terminal; colors range from true color to none, and a text the controlling terminal; colors range from true color to none, and a text
console gets CP437 glyphs console gets CP437 glyphs
@@ -105,13 +105,13 @@ exponential backoff and jitter.
- Authorization by certificate identity: an `auth` block admits named peers - Authorization by certificate identity: an `auth` block admits named peers
(exact or RE2) rather than everything the CA issued, gates the `http` sink's (exact or RE2) rather than everything the CA issued, gates the `http` sink's
stream and status endpoints, and lets a dialer pin the server it talks to stream and status endpoints, and lets a dialer pin the server it talks to
- Password authentication (Argon2id-SCRAM) on the same block: listeners hold - Password authentication (Argon2id-SCRAM) on the same block: no login costs
verifiers, never passwords, and do no KDF work; every login outside proxy a listener KDF work, and with a credentials file it holds verifiers, never
mode is bound to the listener's certificate, so no relay presenting another passwords; every login outside proxy mode is bound to the listener's
one can use it; HTTP listeners issue short-lived bearer tokens. `lw auth` certificate, so no relay presenting another one can use it; HTTP listeners
manages credentials files and logs viewers in; behind a site's issue short-lived bearer tokens. `lw auth` manages credentials files and
TLS-terminating proxy, browsers log in through a shipped login page and a logs viewers in; behind a site's TLS-terminating proxy, browsers log in
dependency-free JS client through a shipped login page and a dependency-free JS client
- Node binding: a chain source can label entries from the sender's certificate - Node binding: a chain source can label entries from the sender's certificate
identity or username instead of from what the sender claims, so origin identity or username instead of from what the sender claims, so origin
attribution is not forgeable attribution is not forgeable
@@ -127,7 +127,7 @@ See [Security](doc/security.md) for configuration and the exact boundary, and
## Documentation ## Documentation
- [Installation](doc/installation.md): building, installing, services, - [Installation](doc/installation.md): building, installing, services,
the container image, packaging the container image, packaging, the end-to-end test scripts
- [Deployment](doc/deployment.md): `deploy/lw-deploy.sh` for edges, - [Deployment](doc/deployment.md): `deploy/lw-deploy.sh` for edges,
aggregators, containers, services, jails aggregators, containers, services, jails
- [Architecture](doc/architecture.md): component model, data flow, - [Architecture](doc/architecture.md): component model, data flow,
@@ -176,8 +176,9 @@ journalctl -f | ./bin/lw --sink http:host=127.0.0.1,port=8080
./bin/lw --source 'file:directory=/var/log/myapp,pattern=*.log' \ ./bin/lw --source 'file:directory=/var/log/myapp,pattern=*.log' \
--format json:sanitizer_policy=json > all.json --format json:sanitizer_policy=json > all.json
# the same with a preset; on a terminal the level names are in color # the same files with a preset, raw to stdout; on a terminal the level
./bin/lw --preset tail:path=/var/log/myapp # names are in color
./bin/lw --preset 'tail:path=/var/log/myapp/*.log'
# compose on screen from a preset menu; r runs, t taps, o prints # compose on screen from a preset menu; r runs, t taps, o prints
./bin/lw -T ./bin/lw -T
@@ -235,8 +236,8 @@ Debian, Ubuntu and Arch. See [Installation](doc/installation.md).
## System Requirements ## System Requirements
- **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+) - **Operating systems**: Linux (kernel 6.10+), FreeBSD (14.0+)
- **Architecture**: amd64 - **Architecture**: amd64; arm64 builds but is untested
- **Go**: 1.27.2+ to build from source - **To build**: Go 1.27.2+, and GNU make or BSD make
Network sources and sinks use IPv4 or IPv6, following the address family of Network sources and sinks use IPv4 or IPv6, following the address family of
the configured host. the configured host.
+5
View File
@@ -1,3 +1,8 @@
// Command lw is LogWisp: it runs the pipelines its file, LOGWISP_ variables
// and flags describe, rebuilt on SIGHUP, SIGUSR1 or an auto_reload, or
// composes them on screen with --tui. Its auth, tls, preset and config commands
// manage credentials and certificates, print a preset's pipeline as TOML and
// write the annotated configuration file.
package main package main
import ( import (
+9 -10
View File
@@ -30,7 +30,7 @@
# quiet = false # No log of lw's own; pipeline output still flows # quiet = false # No log of lw's own; pipeline output still flows
# status_reporter = true # Periodic status logging (30s, DEBUG level); false without a file # status_reporter = true # Periodic status logging (30s, DEBUG level); false without a file
# auto_reload = false # Watch file; read at startup (signals always reread it) # auto_reload = false # Watch file; read at startup (signals always reread it)
# color = "auto" # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR unset # color = "auto" # auto|always|never: level names on console sinks; auto on a terminal, NO_COLOR empty or unset, TERM not dumb
############################################################################### ###############################################################################
### Logging (lw's own log) ### Logging (lw's own log)
@@ -61,7 +61,7 @@
############################################################################### ###############################################################################
# [[pipelines]] # [[pipelines]]
# name = "default" # example # name = "default" # Required (example)
###============================================================================ ###============================================================================
### Flow: the stages between sources and sinks, validated whether on or off ### Flow: the stages between sources and sinks, validated whether on or off
@@ -83,7 +83,7 @@
# [pipelines.flow.format] # [pipelines.flow.format]
# type = "raw" # raw|txt|json # type = "raw" # raw|txt|json
# flags = 0 # Formatter flags; 0 = by type # flags = 0 # For experts, a sum of bits: 1 messages verbatim, 16 no time, 32 no level; 0 = by type
# timestamp_format = "" # Go time layout; empty = RFC 3339 with nanoseconds # timestamp_format = "" # Go time layout; empty = RFC 3339 with nanoseconds
# sanitizer_policy = "" # raw|json|txt|shell; empty = by type # sanitizer_policy = "" # raw|json|txt|shell; empty = by type
@@ -146,7 +146,7 @@
### certificate to the user (needs tls.client_auth) ### certificate to the user (needs tls.client_auth)
### allow mtls: exact identities. Empty allow AND ### allow mtls: exact identities. Empty allow AND
### allow_patterns admits any identity the CA vouches ### allow_patterns admits any identity the CA vouches
### for (logged WARN) ### for (a listener logs a WARN)
### allow_patterns mtls: RE2 patterns; anchor them yourself (^...$) ### allow_patterns mtls: RE2 patterns; anchor them yourself (^...$)
### node_binding chain sources: binds the node label to the identity ### node_binding chain sources: binds the node label to the identity
### or username, overriding trust_node; unset = force ### or username, overriding trust_node; unset = force
@@ -205,7 +205,8 @@
# pattern = "*" # Glob of the whole file name (* and ? only) # pattern = "*" # Glob of the whole file name (* and ? only)
# check_interval_ms = 100 # Directory rescan, at least 10; open files are polled every 100 ms # check_interval_ms = 100 # Directory rescan, at least 10; open files are polled every 100 ms
## raw = true never parses a line; with format type "raw" each line is relayed ## raw = true never parses a line; with format type "raw" each line is relayed
## as written, except that empty lines are skipped and CRLF becomes LF. ## as written, except that empty lines are skipped, CRLF becomes LF and a line
## over 1 MiB, or one still being written when read, is split.
# raw = false # raw = false
# from = "end" # end|start: where a newly found file is read from # from = "end" # end|start: where a newly found file is read from
@@ -300,7 +301,7 @@
# node_binding = "" # node_binding = ""
# credentials_file = "" # scram: written by `lw auth add-user` # credentials_file = "" # scram: written by `lw auth add-user`
# token_lifetime_ms = 0 # token_lifetime_ms = 0
# trusted_proxies = [] # the http sink only # trusted_proxies = [] # refused here: the http sink only
# username = "" # username = ""
# password_file = "" # password_file = ""
# [pipelines.plugin_sources.config.acl] # [pipelines.plugin_sources.config.acl]
@@ -380,7 +381,6 @@
# identity = "" # identity = ""
# allow = [] # allow = []
# allow_patterns = [] # allow_patterns = []
# node_binding = "" # chain sources only
# credentials_file = "" # scram: log in with `lw auth token` # credentials_file = "" # scram: log in with `lw auth token`
# token_lifetime_ms = 0 # token_lifetime_ms = 0
# trusted_proxies = [] # scram: proxies ending the browsers' TLS; tls may then be off # trusted_proxies = [] # scram: proxies ending the browsers' TLS; tls may then be off
@@ -423,7 +423,6 @@
# identity = "" # identity = ""
# allow = [] # allow = []
# allow_patterns = [] # allow_patterns = []
# node_binding = "" # chain sources only
# credentials_file = "" # scram: readers use `lw auth stream` # credentials_file = "" # scram: readers use `lw auth stream`
# username = "" # username = ""
# password_file = "" # password_file = ""
@@ -443,7 +442,7 @@
# [pipelines.plugin_sinks.config] # [pipelines.plugin_sinks.config]
# host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1" # host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1"
# port = 9440 # Required (example) # port = 9440 # Required (example)
# node = "" # Origin label, kept across hops; empty = the host name # node = "" # Origin label, kept across hops; empty = this machine's host name
# buffer_size = 1000 # buffer_size = 1000
# dial_timeout_ms = 5000 # dial_timeout_ms = 5000
# write_timeout_ms = 5000 # write_timeout_ms = 5000
@@ -476,7 +475,7 @@
# host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1" # host = "aggregator.example.org" # Required (example); an IPv6 literal bare, e.g. "2001:db8::1"
# port = 9441 # Required (example) # port = 9441 # Required (example)
# ingest_path = "/ingest" # Must start with "/"; not "/auth" # ingest_path = "/ingest" # Must start with "/"; not "/auth"
# node = "" # Origin label, kept across hops; empty = the host name # node = "" # Origin label, kept across hops; empty = this machine's host name
# buffer_size = 1000 # buffer_size = 1000
# max_batch_count = 100 # max_batch_count = 100
# max_batch_bytes = 1048576 # max_batch_bytes = 1048576
+5
View File
@@ -13,3 +13,8 @@ deploy/lw-deploy.sh --yes ... # no questions: defaults, or fail on a miss
``` ```
Roles, runtimes, scenarios and what is not covered: [doc/deployment.md](../doc/deployment.md). Roles, runtimes, scenarios and what is not covered: [doc/deployment.md](../doc/deployment.md).
`package/` holds the service files (`logwisp.service`, `.sysusers`,
`.tmpfiles`, `logwisp.rc`) that `make install` and this script install, the
shell completions, and the deb, Arch and FreeBSD packaging:
[doc/installation.md](../doc/installation.md#installing).
+3 -1
View File
@@ -105,7 +105,9 @@ Containers (Linux)
publishes nothing and listens on --listen publishes nothing and listens on --listen
Native services Native services
--bin FILE lw binary to install (default: the installed one, else bin/lw) --bin FILE lw binary to install (default: the installed one, else
bin/lw); manual: the lw that adds users and checks
(default: lw on PATH, else bin/lw)
--jail NAME FreeBSD: install into this running jail through jexec --jail NAME FreeBSD: install into this running jail through jexec
--package-dir DIR service files (default deploy/package) --package-dir DIR service files (default deploy/package)
+1 -1
View File
@@ -4,7 +4,7 @@ The [README](../README.md) has LogWisp's features, a quick start and the
system requirements. system requirements.
- [Installation](installation.md): building, installing, services, - [Installation](installation.md): building, installing, services,
the container image, packaging the container image, packaging, the end-to-end test scripts
- [Deployment](deployment.md): `deploy/lw-deploy.sh` for edges, - [Deployment](deployment.md): `deploy/lw-deploy.sh` for edges,
aggregators, containers, services, jails aggregators, containers, services, jails
- [Architecture](architecture.md): component model, data flow, - [Architecture](architecture.md): component model, data flow,
+43 -16
View File
@@ -45,18 +45,23 @@ Package map:
- `internal/filter`: regex filter and filter chain. - `internal/filter`: regex filter and filter chain.
- `internal/format`: adapter over the `lixenwraith/log` formatter and - `internal/format`: adapter over the `lixenwraith/log` formatter and
sanitizer. sanitizer.
- `internal/source/*`, `internal/sink/*`: the source and sink plugins. - `internal/source`, `internal/sink`: the `Source` and `Sink` interfaces and
their stats, and the stop helpers network sinks share (`sink.FlushBound`,
`FlushContext`, `Drain`); their subpackages are the plugins.
- `internal/plugin`: global factory registry populated by plugin `init()`. - `internal/plugin`: global factory registry populated by plugin `init()`.
- `internal/chain`: chain wire protocol: hello preamble, entry codec, backoff. - `internal/chain`: chain wire protocol: hello preamble, entry codec, backoff.
- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`. - `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`; also
the certificates `lw tls` writes and a listener makes at startup, pins and
startup warnings.
- `internal/authz`: the single seam between `AuthOptions` and the network - `internal/authz`: the single seam between `AuthOptions` and the network
plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`; `Authorize` plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`; `Authorize`
refuses under `scram`. refuses under `scram`.
- `internal/netacl`: a listener's `acl` block: address rules, PROXY headers - `internal/netacl`: a listener's `acl` block: address rules, PROXY headers
and per-client limits; its `Table` is SCRAM throttling's limiter too. and per-client limits; its `Table` is SCRAM throttling's limiter too.
- `internal/session`: session manager and per-instance proxy. - `internal/session`: session manager and per-instance proxy.
- `internal/core`: shared types (`LogEntry`, `TransportEvent`), capabilities, - `internal/core`: shared types (`LogEntry`, `TransportEvent`, the viewer's
constants. `Tap`), the level table `core.Levels`, capabilities, constants, and
`Network` and `Listen`, which keep a socket to its host's address family.
- `internal/tokenbucket`: rate limiter primitive. - `internal/tokenbucket`: rate limiter primitive.
- `internal/tui`: `lw --tui`, the diagram and the live view of what `t` - `internal/tui`: `lw --tui`, the diagram and the live view of what `t`
runs; `cmd/lw/screen.go` runs the pipelines for it. runs; `cmd/lw/screen.go` runs the pipelines for it.
@@ -65,6 +70,8 @@ Package map:
terminal acts on, which the console sink shares. terminal acts on, which the console sink shares.
- `internal/version`: version, commit and build time, stamped at build. - `internal/version`: version, commit and build time, stamped at build.
- `internal/testutil`: fixtures for tests only. - `internal/testutil`: fixtures for tests only.
- `config`: embeds `logwisp.toml`, the annotated default configuration that
`lw config init` writes and the packages install.
## Plugin Registration ## Plugin Registration
@@ -98,6 +105,11 @@ instance of a single type fails, naming both pipelines.
4. **Dispatch** sends the event to every sink's input channel with a 4. **Dispatch** sends the event to every sink's input channel with a
non-blocking send, except to a sink declaring `CapBackpressure`. non-blocking send, except to a sink declaring `CapBackpressure`.
A heartbeat joins at step 4: the flow writes it in its format with a
structured `Entry`, and the pipeline offers it to the tap and dispatches it,
so no rate limit or filter drops it and `flow.total_processed` does not count
it.
`LogEntry` fields: `LogEntry` fields:
- `Time`: entry timestamp. - `Time`: entry timestamp.
@@ -161,7 +173,10 @@ shutdown (`dropped_batches`).
`Pipeline.Stop` is ordered so in-flight data drains: `Pipeline.Stop` is ordered so in-flight data drains:
1. Release dispatch waits on backpressure sinks; stop all sources 1. Release dispatch waits on backpressure sinks; stop all sources
concurrently, each closing its subscriber channels. concurrently, each closing its subscriber channels. From the release on,
dispatch picks at random between a backpressure sink's input and the
release, so it drops about half of the entries still in flight for one
(`total_dropped_by_sink`) even with room.
2. Wait for the run loop, which ends when every subscription channel closes. 2. Wait for the run loop, which ends when every subscription channel closes.
3. Stop all sinks concurrently: console and file sinks write their queue; 3. Stop all sinks concurrently: console and file sinks write their queue;
network sinks deliver theirs within `sink.FlushBound`, their write or network sinks deliver theirs within `sink.FlushBound`, their write or
@@ -197,11 +212,13 @@ TLS is built in exactly one place, `internal/tlsx`, which exposes
## Sessions ## Sessions
Each pipeline owns a `session.Manager`. Plugins receive a `session.Proxy` Each pipeline owns a `session.Manager`. Plugins receive a `session.Proxy`
scoped to their instance id, so one plugin cannot see or remove another's scoped to their id, so a plugin sees and removes only the sessions under its
sessions. A session records the remote address, creation and last-activity id; ids are unique per list, so a source and a sink that share one (the
timestamps, and metadata — including `tls` and `tls_peer_cn` for TLS peers, built-in pipe's two `console`s) share that scope. A session records the
`auth_method` / `auth_identity` for authorized ones, and `peer_addr`, the proxy, remote address, creation and last-activity timestamps, and metadata —
for a client a PROXY header names. including `tls` and `tls_peer_cn` for TLS peers, `auth_method` /
`auth_identity` for authorized ones, and `peer_addr`, the proxy, for a client
a PROXY header names.
Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The
HTTP sink's broker treats a vanished session as an eviction signal and closes HTTP sink's broker treats a vanished session as an eviction signal and closes
@@ -277,7 +294,10 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key",
it: answer it in `dirs` and call again. it: answer it in `dirs` and call again.
- Pipelines are a JSON list in the file's shape (`name`, `flow`, - Pipelines are a JSON list in the file's shape (`name`, `flow`,
`plugin_sources`, `plugin_sinks`); a misspelled key is refused. A node is `plugin_sources`, `plugin_sinks`); a misspelled key is refused. A node is
`{role, id, index}`: a source's or sink's id, a filter's index. `{role, id, index}`: `source` or `sink` by its id, `filters` by its index,
or the flow stage `rate_limit`, `format` or `heartbeat`, which `set` turns
on and `remove` off; `add` takes `source`, `sink`, or `filters` with the
type `include` or `exclude`.
## Resource Management ## Resource Management
@@ -285,18 +305,25 @@ answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key",
load. load.
- `max_connections` caps connections on the `tcp` sink and the `tcp_chain` - `max_connections` caps connections on the `tcp` sink and the `tcp_chain`
source, and streams on the `http` sink. Admission is a load-then-check, so a source, and streams on the `http` sink. Admission is a load-then-check, so a
burst can over-admit by roughly one. burst can over-admit by the connections that arrive before the count rises.
- A listener's `acl` block caps each client: `max_connections_per_client`,
and on the `http` sink and `http_chain` source
`requests_per_second_per_client` ([Security](security.md#per-client-limits)).
- Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an - Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an
oversized line terminates a `tcp_chain` link and fails an `http_chain` oversized line terminates a `tcp_chain` link and fails an `http_chain`
batch with `400`. batch with `400`.
- The `http_chain` source caps each request body at `max_body_bytes`. - The `http_chain` source caps each request body at `max_body_bytes`.
- File sinks rotate on size, cap total rotated size, and honour a retention - File sinks rotate on size, cap total rotated size, honour a retention
window. window, and with `min_disk_free_mb` delete the oldest files to keep that
much disk free.
## Performance Notes ## Performance Notes
- In-memory entry processing is sub-millisecond; the formatter mutex is the - In-memory entry processing is sub-millisecond. Besides the formatter's
only shared serialization point in the hot path. mutex, an entry can take the rate limiter's bucket lock (`policy = "drop"`)
and the pipeline's session-manager lock, which the network plugins and the
console and random sources take as entries pass, to refresh session
activity.
- File tailing detects new content within roughly 100 ms (fixed poll), while - File tailing detects new content within roughly 100 ms (fixed poll), while
`check_interval_ms` governs how quickly a *newly created* file is noticed. `check_interval_ms` governs how quickly a *newly created* file is noticed.
- `http_chain` trades latency for efficiency: entries wait up to - `http_chain` trades latency for efficiency: entries wait up to
+25 -15
View File
@@ -20,7 +20,7 @@ nodes can filter and reformat as if the entries were local.
┌───────────────┐ │ http_chain sink ───┼──► upstream collector ┌───────────────┐ │ http_chain sink ───┼──► upstream collector
│ file source │ HTTP/TLS │ │ │ file source │ HTTP/TLS │ │
│ ↓ ├────────────►│ http_chain source │ │ ↓ ├────────────►│ http_chain source │
│ http_chain sink│ :15802 └────────────────────┘ │http_chain sink│ :15802 └────────────────────┘
└───────────────┘ └───────────────┘
``` ```
@@ -44,7 +44,7 @@ Chained entries carry a `node` label identifying where they originated.
Relays preserve `node`, so a label survives any number of hops and identifies Relays preserve `node`, so a label survives any number of hops and identifies
the original producer, unless a hop's source overwrites it: `trust_node = the original producer, unless a hop's source overwrites it: `trust_node =
false`, or `node_binding = "force"` (the default with an `auth` block). false`, or `node_binding = "force"` (the default under `mtls` or `scram`).
Formatters write the label as a prefix on the source Formatters write the label as a prefix on the source
([Formatters](formatters.md#node-identity-in-output)). ([Formatters](formatters.md#node-identity-in-output)).
@@ -55,9 +55,10 @@ under `scram`. `auth.node_binding`:
- `none`: `trust_node` governs the connection label and the per-entry `node` - `none`: `trust_node` governs the connection label and the per-entry `node`
field. field.
- `assert`: the connection label must equal the identity, or the peer is - `assert`: the connection label must equal the identity, or the peer is
rejected; `trust_node` governs the per-entry `node` field. rejected; `trust_node` governs the per-entry `node` field, the label standing
- `force` (default with an `auth` block): the identity is the connection label in wherever it would put the remote address.
and overwrites every per-entry `node` field. - `force` (default under `mtls` or `scram`): the identity is the connection
label and overwrites every per-entry `node` field.
Pick `force` at an ingest boundary you do not trust — it is the only setting Pick `force` at an ingest boundary you do not trust — it is the only setting
where a compromised edge cannot mislabel its entries, including through the where a compromised edge cannot mislabel its entries, including through the
@@ -87,17 +88,23 @@ A persistent connection carrying newline-delimited JSON.
{"logwisp":1,"node":"edge-01"} {"logwisp":1,"node":"edge-01"}
``` ```
3. The listener reads that line within `hello_timeout_ms` and rejects the 3. The listener reads that line, at most 4 KiB, within `hello_timeout_ms` and
connection if it is malformed or declares a different protocol version. rejects the connection if it is malformed or declares a different protocol
version.
4. Every subsequent line is one JSON-encoded `LogEntry`. 4. Every subsequent line is one JSON-encoded `LogEntry`.
Line size is bounded at 1 MiB. An oversized line is a protocol violation and Line size is bounded at 1 MiB. An oversized line is a protocol violation and
terminates the connection, because the scanner cannot resynchronize afterwards. terminates the connection, because the scanner cannot resynchronize afterwards.
The cap applies to the encoded line, envelope and JSON escapes included, so a The cap applies to the encoded line, envelope and JSON escapes included, so a
message near 1 MiB exceeds it (the console and file sources cut lines at message near 1 MiB exceeds it (the console and file sources cut lines at
1 MiB). On `tcp_chain` such an entry ends the link and is lost, together with 1 MiB). On `tcp_chain` such an entry ends the link. If the socket took the
any line the sink writes before it notices the close. On `http_chain` it fails whole line, the entry is lost, together with any line the sink writes before it
its whole batch with `400`, which the sink drops. notices the close. If the line overruns the cap by more than the socket buffers
hold, as a message of control characters can (most escape to six bytes), the
write fails and the sink resends the line on every reconnect: the link stalls
until a stop or reload, and later entries count in `total_dropped_by_sink`. The
sink does not yet drop a line the source must refuse. On `http_chain` such an
entry fails its whole batch with `400`, which the sink drops.
Under `auth.type = "scram"` the hello also carries the SCRAM client-first Under `auth.type = "scram"` the hello also carries the SCRAM client-first
message, and three more lines precede the entries, each one JSON object of at message, and three more lines precede the entries, each one JSON object of at
@@ -119,18 +126,20 @@ binary ignores it and the dialer gives up after 10 s.
### HTTP transport ### HTTP transport
Batches of NDJSON delivered by `POST`, with the preamble expressed as headers. Batches of NDJSON delivered by `POST` to `ingest_path` (default `/ingest`), with
the preamble expressed as headers.
- Request headers - Request headers
- `X-Logwisp-Protocol`: the protocol version; must be `1`. - `X-Logwisp-Protocol`: the protocol version; must be `1`.
- `X-Logwisp-Node`: the origin node label. - `X-Logwisp-Node`: the sending sink's `node`, the connection label, as in the
TCP hello.
- `Content-Type`: `application/x-ndjson`. - `Content-Type`: `application/x-ndjson`.
- `Authorization`: `Bearer <token>`, under `scram`. - `Authorization`: `Bearer <token>`, under `scram`.
- Response header - Response header
- `X-Logwisp-Accepted`: the number of entries ingested. - `X-Logwisp-Accepted`: the number of entries ingested.
Responses: `204` on success; `400` for a bad protocol version or a malformed Responses: `204` on success; `400` for a bad protocol version or a transfer
body, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for error, a line over 1 MiB included; `413` past `max_body_bytes`; `404` for
another path; `405` for a non-`POST` method; `429` (with `Retry-After`) past another path; `405` for a non-`POST` method; `429` (with `Retry-After`) past
`acl.requests_per_second_per_client`; `401` for a missing, invalid or expired `acl.requests_per_second_per_client`; `401` for a missing, invalid or expired
token; `403` for a refused certificate or node label. token; `403` for a refused certificate or node label.
@@ -157,7 +166,8 @@ oversized body.
``` ```
`node`, `level`, and `fields` are omitted when empty. A missing `time` is filled `node`, `level`, and `fields` are omitted when empty. A missing `time` is filled
in at ingest. in at ingest. A line that does not decode is skipped and counted in
`parse_errors`; the connection or batch stands.
## Delivery Semantics ## Delivery Semantics
+36 -31
View File
@@ -74,23 +74,24 @@ pipelines have [their own flags](#pipelines):
- `--schema`: print every setting, pipeline flag, flow stage, plugin option, - `--schema`: print every setting, pipeline flag, flow stage, plugin option,
preset and output form as one JSON document, and exit; tools build their preset and output form as one JSON document, and exit; tools build their
option lists from it: option lists from it:
- a key: its kind, default, rules and help; a table (`tls`, `auth`, `acl`) - a key: its kind, default, rules and help, and a setting's `variable`; a
holds its own keys, those of the plugin's side (listener or dialer), table (`tls`, `auth`, `acl`) holds its own keys, those of the plugin's
HTTP's only on a plugin that speaks HTTP. Every listener lists side (listener, chain source or dialer), HTTP's only on a plugin that
`auth.node_binding`, which only chain sources use, and every HTTP speaks HTTP: `auth.node_binding` only on the chain sources, and
listener `auth.trusted_proxies`, which only the `http` sink admits `auth.trusted_proxies` on both HTTP listeners, though only the `http`
sink admits it
- a flow stage: its keys, and `on`, the values that turn it on - a flow stage: its keys, and `on`, the values that turn it on
- a preset key: its default and help, and the choices (`enum`) and `hint` - a preset key: its default and help, and the choices (`enum`) and `hint`
of the option it sets of the option it sets
- `forms`: `command`, `environment` and `file`, each with its hint - `forms`: `command`, `environment` and `file`, each with its hint
- `--color [WHEN]`, WHEN `auto`, `always` or `never`: level names in color on - `--color [WHEN]`, WHEN `auto`, `always` or `never`: level names in color on
console sinks, default `auto` (a terminal, `NO_COLOR` unset, `TERM` not console sinks, default `auto` (a terminal, `NO_COLOR` empty or unset,
`dumb`); bare, `always`. A console sink's own `color` wins `TERM` not `dumb`); bare, `always`. A console sink's own `color` wins
([console sink](sinks.md#console)) ([console sink](sinks.md#console))
- `-q`, `--quiet`: silence lw's own log and notices, default `false`; - `-q`, `--quiet`: silence lw's own log and notices, default `false`;
pipeline output still flows pipeline output still flows
- `--status_reporter=BOOL`: periodic status logging, default `true` with a - `--status_reporter=BOOL`: pipeline statistics every 30 s, logged at
configuration file, `false` without `debug`, default `true` with a configuration file, `false` without
- `--auto_reload=BOOL`: reload when the configuration file changes, default - `--auto_reload=BOOL`: reload when the configuration file changes, default
`false` `false`
- `-V`, `--version`: print the version and exit - `-V`, `--version`: print the version and exit
@@ -105,12 +106,15 @@ The last `-c` wins; a missing or empty path is an error.
default `stderr`, so stdout carries only data default `stderr`, so stdout carries only data
- `--logging.level`: `debug`, `info`, `warn` or `error`, default `info` with a - `--logging.level`: `debug`, `info`, `warn` or `error`, default `info` with a
configuration file, `warn` without configuration file, `warn` without
- `--logging.format`: `raw`, `txt` or `json` - `--logging.format`: `raw`, `txt` or `json`, default `txt`
- `--logging.sanitization`: `raw`, `json`, `txt` or `shell` - `--logging.sanitization`: `raw`, `json`, `txt` or `shell`, default `raw`
- `--logging.file.directory`: a path - `--logging.file.directory`: a path, default `./log`; the limits below
- `--logging.file.name`: a string delete any `.log` file in it
- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers ([Application Logging](configuration.md#application-logging))
- `--logging.file.retention_hours`: a float - `--logging.file.name`: a string, default `logwisp`
- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers,
default `100` and `1000`
- `--logging.file.retention_hours`: a float, default `168`; `0` keeps files
`--logging.console.target` is accepted but has no effect; the console `--logging.console.target` is accepted but has no effect; the console
destination is derived from `--logging.output`. destination is derived from `--logging.output`.
@@ -308,8 +312,8 @@ Any pipeline flag on the command line makes LogWisp ignore all of them.
A container needs no configuration file. Without one lw logs less A container needs no configuration file. Without one lw logs less
([Built-in Defaults](#built-in-defaults)), so pass `LOGWISP_LOGGING_LEVEL=info` ([Built-in Defaults](#built-in-defaults)), so pass `LOGWISP_LOGGING_LEVEL=info`
and `LOGWISP_STATUS_REPORTER=true`; the image sets neither, as a variable would (`debug`, with `LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the
override a mounted file's value: image sets neither, as a variable would override a mounted file's value:
```bash ```bash
docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
@@ -318,7 +322,7 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
-e LOGWISP_FILTER='exclude:patterns=DEBUG' \ -e LOGWISP_FILTER='exclude:patterns=DEBUG' \
-e LOGWISP_SINK='http:host=0.0.0.0,port=8080' \ -e LOGWISP_SINK='http:host=0.0.0.0,port=8080' \
-e LOGWISP_SINK_1=console \ -e LOGWISP_SINK_1=console \
-e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true \ -e LOGWISP_LOGGING_LEVEL=info \
logwisp:dev logwisp:dev
``` ```
@@ -335,8 +339,9 @@ docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
## Exit Codes ## Exit Codes
- `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version` / - `0`: the end of input, a clean shutdown (`SIGINT`, `SIGTERM`), `--version`,
`--help`, `--dump`, or a valid `--check` `--help`, `--schema`, `--dump`, a valid `--check`, or a `--tui` screen
closed without `r`
- `1`: general error: a configuration load or validation failure (`--check` - `1`: general error: a configuration load or validation failure (`--check`
included, and a bad value on the command line), a logger init failure, a included, and a bad value on the command line), a logger init failure, a
service bootstrap failure service bootstrap failure
@@ -360,7 +365,7 @@ lw < app.log > copy.log
- Each line of stdin is one entry and one line of stdout. - Each line of stdin is one entry and one line of stdout.
- The terminator (`\n` or `\r\n`) is written back as `\n`; an unterminated - The terminator (`\n` or `\r\n`) is written back as `\n`; an unterminated
last line is kept and terminated. last line is kept and terminated.
- Blank lines are skipped. - Empty lines are skipped; a line of spaces is an entry.
- A line over 1 MiB continues in the next entry; no byte is lost. - A line over 1 MiB continues in the next entry; no byte is lost.
- Nothing is dropped: a slow reader slows the reading of stdin. - Nothing is dropped: a slow reader slows the reading of stdin.
- On a terminal, control characters are written as `<hex>` (`ESC` becomes - On a terminal, control characters are written as `<hex>` (`ESC` becomes
@@ -504,16 +509,16 @@ WebAssembly; `t` runs them under the screen and shows what flows.
- The bar counts the shown pipeline's entries in, filtered by the flow, - The bar counts the shown pipeline's entries in, filtered by the flow,
dropped by sinks, and missed by the screen: the view never slows a dropped by sinks, and missed by the screen: the view never slows a
pipeline. pipeline.
- Live view: from 40 rows it sits under the diagram; shorter, it takes the - Live view: from 40 rows it sits under the diagram, unless the inspector is
screen while it has the keys. `t` or Esc gives them back to the diagram open below the canvas; otherwise it takes the screen while it has the keys.
(Esc first closes an open detail). `t` or Esc gives them back to the diagram (Esc first closes an open detail).
- The canvas cursor picks its point, so the view opens on the first - The canvas cursor picks its point, the first source until it moves: a
source: a source shows its entries, a flow stage (or the filter the source shows its entries, a flow stage (or the filter the inspector is
inspector is on) what reaches it, the heartbeat its own, the sinks (or on) what reaches it, the heartbeat its own, the sinks (or no part) what
no part) what they got. `v` switches between all (what the flow dropped the flow offered them (a full sink drops its copy alone). `v` switches
dimmed), what passed and what was dropped there, with no restart; the between all (what the flow dropped dimmed), what passed and what was
title says which. lw's own log shows from warn up at every point; `s` dropped there, with no restart; the title says which. lw's own log shows
also offers it alone, debug too. from warn up at every point; `s` also offers it alone, debug too.
- A row is the entry's time, level, tag, message and fields, read from any - A row is the entry's time, level, tag, message and fields, read from any
line shape: LogWisp's json, lw's own log, vif records, pino, zap, slog, line shape: LogWisp's json, lw's own log, vif records, pino, zap, slog,
docker, journald, ECS, GCP, logfmt, or text after an RFC3339 stamp and docker, journald, ECS, GCP, logfmt, or text after an RFC3339 stamp and
+30 -19
View File
@@ -1,7 +1,8 @@
# Configuration Reference # Configuration Reference
LogWisp is configured with TOML. A complete annotated file listing every option LogWisp is configured with TOML. A complete annotated file listing every option
and its default ships as [`config/logwisp.toml`](../config/logwisp.toml). and its default ships as [`config/logwisp.toml`](../config/logwisp.toml);
`lw config init` writes it ([CLI](cli.md#lw-config)).
## Configuration Precedence ## Configuration Precedence
@@ -45,8 +46,8 @@ Top-level keys, each as type and default:
- `auto_reload` (bool, `false`): watch the config file and reload pipelines on - `auto_reload` (bool, `false`): watch the config file and reload pipelines on
change. change.
- `color` (string, `"auto"`): level names in color on console sinks, which may - `color` (string, `"auto"`): level names in color on console sinks, which may
set their own: `"auto"` (a terminal, `NO_COLOR` unset, `TERM` not `dumb`), set their own: `"auto"` (a terminal, `NO_COLOR` empty or unset, `TERM` not
`"always"` or `"never"`; see the [console sink](sinks.md#console). `dumb`), `"always"` or `"never"`; see the [console sink](sinks.md#console).
`--version`, `--check`, `--dump`, `--schema` and `--tui` are switches, not `--version`, `--check`, `--dump`, `--schema` and `--tui` are switches, not
settings; the matching variables are ignored. settings; the matching variables are ignored.
@@ -80,8 +81,9 @@ retention_hours = 168.0
- `all`: files plus split console. - `all`: files plus split console.
- `none`: no application logging. - `none`: no application logging.
`[logging.file]` applies only to the `file` and `all` modes. Its limits delete `[logging.file]` applies only to the `file` and `all` modes. Its limits, and a
any `.log` file in `directory`, lw's or not: give lw a directory of its own. floor of 10 MB free on the disk that no key changes, delete any `.log` file in
`directory`, lw's or not: give lw a directory of its own.
`retention_hours = 0` keeps files. The `stdout`, `split` and `all` modes mix `retention_hours = 0` keeps files. The `stdout`, `split` and `all` modes mix
the log into a console sink's stdout. `[logging.console].target` is validated the log into a console sink's stdout. `[logging.console].target` is validated
(`stdout`, `stderr`, `split`) but not applied: `logging.output` chooses the (`stdout`, `stderr`, `split`) but not applied: `logging.output` chooses the
@@ -220,11 +222,12 @@ pipelines[0].plugin_sinks[http].config: unknown key "tls.enabeld"
- `auth` enums (`identity`, `node_binding`) are checked whatever `type` is; a - `auth` enums (`identity`, `node_binding`) are checked whatever `type` is; a
block naming peers or credentials with `type` `none` or unset is refused block naming peers or credentials with `type` `none` or unset is refused
([Security](security.md#validation-1)). ([Security](security.md#validation-1)).
- Once the rest is valid, the file itself: the first key the schema does not - The file itself, after the rest at start and by `lw --check` but before it
declare outside a plugin's `config` is refused (`config file "…": unknown at a reload: the first key the schema does not declare outside a plugin's
key "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path `config` is refused (`config file "…": unknown key
would drop the whole table it heads. A top-level `config_file`, `version`, "pipelines[0].plugin_sinks[0].confg"`), since a misspelled table path would
`check` or `dump` is ignored. drop the whole table it heads. A top-level `config_file`, `version`, `check`
or `dump` is ignored.
- Files a plugin names (certificates, keys, CA files, credentials, password - Files a plugin names (certificates, keys, CA files, credentials, password
files) are read when it is built: at start, at reload and by `lw --check`. files) are read when it is built: at start, at reload and by `lw --check`.
Those failures name the pipeline and plugin. Those failures name the pipeline and plugin.
@@ -234,14 +237,15 @@ exits 1 ([Networking](networking.md#network-plugins)).
## Hot Reload ## Hot Reload
`auto_reload = true` watches the configuration file. `SIGHUP` or `SIGUSR1` `auto_reload = true` watches the configuration file: lw reads it every second
rereads it even when watching is off and always rebuilds, so certificates and and applies a change once two reads in a row agree, so within about 2 s.
credentials rotate without a TOML edit. CLI and environment overrides, `SIGHUP` or `SIGUSR1` rereads it even when watching is off and always
pipelines included, are captured at startup and keep their precedence. A rebuilds, so certificates and credentials rotate without a TOML edit. CLI and
discovered default file that does not exist is not an error, so an instance environment overrides, pipelines included, are captured at startup and keep
configured by flags or variables alone still rebuilds; once read, though, its their precedence. A discovered default file that does not exist is not an
removal fails the reload and the running service stays as it is; restart to error, so an instance configured by flags or variables alone still rebuilds;
run without the file. once read, though, its removal fails the reload and the running service stays
as it is; restart to run without the file.
Reload builds a new service from a detached snapshot of the configuration, Reload builds a new service from a detached snapshot of the configuration,
validated again, and shuts the old one down only if that succeeds: parse, validated again, and shuts the old one down only if that succeeds: parse,
@@ -271,6 +275,13 @@ Each TOML type, its Go type, and its command-line / environment form:
- Boolean (`bool`): `true` / `false`, or a bare flag for `true`. - Boolean (`bool`): `true` / `false`, or a bare flag for `true`.
- Array (`[]T`): only in [pipeline specs](cli.md#pipelines), as a repeated key. - Array (`[]T`): only in [pipeline specs](cli.md#pipelines), as a repeated key.
- Table (struct): nested path with `.` (flags) or `_` (environment). - Table (struct): nested path with `.` (flags) or `_` (environment).
- Path (`string`: a plugin's `directory` and `*_file` keys,
`logging.file.directory`): `~` alone or a leading `~/` is the home directory
where lw opens it (`--dump` prints it as typed); `~user` stays as typed; a
relative path starts at lw's working directory. The configuration file's own
path (`-c`, `LOGWISP_CONFIG_*`) is read as typed.
Integer keys refuse fractions and out-of-range values; number keys refuse inf Integer keys refuse fractions and out-of-range values; number keys refuse inf
and NaN. Configuration files are TOML only. and NaN. A suffix names a key's unit: `_ms` milliseconds, `_mb` megabytes of
1,000,000 bytes, `_bytes` bytes, `_hours` hours. Configuration files are TOML
only.
+32 -23
View File
@@ -10,7 +10,9 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
## Roles ## Roles
- `edge` - `edge`
- tails each `--log-dir` (not recursive) for `--log-pattern` (default `*.log`) - tails each `--log-dir` (not recursive) for `--log-pattern` (default
`*.log`), a newly found file from its end, or with `--from start` from its
first line
- forwards to `--aggregator` over `tcp_chain` (port 9440) or, with - forwards to `--aggregator` over `tcp_chain` (port 9440) or, with
`--chain http`, `http_chain` (port 9441) `--chain http`, `http_chain` (port 9441)
- labels its entries `--node`: the short host name, or the jail name - labels its entries `--node`: the short host name, or the jail name
@@ -27,15 +29,16 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
- `docker`, `podman` (Linux) - `docker`, `podman` (Linux)
- the container runs `--read-only --cap-drop ALL --security-opt - the container runs `--read-only --cap-drop ALL --security-opt
no-new-privileges -u 65532:65532 --restart unless-stopped` no-new-privileges -u 65532:65532 --restart unless-stopped`
- `--config-dir` (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`, - `--name` (default `logwisp-ROLE`) names the container; `--config-dir`
and the credentials files, owned by 65532 and mounted read-only at (default `/etc/logwisp/NAME`) holds `logwisp.toml`, `tls/`, and the
`/etc/logwisp` credentials files, owned by 65532 and mounted read-only at `/etc/logwisp`
- log directories mount read-only and the output directory writable, each at - log directories mount read-only and the output directory writable, each at
its host path; `--log-group` adds that group's id to the container its host path; `--log-group` adds that group's id to the container
- listener ports publish on `--listen`, an IP address; the process inside - listener ports publish on `--listen`, an IP address; the process inside
binds `0.0.0.0` binds `0.0.0.0`
- `--network NET` joins a user network, created when missing, where edges - `--network NET` joins a user network, created when missing, where edges
reach an aggregator by container name; `host` publishes nothing reach an aggregator by container name; `host` publishes nothing, and the
process binds `--listen` itself
- `--image` defaults to `logwisp:dev`, the `make image` tag; `--build` builds - `--image` defaults to `logwisp:dev`, the `make image` tag; `--build` builds
it from the repository. Behind an HTTPS-intercepting proxy add it from the repository. Behind an HTTPS-intercepting proxy add
`--build-ca FILE` and, for a proxy on loopback, `--build-ca FILE` and, for a proxy on loopback,
@@ -47,10 +50,11 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
script does not relabel, since `:z` on a shared directory such as script does not relabel, since `:z` on a shared directory such as
`/var/log` would relabel it for every other service too `/var/log` would relabel it for every other service too
- `native`, Linux - `native`, Linux
- needs systemd; installs `lw` to `/usr/local/bin` unless one is installed - needs systemd; installs `deploy/package/logwisp.service`,
(`--bin FILE`, or `--build` for `make build`), and `logwisp.sysusers` and `logwisp.tmpfiles` under `/etc`, and `--bin FILE`
`deploy/package/logwisp.service`, `logwisp.sysusers` and or the repository's `bin/lw` (`--build` runs `make build` first) to
`logwisp.tmpfiles` under `/etc` `/usr/local/bin`, unless neither flag is given and an `lw` is in
`/usr/local/bin` or `/usr/bin`
- a package's unit (`/usr/lib/systemd/system/logwisp.service`) is used as it - a package's unit (`/usr/lib/systemd/system/logwisp.service`) is used as it
is, with the `lw` its `ExecStart` runs; `--bin` and `--build` then stop the is, with the `lw` its `ExecStart` runs; `--bin` and `--build` then stop the
run, since the service would never run that binary run, since the service would never run that binary
@@ -58,14 +62,17 @@ certificates: `lw tls` makes them ([CLI](cli.md#lw-tls)).
- refuses log and output directories under `/tmp` and `/var/tmp`: the unit's - refuses log and output directories under `/tmp` and `/var/tmp`: the unit's
`PrivateTmp=yes` gives the service its own, empty ones `PrivateTmp=yes` gives the service its own, empty ones
- adds `logwisp.service.d/deploy.conf` when the node needs it: - adds `logwisp.service.d/deploy.conf` when the node needs it:
`ReadWritePaths` for an output outside `/var/log/logwisp`, `ReadWritePaths` for an output other than `/var/log/logwisp` or
`ProtectHome=read-only` for logs or output under `/home`, `/var/lib/logwisp`, `ProtectHome=read-only` for logs or output under
`/home`, `/root` or `/run/user`,
`SupplementaryGroups` for `--log-group`, `CAP_NET_BIND_SERVICE` for a port `SupplementaryGroups` for `--log-group`, `CAP_NET_BIND_SERVICE` for a port
below 1024 below 1024
- `native`, FreeBSD - `native`, FreeBSD
- installs `deploy/package/logwisp.rc` as `/usr/local/etc/rc.d/logwisp`, - installs `lw` to `/usr/local/bin` unless one is there (`--bin` and
creates the `logwisp` user with `pw`, enables it with `sysrc`; LogWisp logs `--build` as on Linux), and `deploy/package/logwisp.rc` as
to syslog under the tag `logwisp` `/usr/local/etc/rc.d/logwisp`; creates the `logwisp` user with `pw` and
adds it to `--log-group`, enables the service with `sysrc`; LogWisp logs to
syslog under the tag `logwisp`
- writes `/usr/local/etc/logwisp/logwisp.toml` - writes `/usr/local/etc/logwisp/logwisp.toml`
- `--jail NAME` writes the files and runs the commands through `jexec`, so - `--jail NAME` writes the files and runs the commands through `jexec`, so
the jail's own symlinks cannot redirect a write onto the host; the jail the jail's own symlinks cannot redirect a write onto the host; the jail
@@ -98,9 +105,9 @@ needs `--force`.
certificate must name the address dialed (an IP SAN for an IP), or the certificate must name the address dialed (an IP SAN for an IP), or the
name given as `--server-name` name given as `--server-name`
- `--auth` on the chain link - `--auth` on the chain link
- `mtls`: edges present `--cert-file`; the aggregator admits the `--allow` - `mtls`: edges present `--cert-file`; the aggregator verifies them with
CNs (none: any certificate the CA issued); an edge's `--allow` pins the `--ca-file` and admits the `--allow` CNs (none: any certificate the CA
aggregator's CN issued); an edge's `--allow` pins the aggregator's CN
- `scram`: the aggregator creates each `--add-user`, its password generated - `scram`: the aggregator creates each `--add-user`, its password generated
into `--secrets-dir`, an absolute path (default `~/logwisp-secrets`), and into `--secrets-dir`, an absolute path (default `~/logwisp-secrets`), and
handed to `lw auth add-user`, natively or through the image; copy each handed to `lw auth add-user`, natively or through the image; copy each
@@ -108,8 +115,9 @@ needs `--force`.
`.pass` is the default `--username` `.pass` is the default `--username`
- either binds the node label to the authenticated identity - either binds the node label to the authenticated identity
- `--sink-auth` on the `http` and `tcp` outputs - `--sink-auth` on the `http` and `tcp` outputs
- `mtls` with `--sink-allow`, or `scram` with `--add-viewer` (a separate - `mtls` with `--ca-file` and `--sink-allow`, or `scram` with `--add-viewer`
`viewers.toml`); viewers log in with `lw auth token` and `lw auth stream` (a separate `viewers.toml`); viewers log in with `lw auth token` and
`lw auth stream`
- without it the outputs admit anyone who reaches them: bind `--listen` to a - without it the outputs admit anyone who reaches them: bind `--listen` to a
trusted address trusted address
- without `--auth` or `--sink-auth`, the auth flags imply their mode: - without `--auth` or `--sink-auth`, the auth flags imply their mode:
@@ -156,10 +164,11 @@ Variants:
- the aggregator on the host itself: drop `--jail logs` - the aggregator on the host itself: drop `--jail logs`
- no daemon per jail: one `standalone` or `edge` on the host tails each jail's - no daemon per jail: one `standalone` or `edge` on the host tails each jail's
logs through the host path (`--log-dir JAIL_PATH/var/log`); the entries then logs through the host path (`--log-dir JAIL_PATH/var/log`); the entries then
carry the host's label and the bare file name, so files of the same name in carry the bare file name and, from an edge, the host's label, so files of
two jails cannot be told apart the same name in two jails cannot be told apart
- a Linux aggregator: run step 2 with `--runtime docker` on that host and point - a Linux aggregator: run step 2 on that host with `--runtime docker` in place
the edges at it; the chain protocol is the same of `--runtime native --jail logs`, and point the edges at it; the chain
protocol is the same
## Linux host with containers ## Linux host with containers
+9 -7
View File
@@ -129,9 +129,10 @@ time, which is exactly what you want in a log hot path.
- Anchors: `^app\.log ` (the text starts with the source), `ERROR$` - Anchors: `^app\.log ` (the text starts with the source), `ERROR$`
- Any characters: `.*exception.*` - Any characters: `.*exception.*`
Remember that TOML basic strings process escapes, so a regex backslash needs TOML strings process escapes, so a regex backslash needs doubling:
doubling: `"\\berror\\b"`. TOML literal strings avoid the issue: `"\\berror\\b"`. LogWisp's TOML reader has no literal (single-quoted)
`'\berror\b'`. strings, so `'\berror\b'` does not load, and a single `\b` is TOML's
backspace: `"\berror\b"` loads and never matches.
Anchors apply to the assembled match text, which begins with the source name, Anchors apply to the assembled match text, which begins with the source name,
so `^ERROR` never matches an entry that has a source; use `\bERROR\b`. so `^ERROR` never matches an entry that has a source; use `\bERROR\b`.
@@ -176,10 +177,11 @@ Each filter reports `type`, `logic`, `pattern_count`, `total_processed`,
## Performance ## Performance
Patterns compile once at startup. Every entry that reaches the filter stage is Patterns compile when a pipeline is built, at startup and on each reload.
evaluated against every filter until one rejects it, so cost scales with the Every entry that reaches the filter stage is evaluated against every filter
number of patterns and their complexity. Prefer literal substrings and simple until one rejects it, and each filter runs all of its patterns whatever its
alternations over broad `.*` wildcards. `logic`, so cost scales with the number of patterns and their complexity.
Prefer literal substrings and simple alternations over broad `.*` wildcards.
Filters log every pattern match and decision at DEBUG: useful when a filter Filters log every pattern match and decision at DEBUG: useful when a filter
misbehaves, expensive on a busy pipeline, where `logging.level` stays at misbehaves, expensive on a busy pipeline, where `logging.level` stays at
+42 -22
View File
@@ -11,8 +11,11 @@ flags = 0
timestamp_format = "" timestamp_format = ""
``` ```
One formatter serves the whole pipeline. Sinks receive an identical payload, One formatter serves the whole pipeline. Sinks receive an identical payload
one record ending in exactly one newline; there is no per-sink formatting. ending in a newline; under `raw` a message's own newlines stay, so a record
can span lines. Apart from the `http` sink's SSE framing, only the console
sink alters what it writes, escaping control characters on a terminal and
painting level names ([console](sinks.md#console)).
When you need two shapes of the same data, run two pipelines, or chain to a When you need two shapes of the same data, run two pipelines, or chain to a
node that formats differently. node that formats differently.
@@ -38,7 +41,10 @@ bypasses formatting and sanitization: the message reaches the sink as the
source produced it, control characters included, with no timestamp, level or source produced it, control characters included, with no timestamp, level or
source prefix added, and a newline appended unless the message ends in one. source prefix added, and a newline appended unless the message ends in one.
An entry that also carries `fields` gets the fields JSON appended verbatim An entry that also carries `fields` gets the fields JSON appended verbatim
after a single space; `raw` never drops data and never re-encodes it. after a single space, so `FlagRaw` (`1`) never drops data or re-encodes it.
Heartbeats format without it, so in `raw` their `include_stats` fields print
in Go's map syntax (`heartbeat map[beat_count:0 type:heartbeat]`), as any
entry's fields do under a non-zero `flags` without `1`.
```toml ```toml
[pipelines.flow.format] [pipelines.flow.format]
@@ -48,14 +54,22 @@ type = "raw"
The closest to verbatim transport is the `console` source, or the `file` The closest to verbatim transport is the `console` source, or the `file`
source with `raw = true`. The whole line, terminator removed, becomes the source with `raw = true`. The whole line, terminator removed, becomes the
message and `fields` stays empty; `raw` writes it back with one `\n`. It is not message and `fields` stays empty; `raw` writes it back with one `\n`. It is not
byte-exact: blank lines are skipped, CRLF becomes LF, and a line over 1 MiB byte-exact: empty lines are skipped, CRLF becomes LF, and a line over 1 MiB,
continues in the next entry. The `file` source's JSON branch splits a line or one the `file` source reads while it is still being written, continues in
into message and fields, so `raw` writes `<msg> <fields>` rather than the the next entry. The `file` source's JSON branch splits a line into message
original object. and fields, so `raw` writes `<msg> <fields>` rather than the original object.
### txt ### txt
Human-readable line output with a timestamp and level. Human-readable lines, `<time> <level> <source> <message>`, any fields after
the message. The message and the fields are each double-quoted when empty or
holding whitespace, a non-printable character, a quote, or a shell or `=`
character (`$`, `;`, `<`, `*`, ...), with `"` and `\` backslash-escaped
inside; the time and source never are:
```
2026-01-02T15:04:05.123Z ERROR edge-01/app.log "connection refused"
```
```toml ```toml
[pipelines.flow.format] [pipelines.flow.format]
@@ -103,8 +117,8 @@ you need to override the defaults.
With `flags = 0` the formatter selects `1` for `type = "raw"` and `6` With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
(timestamp + level) for every other type. `8` is added automatically whenever (timestamp + level) for every other type. `8` is added automatically whenever
an entry carries parseable `fields` and `1` is not set (it shapes `json` only); an entry's `fields` is a non-empty JSON object and `1` is not set (it shapes
`1` always wins. `json` only); `1` always wins.
`txt` and `json` show timestamp and level unless a suppress bit says otherwise `txt` and `json` show timestamp and level unless a suppress bit says otherwise
(suppress wins over show), so `2` and `4` alone change nothing. Examples: (suppress wins over show), so `2` and `4` alone change nothing. Examples:
@@ -123,15 +137,19 @@ Each policy, then what to use it with:
- `txt`: writes `<` and non-printable characters as `<hex>` of their bytes - `txt`: writes `<` and non-printable characters as `<hex>` of their bytes
(`<b>` becomes `<3c>b>`). (`<b>` becomes `<3c>b>`).
- File and console sinks. - File and console sinks.
- `json`: escapes control characters for safe JSON embedding. - `json`: writes control characters as JSON escapes (`\t`, `\u001b`) into
the text, which `type = "json"` quotes again: a reader that decodes the
record gets the escape, never the character.
- `type = "json"`. - `type = "json"`.
- `shell`: strips shell metacharacters, whitespace, and control characters. - `shell`: strips shell metacharacters, whitespace, and control characters.
- Data that will be passed to a command. - Data that will be passed to a command.
Every policy but `raw` replaces invalid UTF-8 with U+FFFD. With Every policy but `raw` replaces invalid UTF-8 with U+FFFD. `type = "json"` is
`type = "raw"` and `flags = 0` no policy applies (`1` bypasses it). When valid JSON under any policy, since its own quoting escapes control characters
`sanitizer_policy` is omitted, it is the policy of the same name as `type`, so and replaces invalid UTF-8; with policy `raw` the decoded message keeps its
the safe pairing is the default. control characters. With `type = "raw"` and `flags = 0` no policy applies (`1`
bypasses it). When `sanitizer_policy` is omitted, it is the policy of the same
name as `type`, so the safe pairing is the default.
> `shell` strips dangerous characters but is **not** sufficient to make a string > `shell` strips dangerous characters but is **not** sufficient to make a string
> safe for shell construction. Pass arguments through `exec` argv instead of > safe for shell construction. Pass arguments through `exec` argv instead of
@@ -150,13 +168,15 @@ downstream parsers and grep patterns need to know.
## Structured Fields ## Structured Fields
When an entry carries `Fields` (raw JSON) and `FlagRaw` is not set, the When an entry's `fields` is a non-empty JSON object and `1` is not set, the
formatter parses it and adds the `StructuredJSON` flag. Fields reach a pipeline formatter parses it and adds `8`; without `1`, any other `fields` value
from the `file` source when a tailed line parses as JSON with a `fields` key, (`[1,2]`, `{}`, `"s"`) is dropped. Fields reach a pipeline from the `file`
from the `tcp_chain` and `http_chain` sources when the upstream entry carried source when a tailed line parses as JSON with a `fields` key, from the
them, and from the heartbeat when `include_stats = true`. In `json` output `tcp_chain` and `http_chain` sources when the upstream entry carried them, and
such an entry is `{"time":…,"level":…,"trace":…,"message":…,"fields":{…}}`; from the heartbeat when `include_stats = true`. In `json` output such an entry
without fields the message sits in `"fields":[…]`. is `{"time":…,"level":…,"trace":…,"message":…,"fields":{…}}`; without fields
the message sits in `"fields":[…]`. In `txt` the fields follow the message in
Go's map syntax, keys sorted: `"disk full" "map[dev:sda pct:99]"`.
## Choosing a Configuration ## Choosing a Configuration
+29 -22
View File
@@ -24,7 +24,7 @@ make build # bin/lw; plain `make` only lists the targets
The Makefile works with GNU make and BSD make alike. Targets: The Makefile works with GNU make and BSD make alike. Targets:
- Build - Build
- `make build`: `bin/lw`, with version, commit and build time from git - `make build`: `bin/lw`, with version and commit from git, and the build time
- `make release`: the same, static (`CGO_ENABLED=0`), `-trimpath`, stripped - `make release`: the same, static (`CGO_ENABLED=0`), `-trimpath`, stripped
- `make dev`: built with the race detector - `make dev`: built with the race detector
- `make wasm`: `bin/lwconf/`, the configuration engine a website drops in: - `make wasm`: `bin/lwconf/`, the configuration engine a website drops in:
@@ -36,8 +36,8 @@ The Makefile works with GNU make and BSD make alike. Targets:
[Architecture](architecture.md#configuration-engine) [Architecture](architecture.md#configuration-engine)
- `make version`: the metadata a build would embed - `make version`: the metadata a build would embed
- `make completion`: regenerate the bash, zsh and fish completion in - `make completion`: regenerate the bash, zsh and fish completion in
`deploy/package/completion/` from `cmd/lw`'s `shorts` and `commands` `deploy/package/completion/` from lw's own tables (its flags, commands
tables; `go test ./cmd/lw` fails while they differ and presets); `go test ./cmd/lw` fails while they differ
- `make clean`: remove `bin/` - `make clean`: remove `bin/`
- Check - Check
- `make test`: the Go tests - `make test`: the Go tests
@@ -79,8 +79,9 @@ checkout.
first, or in one command: `make release install`. It honours `DESTDIR`, first, or in one command: `make release install`. It honours `DESTDIR`,
`PREFIX`, `SYSCONFDIR`, `BINDIR` (default `$PREFIX/bin`, the path the systemd `PREFIX`, `SYSCONFDIR`, `BINDIR` (default `$PREFIX/bin`, the path the systemd
unit and the rc.d script run `lw` from), `MANDIR`, `DOCDIR` and `LICENSEDIR` unit and the rc.d script run `lw` from), `MANDIR`, `DOCDIR` and `LICENSEDIR`
(under `$PREFIX/share` by default) and `SERVICE`; `make help` prints the (under `$PREFIX/share` by default) and `SERVICE`; `make help` prints
values in effect: `PREFIX`, `SYSCONFDIR`, `DESTDIR`, `SERVICE` and `INSTALL_OS` as they are in
effect:
- Linux defaults: `PREFIX=/usr`, `SYSCONFDIR=/etc`. Outside a package - Linux defaults: `PREFIX=/usr`, `SYSCONFDIR=/etc`. Outside a package
manager, prefer `PREFIX=/usr/local`; systemd reads units, sysusers and manager, prefer `PREFIX=/usr/local`; systemd reads units, sysusers and
@@ -190,9 +191,11 @@ rc.conf variables:
`make deb` builds a release and packages what `make install` stages, in `make deb` builds a release and packages what `make install` stages, in
Debian's layout, as `bin/logwisp_VERSION_ARCH.deb`, the architecture taken Debian's layout, as `bin/logwisp_VERSION_ARCH.deb`, the architecture taken
from the binary. A release tag gives the version; an untagged or modified from the binary. A release tag gives the version, and a commit past one
tree's sorts below every release and by build time, so each build upgrades sorts between it and the next release. A modified tree, or one with no tag
the last. `DEB_MAINTAINER` names the maintainer. in its history, adds the build time, so each rebuild upgrades the last; with
no tag, the version sorts below every release. `DEB_MAINTAINER` names the
maintainer.
```bash ```bash
make deb make deb
@@ -213,8 +216,8 @@ sudo systemctl enable --now logwisp
`make arch` builds the PKGBUILD in `deploy/package/arch/` from the working `make arch` builds the PKGBUILD in `deploy/package/arch/` from the working
tree, committed or not, into `bin/arch/`, then prints the `pacman -U` line tree, committed or not, into `bin/arch/`, then prints the `pacman -U` line
that installs the package. It needs `base-devel` and `go`, and runs as a that installs the package. It needs a git checkout, `base-devel`, `git` and
user: `makepkg` refuses root. `go`, and runs as a user: `makepkg` refuses root.
```bash ```bash
make arch make arch
@@ -343,10 +346,10 @@ docker run -d --name logwisp \
``` ```
Without a file lw logs less ([CLI](cli.md#built-in-defaults)): add Without a file lw logs less ([CLI](cli.md#built-in-defaults)): add
`-e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true`; the image `-e LOGWISP_LOGGING_LEVEL=info` (`debug`, with
sets neither, as a variable would override a mounted file's value. Without `-e LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the image sets
`-i`, stdin is `/dev/null`, so a container needs a source other than neither, as a variable would override a mounted file's value. Without `-i`,
`console`. stdin is `/dev/null`, so a container needs a source other than `console`.
- Secrets: only as mounted files, at `/run/secrets` (Docker or Compose - Secrets: only as mounted files, at `/run/secrets` (Docker or Compose
secrets, a Kubernetes secret volume), named by `tls.key_file`, secrets, a Kubernetes secret volume), named by `tls.key_file`,
@@ -354,8 +357,8 @@ docker run -d --name logwisp \
the environment: `docker inspect` and `/proc` show it. Make each file the environment: `docker inspect` and `/proc` show it. Make each file
readable by 65532 only (`chown 65532` and mode `0400`). readable by 65532 only (`chown 65532` and mode `0400`).
- Privileges: keep the image's non-root user, a read-only root filesystem, - Privileges: keep the image's non-root user, a read-only root filesystem,
`--cap-drop ALL` and `no-new-privileges`. Without capabilities a listener `--cap-drop ALL` and `no-new-privileges`. Without capabilities, a port of
needs a port of 1024 or above; publish it on any host port. 1024 or above binds on every runtime; publish it on any host port.
- Listeners bind `0.0.0.0` inside the container; `127.0.0.1` would be - Listeners bind `0.0.0.0` inside the container; `127.0.0.1` would be
unreachable through `-p`. unreachable through `-p`.
- Writable data: a `file` sink or `logging.file` needs a volume writable by - Writable data: a `file` sink or `logging.file` needs a volume writable by
@@ -413,18 +416,22 @@ starting it, run `lw --check -c FILE`; see
## Test Scripts ## Test Scripts
The end-to-end scripts in `test/` run against `bin/lw` and share The end-to-end scripts in `test/` run against `bin/lw` (`LOGWISP_BIN` names
`test/lib.sh`. `make e2e` runs each with `--auto` and reports it as PASS, FAIL another) and share `test/lib.sh`. `make e2e` runs each with `--auto` and
or SKIP (exit 77, e.g. no IPv6 loopback); only a failure fails the target. reports it as PASS, FAIL or SKIP (exit 77, e.g. no IPv6 loopback); only a
failure fails the target.
- Modes: - Modes:
- without flags: start the daemons, print a guide of the ports and one-line - without flags: start the daemons, print a guide of the ports and one-line
commands (run `. test/run-X/env` first for `$LW`, `$CA` and the token commands (where the guide says so, run `. test/run-X/env` first for `$LW`,
helper), and wait for Ctrl-C `$CA` and the token helper), and wait for Ctrl-C. `acl-test.sh`,
`passthrough-test.sh`, `pipe-test.sh` and `preset-test.sh` are one-shot:
either mode runs the checks and exits.
- `--auto`: run the checks and tear down; `--keep` leaves the daemons up - `--auto`: run the checks and tear down; `--keep` leaves the daemons up
after a pass after a pass
- `--follow`: show only WARN, ERROR and key events from the daemons - `--follow`: show only WARN, ERROR and key events from the daemons
- Daemons log only to `test/run*/log/NAME.out`, never to the terminal. - Daemons log only to `log/NAME.out` in the run directory, never to the
terminal.
- Colour only on a terminal; `NO_COLOR` turns it off, `FORCE_COLOR=1` on. - Colour only on a terminal; `NO_COLOR` turns it off, `FORCE_COLOR=1` on.
- Exit status: 0 all passed, 1 a failure, 77 skipped. - Exit status: 0 all passed, 1 a failure, 77 skipped.
- Each script owns a port range and a gitignored run directory: - Each script owns a port range and a gitignored run directory:
+41 -17
View File
@@ -1,4 +1,4 @@
.Dd October 11, 2026 .Dd October 14, 2026
.Dt LW 1 .Dt LW 1
.Os .Os
.Sh NAME .Sh NAME
@@ -53,7 +53,7 @@ joins a console source to a console sink in the raw format, without a filter
or rate limit; a configuration file that or rate limit; a configuration file that
defines no pipelines runs it too. defines no pipelines runs it too.
Each line is written with one newline, whether it ended in a newline, a Each line is written with one newline, whether it ended in a newline, a
carriage return and newline, or nothing; blank lines are skipped, and a line carriage return and newline, or nothing; empty lines are skipped, and a line
over 1 MiB continues in the next one. over 1 MiB continues in the next one.
When standard input is a terminal, When standard input is a terminal,
.Nm .Nm
@@ -74,7 +74,8 @@ Under a supervisor, standard input is usually
.Pa /dev/null : .Pa /dev/null :
a console source there ends at once. a console source there ends at once.
.Pp .Pp
The console source and sink never drop a line. While their pipeline runs, the console source and sink never drop a line;
a stop or reload can drop lines still in flight.
A busy pipeline slows the reading of standard input; a console sink whose A busy pipeline slows the reading of standard input; a console sink whose
output is slow makes its pipeline wait, other sinks included. output is slow makes its pipeline wait, other sinks included.
On a terminal the console sink writes control characters, escape sequences On a terminal the console sink writes control characters, escape sequences
@@ -133,7 +134,10 @@ The options are as follows:
.It Fl c Ar file , Fl \-config Ar file , Fl \-config= Ns Ar file .It Fl c Ar file , Fl \-config Ar file , Fl \-config= Ns Ar file
Read the configuration from Read the configuration from
.Ar file , .Ar file ,
which must exist. which must exist; a relative path that climbs above the current directory
with
.Ql ..
is refused, so give such a path absolute.
The last one given wins. The last one given wins.
.It Fl t , Fl \-check .It Fl t , Fl \-check
Build every pipeline and plugin as a start would, loading options, TLS files Build every pipeline and plugin as a start would, loading options, TLS files
@@ -197,7 +201,7 @@ and the terminal reminder; console sinks still write.
.It Fl \-color Op Ar when .It Fl \-color Op Ar when
Color the level names console sinks write: Color the level names console sinks write:
.Cm auto .Cm auto
.Pq the default: on a terminal, with Ev NO_COLOR unset and Ev TERM not Cm dumb , .Pq the default: on a terminal, with Ev NO_COLOR empty or unset and Ev TERM not Cm dumb ,
.Cm always .Cm always
.Pq what a bare Fl \-color No means .Pq what a bare Fl \-color No means
or or
@@ -562,7 +566,9 @@ and
is required; both may be given. is required; both may be given.
.El .El
.Pp .Pp
A listener needs no files: A listener with
.Cm tls.enabled=true
needs no files:
.Cm tls.self_signed=true , .Cm tls.self_signed=true ,
or or
.Cm tls.issuer_cert_file .Cm tls.issuer_cert_file
@@ -601,8 +607,8 @@ when it exists; otherwise one is generated and written there, or printed
once to standard output. once to standard output.
.Fl \-generate .Fl \-generate
replaces an existing password without reading one. replaces an existing password without reading one.
A missing or empty credentials file is created with mode 0600; an existing A missing credentials file is created with mode 0600; an existing one keeps
one keeps its owner and mode. its owner and mode, including an empty one made beforehand to choose them.
.It Cm remove-user Fl \-credentials Ar file Fl u Ar name .It Cm remove-user Fl \-credentials Ar file Fl u Ar name
Remove a user. Remove a user.
The last user cannot be removed. The last user cannot be removed.
@@ -694,16 +700,28 @@ counting from 1, adds more, as in
.It Ev LOGWISP_FORMAT , LOGWISP_RATE_LIMIT , LOGWISP_HEARTBEAT .It Ev LOGWISP_FORMAT , LOGWISP_RATE_LIMIT , LOGWISP_HEARTBEAT
The pipeline's format, rate limit and heartbeat specifications. The pipeline's format, rate limit and heartbeat specifications.
.It Ev NO_COLOR , TERM .It Ev NO_COLOR , TERM
Set to anything, Set to anything but the empty string,
.Ev NO_COLOR .Ev NO_COLOR
turns turns
.Cm auto .Cm auto
color off, as does color off, as does
.Ev TERM=dumb . .Ev TERM=dumb ,
which also keeps the
.Fl \-tui
screen from starting.
.It Ev LC_ALL , LC_CTYPE , LANG
The first one set names the locale; one other than UTF-8 makes the
.Fl \-tui
screen draw in ASCII.
.It Ev HOME .It Ev HOME
Where Where
.Pa ~/.config/logwisp/logwisp.toml .Pa ~/.config/logwisp/logwisp.toml
is looked for. is looked for, and what a leading
.Ql ~
names in a plugin's directory and file options, a preset's
.Cm path
and
.Fl \-logging.file.directory .
.El .El
.Pp .Pp
Secrets never go in the environment: TLS keys, credentials files and Secrets never go in the environment: TLS keys, credentials files and
@@ -761,8 +779,12 @@ a full one is renamed with its time, as
.Nm .Nm
exits 0 at the end of input, after a clean shutdown, and for exits 0 at the end of input, after a clean shutdown, and for
.Fl \-version , .Fl \-version ,
.Fl \-schema ,
help, help,
.Fl \-dump .Fl \-dump ,
a
.Fl \-tui
screen left by printing or quitting,
and a configuration and a configuration
.Fl \-check .Fl \-check
accepts; 1 on a failure, such as a configuration that does not load, or a accepts; 1 on a failure, such as a configuration that does not load, or a
@@ -796,7 +818,9 @@ Shut down gracefully.
.It Dv SIGHUP , SIGUSR1 .It Dv SIGHUP , SIGUSR1
Reread the configuration file, certificates and credentials, and rebuild Reread the configuration file, certificates and credentials, and rebuild
every pipeline. every pipeline.
An invalid configuration leaves the running service untouched. An invalid configuration leaves the running service untouched; one that builds
but fails to start, as on a port another process holds, leaves nothing running
until a later reload succeeds.
Before the service runs, Before the service runs,
.Dv SIGHUP .Dv SIGHUP
is ignored, as under is ignored, as under
@@ -831,9 +855,9 @@ $ tail \-F app.log | lw \-\-format txt
Serve standard input as a live stream on the loopback address only, since Serve standard input as a live stream on the loopback address only, since
the the
.Cm http .Cm http
sink listens on every address by default, and read it from another terminal sink listens on every IPv4 address by default, and read it from another
as server-sent events; a client gets the lines that arrive while it is terminal as server-sent events; a client gets the lines that arrive while it
connected: is connected:
.Bd -literal -offset indent .Bd -literal -offset indent
$ journalctl \-f | lw \-\-sink http:host=127.0.0.1,port=8080 $ journalctl \-f | lw \-\-sink http:host=127.0.0.1,port=8080
$ curl \-N http://127.0.0.1:8080/stream $ curl \-N http://127.0.0.1:8080/stream
@@ -924,7 +948,7 @@ generate the password into a file, then reload the server:
.Xr daemon 8 .Xr daemon 8
.Pp .Pp
The guides to configuration, plugins, security and deployment, installed in The guides to configuration, plugins, security and deployment, installed in
.Pa share/doc/logwisp .Pa share/doc/logwisp/doc
under the installation prefix and kept at under the installation prefix and kept at
.Lk https://github.com/lixenwraith/logwisp/tree/main/doc .Lk https://github.com/lixenwraith/logwisp/tree/main/doc
.Sh SECURITY CONSIDERATIONS .Sh SECURITY CONSIDERATIONS
+23 -16
View File
@@ -8,8 +8,8 @@ see [Chaining](chaining.md).
Every listener and dialer keeps strictly to the family of its `host`: Every listener and dialer keeps strictly to the family of its `host`:
- IPv4 literal (`127.0.0.1`, the default `0.0.0.0`): IPv4 only (`tcp4`). An - IPv4 literal (`127.0.0.1`, the default `0.0.0.0`): IPv4 only (`tcp4`). A
empty `host` is the IPv4 wildcard, as `0.0.0.0`. listener's empty `host` is the IPv4 wildcard, as `0.0.0.0`.
- IPv6 literal (`::1`, `::`, `fe80::1%eth0`): IPv6 only (`tcp6`). - IPv6 literal (`::1`, `::`, `fe80::1%eth0`): IPv6 only (`tcp6`).
- `::` takes no IPv4 connections, on Linux and FreeBSD alike, whatever the - `::` takes no IPv4 connections, on Linux and FreeBSD alike, whatever the
system's `bindv6only` default. system's `bindv6only` default.
@@ -38,8 +38,9 @@ Every listener and dialer keeps strictly to the family of its `host`:
- SCRAM throttling counts an IPv6 client by its /64, so the hosts on a link - SCRAM throttling counts an IPv6 client by its /64, so the hosts on a link
share one link-local budget per interface (zone): each can pick any share one link-local budget per interface (zone): each can pick any
`fe80::/64` address. `fe80::/64` address.
- A peer with a zone never matches `auth.trusted_proxies`: put the proxy on - A peer with a zone never matches `auth.trusted_proxies` or
loopback or a routed address. `acl.proxy_from`: put the proxy on loopback or a routed address.
`acl.allow` and `deny` match it without its zone.
## Network Plugins ## Network Plugins
@@ -79,7 +80,9 @@ deadline where the default is `0` (none); elsewhere it selects the default.
under `scram` the whole login. under `scram` the whole login.
- `read_timeout_ms`, default `0` (none): idle time between entries. - `read_timeout_ms`, default `0` (none): idle time between entries.
- `http_chain` source - `http_chain` source
- `read_timeout_ms`, default `30000`: reading a whole request body. - `read_timeout_ms`, default `30000`: reading a whole request, headers and
body; also how long an idle keep-alive connection waits for its next
request.
- `tcp_chain` sink - `tcp_chain` sink
- `dial_timeout_ms`, default `5000`: TCP connect. - `dial_timeout_ms`, default `5000`: TCP connect.
- `write_timeout_ms`, default `5000`: one line write. - `write_timeout_ms`, default `5000`: one line write.
@@ -89,14 +92,16 @@ deadline where the default is `0` (none); elsewhere it selects the default.
Fixed, non-configurable bounds: Fixed, non-configurable bounds:
- TLS handshake: 10 s, on all TLS listeners and dialers. - TLS handshake: 10 s, on all TLS listeners and dialers; on the `http_chain`
source, `read_timeout_ms` when shorter.
- HTTP read-header timeout: 10 s, on the `http` sink and `http_chain` source. - HTTP read-header timeout: 10 s, on the `http` sink and `http_chain` source.
- HTTP server shutdown grace: 2 s on the `http_chain` source and the `http` - HTTP server shutdown grace: 2 s on the `http_chain` source and the `http`
sink; on the `http` sink, `write_timeout_ms` instead when set below 2 s. sink; on the `http` sink, `write_timeout_ms` instead when set below 2 s.
- Single entry line: at most 1 MiB, on chain listeners. - Single entry line: at most 1 MiB, on chain listeners.
- SCRAM login: 10 s, on the `tcp` sink, chain sinks, `lw auth`, and each - SCRAM login: 10 s, on the `tcp` sink, chain sinks, `lw auth`, and each
`/auth` request. `/auth` request.
- SCRAM line or `/auth` body: at most 4 KiB, on `scram` listeners and dialers. - Hello and SCRAM lines, `/auth` bodies: at most 4 KiB, on `tcp_chain` sources
and on `scram` listeners and dialers.
The `http` sink deliberately leaves the server's `WriteTimeout` unset, since it The `http` sink deliberately leaves the server's `WriteTimeout` unset, since it
would terminate long-lived SSE streams; per-event deadlines come from would terminate long-lived SSE streams; per-event deadlines come from
@@ -108,9 +113,10 @@ would terminate long-lived SSE streams; per-event deadlines come from
`tcp_chain` source, and concurrent streams on the `http` sink (other requests `tcp_chain` source, and concurrent streams on the `http` sink (other requests
are not counted); `0` means unlimited. are not counted); `0` means unlimited.
- Admission is a load-then-check, so a burst can over-admit by roughly one. - A connection or stream counts only after the check that admits it, so a
- On the `tcp` sink and `tcp_chain` source the count is taken at accept, so it burst can over-admit by several, more the larger the burst.
bounds concurrent TLS handshakes as well as established sessions. - On the `tcp` sink and `tcp_chain` source a connection counts before its TLS
handshake, so the cap bounds handshakes in progress as well as sessions.
- Over the limit a TCP connection is closed and counted in `rejected_conns`; - Over the limit a TCP connection is closed and counted in `rejected_conns`;
an `http` stream request is answered `503` and counted in an `http` stream request is answered `503` and counted in
`rejected_clients`, its connection kept. `rejected_clients`, its connection kept.
@@ -155,10 +161,11 @@ backoff_min_ms = 500
backoff_max_ms = 30000 backoff_max_ms = 30000
``` ```
The connection is established lazily, so an edge node starts cleanly even when The connection is established lazily, when the sink first has something to
its relay is down and connects as soon as the relay appears. The `tcp_chain` send, so an edge node starts cleanly even when its relay is down, and connects
sink reports `connected` and `reconnects`; the `http_chain` sink reports at its next attempt once the relay is up. The `tcp_chain` sink reports
`request_errors` and `dropped_batches`. `connected` and `reconnects`; the `http_chain` sink reports `request_errors`
and `dropped_batches`.
Server-side sinks (`tcp`, `http`) do not reconnect; clients are expected to Server-side sinks (`tcp`, `http`) do not reconnect; clients are expected to
retry. Browsers reconnect SSE streams automatically. retry. Browsers reconnect SSE streams automatically.
@@ -196,7 +203,7 @@ preamble, headers, and entry encoding.
- The listener's `acl` refused it: `acl_denied` or `acl_limited` rises, and - The listener's `acl` refused it: `acl_denied` or `acl_limited` rises, and
its log has `Connection refused by acl` at WARN with a `reason`, once a its log has `Connection refused by acl` at WARN with a `reason`, once a
minute at most. The dialer sees `EOF` or `connection reset by peer` (curl: minute at most. The dialer sees `EOF` or `connection reset by peer` (curl:
exit 52 or 56). exit 52 or 56; over TLS, 35).
- `address rules` — `allow` or `deny` excludes the client: the one a PROXY - `address rules` — `allow` or `deny` excludes the client: the one a PROXY
header names (`remote_addr`, the proxy as `peer_addr`), else the socket header names (`remote_addr`, the proxy as `peer_addr`), else the socket
peer. Behind the `http` sink's `auth.trusted_proxies` that is the proxy: peer. Behind the `http` sink's `auth.trusted_proxies` that is the proxy:
@@ -337,7 +344,7 @@ preamble, headers, and entry encoding.
batch is sent. batch is sent.
**Entries arriving under an unexpected node label** **Entries arriving under an unexpected node label**
- `auth.node_binding` defaults to `force` with any `auth` block, which - `auth.node_binding` defaults to `force` under `mtls` or `scram`, which
relabels every entry with the sender's certificate identity or username. If relabels every entry with the sender's certificate identity or username. If
a dashboard suddenly shows a different label, that is why. Use a dashboard suddenly shows a different label, that is why. Use
`node_binding = "assert"` to keep upstream origin labels on relay-to-relay `node_binding = "assert"` to keep upstream origin labels on relay-to-relay
+43 -21
View File
@@ -25,7 +25,8 @@ A supervisor's stdin is `/dev/null` (systemd's default, `docker run` without
`-i`), where a console source ends at once. So a service defines its `-i`), where a console source ends at once. So a service defines its
pipelines: a file without `[[pipelines]]` runs the built-in stdin-to-stdout pipelines: a file without `[[pipelines]]` runs the built-in stdin-to-stdout
pipeline and exits 0 at once, which `Restart=on-failure` does not restart, and pipeline and exits 0 at once, which `Restart=on-failure` does not restart, and
a container configured through variables needs a `LOGWISP_SOURCE`. a container configured through variables needs a `LOGWISP_SOURCE`, or a
`LOGWISP_PRESET` with a source of its own (`aggregator`, or one given a `path`).
**systemd** **systemd**
@@ -50,10 +51,12 @@ sudo service logwisp status
auto_reload = true auto_reload = true
``` ```
or send a signal: or send `SIGHUP` or `SIGUSR1`; the packaged unit and rc.d script send `SIGHUP`
on reload:
```bash ```bash
kill -HUP $(pidof lw) kill -HUP $(pidof lw)
sudo systemctl reload logwisp # rc.d: sudo service logwisp reload
``` ```
Signals reread the selected TOML file even when `auto_reload` is disabled. Signals reread the selected TOML file even when `auto_reload` is disabled.
@@ -67,10 +70,13 @@ running`.
What reload does *not* do: What reload does *not* do:
- Re-apply `logging.*`; application logging is configured once at startup. - Re-apply `logging.*`, `quiet` or `auto_reload`; they take effect at startup
only.
- Preserve connections. Listeners close and reopen, and every SSE, TCP, and - Preserve connections. Listeners close and reopen, and every SSE, TCP, and
chain client is disconnected. Chain sinks reconnect on their own backoff; chain client is disconnected. Chain sinks reconnect on their own backoff;
browsers reconnect SSE automatically; raw TCP consumers must retry themselves. browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
- Keep bearer tokens. Each reload draws a new token key, so every token is
revoked: run `lw auth token` again, and browser viewers log in again.
- Notice rotated certificates on its own: certificate files are read at plugin - Notice rotated certificates on its own: certificate files are read at plugin
construction and `auto_reload` watches only the configuration file, so construction and `auto_reload` watches only the configuration file, so
rotation requires `SIGHUP`. rotation requires `SIGHUP`.
@@ -179,6 +185,11 @@ Each counter, where it is reported, and what a rise means:
- `reconnects` (`tcp_chain` sink): an unstable link or a flapping downstream - `reconnects` (`tcp_chain` sink): an unstable link or a flapping downstream
- `dropped_batches` (`http_chain` sink): downstream is rejecting batches - `dropped_batches` (`http_chain` sink): downstream is rejecting batches
permanently permanently
- `request_errors` (`http_chain` sink): failed posts, each retried under
backoff unless refused for good
- `rejected_requests` (`http_chain` source): batches refused for auth,
protocol version or node binding, or a body over `max_body_bytes`, holding
a line over 1 MiB, or cut short
- `synthesized` (chain sinks): events reach the sink without structure - `synthesized` (chain sinks): events reach the sink without structure
- `acl_denied`, `acl_limited` (a listener whose `acl` block sets a rule or - `acl_denied`, `acl_limited` (a listener whose `acl` block sets a rule or
limit; the `http` status always): connections or requests the address limit; the `http` status always): connections or requests the address
@@ -189,10 +200,10 @@ Each counter, where it is reported, and what a rise means:
LogWisp's own log goes to stderr by default. Without a configuration file lw LogWisp's own log goes to stderr by default. Without a configuration file lw
logs less ([CLI](cli.md#built-in-defaults)), so a container configured only logs less ([CLI](cli.md#built-in-defaults)), so a container configured only
through variables passes `LOGWISP_LOGGING_LEVEL=info` and through variables passes `LOGWISP_LOGGING_LEVEL=info` (`debug`, with
`LOGWISP_STATUS_REPORTER=true`; the image sets neither, as a variable would `LOGWISP_STATUS_REPORTER=true`, for the 30 s statistics); the image sets
override a mounted file's value. `lw-deploy.sh` always passes `-c`, so its neither, as a variable would override a mounted file's value. `lw-deploy.sh`
nodes log at `info`. To files: always passes `-c`, so its nodes log at `info`. To files:
```toml ```toml
[logging] [logging]
@@ -268,33 +279,42 @@ pipeline `total_dropped_by_sink` (sink backed up?), sink `total_processed`.
**A pipeline stalls behind a console sink** **A pipeline stalls behind a console sink**
A console sink never drops: when its output is slow (a paused terminal, a full A console sink never drops while its pipeline runs: when its output is slow
pipe, a stalled log collector) its pipeline waits, and the pipeline's other (a paused terminal, a full pipe, a stalled log collector) its pipeline waits,
sinks with it. Stdin is read slower; other sources queue, then drop and count and the pipeline's other sinks with it. Stdin is read slower; other sources
`dropped_entries`. Where a service's stdout may stall, give it a `file` sink queue, then drop and count `dropped_entries`. Where a service's stdout may
instead. A closed stdout pipe ends lw with `SIGPIPE`. stall, give it a `file` sink instead. A closed stdout pipe ends lw with
`SIGPIPE`.
**File source reads nothing** **File source reads nothing**
- The watcher seeks to end-of-file on start; only content appended afterwards is - A watcher seeks to end-of-file when its file is first found, at start or by
read. Positions are in memory, so a restart re-seeks to end and anything a later scan, so a new file's lines written before that scan are not read.
written during the downtime is lost. `from = "start"` reads each file whole Positions are in memory, so a restart or a reload re-seeks to end, and
instead, and replays it on every restart. anything written meanwhile is lost. `from = "start"` reads each file whole
instead, and replays it on every restart and reload.
- `pattern` is a filename glob with `*` and `?` only, and matching is not - `pattern` is a filename glob with `*` and `?` only, and matching is not
recursive. recursive.
- `check_interval_ms` governs how quickly a *new file* is noticed; tailing an - `check_interval_ms` governs how quickly a *new file* is noticed; tailing an
open file polls at a fixed 100 ms. open file polls at a fixed 100 ms.
- lw's log says why: INFO `Waiting for directory` while the directory is
missing, WARN `Cannot read directory` or `Cannot read file` when permissions
or a service sandbox refuse it, and WARN `Not following lw's own output` for
the file lw's stdout or stderr goes to.
**High memory use** **High memory use**
Buffers are bounded, so unbounded growth almost always means many buffers: Buffers are bounded, so unbounded growth almost always means many buffers:
count sinks × `buffer_size`, plus clients × `client_buffer_size`. A `tcp_chain` count each source's and sink's `buffer_size` (a fixed 1000 on the `file`,
sink blocked on an unreachable downstream also holds its full input queue. `random` and `null` sources and the `null` sink), clients ×
`client_buffer_size`, and each `http` sink's `replay_lines`. A `tcp_chain` sink
blocked on an unreachable downstream also holds its full input queue.
**Chain link not delivering** **Chain link not delivering**
Check `connected` and `reconnects` on a `tcp_chain` sink, `parse_errors` on the Check `connected` and `reconnects` on a `tcp_chain` sink, `request_errors` on
source, and remember `http_chain` waits up to `flush_interval_ms`. For TLS an `http_chain` sink, `rejected_requests` (`http_chain`) and `parse_errors` on
the source, and remember `http_chain` waits up to `flush_interval_ms`. For TLS
problems see [Networking](networking.md#troubleshooting). problems see [Networking](networking.md#troubleshooting).
**Environment variable override has no effect** **Environment variable override has no effect**
@@ -342,7 +362,9 @@ than masking parts of them. See [Filters](filters.md#common-recipes).
1. Run the new binary's `lw --check -c FILE` against the current 1. Run the new binary's `lw --check -c FILE` against the current
configuration: unknown or renamed keys fail there. configuration: unknown or renamed keys fail there.
2. Stop the old process, install the new binary, start it. 2. Stop the old process, install the new binary, start it. A `self_signed`
listener makes a new key at every start: give its dialers the new
`pin_sha256` from its WARN line.
3. Confirm each pipeline started and that counters are advancing. 3. Confirm each pipeline started and that counters are advancing.
**Backup** **Backup**
+58 -46
View File
@@ -26,10 +26,10 @@ configure it, and — equally important — what it does not yet do.
[Browsers behind a TLS-terminating proxy](#browsers-behind-a-tls-terminating-proxy). [Browsers behind a TLS-terminating proxy](#browsers-behind-a-tls-terminating-proxy).
- Server pinning by dialers: certificate identity (`mtls`), bound certificate - Server pinning by dialers: certificate identity (`mtls`), bound certificate
(`scram`). (`scram`).
- Address rules and per-client connection and request limits on every - Address rules and per-client limits (connections on every listener, a
listener, applied before TLS to the client a PROXY header (v1 or v2) from a request rate on HTTP listeners), applied before TLS to the client a PROXY
listed L4 proxy names, and per request to the client an L7 proxy forwards: header (v1 or v2) from a listed L4 proxy names, and per request to the
see [The ACL Block](#the-acl-block). client an L7 proxy forwards: see [The ACL Block](#the-acl-block).
- Startup warnings for expiring certificates and risky settings: see - Startup warnings for expiring certificates and risky settings: see
[Startup Warnings](#startup-warnings). [Startup Warnings](#startup-warnings).
- Unknown configuration keys rejected: a typo in `tls`, `auth`, `acl` or a - Unknown configuration keys rejected: a typo in `tls`, `auth`, `acl` or a
@@ -40,8 +40,8 @@ configure it, and — equally important — what it does not yet do.
comments; comments;
- a string in the file is one list entry, so a filter pattern holding a - a string in the file is one list entry, so a filter pattern holding a
comma is not split into patterns that match nothing; comma is not split into patterns that match nothing;
- the file is opened without blocking, so a FIFO swapped in for it cannot - the file must be a regular file and is opened without blocking, so a
hang the watcher or a `SIGHUP` reload; FIFO in its place fails a reload rather than hanging it;
- a permission change on the file is reported once and does not stop - a permission change on the file is reported once and does not stop
auto-reload; auto-reload;
- parse and conversion errors name the key, never the value, so a secret - parse and conversion errors name the key, never the value, so a secret
@@ -60,12 +60,15 @@ configure it, and — equally important — what it does not yet do.
and `auth_rejected` only. and `auth_rejected` only.
- More than one identity mode per listener: `auth.identity` takes one of - More than one identity mode per listener: `auth.identity` takes one of
`cn`, `san_dns`, `san_uri`, `san_email`. `cn`, `san_dns`, `san_uri`, `san_email`.
- One read of the configuration file: lw's unknown-key check opens it again,
blocking, so a FIFO swapped in between the two reads hangs that load.
Two credentials are supported. Certificates (`mtls`) are the one the transport Two credentials are supported. Certificates (`mtls`) are the one the transport
already carries: the `tls` block establishes that a peer chains to your CA, and already carries: the `tls` block establishes that a peer chains to your CA, and
the `auth` block decides which of those peers may do what. Passwords (`scram`) the `auth` block decides which of those peers may do what. Passwords (`scram`)
need no client PKI: the listener holds verifiers, never passwords, and every need no client PKI: a credentials file holds verifiers, never passwords, and a
login is bound to the listener's certificate. login is bound to the listener's certificate unless the listener sits behind
`trusted_proxies`.
## The TLS Block ## The TLS Block
@@ -139,7 +142,8 @@ built:
`ca_file`, `server_name`, `insecure_skip_verify` or `pin_sha256` on a `ca_file`, `server_name`, `insecure_skip_verify` or `pin_sha256` on a
listener listener
- `client_auth = true` with no `client_ca_file` - `client_auth = true` with no `client_ca_file`
- only one of `cert_file` / `key_file` - only one of `cert_file` / `key_file`, or of `issuer_cert_file` /
`issuer_key_file`
- a `min_version` that is neither `"1.2"` nor `"1.3"` - a `min_version` that is neither `"1.2"` nor `"1.3"`
- a key the plugin does not know, at any depth: `unknown key "tls.enabeld"`. A - a key the plugin does not know, at any depth: `unknown key "tls.enabeld"`. A
misspelled option must not silently leave a protection switched off misspelled option must not silently leave a protection switched off
@@ -212,8 +216,8 @@ Options by where they apply, each with its default:
Under `mtls` the identity is one string pulled from the peer's verified leaf Under `mtls` the identity is one string pulled from the peer's verified leaf
certificate; under `scram` it is the username. The handshake has already certificate; under `scram` it is the username. The handshake has already
checked the chain, signature, and validity window, so this is pure field checked the chain, signature, and validity window (on a dialer with
selection. `pin_sha256`, only the key), so this is pure field selection.
Modes, each with the field it reads and its typical use: Modes, each with the field it reads and its typical use:
@@ -229,17 +233,17 @@ identity is a refusal, not an empty match.
### The allow list ### The allow list
`allow` is an exact-match set; `allow_patterns` holds RE2 patterns. An identity `allow` is an exact-match set; `allow_patterns` holds RE2 patterns. An identity
passes if it appears in either. Under `scram` the credentials file is the allow passes if it appears in either. Under `scram` the listener's users are the
list, and both keys are refused. allow list, and both keys are refused.
Leaving **both** empty under `type = "mtls"` admits any identity the CA vouches Leaving **both** empty under `type = "mtls"` admits any identity the CA vouches
for. That is deliberate — it is how you enable node binding without enumerating for. That is deliberate — it is how you enable node binding without enumerating
a whole fleet — but it is announced rather than silent: a whole fleet — but it is announced rather than silent:
``` ```
WARN msg="Auth policy admits any identity the configured CA vouches for" WARN msg "Auth policy admits any identity the configured CA vouches for"
component=tcp_chain_source instance_id=in_tcp component tcp_chain_source instance_id in_tcp
hint="set auth.allow or auth.allow_patterns to authorize named peers" hint "set auth.allow or auth.allow_patterns to authorize named peers"
``` ```
Anchor your patterns. `allow_patterns = ["edge-\\d{2}"]` matches Anchor your patterns. `allow_patterns = ["edge-\\d{2}"]` matches
@@ -275,8 +279,9 @@ attribution survives.
When binding is active the source says so at startup: When binding is active the source says so at startup:
``` ```
INFO msg="Node labels bound to peer identity; trust_node is ignored" INFO msg "Node labels bound to peer identity; trust_node is ignored"
component=tcp_chain_source node_binding=force trust_node=true component tcp_chain_source instance_id in_tcp
node_binding force trust_node true
``` ```
### Dialer-side pinning ### Dialer-side pinning
@@ -284,7 +289,8 @@ INFO msg="Node labels bound to peer identity; trust_node is ignored"
Under `mtls`, the block on a chain sink pins the *server's* identity. Hostname Under `mtls`, the block on a chain sink pins the *server's* identity. Hostname
verification already proves the server holds a certificate valid for the address verification already proves the server holds a certificate valid for the address
you dialed; pinning additionally requires that certificate to name an identity you dialed; pinning additionally requires that certificate to name an identity
you listed. you listed. With `pin_sha256` the key's holder writes that certificate, so the
pin is the whole check.
```toml ```toml
[pipelines.plugin_sinks.config.auth] [pipelines.plugin_sinks.config.auth]
@@ -319,8 +325,9 @@ files (`credentials_file`, `password_file`) when the plugin is built:
`identity` but no `tls.client_auth` `identity` but no `tls.client_auth`
- a `scram` dialer without `username` and `password_file`, with - a `scram` dialer without `username` and `password_file`, with
`credentials_file`, `token_lifetime_ms`, `trusted_proxies` or `identity` `credentials_file`, `token_lifetime_ms`, `trusted_proxies` or `identity`
(server pinning is `mtls` only), or with a password file that is empty or (server pinning is `mtls` only)
over 1024 bytes - when the plugin is built: a `password_file` holding no password or over
1024 bytes, or under 8 bytes for a listener's one user
- `trusted_proxies` on anything but the `http` sink, with `identity`, or with - `trusted_proxies` on anything but the `http` sink, with `identity`, or with
an entry that is neither an address nor a CIDR; `login_page` or an entry that is neither an address nor a CIDR; `login_page` or
`viewer_page` without `trusted_proxies`, `viewer_page` without `login_page` `viewer_page` without `trusted_proxies`, `viewer_page` without `login_page`
@@ -576,10 +583,10 @@ allow = ["relay.example.org"]
Startup logs report the transport flags and the compiled policy: Startup logs report the transport flags and the compiled policy:
``` ```
INFO msg="TCP chain source initialized" ... tls=true mtls=true INFO msg "TCP chain source initialized" ... tls true mtls true
auth="mtls identity=cn allow=[2 exact, 0 pattern(s)] node_binding=force" auth "mtls identity=cn allow=[2 exact, 0 pattern(s)] node_binding=force"
INFO msg="TCP chain sink initialized" ... tls=true mtls=true INFO msg "TCP chain sink initialized" ... tls true mtls true
auth="mtls identity=cn allow=[1 exact, 0 pattern(s)] node_binding=none" auth "mtls identity=cn allow=[1 exact, 0 pattern(s)] node_binding=none"
``` ```
A client that presents no certificate is refused during the handshake A client that presents no certificate is refused during the handshake
@@ -588,8 +595,8 @@ A client whose certificate is valid but whose identity is not authorized gets
past the handshake and is refused by the policy: past the handshake and is refused by the policy:
``` ```
WARN msg="Connection rejected by auth policy" component=tcp_chain_source WARN msg "Connection rejected by auth policy" component tcp_chain_source ...
remote_addr=<addr> error="auth: refused: identity \"edge-99\" is not allowed" remote_addr <addr> error "auth: refused: identity \"edge-99\" is not allowed"
``` ```
Policy rejections are counted in the plugin's `auth_rejected` statistic; the Policy rejections are counted in the plugin's `auth_rejected` statistic; the
@@ -627,8 +634,8 @@ username = "edge-01"
password_file = "/etc/logwisp/edge-01.pass" password_file = "/etc/logwisp/edge-01.pass"
``` ```
Startup reports `auth="scram users=2 certificate=independent node_binding=force"` Startup reports `auth "scram users=2 certificate=independent node_binding=force"`
on such a chain source, and `auth="scram user=edge-01"` on the dialer. on such a chain source, and `auth "scram user=edge-01"` on the dialer.
### Mechanism ### Mechanism
@@ -670,7 +677,8 @@ requires `tls.client_auth`) binds the two: the certificate's identity field must
equal the username, so every peer needs its own certificate *and* its own equal the username, so every peer needs its own certificate *and* its own
password, like `clientcert=verify-full`. A mismatch fails the login; an HTTP password, like `clientcert=verify-full`. A mismatch fails the login; an HTTP
request whose certificate does not match its token's user gets `403`. Startup request whose certificate does not match its token's user gets `403`. Startup
shows the mode as `certificate=independent` or `certificate=cn=username`. shows the mode as `certificate=independent`, or as the bound field, such as
`certificate=cn=username`.
### Tokens ### Tokens
@@ -717,12 +725,14 @@ stored_key = "<base64>"
username = "edge-01" username = "edge-01"
``` ```
Write it with [`lw auth add-user`](cli.md#lw-auth) rather than by Write it with [`lw auth add-user`](cli.md#lw-auth) rather than by hand. A
hand. The whole file is validated when the plugin is built: a `decoy_key` of at username, in the file or in `auth.username`, is 1 to 256 bytes of UTF-8
least 32 bytes, at least one user, unique names, no unknown keys, and one Argon2 without `,`, `=` or a control character. The whole file is validated when the
profile and salt length for every user, since mixed profiles would tell a prober plugin is built: a `decoy_key` of at least 32 bytes, at least one user, unique
which users exist. `decoy_key` keeps unknown-user challenges stable across names, no unknown keys, and one Argon2 profile and salt length for every user,
restarts and edits; the CLI creates it once and preserves it. A single probe since mixed profiles would tell a prober which users exist. `decoy_key` keeps
unknown-user challenges stable across restarts and edits; the CLI creates it
once and preserves it. A single probe
cannot tell a real user from an unknown one, but a prober polling a name sees cannot tell a real user from an unknown one, but a prober polling a name sees
its salt change when the user is added, rotated or removed. The file is read its salt change when the user is added, rotated or removed. The file is read
only when the plugin is built and `auto_reload` does not watch it: send only when the plugin is built and `auto_reload` does not watch it: send
@@ -756,7 +766,7 @@ edge. `password_file` is read by what it names, on either side:
on that terminal, never in the log or on stderr: logs persist, are on that terminal, never in the log or on stderr: logs persist, are
forwarded, and stderr is often a file. forwarded, and stderr is often a file.
- With no terminal (systemd, cron) the plugin fails. `lw ... &` stops on - With no terminal (systemd, cron) the plugin fails. `lw ... &` stops on
`SIGTTIN` at the prompt. `SIGTTOU` at the prompt.
- Under `lw --tui`'s `t`, the screen asks in a masked dialog instead; a - Under `lw --tui`'s `t`, the screen asks in a masked dialog instead; a
generated password stays shown until a key closes it. generated password stays shown until a key closes it.
@@ -865,9 +875,9 @@ location /logs/ {
- The token lives in a page variable only, never in storage or the URL. It - The token lives in a page variable only, never in storage or the URL. It
lasts as long as the page and at most the token lifetime: a reload, another lasts as long as the page and at most the token lifetime: a reload, another
tab, or a reconnect after expiry asks again; sign out revokes it. tab, or a reconnect after expiry asks again; sign out revokes it.
- The login page says that cookies are unavailable and links to the viewer. - The login page says that cookies are unavailable and, when the sink serves
A page not served over HTTPS keeps no `Secure` cookie either: both pages the viewer, links to it. A page not served over HTTPS keeps no `Secure`
name HTTPS instead and offer no sign-in. cookie either: both pages name HTTPS instead and offer no sign-in.
- **Integrating `scram.js`.** A site with its own CSP can copy `scram.js` from - **Integrating `scram.js`.** A site with its own CSP can copy `scram.js` from
`internal/sink/http/web/` into its bundle: one dependency-free ES module in `internal/sink/http/web/` into its bundle: one dependency-free ES module in
plain JS (no WebAssembly, so no `'wasm-unsafe-eval'`), where `base` is the plain JS (no WebAssembly, so no `'wasm-unsafe-eval'`), where `base` is the
@@ -1007,10 +1017,11 @@ layer.
identities you list may connect, so one CA can serve several trust domains and identities you list may connect, so one CA can serve several trust domains and
a single peer can be withdrawn without touching the others. a single peer can be withdrawn without touching the others.
**`auth` with `type = "scram"`** — a password check, per listener: only the **`auth` with `type = "scram"`** — a password check, per listener: only its
users in its credentials file may connect, the login cannot be relayed through users (a credentials file's, or its one `username`) may connect, the login
another certificate (proxy mode, where TLS ends at the site's proxy, aside), cannot be relayed through another certificate (proxy mode, where TLS ends at
and with `identity` each user is tied to its own certificate. the site's proxy, aside), and with `identity` each user is tied to its own
certificate.
Under either method the chain `node` label can be bound to the identity, so a Under either method the chain `node` label can be bound to the identity, so a
compromised edge cannot attribute its entries to another host, and the `http` compromised edge cannot attribute its entries to another host, and the `http`
@@ -1093,12 +1104,13 @@ where TLS ends at the trusted proxy; `mtls` also requires `client_auth`.
**Log content** **Log content**
Logs routinely contain secrets that were never meant to leave the host. Filters Logs routinely contain secrets that were never meant to leave the host. Filters
are the available tool: are the available tool; patterns are case-sensitive, so `(?i)` makes one catch
`Authorization: Bearer` too:
```toml ```toml
[[pipelines.flow.filters]] [[pipelines.flow.filters]]
type = "exclude" type = "exclude"
patterns = ["password", "api[_-]?key", "authorization", "bearer ", "secret"] patterns = ["(?i)password|api[_-]?key|authorization|bearer |secret"]
``` ```
Choose a sanitizer policy that matches the sink — `json` for JSON output, Choose a sanitizer policy that matches the sink — `json` for JSON output,
+22 -15
View File
@@ -18,7 +18,8 @@ whose input queue is full drops the event for itself only
## console ## console
Writes formatted payloads to stdout or stderr. It is the sink of the built-in Writes formatted payloads to stdout or stderr. It is the sink of the built-in
`pipe` pipeline, and of every spec pipeline given no `--sink`. `pipe` pipeline, and of every pipeline from flags or variables that gets no
sink from `--sink`, `LOGWISP_SINK` or its preset.
```toml ```toml
[[pipelines.plugin_sinks]] [[pipelines.plugin_sinks]]
@@ -40,8 +41,8 @@ Options, each as type and default:
bytes, `ESC` as `<1b>`: `auto` when the output is a terminal, `always`, or bytes, `ESC` as `<1b>`: `auto` when the output is a terminal, `always`, or
`never`. `never`.
- `color` (string, the top-level `color`, itself `auto`): paint level names: - `color` (string, the top-level `color`, itself `auto`): paint level names:
`auto` on a terminal with `NO_COLOR` unset and `TERM` not `dumb`, `always`, `auto` on a terminal with `NO_COLOR` empty or unset and `TERM` not `dumb`,
or `never`. `always`, or `never`.
Payloads are written as the formatter made them, one record per line. Payloads are written as the formatter made them, one record per line.
@@ -60,10 +61,12 @@ Payloads are written as the formatter made them, one record per line.
sink's own, written after escaping. A line whose source found no level sink's own, written after escaping. A line whose source found no level
stays plain. The codes go into whatever the format makes, JSON included: stays plain. The codes go into whatever the format makes, JSON included:
for output another program parses keep `auto` or `never`, not `always`. for output another program parses keep `auto` or `never`, not `always`.
- Backpressure: the sink never drops. When its output is slow (a slow reader, - Backpressure: the sink never drops while its pipeline runs; a stop or
a paused terminal, a stalled log collector) the pipeline waits for it, with reload can drop entries still in flight (`total_dropped_by_sink`). When its
its other sinks. A service whose stdout may stall writes to a `file` sink output is slow (a slow reader, a paused terminal, a stalled log collector)
instead. A closed stdout pipe ends lw with `SIGPIPE`, as it ends `cat`. the pipeline waits for it, with its other sinks. A service whose stdout may
stall writes to a `file` sink instead. A closed stdout pipe ends lw with
`SIGPIPE`, as it ends `cat`.
- On stop it writes what is queued. - On stop it writes what is queued.
--- ---
@@ -90,9 +93,9 @@ flush_interval_ms = 100
Options, each as type and default: Options, each as type and default:
- `directory` (string, **required**): output directory. The sink writes - `directory` (string, **required**): output directory. The sink writes
`NAME.log` and rotates it to `NAME_YYMMDD_HHMMSS.log`. The limits below `NAME.log` and rotates it to `NAME_YYMMDD_HHMMSS.log`, adding
delete *any* `.log` file in the directory, so give each file sink a `_1`, `_2`, ... when that name is taken. The limits below delete *any*
directory of its own. `.log` file in the directory, so give each file sink a directory of its own.
- `name` (string, **required**): file name base. - `name` (string, **required**): file name base.
- `max_size_mb` (int, `100`): rotate when the active file reaches this size. - `max_size_mb` (int, `100`): rotate when the active file reaches this size.
- `max_total_size_mb` (int, `1000`): above this total, the oldest `.log` files - `max_total_size_mb` (int, `1000`): above this total, the oldest `.log` files
@@ -103,7 +106,8 @@ Options, each as type and default:
- `retention_hours` (float, `168.0`): `.log` files there older than this are - `retention_hours` (float, `168.0`): `.log` files there older than this are
deleted; `0` keeps them. deleted; `0` keeps them.
- `buffer_size` (int, `1000`): sink input queue depth, and the depth of the - `buffer_size` (int, `1000`): sink input queue depth, and the depth of the
internal writer's queue, which drops and counts when full. internal writer's queue, which drops when full. That drop is reported
nowhere, and `total_processed` still counts the entry. At most `1048576`.
- `flush_interval_ms` (int, `100`): forced flush interval. - `flush_interval_ms` (int, `100`): forced flush interval.
The sink drives an internal writer configured for raw output with timestamps and The sink drives an internal writer configured for raw output with timestamps and
@@ -115,8 +119,8 @@ it hands the writer what is queued before closing the file.
## null ## null
Discards; holds a sink's place until one is chosen, so a pipeline can be set Discards; holds a sink's place until one is chosen, so a pipeline can be set
up before its sink is. It counts entries and bytes, so it also isolates a up before its sink is. It counts entries and bytes (`total_processed`,
source or flow when measuring it. `total_bytes`), so it also isolates a source or flow when measuring it.
```toml ```toml
[[pipelines.plugin_sinks]] [[pipelines.plugin_sinks]]
@@ -172,7 +176,8 @@ Options, each as type and default:
- `buffer_size` (int, `1000`): sink input queue depth. - `buffer_size` (int, `1000`): sink input queue depth.
- `client_buffer_size` (int, `256`): per-client send queue depth. - `client_buffer_size` (int, `256`): per-client send queue depth.
- `write_timeout_ms` (int, `0`): per-event write deadline; `0` = none. - `write_timeout_ms` (int, `0`): per-event write deadline; `0` = none.
- `max_connections` (int, `0`): concurrent stream cap; `0` = unlimited. - `max_connections` (int, `0`): concurrent stream cap; `0` = unlimited. A
stream past it gets `503` and counts in `rejected_clients`.
- `replay_lines` (int, `0`; `1000` from the `serve` preset): the last entries - `replay_lines` (int, `0`; `1000` from the `serve` preset): the last entries
a new stream gets after its `connected` event, before the live ones, so a a new stream gets after its `connected` event, before the live ones, so a
browser opened late sees a backlog; `0` = none. browser opened late sees a backlog; `0` = none.
@@ -231,7 +236,8 @@ Options, each as type and default:
- Stream responses carry `Content-Type: text/event-stream`, - Stream responses carry `Content-Type: text/event-stream`,
`Cache-Control: no-cache` and `X-Accel-Buffering: no`. On connect the client `Cache-Control: no-cache` and `X-Accel-Buffering: no`. On connect the client
receives an `event: connected` frame carrying its client id, session id, receives an `event: connected` frame carrying its client id, session id,
sink instance id, endpoint paths, and buffer size. sink instance id, endpoint paths, and `client_buffer_size` (as
`buffer_size`).
- Payloads are framed per the SSE spec, one `data:` line per line break in the - Payloads are framed per the SSE spec, one `data:` line per line break in the
payload (CRLF, LF or a lone CR), so multi-line entries stream correctly and payload (CRLF, LF or a lone CR), so multi-line entries stream correctly and
no entry can inject an `event:`, `id:` or `retry:` field. no entry can inject an `event:`, `id:` or `retry:` field.
@@ -312,6 +318,7 @@ Options, each as type and default:
`keep_alive_period_ms` idle; `false` sends none. `keep_alive_period_ms` idle; `false` sends none.
- `keep_alive_period_ms` (int, `30000`): keep-alive idle period. - `keep_alive_period_ms` (int, `30000`): keep-alive idle period.
- `max_connections` (int, `0`): concurrent connection cap; `0` = unlimited. - `max_connections` (int, `0`): concurrent connection cap; `0` = unlimited.
A connection past it is closed at once and counts in `rejected_conns`.
- `tls`, `auth`: as on `http`. - `tls`, `auth`: as on `http`.
- `acl` (table): as on `http`, without `requests_per_second_per_client`. - `acl` (table): as on `http`, without `requests_per_second_per_client`.
+45 -26
View File
@@ -6,7 +6,8 @@ type-specific `config` table. Registered types: `file`, `console`, `random`,
`null`, `tcp_chain`, `http_chain`. `null`, `tcp_chain`, `http_chain`.
Publication is non-blocking for every source but `console`: a full subscriber Publication is non-blocking for every source but `console`: a full subscriber
channel drops the entry and counts it in `dropped_entries`. The `console` channel drops the entry and counts it in `dropped_entries`. That channel holds
`buffer_size` entries, or 1000 on a source without that option. The `console`
source waits instead (see [console](#console)). source waits instead (see [console](#console)).
--- ---
@@ -31,7 +32,7 @@ Options, each as type and default:
- `directory` (string, required): directory to scan; not recursive. A leading - `directory` (string, required): directory to scan; not recursive. A leading
`~/`, here as in every file or directory option, is the home directory; a `~/`, here as in every file or directory option, is the home directory; a
relative path starts at lw's working directory; a missing one is waited for. relative path starts at lw's working directory.
- `pattern` (string, `*`): glob over filenames; `*` and `?` only. - `pattern` (string, `*`): glob over filenames; `*` and `?` only.
- `check_interval_ms` (int, `100`): directory rescan interval; minimum `10`. - `check_interval_ms` (int, `100`): directory rescan interval; minimum `10`.
An open file is polled every 100 ms whatever this is. An open file is polled every 100 ms whatever this is.
@@ -44,30 +45,37 @@ Options, each as type and default:
- Each matched file gets its own watcher, and its entries the file's base - Each matched file gets its own watcher, and its entries the file's base
name as `Source`. Watchers for files that disappear are stopped and removed name as `Source`. Watchers for files that disappear are stopped and removed
on the next scan. on the next scan.
- A missing directory is waited for: INFO `Waiting for directory`, then
`Directory readable` once a scan reads it. A directory or file that cannot be
read is retried at each scan or poll, with WARN `Cannot read directory` or
`Cannot read file` when the failure starts or changes, not at each retry.
- The file lw's stdout or stderr is redirected to is never followed, with a - The file lw's stdout or stderr is redirected to is never followed, with a
warning: `lw > out.log` in the directory would read its own output back. warning: `lw > out.log` in the directory would read its own output back.
- A line longer than 1 MiB continues in the next entry, as on the console - A line longer than 1 MiB continues in the next entry, as on the console
source. source; so does the rest of a line a poll reads while it is still being
written.
- A new watcher seeks to end-of-file. Positions live in memory only, so a - A new watcher seeks to end-of-file. Positions live in memory only, so a
restart or a reload resumes from the current end of each file, and content restart or a reload resumes from the current end of each file, and content
written meanwhile is not read. `from = "start"` instead reads each file whole written meanwhile is not read. `from = "start"` instead reads each file whole
when its watcher is created, which a process writing beside LogWisp needs, at when its watcher is created, which a process writing beside LogWisp needs, at
the cost of replaying files already on disk at every restart and reload. the cost of replaying files already on disk at every restart and reload.
- Rotation is detected from size decrease, modification-time reset, a position - Rotation is detected from size decrease, a modification time that went back
beyond end-of-file, or an inode change. An inode change where the new file is without growth, a position more than 1 KiB beyond end-of-file, or an inode
at least as large as the recorded position is treated as an atomic save, not change. An inode change where the new file is non-empty and at least as large
a rotation, and the position is preserved. Each rotation also publishes an as the recorded position is treated as an atomic save, not a rotation, and
INFO entry `Log rotation detected (#N): REASON` with the file's name as the position is preserved. Each rotation also publishes an INFO entry
`Source`; it reaches every sink, so exclude it with a filter `Log rotation detected (#N): REASON` with the file's name as `Source`; it
(`Log rotation detected`) where unwanted. reaches every sink, so exclude it with a filter (`Log rotation detected`)
where unwanted.
- A rotation that renames in place — what a size-capped writer does — puts the - A rotation that renames in place — what a size-capped writer does — puts the
same inode back under a name `pattern` also matches. Its watcher resumes at same inode back under a name `pattern` also matches. Its watcher resumes at
the position the original reached, so `from = "start"` reads the tail an the position the original reached, so `from = "start"` reads the tail an
unfinished read left behind rather than the whole archive a second time. unfinished read left behind rather than the whole archive a second time.
- A line is parsed as JSON only when it is an object whose top-level keys are - A line is parsed as JSON only when it is a non-empty object whose top-level
all drawn from `time`, `level`, `msg` and `fields` — the four an entry can keys are all drawn from `time`, `level`, `msg` and `fields` (the four an
carry. `time` is read as RFC3339Nano. Any other key, and any non-object line, entry can carry), with any `level` or `msg` a string. `time` is read as
is kept whole as text, because parsing it would drop the rest. `level` is RFC3339Nano; the read time stands in when it is missing or does not parse.
Any other line is kept whole as text, so nothing in it is dropped. `level` is
kept as written (a filter sees `Error` or `fatal` as is), and a line without kept as written (a filter sees `Error` or `fatal` as is), and a line without
`level` has none; only text lines get the inference below. `level` has none; only text lines get the inference below.
- A text line's level is its first whole word, in any case, naming one: - A text line's level is its first whole word, in any case, naming one:
@@ -76,20 +84,23 @@ Options, each as type and default:
console sink paints that word and the http sink's viewer filters on it. console sink paints that word and the http sink's viewer filters on it.
- `raw = true` skips the JSON branch. The line, with its terminator (`\n` or - `raw = true` skips the JSON branch. The line, with its terminator (`\n` or
`\r\n`) removed, becomes the message. `fields` stays empty, the time is the `\r\n`) removed, becomes the message. `fields` stays empty, the time is the
read time, and the level is inferred from the text. Blank lines are skipped. read time, and the level is inferred from the text. Empty lines are skipped.
With `format.type = "raw"` each line is written back with one `\n`, which With `format.type = "raw"` each line is written back with one `\n`, which
carries records the envelope cannot hold but is not byte-exact; see carries records the envelope cannot hold but is not byte-exact; see
[Formatters](formatters.md#raw). [Formatters](formatters.md#raw).
**Statistics**: per-watcher size, position, entries read, rotation count, and **Statistics**: `active_watchers`, and `watchers`, one map per followed file:
last read time, plus `active_watchers`. `directory` (the file's path), `size`, `position`, `entries_read`,
`rotations`, `last_read`. The status report shows that list only as
`watchers_count`.
--- ---
## console ## console
Reads newline-delimited entries from standard input. It is the source of the Reads newline-delimited entries from standard input. It is the source of the
built-in `pipe` pipeline, and of every spec pipeline given no `--source`. built-in `pipe` pipeline, and of every pipeline from flags or variables left
without one: no `--source`, and no preset that brings one.
```toml ```toml
[[pipelines.plugin_sources]] [[pipelines.plugin_sources]]
@@ -106,21 +117,24 @@ Option, as type and default:
Lines: Lines:
- Each line is one entry, its terminator (`\n` or `\r\n`) removed; the - Each line is one entry, its terminator (`\n` or `\r\n`) removed; the
formatter writes one back. An unterminated last line is kept. formatter writes one back. An unterminated last line is kept.
- Blank lines are skipped. - Empty lines are skipped; a line of spaces is an entry.
- A line over 1 MiB continues in the next entry; no byte is lost. - A line over 1 MiB continues in the next entry; no byte is lost.
- The level is inferred from the line text, and `Source` is set to `console`. - The level is inferred from the line text, and `Source` is set to `console`.
Delivery and lifetime: Delivery and lifetime:
- Nothing is dropped: the source waits for its pipeline, so a slow pipeline - The source drops nothing while its pipeline runs: it waits, so a slow
slows the reading of stdin. pipeline slows the reading of stdin.
- One reader serves the process. A reload's new source continues where the old - One reader serves the process. A reload's new source continues where the old
one stopped; no line is read twice. Hence at most **one** console source in one stopped, and no line is read twice, but lines the old pipeline still
the whole configuration, checked at validation. holds at the reload can be lost. Hence at most **one** console source in the
whole configuration, checked at validation.
- At the end of input the source ends, and with it a pipeline whose sources - At the end of input the source ends, and with it a pipeline whose sources
have all ended; lw exits once every pipeline has have all ended; lw exits once every pipeline has
([CLI](cli.md#built-in-defaults)). Under a supervisor whose stdin is ([CLI](cli.md#built-in-defaults)). Under a supervisor whose stdin is
`/dev/null` that is at once. `/dev/null` that is at once.
**Statistics**: `ended`, true once standard input has ended.
--- ---
## random ## random
@@ -151,10 +165,13 @@ Options, each as type and default:
longer message. longer message.
- `special` (bool, `false`): with `length` ≥ 3, mix in up to one control byte - `special` (bool, `false`): with `length` ≥ 3, mix in up to one control byte
(`\n` and `\r` among them) and a few non-ASCII bytes, often invalid UTF-8. (`\n` and `\r` among them) and a few non-ASCII bytes, often invalid UTF-8.
`raw` and `txt` carry them as they are; `json` escapes them inside its `raw` and `txt` carry them as they are; `json` escapes the control byte and
message. replaces each invalid byte with U+FFFD.
Levels (`txt` and `json` only) are random among DEBUG, INFO, WARN, ERROR. Levels (`txt` and `json` only) are random among DEBUG, INFO, WARN, ERROR.
`Source` is `random_ID`, ID being the source's `id`.
**Statistics**: `format`, `interval_ms`, `jitter_ms`, `length`, `special`.
--- ---
@@ -295,7 +312,9 @@ node_binding = "force"
Options, each as type and default: Options, each as type and default:
- `host`, `port`, `trust_node`, `tls`, `auth`: as on [tcp_chain](#tcp_chain). - `host`, `port`, `trust_node`, `tls`: as on [tcp_chain](#tcp_chain).
- `auth` (table): as on `tcp_chain`, plus `token_lifetime_ms`, the `scram`
bearer token lifetime: 10 s to 24 h, `0` = 15 minutes.
- `ingest_path` (string, `/ingest`): endpoint path; must start with `/`, and - `ingest_path` (string, `/ingest`): endpoint path; must start with `/`, and
`/auth` is reserved for the login. `/auth` is reserved for the login.
- `buffer_size` (int, `1000`): subscriber channel depth. - `buffer_size` (int, `1000`): subscriber channel depth.
+13 -11
View File
@@ -20,7 +20,7 @@ release tag, and both skeletons download the tagged source.
git tag -a vX.Y.Z -m vX.Y.Z # git tag -s signs it with your key git tag -a vX.Y.Z -m vX.Y.Z # git tag -s signs it with your key
git push origin vX.Y.Z git push origin vX.Y.Z
``` ```
- `make` then stamps `vX.Y.Z` (from `git describe`), and - The Makefile's builds then stamp `vX.Y.Z` (from `git describe`), and
`go install github.com/lixenwraith/logwisp/cmd/lw@vX.Y.Z` works. `go install github.com/lixenwraith/logwisp/cmd/lw@vX.Y.Z` works.
- A GitHub release for the tag is optional; the tag alone serves - A GitHub release for the tag is optional; the tag alone serves
`archive/vX.Y.Z.tar.gz`, which the PKGBUILD downloads. `archive/vX.Y.Z.tar.gz`, which the PKGBUILD downloads.
@@ -33,10 +33,12 @@ release tag, and both skeletons download the tagged source.
- Push `PKGBUILD` and `.SRCINFO` to - Push `PKGBUILD` and `.SRCINFO` to
`ssh://aur@aur.archlinux.org/logwisp.git`; optionally a `logwisp-git` `ssh://aur@aur.archlinux.org/logwisp.git`; optionally a `logwisp-git`
package built from the main branch. package built from the main branch.
3. FreeBSD, `deploy/package/freebsd/` (`sysutils/logwisp`): 3. FreeBSD, `deploy/package/freebsd/` (`sysutils/logwisp`), copied into a
ports tree with `cp -RL` (`files/logwisp.in` links to
`deploy/package/logwisp.rc`):
- `DISTVERSION=X.Y.Z` and `MAINTAINER=` your address. - `DISTVERSION=X.Y.Z` and `MAINTAINER=` your address.
- `make makesum` writes `distinfo`: the module zip that the Go proxy - `make makesum` writes `distinfo`, the checksums of the `.mod` and `.zip`
serves for `GO_MODULE` at the tag. that the Go proxy serves for `GO_MODULE` at the tag.
- The `logwisp` user needs an ID registered in the ports tree. Take a - The `logwisp` user needs an ID registered in the ports tree. Take a
number below 1000 that is free in both `/usr/ports/UIDs` and number below 1000 that is free in both `/usr/ports/UIDs` and
`/usr/ports/GIDs`, and add to the same patch: `/usr/ports/GIDs`, and add to the same patch:
@@ -57,7 +59,7 @@ release tag, and both skeletons download the tagged source.
(lw is static by design) and `maintainer-script-calls-systemctl` (dh's (lw is static by design) and `maintainer-script-calls-systemctl` (dh's
scripts use `deb-systemd-helper` instead). scripts use `deb-systemd-helper` instead).
5. With a tag, the packaging workflow's FreeBSD job can build the port itself 5. With a tag, the packaging workflow's FreeBSD job can build the port itself
from a ports tree (`make stage check-plist` in a copy of from a ports tree (`make stage check-plist` in a `cp -RL` copy of
`deploy/package/freebsd/`), rather than BSD make on the checkout. `deploy/package/freebsd/`), rather than BSD make on the checkout.
## 2. What `lw --tui` and its engine leave for later ## 2. What `lw --tui` and its engine leave for later
@@ -71,16 +73,16 @@ in place. Not yet:
- Per-sink taps: what each sink took, where the tap at the flow's exit shows - Per-sink taps: what each sink took, where the tap at the flow's exit shows
what dispatch offered (a full sink input drops for that sink only). what dispatch offered (a full sink input drops for that sink only).
- Live filter patching: a filter edited while `t` runs takes effect through - Live filter patching: a filter edited while `t` runs takes effect through
`Filter.UpdatePatterns` (unused), not a rerun that rebinds every listener. `Filter.UpdatePatterns` (unused, and racing `Apply`, which reads the
patterns without `f.mu`), not a rerun that rebinds every listener.
- The wide-rune table `internal/tui/view.go` copies from - The wide-rune table `internal/tui/view.go` copies from
`lixenwraith/terminal`'s one-cell rule goes once terminal exports that rule. `lixenwraith/terminal`'s one-cell rule goes once terminal exports that rule.
- A scripted session on a pseudo-terminal for Run, each output and `t` (keys, - A scripted session on a pseudo-terminal for Run, each output and `t` (keys,
the password dialog, `SIGHUP`), on a text console and over ssh. the password dialog, `SIGHUP`), on a text console and over ssh.
- The schema lists `auth.node_binding` on every listener and - The schema lists `auth.trusted_proxies` on both HTTP listeners, though
`auth.trusted_proxies` on both HTTP listeners, though `AuthOptions.Check` `AuthOptions.Check` refuses it off the `http` sink; a side tag for the
refuses a binding off the chain sources (under `mtls` or `scram`) and `http` sink would filter it as `chain_listener` keeps `node_binding` to the
`trusted_proxies` off the `http` sink; side tags for chain sources and the chain sources.
`http` sink would filter them as `http` filters HTTP's keys.
- Optional, in `lixenwraith/color`: export the 16 xterm reference shades - Optional, in `lixenwraith/color`: export the 16 xterm reference shades
(`RGBTo16` maps the site's hues badly, so the 16-color tier is chosen by (`RGBTo16` maps the site's hues badly, so the 16-color tier is chosen by
hand). hand).
+2 -1
View File
@@ -573,7 +573,8 @@ func (p *Policy) Rejected() uint64 {
// Stats reports policy state for a plugin's stats details map. Merge it in // Stats reports policy state for a plugin's stats details map. Merge it in
// with maps.Copy so rejections surface in the status reporter and in the // with maps.Copy so rejections surface in the status reporter and in the
// http sink's status endpoint. allowed counts admissions (certificate checks, // http sink's status endpoint. allowed counts admissions (certificate checks,
// SCRAM logins); rejected counts every refusal. // SCRAM logins), rejected refusals; a throttled or busy login counts in
// auth_throttled or auth_busy, a forwarded client the acl refuses in acl_denied.
func (p *Policy) Stats() map[string]any { func (p *Policy) Stats() map[string]any {
if p == nil { if p == nil {
return map[string]any{"auth": MethodNone} return map[string]any{"auth": MethodNone}
+2 -1
View File
@@ -196,7 +196,8 @@ type asked struct {
tty bool tty bool
} }
// startupOver is set once lw's first service is built // startupOver is set once lw builds the service it runs, after any run on the
// --tui screen
var startupOver atomic.Bool var startupOver atomic.Bool
// EndStartup stops reading descriptors, pipes, FIFOs and terminals: at a // EndStartup stops reading descriptors, pipes, FIFOs and terminals: at a
+4
View File
@@ -1,3 +1,7 @@
// Package chain is the protocol between logwisp nodes: the hello a dialer
// opens with (ProtocolVersion, node, SCRAM), the NDJSON entries after it, the
// headers and /auth path of its HTTP mapping, and the reconnect backoff the
// chain sinks share.
package chain package chain
import ( import (
+4 -3
View File
@@ -1,7 +1,8 @@
// Package compose is the configuration engine lw --tui and the website share: // Package compose is the configuration engine lw --tui and the website share:
// pipelines started empty, from a preset, a configuration or a command line, // pipelines started empty, from a preset, a configuration or a pasted form,
// edited, validated, and written as a command line, environment or file. It // edited, validated, and written as a command line, environment or file that
// reads no file and links no network or terminal code, so it builds as WebAssembly. // reads back to the same pipelines; Schema is lw --schema. It reads no file and
// links no TLS, HTTP or terminal code, so it builds as WebAssembly.
package compose package compose
import ( import (
+8 -2
View File
@@ -1,3 +1,8 @@
// Package config is lw's configuration: the settings and pipeline tables, the
// plugin catalogue whose option structs' tags are the schema, the pipeline
// flags, LOGWISP_ variables and presets that build pipelines, and the rules
// Decode and ValidateConfig apply. Load's Manager merges the file, environment
// and command line, and rereads the file on reload.
package config package config
// --- LogWisp Configuration Options --- // --- LogWisp Configuration Options ---
@@ -31,7 +36,7 @@ type LogConfig struct {
// File output settings (when Output includes "file" or "all") // File output settings (when Output includes "file" or "all")
File *LogFileConfig `toml:"file"` File *LogFileConfig `toml:"file"`
// Console output settings // Accepted for older files: logging.output chooses the console
Console *LogConsoleConfig `toml:"console"` Console *LogConsoleConfig `toml:"console"`
} }
@@ -44,7 +49,8 @@ type LogFileConfig struct {
RetentionHours float64 `toml:"retention_hours" help:"hours a .log file is kept; 0 keeps them"` RetentionHours float64 `toml:"retention_hours" help:"hours a .log file is kept; 0 keeps them"`
} }
// LogConsoleConfig defines settings for console-based application logging // LogConsoleConfig is logging.console, whose target has no effect:
// logging.output chooses the console stream
type LogConsoleConfig struct { type LogConsoleConfig struct {
Target string `toml:"target" lw:"enum=stdout|stderr|split" help:"accepted without effect: logging.output chooses the console"` Target string `toml:"target" lw:"enum=stdout|stderr|split" help:"accepted without effect: logging.output chooses the console"`
} }
+5 -4
View File
@@ -231,9 +231,9 @@ func settingKeys(v reflect.Value, prefix string) []Key {
} }
// Decode reads a plugin's config map into its options: unknown keys, kinds, // Decode reads a plugin's config map into its options: unknown keys, kinds,
// the defaults, the tag rules, the tls, auth and acl rules of its side, then the // the defaults, the tag rules and the options' own Check, then the host, tls,
// options' own Check; a leading ~ in a path is the home directory. It reads no // acl and auth rules of its side; a leading ~ in a path is the home directory.
// file. // It reads no file.
func Decode[T any](role, typ string, m map[string]any) (*T, error) { func Decode[T any](role, typ string, m map[string]any) (*T, error) {
v, err := decode(role, typ, m) v, err := decode(role, typ, m)
if err != nil { if err != nil {
@@ -272,7 +272,8 @@ func decode(role, typ string, m map[string]any) (any, error) {
} }
// Settle gives a decoded table its defaults, applies the tag rules, then its // Settle gives a decoded table its defaults, applies the tag rules, then its
// own Check: Decode's last steps, for the flow stages a file decodes typed // own Check, as Decode does before its side's rules, for the flow stages a
// file decodes typed
func Settle(v any) error { func Settle(v any) error {
Fill(v) Fill(v)
if err := checkTags(reflect.ValueOf(v).Elem(), ""); err != nil { if err := checkTags(reflect.ValueOf(v).Elem(), ""); err != nil {
+2 -1
View File
@@ -238,7 +238,8 @@ func SpecPipelines(specs []Spec, isDir IsDir) ([]PipelineConfig, error) {
} }
// buildPipelines turns specs, in order, into pipelines; specs before the first // buildPipelines turns specs, in order, into pipelines; specs before the first
// --pipeline go to one named "cli". Every call returns fresh maps. // --pipeline go to one named after the preset that starts it, or else "cli".
// Every call returns fresh maps.
func buildPipelines(specs []pipelineSpec, isDir IsDir) ([]PipelineConfig, error) { func buildPipelines(specs []pipelineSpec, isDir IsDir) ([]PipelineConfig, error) {
var pipelines []PipelineConfig var pipelines []PipelineConfig
var seen map[*specKind]bool var seen map[*specKind]bool
+8 -5
View File
@@ -25,7 +25,8 @@ import (
// ValidateConfig checks a configuration without reading a file it names: // ValidateConfig checks a configuration without reading a file it names:
// the settings, then each pipeline's flow stages, and its sources and sinks // the settings, then each pipeline's flow stages, and its sources and sinks
// decoded in full. Its error joins every problem, one a part, each a // decoded in full. Its error joins every problem, one a part, each a
// *KeyError naming the path of the refused key. lw --check opens the files. // *KeyError naming the refused key's path, save "no pipelines configured".
// lw --check opens the files.
func ValidateConfig(cfg *Config) error { func ValidateConfig(cfg *Config) error {
if cfg == nil { if cfg == nil {
return fmt.Errorf("config is nil") return fmt.Errorf("config is nil")
@@ -521,9 +522,11 @@ func (c *FilterConfig) Check() error {
return nil return nil
} }
// Scan decodes a flow stage's command-line map into target, rejecting keys // Scan decodes a table into target: a flow stage from a spec or an edit, a
// target does not declare: a misspelled key must fail, not be dropped. Its // file's root or the website's pipelines. It refuses a key target does not
// values take their kinds as a plugin's do, a wrong one refused at its key. // declare, nested tables' too: a misspelled key must fail, not be dropped.
// Values outside a list of tables take their kinds as a plugin's do, a wrong
// one refused at its key.
func Scan(configMap map[string]any, target any) error { func Scan(configMap map[string]any, target any) error {
if err := checkKeys(configMap, reflect.TypeOf(target), ""); err != nil { if err := checkKeys(configMap, reflect.TypeOf(target), ""); err != nil {
return err return err
@@ -580,7 +583,7 @@ func checkKeys(m map[string]any, t reflect.Type, prefix string) error {
// checkFileKeys rejects keys the configuration file declares that Config does // checkFileKeys rejects keys the configuration file declares that Config does
// not: a misspelled table path such as plugin_sinks.confg.tls would otherwise // not: a misspelled table path such as plugin_sinks.confg.tls would otherwise
// drop the whole table. Plugin config maps are checked by Scan instead. // drop the whole table. Plugin config maps are checked by Decode instead.
func checkFileKeys(path string) error { func checkFileKeys(path string) error {
data, err := os.ReadFile(path) data, err := os.ReadFile(path)
if errors.Is(err, fs.ErrNotExist) { if errors.Is(err, fs.ErrNotExist) {
+6 -1
View File
@@ -1,3 +1,7 @@
// Package core holds what the parts of a pipeline share: the LogEntry and the
// TransportEvent sinks get, the Tap a viewer reads a flow's decisions from,
// the level names, plugin capabilities, size limits and timings, and Network
// and Listen, which keep a listener or dialer to its host's address family.
package core package core
import ( import (
@@ -29,7 +33,8 @@ type TransportEvent struct {
// Tapped is what a pipeline's flow decided about one entry: Stage names the // Tapped is what a pipeline's flow decided about one entry: Stage names the
// stage that dropped Event.Entry (rate_limit, filters at Index, format), or is // stage that dropped Event.Entry (rate_limit, filters at Index, format), or is
// "" when it passed and Event.Payload is what the sinks got. // "" when it passed and Event.Payload is what dispatch offered the sinks; a
// full sink input drops it for that sink only.
type Tapped struct { type Tapped struct {
Pipeline string Pipeline string
Source string // the source's id; "" for a heartbeat Source string // the source's id; "" for a heartbeat
+5 -9
View File
@@ -37,7 +37,6 @@ func NewFilter(cfg config.FilterConfig, logger *log.Logger) (*Filter, error) {
logger: logger, logger: logger,
} }
// Compile patterns
for i, pattern := range cfg.Patterns { for i, pattern := range cfg.Patterns {
re, err := regexp.Compile(pattern) re, err := regexp.Compile(pattern)
if err != nil { if err != nil {
@@ -82,8 +81,8 @@ func (f *Filter) Apply(entry core.LogEntry) bool {
"logic", f.config.Logic, "logic", f.config.Logic,
"entry_level", entry.Level, "entry_level", entry.Level,
"entry_source", entry.Source, "entry_source", entry.Source,
"entry_message", entry.Message[:min(100, len(entry.Message))], // First 100 chars "entry_message", entry.Message[:min(100, len(entry.Message))],
"text_to_match", text[:min(150, len(text))], // First 150 chars "text_to_match", text[:min(150, len(text))],
"patterns", f.config.Patterns) "patterns", f.config.Patterns)
for i, pattern := range f.config.Patterns { for i, pattern := range f.config.Patterns {
@@ -103,7 +102,6 @@ func (f *Filter) Apply(entry core.LogEntry) bool {
"component", "filter", "component", "filter",
"matched", matched) "matched", matched)
// Determine if we should pass or drop
shouldPass := false shouldPass := false
switch f.config.Type { switch f.config.Type {
case config.FilterTypeInclude: case config.FilterTypeInclude:
@@ -137,11 +135,12 @@ func (f *Filter) GetStats() map[string]any {
} }
} }
// UpdatePatterns allows for dynamic, thread-safe updates to the filter's regex patterns // UpdatePatterns replaces the patterns under f.mu, which Apply does not take:
// it is safe only before entries flow
func (f *Filter) UpdatePatterns(patterns []string) error { func (f *Filter) UpdatePatterns(patterns []string) error {
compiled := make([]*regexp.Regexp, 0, len(patterns)) compiled := make([]*regexp.Regexp, 0, len(patterns))
// Compile all patterns first // Compiled before the swap, so a bad pattern leaves the filter as it was
for i, pattern := range patterns { for i, pattern := range patterns {
re, err := regexp.Compile(pattern) re, err := regexp.Compile(pattern)
if err != nil { if err != nil {
@@ -150,7 +149,6 @@ func (f *Filter) UpdatePatterns(patterns []string) error {
compiled = append(compiled, re) compiled = append(compiled, re)
} }
// Update atomically
f.mu.Lock() f.mu.Lock()
f.patterns = compiled f.patterns = compiled
f.config.Patterns = patterns f.config.Patterns = patterns
@@ -166,7 +164,6 @@ func (f *Filter) UpdatePatterns(patterns []string) error {
func (f *Filter) matches(text string) bool { func (f *Filter) matches(text string) bool {
switch f.config.Logic { switch f.config.Logic {
case config.FilterLogicOr: case config.FilterLogicOr:
// Match any pattern
for _, re := range f.patterns { for _, re := range f.patterns {
if re.MatchString(text) { if re.MatchString(text) {
return true return true
@@ -175,7 +172,6 @@ func (f *Filter) matches(text string) bool {
return false return false
case config.FilterLogicAnd: case config.FilterLogicAnd:
// Must match all patterns
for _, re := range f.patterns { for _, re := range f.patterns {
if !re.MatchString(text) { if !re.MatchString(text) {
return false return false
+1 -7
View File
@@ -83,7 +83,6 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
Message: "heartbeat", Message: "heartbeat",
} }
// Add stats if configured
if hg.config.IncludeStats { if hg.config.IncludeStats {
fields := map[string]any{ fields := map[string]any{
"type": "heartbeat", "type": "heartbeat",
@@ -98,13 +97,11 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
entry.Fields = fieldsJSON entry.Fields = fieldsJSON
} }
// Use formatter to generate payload
var payload []byte var payload []byte
var err error var err error
// Check if we need special formatting for heartbeat // Unreachable: heartbeat.format's enum (txt, json, raw) refuses "comment"
if hg.config.Format == "comment" { if hg.config.Format == "comment" {
// SSE comment format - bypass formatter for this special case
if hg.config.IncludeStats { if hg.config.IncludeStats {
beatNum := hg.beatCount.Load() beatNum := hg.beatCount.Load()
payload = []byte(": heartbeat " + t.Format(time.RFC3339) + payload = []byte(": heartbeat " + t.Format(time.RFC3339) +
@@ -115,7 +112,6 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
} else { } else {
// Use flow's formatter for consistent formatting // Use flow's formatter for consistent formatting
if adapter, ok := hg.formatter.(*format.FormatterAdapter); ok { if adapter, ok := hg.formatter.(*format.FormatterAdapter); ok {
// Customize flags for heartbeat if needed
// The formatter shows a timestamp unless told otherwise // The formatter shows a timestamp unless told otherwise
customFlags := formatter.FlagDefault customFlags := formatter.FlagDefault
if !hg.config.IncludeTimestamp { if !hg.config.IncludeTimestamp {
@@ -123,14 +119,12 @@ func (hg *HeartbeatGenerator) generateHeartbeat(t time.Time) core.TransportEvent
} }
payload, err = adapter.FormatWithFlags(entry, customFlags) payload, err = adapter.FormatWithFlags(entry, customFlags)
} else { } else {
// Fallback to standard format
payload, err = hg.formatter.Format(entry) payload, err = hg.formatter.Format(entry)
} }
if err != nil { if err != nil {
hg.logger.Error("msg", "Failed to format heartbeat", hg.logger.Error("msg", "Failed to format heartbeat",
"error", err) "error", err)
// Fallback to simple text
payload = []byte("heartbeat: " + t.Format(time.RFC3339) + "\n") payload = []byte("heartbeat: " + t.Format(time.RFC3339) + "\n")
} }
} }
+4 -4
View File
@@ -1,8 +1,8 @@
// Package netacl is the address seam of the network listeners, beside tlsx // Package netacl is the address seam of the network listeners, beside tlsx
// and authz: a Policy reads the PROXY header of the proxies it trusts, then // and authz: a Policy reads its trusted proxies' PROXY headers, refuses a peer
// refuses a peer by its client's address, or past its client's limits, before // by address or connection count before TLS, and an HTTP request past its rate
// TLS or a byte of the protocol is read. New returns nil when nothing is set, // or from a forwarded client the rules refuse. Its Table also throttles SCRAM.
// and every method tolerates a nil receiver. // New returns nil when nothing is set; every method tolerates a nil receiver.
package netacl package netacl
import ( import (
+6 -1
View File
@@ -1,3 +1,8 @@
// Package pipeline runs one pipeline: its sources through the flow to its
// sinks, each plugin made by the factory its type registered. Sinks start
// before the sources and stop after them, so entries in flight arrive; a full
// sink drops its event unless it declares core.CapBackpressure. A tap gives a
// viewer a copy of what the flow decides.
package pipeline package pipeline
import ( import (
@@ -39,7 +44,7 @@ type Pipeline struct {
cancel context.CancelFunc cancel context.CancelFunc
wg sync.WaitGroup wg sync.WaitGroup
running atomic.Bool running atomic.Bool
halt chan struct{} // closed by Stop: backpressure sinks drop again halt chan struct{} // closed by Stop: backpressure events may then drop, room or not
finished chan struct{} // closed when every source has ended finished chan struct{} // closed when every source has ended
finishedOnce sync.Once finishedOnce sync.Once
} }
+2 -2
View File
@@ -143,8 +143,8 @@ func TestStopReleasesStalledBackpressureSink(t *testing.T) {
} }
// The tap gets each entry's verdict under its source's id: the stage that // The tap gets each entry's verdict under its source's id: the stage that
// dropped it, or what the sinks got. Drops come only when All is set, and // dropped it, or what dispatch offered the sinks. Drops come only when All is
// heartbeats come as passed, from no source. // set, and heartbeats come as passed, from no source.
func TestTapSeesEachVerdict(t *testing.T) { func TestTapSeesEachVerdict(t *testing.T) {
const n = 1000 const n = 1000
verdict := func(msg string) (string, int) { verdict := func(msg string) (string, int) {
+3
View File
@@ -1,3 +1,6 @@
// Package plugin maps each source and sink type to the constructor its
// package registers in init; a type without a row in config's catalogue is
// refused.
package plugin package plugin
import ( import (
+3
View File
@@ -1,3 +1,6 @@
// Package service holds the pipelines of one configuration: it builds all of
// them or none, starts and stops them together, gives a viewer's tap to each,
// and reports when every one has finished.
package service package service
import ( import (
+10 -9
View File
@@ -1,3 +1,7 @@
// Package session tracks the sessions of a pipeline's plugins: a client
// connection, or a plugin's own input or output. A Manager per pipeline sweeps
// the idle ones; each plugin reaches it through a Proxy that sees only its
// instance's sessions.
package session package session
import ( import (
@@ -11,7 +15,7 @@ import (
"github.com/lixenwraith/logwisp/internal/core" "github.com/lixenwraith/logwisp/internal/core"
) )
// Session represents a connection session // Session is a client connection, or a plugin's own input or output
type Session struct { type Session struct {
InstanceID string // Plugin instance identifier InstanceID string // Plugin instance identifier
ID string // Unique session identifier ID string // Unique session identifier
@@ -20,8 +24,7 @@ type Session struct {
LastActivity time.Time // Last activity timestamp LastActivity time.Time // Last activity timestamp
Metadata map[string]any // Optional metadata (e.g., TLS info) Metadata map[string]any // Optional metadata (e.g., TLS info)
// Connection context Source string // the owning plugin's instance ID, which Proxy sets
Source string // Source type: "tcp_source", "http_source", "tcp_sink", etc.
} }
// Manager handles the lifecycle of sessions // Manager handles the lifecycle of sessions
@@ -34,7 +37,7 @@ type Manager struct {
cleanupTicker *time.Ticker cleanupTicker *time.Ticker
done chan struct{} done chan struct{}
// Expiry callbacks by source type // Expiry callbacks by plugin instance
expiryCallbacks map[string]func(sessionID, remoteAddr string) expiryCallbacks map[string]func(sessionID, remoteAddr string)
callbacksMu sync.RWMutex callbacksMu sync.RWMutex
} }
@@ -52,7 +55,6 @@ func NewManager(maxIdleTime time.Duration) *Manager {
expiryCallbacks: make(map[string]func(sessionID, remoteAddr string)), expiryCallbacks: make(map[string]func(sessionID, remoteAddr string)),
} }
// Start cleanup routine
m.startCleanup() m.startCleanup()
return m return m
@@ -145,7 +147,7 @@ func (m *Manager) GetSessionCount() int {
return len(m.sessions) return len(m.sessions)
} }
// GetSessionsBySource returns snapshots of sessions matching a specific source type. // GetSessionsBySource returns snapshots of sessions matching a specific plugin instance.
func (m *Manager) GetSessionsBySource(source string) []*Session { func (m *Manager) GetSessionsBySource(source string) []*Session {
m.mu.RLock() m.mu.RLock()
defer m.mu.RUnlock() defer m.mu.RUnlock()
@@ -159,7 +161,7 @@ func (m *Manager) GetSessionsBySource(source string) []*Session {
return sessions return sessions
} }
// GetActiveSessionsBySource returns snapshots of active sessions for a given source. // GetActiveSessionsBySource returns snapshots of active sessions for a given plugin instance.
func (m *Manager) GetActiveSessionsBySource(source string) []*Session { func (m *Manager) GetActiveSessionsBySource(source string) []*Session {
m.mu.RLock() m.mu.RLock()
defer m.mu.RUnlock() defer m.mu.RUnlock()
@@ -240,7 +242,7 @@ func (m *Manager) RegisterExpiryCallback(source string, callback func(sessionID,
m.expiryCallbacks[source] = callback m.expiryCallbacks[source] = callback
} }
// UnregisterExpiryCallback removes an expiry callback for a given source type // UnregisterExpiryCallback removes an expiry callback for a given plugin instance
func (m *Manager) UnregisterExpiryCallback(source string) { func (m *Manager) UnregisterExpiryCallback(source string) {
m.callbacksMu.Lock() m.callbacksMu.Lock()
defer m.callbacksMu.Unlock() defer m.callbacksMu.Unlock()
@@ -290,7 +292,6 @@ func (m *Manager) cleanupIdleSessions() {
for _, session := range expiredSessions { for _, session := range expiredSessions {
if callback, exists := callbacks[session.Source]; exists { if callback, exists := callbacks[session.Source]; exists {
// Call callback to notify owner
go callback(session.ID, session.RemoteAddr) go callback(session.ID, session.RemoteAddr)
} }
} }
+2 -2
View File
@@ -82,8 +82,8 @@ func NewConsoleSinkPlugin(
output := Stream(opts.Target) output := Stream(opts.Target)
// inline decides auto: a terminal, NO_COLOR unset, TERM not dumb. Paint // inline decides auto: a terminal, NO_COLOR empty or unset, TERM not
// returns its text unchanged exactly when color is off. // dumb. Paint returns its text unchanged exactly when color is off.
printer := inline.New(output) printer := inline.New(output)
if opts.Color != "auto" { if opts.Color != "auto" {
printer.SetColor(opts.Color == "always") printer.SetColor(opts.Color == "always")
+1 -5
View File
@@ -56,7 +56,6 @@ func NewFileSinkPlugin(
logger *log.Logger, logger *log.Logger,
proxy *session.Proxy, proxy *session.Proxy,
) (sink.Sink, error) { ) (sink.Sink, error) {
// Create empty config struct
opts, err := config.Decode[config.FileSinkOptions]("sink", "file", configMap) opts, err := config.Decode[config.FileSinkOptions]("sink", "file", configMap)
if err != nil { if err != nil {
return nil, err return nil, err
@@ -71,7 +70,7 @@ func NewFileSinkPlugin(
writerConfig.RetentionPeriodHrs = opts.RetentionHours writerConfig.RetentionPeriodHrs = opts.RetentionHours
writerConfig.BufferSize = opts.BufferSize writerConfig.BufferSize = opts.BufferSize
writerConfig.FlushIntervalMs = opts.FlushIntervalMs writerConfig.FlushIntervalMs = opts.FlushIntervalMs
// Sink logic // The payload is already formatted: the writer adds nothing and writes the file only
writerConfig.EnableConsole = false writerConfig.EnableConsole = false
writerConfig.EnableFile = true writerConfig.EnableFile = true
writerConfig.ShowTimestamp = false writerConfig.ShowTimestamp = false
@@ -97,7 +96,6 @@ func NewFileSinkPlugin(
} }
fs.lastProcessed.Store(time.Time{}) fs.lastProcessed.Store(time.Time{})
// Create session for file output
fs.session = proxy.CreateSession( fs.session = proxy.CreateSession(
fmt.Sprintf("file:///%s/%s", opts.Directory, opts.Name), fmt.Sprintf("file:///%s/%s", opts.Directory, opts.Name),
map[string]any{ map[string]any{
@@ -173,12 +171,10 @@ func (fs *FileSink) Stop() {
<-fs.exited <-fs.exited
} }
// Remove session
if fs.session != nil { if fs.session != nil {
fs.proxy.RemoveSession(fs.session.ID) fs.proxy.RemoveSession(fs.session.ID)
} }
// Shutdown the writer with timeout
if err := fs.writer.Shutdown(core.LoggerShutdownTimeout); err != nil { if err := fs.writer.Shutdown(core.LoggerShutdownTimeout); err != nil {
fs.logger.Error("msg", "Error shutting down file writer", fs.logger.Error("msg", "Error shutting down file writer",
"component", "file_sink", "component", "file_sink",
+5 -5
View File
@@ -452,7 +452,6 @@ func (h *HTTPSink) handleStream(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Accel-Buffering", "no") w.Header().Set("X-Accel-Buffering", "no")
w.WriteHeader(http.StatusOK) w.WriteHeader(http.StatusOK)
// Connected event with metadata, parity with fasthttp sink
info, _ := json.Marshal(map[string]any{ info, _ := json.Marshal(map[string]any{
"client_id": strconv.FormatUint(id, 10), "client_id": strconv.FormatUint(id, 10),
"session_id": sess.ID, "session_id": sess.ID,
@@ -621,10 +620,11 @@ func (h *HTTPSink) GetStats() sink.SinkStats {
// handlers; absent when auth is disabled // handlers; absent when auth is disabled
type identityKey struct{} type identityKey struct{}
// authMiddleware gates every endpoint on the policy: a client certificate // authMiddleware gates stream and status on the policy: a client
// or a bearer token. The rejection carries no detail: the status endpoint // certificate, a bearer token or, behind a proxy, the session cookie. The
// already exposes host, port, and throughput counters, so a refusal should // rejection carries no detail: the status endpoint already exposes host,
// not add the shape of the policy on top of that. // port, and throughput counters, so a refusal should not add the shape of
// the policy on top of that.
func (h *HTTPSink) authMiddleware(next http.Handler) http.Handler { func (h *HTTPSink) authMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ident, status, err := h.auth.AuthorizeRequest(r) ident, status, err := h.auth.AuthorizeRequest(r)
+1 -1
View File
@@ -373,7 +373,7 @@ func TestProxiedStreamSessionKeepsTheProxy(t *testing.T) {
// The constructor refuses what it cannot serve: the login pages outside // The constructor refuses what it cannot serve: the login pages outside
// scram's proxy mode, a viewer without the login page, a path under /auth, and // scram's proxy mode, a viewer without the login page, a path under /auth, and
// a path ServeMux or a URL would not take literally, which panicked at Start. // a path ServeMux or a URL would not take literally.
func TestUnservableOptionsAreRefused(t *testing.T) { func TestUnservableOptionsAreRefused(t *testing.T) {
scram := map[string]any{"type": "scram", "credentials_file": scramCredentials(t)} scram := map[string]any{"type": "scram", "credentials_file": scramCredentials(t)}
pki := testutil.NewPKI(t, "viewer-01") pki := testutil.NewPKI(t, "viewer-01")
+2 -2
View File
@@ -53,7 +53,7 @@ type HTTPChainSink struct {
tlsEnabled bool tlsEnabled bool
mtls bool mtls bool
// Authorization: pins the downstream server's identity // Authorization: mtls pins the downstream server's identity; scram logs this sink in
auth *authz.Policy auth *authz.Policy
client *http.Client client *http.Client
@@ -189,7 +189,7 @@ func (t *HTTPChainSink) Capabilities() []core.Capability {
caps = append(caps, core.CapTLS) caps = append(caps, core.CapTLS)
} }
if t.auth.Enabled() { if t.auth.Enabled() {
caps = append(caps, core.CapAuth) // pins the server identity caps = append(caps, core.CapAuth) // pins the server, or logs in under scram
} }
return caps return caps
} }
+1 -1
View File
@@ -9,7 +9,7 @@ import (
// Sink represents an output data stream. // Sink represents an output data stream.
type Sink interface { type Sink interface {
// Capabilities returns a slice of supported Source capabilities // Capabilities returns a slice of supported Sink capabilities
Capabilities() []core.Capability Capabilities() []core.Capability
// Input returns the channel for sending transport events to this sink. // Input returns the channel for sending transport events to this sink.
+5 -4
View File
@@ -221,8 +221,8 @@ func (t *TCPSink) Start(ctx context.Context) error {
go t.acceptLoop() go t.acceptLoop()
go t.broadcastLoop() go t.broadcastLoop()
// Pipeline context cancellation mirrors gnet engine stop: cease accepting // Pipeline context cancellation stops accepting and tears down the
// and tear down existing connections // connections, as Stop does
go func() { go func() {
select { select {
case <-ctx.Done(): case <-ctx.Done():
@@ -299,7 +299,8 @@ func (t *TCPSink) acceptLoop() {
} }
if t.config.MaxConnections > 0 && t.activeConns.Load() >= t.config.MaxConnections { if t.config.MaxConnections > 0 && t.activeConns.Load() >= t.config.MaxConnections {
// Load/admit race can over-admit by a conn under burst; acceptable // The count rises only in handleConn, so a burst can over-admit by
// the conns accepted before their handleConn runs; acceptable
t.rejectedConns.Add(1) t.rejectedConns.Add(1)
conn.Close() conn.Close()
continue continue
@@ -328,7 +329,7 @@ func (t *TCPSink) handleConn(conn net.Conn) {
defer t.wg.Done() defer t.wg.Done()
remote := conn.RemoteAddr().String() remote := conn.RemoteAddr().String()
// Counted from accept: max_connections bounds concurrent handshakes too // Counted before the handshake: max_connections bounds concurrent handshakes too
count := t.activeConns.Add(1) count := t.activeConns.Add(1)
defer func() { defer func() {
conn.Close() conn.Close()
+2 -2
View File
@@ -44,7 +44,7 @@ type TCPChainSink struct {
network string network string
tlsConfig *tls.Config tlsConfig *tls.Config
// Authorization: pins the downstream server's identity // Authorization: mtls pins the downstream server's identity; scram logs this sink in
auth *authz.Policy auth *authz.Policy
input chan core.TransportEvent input chan core.TransportEvent
@@ -158,7 +158,7 @@ func (t *TCPChainSink) Capabilities() []core.Capability {
caps = append(caps, core.CapTLS) caps = append(caps, core.CapTLS)
} }
if t.auth.Enabled() { if t.auth.Enabled() {
caps = append(caps, core.CapAuth) // pins the server identity caps = append(caps, core.CapAuth) // pins the server, or logs in under scram
} }
return caps return caps
} }
+1 -1
View File
@@ -158,7 +158,7 @@ func TestRemoveWatcherPreservesReplacement(t *testing.T) {
} }
// A line longer than core.MaxLogEntryBytes continues in the next entry, and the // A line longer than core.MaxLogEntryBytes continues in the next entry, and the
// lines around it are read once: the watcher used to stop at it for good // lines around it are read once
func TestLongLineContinuesInTheNextEntry(t *testing.T) { func TestLongLineContinuesInTheNextEntry(t *testing.T) {
path := filepath.Join(t.TempDir(), "app.log") path := filepath.Join(t.TempDir(), "app.log")
long := strings.Repeat("x", core.MaxLogEntryBytes+10) long := strings.Repeat("x", core.MaxLogEntryBytes+10)
+1 -1
View File
@@ -20,7 +20,7 @@ type Source interface {
// Stop gracefully shuts down the source // Stop gracefully shuts down the source
Stop() Stop()
// SourceStats contains statistics about a source // GetStats returns the source's statistics
GetStats() SourceStats GetStats() SourceStats
} }
+5 -3
View File
@@ -1,6 +1,8 @@
// Package tlsx builds crypto/tls configurations from config.TLSOptions. // Package tlsx is the TLS seam of the network plugins, beside authz and
// It is the single seam between declarative TLS config and the stdlib; // netacl: Server and Client build a plugin's crypto/tls configuration from
// each network plugin calls exactly one constructor. // config.TLSOptions, a listener without files gets a certificate issued on the
// process key, lw tls makes its CA and certificates here, and LogWarnings
// reports settings that work but are likely mistakes.
package tlsx package tlsx
import ( import (
+4 -3
View File
@@ -1,7 +1,8 @@
// Package tui is lw --tui: pipelines composed on a full-screen canvas drawn as // Package tui is lw --tui: pipelines composed on a full-screen canvas drawn as
// the website draws them, then run or written as a command line, environment // the website draws them, run in place with a live view of what their stages
// or file. The pipelines are a compose.Composition; this package only draws // pass and drop, then left to run or written as a command line, environment
// them and turns keys into the engine's edits. // or file. The pipelines are a compose.Composition; this package draws them,
// turns keys into the engine's edits and runs them through a Runner from cmd/lw.
package tui package tui
import ( import (
+2 -2
View File
@@ -187,8 +187,8 @@ func TestARowCountsAllItHolds(t *testing.T) {
// The canvas cursor picks the point and v the verdicts at it: a source's // The canvas cursor picks the point and v the verdicts at it: a source's
// entries, what reaches a stage or a filter, passes it or it drops, the // entries, what reaches a stage or a filter, passes it or it drops, the
// heartbeats, what the sinks got and what the flow dropped on the way; lw's // heartbeats, what the flow offered the sinks and what it dropped on the way;
// own log shows from warn up at every point, and whole at its own // lw's own log shows from warn up at every point, and whole at its own
func TestThePointAndModePickTheRows(t *testing.T) { func TestThePointAndModePickTheRows(t *testing.T) {
c := pipeline(t, []string{"null", "null"}, []string{"null"}) c := pipeline(t, []string{"null", "null"}, []string{"null"})
for _, typ := range []string{"include", "exclude"} { for _, typ := range []string{"include", "exclude"} {