LogWisp is a pipeline-based log transport and processing system written in Go. It collects log entries from files, stdin, or other LogWisp nodes; rate-limits, filters, and formats them; and distributes them to files, consoles, live network streams, or downstream LogWisp nodes.

Documentation Map

Installation: building, installing, and running as a service
Deployment: deploy/lw-deploy.sh for edges, aggregators, containers, services, jails
Architecture: component model, data flow, concurrency, back-pressure
Configuration: TOML structure, precedence, environment and CLI overrides
Sources: every input plugin and its options
Sinks: every output plugin and its options
Filters: pattern-based inclusion and exclusion
Formatters: output shaping and sanitization
Chaining: multi-node topologies and the chain wire protocol
Networking: listeners, dialers, timeouts, connection limits
Security: TLS, mTLS, and peer authorization; threat model and current limits
mTLS Authentication: design and rationale for certificate-based authorization
Password Authentication: design and rationale for Argon2id-SCRAM authentication; mTLS hardening
CLI: flags, signals, exit codes, lw auth; also the lw(1) manual, lw.1
Operations: running, monitoring, tuning, troubleshooting
To Do: planned work in priority order: network access control and the PROXY protocol, packaging

A fully annotated configuration covering every option lives at config/logwisp.toml.

Capabilities

Pipeline

A Unix filter without configuration: stdin to stdout, line for line
Independent named pipelines, each sources → flow → sinks
Fan-in (many sources per pipeline) and fan-out (many sinks per pipeline)
Isolated sinks: a stalled sink drops its own events rather than stall the pipeline; only the console sink waits, so a filter loses no line
Hot reload of pipeline configuration via SIGHUP/SIGUSR1 or a file watch

Inputs

file (directory tail with rotation detection), console (stdin, one reader per process), random (synthetic generator), null, and the chain ingest listeners tcp_chain and http_chain.

Outputs

console (control characters escaped on a terminal), file (rotating), http (Server-Sent Events plus a JSON status endpoint), tcp (broadcast server), null, and the chain forwarders tcp_chain and http_chain.

Processing

Token-bucket rate limiting with an optional per-entry size cap
Chainable include/exclude regex filters with or/and logic
raw, txt, and json formatting with selectable sanitizer policies
Optional flow-level heartbeat entries

Transport security and authentication

TLS 1.2/1.3 on every network source and sink, listener and dialer alike
Mutual TLS: listeners can require and verify client certificates; dialers can present a client identity
Authorization by certificate identity, per listener: named peers rather than everything the CA issued, with the http sink's endpoints gated too
Password authentication (Argon2id-SCRAM) bound to the listener's certificate, with bearer tokens on HTTP listeners and a lw auth CLI, and browser logins behind a TLS-terminating proxy
Node binding, so a chain source labels entries from the sender's certificate identity or username rather than from what the sender claims

See Security for what each layer does and does not give you.

Quick Start

Without a configuration file lw is a filter, exiting 0 at the end of input; a pipeline without --source reads stdin and one without --sink writes stdout:

lw < app.log > copy.log
tail -F app.log | lw --filter include,patterns=ERROR,patterns=WARN

As a service, a file holds the pipelines:

[[pipelines]]
name = "app"

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "app_logs"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "stdout"
type = "console"
[pipelines.plugin_sinks.config]
target = "stdout"

lw -c config.toml

System Requirements

Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
Architecture: amd64
Go: 1.27.1+ to build from source

Network sources and sinks bind and dial IPv4 or IPv6, each keeping strictly to the family of its host (Networking).

License

BSD 3-Clause.