logwisp/README.md

LogWisp

A pipeline-based log transport and processing system written in Go. LogWisp collects log entries from files, stdin, or other LogWisp nodes; rate-limits, filters, and formats them; and distributes them to files, consoles, live network streams, or downstream LogWisp nodes.

Features

Pipeline

• A Unix filter out of the box: with no configuration, lw copies stdin to stdout line for line and exits at the end of input; flags add filters, formats, sources and sinks
• Presets: pipe, tail, serve, edge and aggregator build the common pipelines from a few keys (lw --preset tail:path=/var/log/app); lw --dump turns any command line into a configuration file
• Independent pipelines, each sources → flow → sinks, running concurrently in one process
• Fan-in and fan-out: many sources and many sinks per pipeline
• Isolated sinks: a stalled sink drops and counts its own events rather than stalling the pipeline or its sibling sinks; only the console sink waits, so a filter loses no line to a slow reader
• Hot reload via SIGHUP/SIGUSR1 or a config file watch, with the new configuration validated before the old service is torn down

On screen

lw --tui (-T) composes pipelines on a full-screen terminal, with the configuration engine the website's configurator runs as WebAssembly:

• Start anywhere: from the pipelines lw would run with the same options (-c FILE, pipeline flags, a discovered file), from a preset, or from a pasted lw command line
• Edit with the schema: sources, flow stages and sinks side by side; the inspector edits each part's options from their defaults, every edit is checked, and ! lists the problems
• Run or print: r runs the pipelines; o also prints them as a command line, environment variables or a TOML file once the screen closes (lw -T > pipelines.toml)
• Tap: t runs them in place and opens a live view of every entry with its fate, passed or the stage or filter that dropped it; e edits that stage, and t again reruns as a reload, keeping the running pipelines when the new ones do not build. Listeners bind, dialers connect and files are written for real; a password is asked in a masked dialog
• Read any log: the view parses LogWisp's JSON, lw's own log, vif records, pino, zap, slog, docker, journald, ECS, GCP, logfmt and stamped text into time, level, message and fields, filters them by level or pattern, and shows the payload as the sinks receive it. It never slows a pipeline: what it misses is counted
• Any terminal: cmd | lw -T taps piped input, since the screen draws on the controlling terminal; colors range from true color to none, and a text console gets CP437 glyphs

Inputs

file (a directory's files tailed through rotation, a missing directory waited for, JSON lines parsed), console (stdin, one reader per process), random (generated entries, to check a deployment before real logs), null (nothing; holds a source's place until one is chosen), and the chain ingest listeners tcp_chain and http_chain.

Outputs

console (level names in color and control characters escaped on a terminal), file (rotating with retention), http (Server-Sent Events, a JSON status endpoint and a browser viewer), tcp (broadcast server), null (discards; holds a sink's place until one is chosen), and the chain forwarders tcp_chain and http_chain.

Processing

• Filters: chainable include/exclude RE2 patterns with or/and logic
• Formatters: raw, txt, and json with selectable sanitizer policies
• Rate limiting: token bucket with an optional per-entry size cap
• Heartbeats: flow-level keep-alive entries that reach every sink

Chaining

Multi-node topologies over a versioned protocol. Chain links carry the structured entry, not the formatted text, so a relay can filter and reformat as if the entries were local. Entries keep a node label identifying their origin across any number of hops. Chain sinks reconnect automatically with exponential backoff and jitter.

Transport security and authentication

• TLS 1.2/1.3 on every network source and sink, listener and dialer alike
• Mutual TLS: listeners can require and verify client certificates; dialers can present a client identity
• Certificates without a PKI: lw tls makes a CA and certificates, and a listener can make its own at startup, self-signed (dialers pin its key with pin_sha256) or signed by a CA
• Authorization by certificate identity: an auth block admits named peers (exact or RE2) rather than everything the CA issued, gates the http sink's stream and status endpoints, and lets a dialer pin the server it talks to
• Password authentication (Argon2id-SCRAM) on the same block: no login costs a listener KDF work, and with a credentials file it holds verifiers, never passwords; every login outside proxy mode is bound to the listener's certificate, so no relay presenting another one can use it; HTTP listeners issue short-lived bearer tokens. lw auth manages credentials files and logs viewers in; behind a site's TLS-terminating proxy, browsers log in through a shipped login page and a dependency-free JS client
• Node binding: a chain source can label entries from the sender's certificate identity or username instead of from what the sender claims, so origin attribution is not forgeable
• Address rules: an acl block admits or refuses peers by address, or by the client an L4 proxy's PROXY header names, and limits each client
• Fail-closed configuration: every option is validated when the file loads, unknown keys included, and startup warns about expiring certificates, disabled verification, unanchored allow patterns and world-readable secrets

See Security for configuration and the exact boundary, and Current State for what is left out.

Documentation

• Installation: building, installing, services, the container image, packaging, the end-to-end test scripts
• Deployment: deploy/lw-deploy.sh for edges, aggregators, containers, services, jails
• Architecture: component model, data flow, concurrency, back-pressure, the configuration engine's WebAssembly API
• Configuration: TOML structure, precedence, environment and CLI overrides
• Sources and Sinks: every plugin and its options
• Filters and Formatters: pattern matching, output shaping and sanitization
• Chaining: multi-node topologies and the wire protocol
• Networking: listeners, dialers, timeouts, limits
• Security: TLS, mTLS, password (SCRAM) authentication, address ACLs, certificates made at startup; threat model and current limits
• CLI: filter use, flags, pipeline specifications, presets, environment, signals, exit codes, lw --tui, --schema, lw auth, lw tls, lw preset, lw config; also the lw(1) manual, doc/lw.1
• Operations: running, monitoring, tuning, troubleshooting
• To Do: the maintainer's release and packaging steps, and what lw --tui and the schema leave for later

A fully annotated configuration covering every option ships as config/logwisp.toml.

Quick Start

make build                      # plain `make` lists the targets

Without a configuration file, lw is a filter: stdin to stdout, line for line, exiting 0 at the end of input. A pipeline without --source reads stdin, one without --sink writes stdout:

./bin/lw < app.log > copy.log
tail -F app.log | ./bin/lw --filter include:patterns=ERROR,patterns=WARN

# stdin as a live SSE stream, with a browser viewer at http://127.0.0.1:8080/;
# the http sink binds 0.0.0.0 unless told
journalctl -f | ./bin/lw --sink http:host=127.0.0.1,port=8080

# tail a directory instead of stdin
./bin/lw --source 'file:directory=/var/log/myapp,pattern=*.log' \
    --format json:sanitizer_policy=json > all.json

# the same files with a preset, raw to stdout; on a terminal the level
# names are in color
./bin/lw --preset 'tail:path=/var/log/myapp/*.log'

# compose on screen from a preset menu; r runs, t taps, o prints
./bin/lw -T

# tap a stream: t shows each line parsed, and its fate
journalctl -f -o json | ./bin/lw -T --preset pipe

As a service, a configuration file holds the pipelines:

# logwisp.toml
[[pipelines]]
name = "app"

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "app_logs"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "sse"
type = "http"
[pipelines.plugin_sinks.config]
host = "127.0.0.1"
port = 8080

./bin/lw -c logwisp.toml                # until SIGINT or SIGTERM; logs to stderr
curl -N http://127.0.0.1:8080/stream    # from another shell

In a container, read-only and without capabilities:

make image
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
    -v /etc/logwisp:/etc/logwisp:ro logwisp:dev -c /etc/logwisp/logwisp.toml

make release && sudo make install installs the binary, the manual, shell completion, a systemd unit or FreeBSD rc.d script and its configuration. Without root, make release install installs lw, its manual, completion and documentation under ~/.local, and no service; lw config init writes the annotated configuration it reads. make deb and make arch package them for Debian, Ubuntu and Arch. See Installation.

System Requirements

• Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
• Architecture: amd64; arm64 builds but is untested
• To build: Go 1.27.3+, and GNU make or BSD make

Network sources and sinks use IPv4 or IPv6, following the address family of the configured host.

License

BSD 3-Clause License