A pipeline-based log transport and processing system written in Go. LogWisp collects log entries from files, stdin, or other LogWisp nodes; rate-limits, filters, and formats them; and distributes them to files, consoles, live network streams, or downstream LogWisp nodes.

Features

Pipeline

A Unix filter out of the box: with no configuration, lw copies stdin to stdout line for line and exits at the end of input; flags add filters, formats, sources and sinks
Independent pipelines, each sources → flow → sinks, running concurrently in one process
Fan-in and fan-out: many sources and many sinks per pipeline
Isolated sinks: a stalled sink drops and counts its own events rather than stalling the pipeline or its sibling sinks; only the console sink waits, so a filter loses no line to a slow reader
Hot reload via SIGHUP/SIGUSR1 or a config file watch, with the new configuration validated before the old service is torn down

Inputs

file (directory tail with rotation detection and JSON line parsing), console (stdin, one reader per process), random (synthetic generator), null, and the chain ingest listeners tcp_chain and http_chain.

Outputs

console (control characters escaped on a terminal), file (rotating with retention), http (Server-Sent Events plus a JSON status endpoint), tcp (broadcast server), null, and the chain forwarders tcp_chain and http_chain.

Processing

Filters: chainable include/exclude RE2 patterns with or/and logic
Formatters: raw, txt, and json with selectable sanitizer policies
Rate limiting: token bucket with an optional per-entry size cap
Heartbeats: flow-level keep-alive entries that reach every sink

Chaining

Multi-node topologies over a versioned protocol. Chain links carry the structured entry, not the formatted text, so a relay can filter and reformat as if the entries were local. Entries keep a node label identifying their origin across any number of hops. Chain sinks reconnect automatically with exponential backoff and jitter.

Transport security and authentication

TLS 1.2/1.3 on every network source and sink, listener and dialer alike
Mutual TLS: listeners can require and verify client certificates; dialers can present a client identity
Authorization by certificate identity: an auth block admits named peers (exact or RE2) rather than everything the CA issued, gates the http sink's stream and status endpoints, and lets a dialer pin the server it talks to
Password authentication (Argon2id-SCRAM) on the same block: listeners hold verifiers, never passwords, and do no KDF work; every login outside proxy mode is bound to the listener's certificate, so no relay presenting another one can use it; HTTP listeners issue short-lived bearer tokens. lw auth manages credentials files and logs viewers in; behind a site's TLS-terminating proxy, browsers log in through a shipped login page and a dependency-free JS client
Node binding: a chain source can label entries from the sender's certificate identity or username instead of from what the sender claims, so origin attribution is not forgeable
Fail-closed configuration: unknown keys are rejected, and startup warns about expiring certificates, disabled verification, unanchored allow patterns and world-readable secrets

See Security for configuration and the exact boundary, and the mTLS and SCRAM authentication designs for the rationale and what is deliberately left out.

Documentation

Installation: building, installing, services, the container image, packaging
Deployment: deploy/lw-deploy.sh for edges, aggregators, containers, services, jails
Architecture: component model, data flow, concurrency, back-pressure
Configuration: TOML structure, precedence, environment and CLI overrides
Sources and Sinks: every plugin and its options
Filters and Formatters: pattern matching, output shaping and sanitization
Chaining: multi-node topologies and the wire protocol
Networking: listeners, dialers, timeouts, limits
Security: TLS, mTLS, peer authorization, threat model
Design notes: mTLS and password authentication
CLI: filter use, flags, pipeline specifications, signals, exit codes, lw auth; also the lw(1) manual, doc/lw.1
Operations: running, monitoring, tuning, troubleshooting
To Do: planned work in priority order: network access control and the PROXY protocol, a hardening review of the config libraries, packaging

A fully annotated configuration covering every option ships as config/logwisp.toml.

Quick Start

make build                      # plain `make` lists the targets

Without a configuration file, lw is a filter: stdin to stdout, line for line, exiting 0 at the end of input. A pipeline without --source reads stdin, one without --sink writes stdout:

./bin/lw < app.log > copy.log
tail -F app.log | ./bin/lw --filter include,patterns=ERROR,patterns=WARN

# stdin as a live SSE stream; the http sink binds 0.0.0.0 unless told
journalctl -f | ./bin/lw --sink http,host=127.0.0.1,port=8080

# tail a directory instead of stdin
./bin/lw --source 'file,directory=/var/log/myapp,pattern=*.log' \
    --format json,sanitizer_policy=json > all.json

As a service, a configuration file holds the pipelines:

# logwisp.toml
[[pipelines]]
name = "app"

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "app_logs"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "sse"
type = "http"
[pipelines.plugin_sinks.config]
host = "127.0.0.1"
port = 8080

./bin/lw -c logwisp.toml                # until SIGINT or SIGTERM; logs to stderr
curl -N http://127.0.0.1:8080/stream    # from another shell

In a container, read-only and without capabilities:

make image
docker run --rm --read-only --cap-drop ALL --security-opt no-new-privileges \
    -v /etc/logwisp:/etc/logwisp:ro logwisp:dev -c /etc/logwisp/logwisp.toml

sudo make install installs the binary, the manual, a sample configuration and a systemd unit or FreeBSD rc.d script; see Installation.

System Requirements

Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
Architecture: amd64
Go: 1.27.1+ to build from source

Network sources and sinks use IPv4 or IPv6, following the address family of the configured host.

License

BSD 3-Clause License