logwisp/README.md

LogWisp

A pipeline-based log transport and processing system written in Go. LogWisp collects log entries from files, stdin, or other LogWisp nodes; rate-limits, filters, and formats them; and distributes them to files, consoles, live network streams, or downstream LogWisp nodes.

Features

Pipeline

• Independent pipelines, each sources → flow → sinks, running concurrently in one process
• Fan-in and fan-out: many sources and many sinks per pipeline
• Never blocks: a stalled sink drops its own events and is counted, rather than stalling the pipeline or its sibling sinks
• Hot reload via SIGHUP/SIGUSR1 or a config file watch, with the new configuration validated before the old service is torn down

Inputs

file (directory tail with rotation detection and JSON line parsing), console (stdin), random (synthetic generator), null, and the chain ingest listeners tcp_chain and http_chain.

Outputs

console, file (rotating with retention), http (Server-Sent Events plus a JSON status endpoint), tcp (broadcast server), null, and the chain forwarders tcp_chain and http_chain.

Processing

• Filters: chainable include/exclude RE2 patterns with or/and logic
• Formatters: raw, txt, and json with selectable sanitizer policies
• Rate limiting: token bucket with an optional per-entry size cap
• Heartbeats: flow-level keep-alive entries that reach every sink

Chaining

Multi-node topologies over a versioned protocol. Chain links carry the structured entry, not the formatted text, so a relay can filter and reformat as if the entries were local. Entries keep a node label identifying their origin across any number of hops. Chain sinks reconnect automatically with exponential backoff and jitter.

Transport security and authentication

• TLS 1.2/1.3 on every network source and sink, listener and dialer alike
• Mutual TLS: listeners can require and verify client certificates; dialers can present a client identity
• Authorization by certificate identity: an auth block admits named peers (exact or RE2) rather than everything the CA issued, gates the http sink's stream and status endpoints, and lets a dialer pin the server it talks to
• Password authentication (Argon2id-SCRAM) on the same block: listeners hold verifiers, never passwords, and do no KDF work; every login outside proxy mode is bound to the listener's certificate, so no relay presenting another one can use it; HTTP listeners issue short-lived bearer tokens. logwisp auth manages credentials files and logs viewers in; behind a site's TLS-terminating proxy, browsers log in through a shipped login page and a dependency-free JS client
• Node binding: a chain source can label entries from the sender's certificate identity or username instead of from what the sender claims, so origin attribution is not forgeable
• Fail-closed configuration: unknown keys are rejected, and startup warns about expiring certificates, disabled verification, unanchored allow patterns and world-readable secrets

See Security for configuration and the exact boundary, and the mTLS and SCRAM authentication designs for the rationale and what is deliberately left out.

Documentation

Document	Contents
Installation	Building, installing, running as a service
Architecture	Component model, data flow, concurrency, back-pressure
Configuration	TOML structure, precedence, environment and CLI overrides
Sources	Every input plugin and its options
Sinks	Every output plugin and its options
Filters	Pattern-based inclusion and exclusion
Formatters	Output shaping and sanitization
Chaining	Multi-node topologies and the chain wire protocol
Networking	Listeners, dialers, timeouts, connection limits
Security	TLS, mTLS, and peer authorization; threat model and current limits
mTLS Authentication	Design and rationale for certificate-based authorization
Password Authentication	Design and rationale for Argon2id-SCRAM authentication; mTLS hardening
CLI	Flags, signals, exit codes, logwisp auth
Operations	Running, monitoring, tuning, troubleshooting

A fully annotated configuration covering every option ships as config/logwisp.toml.

Quick Start

make

# logwisp.toml
[[pipelines]]
name = "default"

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "app_logs"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "stdout"
type = "console"
[pipelines.plugin_sinks.config]
target = "stdout"

logwisp -c logwisp.toml

Running with no configuration file starts a self-demonstrating pipeline: a synthetic generator writing JSON to stdout.

System Requirements

• Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
• Architecture: amd64
• Go: 1.27.1+ to build from source

Network sources and sinks bind and dial over IPv4 only.

License

BSD 3-Clause License