LogWisp

LogWisp

A pipeline-based log transport and processing system written in Go. LogWisp collects log entries from files, stdin, or other LogWisp nodes; rate-limits, filters, and formats them; and distributes them to files, consoles, live network streams, or downstream LogWisp nodes.

Features

Pipeline

• Independent pipelines, each sources → flow → sinks, running concurrently in one process
• Fan-in and fan-out: many sources and many sinks per pipeline
• Never blocks: a stalled sink drops its own events and is counted, rather than stalling the pipeline or its sibling sinks
• Hot reload via SIGHUP/SIGUSR1 or a config file watch, with the new configuration validated before the old service is torn down

Inputs

file (directory tail with rotation detection and JSON line parsing), console (stdin), random (synthetic generator), null, and the chain ingest listeners tcp_chain and http_chain.

Outputs

console, file (rotating with retention), http (Server-Sent Events plus a JSON status endpoint), tcp (broadcast server), null, and the chain forwarders tcp_chain and http_chain.

Processing

• Filters: chainable include/exclude RE2 patterns with or/and logic
• Formatters: raw, txt, and json with selectable sanitizer policies
• Rate limiting: token bucket with an optional per-entry size cap
• Heartbeats: flow-level keep-alive entries that reach every sink

Chaining

Multi-node topologies over a versioned protocol. Chain links carry the structured entry, not the formatted text, so a relay can filter and reformat as if the entries were local. Entries keep a node label identifying their origin across any number of hops. Chain sinks reconnect automatically with exponential backoff and jitter.

Transport security

• TLS 1.2/1.3 on every network source and sink, listener and dialer alike
• Mutual TLS: listeners can require and verify client certificates; dialers can present a client identity

mTLS is currently a CA-wide membership check — any certificate the configured CA issued is accepted, and the peer's Common Name is recorded but not used for authorization. See Security for the exact boundary and the mTLS authentication plan for the proposed work. Password, token, and SCRAM authentication were removed during the restructure and are not currently available.

Documentation

Document	Contents
Installation	Building, installing, running as a service
Architecture	Component model, data flow, concurrency, back-pressure
Configuration	TOML structure, precedence, environment and CLI overrides
Sources	Every input plugin and its options
Sinks	Every output plugin and its options
Filters	Pattern-based inclusion and exclusion
Formatters	Output shaping and sanitization
Chaining	Multi-node topologies and the chain wire protocol
Networking	Listeners, dialers, timeouts, connection limits
Security	TLS and mTLS configuration, threat model, current limits
mTLS Authentication Plan	Design for certificate-based authorization
CLI	Flags, signals, exit codes
Operations	Running, monitoring, tuning, troubleshooting

A fully annotated configuration covering every option ships as config/logwisp.toml.

Quick Start

make

# logwisp.toml
[[pipelines]]
name = "default"

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "app_logs"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "stdout"
type = "console"
[pipelines.plugin_sinks.config]
target = "stdout"

logwisp -c logwisp.toml

Running with no configuration file starts a self-demonstrating pipeline: a synthetic generator writing JSON to stdout.

System Requirements

• Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
• Architecture: amd64
• Go: 1.26+ to build from source

Network sources and sinks bind and dial over IPv4 only.

License

BSD 3-Clause License