logwisp/doc/architecture.md

Architecture Overview

LogWisp moves log entries through independent pipelines. Everything else — plugins, sessions, TLS, statistics — hangs off that spine.

Component Hierarchy

main
├── Service
│   ├── Pipeline "app"
│   │   ├── Registry           creates each plugin from its factory
│   │   ├── Session Manager    per-pipeline connection/session bookkeeping
│   │   ├── Sources[]          plugin instances, keyed by id
│   │   ├── Flow
│   │   │   ├── Rate Limiter   optional, token bucket
│   │   │   ├── Filter Chain   optional, ordered
│   │   │   ├── Formatter      raw | txt | json, with sanitizer
│   │   │   └── Heartbeat      optional generator
│   │   └── Sinks[]            plugin instances, keyed by id
│   └── Pipeline "audit"
│       └── ...
└── Status Reporter            cmd/lw, optional, 30 s: Service.GetGlobalStats

Package map:

• cmd/lw: entry point: the command line (cli.go: lw's flags and the auth, tls, preset and config commands), help, completion, logger bootstrap, signal and reload loop, status reporter.
• cmd/lwconf: internal/compose as WebAssembly, and lwconf.js, which loads it: see Configuration Engine.
• internal/config: option schema (tags, catalogue) and Decode; loading from file, environment and flags; presets; offline validation of every setting and plugin option.
• internal/compose: the configuration engine lw --tui and cmd/lwconf share. It edits pipelines and writes and reads back the command-line, environment and file forms; it reads no file and links no TLS, HTTP or terminal code.
• internal/service: owns the pipeline set; start, stop, shutdown, global stats.
• internal/pipeline: pipeline runtime and per-pipeline plugin registry.
• internal/flow: rate limiter, filter chain invocation, formatting, heartbeat.
• internal/filter: regex filter and filter chain.
• internal/format: adapter over the lixenwraith/log formatter and sanitizer.
• internal/source, internal/sink: the Source and Sink interfaces and their stats, and the stop helpers network sinks share (sink.FlushBound, FlushContext, Drain); their subpackages are the plugins.
• internal/plugin: global factory registry populated by plugin init().
• internal/chain: chain wire protocol: hello preamble, entry codec, backoff.
• internal/tlsx: the single seam between TLSOptions and crypto/tls; also the certificates lw tls writes and a listener makes at startup, pins and startup warnings.
• internal/authz: the single seam between AuthOptions and the network plugins: Admit, AuthorizeRequest, Greet, Prepare; Authorize refuses under scram.
• internal/netacl: a listener's acl block: address rules, PROXY headers and per-client limits; its Table is SCRAM throttling's limiter too.
• internal/session: session manager and per-instance proxy.
• internal/core: shared types (LogEntry, TransportEvent, the viewer's Tap), the level table core.Levels, capabilities, constants, and Network and Listen, which keep a socket to its host's address family.
• internal/tokenbucket: rate limiter primitive.
• internal/tui: lw --tui, the diagram and the live view of what t runs; cmd/lw/screen.go runs the pipelines for it.
• internal/logline: reads a line of any common shape into time, level, tag, message and fields for the live view; holds the one rule for runes a terminal acts on, which the console sink shares.
• internal/version: version, commit and build time, stamped at build.
• internal/testutil: fixtures for tests only.
• config: embeds logwisp.toml, the annotated default configuration that lw config init writes and the packages install.

Plugin Registration

Every plugin registers itself in an init() function, and cmd/lw/bootstrap.go blank-imports each package to trigger it. Adding a plugin means writing the package, calling plugin.RegisterSource / plugin.RegisterSink, adding one blank import, and adding the type's row to the catalogue in internal/config/schema.go, which registration requires.

The row holds the type's options struct, whose tags declare its defaults and rules, and whether the type is single. The console source is single, because a process has only one stdin; loading a configuration with a second instance of a single type fails, naming both pipelines.

Data Flow

Entry lifecycle

1. Source produces a core.LogEntry and publishes it to every subscriber channel it has handed out; the pipeline subscribes before the source starts. Publication is non-blocking: a full subscriber channel increments the source's dropped_entries counter. The console source alone waits, since stdin can simply be read later.
2. Flow applies, in order: rate limit → filter chain → formatter. A rate limit or filter drop ends the entry's life and increments flow.total_dropped. The verdict, passed or the stage that dropped it (a filter by its index), goes to the pipeline's tap when a viewer set one, as lw --tui's live view does.
3. The formatter output becomes a core.TransportEvent, which carries both the formatted Payload and the original structured Entry.
4. Dispatch sends the event to every sink's input channel with a non-blocking send, except to a sink declaring CapBackpressure.

A heartbeat joins at step 4: the flow writes it in its format with a structured Entry, and the pipeline offers it to the tap and dispatches it, so no rate limit or filter drops it and flow.total_processed does not count it.

LogEntry fields:

• Time: entry timestamp.
• Node: origin node label for chained topologies; stamped at the first hop, preserved by relays.
• Source: origin identifier within the node (filename, plugin id, …).
• Level: TRACE, DEBUG, INFO, WARN or ERROR, when detected; the words naming each are one table, core.Levels.
• Message: log content.
• Fields: optional structured metadata as raw JSON.
• RawSize: original byte size, used by the entry-size cap.

Carrying Entry alongside Payload is what makes chain sinks format-independent: a tcp_chain or http_chain sink re-serializes the structured entry rather than shipping whatever text the local formatter chose.

Back-pressure and drops

The drop policy is not configurable: never block, except where waiting loses nothing. Each stage drops and counts instead of waiting:

• Source → subscriber: a full channel drops the entry; source dropped_entries. The console source waits instead and reads stdin later.
• Flow: the rate limiter or a filter rejection drops it; flow.total_dropped.
• Pipeline → sink: a full sink input drops it for that sink only; pipeline total_dropped_by_sink. A sink declaring core.CapBackpressure, the console and null sinks, makes dispatch wait instead, so a filter loses no line to a slow reader and its pipeline runs at the reader's pace; Stop ends the wait.
• TCP/HTTP sink → client queue: a full queue drops it for that client only; sink dropped_writes.
• Flow → tap: a full tap drops the record, copying nothing, which the live view counts as "screen missed". It copies the first 16 KiB of a longer message, fields or payload and 256 bytes of a level, source or node, so a viewer's rows pin no source's line, and takes dropped entries only while the view shows them.

The tcp_chain sink is the one deliberate exception. It holds a line across reconnects until it is written or the sink stops (shutdown or reload, after sink.FlushBound), so a downstream outage propagates backwards as a full input buffer and surfaces as total_dropped_by_sink on the pipeline rather than as silent data loss inside the sink. The http_chain sink retries a batch with backoff, and drops it only on a non-retryable response or on shutdown (dropped_batches).

Concurrency Model

• One goroutine per source drains that source's subscription and feeds the flow.
• The flow's formatter holds a mutex; the underlying formatter reuses an internal buffer and is not goroutine-safe.
• Each network sink runs one broadcast/broker goroutine plus, per connection, a writer goroutine (and for TCP, a reader goroutine that exists only to detect disconnects and refresh session activity).
• Chain sinks run one run-loop goroutine that owns the pending line or batch; the tcp_chain sink adds a reader goroutine per link that closes the link when the source writes anything or closes it.
• Statistics are atomics; configuration and registries use RW mutexes; shutdown is context cancellation plus wait groups.

Shutdown ordering

Pipeline.Stop is ordered so in-flight data drains:

1. Release dispatch waits on backpressure sinks; stop all sources concurrently, each closing its subscriber channels. From the release on, dispatch picks at random between a backpressure sink's input and the release, so it drops about half of the entries still in flight for one (total_dropped_by_sink) even with room.
2. Wait for the run loop, which ends when every subscription channel closes.
3. Stop all sinks concurrently: console and file sinks write their queue; network sinks deliver theirs within sink.FlushBound, their write or request timeout when set below 2 s, else 2 s. Then cancel the pipeline context.

A finite source ends on its own: the console source at the end of stdin. The run loop then returns and the pipeline's Finished channel closes; once every pipeline has finished, the service's Done closes and main shuts down as on SIGTERM, exiting 0.

Network Architecture

Each listener and dialer keeps strictly to the family of its host literal: tcp4 for IPv4, IPv6-only tcp6 for IPv6 (:: too); a hostname resolves. See Networking.

The network plugins, by role:

• Listeners
  • tcp sink: raw broadcast of formatted payloads.
  • http sink: HTTP/1.1 SSE; HTTP/2 negotiated via ALPN when TLS is on.
  • tcp_chain source: chain protocol, persistent NDJSON stream.
  • http_chain source: chain protocol, NDJSON batches over POST.
• Dialers
  • tcp_chain sink: chain protocol, persistent stream, auto-reconnect.
  • http_chain sink: chain protocol, batched POST with retry.

TLS is built in exactly one place, internal/tlsx, which exposes Server(opts, host) for listeners and Client(opts, host) for dialers. See Security.

Sessions

Each pipeline owns a session.Manager. Plugins receive a session.Proxy scoped to their id, so a plugin sees and removes only the sessions under its id; ids are unique per list, so a source and a sink that share one (the built-in pipe's two consoles) share that scope. A session records the remote address, creation and last-activity timestamps, and metadata — including tls and tls_peer_cn for TLS peers, auth_method / auth_identity for authorized ones, and peer_addr, the proxy, for a client a PROXY header names.

Idle sessions are reaped every 5 minutes against a 30-minute idle limit. The HTTP sink's broker treats a vanished session as an eviction signal and closes the corresponding SSE client.

Authorization decisions do not read session metadata — internal/authz makes them from the handshake, the SCRAM exchange or a bearer token, at the point of connection or request, and their outcome is recorded in the session, so a session exists only for a peer that was already admitted. See Security and Chaining.

Configuration Reload

Reload (signal or file watch) rebuilds the entire service:

1. Signals (SIGHUP, SIGUSR1) reread the file; a watch takes the published snapshot. Every candidate is validated again.
2. Build a new service from it. If construction fails, the old service keeps running untouched.
3. Shut the old service down, start the new one, and restart the status reporter if it is enabled.

Because this is a full rebuild, listening sockets close and reopen and all clients are disconnected. Application logging is configured once at startup and is not re-applied on reload. t on the lw --tui screen reruns through the same path, giving each new service the screen's tap before it starts.

Watch errors do not rebuild services. Queued changes are combined, and an unchanged pipeline/status snapshot is skipped. Signals always rebuild for certificate rotation. Failed candidates release their session workers. Once the old service has stopped, a bind/start failure requires another corrected reload or process restart.

Configuration Engine

internal/compose is what lw --tui edits with and what the website runs as WebAssembly, so no part of LogWisp is rewritten in JavaScript. make wasm writes the files a site drops in (Installation): lwconf.wasm is 7.1 MB, 1.9 MB at gzip -9, and the toolchain's wasm_exec.js uses no eval, so the page needs only 'wasm-unsafe-eval' in its script-src.

lwconf.js is one dependency-free ES module that knows the calls and shapes, never a key:

• load({execURL, execSRI, wasmURL, wasmSRI, progress}) fetches both by their integrity and resolves to the engine; progress hears the module's bytes as they arrive, which fetch's own integrity check would hold back.
• fields(schema, role, type) is a node's form rows, a table's a group.
• value, at and group read a node's value and sort problems by node.

The engine's calls take strings (lwconf.js writes objects as JSON) and answer {"value": ...} or {"error", "path", "pipeline", "node", "key", "ask"}, as far as the engine places the refusal:

• schema(): what lw --schema prints.
• preset(name, values, dirs): a preset's pipelines; values maps a key to a string, a list key to strings.
• parse(text, form, dirs): {pipelines, left_out}, the pipelines of a form's pasted text and the names of what it held besides them: a file's settings, an environment's LOGWISP_ variables no pipeline flag has. Text in another form's shape is refused naming that form.
• validate(pipelines): {pipelines, problems}, the pipelines settled (kinds, ids) and every problem of each part, each key's and the first rule between a table's keys (a value of the wrong kind too), as {path, pipeline, node, key, text}.
• emit(pipelines, form): the text of command, environment or file.
• edit(pipelines, op): {pipelines, node}, the node left chosen. op is {op, pipeline, ...}:
  • add_pipeline and rename_pipeline (name), remove_pipeline
  • add (role, type), retype (node, type), remove (node)
  • move_filter (from, to)
  • set (node, key, values), unset (node, key)
• dirs answers whether a path a preset names is a directory, {"/var/log/app.log": false}; unanswered, a path ending in / is one, one with a glob a pattern, and any other is refused at path with ask naming it: answer it in dirs and call again.
• Pipelines are a JSON list in the file's shape (name, flow, plugin_sources, plugin_sinks); a misspelled key is refused. A node is {role, id, index}: source or sink by its id, filters by its index, or the flow stage rate_limit, format or heartbeat, which set turns on and remove off; add takes source, sink, or filters with the type include, exclude or "" for the default.
• Stages and options hold the keys set, so the forms carry no default an edit did not set: a stage key's zero is unset, and an option's zero is unset where it takes the key's default.

Resource Management

• Every buffer is bounded; the drop-not-block policy keeps memory flat under load.
• max_connections caps connections on the tcp sink and the tcp_chain source, and streams on the http sink. Admission is a load-then-check, so a burst can over-admit by the connections that arrive before the count rises.
• A listener's acl block caps each client: max_connections_per_client, and on the http sink and http_chain source requests_per_second_per_client (Security).
• Chain listeners bound a single line at core.MaxLogEntryBytes (1 MiB); an oversized line terminates a tcp_chain link and fails an http_chain batch with 400.
• The http_chain source caps each request body at max_body_bytes.
• File sinks rotate on size, cap total rotated size, honour a retention window, and with min_disk_free_mb delete the oldest files to keep that much disk free.

Performance Notes

• In-memory entry processing is sub-millisecond. Besides the formatter's mutex, an entry can take the rate limiter's bucket lock (policy = "drop") and the pipeline's session-manager lock, which the network plugins and the console and random sources take as entries pass, to refresh session activity.
• File tailing detects new content within roughly 100 ms (fixed poll), while check_interval_ms governs how quickly a newly created file is noticed.
• http_chain trades latency for efficiency: entries wait up to flush_interval_ms (default 1 s) before a batch is sent.
• Scale out with more pipelines per process, more sinks per pipeline, or more nodes chained together.