lixen/logwisp
SHA256
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.34.0 doc sweep and update

Browse Source
This commit is contained in:
lixen committed 2026-10-11 03:27:10 -04:00
1 parent 87f5576ccf
commit 1abffcec6c
41 files changed
+1447 -2287
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/architecture.md
+108 -51
View File
@@ -7,27 +7,36 @@ plugins, sessions, TLS, statistics — hangs off that spine.
```
main
└── Service
├── Pipeline "app"
│ ├── Registry instance tracking, single-instance enforcement
│ ├── Session Manager per-pipeline connection/session bookkeeping
│ ├── Sources[] plugin instances, keyed by id
│ ├── Flow
│ │ ├── Rate Limiter optional, token bucket
│ │ ├── Filter Chain optional, ordered
│ │ ├── Formatter raw | txt | json, with sanitizer
│ │ └── Heartbeat optional generator
│ └── Sinks[] plugin instances, keyed by id
├── Pipeline "audit"
│ └── ...
└── Status Reporter optional, 30s interval
├── Service
│ ├── Pipeline "app"
│ │ ├── Registry creates each plugin from its factory
│ │ ├── Session Manager per-pipeline connection/session bookkeeping
│ │ ├── Sources[] plugin instances, keyed by id
│ │ ├── Flow
│ │ │ ├── Rate Limiter optional, token bucket
│ │ │ ├── Filter Chain optional, ordered
│ │ │ ├── Formatter raw | txt | json, with sanitizer
│ │ │ └── Heartbeat optional generator
│ │ └── Sinks[] plugin instances, keyed by id
│ └── Pipeline "audit"
│ └── ...
└── Status Reporter cmd/lw, optional, 30 s: Service.GetGlobalStats
```
Package map:
- `cmd/lw`: entry point, help, logger bootstrap, signal loop, status
reporter.
- `internal/config`: typed config schema, loading, top-level validation.
- `cmd/lw`: entry point: the command line (`cli.go`: lw's flags and the
`auth`, `tls`, `preset` and `config` commands), help, completion, logger
bootstrap, signal and reload loop, status reporter.
- `cmd/lwconf`: `internal/compose` as WebAssembly, and `lwconf.js`, which
loads it: see [Configuration Engine](#configuration-engine).
- `internal/config`: option schema (tags, catalogue) and `Decode`; loading
from file, environment and flags; presets; offline validation of every
setting and plugin option.
- `internal/compose`: the configuration engine `lw --tui` and `cmd/lwconf`
share. It edits pipelines and writes and reads back the command-line,
environment and file forms; it reads no file and links no TLS, HTTP or
terminal code.
- `internal/service`: owns the pipeline set; start, stop, shutdown, global
stats.
- `internal/pipeline`: pipeline runtime and per-pipeline plugin registry.
@@ -41,7 +50,8 @@ Package map:
- `internal/chain`: chain wire protocol: hello preamble, entry codec, backoff.
- `internal/tlsx`: the single seam between `TLSOptions` and `crypto/tls`.
- `internal/authz`: the single seam between `AuthOptions` and the network
plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`.
plugins: `Admit`, `AuthorizeRequest`, `Greet`, `Prepare`; `Authorize`
refuses under `scram`.
- `internal/netacl`: a listener's `acl` block: address rules, PROXY headers
and per-client limits; its `Table` is SCRAM throttling's limiter too.
- `internal/session`: session manager and per-instance proxy.
@@ -53,15 +63,16 @@ Package map:
- `internal/logline`: reads a line of any common shape into time, level,
tag, message and fields for the live view; holds the one rule for runes a
terminal acts on, which the console sink shares.
- `internal/version`: version, commit and build time, stamped at build.
- `internal/testutil`: fixtures for tests only.
## Plugin Registration
Every plugin registers itself in an `init()` function, and
`cmd/lw/bootstrap.go` blank-imports each package to trigger those
`init()`s. Adding a plugin therefore means writing the package, calling
`plugin.RegisterSource` / `plugin.RegisterSink`, adding one blank import, and
adding the type's row to the catalogue in `internal/config/schema.go`, which
registration requires.
`cmd/lw/bootstrap.go` blank-imports each package to trigger it. Adding a
plugin means writing the package, calling `plugin.RegisterSource` /
`plugin.RegisterSink`, adding one blank import, and adding the type's row to
the catalogue in `internal/config/schema.go`, which registration requires.
The row holds the type's options struct, whose tags declare its defaults and
rules, and whether the type is single. The `console` source is single,
@@ -77,10 +88,11 @@ instance of a single type fails, naming both pipelines.
starts. Publication is non-blocking: a full subscriber channel increments
the source's `dropped_entries` counter. The console source alone waits,
since stdin can simply be read later.
2. **Flow** applies, in order: rate limit → filter chain → formatter. A drop at
any stage ends the entry's life and increments `flow.total_dropped`. The
verdict, passed or the stage that dropped it (a filter by its index), goes
to the pipeline's tap when a viewer set one, as `lw --tui`'s live view does.
2. **Flow** applies, in order: rate limit → filter chain → formatter. A rate
limit or filter drop ends the entry's life and increments
`flow.total_dropped`. The verdict, passed or the stage that dropped it (a
filter by its index), goes to the pipeline's tap when a viewer set one, as
`lw --tui`'s live view does.
3. The formatter output becomes a `core.TransportEvent`, which carries both the
formatted `Payload` and the original structured `Entry`.
4. **Dispatch** sends the event to every sink's input channel with a
@@ -109,8 +121,7 @@ loses nothing. Each stage drops and counts instead of waiting:
- Source → subscriber: a full channel drops the entry; source
`dropped_entries`. The console source waits instead and reads stdin later.
- Flow: the rate limiter, a filter rejection or a format error drops it;
`flow.total_dropped`.
- Flow: the rate limiter or a filter rejection drops it; `flow.total_dropped`.
- Pipeline → sink: a full sink input drops it for that sink only; pipeline
`total_dropped_by_sink`. A sink declaring `core.CapBackpressure`, the console
and null sinks, makes dispatch wait instead, so a filter loses no line to a
@@ -124,11 +135,12 @@ loses nothing. Each stage drops and counts instead of waiting:
the view shows them.
The `tcp_chain` sink is the one deliberate exception. It holds a line across
reconnects until it is written or the process shuts down, so a downstream
outage propagates backwards as a full input buffer and surfaces as
`total_dropped_by_sink` on the pipeline rather than as silent data loss inside
the sink. The `http_chain` sink retries a batch with backoff, and drops it only
on a non-retryable response or on shutdown (`dropped_batches`).
reconnects until it is written or the sink stops (shutdown or reload, after
`sink.FlushBound`), so a downstream outage propagates backwards as a full
input buffer and surfaces as `total_dropped_by_sink` on the pipeline rather
than as silent data loss inside the sink. The `http_chain` sink retries a
batch with backoff, and drops it only on a non-retryable response or on
shutdown (`dropped_batches`).
## Concurrency Model
@@ -138,20 +150,23 @@ on a non-retryable response or on shutdown (`dropped_batches`).
- Each network sink runs one broadcast/broker goroutine plus, per connection, a
writer goroutine (and for TCP, a reader goroutine that exists only to detect
disconnects and refresh session activity).
- Chain sinks run a single run-loop goroutine that exclusively owns the
connection or the pending batch, so no locking is needed around either.
- Chain sinks run one run-loop goroutine that owns the pending line or batch;
the `tcp_chain` sink adds a reader goroutine per link that closes the link
when the source writes anything or closes it.
- Statistics are atomics; configuration and registries use RW mutexes;
shutdown is context cancellation plus wait groups.
### Shutdown ordering
`Pipeline.Stop` is deliberately ordered so in-flight data drains:
`Pipeline.Stop` is ordered so in-flight data drains:
1. Release dispatch waits on backpressure sinks; stop all sources
concurrently, each closing its subscriber channels.
2. Wait for the run loop, which ends when every subscription channel closes.
3. Stop all sinks concurrently; console and file sinks write their queue
first. Then cancel the pipeline context.
3. Stop all sinks concurrently: console and file sinks write their queue;
network sinks deliver theirs within `sink.FlushBound`, their write or
request timeout when set below 2 s, else 2 s. Then cancel the pipeline
context.
A finite source ends on its own: the console source at the end of stdin. The
run loop then returns and the pipeline's `Finished` channel closes; once every
@@ -176,7 +191,7 @@ The network plugins, by role:
- `http_chain` sink: chain protocol, batched POST with retry.
TLS is built in exactly one place, `internal/tlsx`, which exposes
`Server(opts)` for listeners and `Client(opts, host)` for dialers. See
`Server(opts, host)` for listeners and `Client(opts, host)` for dialers. See
[Security](security.md).
## Sessions
@@ -194,18 +209,17 @@ the corresponding SSE client.
Authorization decisions do not read session metadata — `internal/authz` makes
them from the handshake, the SCRAM exchange or a bearer token, at the point of
connection or request, and their outcome is *recorded* in the session. That
ordering matters: a session exists only for a peer that was already admitted.
See the [mTLS](mtls-auth-plan.md) and [SCRAM](scram-auth-plan.md)
authentication designs.
connection or request, and their outcome is *recorded* in the session, so a
session exists only for a peer that was already admitted. See
[Security](security.md#the-auth-block) and
[Chaining](chaining.md#wire-protocol).
## Configuration Reload
Reload (signal or file watch) rebuilds the entire service:
1. Signals reread the selected file; watches consume a newly published snapshot.
The application owns its manager and detached snapshots; no global mutable
target is shared with the running service. Validate every candidate again.
1. Signals (`SIGHUP`, `SIGUSR1`) reread the file; a watch takes the published
snapshot. Every candidate is validated again.
2. Build a **new** service from it. If construction fails, the old service keeps
running untouched.
3. Shut the old service down, start the new one, and restart the status
@@ -222,15 +236,58 @@ certificate rotation. Failed candidates release their session workers. Once the
old service has stopped, a bind/start failure requires another corrected reload
or process restart.
## Configuration Engine
`internal/compose` is what `lw --tui` edits with and what the website runs as
WebAssembly, so no part of LogWisp is rewritten in JavaScript. `make wasm`
writes the files a site drops in
([Installation](installation.md#building-from-source)): `lwconf.wasm` is
7.1 MB, 1.9 MB at gzip -9, and the toolchain's `wasm_exec.js` uses no eval,
so the page needs only `'wasm-unsafe-eval'` in its `script-src`.
`lwconf.js` is one dependency-free ES module that knows the calls and shapes,
never a key:
- `load({execURL, execSRI, wasmURL, wasmSRI})` fetches both by their
integrity and resolves to the engine.
- `fields(schema, role, type)` is a node's form rows, a table's a group.
- `value`, `at` and `group` read a node's value and sort problems by node.
The engine's calls take strings (`lwconf.js` writes objects as JSON) and
answer `{"value": ...}` or `{"error", "path", "pipeline", "node", "key"}`, as
far as the engine places the refusal:
- `schema()`: what `lw --schema` prints.
- `preset(name, values, dirs)`: a preset's pipelines; `values` maps a key to
a string, a list key to strings.
- `parse(text, form, dirs)`: the pipelines of a form's pasted text.
- `validate(pipelines)`: `{pipelines, problems}`, the pipelines settled
(defaults, kinds, ids) and the first problem of each part (a value of the
wrong kind too) as `{path, pipeline, node, key, text}`.
- `emit(pipelines, form)`: the text of `command`, `environment` or `file`.
- `edit(pipelines, op)`: `{pipelines, node}`, the node left chosen. `op` is
`{op, pipeline, ...}`:
- `add_pipeline` and `rename_pipeline` (`name`), `remove_pipeline`
- `add` (`role`, `type`), `retype` (`node`, `type`), `remove` (`node`)
- `move_filter` (`from`, `to`)
- `set` (`node`, `key`, `values`), `unset` (`node`, `key`)
- `dirs` answers whether a path a preset names is a directory,
`{"/var/log/app.log": false}`; unanswered, a path ending in `/` is one, one
with a glob a pattern, and any other is refused at `path`.
- Pipelines are a JSON list in the file's shape (`name`, `flow`,
`plugin_sources`, `plugin_sinks`); a misspelled key is refused. A node is
`{role, id, index}`: a source's or sink's id, a filter's index.
## Resource Management
- Every buffer is bounded; the drop-not-block policy keeps memory flat under
load.
- The `tcp` and `http` sinks and the `tcp_chain` source accept a
`max_connections` cap. Admission is a load-then-check, so a burst can
over-admit by roughly one connection.
- `max_connections` caps connections on the `tcp` sink and the `tcp_chain`
source, and streams on the `http` sink. Admission is a load-then-check, so a
burst can over-admit by roughly one.
- Chain listeners bound a single line at `core.MaxLogEntryBytes` (1 MiB); an
oversized line is a protocol violation and terminates the connection.
oversized line terminates a `tcp_chain` link and fails an `http_chain`
batch with `400`.
- The `http_chain` source caps each request body at `max_body_bytes`.
- File sinks rotate on size, cap total rotated size, and honour a retention
window.