logwisp/doc/operations.md

Operations Guide

Running, monitoring, and maintaining LogWisp.

Starting

# a filter: stdin to stdout, exits 0 at the end of input
lw < app.log > copy.log
tail -F app.log | lw --filter include:patterns=ERROR

# a service: the file's pipelines, until SIGINT or SIGTERM
lw -c /etc/logwisp/logwisp.toml

lw exits on its own only at the end of input (CLI). A reload stops the old sinks as shutdown does, so a stalled client or downstream delays it by up to 2 s.

There is no built-in daemon mode. Run LogWisp in the foreground under a supervisor — systemd, rc.d, or a container runtime — which is where restart, log capture, and resource limits belong. See Installation.

A supervisor's stdin is /dev/null (systemd's default, docker run without -i), where a console source ends at once. So a service defines its pipelines: a file without [[pipelines]] runs the built-in stdin-to-stdout pipeline and exits 0 at once, which Restart=on-failure does not restart, and a container configured through variables needs a LOGWISP_SOURCE, or a LOGWISP_PRESET with a source of its own (aggregator, or one given a path).

systemd

sudo systemctl start logwisp
sudo systemctl status logwisp
sudo journalctl -u logwisp -f

FreeBSD rc.d

sudo service logwisp start
sudo service logwisp status

Configuration Changes

Hot reload

auto_reload = true

or send SIGHUP or SIGUSR1; the packaged unit and rc.d script send SIGHUP on reload:

kill -HUP $(pidof lw)
sudo systemctl reload logwisp     # rc.d: sudo service logwisp reload

Signals reread the selected TOML file even when auto_reload is disabled. Reload validates a detached snapshot and builds a new service before tearing the old one down, so a bad configuration leaves the running service intact. An invalid one logs Failed to read reload configuration with the validator's validate configuration: <key path>: ... error. One whose files fail when its plugins are built (a missing certificate, an unreadable credentials file) logs Failed to bootstrap new service, keeping old service running.

What reload does not do:

• Re-apply logging.*, quiet or auto_reload; they take effect at startup only.
• Preserve connections. Listeners close and reopen, and every SSE, TCP, and chain client is disconnected. Chain sinks reconnect on their own backoff; browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
• Keep bearer tokens. Each reload draws a new token key, so every token is revoked: run lw auth token again, and browser viewers log in again.
• Notice rotated certificates on its own: certificate files are read at plugin construction and auto_reload watches only the configuration file, so rotation requires SIGHUP.

Standard input survives a reload: one reader serves the process, so the new console source continues where the old one stopped, and no line is read twice.

Plan reloads on a busy relay as you would a restart: a listener that fails to bind does so after the old service stopped, leaving lw without working pipelines until a corrected configuration reloads (Hot Reload).

Checking a configuration

Before a reload, run lw --check -c FILE on the edited file: a failed reload keeps the running service, and --check says why without touching it (CLI).

Monitoring

Status reporter

Enabled by default with a configuration file (off without one), every 30 seconds. It logs at DEBUG, so it produces nothing unless logging.level = "debug" — a common surprise.

status_reporter = true

[logging]
level = "debug"

It emits a service summary and then walks each pipeline, flattening scalar statistics into log fields and recursing into flow, rate limiter, filter, source, and sink stats. Each plugin's details are spread into its line, so the auth and TLS rejection counters show up there.

Disable with status_reporter = false.

HTTP status endpoint

When a pipeline has an http sink:

curl -s http://127.0.0.1:8080/status | jq .

{
  "endpoints": {
    "status": "/status",
    "stream": "/stream"
  },
  "instance_id": "sse",
  "server": {
    "acl": "none",
    "active_clients": 3,
    "auth": "none",
    "buffer_size": 1000,
    "client_buffer_size": 256,
    "host": "0.0.0.0",
    "max_connections": 32,
    "port": 8080,
    "tls": false,
    "type": "http",
    "uptime_seconds": 8130,
    "write_timeout_ms": 5000
  },
  "service": "LogWisp",
  "statistics": {
    "acl_denied": 0,
    "acl_limited": 0,
    "auth_rejected": 0,
    "dropped_writes": 12,
    "rejected_clients": 0,
    "total_processed": 15234
  },
  "version": "vX.Y.Z"
}

This endpoint is scoped to one sink, not to the whole process, and without an auth block it is unauthenticated. Bind it to a trusted interface, or query it with a client certificate or a token from lw auth token.

Metrics worth watching

Each counter, where it is reported, and what a rise means:

• dropped_entries (source): downstream cannot keep up with the source; the console source never drops, it reads stdin slower
• total_dropped (flow): the rate limit or filters are discarding entries, often by intent
• total_dropped_by_sink (pipeline): a sink's input queue is full; a console sink waits instead, so its entries count only when shutdown cuts the wait
• dropped_writes (tcp and http sinks): a client's queue overflowed, because it is too slow or one burst exceeded client_buffer_size
• rejected_conns (tcp sink, tcp_chain source), rejected_clients (http sink): max_connections is being hit; rejected_conns also counts auth refusals
• tls_handshake_errors (tcp sink, tcp_chain source): a certificate or version mismatch, or scanning
• parse_errors (chain sources): malformed entry lines from upstream, each skipped
• reconnects (tcp_chain sink): an unstable link or a flapping downstream
• dropped_batches (http_chain sink): downstream is rejecting batches permanently
• request_errors (http_chain sink): failed posts, each retried under backoff unless refused for good
• rejected_requests (http_chain source): batches refused for auth, protocol version or node binding, or a body over max_body_bytes, holding a line over 1 MiB, or cut short
• synthesized (chain sinks): events reach the sink without structure
• acl_denied, acl_limited (a listener whose acl block sets a rule or limit; the http status always): connections or requests the address rules or PROXY-header checks refused, and those over a per-client limit; acl_proxy_headers counts the PROXY headers read

Log Management

LogWisp's own log goes to stderr by default. Without a configuration file lw logs less (CLI), so a container configured only through variables passes LOGWISP_LOGGING_LEVEL=info (debug, with LOGWISP_STATUS_REPORTER=true, for the 30 s statistics); the image sets neither, as a variable would override a mounted file's value. lw-deploy.sh always passes -c, so its nodes log at info. To files:

[logging]
output = "file"
level  = "info"

[logging.file]
directory         = "/var/log/logwisp"
name              = "logwisp"
max_size_mb       = 100
max_total_size_mb = 1000
retention_hours   = 168.0

Rotation is automatic on size, with a total-size cap and a retention window. There is no signal to reopen log files, so do not move files out from under LogWisp and expect it to reattach — let it rotate, or restart it.

Production level: info, or warn on a busy relay. Avoid debug under load: the filter stage logs several lines per entry evaluated.

Performance Tuning

Buffers

• total_dropped_by_sink climbing while the sink is healthy: a burst outran its queue; raise buffer_size
• dropped_writes on a network sink:
  • a client slower than the sustained rate: drops are intended
  • a burst above client_buffer_size, lost even to a fast reader: keep client_buffer_size at or above the burst a drop rate limit lets through, its burst_entries (entries_per_second when 0)
• the HTTP status endpoint reports both queue bounds beside the counters

[pipelines.plugin_sinks.config]
buffer_size        = 5000
client_buffer_size = 1024

Rate limiting

[pipelines.flow.rate_limit]
entries_per_second   = 1000.0
burst_entries        = 2000.0
policy               = "drop"
max_entry_size_bytes = 65536

Enforcing max_entry_size_bytes needs a rate above 0 and policy = "drop" (a file's default is pass); see Rate limiting.

Formatting

raw is the cheapest and skips sanitization; json costs the most. The formatter serializes on a mutex, so it is the one shared bottleneck in a pipeline — splitting work across pipelines parallelizes it.

Chain batching

http_chain trades latency for efficiency. Lower flush_interval_ms for freshness, raise max_batch_count and max_batch_bytes for throughput. Use tcp_chain when per-entry latency matters.

Troubleshooting

Nothing appears at the sink

Walk the pipeline in order and read the counters: source total_entries (is anything being produced?), flow total_dropped (filters or rate limit?), pipeline total_dropped_by_sink (sink backed up?), sink total_processed.

A pipeline stalls behind a console sink

A console sink never drops while its pipeline runs: when its output is slow (a paused terminal, a full pipe, a stalled log collector) its pipeline waits, and the pipeline's other sinks with it. Stdin is read slower; other sources queue, then drop and count dropped_entries. Where a service's stdout may stall, give it a file sink instead. A closed stdout pipe ends lw with SIGPIPE.

File source reads nothing

• A watcher seeks to end-of-file when its file is first found, at start or by a later scan, so a new file's lines written before that scan are not read. Positions are in memory, so a restart or a reload re-seeks to end, and anything written meanwhile is lost. from = "start" reads each file whole instead, and replays it on every restart and reload.
• pattern is a filename glob with * and ? only, and matching is not recursive.
• check_interval_ms governs how quickly a new file is noticed; tailing an open file polls at a fixed 100 ms.
• lw's log says why: INFO Waiting for directory while the directory is missing, WARN Cannot read directory or Cannot read file when permissions or a service sandbox refuse it, and WARN Not following lw's own output for the file lw's stdout or stderr goes to.

High memory use

Buffers are bounded, so unbounded growth almost always means many buffers: count each source's and sink's buffer_size (a fixed 1000 on the file, random and null sources and the null sink), clients × client_buffer_size, and each http sink's replay_lines. A tcp_chain sink blocked on an unreachable downstream also holds its full input queue.

Chain link not delivering

Check connected and reconnects on a tcp_chain sink, request_errors on an http_chain sink, rejected_requests (http_chain) and parse_errors on the source, and remember http_chain waits up to flush_interval_ms. For TLS problems see Networking.

Environment variable override has no effect

Only LOGWISP_-prefixed names are read (LOGWISP_QUIET, LOGWISP_LOGGING_LEVEL). Array-indexed paths, such as a key inside [[pipelines]], cannot be set from the environment or the command line; define whole pipelines there with the pipeline flags or pipeline variables.

Security Operations

Certificate rotation

openssl x509 -in /etc/logwisp/tls/relay.crt -noout -enddate

Certificates and SCRAM credentials files load at plugin construction, so rotation is: write the new files, then kill -HUP. Startup and every reload warn at WARN once a certificate is within 30 days of expiry; automate the check anyway. For passwords see rotation and revocation.

Access review

With tls alone, any certificate signed by the configured client_ca_file is accepted, so "access review" means reviewing what your CA has issued. Add an auth block with an explicit allow list, or scram with its credentials file, and the review becomes those files: the identities or users listed there are the ones that can connect, and removing one plus a SIGHUP is the revocation path. Authorized identities are recorded in session metadata as auth_identity; rejections are counted in auth_rejected and logged at WARN. See Security.

Secret leakage

Filters are the only redaction mechanism, and they drop whole entries rather than masking parts of them. See Filters.

Maintenance

Upgrades

1. Run the new binary's lw --check -c FILE against the current configuration: unknown or renamed keys fail there.
2. Stop the old process, install the new binary, start it. A self_signed listener makes a new key at every start: give its dialers the new pin_sha256 from its WARN line.
3. Confirm each pipeline started and that counters are advancing.

Backup

Configuration files and TLS material are the only durable state worth backing up. LogWisp keeps no persistent runtime state: file read positions live in memory, connections are re-established on restart, and in-flight entries are lost.

Redundancy

Because there is no persistence, availability comes from topology, not from LogWisp itself. Give each edge two chain sinks pointing at two relays if you need to survive a relay outage, and accept that this duplicates entries downstream.