logwisp/doc/operations.md

Operations Guide

Running, monitoring, and maintaining LogWisp.

Starting

# a filter: stdin to stdout, exits 0 at the end of input
lw < app.log > copy.log
tail -F app.log | lw --filter include,patterns=ERROR

# a service: the file's pipelines, until SIGINT or SIGTERM
lw -c /etc/logwisp/logwisp.toml

lw exits on its own only at the end of input: once every pipeline has finished, a pipeline finishing when all its sources have ended. Only a console source ends, when stdin does; file and network sources run until a signal, and lw shuts down gracefully, exit 0, either way. Every sink writes what is queued before the exit; a network sink gives its clients or its downstream its write_timeout_ms (request_timeout_ms for http_chain), at most 2 s, to take it. A reload stops the old sinks the same way, so a stalled client or downstream delays it by that much. See CLI.

There is no built-in daemon mode. Run LogWisp in the foreground under a supervisor — systemd, rc.d, or a container runtime — which is where restart, log capture, and resource limits belong. See Installation.

A supervisor's stdin is /dev/null (systemd's default, docker run without -i), where a console source ends at once. So a service defines its pipelines: a file without [[pipelines]] runs the built-in stdin-to-stdout pipeline and exits 0 at once, which Restart=on-failure does not restart, and a container configured through variables needs a LOGWISP_SOURCE.

systemd

sudo systemctl start logwisp
sudo systemctl status logwisp
sudo journalctl -u logwisp -f

FreeBSD rc.d

sudo service logwisp start
sudo service logwisp status

Configuration Changes

Hot reload

auto_reload = true

or send a signal:

kill -HUP $(pidof lw)

Signals reread the selected TOML file even when auto_reload is disabled. Reload validates a detached snapshot and constructs a new service before tearing the old one down, so invalid configuration leaves the running service intact and logs the failure:

ERROR msg="Failed to bootstrap new service, keeping old service running" error=...

What reload does not do:

• Re-apply logging.*; application logging is configured once at startup.
• Preserve connections. Listeners close and reopen, and every SSE, TCP, and chain client is disconnected. Chain sinks reconnect on their own backoff; browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
• Notice rotated certificates on its own: certificate files are read at plugin construction and auto_reload watches only the configuration file, so rotation requires SIGHUP.

Standard input survives a reload: one reader serves the process, so the new console source continues where the old one stopped, and no line is read twice.

Plan reloads on a busy relay the way you would plan a restart. Listener bind/start failures happen after the old service stops; these can leave the application without working pipelines until a corrected configuration reloads. File-watch errors do not restart services, and queued changes are combined.

Checking a configuration

lw --check builds every pipeline and plugin as a start would, without binding a port or opening, reading or creating a log file, prints the startup warnings, and exits 0, 1 on an invalid configuration, or 2 when a named file is missing:

lw --check -c candidate.toml

Before a reload, check the edited file this way: a failed reload keeps the running service, but --check says why without touching it. Failures name the pipeline and the offending key:

ERROR msg="Failed to create pipeline" pipeline=app error="failed to create sink out: port: must be 1-65535, got 0"

Remember that most validation lives in plugin constructors, so a config only proves itself when the pipeline is actually built.

Monitoring

Status reporter

Enabled by default with a configuration file (off without one), every 30 seconds. It logs at DEBUG, so it produces nothing unless logging.level = "debug" — a common surprise.

status_reporter = true

[logging]
level = "debug"

It emits a service summary and then walks each pipeline, flattening scalar statistics into log fields and recursing into flow, rate limiter, filter, source, and sink stats. Each plugin's details are spread into its line, so the auth and TLS rejection counters show up there.

Disable with status_reporter = false.

HTTP status endpoint

When a pipeline has an http sink:

curl -s http://127.0.0.1:8080/status | jq .

{
  "service": "LogWisp",
  "version": "v0.16.0",
  "instance_id": "sse",
  "server": {
    "type": "http",
    "host": "0.0.0.0",
    "port": 8080,
    "tls": false,
    "active_clients": 3,
    "buffer_size": 1000,
    "client_buffer_size": 256,
    "max_connections": 32,
    "write_timeout_ms": 5000,
    "uptime_seconds": 8130
  },
  "endpoints": { "stream": "/stream", "status": "/status" },
  "statistics": {
    "total_processed": 15234,
    "dropped_writes": 12,
    "rejected_clients": 0
  }
}

This endpoint is scoped to one sink, not to the whole process, and without an auth block it is unauthenticated. Bind it to a trusted interface, or query it with a client certificate or a token from lw auth token.

Metrics worth watching

Each counter, where it is reported, and what a rise means:

• dropped_entries (source): downstream cannot keep up with the source; the console source never drops, it reads stdin slower
• total_dropped (flow): the rate limit or filters are discarding entries, often by intent
• total_dropped_by_sink (pipeline): a sink's input queue is full; a console sink waits instead, so its entries count only when shutdown cuts the wait
• dropped_writes (tcp and http sinks): a client's queue overflowed, because it is too slow or one burst exceeded client_buffer_size
• rejected_conns (tcp sink, tcp_chain source), rejected_clients (http sink): max_connections is being hit; rejected_conns also counts auth refusals
• tls_handshake_errors (tcp sink, tcp_chain source): a certificate or version mismatch, or scanning
• parse_errors (chain sources): protocol or version skew upstream
• reconnects (tcp_chain sink): an unstable link or a flapping downstream
• dropped_batches (http_chain sink): downstream is rejecting batches permanently
• synthesized (chain sinks): events reach the sink without structure

Log Management

LogWisp's own operational log goes to stderr by default, at info with a configuration file and at warn without one; explicit values win. A container configured only through variables or flags therefore logs at warn: pass -e LOGWISP_LOGGING_LEVEL=info. lw-deploy.sh always passes -c, so its nodes log at info. To files:

[logging]
output = "file"
level  = "info"

[logging.file]
directory         = "/var/log/logwisp"
name              = "logwisp"
max_size_mb       = 100
max_total_size_mb = 1000
retention_hours   = 168.0

Rotation is automatic on size, with a total-size cap and a retention window. There is no signal to reopen log files, so do not move files out from under LogWisp and expect it to reattach — let it rotate, or restart it.

Production level: info, or warn on a busy relay. Avoid debug under load: the filter stage logs several lines per entry evaluated.

Performance Tuning

Buffers

Raise buffer_size when total_dropped_by_sink is climbing but the sink itself is healthy — that is a burst-absorption problem.

dropped_writes on a network sink has two causes that a counter alone does not separate. A consumer slower than the sustained rate cannot be bought off with buffer, and drops are the intended outcome. A burst the consumer would have drained, arriving faster than it reads, is configuration: the sink queues a whole burst while the client writes one frame at a time, so the part of a burst above client_buffer_size is lost even to a loopback reader. Where a rate_limit bounds the pipeline, its burst is that number — keep client_buffer_size at or above it and the second cause disappears. The HTTP status endpoint reports both queue bounds alongside the counters so an operator can tell which one is in play.

[pipelines.plugin_sinks.config]
buffer_size        = 5000
client_buffer_size = 1024

Rate limiting

[pipelines.flow.rate_limit]
rate                 = 1000.0
burst                = 2000.0
policy               = "drop"
max_entry_size_bytes = 65536

Two behaviours to keep in mind: the limiter does not exist at all when rate <= 0, and policy = "pass" short-circuits the size cap as well as the rate check. Enforcing max_entry_size_bytes therefore requires rate > 0 and policy = "drop".

Formatting

raw is the cheapest and skips sanitization; json costs the most. The formatter serializes on a mutex, so it is the one shared bottleneck in a pipeline — splitting work across pipelines parallelizes it.

Chain batching

http_chain trades latency for efficiency. Lower flush_interval_ms for freshness, raise max_batch_count and max_batch_bytes for throughput. Use tcp_chain when per-entry latency matters.

Troubleshooting

Nothing appears at the sink

Walk the pipeline in order and read the counters: source total_entries (is anything being produced?), flow total_dropped (filters or rate limit?), pipeline total_dropped_by_sink (sink backed up?), sink total_processed.

A pipeline stalls behind a console sink

A console sink never drops: when its output is slow (a paused terminal, a full pipe, a stalled log collector) its pipeline waits, and the pipeline's other sinks with it. Stdin is read slower; other sources queue, then drop and count dropped_entries. Where a service's stdout may stall, give it a file sink instead. A closed stdout pipe ends lw with SIGPIPE.

File source reads nothing

• The watcher seeks to end-of-file on start; only content appended afterwards is read. Positions are in memory, so a restart re-seeks to end and anything written during the downtime is lost. from = "start" reads each file whole instead, and replays it on every restart.
• pattern is a filename glob with * and ? only, and matching is not recursive.
• check_interval_ms governs how quickly a new file is noticed; tailing an open file polls at a fixed 100 ms.

High memory use

Buffers are bounded, so unbounded growth almost always means many buffers: count sinks × buffer_size, plus clients × client_buffer_size. A tcp_chain sink blocked on an unreachable downstream also holds its full input queue.

Chain link not delivering

Check connected and reconnects on a tcp_chain sink, parse_errors on the source, and remember http_chain waits up to flush_interval_ms. For TLS problems see Networking.

Environment variable override has no effect

LogWisp reads LOGWISP_QUIET, LOGWISP_LOGGING_LEVEL, and other prefixed names. Bare names used by older versions must be renamed. Array-indexed paths, such as a key inside [[pipelines]], cannot be set from the environment or the command line; define whole pipelines there with the pipeline flags or pipeline variables.

Security Operations

Certificate rotation

openssl x509 -in /etc/logwisp/tls/relay.crt -noout -enddate

Certificates and SCRAM credentials files load at plugin construction, so rotation is: write the new files, then kill -HUP. Startup and every reload warn at WARN once a certificate is within 30 days of expiry; automate the check anyway. For passwords see rotation and revocation.

Access review

With tls alone, any certificate signed by the configured client_ca_file is accepted, so "access review" means reviewing what your CA has issued. Add an auth block with an explicit allow list, or scram with its credentials file, and the review becomes those files: the identities or users listed there are the ones that can connect, and removing one plus a SIGHUP is the revocation path. Authorized identities are recorded in session metadata as auth_identity; rejections are counted in auth_rejected and logged at WARN. See Security.

Secret leakage

Filters are the only redaction mechanism, and they drop whole entries rather than masking parts of them. See Filters.

Maintenance

Upgrades

1. Read the changelog for configuration-schema changes.
2. Start the new binary against the current configuration in a scratch environment.
3. Stop the old process, install the new binary, start it.
4. Confirm each pipeline started and that counters are advancing.

Backup

Configuration files and TLS material are the only durable state worth backing up. LogWisp keeps no persistent runtime state: file read positions live in memory, connections are re-established on restart, and in-flight entries are lost.

Redundancy

Because there is no persistence, availability comes from topology, not from LogWisp itself. Give each edge two chain sinks pointing at two relays if you need to survive a relay outage, and accept that this duplicates entries downstream.