logwisp/doc/operations.md

# Operations Guide

Running, monitoring, and maintaining LogWisp.

## Starting

```bash
# a filter: stdin to stdout, exits 0 at the end of input
lw < app.log > copy.log
tail -F app.log | lw --filter include,patterns=ERROR

# a service: the file's pipelines, until SIGINT or SIGTERM
lw -c /etc/logwisp/logwisp.toml
```

lw exits on its own only at the end of input: once every pipeline has
finished, a pipeline finishing when all its sources have ended. Only a console
source ends, when stdin does; file and network sources run until a signal, and
lw shuts down gracefully, exit 0, either way.
Every sink writes what is queued before the exit; a network sink gives its
clients or its downstream its `write_timeout_ms` (`request_timeout_ms` for
`http_chain`), at most 2 s, to take it. A reload stops the old sinks the same
way, so a stalled client or downstream delays it by that much. See
[CLI](cli.md#built-in-defaults).

There is no built-in daemon mode. Run LogWisp in the foreground under a
supervisor — systemd, rc.d, or a container runtime — which is where restart,
log capture, and resource limits belong. See [Installation](installation.md).

A supervisor's stdin is `/dev/null` (systemd's default, `docker run` without
`-i`), where a console source ends at once. So a service defines its
pipelines: a file without `[[pipelines]]` runs the built-in stdin-to-stdout
pipeline and exits 0 at once, which `Restart=on-failure` does not restart, and
a container configured through variables needs a `LOGWISP_SOURCE`.

**systemd**

```bash
sudo systemctl start logwisp
sudo systemctl status logwisp
sudo journalctl -u logwisp -f
```

**FreeBSD rc.d**

```bash
sudo service logwisp start
sudo service logwisp status
```

## Configuration Changes

### Hot reload

```toml
auto_reload = true
```

or send a signal:

```bash
kill -HUP $(pidof lw)
```

Signals reread the selected TOML file even when `auto_reload` is disabled.
Reload validates a detached snapshot and constructs a new service **before**
tearing the old one down, so invalid configuration leaves the running service
intact and logs the failure:

```
ERROR msg="Failed to bootstrap new service, keeping old service running" error=...
```

What reload does *not* do:

- Re-apply `logging.*`; application logging is configured once at startup.
- Preserve connections. Listeners close and reopen, and every SSE, TCP, and
  chain client is disconnected. Chain sinks reconnect on their own backoff;
  browsers reconnect SSE automatically; raw TCP consumers must retry themselves.
- Notice rotated certificates on its own: certificate files are read at plugin
  construction and `auto_reload` watches only the configuration file, so
  rotation requires `SIGHUP`.

Standard input survives a reload: one reader serves the process, so the new
console source continues where the old one stopped, and no line is read twice.

Plan reloads on a busy relay the way you would plan a restart.
Listener bind/start failures happen after the old service stops; these can leave
the application without working pipelines until a corrected configuration reloads.
File-watch errors do not restart services, and queued changes are combined.

### Checking a configuration

`lw --check` builds every pipeline and plugin as a start would, without binding
a port or opening, reading or creating a log file, prints the startup warnings,
and exits 0, 1 on an invalid configuration, or 2 when a named file is missing:

```bash
lw --check -c candidate.toml
```

Before a reload, check the edited file this way: a failed reload keeps the
running service, but `--check` says why without touching it. Failures name the
pipeline and the offending key:

```
ERROR msg="Failed to create pipeline" pipeline=app error="failed to create sink out: port: must be 1-65535, got 0"
```

Remember that most validation lives in plugin constructors, so a config only
proves itself when the pipeline is actually built.

## Monitoring

### Status reporter

Enabled by default with a configuration file (off without one), every 30
seconds. It logs at **DEBUG**, so it produces nothing unless
`logging.level = "debug"` — a common surprise.

```toml
status_reporter = true

[logging]
level = "debug"
```

It emits a service summary and then walks each pipeline, flattening scalar
statistics into log fields and recursing into flow, rate limiter, filter,
source, and sink stats. Each plugin's `details` are spread into its line, so
the auth and TLS rejection counters show up there.

Disable with `status_reporter = false`.

### HTTP status endpoint

When a pipeline has an `http` sink:

```bash
curl -s http://127.0.0.1:8080/status | jq .
```

```json
{
  "service": "LogWisp",
  "version": "v0.16.0",
  "instance_id": "sse",
  "server": {
    "type": "http",
    "host": "0.0.0.0",
    "port": 8080,
    "tls": false,
    "active_clients": 3,
    "buffer_size": 1000,
    "client_buffer_size": 256,
    "max_connections": 32,
    "write_timeout_ms": 5000,
    "uptime_seconds": 8130
  },
  "endpoints": { "stream": "/stream", "status": "/status" },
  "statistics": {
    "total_processed": 15234,
    "dropped_writes": 12,
    "rejected_clients": 0
  }
}
```

This endpoint is scoped to one sink, not to the whole process, and without an
`auth` block it is **unauthenticated**. Bind it to a trusted interface, or
query it with a client certificate or a token from `lw auth token`.

### Metrics worth watching

Each counter, where it is reported, and what a rise means:

- `dropped_entries` (source): downstream cannot keep up with the source; the
  console source never drops, it reads stdin slower
- `total_dropped` (flow): the rate limit or filters are discarding entries,
  often by intent
- `total_dropped_by_sink` (pipeline): a sink's input queue is full; a console
  sink waits instead, so its entries count only when shutdown cuts the wait
- `dropped_writes` (`tcp` and `http` sinks): a client's queue overflowed,
  because it is too slow or one burst exceeded `client_buffer_size`
- `rejected_conns` (`tcp` sink, `tcp_chain` source), `rejected_clients`
  (`http` sink): `max_connections` is being hit; `rejected_conns` also counts
  auth refusals
- `tls_handshake_errors` (`tcp` sink, `tcp_chain` source): a certificate or
  version mismatch, or scanning
- `parse_errors` (chain sources): protocol or version skew upstream
- `reconnects` (`tcp_chain` sink): an unstable link or a flapping downstream
- `dropped_batches` (`http_chain` sink): downstream is rejecting batches
  permanently
- `synthesized` (chain sinks): events reach the sink without structure

## Log Management

LogWisp's own operational log goes to stderr by default, at `info` with a
configuration file and at `warn` without one; explicit values win. A container
configured only through variables or flags therefore logs at `warn`: pass
`-e LOGWISP_LOGGING_LEVEL=info`. `lw-deploy.sh` always passes `-c`, so its
nodes log at `info`. To files:

```toml
[logging]
output = "file"
level  = "info"

[logging.file]
directory         = "/var/log/logwisp"
name              = "logwisp"
max_size_mb       = 100
max_total_size_mb = 1000
retention_hours   = 168.0
```

Rotation is automatic on size, with a total-size cap and a retention window.
There is no signal to reopen log files, so do not move files out from under
LogWisp and expect it to reattach — let it rotate, or restart it.

Production level: `info`, or `warn` on a busy relay. Avoid `debug` under load:
the filter stage logs several lines per entry evaluated.

## Performance Tuning

### Buffers

Raise `buffer_size` when `total_dropped_by_sink` is climbing but the sink itself
is healthy — that is a burst-absorption problem.

`dropped_writes` on a network sink has two causes that a counter alone does not
separate. A consumer slower than the sustained rate cannot be bought off with
buffer, and drops are the intended outcome. A burst the consumer would have
drained, arriving faster than it reads, is configuration: the sink queues a
whole burst while the client writes one frame at a time, so the part of a burst
above `client_buffer_size` is lost even to a loopback reader.
Where a `rate_limit` bounds the pipeline, its `burst` is that number — keep
`client_buffer_size` at or above it and the second cause disappears. The HTTP
status endpoint reports both queue bounds alongside the counters so an operator
can tell which one is in play.

```toml
[pipelines.plugin_sinks.config]
buffer_size        = 5000
client_buffer_size = 1024
```

### Rate limiting

```toml
[pipelines.flow.rate_limit]
rate                 = 1000.0
burst                = 2000.0
policy               = "drop"
max_entry_size_bytes = 65536
```

Two behaviours to keep in mind: the limiter does not exist at all when
`rate <= 0`, and `policy = "pass"` short-circuits the size cap as well as the
rate check. Enforcing `max_entry_size_bytes` therefore requires `rate > 0` and
`policy = "drop"`.

### Formatting

`raw` is the cheapest and skips sanitization; `json` costs the most. The
formatter serializes on a mutex, so it is the one shared bottleneck in a
pipeline — splitting work across pipelines parallelizes it.

### Chain batching

`http_chain` trades latency for efficiency. Lower `flush_interval_ms` for
freshness, raise `max_batch_count` and `max_batch_bytes` for throughput. Use
`tcp_chain` when per-entry latency matters.

## Troubleshooting

**Nothing appears at the sink**

Walk the pipeline in order and read the counters: source `total_entries` (is
anything being produced?), flow `total_dropped` (filters or rate limit?),
pipeline `total_dropped_by_sink` (sink backed up?), sink `total_processed`.

**A pipeline stalls behind a console sink**

A console sink never drops: when its output is slow (a paused terminal, a full
pipe, a stalled log collector) its pipeline waits, and the pipeline's other
sinks with it. Stdin is read slower; other sources queue, then drop and count
`dropped_entries`. Where a service's stdout may stall, give it a `file` sink
instead. A closed stdout pipe ends lw with `SIGPIPE`.

**File source reads nothing**

- The watcher seeks to end-of-file on start; only content appended afterwards is
  read. Positions are in memory, so a restart re-seeks to end and anything
  written during the downtime is lost. `from = "start"` reads each file whole
  instead, and replays it on every restart.
- `pattern` is a filename glob with `*` and `?` only, and matching is not
  recursive.
- `check_interval_ms` governs how quickly a *new file* is noticed; tailing an
  open file polls at a fixed 100 ms.

**High memory use**

Buffers are bounded, so unbounded growth almost always means many buffers:
count sinks × `buffer_size`, plus clients × `client_buffer_size`. A `tcp_chain`
sink blocked on an unreachable downstream also holds its full input queue.

**Chain link not delivering**

Check `connected` and `reconnects` on a `tcp_chain` sink, `parse_errors` on the
source, and remember `http_chain` waits up to `flush_interval_ms`. For TLS
problems see [Networking](networking.md#troubleshooting).

**Environment variable override has no effect**

LogWisp reads `LOGWISP_QUIET`, `LOGWISP_LOGGING_LEVEL`, and other prefixed
names. Bare names used by older versions must be renamed. Array-indexed paths,
such as a key inside `[[pipelines]]`, cannot be set from the environment or the
command line; define whole pipelines there with the
[pipeline flags](cli.md#pipelines) or
[pipeline variables](cli.md#pipeline-variables).

## Security Operations

**Certificate rotation**

```bash
openssl x509 -in /etc/logwisp/tls/relay.crt -noout -enddate
```

Certificates and SCRAM credentials files load at plugin construction, so
rotation is: write the new files, then `kill -HUP`. Startup and every reload
warn at WARN once a certificate is within 30 days of expiry; automate the check
anyway. For passwords see
[rotation and revocation](security.md#rollout-rotation-and-revocation).

**Access review**

With `tls` alone, any certificate signed by the configured `client_ca_file` is
accepted, so "access review" means reviewing what your CA has issued. Add an
`auth` block with an explicit `allow` list, or `scram` with its credentials
file, and the review becomes those files: the identities or users listed there
are the ones that can connect, and removing one plus a `SIGHUP` is the
revocation path. Authorized identities are recorded in session metadata as
`auth_identity`; rejections are counted in `auth_rejected` and logged at WARN.
See [Security](security.md#the-auth-block).

**Secret leakage**

Filters are the only redaction mechanism, and they drop whole entries rather
than masking parts of them. See [Filters](filters.md#common-recipes).

## Maintenance

**Upgrades**

1. Read the changelog for configuration-schema changes.
2. Start the new binary against the current configuration in a scratch
   environment.
3. Stop the old process, install the new binary, start it.
4. Confirm each pipeline started and that counters are advancing.

**Backup**

Configuration files and TLS material are the only durable state worth backing
up. LogWisp keeps no persistent runtime state: file read positions live in
memory, connections are re-established on restart, and in-flight entries are
lost.

**Redundancy**

Because there is no persistence, availability comes from topology, not from
LogWisp itself. Give each edge two chain sinks pointing at two relays if you
need to survive a relay outage, and accept that this duplicates entries
downstream.