lixen/logwisp
SHA256
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.22.0 several enhancements and fixes for security, network, doc, test, packaging
Test / test (push) Canceled after 0s
Details
Test / e2e (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
lixen committed 2026-10-03 14:52:16 -04:00
1 parent 9ab8c356d2
commit 71d1b82f83
102 files changed
+7102 -2178
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

doc/cli.md
+177 -86
View File
@@ -1,19 +1,21 @@
# Command Line Interface
```
logwisp [options]
logwisp auth <command> [flags]
logwisp help | -h | --help
logwisp --version
lw [options]
lw --check [options]
lw auth <command> [flags]
lw help | -h | --help
lw --version
```
`logwisp auth` manages SCRAM credentials and logs in to `scram` listeners; see
[below](#logwisp-auth). There is no certificate-generation command: use
`lw auth` manages SCRAM credentials and logs in to `scram` listeners; see
[below](#lw-auth). There is no certificate-generation command: use
`openssl` or your PKI tooling — see [Security](security.md#enabling-mtls).
## Options
Any scalar configuration key is settable as a flag using its TOML path:
Any scalar configuration key is settable as a flag using its TOML path;
pipelines have [their own flags](#pipelines):
```
--<path>=<value> e.g. --logging.level=debug
@@ -23,49 +25,92 @@ Any scalar configuration key is settable as a flag using its TOML path:
### Common
| Flag | Description | Default |
|------|-------------|---------|
| `-c <path>` | Configuration file | `./logwisp.toml` |
| `--config <path>` / `--config=<path>` | Configuration file | `./logwisp.toml` |
| `--quiet` | Suppress all application output | `false` |
| `--status_reporter=<bool>` | Periodic status logging | `true` |
| `--auto_reload=<bool>` | Reload config when the file changes | `false` |
| `--version` | Print version and exit | — |
| `-h`, `--help`, `help` | Print usage and exit | — |
- `-c <path>`, `--config <path>`, `--config=<path>`: the configuration file,
default `./logwisp.toml`
- `--check`: build every pipeline and plugin, bind nothing, report and exit;
see [Usage Patterns](#usage-patterns)
- `--quiet`: suppress all application output, default `false`
- `--status_reporter=<bool>`: periodic status logging, default `true`
- `--auto_reload=<bool>`: reload when the configuration file changes, default
`false`
- `--version`: print the version and exit
- `-h`, `--help`, `help`: print usage and exit
The last file-selection flag wins. `-c=<path>` also works. A missing or empty
path returns an error, and `--` ends option parsing.
### Logging
| Flag | Values |
|------|--------|
| `--logging.output` | `file`, `stdout`, `stderr`, `split`, `all`, `none` |
| `--logging.level` | `debug`, `info`, `warn`, `error` |
| `--logging.format` | `raw`, `txt`, `json` |
| `--logging.sanitization` | `raw`, `json`, `txt`, `shell` |
| `--logging.file.directory` | path |
| `--logging.file.name` | string |
| `--logging.file.max_size_mb` | integer |
| `--logging.file.max_total_size_mb` | integer |
| `--logging.file.retention_hours` | float |
- `--logging.output`: `file`, `stdout`, `stderr`, `split`, `all` or `none`
- `--logging.level`: `debug`, `info`, `warn` or `error`
- `--logging.format`: `raw`, `txt` or `json`
- `--logging.sanitization`: `raw`, `json`, `txt` or `shell`
- `--logging.file.directory`: a path
- `--logging.file.name`: a string
- `--logging.file.max_size_mb`, `--logging.file.max_total_size_mb`: integers
- `--logging.file.retention_hours`: a float
`--logging.console.target` is accepted but has no effect; the console
destination is derived from `--logging.output`.
### Pipelines
Pipelines, sources, sinks, and filters **cannot** be configured from the command
line. Array-indexed paths such as `--pipelines.0.name=app` or
`--pipelines.0.plugin_sinks.0.type=null` are reported as unrecognized and
ignored:
Pipeline flags define whole pipelines. They replace the configuration file's
pipelines and the built-in default; every other key keeps its precedence.
- `--pipeline NAME` starts a pipeline; specs before the first one go to a
pipeline named `cli`
- `--source SPEC`, `--sink SPEC` and `--filter SPEC` add a stage, repeatable
- `--format SPEC`, `--rate-limit SPEC` and `--heartbeat SPEC`, once per pipeline
- both `--flag SPEC` and `--flag=SPEC` work; `--` ends options
A SPEC is a comma-separated list:
- source, sink, filter and format specs start with a TYPE
- the plugin type: [Sources](sources.md), [Sinks](sinks.md)
- `include` or `exclude`: [Filters](filters.md)
- `json`, `txt` or `raw`: [Formatters](formatters.md)
- rate-limit and heartbeat specs have no TYPE
- the rest are `key=value` pairs with the TOML keys of the plugin's `config`
table or of the [flow stage](configuration.md#flow-stages)
- a dotted key reaches a nested table: `tls.cert_file=...`, `auth.type=scram`
- a repeated key makes a list: `patterns=ERROR,patterns=WARN`
- each value is one list entry, commas included: `auth.allow_patterns=^a{1\,3}$`
is one regex, `auth.allow=a,auth.allow=b` two entries
- values convert to the option's type: `port=8080`, `raw=true`
- `\` escapes `,`, `=` and `\` in a value; any other backslash stays, so regex
escapes such as `\d` pass unchanged
- `id=NAME` names a source or sink; the default is its TYPE, then `TYPE_2`,
`TYPE_3`, ...
- naming a stage turns it on: `--rate-limit` defaults `policy` to `drop`,
`--heartbeat` defaults `enabled` to `true`
Errors name the flag and the offending part, and nothing starts:
```
Warning: unrecognized flags ignored: [pipelines.0.name]
--sink http,port: missing "="
--rate-limit rate=100,polcy=drop: unknown key "polcy"
```
Use a configuration file. Older documentation described CLI pipeline overrides
that the current loader does not implement.
A misspelled plugin key fails when the plugin is built, before any listener
opens: `failed to create sink http: ... unknown key "tls.enabeld"`.
```bash
# tail a directory, serve it over SSE
lw --source 'file,directory=/var/log/app,pattern=*.log' \
--sink http,host=0.0.0.0,port=8080
# errors and warnings as JSON, over TLS with SCRAM logins
lw --source file,directory=/var/log/app \
--filter include,patterns=ERROR,patterns=WARN --format json \
--sink "http,port=8443,auth.type=scram,auth.credentials_file=/etc/logwisp/users.toml,\
tls.enabled=true,tls.cert_file=/etc/logwisp/server.crt,tls.key_file=/etc/logwisp/server.key"
# two pipelines
lw --pipeline app --source file,directory=/var/log/app --sink console \
--pipeline relay --source tcp_chain,port=9000 \
--sink file,directory=/var/log/relay,name=relay
```
## Environment Variables
@@ -83,28 +128,54 @@ forms when upgrading; see [Configuration](configuration.md#environment-variables
The path resolver reads these variables directly:
| Variable | Effect |
|----------|--------|
| `LOGWISP_CONFIG_FILE` | Configuration file path; joined onto `LOGWISP_CONFIG_DIR` when both are set |
| `LOGWISP_CONFIG_DIR` | Configuration directory; alone, implies `<dir>/logwisp.toml` |
- `LOGWISP_CONFIG_FILE`: the configuration file path, joined onto
`LOGWISP_CONFIG_DIR` when both are set
- `LOGWISP_CONFIG_DIR`: the configuration directory; alone, it implies
`<dir>/logwisp.toml`
As with flags, array elements cannot be set this way.
### Pipeline Variables
One pipeline can come from the environment, with the [SPEC](#pipelines) syntax.
Any pipeline flag on the command line makes LogWisp ignore all of them.
- `LOGWISP_PIPELINE`: the pipeline's name, default `cli`
- `LOGWISP_SOURCE`, `LOGWISP_SINK`, `LOGWISP_FILTER`: one stage each
- `LOGWISP_SOURCE_1` .. `LOGWISP_SOURCE_N` add more, after the unnumbered
one in numeric order; likewise `LOGWISP_SINK_N` and `LOGWISP_FILTER_N`
- `LOGWISP_FORMAT`, `LOGWISP_RATE_LIMIT`, `LOGWISP_HEARTBEAT`
- an empty variable counts as unset
A container needs no configuration file:
```bash
docker run --rm -p 8080:8080 -v /var/log/app:/logs:ro \
-e LOGWISP_PIPELINE=app \
-e LOGWISP_SOURCE='file,directory=/logs,pattern=*.log' \
-e LOGWISP_FILTER='exclude,patterns=DEBUG' \
-e LOGWISP_SINK='http,host=0.0.0.0,port=8080' \
-e LOGWISP_SINK_1=console \
logwisp
```
## Precedence
1. Command-line flags
2. Environment variables
3. Configuration file
4. Built-in defaults
- Pipelines
- pipeline flags, if any are given
- else pipeline variables, if any are set
- else the configuration file's `[[pipelines]]`
- else the built-in default
- Every other key
1. command-line flags
2. environment variables
3. configuration file
4. built-in defaults
A reload rereads the file and keeps the command-line or environment pipelines.
## Signals
| Signal | Action |
|--------|--------|
| `SIGINT` | Graceful shutdown |
| `SIGTERM` | Graceful shutdown |
| `SIGHUP` | Reload configuration |
| `SIGUSR1` | Reload configuration |
- `SIGINT`, `SIGTERM`: graceful shutdown
- `SIGHUP`, `SIGUSR1`: reload the configuration
`SIGHUP` is ignored during startup, before the signal handler is installed, so
LogWisp survives a terminal hang-up like `nohup`. Once running, it triggers a
@@ -115,11 +186,10 @@ service untouched; see [Configuration](configuration.md#hot-reload).
## Exit Codes
| Code | Meaning |
|------|---------|
| `0` | Clean shutdown, or `--version` / `--help` |
| `1` | General error: config load or validation failure, logger init failure, service bootstrap failure |
| `2` | Explicitly requested configuration file not found |
- `0`: clean shutdown, `--version` / `--help`, or a valid `--check`
- `1`: general error: a configuration load or validation failure (`--check`
included), a logger init failure, a service bootstrap failure
- `2`: an explicitly requested configuration file is not found
Exit code 2 applies only when the file was named explicitly (`-c`,
`--config=`, or the `LOGWISP_CONFIG_*` variables). A missing discovered default
@@ -133,8 +203,9 @@ a rate limit of 5 entries/second with a burst of 10 and `policy = "drop"`, and a
`console` sink on stdout. It is a self-demonstrating idle mode, not a useful
production configuration.
Note that as soon as your file defines `[[pipelines]]`, that entire default
pipeline — rate limit included — is replaced rather than merged.
As soon as the file defines `[[pipelines]]`, or pipeline flags or variables are
given, that entire default pipeline — rate limit included — is replaced rather
than merged.
## Usage Patterns
@@ -142,26 +213,29 @@ pipeline — rate limit included — is replaced rather than merged.
```bash
# verbose, everything to stderr
logwisp -c dev.toml --logging.output=stderr --logging.level=debug
lw -c dev.toml --logging.output=stderr --logging.level=debug
# no config at all: synthetic generator to stdout
logwisp
lw
```
**Configuration check**
```bash
# starts the service; a config error exits non-zero before any pipeline runs
logwisp -c /etc/logwisp/logwisp.toml --logging.level=debug
```
`lw --check` validates without running: it builds every pipeline and plugin as
a start would (options, TLS files, credentials files, startup warnings), but
binds no port and opens, reads or creates no log file; it prints
`configuration ok` and exits 0, or the error and exits 1 (2 when a named file
is missing, see [Exit Codes](#exit-codes)):
There is no dry-run or validate-only mode. The closest approximation is starting
with debug logging and stopping once the pipelines report as started.
```bash
lw --check -c /etc/logwisp/logwisp.toml
lw --check --source file,directory=/var/log/app --sink http,port=8080
```
**Production**
```bash
logwisp -c /etc/logwisp/logwisp.toml --logging.output=file
lw -c /etc/logwisp/logwisp.toml --logging.output=file
```
Run under a supervisor (systemd, rc.d) rather than backgrounding it — there is
@@ -171,29 +245,38 @@ no `--background` flag; earlier releases had one and it was removed. See
**Reload**
```bash
kill -HUP $(pidof logwisp)
kill -USR1 $(pidof logwisp)
kill -HUP $(pidof lw)
kill -USR1 $(pidof lw)
```
## `logwisp auth`
## `lw auth`
Manages the credentials files of `auth.type = "scram"` listeners and logs in to
them as a viewer. See
[Password Authentication](security.md#password-authentication-scram).
| Command | Does |
|---------|------|
| `add-user -credentials FILE -user NAME [-password-file FILE] [-generate]` | Adds a user, or replaces its password |
| `remove-user -credentials FILE -user NAME` | Removes a user; refuses the last one |
| `token -url https://HOST:PORT[/PATH] -user NAME -password-file FILE [-unbound] [TLS flags]` | Logs in to an `http` sink or `http_chain` source and prints a bearer token |
| `stream -addr HOST:PORT -user NAME -password-file FILE [TLS flags]` | Logs in to a `tcp` sink and copies its stream to stdout until interrupted |
```
lw auth add-user -credentials FILE -user NAME [-password-file FILE] [-generate]
lw auth remove-user -credentials FILE -user NAME
lw auth token -url https://HOST:PORT[/PATH] -user NAME -password-file FILE
[-unbound] [TLS flags]
lw auth stream -addr HOST:PORT -user NAME -password-file FILE [TLS flags]
```
`logwisp auth <command> -h` lists a command's flags. The exit status is `0` on
- `add-user`: adds a user, or replaces its password
- `remove-user`: removes a user; refuses the last one
- `token`: logs in to an `http` sink or `http_chain` source and prints a
bearer token
- `stream`: logs in to a `tcp` sink and copies its stream to stdout until
interrupted
`lw auth <command> -h` lists a command's flags. The exit status is `0` on
success, `1` on failure and `2` on a usage error.
**`add-user`** creates the credentials file, with a fresh `decoy_key`, when it
does not exist or is empty (create it empty first to choose its owner). The password comes from `-password-file` when that file exists
(at least 8 bytes; one trailing line break is trimmed). Otherwise a random
does not exist or is empty (create it empty first to choose its owner). The
password comes from `-password-file` when that file exists (at least 8 bytes;
one trailing line break is trimmed). Otherwise a random
26-character password (130 bits) is generated and written to `-password-file`,
or printed once to stdout when there is none. Replacing an existing user's
password needs an existing `-password-file` or `-generate`, so a mistyped path
@@ -215,29 +298,37 @@ There is no flag to skip verification: an unverified server could relay the
login. Redirects are not followed. `-unbound` logs in to an `http` sink behind
a TLS-terminating proxy (`auth.trusted_proxies`), still pinning the proxy's
certificate across the two requests; only then may `-url` carry the path the
proxy mounts LogWisp at. `stream` exits `0`
on `SIGINT` or `SIGTERM`, and `1` when the server ends the stream (a reload or
shutdown).
proxy mounts LogWisp at. `stream` exits `0` on `SIGINT` or `SIGTERM`, and `1`
when the server ends the stream (a reload or shutdown).
Addresses, as for the plugins ([Networking](networking.md#address-family)):
- An IPv6 address goes in brackets: `-addr [::1]:PORT`,
`-url https://[::1]:PORT`; in a URL a link-local zone is escaped,
`https://[fe80::1%25eth0]:PORT`.
- The dial keeps to the address's family; a hostname resolves.
- `-server-name` defaults to the host; an address, without brackets or zone,
must then be among the certificate's IP SANs.
```bash
# listener host: a file the service user can read, users, then apply
install -m 0640 -o root -g logwisp /dev/null /etc/logwisp/users.toml
logwisp auth add-user -credentials /etc/logwisp/users.toml -user edge-01 \
lw auth add-user -credentials /etc/logwisp/users.toml -user edge-01 \
-password-file /etc/logwisp/edge-01.pass
logwisp auth add-user -credentials /etc/logwisp/users.toml -user viewer \
lw auth add-user -credentials /etc/logwisp/users.toml -user viewer \
-password-file viewer.pass
kill -HUP $(pidof logwisp)
kill -HUP $(pidof lw)
# rotate: new password into the file; SIGHUP, deploy the file, SIGHUP the edge
logwisp auth add-user -credentials /etc/logwisp/users.toml -user edge-01 \
lw auth add-user -credentials /etc/logwisp/users.toml -user edge-01 \
-password-file /etc/logwisp/edge-01.pass -generate
# http sink status, with the token kept out of curl's argv
curl --cacert ca.crt -H @<(printf 'Authorization: Bearer %s\n' \
"$(logwisp auth token -url https://HOST:PORT -user viewer \
"$(lw auth token -url https://HOST:PORT -user viewer \
-password-file viewer.pass -ca-file ca.crt)") https://HOST:PORT/status
# follow a tcp sink
logwisp auth stream -addr HOST:PORT -user viewer -password-file viewer.pass \
lw auth stream -addr HOST:PORT -user viewer -password-file viewer.pass \
-ca-file ca.crt
```