lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.16.1 doc update

Browse Source
This commit is contained in:
lixen
2026-08-29 16:37:20 -04:00
parent 296b351883
commit 5fbd5c71cf
19 changed files with 3066 additions and 1323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/configuration.md
+220 -138
View File
@@ -1,203 +1,285 @@
# Configuration Reference
LogWisp configuration uses TOML format with flexible override mechanisms.
LogWisp is configured with TOML. A complete annotated file listing every option
and its default ships as [`config/logwisp.toml`](../config/logwisp.toml).
## Configuration Precedence
Configuration sources are evaluated in order:
1. **Command-line flags** (highest priority)
2. **Environment variables**
3. **Configuration file**
4. **Built-in defaults** (lowest priority)
Sources are merged in this order, highest priority first:
1. Command-line flags
2. Environment variables
3. Configuration file
4. Built-in defaults
The `pipelines` array is replaced wholesale, not merged: as soon as your file
defines `[[pipelines]]`, the built-in default pipeline (and its default rate
limit and formatter) disappears entirely.
## File Location
LogWisp searches for configuration in order:
1. Path specified via `--config` flag
2. Path from `LOGWISP_CONFIG_FILE` environment variable
3. `~/.config/logwisp/logwisp.toml`
4. `./logwisp.toml` in current directory
The path is resolved before any other configuration is read:
1. `-c <path>` on the command line
2. `--config=<path>` on the command line
3. `$LOGWISP_CONFIG_FILE`, joined onto `$LOGWISP_CONFIG_DIR` when both are set
4. `$LOGWISP_CONFIG_DIR/logwisp.toml`
5. `~/.config/logwisp/logwisp.toml`, if it exists
6. `./logwisp.toml`
Missing file behaviour differs by how it was chosen. An explicitly requested
file that does not exist is a fatal error (exit code 2); a missing discovered
default is not an error, and LogWisp starts on built-in defaults.
> `--config <path>` with a space is **not** recognized as a config path. It is
> parsed as an unknown flag, warned about, and ignored — LogWisp then silently
> falls back to `./logwisp.toml`. Use `-c <path>` or `--config=<path>`.
## Global Settings
Top-level configuration options:
| Setting | Type | Default | Description |
|---------|------|---------|-------------|
| `quiet` | bool | false | Suppress console output |
| `status_reporter` | bool | true | Periodic status logging |
| `auto_reload` | bool | false | Enable file watch for auto-reload |
| `quiet` | bool | `false` | Disable all application logging and console diagnostics |
| `status_reporter` | bool | `true` | Emit a periodic status report every 30 s at DEBUG level |
| `auto_reload` | bool | `false` | Watch the config file and reload pipelines on change |
## Logging Configuration
`--version` prints version information and exits; it is not a persistent
setting.
LogWisp's internal operational logging:
Note that `status_reporter` writes at DEBUG level, so it produces nothing unless
`logging.level = "debug"`.
## Application Logging
This configures LogWisp's own operational log, not the log data it transports.
```toml
[logging]
output = "stdout" # file|stdout|stderr|split|all|none
level = "info" # debug|info|warn|error
output = "stdout" # file | stdout | stderr | split | all | none
level = "info" # debug | info | warn | error
format = "txt" # raw | txt | json
# sanitization = "" # raw | json | txt | shell
[logging.file]
directory = "./log"
name = "logwisp"
max_size_mb = 100
directory = "./log"
name = "logwisp"
max_size_mb = 100
max_total_size_mb = 1000
retention_hours = 168.0
[logging.console]
target = "stdout" # stdout|stderr|split
retention_hours = 168.0
```
### Output Modes
### Output modes
- **file**: Write to log files only
- **stdout**: Write to standard output
- **stderr**: Write to standard error
- **split**: INFO/DEBUG to stdout, WARN/ERROR to stderr
- **all**: Write to both file and console
- **none**: Disable all logging
| Mode | Behaviour |
|------|-----------|
| `file` | Files only |
| `stdout` | Standard output only |
| `stderr` | Standard error only |
| `split` | DEBUG/INFO to stdout, WARN/ERROR to stderr |
| `all` | Files plus split console |
| `none` | No application logging |
`[logging.file]` applies only to the `file` and `all` modes.
> `[logging.console].target` is accepted and validated (`stdout`, `stderr`,
> `split`) but **not applied**. The console destination is derived from
> `logging.output`. The key is retained for compatibility; setting it has no
> effect.
`quiet = true` overrides every logging setting and disables both file and
console output.
## Pipeline Configuration
Each `[[pipelines]]` section defines an independent processing pipeline:
```toml
[[pipelines]]
name = "pipeline-name"
name = "app" # required, unique across pipelines
# Rate limiting (optional)
# --- flow: everything between sources and sinks ---
[pipelines.flow.rate_limit]
rate = 1000.0
burst = 2000.0
policy = "drop" # pass|drop
max_entry_size_bytes = 0 # 0=unlimited
rate = 1000.0
burst = 2000.0
policy = "drop"
max_entry_size_bytes = 65536
# Format configuration (optional)
[pipelines.flow.format]
type = "json" # raw|json|txt
sanitizer_policy = "json"
[[pipelines.plugin_sources]]
id = "my_source"
type = "file"
[pipelines.plugin_sources.config]
# ... source-specific config
# Filters (optional)
[[pipelines.flow.filters]]
type = "include"
logic = "or"
type = "include"
logic = "or"
patterns = ["ERROR", "WARN"]
# Sinks (required, 1+)
[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"
[pipelines.flow.heartbeat]
enabled = true
interval_ms = 30000
# --- sources: one or more ---
[[pipelines.plugin_sources]]
id = "app_logs" # unique within the pipeline
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
# --- sinks: one or more ---
[[pipelines.plugin_sinks]]
id = "my_sink"
id = "sse"
type = "http"
[pipelines.plugin_sinks.config]
# ... sink-specific config
port = 8080
```
Every source and sink is a plugin instance with three keys:
| Key | Meaning |
|-----|---------|
| `id` | Instance identifier, unique within the pipeline; appears in logs and stats |
| `type` | Registered plugin type |
| `config` | Plugin-specific table; see [Sources](sources.md) and [Sinks](sinks.md) |
`config_file` is reserved on both structures for a future include mechanism and
is not implemented.
### Flow stages
| Block | Optional | Reference |
|-------|----------|-----------|
| `flow.rate_limit` | yes | below |
| `flow.filters` | yes | [Filters](filters.md) |
| `flow.format` | yes (defaults to `raw`) | [Formatters](formatters.md) |
| `flow.heartbeat` | yes | below |
#### Rate limiting
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `rate` | float | `0` | Entries per second; `<= 0` disables the limiter entirely |
| `burst` | float | `rate` | Token bucket capacity |
| `policy` | string | `pass` | `pass` allows everything through, `drop` discards over-limit entries |
| `max_entry_size_bytes` | int | `0` | Per-entry byte cap; `0` = unlimited |
Two behaviours are easy to trip over:
- The limiter is constructed only when `rate > 0`. With `rate = 0`,
`max_entry_size_bytes` is never enforced.
- `policy = "pass"` short-circuits the whole check, including the size cap.
To enforce a size cap you need `rate > 0` **and** `policy = "drop"`.
#### Heartbeat
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `enabled` | bool | `false` | Enable heartbeat generation |
| `interval_ms` | int | `1000` | Interval; minimum `100` |
| `include_timestamp` | bool | `false` | `false` formats with level only, no timestamp |
| `include_stats` | bool | `false` | Attach `beat_count` and measured `interval_ms` as fields |
| `format` | string | `txt` | `txt`, `json`, or `raw` |
Heartbeats are ordinary entries with source `heartbeat` and level `INFO`. They
are generated after the flow's filter and rate-limit stages, so filters do not
suppress them, and they reach every sink in the pipeline.
> `format = "comment"` (SSE comment framing) appears in older documentation and
> in a code path in the generator, but the validator rejects it and the pipeline
> fails to start. Use `txt`, `json`, or `raw`.
## Environment Variables
All configuration options support environment variable overrides:
Environment overrides are derived from the TOML path: `.` becomes `_` and the
result is uppercased.
### Naming Convention
- Prefix: `LOGWISP_`
- Path separator: `_` (underscore)
- Array indices: Numeric suffix (0-based)
- Case: UPPERCASE
### Mapping Examples
| TOML Path | Environment Variable |
| TOML path | Environment variable |
|-----------|---------------------|
| `quiet` | `LOGWISP_QUIET` |
| `logging.level` | `LOGWISP_LOGGING_LEVEL` |
| `pipelines[0].name` | `LOGWISP_PIPELINES_0_NAME` |
| `pipelines[0].plugin_sources[0].type` | `LOGWISP_PIPELINES_0_PLUGIN_SOURCES_0_TYPE` |
| `quiet` | `QUIET` |
| `status_reporter` | `STATUS_REPORTER` |
| `logging.level` | `LOGGING_LEVEL` |
| `logging.file.directory` | `LOGGING_FILE_DIRECTORY` |
> **The `LOGWISP_` prefix is not currently applied.** The configuration loader
> requests it, but supplying a custom path-to-variable transform replaces the
> prefixing step rather than composing with it, so LogWisp reads bare
> `QUIET`, `LOGGING_LEVEL`, and so on from the environment. Treat this as
> current behaviour to be aware of — bare names like `QUIET` can collide with
> unrelated variables — rather than as intended design.
>
> The two exceptions are `LOGWISP_CONFIG_FILE` and `LOGWISP_CONFIG_DIR`, which
> are read directly by the path resolver and **do** carry the prefix.
Only scalar paths that exist in the configuration schema can be set this way.
Array elements cannot: `PIPELINES_0_NAME` has no effect.
## Command-Line Overrides
All configuration options can be overridden via CLI flags:
Any scalar configuration path is settable as a flag using its TOML path:
```bash
logwisp --quiet \
--logging.level=debug \
--pipelines.0.name=myapp \
--pipelines.0.plugin_sources.0.type=console
logwisp --logging.level=debug --status_reporter=false
logwisp --logging.level debug # space form also works
logwisp --quiet # bare flag means true
```
## Configuration Validation
Unrecognized flags are reported on stderr before the logger exists and are then
ignored:
LogWisp validates configuration at startup:
- Rpipelines non-empty, name non-empty, ≥1 source, ≥1 sink, logging enum values.equired fields presence
```
Warning: unrecognized flags ignored: [pipelines.0.name]
```
Partial check in plugin constructor:
- Type correctness
- Port conflicts
- Path accessibility
- Pattern compilation
- Network address formats
> Array-indexed paths are **not** settable from the command line.
> `--pipelines.0.name=x`, `--pipelines.0.plugin_sinks.0.type=null`, and similar
> flags are reported as unrecognized and ignored. Pipelines, sources, sinks, and
> filters can only be defined in the configuration file. Older documentation
> claimed otherwise.
## Validation
Startup validation is intentionally split.
`internal/config` validates only global structure:
- at least one pipeline
- unique, non-empty pipeline names
- at least one source and one sink per pipeline
- `logging.output`, `logging.level`, `logging.format`, `logging.sanitization`,
and `logging.console.target` enum membership
Everything else is validated by the plugin constructor that owns it — port
range, required paths, path prefixes, enum values, regex compilation, TLS file
loading. A failure there aborts pipeline construction with a message naming the
pipeline, plugin id, and offending key.
There is **no** cross-pipeline port-conflict detection. Two sinks bound to the
same port fail at listener bind time, when the pipeline starts.
## Hot Reload
Enable configuration hot reload:
```toml
auto_reload = true
```
Or via command line:
```bash
logwisp --auto-reload
```
or send `SIGHUP` / `SIGUSR1`.
Reload triggers:
- File modification detection
- SIGHUP or SIGUSR1 signals
Reload rebuilds the whole service: a new service is constructed from the new
configuration first, and only if that succeeds is the old one shut down. A
configuration error therefore leaves the running service untouched.
Reloadable items:
- Pipeline configurations
- Sources and sinks
- Filters and formatters
- Rate limits
| Reloaded | Not reloaded |
|----------|--------------|
| Pipelines, sources, sinks | `logging.*` (applied once at startup) |
| Filters, formatters, rate limits, heartbeats | `quiet` |
| `status_reporter` | `auto_reload` (the watcher is not restarted) |
Non-reloadable (requires restart):
- Logging configuration
- Global settings
Because the rebuild is total, listeners close and reopen and every connected
client is disconnected. Chain sinks reconnect on their own backoff schedule.
## Default Configuration
## Type Reference
Minimal working configuration:
```toml
[[pipelines]]
name = "default"
[[pipelines.plugin_sources]]
id = "default_source"
type = "file"
[pipelines.plugin_sources.config]
directory = "./"
pattern = "*.log"
[[pipelines.plugin_sinks]]
id = "default_sink"
type = "console"
[pipelines.plugin_sinks.config]
target = "stdout"
```
## Configuration Schema
### Type Reference
| TOML Type | Go Type | Environment Format |
|-----------|---------|-------------------|
| String | string | Plain text |
| Integer | int64 | Numeric string |
| Float | float64 | Decimal string |
| Boolean | bool | true/false |
| Array | []T | JSON array string |
| Table | struct | Nested with `_` |
| TOML type | Go type | Command-line / environment form |
|-----------|---------|-------------------------------|
| String | `string` | Plain text |
| Integer | `int64` | Decimal string |
| Float | `float64` | Decimal string |
| Boolean | `bool` | `true` / `false`, or a bare flag for `true` |
| Array | `[]T` | Not settable outside the file |
| Table | struct | Nested path with `.` (flags) or `_` (environment) |