lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.2.0 restructured to pipeline architecture, dirty

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-11 04:52:41 -04:00
parent 5936f82970
commit b503816de3
51 changed files with 4132 additions and 5936 deletions
Download Patch File Download Diff File Expand all files Collapse all files

299
doc/environment.md
View File

@ -1,275 +1,148 @@
# Environment Variables
LogWisp supports comprehensive configuration through environment variables, allowing deployment without configuration files or dynamic overrides in containerized environments.
Configure LogWisp through environment variables for containerized deployments.
## Naming Convention
Environment variables follow a structured pattern:
- **Prefix**: `LOGWISP_`
- **Path separator**: `_` (underscore)
- **Array indices**: Numeric suffix (0-based)
- **Case**: UPPERCASE
### Examples:
- Config file setting: `logging.level = "debug"`
- Environment variable: `LOGWISP_LOGGING_LEVEL=debug`
- Array element: `streams[0].name = "app"`
- Environment variable: `LOGWISP_STREAMS_0_NAME=app`
Examples:
- `logging.level` → `LOGWISP_LOGGING_LEVEL`
- `pipelines[0].name` → `LOGWISP_PIPELINES_0_NAME`
## General Variables
### `LOGWISP_CONFIG_FILE`
Path to the configuration file.
- **Default**: `~/.config/logwisp.toml`
- **Example**: `LOGWISP_CONFIG_FILE=/etc/logwisp/config.toml`
Configuration file path.
```bash
export LOGWISP_CONFIG_FILE=/etc/logwisp/config.toml
```
### `LOGWISP_CONFIG_DIR`
Directory containing configuration files.
- **Usage**: Combined with `LOGWISP_CONFIG_FILE` for relative paths
- **Example**:
```bash
export LOGWISP_CONFIG_DIR=/etc/logwisp
export LOGWISP_CONFIG_FILE=production.toml
# Loads: /etc/logwisp/production.toml
```
Configuration directory.
```bash
export LOGWISP_CONFIG_DIR=/etc/logwisp
export LOGWISP_CONFIG_FILE=production.toml
```
### `LOGWISP_DISABLE_STATUS_REPORTER`
Disable the periodic status reporter.
- **Values**: `1` (disable), `0` or unset (enable)
- **Default**: `0` (enabled)
- **Example**: `LOGWISP_DISABLE_STATUS_REPORTER=1`
### `LOGWISP_BACKGROUND`
Internal marker for background process detection.
- **Note**: Set automatically by `--background` flag
- **Values**: `1` (background), unset (foreground)
Disable periodic status reporting.
```bash
export LOGWISP_DISABLE_STATUS_REPORTER=1
```
## Logging Variables
### `LOGWISP_LOGGING_OUTPUT`
LogWisp's operational log output mode.
- **Values**: `file`, `stdout`, `stderr`, `both`, `none`
- **Example**: `LOGWISP_LOGGING_OUTPUT=both`
### `LOGWISP_LOGGING_LEVEL`
Minimum log level for operational logs.
- **Values**: `debug`, `info`, `warn`, `error`
- **Example**: `LOGWISP_LOGGING_LEVEL=debug`
### File Logging
```bash
# Output mode
LOGWISP_LOGGING_OUTPUT=both
# Log level
LOGWISP_LOGGING_LEVEL=debug
# File logging
LOGWISP_LOGGING_FILE_DIRECTORY=/var/log/logwisp
LOGWISP_LOGGING_FILE_NAME=logwisp
LOGWISP_LOGGING_FILE_MAX_SIZE_MB=100
LOGWISP_LOGGING_FILE_MAX_TOTAL_SIZE_MB=1000
LOGWISP_LOGGING_FILE_RETENTION_HOURS=168 # 7 days
LOGWISP_LOGGING_FILE_RETENTION_HOURS=168
# Console logging
LOGWISP_LOGGING_CONSOLE_TARGET=stderr
LOGWISP_LOGGING_CONSOLE_FORMAT=json
```
### Console Logging
## Pipeline Configuration
### Basic Pipeline
```bash
LOGWISP_LOGGING_CONSOLE_TARGET=stderr # stdout, stderr, split
LOGWISP_LOGGING_CONSOLE_FORMAT=txt # txt, json
```
# Pipeline name
LOGWISP_PIPELINES_0_NAME=app
## Stream Configuration
# Source configuration
LOGWISP_PIPELINES_0_SOURCES_0_TYPE=directory
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATH=/var/log/app
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATTERN="*.log"
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_CHECK_INTERVAL_MS=100
Streams are configured using array indices (0-based).
### Basic Stream Settings
```bash
# First stream (index 0)
LOGWISP_STREAMS_0_NAME=app
LOGWISP_STREAMS_0_MONITOR_CHECK_INTERVAL_MS=100
# Second stream (index 1)
LOGWISP_STREAMS_1_NAME=system
LOGWISP_STREAMS_1_MONITOR_CHECK_INTERVAL_MS=1000
```
### Monitor Targets
```bash
# Single file target
LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/app.log
LOGWISP_STREAMS_0_MONITOR_TARGETS_0_IS_FILE=true
# Directory with pattern
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_PATH=/var/log/myapp
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_PATTERN="*.log"
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_IS_FILE=false
# Sink configuration
LOGWISP_PIPELINES_0_SINKS_0_TYPE=http
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_PORT=8080
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_BUFFER_SIZE=1000
```
### Filters
```bash
# Include filter
LOGWISP_STREAMS_0_FILTERS_0_TYPE=include
LOGWISP_STREAMS_0_FILTERS_0_LOGIC=or
LOGWISP_STREAMS_0_FILTERS_0_PATTERNS='["ERROR","WARN","CRITICAL"]'
LOGWISP_PIPELINES_0_FILTERS_0_TYPE=include
LOGWISP_PIPELINES_0_FILTERS_0_LOGIC=or
LOGWISP_PIPELINES_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
# Exclude filter
LOGWISP_STREAMS_0_FILTERS_1_TYPE=exclude
LOGWISP_STREAMS_0_FILTERS_1_PATTERNS='["DEBUG","TRACE"]'
LOGWISP_PIPELINES_0_FILTERS_1_TYPE=exclude
LOGWISP_PIPELINES_0_FILTERS_1_PATTERNS='["DEBUG"]'
```
### HTTP Server
### HTTP Sink Options
```bash
LOGWISP_STREAMS_0_HTTPSERVER_ENABLED=true
LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
LOGWISP_STREAMS_0_HTTPSERVER_BUFFER_SIZE=1000
LOGWISP_STREAMS_0_HTTPSERVER_STREAM_PATH=/stream
LOGWISP_STREAMS_0_HTTPSERVER_STATUS_PATH=/status
# Basic
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_STREAM_PATH=/stream
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_STATUS_PATH=/status
# Heartbeat
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_ENABLED=true
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INTERVAL_SECONDS=30
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_FORMAT=comment
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INCLUDE_TIMESTAMP=true
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INCLUDE_STATS=false
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_ENABLED=true
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_INTERVAL_SECONDS=30
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_FORMAT=comment
# Rate Limiting
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_ENABLED=true
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=10.0
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_BURST_SIZE=20
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_LIMIT_BY=ip
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_MAX_CONNECTIONS_PER_IP=5
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_ENABLED=true
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_REQUESTS_PER_SECOND=10.0
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_BURST_SIZE=20
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_LIMIT_BY=ip
```
### TCP Server
```bash
LOGWISP_STREAMS_0_TCPSERVER_ENABLED=true
LOGWISP_STREAMS_0_TCPSERVER_PORT=9090
LOGWISP_STREAMS_0_TCPSERVER_BUFFER_SIZE=5000
# Rate Limiting
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_ENABLED=true
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=5.0
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_BURST_SIZE=10
```
## Complete Example
Here's a complete example configuring two streams via environment variables:
## Example
```bash
#!/bin/bash
# Logging configuration
# Logging
export LOGWISP_LOGGING_OUTPUT=both
export LOGWISP_LOGGING_LEVEL=info
export LOGWISP_LOGGING_FILE_DIRECTORY=/var/log/logwisp
export LOGWISP_LOGGING_FILE_MAX_SIZE_MB=100
# Stream 0: Application logs
export LOGWISP_STREAMS_0_NAME=app
export LOGWISP_STREAMS_0_MONITOR_CHECK_INTERVAL_MS=50
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/myapp
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATTERN="*.log"
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_IS_FILE=false
# Pipeline 0: Application logs
export LOGWISP_PIPELINES_0_NAME=app
export LOGWISP_PIPELINES_0_SOURCES_0_TYPE=directory
export LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATH=/var/log/myapp
export LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATTERN="*.log"
# Stream 0: Filters
export LOGWISP_STREAMS_0_FILTERS_0_TYPE=include
export LOGWISP_STREAMS_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
# Filters
export LOGWISP_PIPELINES_0_FILTERS_0_TYPE=include
export LOGWISP_PIPELINES_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
# Stream 0: HTTP server
export LOGWISP_STREAMS_0_HTTPSERVER_ENABLED=true
export LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
export LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_ENABLED=true
export LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=25.0
# HTTP sink
export LOGWISP_PIPELINES_0_SINKS_0_TYPE=http
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_PORT=8080
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_ENABLED=true
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_REQUESTS_PER_SECOND=25.0
# Stream 1: System logs
export LOGWISP_STREAMS_1_NAME=system
export LOGWISP_STREAMS_1_MONITOR_CHECK_INTERVAL_MS=1000
export LOGWISP_STREAMS_1_MONITOR_TARGETS_0_PATH=/var/log/syslog
export LOGWISP_STREAMS_1_MONITOR_TARGETS_0_IS_FILE=true
# Pipeline 1: System logs
export LOGWISP_PIPELINES_1_NAME=system
export LOGWISP_PIPELINES_1_SOURCES_0_TYPE=file
export LOGWISP_PIPELINES_1_SOURCES_0_OPTIONS_PATH=/var/log/syslog
# Stream 1: TCP server
export LOGWISP_STREAMS_1_TCPSERVER_ENABLED=true
export LOGWISP_STREAMS_1_TCPSERVER_PORT=9090
# TCP sink
export LOGWISP_PIPELINES_1_SINKS_0_TYPE=tcp
export LOGWISP_PIPELINES_1_SINKS_0_OPTIONS_PORT=9090
# Start LogWisp
logwisp
```
## Docker/Kubernetes Usage
## Precedence
Environment variables are ideal for containerized deployments:
### Docker
```dockerfile
FROM logwisp:latest
ENV LOGWISP_LOGGING_OUTPUT=stdout
ENV LOGWISP_STREAMS_0_NAME=container
ENV LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/app
ENV LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
```
### Docker Compose
```yaml
version: '3'
services:
logwisp:
image: logwisp:latest
environment:
- LOGWISP_LOGGING_OUTPUT=stdout
- LOGWISP_STREAMS_0_NAME=webapp
- LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/logs
- LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
volumes:
- ./logs:/logs:ro
ports:
- "8080:8080"
```
### Kubernetes ConfigMap
```yaml
apiVersion: v1
kind: ConfigMap
metadata:
name: logwisp-config
data:
LOGWISP_LOGGING_LEVEL: "info"
LOGWISP_STREAMS_0_NAME: "k8s-app"
LOGWISP_STREAMS_0_HTTPSERVER_PORT: "8080"
```
## Precedence Rules
When the same setting is configured multiple ways, this precedence applies:
1. **Command-line flags** (highest priority)
2. **Environment variables**
3. **Configuration file**
4. **Default values** (lowest priority)
Example:
```bash
# Config file has: logging.level = "info"
export LOGWISP_LOGGING_LEVEL=warn
logwisp --log-level debug
# Result: log level will be "debug" (CLI flag wins)
```
## Debugging
To see which environment variables LogWisp recognizes:
```bash
# List all LOGWISP variables
env | grep ^LOGWISP_
# Test configuration parsing
LOGWISP_LOGGING_LEVEL=debug logwisp --version
```
## Security Considerations
- **Sensitive Values**: Avoid putting passwords or tokens in environment variables
- **Process Visibility**: Environment variables may be visible to other processes
- **Container Security**: Use secrets management for sensitive configuration
- **Logging**: Be careful not to log environment variable values
## See Also
- [Configuration Guide](configuration.md) - Complete configuration reference
- [CLI Options](cli.md) - Command-line interface
- [Docker Deployment](integrations.md#docker) - Container-specific guidance
1. Command-line flags (highest)
2. Environment variables
3. Configuration file
4. Defaults (lowest)