logwisp/doc/configuration.md

Configuration Guide

LogWisp uses TOML format with a flexible source → filter → sink pipeline architecture.

Configuration File Location

1. Command line: --config /path/to/config.toml
2. Environment: $LOGWISP_CONFIG_FILE
3. User config: ~/.config/logwisp.toml
4. Current directory: ./logwisp.toml

Configuration Structure

# Optional: LogWisp's own logging
[logging]
output = "stderr"
level = "info"

# Required: At least one pipeline
[[pipelines]]
name = "default"

# Sources (required)
[[pipelines.sources]]
type = "directory"
options = { ... }

# Filters (optional)
[[pipelines.filters]]
type = "include"
patterns = [...]

# Sinks (required)
[[pipelines.sinks]]
type = "http"
options = { ... }

Logging Configuration

Controls LogWisp's operational logging:

[logging]
output = "stderr"  # file, stdout, stderr, both, none
level = "info"     # debug, info, warn, error

[logging.file]
directory = "./logs"
name = "logwisp"
max_size_mb = 100
max_total_size_mb = 1000
retention_hours = 168

[logging.console]
target = "stderr"  # stdout, stderr, split
format = "txt"     # txt or json

Pipeline Configuration

Each [[pipelines]] section defines an independent processing pipeline.

Sources

Input data sources:

Directory Source

[[pipelines.sources]]
type = "directory"
options = {
    path = "/var/log/myapp",      # Directory to monitor
    pattern = "*.log",            # File pattern (glob)
    check_interval_ms = 100       # Check interval (10-60000)
}

File Source

[[pipelines.sources]]
type = "file"
options = {
    path = "/var/log/app.log"     # Specific file
}

Stdin Source

[[pipelines.sources]]
type = "stdin"
options = {}

Filters

Control which log entries pass through:

# Include filter - only matching logs pass
[[pipelines.filters]]
type = "include"
logic = "or"      # or: match any, and: match all
patterns = [
    "ERROR",
    "(?i)warn",   # Case-insensitive
    "\\bfatal\\b" # Word boundary
]

# Exclude filter - matching logs are dropped
[[pipelines.filters]]
type = "exclude"
patterns = ["DEBUG", "health-check"]

Sinks

Output destinations:

HTTP Sink (SSE)

[[pipelines.sinks]]
type = "http"
options = {
    port = 8080,
    buffer_size = 1000,
    stream_path = "/stream",
    status_path = "/status",
    
    # Heartbeat
    heartbeat = {
        enabled = true,
        interval_seconds = 30,
        format = "comment",  # comment or json
        include_timestamp = true
    },
    
    # Rate limiting
    rate_limit = {
        enabled = true,
        requests_per_second = 10.0,
        burst_size = 20,
        limit_by = "ip",  # ip or global
        max_connections_per_ip = 5
    }
}

TCP Sink

[[pipelines.sinks]]
type = "tcp"
options = {
    port = 9090,
    buffer_size = 5000,
    heartbeat = { enabled = true, interval_seconds = 60 },
    rate_limit = { enabled = true, requests_per_second = 5.0 }
}

File Sink

[[pipelines.sinks]]
type = "file"
options = {
    directory = "/var/log/logwisp",
    name = "app",
    max_size_mb = 100,
    retention_hours = 168.0,
    buffer_size = 2000
}

Console Sinks

[[pipelines.sinks]]
type = "stdout"  # or "stderr"
options = { buffer_size = 500 }

Complete Examples

Basic Application Monitoring

[[pipelines]]
name = "app"

[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/app", pattern = "*.log" }

[[pipelines.sinks]]
type = "http"
options = { port = 8080 }

Production with Filtering

[logging]
output = "file"
level = "info"

[[pipelines]]
name = "production"

[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/app", pattern = "*.log", check_interval_ms = 50 }

[[pipelines.filters]]
type = "include"
patterns = ["ERROR", "WARN", "CRITICAL"]

[[pipelines.filters]]
type = "exclude"
patterns = ["/health", "/metrics"]

[[pipelines.sinks]]
type = "http"
options = {
    port = 8080,
    rate_limit = { enabled = true, requests_per_second = 25.0 }
}

[[pipelines.sinks]]
type = "file"
options = { directory = "/var/log/archive", name = "errors" }

Multi-Source Aggregation

[[pipelines]]
name = "aggregated"

[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/nginx", pattern = "*.log" }

[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/app", pattern = "*.log" }

[[pipelines.sources]]
type = "stdin"
options = {}

[[pipelines.sinks]]
type = "tcp"
options = { port = 9090 }

Router Mode

# Run with: logwisp --router

[[pipelines]]
name = "api"
[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/api", pattern = "*.log" }
[[pipelines.sinks]]
type = "http"
options = { port = 8080 }  # Same port OK in router mode

[[pipelines]]
name = "web"
[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/nginx", pattern = "*.log" }
[[pipelines.sinks]]
type = "http"
options = { port = 8080 }  # Shared port

# Access:
# http://localhost:8080/api/stream
# http://localhost:8080/web/stream
# http://localhost:8080/status

Validation

LogWisp validates on startup:

• Required fields (name, sources, sinks)
• Port conflicts between pipelines
• Pattern syntax
• Path accessibility
• Rate limit values