lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.3.2 doc update

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-12 23:24:41 -04:00
parent e31591ac8d
commit 0accb5f2d3
8 changed files with 648 additions and 183 deletions
Download Patch File Download Diff File Expand all files Collapse all files

227
doc/configuration.md
View File

@ -2,20 +2,86 @@
LogWisp uses TOML format with a flexible **source → filter → sink** pipeline architecture.
## Configuration Methods
LogWisp supports three configuration methods with the following precedence:
1. **Command-line flags** (highest priority)
2. **Environment variables**
3. **Configuration file** (lowest priority)
### Complete Configuration Reference
| Category | CLI Flag | Environment Variable | TOML File |
|----------|----------|---------------------|-----------|
| **Top-level** |
| Router mode | `--router` | `LOGWISP_ROUTER` | `router = true` |
| Background mode | `--background` | `LOGWISP_BACKGROUND` | `background = true` |
| Show version | `--version` | `LOGWISP_VERSION` | `version = true` |
| Quiet mode | `--quiet` | `LOGWISP_QUIET` | `quiet = true` |
| Disable status reporter | `--disable-status-reporter` | `LOGWISP_DISABLE_STATUS_REPORTER` | `disable_status_reporter = true` |
| Config file | `--config <path>` | `LOGWISP_CONFIG_FILE` | N/A |
| Config directory | N/A | `LOGWISP_CONFIG_DIR` | N/A |
| **Logging** |
| Output mode | `--logging.output <mode>` | `LOGWISP_LOGGING_OUTPUT` | `[logging]`<br>`output = "stderr"` |
| Log level | `--logging.level <level>` | `LOGWISP_LOGGING_LEVEL` | `[logging]`<br>`level = "info"` |
| File directory | `--logging.file.directory <path>` | `LOGWISP_LOGGING_FILE_DIRECTORY` | `[logging.file]`<br>`directory = "./logs"` |
| File name | `--logging.file.name <name>` | `LOGWISP_LOGGING_FILE_NAME` | `[logging.file]`<br>`name = "logwisp"` |
| Max file size | `--logging.file.max_size_mb <size>` | `LOGWISP_LOGGING_FILE_MAX_SIZE_MB` | `[logging.file]`<br>`max_size_mb = 100` |
| Max total size | `--logging.file.max_total_size_mb <size>` | `LOGWISP_LOGGING_FILE_MAX_TOTAL_SIZE_MB` | `[logging.file]`<br>`max_total_size_mb = 1000` |
| Retention hours | `--logging.file.retention_hours <hours>` | `LOGWISP_LOGGING_FILE_RETENTION_HOURS` | `[logging.file]`<br>`retention_hours = 168` |
| Console target | `--logging.console.target <target>` | `LOGWISP_LOGGING_CONSOLE_TARGET` | `[logging.console]`<br>`target = "stderr"` |
| Console format | `--logging.console.format <format>` | `LOGWISP_LOGGING_CONSOLE_FORMAT` | `[logging.console]`<br>`format = "txt"` |
| **Pipelines** |
| Pipeline name | `--pipelines.N.name <name>` | `LOGWISP_PIPELINES_N_NAME` | `[[pipelines]]`<br>`name = "default"` |
| Source type | `--pipelines.N.sources.N.type <type>` | `LOGWISP_PIPELINES_N_SOURCES_N_TYPE` | `[[pipelines.sources]]`<br>`type = "directory"` |
| Source options | `--pipelines.N.sources.N.options.<key> <value>` | `LOGWISP_PIPELINES_N_SOURCES_N_OPTIONS_<KEY>` | `[[pipelines.sources]]`<br>`options = { ... }` |
| Filter type | `--pipelines.N.filters.N.type <type>` | `LOGWISP_PIPELINES_N_FILTERS_N_TYPE` | `[[pipelines.filters]]`<br>`type = "include"` |
| Filter logic | `--pipelines.N.filters.N.logic <logic>` | `LOGWISP_PIPELINES_N_FILTERS_N_LOGIC` | `[[pipelines.filters]]`<br>`logic = "or"` |
| Filter patterns | `--pipelines.N.filters.N.patterns <json>` | `LOGWISP_PIPELINES_N_FILTERS_N_PATTERNS` | `[[pipelines.filters]]`<br>`patterns = [...]` |
| Sink type | `--pipelines.N.sinks.N.type <type>` | `LOGWISP_PIPELINES_N_SINKS_N_TYPE` | `[[pipelines.sinks]]`<br>`type = "http"` |
| Sink options | `--pipelines.N.sinks.N.options.<key> <value>` | `LOGWISP_PIPELINES_N_SINKS_N_OPTIONS_<KEY>` | `[[pipelines.sinks]]`<br>`options = { ... }` |
| Auth type | `--pipelines.N.auth.type <type>` | `LOGWISP_PIPELINES_N_AUTH_TYPE` | `[pipelines.auth]`<br>`type = "none"` |
Note: `N` represents array indices (0-based).
## Configuration File Location
1. Command line: `--config /path/to/config.toml`
2. Environment: `$LOGWISP_CONFIG_FILE`
3. User config: `~/.config/logwisp.toml`
2. Environment: `$LOGWISP_CONFIG_FILE` and `$LOGWISP_CONFIG_DIR`
3. User config: `~/.config/logwisp/logwisp.toml`
4. Current directory: `./logwisp.toml`
## Configuration Structure
```toml
# Optional: Enable router mode
router = false
# Optional: Background mode
background = false
# Optional: Quiet mode
quiet = false
# Optional: Disable status reporter
disable_status_reporter = false
# Optional: LogWisp's own logging
[logging]
output = "stderr"
level = "info"
output = "stderr" # file, stdout, stderr, both, none
level = "info" # debug, info, warn, error
[logging.file]
directory = "./logs"
name = "logwisp"
max_size_mb = 100
max_total_size_mb = 1000
retention_hours = 168
[logging.console]
target = "stderr" # stdout, stderr, split
format = "txt" # txt or json
# Required: At least one pipeline
[[pipelines]]
@ -37,27 +103,6 @@ type = "http"
options = { ... }
```
## Logging Configuration
Controls LogWisp's operational logging:
```toml
[logging]
output = "stderr" # file, stdout, stderr, both, none
level = "info" # debug, info, warn, error
[logging.file]
directory = "./logs"
name = "logwisp"
max_size_mb = 100
max_total_size_mb = 1000
retention_hours = 168
[logging.console]
target = "stderr" # stdout, stderr, split
format = "txt" # txt or json
```
## Pipeline Configuration
Each `[[pipelines]]` section defines an independent processing pipeline.
@ -93,6 +138,39 @@ type = "stdin"
options = {}
```
#### HTTP Source
```toml
[[pipelines.sources]]
type = "http"
options = {
port = 8081, # Port to listen on
ingest_path = "/ingest", # Path for POST requests
buffer_size = 1000, # Input buffer size
rate_limit = { # Optional rate limiting
enabled = true,
requests_per_second = 10.0,
burst_size = 20,
limit_by = "ip"
}
}
```
#### TCP Source
```toml
[[pipelines.sources]]
type = "tcp"
options = {
port = 9091, # Port to listen on
buffer_size = 1000, # Input buffer size
rate_limit = { # Optional rate limiting
enabled = true,
requests_per_second = 5.0,
burst_size = 10,
limit_by = "ip"
}
}
```
### Filters
Control which log entries pass through:
@ -133,7 +211,8 @@ options = {
enabled = true,
interval_seconds = 30,
format = "comment", # comment or json
include_timestamp = true
include_timestamp = true,
include_stats = false
},
# Rate limiting
@ -142,7 +221,10 @@ options = {
requests_per_second = 10.0,
burst_size = 20,
limit_by = "ip", # ip or global
max_connections_per_ip = 5
max_connections_per_ip = 5,
max_total_connections = 100,
response_code = 429,
response_message = "Rate limit exceeded"
}
}
```
@ -154,8 +236,45 @@ type = "tcp"
options = {
port = 9090,
buffer_size = 5000,
heartbeat = { enabled = true, interval_seconds = 60 },
rate_limit = { enabled = true, requests_per_second = 5.0 }
heartbeat = { enabled = true, interval_seconds = 60, format = "json" },
rate_limit = { enabled = true, requests_per_second = 5.0, burst_size = 10 }
}
```
#### HTTP Client Sink
```toml
[[pipelines.sinks]]
type = "http_client"
options = {
url = "https://remote-log-server.com/ingest",
buffer_size = 1000,
batch_size = 100,
batch_delay_ms = 1000,
timeout_seconds = 30,
max_retries = 3,
retry_delay_ms = 1000,
retry_backoff = 2.0,
headers = {
"Authorization" = "Bearer <API_KEY_HERE>",
"X-Custom-Header" = "value"
},
insecure_skip_verify = false
}
```
#### TCP Client Sink
```toml
[[pipelines.sinks]]
type = "tcp_client"
options = {
address = "remote-server.com:9090",
buffer_size = 1000,
dial_timeout_seconds = 10,
write_timeout_seconds = 30,
keep_alive_seconds = 30,
reconnect_delay_ms = 1000,
max_reconnect_delay_seconds = 30,
reconnect_backoff = 1.5
}
```
@ -167,7 +286,9 @@ options = {
directory = "/var/log/logwisp",
name = "app",
max_size_mb = 100,
max_total_size_mb = 1000,
retention_hours = 168.0,
min_disk_free_mb = 1000,
buffer_size = 2000
}
```
@ -176,7 +297,10 @@ options = {
```toml
[[pipelines.sinks]]
type = "stdout" # or "stderr"
options = { buffer_size = 500 }
options = {
buffer_size = 500,
target = "stdout" # stdout, stderr, or split
}
```
## Complete Examples
@ -196,7 +320,7 @@ type = "http"
options = { port = 8080 }
```
### Production with Filtering
### Filtering
```toml
[logging]
@ -248,6 +372,10 @@ options = { path = "/var/log/app", pattern = "*.log" }
type = "stdin"
options = {}
[[pipelines.sources]]
type = "http"
options = { port = 8081, ingest_path = "/logs" }
[[pipelines.sinks]]
type = "tcp"
options = { port = 9090 }
@ -257,6 +385,7 @@ options = { port = 9090 }
```toml
# Run with: logwisp --router
router = true
[[pipelines]]
name = "api"
@ -282,11 +411,33 @@ options = { port = 8080 } # Shared port
# http://localhost:8080/status
```
## Validation
### Remote Log Forwarding
LogWisp validates on startup:
- Required fields (name, sources, sinks)
- Port conflicts between pipelines
- Pattern syntax
- Path accessibility
- Rate limit values
```toml
[[pipelines]]
name = "forwarder"
[[pipelines.sources]]
type = "directory"
options = { path = "/var/log/app", pattern = "*.log" }
[[pipelines.filters]]
type = "include"
patterns = ["ERROR", "WARN"]
[[pipelines.sinks]]
type = "http_client"
options = {
url = "https://log-aggregator.example.com/ingest",
batch_size = 100,
batch_delay_ms = 5000,
headers = { "Authorization" = "Bearer <API_KEY_HERE>" }
}
[[pipelines.sinks]]
type = "tcp_client"
options = {
address = "backup-logger.example.com:9090",
reconnect_delay_ms = 5000
}
```