v0.13.0 doc update, refactor, http/tcp chain source and sink added
This commit is contained in:
+15
-18
@@ -6,24 +6,20 @@ A pipeline-based log transport and processing system built in Go. LogWisp provid
|
||||
|
||||
### Core Capabilities
|
||||
- **Pipeline Architecture**: Independent processing pipelines with source(s) → filter → format → sink(s) flow
|
||||
- **Multiple Input Sources**: Directory monitoring, stdin, HTTP, TCP
|
||||
- **Flexible Output Sinks**: Console, file, HTTP SSE, TCP streaming, HTTP/TCP forwarding
|
||||
- **Multiple Input Sources**: File monitoring, console (stdin), random log generation, null
|
||||
- **Flexible Output Sinks**: Console, file, HTTP SSE, TCP streaming, null
|
||||
- **Real-time Processing**: Sub-millisecond latency with configurable buffering
|
||||
- **Hot Configuration Reload**: Update pipelines without service restart
|
||||
- **Session Management**: Built-in session tracking for multiple client connections
|
||||
|
||||
### Data Processing
|
||||
- **Pattern-based Filtering**: Chainable include/exclude filters with regex support
|
||||
- **Multiple Formatters**: Raw, JSON, and template-based text formatting
|
||||
- **Multiple Formatters**: Raw, JSON, and text formatting with integrated sanitizer policies
|
||||
- **Rate Limiting**: Pipeline rate controls
|
||||
- **Heartbeat Generation**: Flow-level heartbeat events for keep-alives
|
||||
|
||||
### Security & Reliability
|
||||
- **Authentication**: mTLS support
|
||||
- **Access Control**: IP whitelisting/blacklisting, connection limits
|
||||
- **TLS Encryption**: Full TLS 1.2/1.3 support for HTTP connections
|
||||
- **Automatic Reconnection**: Resilient client connections with exponential backoff
|
||||
- **File Rotation**: Size-based rotation with retention policies
|
||||
|
||||
### Operational Features
|
||||
- **Status Monitoring**: Real-time statistics and health endpoints
|
||||
- **Signal Handling**: Graceful shutdown and configuration reload via signals
|
||||
- **Background Mode**: Daemon operation with proper signal handling
|
||||
@@ -38,8 +34,7 @@ A pipeline-based log transport and processing system built in Go. LogWisp provid
|
||||
- [Output Sinks](sinks.md) - Sink types and output options
|
||||
- [Filters](filters.md) - Pattern-based log filtering
|
||||
- [Formatters](formatters.md) - Log formatting and transformation
|
||||
- [Security](security.md) - IP-based access control configuration and mTLS
|
||||
- [Networking](networking.md) - TLS, rate limiting, and network features
|
||||
- [Networking & Security](networking.md) - Network features (Note: TLS and Auth are currently placeholders in the new architecture)
|
||||
- [Command Line Interface](cli.md) - CLI flags and subcommands
|
||||
- [Operations Guide](operations.md) - Running and maintaining LogWisp
|
||||
|
||||
@@ -51,15 +46,17 @@ Install LogWisp and create a basic configuration:
|
||||
[[pipelines]]
|
||||
name = "default"
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
[pipelines.sources.directory]
|
||||
path = "./"
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "default_source"
|
||||
type = "file"
|
||||
[pipelines.plugin_sources.config]
|
||||
directory = "./"
|
||||
pattern = "*.log"
|
||||
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "default_sink"
|
||||
type = "console"
|
||||
[pipelines.sinks.console]
|
||||
[pipelines.plugin_sinks.config]
|
||||
target = "stdout"
|
||||
```
|
||||
|
||||
@@ -73,4 +70,4 @@ Run with: `logwisp -c config.toml`
|
||||
|
||||
## License
|
||||
|
||||
BSD 3-Clause License
|
||||
BSD 3-Clause License
|
||||
|
||||
+27
-24
@@ -13,13 +13,15 @@ Each pipeline operates independently with a source → filter → format → sin
|
||||
```
|
||||
Service (Main Process)
|
||||
├── Pipeline 1
|
||||
│ ├── Sources (1 or more)
|
||||
│ ├── Rate Limiter (optional)
|
||||
│ ├── Filter Chain (optional)
|
||||
│ ├── Formatter (optional)
|
||||
│ └── Sinks (1 or more)
|
||||
│ ├── Plugin Sources (1 or more)
|
||||
│ ├── Flow
|
||||
│ │ ├── Heartbeat Generator (optional)
|
||||
│ │ ├── Rate Limiter (optional)
|
||||
│ │ ├── Filter Chain (optional)
|
||||
│ │ └── Formatter (optional)
|
||||
│ └── Plugin Sinks (1 or more)
|
||||
├── Pipeline 2
|
||||
│ └── [Same structure]
|
||||
│ └── [Similar structure]
|
||||
└── Status Reporter (optional)
|
||||
```
|
||||
|
||||
@@ -27,11 +29,11 @@ Service (Main Process)
|
||||
|
||||
### Processing Stages
|
||||
|
||||
1. **Source Stage**: Sources monitor inputs and generate log entries
|
||||
2. **Rate Limiting**: Optional pipeline-level rate control
|
||||
3. **Filtering**: Pattern-based inclusion/exclusion
|
||||
4. **Formatting**: Transform entries to desired output format
|
||||
5. **Distribution**: Fan-out to multiple sinks
|
||||
1. **Source Stage**: Plugin sources monitor inputs and generate log entries
|
||||
2. **Flow - Rate Limiting**: Optional pipeline-level rate control
|
||||
3. **Flow - Filtering**: Pattern-based inclusion/exclusion
|
||||
4. **Flow - Formatting**: Transform entries to desired output format with sanitization
|
||||
5. **Distribution**: Fan-out to multiple plugin sinks
|
||||
|
||||
### Entry Lifecycle
|
||||
|
||||
@@ -50,14 +52,16 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
- Sinks: Independent buffers per sink
|
||||
- Network components: Additional TCP/HTTP buffers
|
||||
|
||||
*"Sink dispatch uses non-blocking sends. When a sink's input buffer is full, the event is dropped for that sink only and counted in pipeline statistics (`total_dropped_by_sink`). The policy is uniform and not configurable; a slow sink does not stall the pipeline or other sinks."*
|
||||
|
||||
## Component Types
|
||||
|
||||
### Sources (Input)
|
||||
|
||||
- **Directory Source**: File system monitoring with rotation detection
|
||||
- **Stdin Source**: Standard input processing
|
||||
- **HTTP Source**: REST endpoint for log ingestion
|
||||
- **TCP Source**: Raw TCP socket listener
|
||||
- **File Source**: File system directory monitoring with rotation detection
|
||||
- **Console Source**: Standard input processing (stdin)
|
||||
- **Random Source**: Generates random log entries for testing
|
||||
- **Null Source**: Discards logs, used for testing
|
||||
|
||||
### Sinks (Output)
|
||||
|
||||
@@ -65,14 +69,13 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
- **File Sink**: Rotating file writer
|
||||
- **HTTP Sink**: Server-Sent Events (SSE) streaming
|
||||
- **TCP Sink**: TCP server for client connections
|
||||
- **HTTP Client Sink**: Forward to remote HTTP endpoints
|
||||
- **TCP Client Sink**: Forward to remote TCP servers
|
||||
- **Null Sink**: Discards all received events
|
||||
|
||||
### Processing Components
|
||||
|
||||
- **Rate Limiter**: Token bucket algorithm for flow control
|
||||
- **Filter Chain**: Sequential pattern matching
|
||||
- **Formatters**: Raw, JSON, or template-based text transformation
|
||||
- **Formatters**: Raw, JSON, or text transformation with sanitizer policies
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
@@ -95,8 +98,7 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
### Connection Patterns
|
||||
|
||||
**Chaining Design**:
|
||||
- TCP Client Sink → TCP Source: Direct TCP forwarding
|
||||
- HTTP Client Sink → HTTP Source: HTTP-based forwarding
|
||||
- Future plan
|
||||
|
||||
**Monitoring Design**:
|
||||
- TCP Sink: Debugging interface
|
||||
@@ -106,8 +108,6 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
|
||||
- HTTP/1.1 and HTTP/2 for HTTP connections
|
||||
- Raw TCP connections
|
||||
- TLS 1.2/1.3 for HTTPS connections (HTTP only)
|
||||
- Server-Sent Events for real-time streaming
|
||||
|
||||
## Resource Management
|
||||
|
||||
@@ -125,6 +125,10 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
|
||||
### Connection Management
|
||||
|
||||
- HTTP sink silenty drops IPv6 connections (deliberate IPv4-only enforcement)
|
||||
|
||||
*Note:* Placeholder; below features are removed in restructuring and will be added in the future release.
|
||||
|
||||
- Per-IP connection limits
|
||||
- Global connection caps
|
||||
- Automatic reconnection with exponential backoff
|
||||
@@ -136,7 +140,6 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
|
||||
- Panic recovery in pipeline processing
|
||||
- Independent pipeline operation
|
||||
- Automatic source restart on failure
|
||||
- Sink failure isolation
|
||||
|
||||
### Data Integrity
|
||||
@@ -165,4 +168,4 @@ Each component maintains internal buffers to handle burst traffic:
|
||||
- Horizontal: Multiple LogWisp instances with different configurations
|
||||
- Vertical: Multiple pipelines per instance
|
||||
- Fan-out: Multiple sinks per pipeline
|
||||
- Fan-in: Multiple sources per pipeline
|
||||
- Fan-in: Multiple sources per pipeline
|
||||
|
||||
+13
-57
@@ -15,30 +15,8 @@ logwisp [options]
|
||||
|
||||
| Command | Description |
|
||||
|---------|-------------|
|
||||
| `tls` | Generate TLS certificates |
|
||||
| `version` | Display version information |
|
||||
| `help` | Show help information |
|
||||
|
||||
### tls Command
|
||||
|
||||
Generate TLS certificates.
|
||||
|
||||
```bash
|
||||
logwisp tls [options]
|
||||
```
|
||||
|
||||
**Options:**
|
||||
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-ca` | Generate CA certificate | - |
|
||||
| `-server` | Generate server certificate | - |
|
||||
| `-client` | Generate client certificate | - |
|
||||
| `-host` | Comma-separated hosts/IPs | localhost |
|
||||
| `-o` | Output file prefix | Required |
|
||||
| `-ca-cert` | CA certificate file | Required for server/client |
|
||||
| `-ca-key` | CA key file | Required for server/client |
|
||||
| `-days` | Certificate validity days | 365 |
|
||||
| `--version` | Display version information |
|
||||
| `--help` | Show help information |
|
||||
|
||||
### version Command
|
||||
|
||||
@@ -63,10 +41,9 @@ Output includes:
|
||||
| Flag | Description | Default |
|
||||
|------|-------------|---------|
|
||||
| `-c, --config` | Configuration file path | `./logwisp.toml` |
|
||||
| `-b, --background` | Run as daemon | false |
|
||||
| `-q, --quiet` | Suppress console output | false |
|
||||
| `--disable-status-reporter` | Disable status logging | false |
|
||||
| `--config-auto-reload` | Enable config hot reload | false |
|
||||
| `--status-reporter` | Status logging | true |
|
||||
| `--auto-reload` | Enable config hot reload | false |
|
||||
|
||||
### Logging Options
|
||||
|
||||
@@ -91,9 +68,9 @@ Configure pipelines via CLI (N = array index, 0-based).
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `--pipelines.N.name` | Pipeline name |
|
||||
| `--pipelines.N.sources.N.type` | Source type |
|
||||
| `--pipelines.N.filters.N.type` | Filter type |
|
||||
| `--pipelines.N.sinks.N.type` | Sink type |
|
||||
| `--pipelines.N.plugin_sources.N.type` | Source type |
|
||||
| `--pipelines.N.flow.filters.N.type` | Filter type |
|
||||
| `--pipelines.N.plugin_sinks.N.type` | Sink type |
|
||||
|
||||
## Flag Formats
|
||||
|
||||
@@ -102,7 +79,7 @@ Configure pipelines via CLI (N = array index, 0-based).
|
||||
```bash
|
||||
logwisp --quiet
|
||||
logwisp --quiet=true
|
||||
logwisp --quiet=false
|
||||
logwisp --pipelines.0.plugin_sources.0.type=console
|
||||
```
|
||||
|
||||
### String Flags
|
||||
@@ -117,13 +94,13 @@ logwisp -c config.toml
|
||||
```bash
|
||||
logwisp --logging.level=debug
|
||||
logwisp --pipelines.0.name=myapp
|
||||
logwisp --pipelines.0.sources.0.type=stdin
|
||||
logwisp --pipelines.0.sources.0.type=console
|
||||
```
|
||||
|
||||
### Array Values (JSON)
|
||||
|
||||
```bash
|
||||
logwisp --pipelines.0.filters.0.patterns='["ERROR","WARN"]'
|
||||
logwisp --pipelines.0.flow.filters.0.patterns='["ERROR","WARN"]'
|
||||
```
|
||||
|
||||
## Environment Variables
|
||||
@@ -171,7 +148,7 @@ export LOGWISP_PIPELINES_0_NAME=myapp
|
||||
logwisp --logging.output=stderr --logging.level=debug
|
||||
|
||||
# Quick test with stdin
|
||||
logwisp --pipelines.0.sources.0.type=stdin --pipelines.0.sinks.0.type=console
|
||||
logwisp --pipelines.0.plugin_sources.0.type=console --pipelines.0.plugin_sinks.0.type=console
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
@@ -194,19 +171,6 @@ logwisp --config test.toml --logging.level=debug --disable-status-reporter
|
||||
logwisp --config test.toml --quiet
|
||||
```
|
||||
|
||||
### Quick Commands
|
||||
|
||||
```bash
|
||||
# Generate admin password
|
||||
logwisp auth -u admin -b
|
||||
|
||||
# Create self-signed certs
|
||||
logwisp tls -server -host localhost -o server
|
||||
|
||||
# Check version
|
||||
logwisp version
|
||||
```
|
||||
|
||||
## Help System
|
||||
|
||||
### General Help
|
||||
@@ -217,14 +181,6 @@ logwisp -h
|
||||
logwisp help
|
||||
```
|
||||
|
||||
### Command Help
|
||||
|
||||
```bash
|
||||
logwisp auth --help
|
||||
logwisp tls --help
|
||||
logwisp help auth
|
||||
```
|
||||
|
||||
## Special Flags
|
||||
|
||||
### Internal Flags
|
||||
@@ -235,6 +191,6 @@ These flags are for internal use:
|
||||
|
||||
### Hidden Behaviors
|
||||
|
||||
- SIGHUP ignored by default (nohup behavior)
|
||||
- SIGHUP ignored ignored during startup (after startup triggers config reload)
|
||||
- Automatic panic recovery in pipelines
|
||||
- Resource cleanup on shutdown
|
||||
- Resource cleanup on shutdown
|
||||
|
||||
+29
-24
@@ -24,10 +24,9 @@ Top-level configuration options:
|
||||
|
||||
| Setting | Type | Default | Description |
|
||||
|---------|------|---------|-------------|
|
||||
| `background` | bool | false | Run as daemon process |
|
||||
| `quiet` | bool | false | Suppress console output |
|
||||
| `disable_status_reporter` | bool | false | Disable periodic status logging |
|
||||
| `config_auto_reload` | bool | false | Enable file watch for auto-reload |
|
||||
| `status_reporter` | bool | true | Periodic status logging |
|
||||
| `auto_reload` | bool | false | Enable file watch for auto-reload |
|
||||
|
||||
## Logging Configuration
|
||||
|
||||
@@ -47,7 +46,6 @@ retention_hours = 168.0
|
||||
|
||||
[logging.console]
|
||||
target = "stdout" # stdout|stderr|split
|
||||
format = "txt" # txt|json
|
||||
```
|
||||
|
||||
### Output Modes
|
||||
@@ -68,30 +66,34 @@ Each `[[pipelines]]` section defines an independent processing pipeline:
|
||||
name = "pipeline-name"
|
||||
|
||||
# Rate limiting (optional)
|
||||
[pipelines.rate_limit]
|
||||
[pipelines.flow.rate_limit]
|
||||
rate = 1000.0
|
||||
burst = 2000.0
|
||||
policy = "drop" # pass|drop
|
||||
max_entry_size_bytes = 0 # 0=unlimited
|
||||
|
||||
# Format configuration (optional)
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "json" # raw|json|txt
|
||||
sanitizer_policy = "json"
|
||||
|
||||
# Sources (required, 1+)
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "my_source"
|
||||
type = "file"
|
||||
[pipelines.plugin_sources.config]
|
||||
# ... source-specific config
|
||||
|
||||
# Filters (optional)
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
|
||||
# Sinks (required, 1+)
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "my_sink"
|
||||
type = "http"
|
||||
[pipelines.plugin_sinks.config]
|
||||
# ... sink-specific config
|
||||
```
|
||||
|
||||
@@ -113,7 +115,7 @@ All configuration options support environment variable overrides:
|
||||
| `quiet` | `LOGWISP_QUIET` |
|
||||
| `logging.level` | `LOGWISP_LOGGING_LEVEL` |
|
||||
| `pipelines[0].name` | `LOGWISP_PIPELINES_0_NAME` |
|
||||
| `pipelines[0].sources[0].type` | `LOGWISP_PIPELINES_0_SOURCES_0_TYPE` |
|
||||
| `pipelines[0].plugin_sources[0].type` | `LOGWISP_PIPELINES_0_PLUGIN_SOURCES_0_TYPE` |
|
||||
|
||||
## Command-Line Overrides
|
||||
|
||||
@@ -123,13 +125,15 @@ All configuration options can be overridden via CLI flags:
|
||||
logwisp --quiet \
|
||||
--logging.level=debug \
|
||||
--pipelines.0.name=myapp \
|
||||
--pipelines.0.sources.0.type=stdin
|
||||
--pipelines.0.plugin_sources.0.type=console
|
||||
```
|
||||
|
||||
## Configuration Validation
|
||||
|
||||
LogWisp validates configuration at startup:
|
||||
- Required fields presence
|
||||
- Rpipelines non-empty, name non-empty, ≥1 source, ≥1 sink, logging enum values.equired fields presence
|
||||
|
||||
Partial check in plugin constructor:
|
||||
- Type correctness
|
||||
- Port conflicts
|
||||
- Path accessibility
|
||||
@@ -141,12 +145,12 @@ LogWisp validates configuration at startup:
|
||||
Enable configuration hot reload:
|
||||
|
||||
```toml
|
||||
config_auto_reload = true
|
||||
auto_reload = true
|
||||
```
|
||||
|
||||
Or via command line:
|
||||
```bash
|
||||
logwisp --config-auto-reload
|
||||
logwisp --auto-reload
|
||||
```
|
||||
|
||||
Reload triggers:
|
||||
@@ -161,7 +165,6 @@ Reloadable items:
|
||||
|
||||
Non-reloadable (requires restart):
|
||||
- Logging configuration
|
||||
- Background mode
|
||||
- Global settings
|
||||
|
||||
## Default Configuration
|
||||
@@ -172,15 +175,17 @@ Minimal working configuration:
|
||||
[[pipelines]]
|
||||
name = "default"
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
[pipelines.sources.directory]
|
||||
path = "./"
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "default_source"
|
||||
type = "file"
|
||||
[pipelines.plugin_sources.config]
|
||||
directory = "./"
|
||||
pattern = "*.log"
|
||||
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "default_sink"
|
||||
type = "console"
|
||||
[pipelines.sinks.console]
|
||||
[pipelines.plugin_sinks.config]
|
||||
target = "stdout"
|
||||
```
|
||||
|
||||
@@ -195,4 +200,4 @@ target = "stdout"
|
||||
| Float | float64 | Decimal string |
|
||||
| Boolean | bool | true/false |
|
||||
| Array | []T | JSON array string |
|
||||
| Table | struct | Nested with `_` |
|
||||
| Table | struct | Nested with `_` |
|
||||
|
||||
+11
-11
@@ -9,7 +9,7 @@ LogWisp filters control which log entries pass through the pipeline using patter
|
||||
Only entries matching patterns pass through.
|
||||
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
logic = "or" # or|and
|
||||
patterns = [
|
||||
@@ -24,7 +24,7 @@ patterns = [
|
||||
Entries matching patterns are dropped.
|
||||
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"DEBUG",
|
||||
@@ -89,12 +89,12 @@ Multiple filters execute sequentially:
|
||||
|
||||
```toml
|
||||
# First filter: Include errors and warnings
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
|
||||
# Second filter: Exclude test environments
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["test-env", "staging"]
|
||||
```
|
||||
@@ -137,14 +137,14 @@ patterns = ["ERROR", "WARN", "FATAL", "CRITICAL"]
|
||||
|
||||
### Application Filtering
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["app1", "app2", "app3"]
|
||||
```
|
||||
|
||||
### Noise Reduction
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"health-check",
|
||||
@@ -156,7 +156,7 @@ patterns = [
|
||||
|
||||
### Security Filtering
|
||||
```toml
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"password",
|
||||
@@ -169,17 +169,17 @@ patterns = [
|
||||
### Multi-stage Filtering
|
||||
```toml
|
||||
# Include production logs
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["prod-", "production"]
|
||||
|
||||
# Include only errors
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "EXCEPTION", "FATAL"]
|
||||
|
||||
# Exclude known issues
|
||||
[[pipelines.filters]]
|
||||
[[pipelines.flow.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["ECONNRESET", "broken pipe"]
|
||||
```
|
||||
```
|
||||
|
||||
+16
-51
@@ -9,11 +9,10 @@ LogWisp formatters transform log entries before output to sinks.
|
||||
Outputs the log message as-is with optional newline.
|
||||
|
||||
```toml
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "raw"
|
||||
|
||||
[pipelines.format.raw]
|
||||
add_new_line = true
|
||||
sanitizer_policy = "raw"
|
||||
flags = 1
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
@@ -21,6 +20,9 @@ add_new_line = true
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `add_new_line` | bool | true | Append newline to messages |
|
||||
| `type` | string | "raw" | raw, json, or txt |
|
||||
| `flags` | int64 | 0 | log/formatter flags override |
|
||||
| `sanitizer_policy` | string | | Sanitizer policy (e.g. "json", "raw", "txt", "shell") |
|
||||
|
||||
### JSON Formatter
|
||||
|
||||
@@ -30,24 +32,11 @@ Produces structured JSON output.
|
||||
[pipelines.format]
|
||||
type = "json"
|
||||
|
||||
[pipelines.format.json]
|
||||
pretty = false
|
||||
timestamp_field = "timestamp"
|
||||
level_field = "level"
|
||||
message_field = "message"
|
||||
source_field = "source"
|
||||
+[pipelines.flow.format]
|
||||
type = "json"
|
||||
sanitizer_policy = "json"
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `pretty` | bool | false | Pretty print JSON |
|
||||
| `timestamp_field` | string | "timestamp" | Field name for timestamp |
|
||||
| `level_field` | string | "level" | Field name for log level |
|
||||
| `message_field` | string | "message" | Field name for message |
|
||||
| `source_field` | string | "source" | Field name for source |
|
||||
|
||||
**Output Structure:**
|
||||
```json
|
||||
{
|
||||
@@ -63,11 +52,9 @@ source_field = "source"
|
||||
Template-based text formatting.
|
||||
|
||||
```toml
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "txt"
|
||||
|
||||
[pipelines.format.txt]
|
||||
template = "[{{.Timestamp | FmtTime}}] [{{.Level | ToUpper}}] {{.Source}} - {{.Message}}"
|
||||
sanitizer_policy = "txt"
|
||||
timestamp_format = "2006-01-02T15:04:05.000Z07:00"
|
||||
```
|
||||
|
||||
@@ -75,8 +62,7 @@ timestamp_format = "2006-01-02T15:04:05.000Z07:00"
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `template` | string | See below | Go template string |
|
||||
| `timestamp_format` | string | RFC3339 | Go time format string |
|
||||
| `timestamp_format` | string | "" | Time format override |
|
||||
|
||||
**Default Template:**
|
||||
```
|
||||
@@ -134,12 +120,12 @@ Each pipeline can have its own formatter:
|
||||
```toml
|
||||
[[pipelines]]
|
||||
name = "json-pipeline"
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "json"
|
||||
|
||||
[[pipelines]]
|
||||
name = "text-pipeline"
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "txt"
|
||||
```
|
||||
|
||||
@@ -182,34 +168,13 @@ Relative performance (fastest to slowest):
|
||||
|
||||
### Structured Logging
|
||||
```toml
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "json"
|
||||
[pipelines.format.json]
|
||||
pretty = false
|
||||
```
|
||||
|
||||
### Human-Readable Logs
|
||||
```toml
|
||||
[pipelines.format]
|
||||
[pipelines.flow.format]
|
||||
type = "txt"
|
||||
[pipelines.format.txt]
|
||||
template = "{{.Timestamp | FmtTime}} [{{.Level}}] {{.Message}}"
|
||||
timestamp_format = "15:04:05"
|
||||
```
|
||||
|
||||
### Syslog Format
|
||||
```toml
|
||||
[pipelines.format]
|
||||
type = "txt"
|
||||
[pipelines.format.txt]
|
||||
template = "{{.Timestamp | FmtTime}} {{.Source}} {{.Level}}: {{.Message}}"
|
||||
timestamp_format = "Jan 2 15:04:05"
|
||||
```
|
||||
|
||||
### Minimal Output
|
||||
```toml
|
||||
[pipelines.format]
|
||||
type = "txt"
|
||||
[pipelines.format.txt]
|
||||
template = "{{.Message}}"
|
||||
```
|
||||
+16
-210
@@ -1,173 +1,40 @@
|
||||
# Networking
|
||||
|
||||
Network configuration for LogWisp connections, including TLS, rate limiting, and access control.
|
||||
*Note: Under redesign*
|
||||
|
||||
## TLS Configuration
|
||||
|
||||
### TLS Support Matrix
|
||||
|
||||
| Component | TLS Support | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| HTTP Source | ✓ | Full TLS 1.2/1.3 |
|
||||
| HTTP Sink | ✓ | Full TLS 1.2/1.3 |
|
||||
| HTTP Client | ✓ | Client certificates |
|
||||
| TCP Source | ✗ | No encryption |
|
||||
| TCP Sink | ✗ | No encryption |
|
||||
| TCP Client | ✗ | No encryption |
|
||||
|
||||
### Server TLS Configuration
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.tls]
|
||||
enabled = true
|
||||
cert_file = "/path/to/server.pem"
|
||||
key_file = "/path/to/server.key"
|
||||
min_version = "TLS1.2" # TLS1.2|TLS1.3
|
||||
client_auth = false
|
||||
client_ca_file = "/path/to/client-ca.pem"
|
||||
verify_client_cert = true
|
||||
```
|
||||
|
||||
### Client TLS Configuration
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http_client.tls]
|
||||
enabled = true
|
||||
server_ca_file = "/path/to/ca.pem" # For server verification
|
||||
server_name = "logs.example.com"
|
||||
insecure_skip_verify = false
|
||||
client_cert_file = "/path/to/client.pem" # For mTLS
|
||||
client_key_file = "/path/to/client.key" # For mTLS
|
||||
```
|
||||
|
||||
### TLS Certificate Generation
|
||||
|
||||
Using the `tls` command:
|
||||
|
||||
```bash
|
||||
# Generate CA certificate
|
||||
logwisp tls -ca -o myca
|
||||
|
||||
# Generate server certificate
|
||||
logwisp tls -server -ca-cert myca.pem -ca-key myca.key -host localhost,server.example.com -o server
|
||||
|
||||
# Generate client certificate
|
||||
logwisp tls -client -ca-cert myca.pem -ca-key myca.key -o client
|
||||
```
|
||||
|
||||
Command options:
|
||||
|
||||
| Flag | Description |
|
||||
|------|-------------|
|
||||
| `-ca` | Generate CA certificate |
|
||||
| `-server` | Generate server certificate |
|
||||
| `-client` | Generate client certificate |
|
||||
| `-host` | Comma-separated hostnames/IPs |
|
||||
| `-o` | Output file prefix |
|
||||
| `-days` | Certificate validity (default: 365) |
|
||||
|
||||
## Network Rate Limiting
|
||||
|
||||
### Configuration Options
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.net_limit]
|
||||
enabled = true
|
||||
max_connections_per_ip = 10
|
||||
max_connections_total = 100
|
||||
requests_per_second = 100.0
|
||||
burst_size = 200
|
||||
response_code = 429
|
||||
response_message = "Rate limit exceeded"
|
||||
ip_whitelist = ["192.168.1.0/24"]
|
||||
ip_blacklist = ["10.0.0.0/8"]
|
||||
```
|
||||
|
||||
### Rate Limiting Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `enabled` | bool | Enable rate limiting |
|
||||
| `max_connections_per_ip` | int | Per-IP connection limit |
|
||||
| `max_connections_total` | int | Global connection limit |
|
||||
| `requests_per_second` | float | Request rate limit |
|
||||
| `burst_size` | int | Token bucket burst capacity |
|
||||
| `response_code` | int | HTTP response code when limited |
|
||||
| `response_message` | string | Response message when limited |
|
||||
|
||||
### IP Access Control
|
||||
|
||||
**Whitelist**: Only specified IPs/networks allowed
|
||||
```toml
|
||||
ip_whitelist = [
|
||||
"192.168.1.0/24", # Local network
|
||||
"10.0.0.0/8", # Private network
|
||||
"203.0.113.5" # Specific IP
|
||||
]
|
||||
```
|
||||
|
||||
**Blacklist**: Specified IPs/networks denied
|
||||
```toml
|
||||
ip_blacklist = [
|
||||
"192.168.1.100", # Blocked host
|
||||
"10.0.0.0/16" # Blocked subnet
|
||||
]
|
||||
```
|
||||
|
||||
Processing order:
|
||||
1. Blacklist (immediate deny if matched)
|
||||
2. Whitelist (must match if configured)
|
||||
3. Rate limiting
|
||||
4. Authentication
|
||||
|
||||
*Note: As of the latest architecture updates, network TLS, mTLS, and Rate Limiting features are undergoing a redesign and are currently acting as placeholders. The documentation below details the structure for future updates.*
|
||||
|
||||
## Connection Management
|
||||
|
||||
### TCP Keep-Alive
|
||||
|
||||
```toml
|
||||
[pipelines.sources.tcp]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "tcp_out"
|
||||
type = "tcp"
|
||||
[pipelines.plugin_sinks.config]
|
||||
keep_alive = true
|
||||
keep_alive_period_ms = 30000 # 30 seconds
|
||||
```
|
||||
|
||||
Benefits:
|
||||
- Detect dead connections
|
||||
- Prevent connection timeout
|
||||
- Maintain NAT mappings
|
||||
|
||||
### Connection Timeouts
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http]
|
||||
read_timeout_ms = 10000 # 10 seconds
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "http_out"
|
||||
type = "http"
|
||||
[pipelines.plugin_sinks.config]
|
||||
write_timeout_ms = 10000 # 10 seconds
|
||||
|
||||
[pipelines.sinks.tcp_client]
|
||||
dial_timeout = 10 # Connection timeout
|
||||
write_timeout = 30 # Write timeout
|
||||
read_timeout = 10 # Read timeout
|
||||
```
|
||||
|
||||
### Connection Limits
|
||||
|
||||
Global limits:
|
||||
```toml
|
||||
max_connections = 100 # Total concurrent connections
|
||||
```
|
||||
|
||||
Per-IP limits:
|
||||
```toml
|
||||
max_connections_per_ip = 10
|
||||
```
|
||||
|
||||
## Heartbeat Configuration
|
||||
|
||||
Keep connections alive with periodic heartbeats:
|
||||
|
||||
### HTTP Sink Heartbeat
|
||||
Keep connections alive with periodic heartbeats. Note that Heartbeat is a flow-level feature in the new architecture.
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http.heartbeat]
|
||||
[pipelines.flow.heartbeat]
|
||||
enabled = true
|
||||
interval_ms = 30000
|
||||
include_timestamp = true
|
||||
@@ -175,37 +42,18 @@ include_stats = false
|
||||
format = "comment" # comment|event|json
|
||||
```
|
||||
|
||||
Formats:
|
||||
- **comment**: SSE comment (`: heartbeat`)
|
||||
- **event**: SSE event with data
|
||||
- **json**: JSON-formatted heartbeat
|
||||
|
||||
### TCP Sink Heartbeat
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.tcp.heartbeat]
|
||||
enabled = true
|
||||
interval_ms = 30000
|
||||
include_timestamp = true
|
||||
include_stats = false
|
||||
format = "json" # json|txt
|
||||
```
|
||||
|
||||
## Network Protocols
|
||||
|
||||
### HTTP/HTTPS
|
||||
|
||||
- HTTP/1.1 and HTTP/2 support
|
||||
- HTTP/1.1 support
|
||||
- Persistent connections
|
||||
- Chunked transfer encoding
|
||||
- Server-Sent Events (SSE)
|
||||
|
||||
### TCP
|
||||
|
||||
- Raw TCP sockets
|
||||
- Newline-delimited protocol
|
||||
- Binary-safe transmission
|
||||
- No encryption available
|
||||
|
||||
## Port Configuration
|
||||
|
||||
@@ -213,9 +61,7 @@ format = "json" # json|txt
|
||||
|
||||
| Service | Default Port | Protocol |
|
||||
|---------|--------------|----------|
|
||||
| HTTP Source | 8081 | HTTP/HTTPS |
|
||||
| HTTP Sink | 8080 | HTTP/HTTPS |
|
||||
| TCP Source | 9091 | TCP |
|
||||
| HTTP Sink | 8080 | HTTP |
|
||||
| TCP Sink | 9090 | TCP |
|
||||
|
||||
### Port Conflict Prevention
|
||||
@@ -223,46 +69,6 @@ format = "json" # json|txt
|
||||
LogWisp validates port usage at startup:
|
||||
- Detects port conflicts across pipelines
|
||||
- Prevents duplicate bindings
|
||||
- Suggests alternative ports
|
||||
|
||||
## Network Security
|
||||
|
||||
### Best Practices
|
||||
|
||||
1. **Use TLS for HTTP** connections when possible
|
||||
2. **Implement rate limiting** to prevent DoS
|
||||
3. **Configure IP whitelists** for restricted access
|
||||
4. **Enable authentication** for all network endpoints
|
||||
5. **Use non-standard ports** to reduce scanning exposure
|
||||
6. **Monitor connection metrics** for anomalies
|
||||
7. **Set appropriate timeouts** to prevent resource exhaustion
|
||||
|
||||
### Security Warnings
|
||||
|
||||
- TCP connections are **always unencrypted**
|
||||
- HTTP Basic/Token auth **requires TLS**
|
||||
- Avoid `skip_verify` in production
|
||||
- Never expose unauthenticated endpoints publicly
|
||||
|
||||
## Load Balancing
|
||||
|
||||
### Client-Side Load Balancing
|
||||
|
||||
Configure multiple endpoints (future feature):
|
||||
```toml
|
||||
[[pipelines.sinks.http_client]]
|
||||
urls = [
|
||||
"https://log1.example.com/ingest",
|
||||
"https://log2.example.com/ingest"
|
||||
]
|
||||
strategy = "round-robin" # round-robin|random|least-conn
|
||||
```
|
||||
|
||||
### Server-Side Considerations
|
||||
|
||||
- Use reverse proxy for load distribution
|
||||
- Configure session affinity if needed
|
||||
- Monitor individual instance health
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
@@ -286,4 +92,4 @@ strategy = "round-robin" # round-robin|random|least-conn
|
||||
**Connection Timeout**
|
||||
- Increase timeout values
|
||||
- Check network latency
|
||||
- Verify keep-alive settings
|
||||
- Verify keep-alive settings
|
||||
|
||||
+9
-25
@@ -2,6 +2,8 @@
|
||||
|
||||
Running, monitoring, and maintaining LogWisp in production.
|
||||
|
||||
*Note: TLS, acccess control under redesign*
|
||||
|
||||
## Starting LogWisp
|
||||
|
||||
### Manual Start
|
||||
@@ -60,7 +62,7 @@ kill -USR1 $(pidof logwisp)
|
||||
|
||||
Test configuration without starting:
|
||||
```bash
|
||||
logwisp --config test.toml --quiet --disable-status-reporter
|
||||
logwisp --config test.toml --quiet --status-reporter=false
|
||||
```
|
||||
|
||||
Check for errors:
|
||||
@@ -168,18 +170,10 @@ Production recommendation: `info` or `warn`
|
||||
Adjust buffers based on load:
|
||||
|
||||
```toml
|
||||
# High-volume source
|
||||
[[pipelines.sources]]
|
||||
type = "http"
|
||||
[pipelines.sources.http]
|
||||
buffer_size = 5000 # Increase for burst traffic
|
||||
|
||||
# Slow consumer sink
|
||||
[[pipelines.sinks]]
|
||||
type = "http_client"
|
||||
[pipelines.sinks.http_client]
|
||||
buffer_size = 10000 # Larger buffer for slow endpoints
|
||||
batch_size = 500 # Larger batches
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "file_in"
|
||||
type = "file"
|
||||
[pipelines.plugin_sources.config]
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
@@ -187,22 +181,12 @@ batch_size = 500 # Larger batches
|
||||
Protect against overload:
|
||||
|
||||
```toml
|
||||
[pipelines.rate_limit]
|
||||
[pipelines.flow.rate_limit]
|
||||
rate = 1000.0 # Entries per second
|
||||
burst = 2000.0 # Burst capacity
|
||||
policy = "drop" # Drop excess entries
|
||||
```
|
||||
|
||||
### Connection Limits
|
||||
|
||||
Prevent resource exhaustion:
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.net_limit]
|
||||
max_connections_total = 1000
|
||||
max_connections_per_ip = 50
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
@@ -340,4 +324,4 @@ Data loss:
|
||||
- Run multiple instances for redundancy
|
||||
- Use load balancer for distribution
|
||||
- Implement monitoring alerts
|
||||
- Document recovery procedures
|
||||
- Document recovery procedures
|
||||
|
||||
+1
-55
@@ -1,58 +1,4 @@
|
||||
# Security
|
||||
|
||||
## mTLS (Mutual TLS)
|
||||
*Note: Security features like mTLS and IP-Based Access Control are currently under redesign and act as placeholders in the new architecture. Future versions will reintroduce full security capabilities.*
|
||||
|
||||
Certificate-based authentication for HTTPS.
|
||||
|
||||
### Server Configuration
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.tls]
|
||||
enabled = true
|
||||
cert_file = "/path/to/server.pem"
|
||||
key_file = "/path/to/server.key"
|
||||
client_auth = true
|
||||
client_ca_file = "/path/to/ca.pem"
|
||||
verify_client_cert = true
|
||||
```
|
||||
|
||||
### Client Configuration
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http_client.tls]
|
||||
enabled = true
|
||||
cert_file = "/path/to/client.pem"
|
||||
key_file = "/path/to/client.key"
|
||||
```
|
||||
|
||||
### Certificate Generation
|
||||
|
||||
Use the `tls` command:
|
||||
```bash
|
||||
# Generate CA
|
||||
logwisp tls -ca -o ca
|
||||
|
||||
# Generate server certificate
|
||||
logwisp tls -server -ca-cert ca.pem -ca-key ca.key -host localhost -o server
|
||||
|
||||
# Generate client certificate
|
||||
logwisp tls -client -ca-cert ca.pem -ca-key ca.key -o client
|
||||
```
|
||||
|
||||
## Access Control
|
||||
|
||||
ogWisp provides IP-based access control for network connections.
|
||||
|
||||
+## IP-Based Access Control
|
||||
|
||||
Configure IP-based access control for sources:
|
||||
```toml
|
||||
[pipelines.sources.http.net_limit]
|
||||
enabled = true
|
||||
ip_whitelist = ["192.168.1.0/24", "10.0.0.0/8"]
|
||||
ip_blacklist = ["192.168.1.100"]
|
||||
```
|
||||
|
||||
Priority order:
|
||||
1. Blacklist (checked first, immediate deny)
|
||||
2. Whitelist (if configured, must match)
|
||||
+22
-151
@@ -9,13 +9,12 @@ LogWisp sinks deliver processed log entries to various destinations.
|
||||
Output to stdout/stderr.
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "console_out"
|
||||
type = "console"
|
||||
|
||||
[pipelines.sinks.console]
|
||||
[pipelines.plugin_sinks.config]
|
||||
target = "stdout" # stdout|stderr|split
|
||||
colorize = false
|
||||
buffer_size = 100
|
||||
buffer_size = 1000
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
@@ -23,8 +22,7 @@ buffer_size = 100
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `target` | string | "stdout" | Output target (stdout/stderr/split) |
|
||||
| `colorize` | bool | false | Enable colored output |
|
||||
| `buffer_size` | int | 100 | Internal buffer size |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
|
||||
**Target Modes:**
|
||||
- **stdout**: All output to standard output
|
||||
@@ -36,10 +34,10 @@ buffer_size = 100
|
||||
Write logs to rotating files.
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "file_out"
|
||||
type = "file"
|
||||
|
||||
[pipelines.sinks.file]
|
||||
[pipelines.plugin_sinks.config]
|
||||
directory = "./logs"
|
||||
name = "output"
|
||||
max_size_mb = 100
|
||||
@@ -74,17 +72,15 @@ flush_interval_ms = 1000
|
||||
SSE (Server-Sent Events) streaming server.
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "http_out"
|
||||
type = "http"
|
||||
|
||||
[pipelines.sinks.http]
|
||||
[pipelines.plugin_sinks.config]
|
||||
host = "0.0.0.0"
|
||||
port = 8080
|
||||
stream_path = "/stream"
|
||||
status_path = "/status"
|
||||
buffer_size = 1000
|
||||
max_connections = 100
|
||||
read_timeout_ms = 10000
|
||||
write_timeout_ms = 10000
|
||||
```
|
||||
|
||||
@@ -97,34 +93,20 @@ write_timeout_ms = 10000
|
||||
| `stream_path` | string | "/stream" | SSE stream endpoint |
|
||||
| `status_path` | string | "/status" | Status endpoint |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `max_connections` | int | 100 | Maximum concurrent clients |
|
||||
| `read_timeout_ms` | int | 10000 | Read timeout |
|
||||
| `write_timeout_ms` | int | 10000 | Write timeout |
|
||||
|
||||
**Heartbeat Configuration:**
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http.heartbeat]
|
||||
enabled = true
|
||||
interval_ms = 30000
|
||||
include_timestamp = true
|
||||
include_stats = false
|
||||
format = "comment" # comment|event|json
|
||||
```
|
||||
|
||||
### TCP Sink
|
||||
|
||||
TCP streaming server for debugging.
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "tcp_out"
|
||||
type = "tcp"
|
||||
|
||||
[pipelines.sinks.tcp]
|
||||
[pipelines.plugin_sinks.config]
|
||||
host = "0.0.0.0"
|
||||
port = 9090
|
||||
buffer_size = 1000
|
||||
max_connections = 100
|
||||
keep_alive = true
|
||||
keep_alive_period_ms = 30000
|
||||
```
|
||||
@@ -136,132 +118,21 @@ keep_alive_period_ms = 30000
|
||||
| `host` | string | "0.0.0.0" | Bind address |
|
||||
| `port` | int | Required | Listen port |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `max_connections` | int | 100 | Maximum concurrent clients |
|
||||
| `keep_alive` | bool | true | Enable TCP keep-alive |
|
||||
| `keep_alive_period_ms` | int | 30000 | Keep-alive interval |
|
||||
| `write_timeout_ms` | int | 10000 | Write timeout |
|
||||
|
||||
**Note:** TCP Sink has no authentication support (debugging only).
|
||||
|
||||
### HTTP Client Sink
|
||||
|
||||
Forward logs to remote HTTP endpoints.
|
||||
### Null Sink
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "http_client"
|
||||
|
||||
[pipelines.sinks.http_client]
|
||||
url = "https://logs.example.com/ingest"
|
||||
buffer_size = 1000
|
||||
batch_size = 100
|
||||
batch_delay_ms = 1000
|
||||
timeout_seconds = 30
|
||||
max_retries = 3
|
||||
retry_delay_ms = 1000
|
||||
retry_backoff = 2.0
|
||||
insecure_skip_verify = false
|
||||
[[pipelines.plugin_sinks]]
|
||||
id = "null_out"
|
||||
type = "null"
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
## Buffer Management
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `url` | string | Required | Target URL |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `batch_size` | int | 100 | Logs per request |
|
||||
| `batch_delay_ms` | int | 1000 | Max wait before sending |
|
||||
| `timeout_seconds` | int | 30 | Request timeout |
|
||||
| `max_retries` | int | 3 | Retry attempts |
|
||||
| `retry_delay_ms` | int | 1000 | Initial retry delay |
|
||||
| `retry_backoff` | float | 2.0 | Exponential backoff multiplier |
|
||||
| `insecure_skip_verify` | bool | false | Skip TLS verification |
|
||||
|
||||
### TCP Client Sink
|
||||
|
||||
Forward logs to remote TCP servers.
|
||||
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "tcp_client"
|
||||
|
||||
[pipelines.sinks.tcp_client]
|
||||
host = "logs.example.com"
|
||||
port = 9090
|
||||
buffer_size = 1000
|
||||
dial_timeout = 10
|
||||
write_timeout = 30
|
||||
read_timeout = 10
|
||||
keep_alive = 30
|
||||
reconnect_delay_ms = 1000
|
||||
max_reconnect_delay_ms = 30000
|
||||
reconnect_backoff = 1.5
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `host` | string | Required | Target host |
|
||||
| `port` | int | Required | Target port |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `dial_timeout` | int | 10 | Connection timeout (seconds) |
|
||||
| `write_timeout` | int | 30 | Write timeout (seconds) |
|
||||
| `read_timeout` | int | 10 | Read timeout (seconds) |
|
||||
| `keep_alive` | int | 30 | TCP keep-alive (seconds) |
|
||||
| `reconnect_delay_ms` | int | 1000 | Initial reconnect delay |
|
||||
| `max_reconnect_delay_ms` | int | 30000 | Maximum reconnect delay |
|
||||
| `reconnect_backoff` | float | 1.5 | Backoff multiplier |
|
||||
|
||||
## Network Sink Features
|
||||
|
||||
### Network Rate Limiting
|
||||
|
||||
Available for HTTP and TCP sinks:
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http.net_limit]
|
||||
enabled = true
|
||||
max_connections_per_ip = 10
|
||||
max_connections_total = 100
|
||||
ip_whitelist = ["192.168.1.0/24"]
|
||||
ip_blacklist = ["10.0.0.0/8"]
|
||||
```
|
||||
|
||||
### TLS Configuration (HTTP Only)
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http.tls]
|
||||
enabled = true
|
||||
cert_file = "/path/to/cert.pem"
|
||||
key_file = "/path/to/key.pem"
|
||||
ca_file = "/path/to/ca.pem"
|
||||
min_version = "TLS1.2"
|
||||
client_auth = false
|
||||
```
|
||||
|
||||
HTTP Client TLS:
|
||||
|
||||
```toml
|
||||
[pipelines.sinks.http_client.tls]
|
||||
enabled = true
|
||||
server_ca_file = "/path/to/ca.pem" # For server verification
|
||||
server_name = "logs.example.com"
|
||||
insecure_skip_verify = false
|
||||
client_cert_file = "/path/to/client.pem" # For mTLS
|
||||
client_key_file = "/path/to/client.key" # For mTLS
|
||||
```
|
||||
|
||||
## Sink Chaining
|
||||
|
||||
Designed connection patterns:
|
||||
|
||||
### Log Aggregation
|
||||
- **HTTP Client Sink → HTTP Source**: HTTP/HTTPS (optional mTLS for HTTPS)
|
||||
- **TCP Client Sink → TCP Source**: Raw TCP
|
||||
|
||||
### Live Monitoring
|
||||
- **HTTP Sink**: Browser-based SSE streaming
|
||||
- **TCP Sink**: Debug interface (telnet/netcat)
|
||||
- Full input buffer: entry dropped for that sink only (counted per pipeline as `total_dropped_by_sink`)"
|
||||
|
||||
## Sink Statistics
|
||||
|
||||
@@ -270,4 +141,4 @@ All sinks track:
|
||||
- Active connections
|
||||
- Failed sends
|
||||
- Retry attempts
|
||||
- Last processed timestamp
|
||||
- Last processed timestamp
|
||||
|
||||
+40
-110
@@ -6,31 +6,29 @@ LogWisp sources monitor various inputs and generate log entries for pipeline pro
|
||||
|
||||
### Directory Source
|
||||
|
||||
Monitors a directory for log files matching a pattern.
|
||||
Monitors a directory for log files matching a pattern. (type: `file`)
|
||||
|
||||
```toml
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
|
||||
[pipelines.sources.directory]
|
||||
path = "/var/log/myapp"
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "file_in"
|
||||
type = "file"
|
||||
[pipelines.plugin_sources.config]
|
||||
directory = "/var/log/myapp"
|
||||
pattern = "*.log" # Glob pattern
|
||||
check_interval_ms = 100 # Poll interval
|
||||
recursive = false # Scan subdirectories
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `path` | string | Required | Directory to monitor |
|
||||
| `directory` | string | Required | Directory to monitor |
|
||||
| `pattern` | string | "*" | File pattern (glob) |
|
||||
| `check_interval_ms` | int | 100 | File check interval in milliseconds |
|
||||
| `recursive` | bool | false | Include subdirectories |
|
||||
|
||||
**Features:**
|
||||
- Automatic file rotation detection
|
||||
- Position tracking (resume after restart)
|
||||
- Automatic rotation detection (inode + size tracking)
|
||||
- In-memory position tracking; on restart, monitoring resumes from the current end of each file (offsets are not persisted)
|
||||
- Concurrent file monitoring
|
||||
- Pattern-based file selection
|
||||
|
||||
@@ -39,10 +37,10 @@ recursive = false # Scan subdirectories
|
||||
Reads log entries from standard input.
|
||||
|
||||
```toml
|
||||
[[pipelines.sources]]
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "console_in"
|
||||
type = "console"
|
||||
|
||||
[pipelines.sources.stdin]
|
||||
[pipelines.plugin_sources.config]
|
||||
buffer_size = 1000
|
||||
```
|
||||
|
||||
@@ -57,107 +55,30 @@ buffer_size = 1000
|
||||
- Automatic level detection
|
||||
- Non-blocking reads
|
||||
|
||||
### HTTP Source
|
||||
|
||||
REST endpoint for log ingestion.
|
||||
### Random Source
|
||||
|
||||
```toml
|
||||
[[pipelines.sources]]
|
||||
type = "http"
|
||||
|
||||
[pipelines.sources.http]
|
||||
host = "0.0.0.0"
|
||||
port = 8081
|
||||
ingest_path = "/ingest"
|
||||
buffer_size = 1000
|
||||
max_body_size = 1048576 # 1MB
|
||||
read_timeout_ms = 10000
|
||||
write_timeout_ms = 10000
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "random_in"
|
||||
type = "random"
|
||||
[pipelines.plugin_sources.config]
|
||||
interval_ms = 500
|
||||
jitter_ms = 0
|
||||
format = "txt"
|
||||
length = 20
|
||||
special = false
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `host` | string | "0.0.0.0" | Bind address |
|
||||
| `port` | int | Required | Listen port |
|
||||
| `ingest_path` | string | "/ingest" | Ingestion endpoint path |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `max_body_size` | int | 1048576 | Maximum request body size |
|
||||
| `read_timeout_ms` | int | 10000 | Read timeout |
|
||||
| `write_timeout_ms` | int | 10000 | Write timeout |
|
||||
|
||||
**Input Formats:**
|
||||
- Single JSON object
|
||||
- JSON array
|
||||
- Newline-delimited JSON (NDJSON)
|
||||
- Plain text (one entry per line)
|
||||
|
||||
### TCP Source
|
||||
|
||||
Raw TCP socket listener for log ingestion.
|
||||
|
||||
```toml
|
||||
[[pipelines.sources]]
|
||||
type = "tcp"
|
||||
|
||||
[pipelines.sources.tcp]
|
||||
host = "0.0.0.0"
|
||||
port = 9091
|
||||
buffer_size = 1000
|
||||
read_timeout_ms = 10000
|
||||
keep_alive = true
|
||||
keep_alive_period_ms = 30000
|
||||
```
|
||||
|
||||
**Configuration Options:**
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `host` | string | "0.0.0.0" | Bind address |
|
||||
| `port` | int | Required | Listen port |
|
||||
| `buffer_size` | int | 1000 | Internal buffer size |
|
||||
| `read_timeout_ms` | int | 10000 | Read timeout |
|
||||
| `keep_alive` | bool | true | Enable TCP keep-alive |
|
||||
| `keep_alive_period_ms` | int | 30000 | Keep-alive interval |
|
||||
|
||||
**Protocol:**
|
||||
- Newline-delimited JSON
|
||||
- One log entry per line
|
||||
- UTF-8 encoding
|
||||
|
||||
## Network Source Features
|
||||
|
||||
### Network Rate Limiting
|
||||
|
||||
Available for HTTP and TCP sources:
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.net_limit]
|
||||
enabled = true
|
||||
max_connections_per_ip = 10
|
||||
max_connections_total = 100
|
||||
requests_per_second = 100.0
|
||||
burst_size = 200
|
||||
response_code = 429
|
||||
response_message = "Rate limit exceeded"
|
||||
ip_whitelist = ["192.168.1.0/24"]
|
||||
ip_blacklist = ["10.0.0.0/8"]
|
||||
```
|
||||
|
||||
### TLS Configuration (HTTP Only)
|
||||
|
||||
```toml
|
||||
[pipelines.sources.http.tls]
|
||||
enabled = true
|
||||
cert_file = "/path/to/cert.pem"
|
||||
key_file = "/path/to/key.pem"
|
||||
min_version = "TLS1.2"
|
||||
client_auth = true
|
||||
client_ca_file = "/path/to/client-ca.pem"
|
||||
verify_client_cert = true
|
||||
```
|
||||
|
||||
| `interval_ms` | int | 500 | Generation interval |
|
||||
| `jitter_ms` | int | 0 | Random jitter interval |
|
||||
| `format` | string | "txt" | "txt", "json", "raw" |
|
||||
| `length` | int | 20 | Log length |
|
||||
| `special` | bool | false | Include special characters |
|
||||
|
||||
## Source Statistics
|
||||
|
||||
All sources track:
|
||||
@@ -168,10 +89,19 @@ All sources track:
|
||||
- Active connections (network sources)
|
||||
- Source-specific metrics
|
||||
|
||||
### Null Source
|
||||
|
||||
```toml
|
||||
[[pipelines.plugin_sources]]
|
||||
id = "null_in"
|
||||
type = "null"
|
||||
[pipelines.plugin_sources.config]
|
||||
```
|
||||
|
||||
## Buffer Management
|
||||
|
||||
Each source maintains internal buffers:
|
||||
- Default size: 1000 entries
|
||||
- Drop policy when full
|
||||
- Configurable per source
|
||||
- Non-blocking writes
|
||||
- Non-blocking writes
|
||||
|
||||
Reference in New Issue
Block a user