v0.2.0 restructured to pipeline architecture, dirty
This commit is contained in:
@ -1,75 +1,26 @@
|
||||
# LogWisp Documentation
|
||||
|
||||
Welcome to the LogWisp documentation. This guide covers all aspects of installing, configuring, and using LogWisp for multi-stream log monitoring.
|
||||
Documentation covers installation, configuration, and usage of LogWisp's pipeline-based log monitoring system.
|
||||
|
||||
## 📚 Documentation Index
|
||||
|
||||
### Getting Started
|
||||
- **[Installation Guide](installation.md)** - How to install LogWisp on various platforms
|
||||
- **[Quick Start](quickstart.md)** - Get up and running in 5 minutes
|
||||
- **[Architecture Overview](architecture.md)** - System design and components
|
||||
- **[Installation Guide](installation.md)** - Platform-specific installation
|
||||
- **[Quick Start](quickstart.md)** - Get running in 5 minutes
|
||||
- **[Architecture Overview](architecture.md)** - Pipeline design
|
||||
|
||||
### Configuration
|
||||
- **[Configuration Guide](configuration.md)** - Complete configuration reference
|
||||
- **[Environment Variables](environment.md)** - Environment variable reference
|
||||
- **[Command Line Options](cli.md)** - CLI flags and parameters
|
||||
- **[Configuration Guide](configuration.md)** - Complete reference
|
||||
- **[Environment Variables](environment.md)** - Container configuration
|
||||
- **[Command Line Options](cli.md)** - CLI reference
|
||||
- **[Sample Configurations](../config/)** - Default & Minimal Config
|
||||
|
||||
### Features
|
||||
- **[Filters Guide](filters.md)** - Pattern-based log filtering
|
||||
- **[Rate Limiting](ratelimiting.md)** - Request and connection limiting
|
||||
- **[Router Mode](router.md)** - Path-based multi-stream routing
|
||||
- **[Authentication](authentication.md)** - Securing your log streams *(planned)*
|
||||
|
||||
### Operations
|
||||
- **[Monitoring & Status](monitoring.md)** - Health checks and statistics
|
||||
- **[Performance Tuning](performance.md)** - Optimization guidelines
|
||||
- **[Troubleshooting](troubleshooting.md)** - Common issues and solutions
|
||||
|
||||
### Advanced Topics
|
||||
- **[Security Best Practices](security.md)** - Hardening your deployment
|
||||
- **[Integration Examples](integrations.md)** - Working with other tools
|
||||
- **[Development Guide](development.md)** - Contributing to LogWisp
|
||||
|
||||
## 🚀 Quick Links
|
||||
|
||||
- **[Example Configurations](examples/)** - Ready-to-use config templates
|
||||
- **[API Reference](api.md)** - SSE/TCP protocol documentation
|
||||
- **[Changelog](../CHANGELOG.md)** - Version history and updates
|
||||
|
||||
## 💡 Common Use Cases
|
||||
|
||||
### Single Application Monitoring
|
||||
Monitor logs from one application with basic filtering:
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "myapp"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/myapp", pattern = "*.log" }]
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
```
|
||||
|
||||
### Multi-Service Architecture
|
||||
Monitor multiple services with different configurations:
|
||||
```bash
|
||||
logwisp --router --config /etc/logwisp/services.toml
|
||||
```
|
||||
|
||||
### High-Security Environments
|
||||
Enable authentication and rate limiting:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 10.0
|
||||
max_connections_per_ip = 3
|
||||
```
|
||||
|
||||
## 🔍 Finding Help
|
||||
|
||||
- **GitHub Issues**: [Report bugs or request features](https://github.com/logwisp/logwisp/issues)
|
||||
- **Discussions**: [Ask questions and share ideas](https://github.com/logwisp/logwisp/discussions)
|
||||
- **Examples**: Check the [examples directory](examples/) for common scenarios
|
||||
- **[Status Monitoring](status.md)** - Health checks
|
||||
- **[Filters Guide](filters.md)** - Pattern-based filtering
|
||||
- **[Rate Limiting](ratelimiting.md)** - Connection protection
|
||||
- **[Router Mode](router.md)** - Multi-pipeline routing
|
||||
- **[Authentication](authentication.md)** - Access control *(planned)*
|
||||
|
||||
## 📝 License
|
||||
|
||||
|
||||
304
doc/architecture.md
Normal file
304
doc/architecture.md
Normal file
@ -0,0 +1,304 @@
|
||||
# Architecture Overview
|
||||
|
||||
LogWisp implements a flexible pipeline architecture for real-time log processing and streaming.
|
||||
|
||||
## Core Architecture
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────────────────────┐
|
||||
│ LogWisp Service │
|
||||
├─────────────────────────────────────────────────────────────────────────┤
|
||||
│ │
|
||||
│ ┌─────────────────────────── Pipeline 1 ───────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ │ Sources Filters Sinks │ │
|
||||
│ │ ┌──────┐ ┌────────┐ ┌──────┐ │ │
|
||||
│ │ │ Dir │──┐ │Include │ ┌────│ HTTP │←── Client 1 │ │
|
||||
│ │ └──────┘ │ │ ERROR │ │ └──────┘ │ │
|
||||
│ │ ├────▶│ WARN │────▶├────┌──────┐ │ │
|
||||
│ │ ┌──────┐ │ └────────┘ │ │ File │ │ │
|
||||
│ │ │ File │──┘ ▼ │ └──────┘ │ │
|
||||
│ │ └──────┘ ┌────────┐ │ ┌──────┐ │ │
|
||||
│ │ │Exclude │ └────│ TCP │←── Client 2 │ │
|
||||
│ │ │ DEBUG │ └──────┘ │ │
|
||||
│ │ └────────┘ │ │
|
||||
│ └──────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────── Pipeline 2 ───────────────────────────┐ │
|
||||
│ │ │ │
|
||||
│ │ ┌──────┐ ┌──────┐ │ │
|
||||
│ │ │Stdin │────────────────────────────│Stdout│ │ │
|
||||
│ │ └──────┘ (No Filters) └──────┘ │ │
|
||||
│ └──────────────────────────────────────────────────────────────────┘ │
|
||||
│ │
|
||||
│ ┌─────────────────────────── Pipeline N ───────────────────────────┐ │
|
||||
│ │ Multiple Sources → Filter Chain → Multiple Sinks │ │
|
||||
│ └──────────────────────────────────────────────────────────────────┘ │
|
||||
└─────────────────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Data Flow
|
||||
|
||||
```
|
||||
Log Entry Flow:
|
||||
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ File │ │ Parse │ │ Filter │ │ Format │
|
||||
│ Watcher │────▶│ Entry │────▶│ Chain │────▶│ Send │
|
||||
└─────────┘ └─────────┘ └─────────┘ └─────────┘
|
||||
│ │ │ │
|
||||
▼ ▼ ▼ ▼
|
||||
Detect Extract Include/ Deliver to
|
||||
Changes Timestamp Exclude Clients
|
||||
& Level Patterns
|
||||
|
||||
|
||||
Entry Processing:
|
||||
|
||||
1. Source Detection 2. Entry Creation 3. Filter Application
|
||||
┌──────────┐ ┌────────────┐ ┌─────────────┐
|
||||
│New Entry │ │ Timestamp │ │ Filter 1 │
|
||||
│Detected │──────────▶│ Level │────────▶│ Include? │
|
||||
└──────────┘ │ Message │ └──────┬──────┘
|
||||
└────────────┘ │
|
||||
▼
|
||||
4. Sink Distribution ┌─────────────┐
|
||||
┌──────────┐ │ Filter 2 │
|
||||
│ HTTP │◀───┐ │ Exclude? │
|
||||
└──────────┘ │ └──────┬──────┘
|
||||
┌──────────┐ │ │
|
||||
│ TCP │◀───┼────────── Entry ◀──────────────────┘
|
||||
└──────────┘ │ (if passed)
|
||||
┌──────────┐ │
|
||||
│ File │◀───┘
|
||||
└──────────┘
|
||||
```
|
||||
|
||||
## Component Details
|
||||
|
||||
### Sources
|
||||
|
||||
Sources monitor inputs and generate log entries:
|
||||
|
||||
```
|
||||
Directory Source:
|
||||
┌─────────────────────────────────┐
|
||||
│ Directory Monitor │
|
||||
├─────────────────────────────────┤
|
||||
│ • Pattern Matching (*.log) │
|
||||
│ • File Rotation Detection │
|
||||
│ • Position Tracking │
|
||||
│ • Concurrent File Watching │
|
||||
└─────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────┐
|
||||
│ File Watcher │ (per file)
|
||||
├──────────────┤
|
||||
│ • Read New │
|
||||
│ • Track Pos │
|
||||
│ • Detect Rot │
|
||||
└──────────────┘
|
||||
```
|
||||
|
||||
### Filters
|
||||
|
||||
Filters process entries through pattern matching:
|
||||
|
||||
```
|
||||
Filter Chain:
|
||||
┌─────────────┐
|
||||
Entry ──────────▶│ Filter 1 │
|
||||
│ (Include) │
|
||||
└──────┬──────┘
|
||||
│ Pass?
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ Filter 2 │
|
||||
│ (Exclude) │
|
||||
└──────┬──────┘
|
||||
│ Pass?
|
||||
▼
|
||||
┌─────────────┐
|
||||
│ Filter N │
|
||||
└──────┬──────┘
|
||||
│
|
||||
▼
|
||||
To Sinks
|
||||
```
|
||||
|
||||
### Sinks
|
||||
|
||||
Sinks deliver processed entries to destinations:
|
||||
|
||||
```
|
||||
HTTP Sink (SSE):
|
||||
┌───────────────────────────────────┐
|
||||
│ HTTP Server │
|
||||
├───────────────────────────────────┤
|
||||
│ ┌─────────┐ ┌─────────┐ │
|
||||
│ │ Stream │ │ Status │ │
|
||||
│ │Endpoint │ │Endpoint │ │
|
||||
│ └────┬────┘ └────┬────┘ │
|
||||
│ │ │ │
|
||||
│ ┌────▼──────────────▼────┐ │
|
||||
│ │ Connection Manager │ │
|
||||
│ ├────────────────────────┤ │
|
||||
│ │ • Rate Limiting │ │
|
||||
│ │ • Heartbeat │ │
|
||||
│ │ • Buffer Management │ │
|
||||
│ └────────────────────────┘ │
|
||||
└───────────────────────────────────┘
|
||||
|
||||
TCP Sink:
|
||||
┌───────────────────────────────────┐
|
||||
│ TCP Server │
|
||||
├───────────────────────────────────┤
|
||||
│ ┌────────────────────────┐ │
|
||||
│ │ gnet Event Loop │ │
|
||||
│ ├────────────────────────┤ │
|
||||
│ │ • Async I/O │ │
|
||||
│ │ • Connection Pool │ │
|
||||
│ │ • Rate Limiting │ │
|
||||
│ └────────────────────────┘ │
|
||||
└───────────────────────────────────┘
|
||||
```
|
||||
|
||||
## Router Mode
|
||||
|
||||
In router mode, multiple pipelines share HTTP ports:
|
||||
|
||||
```
|
||||
Router Architecture:
|
||||
┌─────────────────┐
|
||||
│ HTTP Router │
|
||||
│ Port 8080 │
|
||||
└────────┬────────┘
|
||||
│
|
||||
┌────────────────────┼────────────────────┐
|
||||
│ │ │
|
||||
/app/stream /db/stream /sys/stream
|
||||
│ │ │
|
||||
┌────▼────┐ ┌────▼────┐ ┌────▼────┐
|
||||
│Pipeline │ │Pipeline │ │Pipeline │
|
||||
│ "app" │ │ "db" │ │ "sys" │
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
|
||||
Path Routing:
|
||||
Client Request ──▶ Router ──▶ Parse Path ──▶ Find Pipeline ──▶ Route
|
||||
│
|
||||
▼
|
||||
Extract Pipeline Name
|
||||
from /pipeline/endpoint
|
||||
```
|
||||
|
||||
## Memory Management
|
||||
|
||||
```
|
||||
Buffer Flow:
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│ Source │ │ Pipeline │ │ Sink │
|
||||
│ Buffer │────▶│ Buffer │────▶│ Buffer │
|
||||
│ (1000) │ │ (chan) │ │ (1000) │
|
||||
└──────────┘ └──────────┘ └──────────┘
|
||||
│ │ │
|
||||
▼ ▼ ▼
|
||||
Drop if full Backpressure Drop if full
|
||||
(counted) (blocking) (counted)
|
||||
```
|
||||
|
||||
## Rate Limiting
|
||||
|
||||
```
|
||||
Token Bucket Algorithm:
|
||||
┌─────────────────────────────┐
|
||||
│ Token Bucket │
|
||||
├─────────────────────────────┤
|
||||
│ Capacity: burst_size │
|
||||
│ Refill: requests_per_second │
|
||||
│ │
|
||||
│ ┌─────────────────────┐ │
|
||||
│ │ ● ● ● ● ● ● ○ ○ ○ ○ │ │
|
||||
│ └─────────────────────┘ │
|
||||
│ 6/10 tokens available │
|
||||
└─────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
Request arrives
|
||||
│
|
||||
▼
|
||||
Token available? ──No──▶ Reject (429)
|
||||
│
|
||||
Yes
|
||||
▼
|
||||
Consume token ──▶ Allow request
|
||||
```
|
||||
|
||||
## Concurrency Model
|
||||
|
||||
```
|
||||
Goroutine Structure:
|
||||
|
||||
Main ────┬──── Pipeline 1 ────┬──── Source Reader 1
|
||||
│ ├──── Source Reader 2
|
||||
│ ├──── Filter Processor
|
||||
│ ├──── HTTP Server
|
||||
│ └──── TCP Server
|
||||
│
|
||||
├──── Pipeline 2 ────┬──── Source Reader
|
||||
│ └──── Sink Writers
|
||||
│
|
||||
└──── HTTP Router (if enabled)
|
||||
|
||||
Channel Communication:
|
||||
Source ──chan──▶ Filter ──chan──▶ Sink
|
||||
│ │
|
||||
└── Non-blocking send ────────────┘
|
||||
(drop & count if full)
|
||||
```
|
||||
|
||||
## Configuration Loading
|
||||
|
||||
```
|
||||
Priority Order:
|
||||
1. CLI Flags ─────────┐
|
||||
2. Environment Vars ──┼──▶ Merge ──▶ Final Config
|
||||
3. Config File ───────┤
|
||||
4. Defaults ──────────┘
|
||||
|
||||
Example:
|
||||
CLI: --log-level debug
|
||||
Env: LOGWISP_PIPELINES_0_NAME=app
|
||||
File: pipelines.toml
|
||||
Default: buffer_size = 1000
|
||||
```
|
||||
|
||||
## Security Architecture
|
||||
|
||||
```
|
||||
Security Layers:
|
||||
|
||||
┌─────────────────────────────────────┐
|
||||
│ Network Layer │
|
||||
├─────────────────────────────────────┤
|
||||
│ • Rate Limiting (per IP/global) │
|
||||
│ • Connection Limits │
|
||||
│ • TLS/SSL (planned) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
┌──────────────▼──────────────────────┐
|
||||
│ Authentication Layer │
|
||||
├─────────────────────────────────────┤
|
||||
│ • Basic Auth (planned) │
|
||||
│ • Bearer Tokens (planned) │
|
||||
│ • IP Whitelisting (planned) │
|
||||
└──────────────┬──────────────────────┘
|
||||
│
|
||||
┌──────────────▼──────────────────────┐
|
||||
│ Application Layer │
|
||||
├─────────────────────────────────────┤
|
||||
│ • Input Validation │
|
||||
│ • Path Traversal Prevention │
|
||||
│ • Resource Limits │
|
||||
└─────────────────────────────────────┘
|
||||
```
|
||||
119
doc/cli.md
119
doc/cli.md
@ -1,6 +1,6 @@
|
||||
# Command Line Interface
|
||||
|
||||
LogWisp provides a comprehensive set of command-line options for controlling its behavior without modifying configuration files.
|
||||
LogWisp CLI options for controlling behavior without modifying configuration files.
|
||||
|
||||
## Synopsis
|
||||
|
||||
@ -11,145 +11,92 @@ logwisp [options]
|
||||
## General Options
|
||||
|
||||
### `--config <path>`
|
||||
Specify the configuration file location.
|
||||
Configuration file location.
|
||||
- **Default**: `~/.config/logwisp.toml`
|
||||
- **Example**: `logwisp --config /etc/logwisp/production.toml`
|
||||
|
||||
### `--router`
|
||||
Enable HTTP router mode for path-based routing of multiple streams.
|
||||
- **Default**: `false` (standalone mode)
|
||||
- **Use case**: Consolidate multiple HTTP streams on shared ports
|
||||
Enable HTTP router mode for path-based routing.
|
||||
- **Default**: `false`
|
||||
- **Example**: `logwisp --router`
|
||||
|
||||
### `--version`
|
||||
Display version information and exit.
|
||||
- **Example**: `logwisp --version`
|
||||
Display version information.
|
||||
|
||||
### `--background`
|
||||
Run LogWisp as a background process.
|
||||
- **Default**: `false` (foreground mode)
|
||||
Run as background process.
|
||||
- **Example**: `logwisp --background`
|
||||
|
||||
## Logging Options
|
||||
|
||||
These options override the corresponding configuration file settings.
|
||||
Override configuration file settings:
|
||||
|
||||
### `--log-output <mode>`
|
||||
Control where LogWisp writes its own operational logs.
|
||||
LogWisp's operational log output.
|
||||
- **Values**: `file`, `stdout`, `stderr`, `both`, `none`
|
||||
- **Default**: Configured value or `stderr`
|
||||
- **Example**: `logwisp --log-output both`
|
||||
|
||||
#### Output Modes:
|
||||
- `file`: Write logs only to files
|
||||
- `stdout`: Write logs only to standard output
|
||||
- `stderr`: Write logs only to standard error
|
||||
- `both`: Write logs to both files and console
|
||||
- `none`: Disable logging (⚠️ SECURITY: Not recommended)
|
||||
|
||||
### `--log-level <level>`
|
||||
Set the minimum log level for LogWisp's operational logs.
|
||||
Minimum log level.
|
||||
- **Values**: `debug`, `info`, `warn`, `error`
|
||||
- **Default**: Configured value or `info`
|
||||
- **Example**: `logwisp --log-level debug`
|
||||
|
||||
### `--log-file <path>`
|
||||
Specify the log file path when using file output.
|
||||
- **Default**: Configured value or `./logs/logwisp.log`
|
||||
- **Example**: `logwisp --log-output file --log-file /var/log/logwisp/app.log`
|
||||
Log file path (with file output).
|
||||
- **Example**: `logwisp --log-file /var/log/logwisp/app.log`
|
||||
|
||||
### `--log-dir <directory>`
|
||||
Specify the log directory when using file output.
|
||||
- **Default**: Configured value or `./logs`
|
||||
- **Example**: `logwisp --log-output file --log-dir /var/log/logwisp`
|
||||
Log directory (with file output).
|
||||
- **Example**: `logwisp --log-dir /var/log/logwisp`
|
||||
|
||||
### `--log-console <target>`
|
||||
Control console output destination when using `stdout`, `stderr`, or `both` modes.
|
||||
Console output destination.
|
||||
- **Values**: `stdout`, `stderr`, `split`
|
||||
- **Default**: `stderr`
|
||||
- **Example**: `logwisp --log-output both --log-console split`
|
||||
|
||||
#### Console Targets:
|
||||
- `stdout`: All logs to standard output
|
||||
- `stderr`: All logs to standard error
|
||||
- `split`: INFO/DEBUG to stdout, WARN/ERROR to stderr (planned)
|
||||
- **Example**: `logwisp --log-console split`
|
||||
|
||||
## Examples
|
||||
|
||||
### Basic Usage
|
||||
```bash
|
||||
# Start with default configuration
|
||||
# Default configuration
|
||||
logwisp
|
||||
|
||||
# Use a specific configuration file
|
||||
# Specific configuration
|
||||
logwisp --config /etc/logwisp/production.toml
|
||||
```
|
||||
|
||||
### Development Mode
|
||||
### Development
|
||||
```bash
|
||||
# Enable debug logging to console
|
||||
# Debug mode
|
||||
logwisp --log-output stderr --log-level debug
|
||||
|
||||
# Debug with file output
|
||||
# With file output
|
||||
logwisp --log-output both --log-level debug --log-dir ./debug-logs
|
||||
```
|
||||
|
||||
### Production Deployment
|
||||
### Production
|
||||
```bash
|
||||
# File logging with info level
|
||||
logwisp --log-output file --log-dir /var/log/logwisp --log-level info
|
||||
# File logging
|
||||
logwisp --log-output file --log-dir /var/log/logwisp
|
||||
|
||||
# Background mode with custom config
|
||||
logwisp --background --config /etc/logwisp/prod.toml
|
||||
|
||||
# Router mode for multiple services
|
||||
logwisp --router --config /etc/logwisp/services.toml
|
||||
```
|
||||
|
||||
### Troubleshooting
|
||||
```bash
|
||||
# Maximum verbosity to stderr
|
||||
logwisp --log-output stderr --log-level debug
|
||||
|
||||
# Check version
|
||||
logwisp --version
|
||||
|
||||
# Test configuration without backgrounding
|
||||
logwisp --config test.toml --log-level debug
|
||||
# Background with router
|
||||
logwisp --background --router --config /etc/logwisp/prod.toml
|
||||
```
|
||||
|
||||
## Priority Order
|
||||
|
||||
Configuration values are applied in the following priority order (highest to lowest):
|
||||
|
||||
1. **Command-line flags** - Explicitly specified options
|
||||
2. **Environment variables** - `LOGWISP_*` prefixed variables
|
||||
3. **Configuration file** - TOML configuration
|
||||
4. **Built-in defaults** - Hardcoded fallback values
|
||||
1. **Command-line flags** (highest)
|
||||
2. **Environment variables**
|
||||
3. **Configuration file**
|
||||
4. **Built-in defaults** (lowest)
|
||||
|
||||
## Exit Codes
|
||||
|
||||
- `0`: Successful execution
|
||||
- `1`: General error (configuration, startup failure)
|
||||
- `2`: Invalid command-line arguments
|
||||
- `0`: Success
|
||||
- `1`: General error
|
||||
- `2`: Invalid arguments
|
||||
|
||||
## Signals
|
||||
|
||||
LogWisp responds to the following signals:
|
||||
|
||||
- `SIGINT` (Ctrl+C): Graceful shutdown
|
||||
- `SIGTERM`: Graceful shutdown
|
||||
- `SIGKILL`: Immediate termination (not recommended)
|
||||
|
||||
During graceful shutdown, LogWisp will:
|
||||
1. Stop accepting new connections
|
||||
2. Finish streaming to existing clients
|
||||
3. Flush all buffers
|
||||
4. Close all file handles
|
||||
5. Exit cleanly
|
||||
|
||||
## See Also
|
||||
|
||||
- [Configuration Guide](configuration.md) - Complete configuration reference
|
||||
- [Environment Variables](environment.md) - Environment variable options
|
||||
- [Router Mode](router.md) - Path-based routing details
|
||||
- `SIGTERM`: Graceful shutdown
|
||||
@ -1,10 +1,9 @@
|
||||
# Configuration Guide
|
||||
|
||||
LogWisp uses TOML format for configuration with sensible defaults for all settings.
|
||||
LogWisp uses TOML format with a flexible **source → filter → sink** pipeline architecture.
|
||||
|
||||
## Configuration File Location
|
||||
|
||||
Default search order:
|
||||
1. Command line: `--config /path/to/config.toml`
|
||||
2. Environment: `$LOGWISP_CONFIG_FILE`
|
||||
3. User config: `~/.config/logwisp.toml`
|
||||
@ -13,342 +12,281 @@ Default search order:
|
||||
## Configuration Structure
|
||||
|
||||
```toml
|
||||
# Optional: LogWisp's own logging configuration
|
||||
# Optional: LogWisp's own logging
|
||||
[logging]
|
||||
output = "stderr" # file, stdout, stderr, both, none
|
||||
level = "info" # debug, info, warn, error
|
||||
output = "stderr"
|
||||
level = "info"
|
||||
|
||||
# Required: At least one stream
|
||||
[[streams]]
|
||||
name = "default" # Unique identifier
|
||||
# Required: At least one pipeline
|
||||
[[pipelines]]
|
||||
name = "default"
|
||||
|
||||
[streams.monitor] # Required: What to monitor
|
||||
# ... monitor settings ...
|
||||
# Sources (required)
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { ... }
|
||||
|
||||
[streams.httpserver] # Optional: HTTP/SSE server
|
||||
# ... HTTP settings ...
|
||||
# Filters (optional)
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
patterns = [...]
|
||||
|
||||
[streams.tcpserver] # Optional: TCP server
|
||||
# ... TCP settings ...
|
||||
|
||||
[[streams.filters]] # Optional: Log filtering
|
||||
# ... filter settings ...
|
||||
# Sinks (required)
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { ... }
|
||||
```
|
||||
|
||||
## Logging Configuration
|
||||
|
||||
Controls LogWisp's operational logging (not the logs being monitored).
|
||||
Controls LogWisp's operational logging:
|
||||
|
||||
```toml
|
||||
[logging]
|
||||
output = "stderr" # Where to write LogWisp's logs
|
||||
level = "info" # Minimum log level
|
||||
output = "stderr" # file, stdout, stderr, both, none
|
||||
level = "info" # debug, info, warn, error
|
||||
|
||||
# File output settings (when output includes "file")
|
||||
[logging.file]
|
||||
directory = "./logs" # Log directory
|
||||
name = "logwisp" # Base filename
|
||||
max_size_mb = 100 # Rotate at this size
|
||||
max_total_size_mb = 1000 # Total size limit
|
||||
retention_hours = 168 # Keep for 7 days
|
||||
directory = "./logs"
|
||||
name = "logwisp"
|
||||
max_size_mb = 100
|
||||
max_total_size_mb = 1000
|
||||
retention_hours = 168
|
||||
|
||||
# Console output settings
|
||||
[logging.console]
|
||||
target = "stderr" # stdout, stderr, split
|
||||
format = "txt" # txt or json
|
||||
```
|
||||
|
||||
## Stream Configuration
|
||||
## Pipeline Configuration
|
||||
|
||||
Each `[[streams]]` section defines an independent log monitoring pipeline.
|
||||
Each `[[pipelines]]` section defines an independent processing pipeline.
|
||||
|
||||
### Monitor Settings
|
||||
### Sources
|
||||
|
||||
What files or directories to watch:
|
||||
Input data sources:
|
||||
|
||||
#### Directory Source
|
||||
```toml
|
||||
[streams.monitor]
|
||||
check_interval_ms = 100 # How often to check for new entries
|
||||
|
||||
# Monitor targets (at least one required)
|
||||
targets = [
|
||||
# Watch all .log files in a directory
|
||||
{ path = "/var/log/myapp", pattern = "*.log", is_file = false },
|
||||
|
||||
# Watch a specific file
|
||||
{ path = "/var/log/app.log", is_file = true },
|
||||
|
||||
# Multiple patterns
|
||||
{ path = "/logs", pattern = "app-*.log", is_file = false },
|
||||
{ path = "/logs", pattern = "error-*.txt", is_file = false }
|
||||
]
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = {
|
||||
path = "/var/log/myapp", # Directory to monitor
|
||||
pattern = "*.log", # File pattern (glob)
|
||||
check_interval_ms = 100 # Check interval (10-60000)
|
||||
}
|
||||
```
|
||||
|
||||
### HTTP Server (SSE)
|
||||
|
||||
Server-Sent Events streaming over HTTP:
|
||||
|
||||
#### File Source
|
||||
```toml
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
buffer_size = 1000 # Per-client event buffer
|
||||
stream_path = "/stream" # SSE endpoint
|
||||
status_path = "/status" # Statistics endpoint
|
||||
|
||||
# Keep-alive heartbeat
|
||||
[streams.httpserver.heartbeat]
|
||||
enabled = true
|
||||
interval_seconds = 30
|
||||
format = "comment" # "comment" or "json"
|
||||
include_timestamp = true
|
||||
include_stats = false
|
||||
|
||||
# Rate limiting (optional)
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = false
|
||||
requests_per_second = 10.0
|
||||
burst_size = 20
|
||||
limit_by = "ip" # "ip" or "global"
|
||||
response_code = 429
|
||||
response_message = "Rate limit exceeded"
|
||||
max_connections_per_ip = 5
|
||||
max_total_connections = 100
|
||||
[[pipelines.sources]]
|
||||
type = "file"
|
||||
options = {
|
||||
path = "/var/log/app.log" # Specific file
|
||||
}
|
||||
```
|
||||
|
||||
### TCP Server
|
||||
|
||||
Raw TCP streaming for high performance:
|
||||
|
||||
#### Stdin Source
|
||||
```toml
|
||||
[streams.tcpserver]
|
||||
enabled = true
|
||||
port = 9090
|
||||
buffer_size = 5000 # Larger buffer for TCP
|
||||
|
||||
# Heartbeat (always JSON format for TCP)
|
||||
[streams.tcpserver.heartbeat]
|
||||
enabled = true
|
||||
interval_seconds = 60
|
||||
include_timestamp = true
|
||||
include_stats = false
|
||||
|
||||
# Rate limiting
|
||||
[streams.tcpserver.rate_limit]
|
||||
enabled = false
|
||||
requests_per_second = 5.0
|
||||
burst_size = 10
|
||||
limit_by = "ip"
|
||||
[[pipelines.sources]]
|
||||
type = "stdin"
|
||||
options = {}
|
||||
```
|
||||
|
||||
### Filters
|
||||
|
||||
Control which log entries are streamed:
|
||||
Control which log entries pass through:
|
||||
|
||||
```toml
|
||||
# Include filter - only matching logs pass
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
logic = "or" # "or" = match any, "and" = match all
|
||||
logic = "or" # or: match any, and: match all
|
||||
patterns = [
|
||||
"ERROR",
|
||||
"WARN",
|
||||
"CRITICAL"
|
||||
"(?i)warn", # Case-insensitive
|
||||
"\\bfatal\\b" # Word boundary
|
||||
]
|
||||
|
||||
# Exclude filter - matching logs are dropped
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"DEBUG",
|
||||
"health check"
|
||||
]
|
||||
patterns = ["DEBUG", "health-check"]
|
||||
```
|
||||
|
||||
### Sinks
|
||||
|
||||
Output destinations:
|
||||
|
||||
#### HTTP Sink (SSE)
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = {
|
||||
port = 8080,
|
||||
buffer_size = 1000,
|
||||
stream_path = "/stream",
|
||||
status_path = "/status",
|
||||
|
||||
# Heartbeat
|
||||
heartbeat = {
|
||||
enabled = true,
|
||||
interval_seconds = 30,
|
||||
format = "comment", # comment or json
|
||||
include_timestamp = true
|
||||
},
|
||||
|
||||
# Rate limiting
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 10.0,
|
||||
burst_size = 20,
|
||||
limit_by = "ip", # ip or global
|
||||
max_connections_per_ip = 5
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### TCP Sink
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "tcp"
|
||||
options = {
|
||||
port = 9090,
|
||||
buffer_size = 5000,
|
||||
heartbeat = { enabled = true, interval_seconds = 60 },
|
||||
rate_limit = { enabled = true, requests_per_second = 5.0 }
|
||||
}
|
||||
```
|
||||
|
||||
#### File Sink
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "file"
|
||||
options = {
|
||||
directory = "/var/log/logwisp",
|
||||
name = "app",
|
||||
max_size_mb = 100,
|
||||
retention_hours = 168.0,
|
||||
buffer_size = 2000
|
||||
}
|
||||
```
|
||||
|
||||
#### Console Sinks
|
||||
```toml
|
||||
[[pipelines.sinks]]
|
||||
type = "stdout" # or "stderr"
|
||||
options = { buffer_size = 500 }
|
||||
```
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Minimal Configuration
|
||||
### Basic Application Monitoring
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "simple"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "./logs", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines]]
|
||||
name = "app"
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log" }
|
||||
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
```
|
||||
|
||||
### Production Web Application
|
||||
### Production with Filtering
|
||||
|
||||
```toml
|
||||
[logging]
|
||||
output = "file"
|
||||
level = "info"
|
||||
[logging.file]
|
||||
directory = "/var/log/logwisp"
|
||||
max_size_mb = 500
|
||||
retention_hours = 336 # 14 days
|
||||
|
||||
[[streams]]
|
||||
name = "webapp"
|
||||
[[pipelines]]
|
||||
name = "production"
|
||||
|
||||
[streams.monitor]
|
||||
check_interval_ms = 50
|
||||
targets = [
|
||||
{ path = "/var/log/nginx", pattern = "access.log*" },
|
||||
{ path = "/var/log/nginx", pattern = "error.log*" },
|
||||
{ path = "/var/log/myapp", pattern = "*.log" }
|
||||
]
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log", check_interval_ms = 50 }
|
||||
|
||||
# Only errors and warnings
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = [
|
||||
"\\b(ERROR|error|Error)\\b",
|
||||
"\\b(WARN|WARNING|warn|warning)\\b",
|
||||
"\\b(CRITICAL|FATAL|critical|fatal)\\b",
|
||||
"status=[4-5][0-9][0-9]" # HTTP errors
|
||||
]
|
||||
patterns = ["ERROR", "WARN", "CRITICAL"]
|
||||
|
||||
# Exclude noise
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"/health",
|
||||
"/metrics",
|
||||
"ELB-HealthChecker"
|
||||
]
|
||||
patterns = ["/health", "/metrics"]
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
buffer_size = 2000
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = {
|
||||
port = 8080,
|
||||
rate_limit = { enabled = true, requests_per_second = 25.0 }
|
||||
}
|
||||
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 25.0
|
||||
burst_size = 50
|
||||
max_connections_per_ip = 10
|
||||
[[pipelines.sinks]]
|
||||
type = "file"
|
||||
options = { directory = "/var/log/archive", name = "errors" }
|
||||
```
|
||||
|
||||
### Multi-Service with Router
|
||||
### Multi-Source Aggregation
|
||||
|
||||
```toml
|
||||
[[pipelines]]
|
||||
name = "aggregated"
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/nginx", pattern = "*.log" }
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log" }
|
||||
|
||||
[[pipelines.sources]]
|
||||
type = "stdin"
|
||||
options = {}
|
||||
|
||||
[[pipelines.sinks]]
|
||||
type = "tcp"
|
||||
options = { port = 9090 }
|
||||
```
|
||||
|
||||
### Router Mode
|
||||
|
||||
```toml
|
||||
# Run with: logwisp --router
|
||||
|
||||
# Service 1: API
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "api"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/api", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # All streams can use same port in router mode
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/api", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 } # Same port OK in router mode
|
||||
|
||||
# Service 2: Database
|
||||
[[streams]]
|
||||
name = "database"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/postgresql", pattern = "*.log" }]
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "FATAL", "deadlock", "timeout"]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines]]
|
||||
name = "web"
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/nginx", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 } # Shared port
|
||||
|
||||
# Service 3: System
|
||||
[[streams]]
|
||||
name = "system"
|
||||
[streams.monitor]
|
||||
targets = [
|
||||
{ path = "/var/log/syslog", is_file = true },
|
||||
{ path = "/var/log/auth.log", is_file = true }
|
||||
]
|
||||
[streams.tcpserver]
|
||||
enabled = true
|
||||
port = 9090
|
||||
# Access:
|
||||
# http://localhost:8080/api/stream
|
||||
# http://localhost:8080/web/stream
|
||||
# http://localhost:8080/status
|
||||
```
|
||||
|
||||
### High-Security Configuration
|
||||
|
||||
```toml
|
||||
[logging]
|
||||
output = "file"
|
||||
level = "warn" # Less verbose
|
||||
|
||||
[[streams]]
|
||||
name = "secure"
|
||||
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/secure", pattern = "*.log" }]
|
||||
|
||||
# Only security events
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = [
|
||||
"auth",
|
||||
"sudo",
|
||||
"ssh",
|
||||
"login",
|
||||
"failed",
|
||||
"denied"
|
||||
]
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8443
|
||||
|
||||
# Strict rate limiting
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 2.0
|
||||
burst_size = 3
|
||||
limit_by = "ip"
|
||||
max_connections_per_ip = 1
|
||||
response_code = 403 # Forbidden instead of 429
|
||||
|
||||
# Future: Authentication
|
||||
# [streams.auth]
|
||||
# type = "basic"
|
||||
# [streams.auth.basic_auth]
|
||||
# users_file = "/etc/logwisp/users.htpasswd"
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
### Performance Tuning
|
||||
|
||||
- **check_interval_ms**: Higher values reduce CPU usage
|
||||
- **buffer_size**: Larger buffers handle bursts better
|
||||
- **rate_limit**: Essential for public-facing streams
|
||||
|
||||
### Filter Patterns
|
||||
|
||||
- Use word boundaries: `\\berror\\b` (won't match "errorCode")
|
||||
- Case-insensitive: `(?i)error`
|
||||
- Anchors for speed: `^ERROR` faster than `ERROR`
|
||||
- Test complex patterns before deployment
|
||||
|
||||
### Resource Limits
|
||||
|
||||
- Each stream uses ~10-50MB RAM (depending on buffers)
|
||||
- CPU usage scales with check_interval and file activity
|
||||
- Network bandwidth depends on log volume and client count
|
||||
|
||||
## Validation
|
||||
|
||||
LogWisp validates configuration on startup:
|
||||
- Required fields (name, monitor targets)
|
||||
- Port conflicts between streams
|
||||
- Pattern syntax for filters
|
||||
LogWisp validates on startup:
|
||||
- Required fields (name, sources, sinks)
|
||||
- Port conflicts between pipelines
|
||||
- Pattern syntax
|
||||
- Path accessibility
|
||||
|
||||
## See Also
|
||||
|
||||
- [Environment Variables](environment.md) - Override via environment
|
||||
- [CLI Options](cli.md) - Override via command line
|
||||
- [Filter Guide](filters.md) - Advanced filtering patterns
|
||||
- [Examples](examples/) - Ready-to-use configurations
|
||||
- Rate limit values
|
||||
@ -1,275 +1,148 @@
|
||||
# Environment Variables
|
||||
|
||||
LogWisp supports comprehensive configuration through environment variables, allowing deployment without configuration files or dynamic overrides in containerized environments.
|
||||
Configure LogWisp through environment variables for containerized deployments.
|
||||
|
||||
## Naming Convention
|
||||
|
||||
Environment variables follow a structured pattern:
|
||||
- **Prefix**: `LOGWISP_`
|
||||
- **Path separator**: `_` (underscore)
|
||||
- **Array indices**: Numeric suffix (0-based)
|
||||
- **Case**: UPPERCASE
|
||||
|
||||
### Examples:
|
||||
- Config file setting: `logging.level = "debug"`
|
||||
- Environment variable: `LOGWISP_LOGGING_LEVEL=debug`
|
||||
|
||||
- Array element: `streams[0].name = "app"`
|
||||
- Environment variable: `LOGWISP_STREAMS_0_NAME=app`
|
||||
Examples:
|
||||
- `logging.level` → `LOGWISP_LOGGING_LEVEL`
|
||||
- `pipelines[0].name` → `LOGWISP_PIPELINES_0_NAME`
|
||||
|
||||
## General Variables
|
||||
|
||||
### `LOGWISP_CONFIG_FILE`
|
||||
Path to the configuration file.
|
||||
- **Default**: `~/.config/logwisp.toml`
|
||||
- **Example**: `LOGWISP_CONFIG_FILE=/etc/logwisp/config.toml`
|
||||
Configuration file path.
|
||||
```bash
|
||||
export LOGWISP_CONFIG_FILE=/etc/logwisp/config.toml
|
||||
```
|
||||
|
||||
### `LOGWISP_CONFIG_DIR`
|
||||
Directory containing configuration files.
|
||||
- **Usage**: Combined with `LOGWISP_CONFIG_FILE` for relative paths
|
||||
- **Example**:
|
||||
```bash
|
||||
export LOGWISP_CONFIG_DIR=/etc/logwisp
|
||||
export LOGWISP_CONFIG_FILE=production.toml
|
||||
# Loads: /etc/logwisp/production.toml
|
||||
```
|
||||
Configuration directory.
|
||||
```bash
|
||||
export LOGWISP_CONFIG_DIR=/etc/logwisp
|
||||
export LOGWISP_CONFIG_FILE=production.toml
|
||||
```
|
||||
|
||||
### `LOGWISP_DISABLE_STATUS_REPORTER`
|
||||
Disable the periodic status reporter.
|
||||
- **Values**: `1` (disable), `0` or unset (enable)
|
||||
- **Default**: `0` (enabled)
|
||||
- **Example**: `LOGWISP_DISABLE_STATUS_REPORTER=1`
|
||||
|
||||
### `LOGWISP_BACKGROUND`
|
||||
Internal marker for background process detection.
|
||||
- **Note**: Set automatically by `--background` flag
|
||||
- **Values**: `1` (background), unset (foreground)
|
||||
Disable periodic status reporting.
|
||||
```bash
|
||||
export LOGWISP_DISABLE_STATUS_REPORTER=1
|
||||
```
|
||||
|
||||
## Logging Variables
|
||||
|
||||
### `LOGWISP_LOGGING_OUTPUT`
|
||||
LogWisp's operational log output mode.
|
||||
- **Values**: `file`, `stdout`, `stderr`, `both`, `none`
|
||||
- **Example**: `LOGWISP_LOGGING_OUTPUT=both`
|
||||
|
||||
### `LOGWISP_LOGGING_LEVEL`
|
||||
Minimum log level for operational logs.
|
||||
- **Values**: `debug`, `info`, `warn`, `error`
|
||||
- **Example**: `LOGWISP_LOGGING_LEVEL=debug`
|
||||
|
||||
### File Logging
|
||||
```bash
|
||||
# Output mode
|
||||
LOGWISP_LOGGING_OUTPUT=both
|
||||
|
||||
# Log level
|
||||
LOGWISP_LOGGING_LEVEL=debug
|
||||
|
||||
# File logging
|
||||
LOGWISP_LOGGING_FILE_DIRECTORY=/var/log/logwisp
|
||||
LOGWISP_LOGGING_FILE_NAME=logwisp
|
||||
LOGWISP_LOGGING_FILE_MAX_SIZE_MB=100
|
||||
LOGWISP_LOGGING_FILE_MAX_TOTAL_SIZE_MB=1000
|
||||
LOGWISP_LOGGING_FILE_RETENTION_HOURS=168 # 7 days
|
||||
LOGWISP_LOGGING_FILE_RETENTION_HOURS=168
|
||||
|
||||
# Console logging
|
||||
LOGWISP_LOGGING_CONSOLE_TARGET=stderr
|
||||
LOGWISP_LOGGING_CONSOLE_FORMAT=json
|
||||
```
|
||||
|
||||
### Console Logging
|
||||
## Pipeline Configuration
|
||||
|
||||
### Basic Pipeline
|
||||
```bash
|
||||
LOGWISP_LOGGING_CONSOLE_TARGET=stderr # stdout, stderr, split
|
||||
LOGWISP_LOGGING_CONSOLE_FORMAT=txt # txt, json
|
||||
```
|
||||
# Pipeline name
|
||||
LOGWISP_PIPELINES_0_NAME=app
|
||||
|
||||
## Stream Configuration
|
||||
# Source configuration
|
||||
LOGWISP_PIPELINES_0_SOURCES_0_TYPE=directory
|
||||
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATH=/var/log/app
|
||||
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATTERN="*.log"
|
||||
LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_CHECK_INTERVAL_MS=100
|
||||
|
||||
Streams are configured using array indices (0-based).
|
||||
|
||||
### Basic Stream Settings
|
||||
```bash
|
||||
# First stream (index 0)
|
||||
LOGWISP_STREAMS_0_NAME=app
|
||||
LOGWISP_STREAMS_0_MONITOR_CHECK_INTERVAL_MS=100
|
||||
|
||||
# Second stream (index 1)
|
||||
LOGWISP_STREAMS_1_NAME=system
|
||||
LOGWISP_STREAMS_1_MONITOR_CHECK_INTERVAL_MS=1000
|
||||
```
|
||||
|
||||
### Monitor Targets
|
||||
```bash
|
||||
# Single file target
|
||||
LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/app.log
|
||||
LOGWISP_STREAMS_0_MONITOR_TARGETS_0_IS_FILE=true
|
||||
|
||||
# Directory with pattern
|
||||
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_PATH=/var/log/myapp
|
||||
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_PATTERN="*.log"
|
||||
LOGWISP_STREAMS_0_MONITOR_TARGETS_1_IS_FILE=false
|
||||
# Sink configuration
|
||||
LOGWISP_PIPELINES_0_SINKS_0_TYPE=http
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_PORT=8080
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_BUFFER_SIZE=1000
|
||||
```
|
||||
|
||||
### Filters
|
||||
```bash
|
||||
# Include filter
|
||||
LOGWISP_STREAMS_0_FILTERS_0_TYPE=include
|
||||
LOGWISP_STREAMS_0_FILTERS_0_LOGIC=or
|
||||
LOGWISP_STREAMS_0_FILTERS_0_PATTERNS='["ERROR","WARN","CRITICAL"]'
|
||||
LOGWISP_PIPELINES_0_FILTERS_0_TYPE=include
|
||||
LOGWISP_PIPELINES_0_FILTERS_0_LOGIC=or
|
||||
LOGWISP_PIPELINES_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
|
||||
|
||||
# Exclude filter
|
||||
LOGWISP_STREAMS_0_FILTERS_1_TYPE=exclude
|
||||
LOGWISP_STREAMS_0_FILTERS_1_PATTERNS='["DEBUG","TRACE"]'
|
||||
LOGWISP_PIPELINES_0_FILTERS_1_TYPE=exclude
|
||||
LOGWISP_PIPELINES_0_FILTERS_1_PATTERNS='["DEBUG"]'
|
||||
```
|
||||
|
||||
### HTTP Server
|
||||
### HTTP Sink Options
|
||||
```bash
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_ENABLED=true
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_BUFFER_SIZE=1000
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_STREAM_PATH=/stream
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_STATUS_PATH=/status
|
||||
# Basic
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_STREAM_PATH=/stream
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_STATUS_PATH=/status
|
||||
|
||||
# Heartbeat
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_ENABLED=true
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INTERVAL_SECONDS=30
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_FORMAT=comment
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INCLUDE_TIMESTAMP=true
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_HEARTBEAT_INCLUDE_STATS=false
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_ENABLED=true
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_INTERVAL_SECONDS=30
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_HEARTBEAT_FORMAT=comment
|
||||
|
||||
# Rate Limiting
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_ENABLED=true
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=10.0
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_BURST_SIZE=20
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_LIMIT_BY=ip
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_MAX_CONNECTIONS_PER_IP=5
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_ENABLED=true
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_REQUESTS_PER_SECOND=10.0
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_BURST_SIZE=20
|
||||
LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_LIMIT_BY=ip
|
||||
```
|
||||
|
||||
### TCP Server
|
||||
```bash
|
||||
LOGWISP_STREAMS_0_TCPSERVER_ENABLED=true
|
||||
LOGWISP_STREAMS_0_TCPSERVER_PORT=9090
|
||||
LOGWISP_STREAMS_0_TCPSERVER_BUFFER_SIZE=5000
|
||||
|
||||
# Rate Limiting
|
||||
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_ENABLED=true
|
||||
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=5.0
|
||||
LOGWISP_STREAMS_0_TCPSERVER_RATE_LIMIT_BURST_SIZE=10
|
||||
```
|
||||
|
||||
## Complete Example
|
||||
|
||||
Here's a complete example configuring two streams via environment variables:
|
||||
## Example
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
# Logging configuration
|
||||
# Logging
|
||||
export LOGWISP_LOGGING_OUTPUT=both
|
||||
export LOGWISP_LOGGING_LEVEL=info
|
||||
export LOGWISP_LOGGING_FILE_DIRECTORY=/var/log/logwisp
|
||||
export LOGWISP_LOGGING_FILE_MAX_SIZE_MB=100
|
||||
|
||||
# Stream 0: Application logs
|
||||
export LOGWISP_STREAMS_0_NAME=app
|
||||
export LOGWISP_STREAMS_0_MONITOR_CHECK_INTERVAL_MS=50
|
||||
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/myapp
|
||||
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATTERN="*.log"
|
||||
export LOGWISP_STREAMS_0_MONITOR_TARGETS_0_IS_FILE=false
|
||||
# Pipeline 0: Application logs
|
||||
export LOGWISP_PIPELINES_0_NAME=app
|
||||
export LOGWISP_PIPELINES_0_SOURCES_0_TYPE=directory
|
||||
export LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATH=/var/log/myapp
|
||||
export LOGWISP_PIPELINES_0_SOURCES_0_OPTIONS_PATTERN="*.log"
|
||||
|
||||
# Stream 0: Filters
|
||||
export LOGWISP_STREAMS_0_FILTERS_0_TYPE=include
|
||||
export LOGWISP_STREAMS_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
|
||||
# Filters
|
||||
export LOGWISP_PIPELINES_0_FILTERS_0_TYPE=include
|
||||
export LOGWISP_PIPELINES_0_FILTERS_0_PATTERNS='["ERROR","WARN"]'
|
||||
|
||||
# Stream 0: HTTP server
|
||||
export LOGWISP_STREAMS_0_HTTPSERVER_ENABLED=true
|
||||
export LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
|
||||
export LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_ENABLED=true
|
||||
export LOGWISP_STREAMS_0_HTTPSERVER_RATE_LIMIT_REQUESTS_PER_SECOND=25.0
|
||||
# HTTP sink
|
||||
export LOGWISP_PIPELINES_0_SINKS_0_TYPE=http
|
||||
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_PORT=8080
|
||||
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_ENABLED=true
|
||||
export LOGWISP_PIPELINES_0_SINKS_0_OPTIONS_RATE_LIMIT_REQUESTS_PER_SECOND=25.0
|
||||
|
||||
# Stream 1: System logs
|
||||
export LOGWISP_STREAMS_1_NAME=system
|
||||
export LOGWISP_STREAMS_1_MONITOR_CHECK_INTERVAL_MS=1000
|
||||
export LOGWISP_STREAMS_1_MONITOR_TARGETS_0_PATH=/var/log/syslog
|
||||
export LOGWISP_STREAMS_1_MONITOR_TARGETS_0_IS_FILE=true
|
||||
# Pipeline 1: System logs
|
||||
export LOGWISP_PIPELINES_1_NAME=system
|
||||
export LOGWISP_PIPELINES_1_SOURCES_0_TYPE=file
|
||||
export LOGWISP_PIPELINES_1_SOURCES_0_OPTIONS_PATH=/var/log/syslog
|
||||
|
||||
# Stream 1: TCP server
|
||||
export LOGWISP_STREAMS_1_TCPSERVER_ENABLED=true
|
||||
export LOGWISP_STREAMS_1_TCPSERVER_PORT=9090
|
||||
# TCP sink
|
||||
export LOGWISP_PIPELINES_1_SINKS_0_TYPE=tcp
|
||||
export LOGWISP_PIPELINES_1_SINKS_0_OPTIONS_PORT=9090
|
||||
|
||||
# Start LogWisp
|
||||
logwisp
|
||||
```
|
||||
|
||||
## Docker/Kubernetes Usage
|
||||
## Precedence
|
||||
|
||||
Environment variables are ideal for containerized deployments:
|
||||
|
||||
### Docker
|
||||
```dockerfile
|
||||
FROM logwisp:latest
|
||||
ENV LOGWISP_LOGGING_OUTPUT=stdout
|
||||
ENV LOGWISP_STREAMS_0_NAME=container
|
||||
ENV LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/var/log/app
|
||||
ENV LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
|
||||
```
|
||||
|
||||
### Docker Compose
|
||||
```yaml
|
||||
version: '3'
|
||||
services:
|
||||
logwisp:
|
||||
image: logwisp:latest
|
||||
environment:
|
||||
- LOGWISP_LOGGING_OUTPUT=stdout
|
||||
- LOGWISP_STREAMS_0_NAME=webapp
|
||||
- LOGWISP_STREAMS_0_MONITOR_TARGETS_0_PATH=/logs
|
||||
- LOGWISP_STREAMS_0_HTTPSERVER_PORT=8080
|
||||
volumes:
|
||||
- ./logs:/logs:ro
|
||||
ports:
|
||||
- "8080:8080"
|
||||
```
|
||||
|
||||
### Kubernetes ConfigMap
|
||||
```yaml
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: logwisp-config
|
||||
data:
|
||||
LOGWISP_LOGGING_LEVEL: "info"
|
||||
LOGWISP_STREAMS_0_NAME: "k8s-app"
|
||||
LOGWISP_STREAMS_0_HTTPSERVER_PORT: "8080"
|
||||
```
|
||||
|
||||
## Precedence Rules
|
||||
|
||||
When the same setting is configured multiple ways, this precedence applies:
|
||||
|
||||
1. **Command-line flags** (highest priority)
|
||||
2. **Environment variables**
|
||||
3. **Configuration file**
|
||||
4. **Default values** (lowest priority)
|
||||
|
||||
Example:
|
||||
```bash
|
||||
# Config file has: logging.level = "info"
|
||||
export LOGWISP_LOGGING_LEVEL=warn
|
||||
logwisp --log-level debug
|
||||
|
||||
# Result: log level will be "debug" (CLI flag wins)
|
||||
```
|
||||
|
||||
## Debugging
|
||||
|
||||
To see which environment variables LogWisp recognizes:
|
||||
```bash
|
||||
# List all LOGWISP variables
|
||||
env | grep ^LOGWISP_
|
||||
|
||||
# Test configuration parsing
|
||||
LOGWISP_LOGGING_LEVEL=debug logwisp --version
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- **Sensitive Values**: Avoid putting passwords or tokens in environment variables
|
||||
- **Process Visibility**: Environment variables may be visible to other processes
|
||||
- **Container Security**: Use secrets management for sensitive configuration
|
||||
- **Logging**: Be careful not to log environment variable values
|
||||
|
||||
## See Also
|
||||
|
||||
- [Configuration Guide](configuration.md) - Complete configuration reference
|
||||
- [CLI Options](cli.md) - Command-line interface
|
||||
- [Docker Deployment](integrations.md#docker) - Container-specific guidance
|
||||
1. Command-line flags (highest)
|
||||
2. Environment variables
|
||||
3. Configuration file
|
||||
4. Defaults (lowest)
|
||||
426
doc/filters.md
426
doc/filters.md
@ -1,21 +1,17 @@
|
||||
# Filter Guide
|
||||
|
||||
LogWisp's filtering system allows you to control which log entries are streamed to clients, reducing noise and focusing on what matters.
|
||||
LogWisp filters control which log entries pass through pipelines using regular expressions.
|
||||
|
||||
## How Filters Work
|
||||
|
||||
Filters use regular expressions to match log entries. Each filter can either:
|
||||
- **Include**: Only matching logs pass through (whitelist)
|
||||
- **Include**: Only matching logs pass (whitelist)
|
||||
- **Exclude**: Matching logs are dropped (blacklist)
|
||||
- Multiple filters apply sequentially - all must pass
|
||||
|
||||
Multiple filters are applied sequentially - a log entry must pass ALL filters to be streamed.
|
||||
|
||||
## Filter Configuration
|
||||
|
||||
### Basic Structure
|
||||
## Configuration
|
||||
|
||||
```toml
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include" # or "exclude"
|
||||
logic = "or" # or "and"
|
||||
patterns = [
|
||||
@ -26,115 +22,79 @@ patterns = [
|
||||
|
||||
### Filter Types
|
||||
|
||||
#### Include Filter (Whitelist)
|
||||
Only logs matching the patterns are streamed:
|
||||
|
||||
#### Include Filter
|
||||
```toml
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = [
|
||||
"ERROR",
|
||||
"WARN",
|
||||
"CRITICAL"
|
||||
]
|
||||
# Result: Only ERROR, WARN, or CRITICAL logs are streamed
|
||||
patterns = ["ERROR", "WARN", "CRITICAL"]
|
||||
# Only ERROR, WARN, or CRITICAL logs pass
|
||||
```
|
||||
|
||||
#### Exclude Filter (Blacklist)
|
||||
Logs matching the patterns are dropped:
|
||||
|
||||
#### Exclude Filter
|
||||
```toml
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"DEBUG",
|
||||
"TRACE",
|
||||
"/health"
|
||||
]
|
||||
# Result: DEBUG, TRACE, and health check logs are filtered out
|
||||
patterns = ["DEBUG", "TRACE", "/health"]
|
||||
# DEBUG, TRACE, and health checks are dropped
|
||||
```
|
||||
|
||||
### Logic Operators
|
||||
|
||||
#### OR Logic (Default)
|
||||
Log matches if ANY pattern matches:
|
||||
- **OR**: Match ANY pattern (default)
|
||||
- **AND**: Match ALL patterns
|
||||
|
||||
```toml
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
# OR Logic
|
||||
logic = "or"
|
||||
patterns = ["ERROR", "FAIL", "EXCEPTION"]
|
||||
# Matches: "ERROR: disk full" OR "FAIL: connection timeout" OR "NullPointerException"
|
||||
```
|
||||
patterns = ["ERROR", "FAIL"]
|
||||
# Matches: "ERROR: disk full" OR "FAIL: timeout"
|
||||
|
||||
#### AND Logic
|
||||
Log matches only if ALL patterns match:
|
||||
|
||||
```toml
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
# AND Logic
|
||||
logic = "and"
|
||||
patterns = ["database", "timeout", "ERROR"]
|
||||
# Matches: "ERROR: database connection timeout"
|
||||
# Doesn't match: "ERROR: file not found" (missing "database" and "timeout")
|
||||
# Not: "ERROR: file not found"
|
||||
```
|
||||
|
||||
## Pattern Syntax
|
||||
|
||||
LogWisp uses Go's regular expression syntax (RE2):
|
||||
|
||||
### Basic Patterns
|
||||
Go regular expressions (RE2):
|
||||
|
||||
```toml
|
||||
"ERROR" # Substring match
|
||||
"(?i)error" # Case-insensitive
|
||||
"\\berror\\b" # Word boundaries
|
||||
"^ERROR" # Start of line
|
||||
"ERROR$" # End of line
|
||||
"error|fail|warn" # Alternatives
|
||||
```
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Log Levels
|
||||
```toml
|
||||
patterns = [
|
||||
"ERROR", # Exact substring match
|
||||
"(?i)error", # Case-insensitive
|
||||
"\\berror\\b", # Word boundaries
|
||||
"^ERROR", # Start of line
|
||||
"ERROR$", # End of line
|
||||
"ERR(OR)?", # Optional group
|
||||
"error|fail|exception" # Alternatives
|
||||
"\\[(ERROR|WARN|INFO)\\]", # [ERROR] format
|
||||
"(?i)\\b(error|warning)\\b", # Word boundaries
|
||||
"level=(error|warn)", # key=value format
|
||||
]
|
||||
```
|
||||
|
||||
### Common Pattern Examples
|
||||
|
||||
#### Log Levels
|
||||
### Application Errors
|
||||
```toml
|
||||
# Standard log levels
|
||||
patterns = [
|
||||
"\\[(ERROR|WARN|INFO|DEBUG)\\]", # [ERROR] format
|
||||
"(?i)\\b(error|warning|info|debug)\\b", # Word boundaries
|
||||
"level=(error|warn|info|debug)", # key=value format
|
||||
"<(Error|Warning|Info|Debug)>" # XML-style
|
||||
]
|
||||
|
||||
# Severity patterns
|
||||
patterns = [
|
||||
"(?i)(fatal|critical|severe)",
|
||||
"(?i)(error|fail|exception)",
|
||||
"(?i)(warn|warning|caution)",
|
||||
"panic:", # Go panics
|
||||
"Traceback", # Python errors
|
||||
]
|
||||
```
|
||||
|
||||
#### Application Errors
|
||||
```toml
|
||||
# Java/JVM
|
||||
# Java
|
||||
patterns = [
|
||||
"Exception",
|
||||
"\\.java:[0-9]+", # Stack trace lines
|
||||
"at com\\.mycompany\\.", # Company packages
|
||||
"NullPointerException|ClassNotFoundException"
|
||||
"at .+\\.java:[0-9]+",
|
||||
"NullPointerException"
|
||||
]
|
||||
|
||||
# Python
|
||||
patterns = [
|
||||
"Traceback \\(most recent call last\\)",
|
||||
"Traceback",
|
||||
"File \".+\\.py\", line [0-9]+",
|
||||
"(ValueError|TypeError|KeyError)"
|
||||
"ValueError|TypeError"
|
||||
]
|
||||
|
||||
# Go
|
||||
@ -143,297 +103,73 @@ patterns = [
|
||||
"goroutine [0-9]+",
|
||||
"runtime error:"
|
||||
]
|
||||
```
|
||||
|
||||
# Node.js
|
||||
### Performance Issues
|
||||
```toml
|
||||
patterns = [
|
||||
"Error:",
|
||||
"at .+ \\(.+\\.js:[0-9]+:[0-9]+\\)",
|
||||
"UnhandledPromiseRejection"
|
||||
"took [0-9]{4,}ms", # >999ms operations
|
||||
"timeout|timed out",
|
||||
"slow query",
|
||||
"high cpu|cpu usage: [8-9][0-9]%"
|
||||
]
|
||||
```
|
||||
|
||||
#### Performance Issues
|
||||
### HTTP Patterns
|
||||
```toml
|
||||
patterns = [
|
||||
"took [0-9]{4,}ms", # Operations over 999ms
|
||||
"duration>[0-9]{3,}s", # Long durations
|
||||
"timeout|timed out", # Timeouts
|
||||
"slow query", # Database
|
||||
"memory pressure", # Memory issues
|
||||
"high cpu|cpu usage: [8-9][0-9]%" # CPU issues
|
||||
]
|
||||
```
|
||||
|
||||
#### Security Patterns
|
||||
```toml
|
||||
patterns = [
|
||||
"(?i)(unauthorized|forbidden|denied)",
|
||||
"(?i)(auth|authentication) fail",
|
||||
"invalid (token|session|credentials)",
|
||||
"SQL injection|XSS|CSRF",
|
||||
"brute force|rate limit",
|
||||
"suspicious activity"
|
||||
]
|
||||
```
|
||||
|
||||
#### HTTP Patterns
|
||||
```toml
|
||||
# Error status codes
|
||||
patterns = [
|
||||
"status[=:][4-5][0-9]{2}", # status=404, status:500
|
||||
"HTTP/[0-9.]+ [4-5][0-9]{2}", # HTTP/1.1 404
|
||||
"\"status\":\\s*[4-5][0-9]{2}" # JSON "status": 500
|
||||
]
|
||||
|
||||
# Specific endpoints
|
||||
patterns = [
|
||||
"\"(GET|POST|PUT|DELETE) /api/",
|
||||
"/api/v[0-9]+/users",
|
||||
"path=\"/admin"
|
||||
"status[=:][4-5][0-9]{2}", # 4xx/5xx codes
|
||||
"HTTP/[0-9.]+ [4-5][0-9]{2}",
|
||||
"\"/api/v[0-9]+/", # API paths
|
||||
]
|
||||
```
|
||||
|
||||
## Filter Chains
|
||||
|
||||
Multiple filters create a processing chain. Each filter must pass for the log to be streamed.
|
||||
|
||||
### Example: Error Monitoring
|
||||
### Error Monitoring
|
||||
```toml
|
||||
# Step 1: Include only errors and warnings
|
||||
[[streams.filters]]
|
||||
# Include errors
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = [
|
||||
"(?i)\\b(error|fail|exception)\\b",
|
||||
"(?i)\\b(warn|warning)\\b",
|
||||
"(?i)\\b(critical|fatal|severe)\\b"
|
||||
]
|
||||
patterns = ["(?i)\\b(error|fail|critical)\\b"]
|
||||
|
||||
# Step 2: Exclude known non-issues
|
||||
[[streams.filters]]
|
||||
# Exclude known non-issues
|
||||
[[pipelines.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"Error: Expected behavior",
|
||||
"Warning: Deprecated API",
|
||||
"INFO.*error in message" # INFO logs talking about errors
|
||||
]
|
||||
|
||||
# Step 3: Exclude noisy sources
|
||||
[[streams.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"/health",
|
||||
"/metrics",
|
||||
"ELB-HealthChecker",
|
||||
"Googlebot"
|
||||
]
|
||||
patterns = ["Error: Expected", "/health"]
|
||||
```
|
||||
|
||||
### Example: API Monitoring
|
||||
### API Monitoring
|
||||
```toml
|
||||
# Include only API calls
|
||||
[[streams.filters]]
|
||||
# Include API calls
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
patterns = [
|
||||
"/api/",
|
||||
"/v[0-9]+/"
|
||||
]
|
||||
patterns = ["/api/", "/v[0-9]+/"]
|
||||
|
||||
# Exclude successful requests
|
||||
[[streams.filters]]
|
||||
# Exclude successful
|
||||
[[pipelines.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"\" 200 ", # HTTP 200 OK
|
||||
"\" 201 ", # HTTP 201 Created
|
||||
"\" 204 ", # HTTP 204 No Content
|
||||
"\" 304 " # HTTP 304 Not Modified
|
||||
]
|
||||
|
||||
# Exclude OPTIONS requests (CORS)
|
||||
[[streams.filters]]
|
||||
type = "exclude"
|
||||
patterns = [
|
||||
"OPTIONS "
|
||||
]
|
||||
patterns = ["\" 2[0-9]{2} "]
|
||||
```
|
||||
|
||||
### Example: Security Audit
|
||||
```toml
|
||||
# Include security-relevant events
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = [
|
||||
"(?i)auth",
|
||||
"(?i)login|logout",
|
||||
"(?i)sudo|root",
|
||||
"(?i)ssh|sftp|ftp",
|
||||
"(?i)firewall|iptables",
|
||||
"COMMAND=", # sudo commands
|
||||
"USER=", # user actions
|
||||
"SELINUX"
|
||||
]
|
||||
## Performance Tips
|
||||
|
||||
# Must also contain failure/success indicators
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
logic = "or"
|
||||
patterns = [
|
||||
"(?i)(fail|denied|error)",
|
||||
"(?i)(success|accepted|granted)",
|
||||
"(?i)(invalid|unauthorized)"
|
||||
]
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Pattern Complexity
|
||||
|
||||
Simple patterns are fast (~1μs per check):
|
||||
```toml
|
||||
patterns = ["ERROR", "WARN", "FATAL"]
|
||||
```
|
||||
|
||||
Complex patterns are slower (~10-100μs per check):
|
||||
```toml
|
||||
patterns = [
|
||||
"^\\[\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\]\\s+\\[(ERROR|WARN)\\]\\s+\\[([^\\]]+)\\]\\s+(.+)$"
|
||||
]
|
||||
```
|
||||
|
||||
### Optimization Tips
|
||||
|
||||
1. **Use anchors when possible**:
|
||||
```toml
|
||||
"^ERROR" # Faster than "ERROR"
|
||||
```
|
||||
|
||||
2. **Avoid nested quantifiers**:
|
||||
```toml
|
||||
# BAD: Can cause exponential backtracking
|
||||
"((a+)+)+"
|
||||
|
||||
# GOOD: Linear time
|
||||
"a+"
|
||||
```
|
||||
|
||||
3. **Use non-capturing groups**:
|
||||
```toml
|
||||
"(?:error|warn)" # Instead of "(error|warn)"
|
||||
```
|
||||
|
||||
4. **Order patterns by frequency**:
|
||||
```toml
|
||||
# Most common first
|
||||
patterns = ["ERROR", "WARN", "INFO", "DEBUG"]
|
||||
```
|
||||
|
||||
5. **Prefer character classes**:
|
||||
```toml
|
||||
"[0-9]" # Instead of "\\d"
|
||||
"[a-zA-Z]" # Instead of "\\w"
|
||||
```
|
||||
1. **Use anchors**: `^ERROR` faster than `ERROR`
|
||||
2. **Avoid nested quantifiers**: `((a+)+)+`
|
||||
3. **Non-capturing groups**: `(?:error|warn)`
|
||||
4. **Order by frequency**: Most common first
|
||||
5. **Simple patterns**: Faster than complex regex
|
||||
|
||||
## Testing Filters
|
||||
|
||||
### Test Configuration
|
||||
Create a test configuration with sample logs:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "test"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "./test-logs", pattern = "*.log" }]
|
||||
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = ["YOUR_PATTERN_HERE"]
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8888
|
||||
```
|
||||
|
||||
### Generate Test Logs
|
||||
```bash
|
||||
# Create test log entries
|
||||
echo "[ERROR] Database connection failed" >> test-logs/app.log
|
||||
echo "[INFO] User logged in" >> test-logs/app.log
|
||||
echo "[WARN] High memory usage: 85%" >> test-logs/app.log
|
||||
# Test configuration
|
||||
echo "[ERROR] Test" >> test.log
|
||||
echo "[INFO] Test" >> test.log
|
||||
|
||||
# Run LogWisp with debug logging
|
||||
logwisp --config test.toml --log-level debug
|
||||
# Run with debug
|
||||
logwisp --log-level debug
|
||||
|
||||
# Check what passes through
|
||||
curl -N http://localhost:8888/stream
|
||||
```
|
||||
|
||||
### Debug Filter Behavior
|
||||
Enable debug logging to see filter decisions:
|
||||
|
||||
```bash
|
||||
logwisp --log-level debug --log-output stderr
|
||||
```
|
||||
|
||||
Look for messages like:
|
||||
```
|
||||
Entry filtered out component=filter_chain filter_index=0 filter_type=include
|
||||
Entry passed all filters component=filter_chain
|
||||
```
|
||||
|
||||
## Common Pitfalls
|
||||
|
||||
### Case Sensitivity
|
||||
By default, patterns are case-sensitive:
|
||||
```toml
|
||||
# Won't match "error" or "Error"
|
||||
patterns = ["ERROR"]
|
||||
|
||||
# Use case-insensitive flag
|
||||
patterns = ["(?i)error"]
|
||||
```
|
||||
|
||||
### Partial Matches
|
||||
Patterns match substrings by default:
|
||||
```toml
|
||||
# Matches "ERROR", "ERRORS", "TERROR"
|
||||
patterns = ["ERROR"]
|
||||
|
||||
# Use word boundaries for exact words
|
||||
patterns = ["\\bERROR\\b"]
|
||||
```
|
||||
|
||||
### Special Characters
|
||||
Remember to escape regex special characters:
|
||||
```toml
|
||||
# Won't work as expected
|
||||
patterns = ["[ERROR]"]
|
||||
|
||||
# Correct: escape brackets
|
||||
patterns = ["\\[ERROR\\]"]
|
||||
```
|
||||
|
||||
### Performance Impact
|
||||
Too many complex patterns can impact performance:
|
||||
```toml
|
||||
# Consider splitting into multiple streams instead
|
||||
[[streams.filters]]
|
||||
patterns = [
|
||||
# 50+ complex patterns...
|
||||
]
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start Simple**: Begin with basic patterns and refine as needed
|
||||
2. **Test Thoroughly**: Use test logs to verify filter behavior
|
||||
3. **Monitor Performance**: Check filter statistics in `/status`
|
||||
4. **Document Patterns**: Comment complex patterns for maintenance
|
||||
5. **Use Multiple Streams**: Instead of complex filters, consider separate streams
|
||||
6. **Regular Review**: Periodically review and optimize filter rules
|
||||
|
||||
## See Also
|
||||
|
||||
- [Configuration Guide](configuration.md) - Complete configuration reference
|
||||
- [Performance Tuning](performance.md) - Optimization guidelines
|
||||
- [Examples](examples/) - Real-world filter configurations
|
||||
# Check output
|
||||
curl -N http://localhost:8080/stream
|
||||
```
|
||||
@ -1,137 +1,54 @@
|
||||
# Installation Guide
|
||||
|
||||
This guide covers installing LogWisp on various platforms and deployment scenarios.
|
||||
Installation process on tested platforms.
|
||||
|
||||
## Requirements
|
||||
|
||||
### System Requirements
|
||||
- **OS**: Linux, FreeBSD
|
||||
- **Architecture**: amd64
|
||||
- **Go**: 1.23+ (for building)
|
||||
|
||||
- **OS**: Linux, macOS, FreeBSD, Windows (with WSL)
|
||||
- **Architecture**: amd64, arm64
|
||||
- **Memory**: 64MB minimum, 256MB recommended
|
||||
- **Disk**: 10MB for binary, plus log storage
|
||||
- **Go**: 1.23+ (for building from source)
|
||||
|
||||
### Runtime Dependencies
|
||||
|
||||
LogWisp is a single static binary with no runtime dependencies. It only requires:
|
||||
- Read access to monitored log files
|
||||
- Network access for serving streams
|
||||
- Write access for operational logs (optional)
|
||||
|
||||
## Installation Methods
|
||||
## Installation
|
||||
|
||||
### Pre-built Binaries
|
||||
|
||||
Download the latest release:
|
||||
|
||||
```bash
|
||||
# Linux (amd64)
|
||||
# Linux amd64
|
||||
wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-linux-amd64
|
||||
chmod +x logwisp-linux-amd64
|
||||
sudo mv logwisp-linux-amd64 /usr/local/bin/logwisp
|
||||
|
||||
# macOS (Intel)
|
||||
# macOS Intel
|
||||
wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-darwin-amd64
|
||||
chmod +x logwisp-darwin-amd64
|
||||
sudo mv logwisp-darwin-amd64 /usr/local/bin/logwisp
|
||||
|
||||
# macOS (Apple Silicon)
|
||||
wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-darwin-arm64
|
||||
chmod +x logwisp-darwin-arm64
|
||||
sudo mv logwisp-darwin-arm64 /usr/local/bin/logwisp
|
||||
```
|
||||
|
||||
Verify installation:
|
||||
```bash
|
||||
# Verify
|
||||
logwisp --version
|
||||
```
|
||||
|
||||
### From Source
|
||||
|
||||
Build from source code:
|
||||
|
||||
```bash
|
||||
# Clone repository
|
||||
git clone https://github.com/yourusername/logwisp.git
|
||||
cd logwisp
|
||||
|
||||
# Build
|
||||
make build
|
||||
|
||||
# Install
|
||||
sudo make install
|
||||
|
||||
# Or install to custom location
|
||||
make install PREFIX=/opt/logwisp
|
||||
```
|
||||
|
||||
### Using Go Install
|
||||
|
||||
Install directly with Go:
|
||||
### Go Install
|
||||
|
||||
```bash
|
||||
go install github.com/yourusername/logwisp/src/cmd/logwisp@latest
|
||||
go install github.com/lixenwraith/logwisp/src/cmd/logwisp@latest
|
||||
```
|
||||
Note: Binary created with this method will not contain version information.
|
||||
|
||||
Note: This installs to `$GOPATH/bin` (usually `~/go/bin`)
|
||||
## Platform-Specific
|
||||
|
||||
### Docker
|
||||
|
||||
Official Docker image:
|
||||
### Linux (systemd)
|
||||
|
||||
```bash
|
||||
# Pull image
|
||||
docker pull yourusername/logwisp:latest
|
||||
|
||||
# Run with volume mount
|
||||
docker run -d \
|
||||
--name logwisp \
|
||||
-p 8080:8080 \
|
||||
-v /var/log:/logs:ro \
|
||||
-v $PWD/config.toml:/config/logwisp.toml:ro \
|
||||
yourusername/logwisp:latest \
|
||||
--config /config/logwisp.toml
|
||||
```
|
||||
|
||||
Build your own image:
|
||||
|
||||
```dockerfile
|
||||
FROM golang:1.23-alpine AS builder
|
||||
WORKDIR /build
|
||||
COPY . .
|
||||
RUN go build -o logwisp ./src/cmd/logwisp
|
||||
|
||||
FROM alpine:latest
|
||||
RUN apk --no-cache add ca-certificates
|
||||
COPY --from=builder /build/logwisp /usr/local/bin/
|
||||
ENTRYPOINT ["logwisp"]
|
||||
```
|
||||
|
||||
## Platform-Specific Instructions
|
||||
|
||||
### Linux
|
||||
|
||||
#### Debian/Ubuntu
|
||||
|
||||
Create package (planned):
|
||||
```bash
|
||||
# Future feature
|
||||
sudo apt install logwisp
|
||||
```
|
||||
|
||||
Manual installation:
|
||||
```bash
|
||||
# Download binary
|
||||
wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-linux-amd64 -O logwisp
|
||||
chmod +x logwisp
|
||||
sudo mv logwisp /usr/local/bin/
|
||||
|
||||
# Create config directory
|
||||
sudo mkdir -p /etc/logwisp
|
||||
sudo cp config/logwisp.toml.example /etc/logwisp/logwisp.toml
|
||||
|
||||
# Create systemd service
|
||||
# Create service
|
||||
sudo tee /etc/systemd/system/logwisp.service << EOF
|
||||
[Unit]
|
||||
Description=LogWisp Log Monitoring Service
|
||||
@ -140,21 +57,10 @@ After=network.target
|
||||
[Service]
|
||||
Type=simple
|
||||
User=logwisp
|
||||
Group=logwisp
|
||||
ExecStart=/usr/local/bin/logwisp --config /etc/logwisp/logwisp.toml
|
||||
Restart=always
|
||||
RestartSec=5
|
||||
StandardOutput=journal
|
||||
StandardError=journal
|
||||
SyslogIdentifier=logwisp
|
||||
|
||||
# Security hardening
|
||||
NoNewPrivileges=true
|
||||
PrivateTmp=true
|
||||
ProtectSystem=strict
|
||||
ProtectHome=true
|
||||
ReadOnlyPaths=/var/log
|
||||
ReadWritePaths=/var/log/logwisp
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@ -163,9 +69,12 @@ EOF
|
||||
# Create user
|
||||
sudo useradd -r -s /bin/false logwisp
|
||||
|
||||
# Create log directory
|
||||
sudo mkdir -p /var/log/logwisp
|
||||
sudo chown logwisp:logwisp /var/log/logwisp
|
||||
# Create service user
|
||||
sudo useradd -r -s /bin/false logwisp
|
||||
|
||||
# Create configuration directory
|
||||
sudo mkdir -p /etc/logwisp
|
||||
sudo chown logwisp:logwisp /etc/logwisp
|
||||
|
||||
# Enable and start
|
||||
sudo systemctl daemon-reload
|
||||
@ -173,93 +82,11 @@ sudo systemctl enable logwisp
|
||||
sudo systemctl start logwisp
|
||||
```
|
||||
|
||||
#### Red Hat/CentOS/Fedora
|
||||
### FreeBSD (rc.d)
|
||||
|
||||
```bash
|
||||
# Similar to Debian, but use:
|
||||
sudo yum install wget # or dnf on newer versions
|
||||
|
||||
# SELinux context (if enabled)
|
||||
sudo semanage fcontext -a -t bin_t /usr/local/bin/logwisp
|
||||
sudo restorecon -v /usr/local/bin/logwisp
|
||||
```
|
||||
|
||||
#### Arch Linux
|
||||
|
||||
AUR package (community maintained):
|
||||
```bash
|
||||
# Future feature
|
||||
yay -S logwisp
|
||||
```
|
||||
|
||||
### macOS
|
||||
|
||||
#### Homebrew
|
||||
|
||||
Formula (planned):
|
||||
```bash
|
||||
# Future feature
|
||||
brew install logwisp
|
||||
```
|
||||
|
||||
#### Manual Installation
|
||||
|
||||
```bash
|
||||
# Download and install
|
||||
curl -L https://github.com/yourusername/logwisp/releases/latest/download/logwisp-darwin-$(uname -m) -o logwisp
|
||||
chmod +x logwisp
|
||||
sudo mv logwisp /usr/local/bin/
|
||||
|
||||
# Create LaunchDaemon
|
||||
sudo tee /Library/LaunchDaemons/com.logwisp.plist << EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key>
|
||||
<string>com.logwisp</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array>
|
||||
<string>/usr/local/bin/logwisp</string>
|
||||
<string>--config</string>
|
||||
<string>/usr/local/etc/logwisp/logwisp.toml</string>
|
||||
</array>
|
||||
<key>RunAtLoad</key>
|
||||
<true/>
|
||||
<key>KeepAlive</key>
|
||||
<true/>
|
||||
<key>StandardOutPath</key>
|
||||
<string>/usr/local/var/log/logwisp.log</string>
|
||||
<key>StandardErrorPath</key>
|
||||
<string>/usr/local/var/log/logwisp.error.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
|
||||
# Load service
|
||||
sudo launchctl load /Library/LaunchDaemons/com.logwisp.plist
|
||||
```
|
||||
|
||||
### FreeBSD
|
||||
|
||||
#### Ports
|
||||
|
||||
```bash
|
||||
# Future feature
|
||||
cd /usr/ports/sysutils/logwisp
|
||||
make install clean
|
||||
```
|
||||
|
||||
#### Manual Installation
|
||||
|
||||
```bash
|
||||
# Download
|
||||
fetch https://github.com/yourusername/logwisp/releases/latest/download/logwisp-freebsd-amd64
|
||||
chmod +x logwisp-freebsd-amd64
|
||||
mv logwisp-freebsd-amd64 /usr/local/bin/logwisp
|
||||
|
||||
# RC script
|
||||
cat > /usr/local/etc/rc.d/logwisp << 'EOF'
|
||||
# Create service script
|
||||
sudo tee /usr/local/etc/rc.d/logwisp << 'EOF'
|
||||
#!/bin/sh
|
||||
|
||||
# PROVIDE: logwisp
|
||||
@ -273,319 +100,125 @@ rcvar="${name}_enable"
|
||||
command="/usr/local/bin/logwisp"
|
||||
command_args="--config /usr/local/etc/logwisp/logwisp.toml"
|
||||
pidfile="/var/run/${name}.pid"
|
||||
start_cmd="logwisp_start"
|
||||
stop_cmd="logwisp_stop"
|
||||
|
||||
logwisp_start()
|
||||
{
|
||||
echo "Starting logwisp service..."
|
||||
/usr/sbin/daemon -c -f -p ${pidfile} ${command} ${command_args}
|
||||
}
|
||||
|
||||
logwisp_stop()
|
||||
{
|
||||
if [ -f ${pidfile} ]; then
|
||||
echo "Stopping logwisp service..."
|
||||
kill $(cat ${pidfile})
|
||||
rm -f ${pidfile}
|
||||
fi
|
||||
}
|
||||
|
||||
load_rc_config $name
|
||||
: ${logwisp_enable:="NO"}
|
||||
: ${logwisp_config:="/usr/local/etc/logwisp/logwisp.toml"}
|
||||
|
||||
run_rc_command "$1"
|
||||
EOF
|
||||
|
||||
chmod +x /usr/local/etc/rc.d/logwisp
|
||||
# Make executable
|
||||
sudo chmod +x /usr/local/etc/rc.d/logwisp
|
||||
|
||||
# Enable
|
||||
sysrc logwisp_enable="YES"
|
||||
service logwisp start
|
||||
```
|
||||
# Create service user
|
||||
sudo pw useradd logwisp -d /nonexistent -s /usr/sbin/nologin
|
||||
|
||||
### Windows
|
||||
# Create configuration directory
|
||||
sudo mkdir -p /usr/local/etc/logwisp
|
||||
sudo chown logwisp:logwisp /usr/local/etc/logwisp
|
||||
|
||||
#### Windows Subsystem for Linux (WSL)
|
||||
# Enable service
|
||||
sudo sysrc logwisp_enable="YES"
|
||||
|
||||
```bash
|
||||
# Inside WSL, follow Linux instructions
|
||||
wget https://github.com/yourusername/logwisp/releases/latest/download/logwisp-linux-amd64
|
||||
chmod +x logwisp-linux-amd64
|
||||
./logwisp-linux-amd64
|
||||
```
|
||||
|
||||
#### Native Windows (planned)
|
||||
|
||||
Future support for native Windows service.
|
||||
|
||||
## Container Deployment
|
||||
|
||||
### Docker Compose
|
||||
|
||||
```yaml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
logwisp:
|
||||
image: yourusername/logwisp:latest
|
||||
container_name: logwisp
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
- "8080:8080"
|
||||
- "9090:9090" # If using TCP
|
||||
volumes:
|
||||
- /var/log:/logs:ro
|
||||
- ./logwisp.toml:/config/logwisp.toml:ro
|
||||
command: ["--config", "/config/logwisp.toml"]
|
||||
environment:
|
||||
- LOGWISP_LOGGING_LEVEL=info
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:8080/status"]
|
||||
interval: 30s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
```
|
||||
|
||||
### Kubernetes
|
||||
|
||||
Deployment manifest:
|
||||
|
||||
```yaml
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: logwisp
|
||||
labels:
|
||||
app: logwisp
|
||||
spec:
|
||||
replicas: 1
|
||||
selector:
|
||||
matchLabels:
|
||||
app: logwisp
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: logwisp
|
||||
spec:
|
||||
containers:
|
||||
- name: logwisp
|
||||
image: yourusername/logwisp:latest
|
||||
args:
|
||||
- --config
|
||||
- /config/logwisp.toml
|
||||
ports:
|
||||
- containerPort: 8080
|
||||
name: http
|
||||
- containerPort: 9090
|
||||
name: tcp
|
||||
volumeMounts:
|
||||
- name: logs
|
||||
mountPath: /logs
|
||||
readOnly: true
|
||||
- name: config
|
||||
mountPath: /config
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /status
|
||||
port: 8080
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /status
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
resources:
|
||||
requests:
|
||||
memory: "128Mi"
|
||||
cpu: "100m"
|
||||
limits:
|
||||
memory: "512Mi"
|
||||
cpu: "500m"
|
||||
volumes:
|
||||
- name: logs
|
||||
hostPath:
|
||||
path: /var/log
|
||||
- name: config
|
||||
configMap:
|
||||
name: logwisp-config
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: logwisp
|
||||
spec:
|
||||
selector:
|
||||
app: logwisp
|
||||
ports:
|
||||
- name: http
|
||||
port: 8080
|
||||
targetPort: 8080
|
||||
- name: tcp
|
||||
port: 9090
|
||||
targetPort: 9090
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: logwisp-config
|
||||
data:
|
||||
logwisp.toml: |
|
||||
[[streams]]
|
||||
name = "k8s"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
# Start service
|
||||
sudo service logwisp start
|
||||
```
|
||||
|
||||
## Post-Installation
|
||||
|
||||
### Verify Installation
|
||||
```bash
|
||||
# Check version
|
||||
logwisp --version
|
||||
|
||||
1. Check version:
|
||||
```bash
|
||||
logwisp --version
|
||||
```
|
||||
# Test configuration
|
||||
logwisp --config /etc/logwisp/logwisp.toml --log-level debug
|
||||
|
||||
2. Test configuration:
|
||||
```bash
|
||||
logwisp --config /etc/logwisp/logwisp.toml --log-level debug
|
||||
```
|
||||
# Check service
|
||||
sudo systemctl status logwisp
|
||||
```
|
||||
|
||||
3. Check service status:
|
||||
```bash
|
||||
# systemd
|
||||
sudo systemctl status logwisp
|
||||
|
||||
# macOS
|
||||
sudo launchctl list | grep logwisp
|
||||
|
||||
# FreeBSD
|
||||
service logwisp status
|
||||
```
|
||||
### Linux Service Status
|
||||
```bash
|
||||
sudo systemctl status logwisp
|
||||
```
|
||||
|
||||
4. Test streaming:
|
||||
```bash
|
||||
curl -N http://localhost:8080/stream
|
||||
```
|
||||
|
||||
### Security Hardening
|
||||
|
||||
1. **Create dedicated user**:
|
||||
```bash
|
||||
sudo useradd -r -s /bin/false -d /var/lib/logwisp logwisp
|
||||
```
|
||||
|
||||
2. **Set file permissions**:
|
||||
```bash
|
||||
sudo chown root:root /usr/local/bin/logwisp
|
||||
sudo chmod 755 /usr/local/bin/logwisp
|
||||
sudo chown -R logwisp:logwisp /etc/logwisp
|
||||
sudo chmod 640 /etc/logwisp/logwisp.toml
|
||||
```
|
||||
|
||||
3. **Configure firewall**:
|
||||
```bash
|
||||
# UFW
|
||||
sudo ufw allow 8080/tcp comment "LogWisp HTTP"
|
||||
|
||||
# firewalld
|
||||
sudo firewall-cmd --permanent --add-port=8080/tcp
|
||||
sudo firewall-cmd --reload
|
||||
```
|
||||
|
||||
4. **Enable SELinux/AppArmor** (if applicable)
|
||||
### FreeBSD Service Status
|
||||
```bash
|
||||
sudo service logwisp status
|
||||
```
|
||||
|
||||
### Initial Configuration
|
||||
|
||||
1. Copy example configuration:
|
||||
```bash
|
||||
sudo cp /usr/local/share/logwisp/examples/logwisp.toml.example /etc/logwisp/logwisp.toml
|
||||
```
|
||||
Create a basic configuration file:
|
||||
|
||||
2. Edit configuration:
|
||||
```bash
|
||||
sudo nano /etc/logwisp/logwisp.toml
|
||||
```
|
||||
```toml
|
||||
# /etc/logwisp/logwisp.toml (Linux)
|
||||
# /usr/local/etc/logwisp/logwisp.toml (FreeBSD)
|
||||
|
||||
3. Set up log monitoring:
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "myapp"
|
||||
[streams.monitor]
|
||||
targets = [
|
||||
{ path = "/var/log/myapp", pattern = "*.log" }
|
||||
]
|
||||
```
|
||||
[[pipelines]]
|
||||
name = "myapp"
|
||||
|
||||
4. Restart service:
|
||||
```bash
|
||||
sudo systemctl restart logwisp
|
||||
```
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = {
|
||||
path = "/path/to/application/logs",
|
||||
pattern = "*.log"
|
||||
}
|
||||
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
```
|
||||
|
||||
Restart service after configuration changes:
|
||||
|
||||
**Linux:**
|
||||
```bash
|
||||
sudo systemctl restart logwisp
|
||||
```
|
||||
|
||||
**FreeBSD:**
|
||||
```bash
|
||||
sudo service logwisp restart
|
||||
```
|
||||
|
||||
## Uninstallation
|
||||
|
||||
### Linux
|
||||
```bash
|
||||
# Stop service
|
||||
sudo systemctl stop logwisp
|
||||
sudo systemctl disable logwisp
|
||||
|
||||
# Remove files
|
||||
sudo rm /usr/local/bin/logwisp
|
||||
sudo rm /etc/systemd/system/logwisp.service
|
||||
sudo rm -rf /etc/logwisp
|
||||
sudo rm -rf /var/log/logwisp
|
||||
|
||||
# Remove user
|
||||
sudo userdel logwisp
|
||||
```
|
||||
|
||||
### macOS
|
||||
### FreeBSD
|
||||
```bash
|
||||
# Stop service
|
||||
sudo launchctl unload /Library/LaunchDaemons/com.logwisp.plist
|
||||
|
||||
# Remove files
|
||||
sudo service logwisp stop
|
||||
sudo sysrc logwisp_enable="NO"
|
||||
sudo rm /usr/local/bin/logwisp
|
||||
sudo rm /Library/LaunchDaemons/com.logwisp.plist
|
||||
sudo rm /usr/local/etc/rc.d/logwisp
|
||||
sudo rm -rf /usr/local/etc/logwisp
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker stop logwisp
|
||||
docker rm logwisp
|
||||
docker rmi yourusername/logwisp:latest
|
||||
```
|
||||
|
||||
## Troubleshooting Installation
|
||||
|
||||
### Permission Denied
|
||||
|
||||
If you get permission errors:
|
||||
```bash
|
||||
# Check file ownership
|
||||
ls -la /usr/local/bin/logwisp
|
||||
|
||||
# Fix permissions
|
||||
sudo chmod +x /usr/local/bin/logwisp
|
||||
|
||||
# Check log directory
|
||||
sudo mkdir -p /var/log/logwisp
|
||||
sudo chown logwisp:logwisp /var/log/logwisp
|
||||
```
|
||||
|
||||
### Service Won't Start
|
||||
|
||||
Check logs:
|
||||
```bash
|
||||
# systemd
|
||||
sudo journalctl -u logwisp -f
|
||||
|
||||
# Manual run
|
||||
sudo -u logwisp /usr/local/bin/logwisp --config /etc/logwisp/logwisp.toml
|
||||
```
|
||||
|
||||
### Port Already in Use
|
||||
|
||||
Find conflicting process:
|
||||
```bash
|
||||
sudo lsof -i :8080
|
||||
# or
|
||||
sudo netstat -tlnp | grep 8080
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Quick Start](quickstart.md) - Get running quickly
|
||||
- [Configuration Guide](configuration.md) - Configure LogWisp
|
||||
- [Troubleshooting](troubleshooting.md) - Common issues
|
||||
- [Security Best Practices](security.md) - Hardening guide
|
||||
sudo pw userdel logwisp
|
||||
```
|
||||
@ -1,511 +0,0 @@
|
||||
# Monitoring & Status Guide
|
||||
|
||||
LogWisp provides comprehensive monitoring capabilities through status endpoints, operational logs, and metrics.
|
||||
|
||||
## Status Endpoints
|
||||
|
||||
### Stream Status
|
||||
|
||||
Each stream exposes its own status endpoint:
|
||||
|
||||
```bash
|
||||
# Standalone mode
|
||||
curl http://localhost:8080/status
|
||||
|
||||
# Router mode
|
||||
curl http://localhost:8080/streamname/status
|
||||
```
|
||||
|
||||
Example response:
|
||||
```json
|
||||
{
|
||||
"service": "LogWisp",
|
||||
"version": "1.0.0",
|
||||
"server": {
|
||||
"type": "http",
|
||||
"port": 8080,
|
||||
"active_clients": 5,
|
||||
"buffer_size": 1000,
|
||||
"uptime_seconds": 3600,
|
||||
"mode": {
|
||||
"standalone": true,
|
||||
"router": false
|
||||
}
|
||||
},
|
||||
"monitor": {
|
||||
"active_watchers": 3,
|
||||
"total_entries": 152341,
|
||||
"dropped_entries": 12,
|
||||
"start_time": "2024-01-20T10:00:00Z",
|
||||
"last_entry_time": "2024-01-20T11:00:00Z"
|
||||
},
|
||||
"filters": {
|
||||
"filter_count": 2,
|
||||
"total_processed": 152341,
|
||||
"total_passed": 48234,
|
||||
"filters": [
|
||||
{
|
||||
"type": "include",
|
||||
"logic": "or",
|
||||
"pattern_count": 3,
|
||||
"total_processed": 152341,
|
||||
"total_matched": 48234,
|
||||
"total_dropped": 0
|
||||
}
|
||||
]
|
||||
},
|
||||
"features": {
|
||||
"heartbeat": {
|
||||
"enabled": true,
|
||||
"interval": 30,
|
||||
"format": "comment"
|
||||
},
|
||||
"rate_limit": {
|
||||
"enabled": true,
|
||||
"total_requests": 8234,
|
||||
"blocked_requests": 89,
|
||||
"active_ips": 12,
|
||||
"total_connections": 5
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Global Status (Router Mode)
|
||||
|
||||
In router mode, a global status endpoint provides aggregated information:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/status
|
||||
```
|
||||
|
||||
## Key Metrics
|
||||
|
||||
### Monitor Metrics
|
||||
|
||||
Track file watching performance:
|
||||
|
||||
| Metric | Description | Healthy Range |
|
||||
|--------|-------------|---------------|
|
||||
| `active_watchers` | Number of files being watched | 1-1000 |
|
||||
| `total_entries` | Total log entries processed | Increasing |
|
||||
| `dropped_entries` | Entries dropped due to buffer full | < 1% of total |
|
||||
| `entries_per_second` | Current processing rate | Varies |
|
||||
|
||||
### Connection Metrics
|
||||
|
||||
Monitor client connections:
|
||||
|
||||
| Metric | Description | Warning Signs |
|
||||
|--------|-------------|---------------|
|
||||
| `active_clients` | Current SSE connections | Near limit |
|
||||
| `tcp_connections` | Current TCP connections | Near limit |
|
||||
| `total_connections` | All active connections | > 80% of max |
|
||||
|
||||
### Filter Metrics
|
||||
|
||||
Understand filtering effectiveness:
|
||||
|
||||
| Metric | Description | Optimization |
|
||||
|--------|-------------|--------------|
|
||||
| `total_processed` | Entries checked | - |
|
||||
| `total_passed` | Entries that passed | Very low = too restrictive |
|
||||
| `total_dropped` | Entries filtered out | Very high = review patterns |
|
||||
|
||||
### Rate Limit Metrics
|
||||
|
||||
Track rate limiting impact:
|
||||
|
||||
| Metric | Description | Action Needed |
|
||||
|--------|-------------|---------------|
|
||||
| `blocked_requests` | Rejected requests | High = increase limits |
|
||||
| `active_ips` | Unique clients | High = scale out |
|
||||
| `blocked_percentage` | Rejection rate | > 10% = review |
|
||||
|
||||
## Operational Logging
|
||||
|
||||
### Log Levels
|
||||
|
||||
Configure LogWisp's operational logging:
|
||||
|
||||
```toml
|
||||
[logging]
|
||||
output = "both" # file and stderr
|
||||
level = "info" # info for production
|
||||
```
|
||||
|
||||
Log levels and their use:
|
||||
- **DEBUG**: Detailed internal operations
|
||||
- **INFO**: Normal operations, connections
|
||||
- **WARN**: Recoverable issues
|
||||
- **ERROR**: Errors requiring attention
|
||||
|
||||
### Important Log Messages
|
||||
|
||||
#### Startup Messages
|
||||
```
|
||||
LogWisp starting version=1.0.0 config_file=/etc/logwisp.toml
|
||||
Stream registered with router stream=app
|
||||
TCP endpoint configured transport=system port=9090
|
||||
HTTP endpoints configured transport=app stream_url=http://localhost:8080/stream
|
||||
```
|
||||
|
||||
#### Connection Events
|
||||
```
|
||||
HTTP client connected remote_addr=192.168.1.100:54231 active_clients=6
|
||||
HTTP client disconnected remote_addr=192.168.1.100:54231 active_clients=5
|
||||
TCP connection opened remote_addr=192.168.1.100:54232 active_connections=3
|
||||
```
|
||||
|
||||
#### Error Conditions
|
||||
```
|
||||
Failed to open file for checking path=/var/log/app.log error=permission denied
|
||||
Scanner error while reading file path=/var/log/huge.log error=token too long
|
||||
Request rate limited ip=192.168.1.100
|
||||
Connection limit exceeded ip=192.168.1.100 connections=5 limit=5
|
||||
```
|
||||
|
||||
#### Performance Warnings
|
||||
```
|
||||
Dropped log entry - subscriber buffer full
|
||||
Dropped entry for slow client remote_addr=192.168.1.100
|
||||
Check interval too small: 5ms (min: 10ms)
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Basic Health Check
|
||||
|
||||
Simple up/down check:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# health_check.sh
|
||||
|
||||
STATUS=$(curl -s -o /dev/null -w "%{http_code}" http://localhost:8080/status)
|
||||
|
||||
if [ "$STATUS" -eq 200 ]; then
|
||||
echo "LogWisp is healthy"
|
||||
exit 0
|
||||
else
|
||||
echo "LogWisp is unhealthy (status: $STATUS)"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Advanced Health Check
|
||||
|
||||
Check specific conditions:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# advanced_health_check.sh
|
||||
|
||||
RESPONSE=$(curl -s http://localhost:8080/status)
|
||||
|
||||
# Check if processing logs
|
||||
ENTRIES=$(echo "$RESPONSE" | jq -r '.monitor.total_entries')
|
||||
if [ "$ENTRIES" -eq 0 ]; then
|
||||
echo "WARNING: No log entries processed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check dropped entries
|
||||
DROPPED=$(echo "$RESPONSE" | jq -r '.monitor.dropped_entries')
|
||||
TOTAL=$(echo "$RESPONSE" | jq -r '.monitor.total_entries')
|
||||
DROP_PERCENT=$(( DROPPED * 100 / TOTAL ))
|
||||
|
||||
if [ "$DROP_PERCENT" -gt 5 ]; then
|
||||
echo "WARNING: High drop rate: ${DROP_PERCENT}%"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check connections
|
||||
CONNECTIONS=$(echo "$RESPONSE" | jq -r '.server.active_clients')
|
||||
echo "OK: Processing logs, $CONNECTIONS active clients"
|
||||
exit 0
|
||||
```
|
||||
|
||||
### Container Health Check
|
||||
|
||||
Docker/Kubernetes configuration:
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
|
||||
CMD curl -f http://localhost:8080/status || exit 1
|
||||
```
|
||||
|
||||
```yaml
|
||||
# Kubernetes
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /status
|
||||
port: 8080
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 30
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /status
|
||||
port: 8080
|
||||
initialDelaySeconds: 5
|
||||
periodSeconds: 10
|
||||
```
|
||||
|
||||
## Monitoring Integration
|
||||
|
||||
### Prometheus Metrics
|
||||
|
||||
Export metrics in Prometheus format:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# prometheus_exporter.sh
|
||||
|
||||
while true; do
|
||||
STATUS=$(curl -s http://localhost:8080/status)
|
||||
|
||||
# Extract metrics
|
||||
CLIENTS=$(echo "$STATUS" | jq -r '.server.active_clients')
|
||||
ENTRIES=$(echo "$STATUS" | jq -r '.monitor.total_entries')
|
||||
DROPPED=$(echo "$STATUS" | jq -r '.monitor.dropped_entries')
|
||||
|
||||
# Output Prometheus format
|
||||
cat << EOF
|
||||
# HELP logwisp_active_clients Number of active streaming clients
|
||||
# TYPE logwisp_active_clients gauge
|
||||
logwisp_active_clients $CLIENTS
|
||||
|
||||
# HELP logwisp_total_entries Total log entries processed
|
||||
# TYPE logwisp_total_entries counter
|
||||
logwisp_total_entries $ENTRIES
|
||||
|
||||
# HELP logwisp_dropped_entries Total log entries dropped
|
||||
# TYPE logwisp_dropped_entries counter
|
||||
logwisp_dropped_entries $DROPPED
|
||||
EOF
|
||||
|
||||
sleep 60
|
||||
done
|
||||
```
|
||||
|
||||
### Grafana Dashboard
|
||||
|
||||
Key panels for Grafana:
|
||||
|
||||
1. **Active Connections**
|
||||
- Query: `logwisp_active_clients`
|
||||
- Visualization: Graph
|
||||
- Alert: > 80% of max
|
||||
|
||||
2. **Log Processing Rate**
|
||||
- Query: `rate(logwisp_total_entries[5m])`
|
||||
- Visualization: Graph
|
||||
- Alert: < 1 entry/min
|
||||
|
||||
3. **Drop Rate**
|
||||
- Query: `rate(logwisp_dropped_entries[5m]) / rate(logwisp_total_entries[5m])`
|
||||
- Visualization: Gauge
|
||||
- Alert: > 5%
|
||||
|
||||
4. **Rate Limit Rejections**
|
||||
- Query: `rate(logwisp_blocked_requests[5m])`
|
||||
- Visualization: Graph
|
||||
- Alert: > 10/min
|
||||
|
||||
### Datadog Integration
|
||||
|
||||
Send custom metrics:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# datadog_metrics.sh
|
||||
|
||||
while true; do
|
||||
STATUS=$(curl -s http://localhost:8080/status)
|
||||
|
||||
# Send metrics to Datadog
|
||||
echo "$STATUS" | jq -r '
|
||||
"logwisp.connections:\(.server.active_clients)|g",
|
||||
"logwisp.entries:\(.monitor.total_entries)|c",
|
||||
"logwisp.dropped:\(.monitor.dropped_entries)|c"
|
||||
' | while read metric; do
|
||||
echo "$metric" | nc -u -w1 localhost 8125
|
||||
done
|
||||
|
||||
sleep 60
|
||||
done
|
||||
```
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### CPU Usage
|
||||
|
||||
Monitor CPU usage by component:
|
||||
|
||||
```bash
|
||||
# Check process CPU
|
||||
top -p $(pgrep logwisp) -b -n 1
|
||||
|
||||
# Profile CPU usage
|
||||
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30
|
||||
```
|
||||
|
||||
Common CPU consumers:
|
||||
- File watching (reduce check_interval_ms)
|
||||
- Regex filtering (simplify patterns)
|
||||
- JSON encoding (reduce clients)
|
||||
|
||||
### Memory Usage
|
||||
|
||||
Track memory consumption:
|
||||
|
||||
```bash
|
||||
# Check process memory
|
||||
ps aux | grep logwisp
|
||||
|
||||
# Detailed memory stats
|
||||
cat /proc/$(pgrep logwisp)/status | grep -E "Vm(RSS|Size)"
|
||||
```
|
||||
|
||||
Memory optimization:
|
||||
- Reduce buffer sizes
|
||||
- Limit connections
|
||||
- Simplify filters
|
||||
|
||||
### Network Bandwidth
|
||||
|
||||
Monitor streaming bandwidth:
|
||||
|
||||
```bash
|
||||
# Network statistics
|
||||
netstat -i
|
||||
iftop -i eth0 -f "port 8080"
|
||||
|
||||
# Connection count
|
||||
ss -tan | grep :8080 | wc -l
|
||||
```
|
||||
|
||||
## Alerting
|
||||
|
||||
### Basic Alerts
|
||||
|
||||
Essential alerts to configure:
|
||||
|
||||
| Alert | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| Service Down | Status endpoint fails | Critical |
|
||||
| High Drop Rate | > 10% entries dropped | Warning |
|
||||
| No Log Activity | 0 entries/min for 5 min | Warning |
|
||||
| Connection Limit | > 90% of max connections | Warning |
|
||||
| Rate Limit High | > 20% requests blocked | Warning |
|
||||
|
||||
### Alert Script
|
||||
|
||||
Example monitoring script:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# monitor_alerts.sh
|
||||
|
||||
check_alert() {
|
||||
local name=$1
|
||||
local condition=$2
|
||||
local message=$3
|
||||
|
||||
if eval "$condition"; then
|
||||
echo "ALERT: $name - $message"
|
||||
# Send to alerting system
|
||||
# curl -X POST https://alerts.example.com/...
|
||||
fi
|
||||
}
|
||||
|
||||
while true; do
|
||||
STATUS=$(curl -s http://localhost:8080/status)
|
||||
|
||||
if [ -z "$STATUS" ]; then
|
||||
check_alert "SERVICE_DOWN" "true" "LogWisp not responding"
|
||||
sleep 60
|
||||
continue
|
||||
fi
|
||||
|
||||
# Extract metrics
|
||||
DROPPED=$(echo "$STATUS" | jq -r '.monitor.dropped_entries')
|
||||
TOTAL=$(echo "$STATUS" | jq -r '.monitor.total_entries')
|
||||
CLIENTS=$(echo "$STATUS" | jq -r '.server.active_clients')
|
||||
|
||||
# Check conditions
|
||||
check_alert "HIGH_DROP_RATE" \
|
||||
"[ $((DROPPED * 100 / TOTAL)) -gt 10 ]" \
|
||||
"Drop rate above 10%"
|
||||
|
||||
check_alert "HIGH_CONNECTIONS" \
|
||||
"[ $CLIENTS -gt 90 ]" \
|
||||
"Near connection limit: $CLIENTS/100"
|
||||
|
||||
sleep 60
|
||||
done
|
||||
```
|
||||
|
||||
## Troubleshooting with Monitoring
|
||||
|
||||
### No Logs Appearing
|
||||
|
||||
Check monitor stats:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.monitor'
|
||||
```
|
||||
|
||||
Look for:
|
||||
- `active_watchers` = 0 (no files found)
|
||||
- `total_entries` not increasing (files not updating)
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
Enable debug logging:
|
||||
```bash
|
||||
logwisp --log-level debug --log-output stderr
|
||||
```
|
||||
|
||||
Watch for:
|
||||
- Frequent "checkFile" messages (reduce check_interval)
|
||||
- Many filter operations (optimize patterns)
|
||||
|
||||
### Memory Growth
|
||||
|
||||
Monitor over time:
|
||||
```bash
|
||||
while true; do
|
||||
ps aux | grep logwisp | grep -v grep
|
||||
curl -s http://localhost:8080/status | jq '.server.active_clients'
|
||||
sleep 10
|
||||
done
|
||||
```
|
||||
|
||||
### Connection Issues
|
||||
|
||||
Check connection stats:
|
||||
```bash
|
||||
# Current connections
|
||||
curl -s http://localhost:8080/status | jq '.server'
|
||||
|
||||
# Rate limit stats
|
||||
curl -s http://localhost:8080/status | jq '.features.rate_limit'
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Regular Monitoring**: Check status endpoints every 30-60 seconds
|
||||
2. **Set Alerts**: Configure alerts for critical conditions
|
||||
3. **Log Rotation**: Rotate LogWisp's own logs to prevent disk fill
|
||||
4. **Baseline Metrics**: Establish normal ranges for your environment
|
||||
5. **Capacity Planning**: Monitor trends for scaling decisions
|
||||
6. **Test Monitoring**: Verify alerts work before issues occur
|
||||
|
||||
## See Also
|
||||
|
||||
- [Performance Tuning](performance.md) - Optimization guide
|
||||
- [Troubleshooting](troubleshooting.md) - Common issues
|
||||
- [Configuration Guide](configuration.md) - Monitoring configuration
|
||||
- [Integration Examples](integrations.md) - Monitoring system integration
|
||||
@ -1,26 +1,20 @@
|
||||
# Quick Start Guide
|
||||
|
||||
Get LogWisp up and running in 5 minutes!
|
||||
Get LogWisp up and running in 5 minutes with the new pipeline architecture!
|
||||
|
||||
## Installation
|
||||
|
||||
### From Source
|
||||
```bash
|
||||
# Clone the repository
|
||||
git clone https://github.com/yourusername/logwisp.git
|
||||
git clone https://github.com/lixenwraith/logwisp.git
|
||||
cd logwisp
|
||||
|
||||
# Build and install
|
||||
make install
|
||||
|
||||
# Or just build
|
||||
make build
|
||||
./logwisp --version
|
||||
```
|
||||
|
||||
### Using Go Install
|
||||
|
||||
```bash
|
||||
go install github.com/yourusername/logwisp/src/cmd/logwisp@latest
|
||||
go install github.com/lixenwraith/logwisp/src/cmd/logwisp@latest
|
||||
```
|
||||
|
||||
## Basic Usage
|
||||
@ -35,22 +29,19 @@ logwisp
|
||||
|
||||
### 2. Stream Logs
|
||||
|
||||
In another terminal, connect to the log stream:
|
||||
Connect to the log stream:
|
||||
|
||||
```bash
|
||||
# Using curl (SSE stream)
|
||||
# SSE stream
|
||||
curl -N http://localhost:8080/stream
|
||||
|
||||
# Check status
|
||||
curl http://localhost:8080/status | jq .
|
||||
```
|
||||
|
||||
### 3. Create Some Logs
|
||||
|
||||
Generate test logs to see streaming in action:
|
||||
### 3. Generate Test Logs
|
||||
|
||||
```bash
|
||||
# In a third terminal
|
||||
echo "[ERROR] Something went wrong!" >> test.log
|
||||
echo "[INFO] Application started" >> test.log
|
||||
echo "[WARN] Low memory warning" >> test.log
|
||||
@ -60,88 +51,78 @@ echo "[WARN] Low memory warning" >> test.log
|
||||
|
||||
### Monitor Specific Directory
|
||||
|
||||
Create a configuration file `~/.config/logwisp.toml`:
|
||||
Create `~/.config/logwisp.toml`:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "myapp"
|
||||
|
||||
[streams.monitor]
|
||||
targets = [
|
||||
{ path = "/var/log/myapp", pattern = "*.log", is_file = false }
|
||||
]
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/myapp", pattern = "*.log" }
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
```
|
||||
|
||||
Run LogWisp:
|
||||
```bash
|
||||
logwisp
|
||||
```
|
||||
|
||||
### Filter Only Errors and Warnings
|
||||
|
||||
Add filters to your configuration:
|
||||
### Filter Only Errors
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "errors"
|
||||
|
||||
[streams.monitor]
|
||||
targets = [
|
||||
{ path = "./", pattern = "*.log" }
|
||||
]
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "./", pattern = "*.log" }
|
||||
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN", "CRITICAL", "FATAL"]
|
||||
patterns = ["ERROR", "WARN", "CRITICAL"]
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
```
|
||||
|
||||
### Multiple Log Sources
|
||||
### Multiple Outputs
|
||||
|
||||
Monitor different applications on different ports:
|
||||
Send logs to both HTTP stream and file:
|
||||
|
||||
```toml
|
||||
# Stream 1: Web application
|
||||
[[streams]]
|
||||
name = "webapp"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/nginx", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines]]
|
||||
name = "multi-output"
|
||||
|
||||
# Stream 2: Database
|
||||
[[streams]]
|
||||
name = "database"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/postgresql", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8081
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log" }
|
||||
|
||||
# HTTP streaming
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
|
||||
# File archival
|
||||
[[pipelines.sinks]]
|
||||
type = "file"
|
||||
options = { directory = "/var/log/archive", name = "app" }
|
||||
```
|
||||
|
||||
### TCP Streaming
|
||||
|
||||
For high-performance streaming, use TCP:
|
||||
For high-performance streaming:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "highperf"
|
||||
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/app", pattern = "*.log" }]
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log" }
|
||||
|
||||
[streams.tcpserver]
|
||||
enabled = true
|
||||
port = 9090
|
||||
buffer_size = 5000
|
||||
[[pipelines.sinks]]
|
||||
type = "tcp"
|
||||
options = { port = 9090, buffer_size = 5000 }
|
||||
```
|
||||
|
||||
Connect with netcat:
|
||||
@ -151,16 +132,15 @@ nc localhost 9090
|
||||
|
||||
### Router Mode
|
||||
|
||||
Consolidate multiple streams on one port using router mode:
|
||||
Run multiple pipelines on shared ports:
|
||||
|
||||
```bash
|
||||
# With the multi-stream config above
|
||||
logwisp --router
|
||||
|
||||
# Access streams at:
|
||||
# http://localhost:8080/webapp/stream
|
||||
# http://localhost:8080/database/stream
|
||||
# http://localhost:8080/status (global status)
|
||||
# Access pipelines at:
|
||||
# http://localhost:8080/myapp/stream
|
||||
# http://localhost:8080/errors/stream
|
||||
# http://localhost:8080/status (global)
|
||||
```
|
||||
|
||||
## Quick Tips
|
||||
@ -170,40 +150,23 @@ logwisp --router
|
||||
logwisp --log-level debug --log-output stderr
|
||||
```
|
||||
|
||||
### Run in Background
|
||||
```bash
|
||||
logwisp --background --config /etc/logwisp/prod.toml
|
||||
```
|
||||
|
||||
### Rate Limiting
|
||||
Protect your streams from abuse:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 10.0
|
||||
burst_size = 20
|
||||
max_connections_per_ip = 5
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = {
|
||||
port = 8080,
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 10.0,
|
||||
burst_size = 20
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### JSON Output Format
|
||||
For structured logging:
|
||||
|
||||
### Console Output
|
||||
```toml
|
||||
[logging.console]
|
||||
format = "json"
|
||||
```
|
||||
|
||||
## What's Next?
|
||||
|
||||
- Read the [Configuration Guide](configuration.md) for all options
|
||||
- Learn about [Filters](filters.md) for advanced pattern matching
|
||||
- Explore [Rate Limiting](ratelimiting.md) for production deployments
|
||||
- Check out [Example Configurations](examples/) for more scenarios
|
||||
|
||||
## Getting Help
|
||||
|
||||
- Run `logwisp --help` for CLI options
|
||||
- Check `http://localhost:8080/status` for runtime statistics
|
||||
- Enable debug logging for troubleshooting
|
||||
- Visit our [GitHub repository](https://github.com/yourusername/logwisp) for issues and discussions
|
||||
[[pipelines.sinks]]
|
||||
type = "stdout" # or "stderr"
|
||||
options = {}
|
||||
```
|
||||
@ -1,136 +1,65 @@
|
||||
# Rate Limiting Guide
|
||||
|
||||
LogWisp provides configurable rate limiting to protect against abuse, prevent resource exhaustion, and ensure fair access to log streams.
|
||||
LogWisp provides configurable rate limiting to protect against abuse and ensure fair access.
|
||||
|
||||
## How Rate Limiting Works
|
||||
## How It Works
|
||||
|
||||
LogWisp uses a **token bucket algorithm** for smooth, burst-tolerant rate limiting:
|
||||
|
||||
1. Each client (or globally) gets a bucket with a fixed capacity
|
||||
2. Tokens are added to the bucket at a configured rate
|
||||
Token bucket algorithm:
|
||||
1. Each client gets a bucket with fixed capacity
|
||||
2. Tokens refill at configured rate
|
||||
3. Each request consumes one token
|
||||
4. If no tokens are available, the request is rejected
|
||||
5. The bucket can accumulate tokens up to its capacity for bursts
|
||||
4. No tokens = request rejected
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Configuration
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true # Enable rate limiting
|
||||
requests_per_second = 10.0 # Token refill rate
|
||||
burst_size = 20 # Maximum tokens (bucket capacity)
|
||||
limit_by = "ip" # "ip" or "global"
|
||||
[[pipelines.sinks]]
|
||||
type = "http" # or "tcp"
|
||||
options = {
|
||||
port = 8080,
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 10.0,
|
||||
burst_size = 20,
|
||||
limit_by = "ip", # or "global"
|
||||
max_connections_per_ip = 5,
|
||||
max_total_connections = 100,
|
||||
response_code = 429,
|
||||
response_message = "Rate limit exceeded"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Complete Options
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
# Core settings
|
||||
enabled = true # Enable/disable rate limiting
|
||||
requests_per_second = 10.0 # Token generation rate (float)
|
||||
burst_size = 20 # Token bucket capacity
|
||||
|
||||
# Limiting strategy
|
||||
limit_by = "ip" # "ip" or "global"
|
||||
|
||||
# Connection limits
|
||||
max_connections_per_ip = 5 # Max concurrent connections per IP
|
||||
max_total_connections = 100 # Max total concurrent connections
|
||||
|
||||
# Response configuration
|
||||
response_code = 429 # HTTP status code when limited
|
||||
response_message = "Rate limit exceeded" # Error message
|
||||
|
||||
# Same options available for TCP
|
||||
[streams.tcpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 5.0
|
||||
burst_size = 10
|
||||
limit_by = "ip"
|
||||
```
|
||||
|
||||
## Limiting Strategies
|
||||
## Strategies
|
||||
|
||||
### Per-IP Limiting (Default)
|
||||
|
||||
Each client IP address gets its own token bucket:
|
||||
|
||||
Each IP gets its own bucket:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
limit_by = "ip"
|
||||
requests_per_second = 10.0
|
||||
burst_size = 20
|
||||
# Client A: 10 req/sec
|
||||
# Client B: 10 req/sec
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Fair access for multiple users
|
||||
- Prevent single client from monopolizing resources
|
||||
- Public-facing endpoints
|
||||
|
||||
**Example behavior:**
|
||||
- Client A: Can make 10 req/sec
|
||||
- Client B: Also can make 10 req/sec
|
||||
- Total: Up to 10 × number of clients
|
||||
|
||||
### Global Limiting
|
||||
|
||||
All clients share a single token bucket:
|
||||
|
||||
All clients share one bucket:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
limit_by = "global"
|
||||
requests_per_second = 50.0
|
||||
burst_size = 100
|
||||
# All clients combined: 50 req/sec
|
||||
```
|
||||
|
||||
**Use cases:**
|
||||
- Protect backend resources
|
||||
- Control total system load
|
||||
- Internal services with known clients
|
||||
|
||||
**Example behavior:**
|
||||
- All clients combined: 50 req/sec max
|
||||
- One aggressive client can consume all tokens
|
||||
|
||||
## Connection Limits
|
||||
|
||||
In addition to request rate limiting, you can limit concurrent connections:
|
||||
|
||||
### Per-IP Connection Limit
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
max_connections_per_ip = 5 # Each IP can have max 5 connections
|
||||
max_connections_per_ip = 5 # Per IP
|
||||
max_total_connections = 100 # Total
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Prevents connection exhaustion attacks
|
||||
- Limits resource usage per client
|
||||
- Checked before rate limits
|
||||
|
||||
### Total Connection Limit
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
max_total_connections = 100 # Max 100 connections total
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
- Protects server resources
|
||||
- Prevents memory exhaustion
|
||||
- Global limit across all IPs
|
||||
|
||||
## Response Behavior
|
||||
|
||||
### HTTP Responses
|
||||
|
||||
When rate limited, HTTP clients receive:
|
||||
|
||||
### HTTP
|
||||
Returns JSON with configured status:
|
||||
```json
|
||||
{
|
||||
"error": "Rate limit exceeded",
|
||||
@ -138,389 +67,59 @@ When rate limited, HTTP clients receive:
|
||||
}
|
||||
```
|
||||
|
||||
With these headers:
|
||||
- Status code: 429 (default) or configured value
|
||||
- Content-Type: application/json
|
||||
### TCP
|
||||
Connections silently dropped.
|
||||
|
||||
Configure custom responses:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
response_code = 503 # Service Unavailable
|
||||
response_message = "Server overloaded, please retry later"
|
||||
```
|
||||
|
||||
### TCP Behavior
|
||||
|
||||
TCP connections are **silently dropped** when rate limited:
|
||||
- No error message sent
|
||||
- Connection immediately closed
|
||||
- Prevents information leakage
|
||||
|
||||
## Configuration Examples
|
||||
## Examples
|
||||
|
||||
### Light Protection
|
||||
|
||||
For internal or trusted environments:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 50.0
|
||||
burst_size = 100
|
||||
limit_by = "ip"
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 50.0,
|
||||
burst_size = 100
|
||||
}
|
||||
```
|
||||
|
||||
### Moderate Protection
|
||||
|
||||
For semi-public endpoints:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 10.0
|
||||
burst_size = 30
|
||||
limit_by = "ip"
|
||||
max_connections_per_ip = 5
|
||||
max_total_connections = 200
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 10.0,
|
||||
burst_size = 30,
|
||||
max_connections_per_ip = 5
|
||||
}
|
||||
```
|
||||
|
||||
### Strict Protection
|
||||
|
||||
For public or sensitive endpoints:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 2.0
|
||||
burst_size = 5
|
||||
limit_by = "ip"
|
||||
max_connections_per_ip = 2
|
||||
max_total_connections = 50
|
||||
response_code = 503
|
||||
response_message = "Service temporarily unavailable"
|
||||
```
|
||||
|
||||
### Debug/Development
|
||||
|
||||
Disable for testing:
|
||||
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = false
|
||||
```
|
||||
|
||||
## Use Case Scenarios
|
||||
|
||||
### Public Log Viewer
|
||||
|
||||
Prevent abuse while allowing legitimate use:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "public-logs"
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 5.0 # 5 new connections per second
|
||||
burst_size = 10 # Allow short bursts
|
||||
limit_by = "ip"
|
||||
max_connections_per_ip = 3 # Max 3 streams per user
|
||||
max_total_connections = 100
|
||||
```
|
||||
|
||||
### Internal Monitoring
|
||||
|
||||
Protect against accidental overload:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "internal-metrics"
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8081
|
||||
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 100.0 # High limit for internal use
|
||||
burst_size = 200
|
||||
limit_by = "global" # Total system limit
|
||||
max_total_connections = 500
|
||||
```
|
||||
|
||||
### High-Security Audit Logs
|
||||
|
||||
Very restrictive access:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "audit"
|
||||
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8443
|
||||
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 0.5 # 1 request every 2 seconds
|
||||
burst_size = 2
|
||||
limit_by = "ip"
|
||||
max_connections_per_ip = 1 # Single connection only
|
||||
max_total_connections = 10
|
||||
response_code = 403 # Forbidden (hide rate limit)
|
||||
response_message = "Access denied"
|
||||
```
|
||||
|
||||
### Multi-Tenant Service
|
||||
|
||||
Different limits per stream:
|
||||
|
||||
```toml
|
||||
# Free tier
|
||||
[[streams]]
|
||||
name = "logs-free"
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 1.0
|
||||
burst_size = 5
|
||||
max_connections_per_ip = 1
|
||||
|
||||
# Premium tier
|
||||
[[streams]]
|
||||
name = "logs-premium"
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 50.0
|
||||
burst_size = 100
|
||||
max_connections_per_ip = 10
|
||||
```
|
||||
|
||||
## Monitoring Rate Limits
|
||||
|
||||
### Status Endpoint
|
||||
|
||||
Check rate limit statistics:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/status | jq '.server.features.rate_limit'
|
||||
```
|
||||
|
||||
Response includes:
|
||||
```json
|
||||
{
|
||||
"enabled": true,
|
||||
"total_requests": 15234,
|
||||
"blocked_requests": 89,
|
||||
"active_ips": 12,
|
||||
"total_connections": 8,
|
||||
"config": {
|
||||
"requests_per_second": 10,
|
||||
"burst_size": 20,
|
||||
"limit_by": "ip"
|
||||
}
|
||||
rate_limit = {
|
||||
enabled = true,
|
||||
requests_per_second = 2.0,
|
||||
burst_size = 5,
|
||||
max_connections_per_ip = 2,
|
||||
response_code = 503
|
||||
}
|
||||
```
|
||||
|
||||
### Debug Logging
|
||||
## Monitoring
|
||||
|
||||
Enable debug logs to see rate limit decisions:
|
||||
Check statistics:
|
||||
```bash
|
||||
curl http://localhost:8080/status | jq '.sinks[0].details.rate_limit'
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
logwisp --log-level debug
|
||||
```
|
||||
|
||||
Look for messages:
|
||||
```
|
||||
Request rate limited ip=192.168.1.100
|
||||
Connection limit exceeded ip=192.168.1.100 connections=5 limit=5
|
||||
Created new IP limiter ip=192.168.1.100 total_ips=3
|
||||
```
|
||||
|
||||
## Testing Rate Limits
|
||||
|
||||
### Test Script
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# Test rate limiting behavior
|
||||
|
||||
URL="http://localhost:8080/stream"
|
||||
PARALLEL=10
|
||||
DURATION=10
|
||||
|
||||
echo "Testing rate limits..."
|
||||
echo "URL: $URL"
|
||||
echo "Parallel connections: $PARALLEL"
|
||||
echo "Duration: ${DURATION}s"
|
||||
echo
|
||||
|
||||
# Function to connect and count lines
|
||||
test_connection() {
|
||||
local id=$1
|
||||
local count=0
|
||||
local start=$(date +%s)
|
||||
|
||||
while (( $(date +%s) - start < DURATION )); do
|
||||
if curl -s -N --max-time 1 "$URL" >/dev/null 2>&1; then
|
||||
((count++))
|
||||
echo "[$id] Connected successfully (total: $count)"
|
||||
else
|
||||
echo "[$id] Rate limited!"
|
||||
fi
|
||||
sleep 0.1
|
||||
done
|
||||
}
|
||||
|
||||
# Run parallel connections
|
||||
for i in $(seq 1 $PARALLEL); do
|
||||
test_connection $i &
|
||||
done
|
||||
|
||||
wait
|
||||
echo "Test complete"
|
||||
```
|
||||
|
||||
### Load Testing
|
||||
|
||||
Using Apache Bench (ab):
|
||||
|
||||
```bash
|
||||
# Test burst handling
|
||||
ab -n 100 -c 20 http://localhost:8080/status
|
||||
|
||||
# Test sustained load
|
||||
ab -n 1000 -c 5 -r http://localhost:8080/status
|
||||
```
|
||||
|
||||
Using curl:
|
||||
|
||||
```bash
|
||||
# Test connection limit
|
||||
for i in {1..10}; do
|
||||
curl -N http://localhost:8080/stream &
|
||||
# Test rate limits
|
||||
for i in {1..20}; do
|
||||
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/status
|
||||
done
|
||||
```
|
||||
|
||||
## Tuning Guidelines
|
||||
## Tuning
|
||||
|
||||
### Setting requests_per_second
|
||||
|
||||
Consider:
|
||||
- Expected legitimate traffic
|
||||
- Server capacity
|
||||
- Client retry behavior
|
||||
|
||||
**Formula**: `requests_per_second = expected_clients × requests_per_client`
|
||||
|
||||
### Setting burst_size
|
||||
|
||||
General rule: `burst_size = 2-3 × requests_per_second`
|
||||
|
||||
Examples:
|
||||
- `10 req/s → burst_size = 20-30`
|
||||
- `1 req/s → burst_size = 3-5`
|
||||
- `100 req/s → burst_size = 200-300`
|
||||
|
||||
### Connection Limits
|
||||
|
||||
Based on available memory:
|
||||
- Each HTTP connection: ~1-2MB
|
||||
- Each TCP connection: ~0.5-1MB
|
||||
|
||||
**Formula**: `max_connections = available_memory / memory_per_connection`
|
||||
|
||||
## Common Issues
|
||||
|
||||
### "All requests blocked"
|
||||
|
||||
Check if:
|
||||
- Rate limits too strict
|
||||
- Burst size too small
|
||||
- Using global limiting with many clients
|
||||
|
||||
### "Memory growth"
|
||||
|
||||
Possible causes:
|
||||
- No connection limits set
|
||||
- Slow clients holding connections
|
||||
- Too high burst_size
|
||||
|
||||
Solutions:
|
||||
```toml
|
||||
max_connections_per_ip = 5
|
||||
max_total_connections = 100
|
||||
```
|
||||
|
||||
### "Legitimate users blocked"
|
||||
|
||||
Consider:
|
||||
- Increasing burst_size for short spikes
|
||||
- Using per-IP instead of global limiting
|
||||
- Different streams for different user tiers
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Information Disclosure
|
||||
|
||||
Rate limit responses can reveal information:
|
||||
|
||||
```toml
|
||||
# Default - informative
|
||||
response_code = 429
|
||||
response_message = "Rate limit exceeded"
|
||||
|
||||
# Security-focused - generic
|
||||
response_code = 503
|
||||
response_message = "Service unavailable"
|
||||
|
||||
# High security - misleading
|
||||
response_code = 403
|
||||
response_message = "Forbidden"
|
||||
```
|
||||
|
||||
### DDoS Protection
|
||||
|
||||
Rate limiting helps but isn't complete DDoS protection:
|
||||
- Use with firewall rules
|
||||
- Consider CDN/proxy rate limiting
|
||||
- Monitor for distributed attacks
|
||||
|
||||
### Resource Exhaustion
|
||||
|
||||
Protect against:
|
||||
- Connection exhaustion
|
||||
- Memory exhaustion
|
||||
- CPU exhaustion
|
||||
|
||||
```toml
|
||||
# Comprehensive protection
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 10.0
|
||||
burst_size = 20
|
||||
max_connections_per_ip = 5
|
||||
max_total_connections = 100
|
||||
limit_by = "ip"
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start Conservative**: Begin with strict limits and relax as needed
|
||||
2. **Monitor Statistics**: Use `/status` endpoint to track behavior
|
||||
3. **Test Thoroughly**: Verify limits work as expected under load
|
||||
4. **Document Limits**: Make rate limits clear to users
|
||||
5. **Provide Retry Info**: Help clients implement proper retry logic
|
||||
6. **Different Tiers**: Consider different limits for different user types
|
||||
7. **Regular Review**: Adjust limits based on usage patterns
|
||||
|
||||
## See Also
|
||||
|
||||
- [Configuration Guide](configuration.md) - Complete configuration reference
|
||||
- [Security Best Practices](security.md) - Security hardening
|
||||
- [Performance Tuning](performance.md) - Optimization guidelines
|
||||
- [Troubleshooting](troubleshooting.md) - Common issues
|
||||
- **requests_per_second**: Expected load
|
||||
- **burst_size**: 2-3× requests_per_second
|
||||
- **Connection limits**: Based on memory
|
||||
552
doc/router.md
552
doc/router.md
@ -1,520 +1,158 @@
|
||||
# Router Mode Guide
|
||||
|
||||
Router mode allows multiple LogWisp streams to share HTTP ports through path-based routing, simplifying deployment and access control.
|
||||
Router mode enables multiple pipelines to share HTTP ports through path-based routing.
|
||||
|
||||
## Overview
|
||||
|
||||
In standard mode, each stream requires its own port:
|
||||
- Stream 1: `http://localhost:8080/stream`
|
||||
- Stream 2: `http://localhost:8081/stream`
|
||||
- Stream 3: `http://localhost:8082/stream`
|
||||
**Standard mode**: Each pipeline needs its own port
|
||||
- Pipeline 1: `http://localhost:8080/stream`
|
||||
- Pipeline 2: `http://localhost:8081/stream`
|
||||
|
||||
In router mode, streams share ports via paths:
|
||||
- Stream 1: `http://localhost:8080/app/stream`
|
||||
- Stream 2: `http://localhost:8080/database/stream`
|
||||
- Stream 3: `http://localhost:8080/system/stream`
|
||||
**Router mode**: Pipelines share ports via paths
|
||||
- Pipeline 1: `http://localhost:8080/app/stream`
|
||||
- Pipeline 2: `http://localhost:8080/database/stream`
|
||||
- Global status: `http://localhost:8080/status`
|
||||
|
||||
## Enabling Router Mode
|
||||
|
||||
Start LogWisp with the `--router` flag:
|
||||
|
||||
```bash
|
||||
logwisp --router --config /etc/logwisp/multi-stream.toml
|
||||
logwisp --router --config /etc/logwisp/multi-pipeline.toml
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
### Basic Router Configuration
|
||||
|
||||
```toml
|
||||
# All streams can use the same port in router mode
|
||||
[[streams]]
|
||||
# All pipelines can use the same port
|
||||
[[pipelines]]
|
||||
name = "app"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/app", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # Same port OK
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/app", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 } # Same port OK
|
||||
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "database"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/postgresql", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # Shared port
|
||||
|
||||
[[streams]]
|
||||
name = "nginx"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/nginx", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # Shared port
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/postgresql", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 } # Shared port
|
||||
```
|
||||
|
||||
### Path Structure
|
||||
## Path Structure
|
||||
|
||||
In router mode, paths are automatically prefixed with the stream name:
|
||||
Paths are prefixed with pipeline name:
|
||||
|
||||
| Stream Name | Configuration Path | Router Mode Path |
|
||||
|------------|-------------------|------------------|
|
||||
| Pipeline | Config Path | Router Path |
|
||||
|----------|-------------|-------------|
|
||||
| `app` | `/stream` | `/app/stream` |
|
||||
| `app` | `/status` | `/app/status` |
|
||||
| `database` | `/stream` | `/database/stream` |
|
||||
| `database` | `/status` | `/database/status` |
|
||||
|
||||
### Custom Paths
|
||||
|
||||
You can customize the paths in each stream:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "api"
|
||||
[streams.httpserver]
|
||||
stream_path = "/logs" # Becomes /api/logs
|
||||
status_path = "/health" # Becomes /api/health
|
||||
```
|
||||
|
||||
## URL Endpoints
|
||||
|
||||
### Stream Endpoints
|
||||
|
||||
Access individual streams:
|
||||
|
||||
```bash
|
||||
# SSE stream for 'app' logs
|
||||
curl -N http://localhost:8080/app/stream
|
||||
|
||||
# Status for 'database' stream
|
||||
curl http://localhost:8080/database/status
|
||||
|
||||
# Custom path example
|
||||
curl -N http://localhost:8080/api/logs
|
||||
```
|
||||
|
||||
### Global Status
|
||||
|
||||
Router mode provides a global status endpoint:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8080/status | jq .
|
||||
```
|
||||
|
||||
Returns aggregated information:
|
||||
```json
|
||||
{
|
||||
"service": "LogWisp Router",
|
||||
"version": "1.0.0",
|
||||
"port": 8080,
|
||||
"total_streams": 3,
|
||||
"streams": {
|
||||
"app": { /* stream stats */ },
|
||||
"database": { /* stream stats */ },
|
||||
"nginx": { /* stream stats */ }
|
||||
},
|
||||
"router": {
|
||||
"uptime_seconds": 3600,
|
||||
"total_requests": 15234,
|
||||
"routed_requests": 15220,
|
||||
"failed_requests": 14
|
||||
}
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = {
|
||||
stream_path = "/logs", # Becomes /app/logs
|
||||
status_path = "/health" # Becomes /app/health
|
||||
}
|
||||
```
|
||||
|
||||
## Port Sharing
|
||||
## Endpoints
|
||||
|
||||
### How It Works
|
||||
### Pipeline Endpoints
|
||||
```bash
|
||||
# SSE stream
|
||||
curl -N http://localhost:8080/app/stream
|
||||
|
||||
1. Router server listens on configured ports
|
||||
2. Examines request path to determine target stream
|
||||
3. Routes request to appropriate stream handler
|
||||
4. Stream handles request as if standalone
|
||||
# Pipeline status
|
||||
curl http://localhost:8080/database/status
|
||||
```
|
||||
|
||||
### Port Assignment Rules
|
||||
### Global Status
|
||||
```bash
|
||||
curl http://localhost:8080/status
|
||||
```
|
||||
|
||||
In router mode:
|
||||
- Multiple streams can use the same port
|
||||
- Router detects and consolidates shared ports
|
||||
- Each unique port gets one router server
|
||||
- TCP servers remain independent (no routing)
|
||||
|
||||
Example with multiple ports:
|
||||
|
||||
```toml
|
||||
# Streams 1-3 share port 8080
|
||||
[[streams]]
|
||||
name = "app"
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
|
||||
[[streams]]
|
||||
name = "db"
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
|
||||
[[streams]]
|
||||
name = "web"
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
|
||||
# Stream 4 uses different port
|
||||
[[streams]]
|
||||
name = "admin"
|
||||
[streams.httpserver]
|
||||
port = 9090
|
||||
|
||||
# Result: 2 router servers (8080 and 9090)
|
||||
Returns:
|
||||
```json
|
||||
{
|
||||
"service": "LogWisp Router",
|
||||
"pipelines": {
|
||||
"app": { /* stats */ },
|
||||
"database": { /* stats */ }
|
||||
},
|
||||
"total_pipelines": 2
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
|
||||
### Microservices Architecture
|
||||
|
||||
Route logs from different services:
|
||||
|
||||
### Microservices
|
||||
```toml
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "frontend"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/frontend", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/frontend", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "backend"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/backend", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
[[pipelines.sources]]
|
||||
type = "directory"
|
||||
options = { path = "/var/log/backend", pattern = "*.log" }
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
|
||||
[[streams]]
|
||||
name = "worker"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/var/log/worker", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080
|
||||
# Access:
|
||||
# http://localhost:8080/frontend/stream
|
||||
# http://localhost:8080/backend/stream
|
||||
```
|
||||
|
||||
Access via:
|
||||
- Frontend logs: `http://localhost:8080/frontend/stream`
|
||||
- Backend logs: `http://localhost:8080/backend/stream`
|
||||
- Worker logs: `http://localhost:8080/worker/stream`
|
||||
|
||||
### Environment-Based Routing
|
||||
|
||||
Different log levels per environment:
|
||||
|
||||
### Environment-Based
|
||||
```toml
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "prod"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/prod", pattern = "*.log" }]
|
||||
[[streams.filters]]
|
||||
[[pipelines.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
|
||||
[[streams]]
|
||||
name = "staging"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/staging", pattern = "*.log" }]
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN", "INFO"]
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
|
||||
[[streams]]
|
||||
[[pipelines]]
|
||||
name = "dev"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/dev", pattern = "*.log" }]
|
||||
# No filters - all logs
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
```
|
||||
|
||||
### Department Access
|
||||
|
||||
Separate streams for different teams:
|
||||
|
||||
```toml
|
||||
[[streams]]
|
||||
name = "engineering"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/apps", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 50.0
|
||||
|
||||
[[streams]]
|
||||
name = "security"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/audit", pattern = "*.log" }]
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 5.0
|
||||
max_connections_per_ip = 1
|
||||
|
||||
[[streams]]
|
||||
name = "support"
|
||||
[streams.monitor]
|
||||
targets = [{ path = "/logs/customer", pattern = "*.log" }]
|
||||
[[streams.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["password", "token", "secret"]
|
||||
[streams.httpserver]
|
||||
port = 8080
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Mixed Mode Deployment
|
||||
|
||||
Combine router and standalone modes:
|
||||
|
||||
```toml
|
||||
# Public streams via router
|
||||
[[streams]]
|
||||
name = "public-api"
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # Router mode
|
||||
|
||||
[[streams]]
|
||||
name = "public-web"
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 8080 # Router mode
|
||||
|
||||
# Internal stream standalone
|
||||
[[streams]]
|
||||
name = "internal"
|
||||
[streams.httpserver]
|
||||
enabled = true
|
||||
port = 9999 # Different port, standalone
|
||||
|
||||
# High-performance TCP
|
||||
[[streams]]
|
||||
name = "metrics"
|
||||
[streams.tcpserver]
|
||||
enabled = true
|
||||
port = 9090 # TCP not affected by router
|
||||
```
|
||||
|
||||
### Load Balancer Integration
|
||||
|
||||
Router mode works well with load balancers:
|
||||
|
||||
```nginx
|
||||
# Nginx configuration
|
||||
upstream logwisp {
|
||||
server logwisp1:8080;
|
||||
server logwisp2:8080;
|
||||
server logwisp3:8080;
|
||||
}
|
||||
|
||||
location /logs/ {
|
||||
proxy_pass http://logwisp/;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Connection "";
|
||||
proxy_buffering off;
|
||||
}
|
||||
```
|
||||
|
||||
Access becomes:
|
||||
- `https://example.com/logs/app/stream`
|
||||
- `https://example.com/logs/database/stream`
|
||||
- `https://example.com/logs/status`
|
||||
|
||||
### Path-Based Access Control
|
||||
|
||||
Use reverse proxy for authentication:
|
||||
|
||||
```nginx
|
||||
# Require auth for security logs
|
||||
location /logs/security/ {
|
||||
auth_basic "Security Logs";
|
||||
auth_basic_user_file /etc/nginx/security.htpasswd;
|
||||
proxy_pass http://localhost:8080/security/;
|
||||
}
|
||||
|
||||
# Public access for status
|
||||
location /logs/app/ {
|
||||
proxy_pass http://localhost:8080/app/;
|
||||
}
|
||||
[[pipelines.sinks]]
|
||||
type = "http"
|
||||
options = { port = 8080 }
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
### Router Mode Limitations
|
||||
1. **HTTP Only**: Router mode only works for HTTP/SSE
|
||||
2. **No TCP Routing**: TCP remains on separate ports
|
||||
3. **Path Conflicts**: Pipeline names must be unique
|
||||
|
||||
1. **HTTP Only**: Router mode only works for HTTP/SSE streams
|
||||
2. **No TCP Routing**: TCP streams remain on separate ports
|
||||
3. **Path Conflicts**: Stream names must be unique
|
||||
4. **Same Config**: All streams on a port share SSL/auth settings
|
||||
## Load Balancer Integration
|
||||
|
||||
### When Not to Use Router Mode
|
||||
|
||||
- High-performance scenarios (use TCP)
|
||||
- Streams need different SSL certificates
|
||||
- Complex authentication per stream
|
||||
- Network isolation requirements
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "Path not found"
|
||||
|
||||
Check available routes:
|
||||
```bash
|
||||
curl http://localhost:8080/invalid-path
|
||||
```
|
||||
|
||||
Response shows available routes:
|
||||
```json
|
||||
{
|
||||
"error": "Not Found",
|
||||
"requested_path": "/invalid-path",
|
||||
"available_routes": [
|
||||
"/status (global status)",
|
||||
"/app/stream (stream: app)",
|
||||
"/app/status (status: app)",
|
||||
"/database/stream (stream: database)",
|
||||
"/database/status (status: database)"
|
||||
]
|
||||
```nginx
|
||||
upstream logwisp {
|
||||
server logwisp1:8080;
|
||||
server logwisp2:8080;
|
||||
}
|
||||
```
|
||||
|
||||
### "Port conflict"
|
||||
|
||||
If you see port conflicts:
|
||||
1. Ensure `--router` flag is used
|
||||
2. Check all streams have `httpserver.enabled = true`
|
||||
3. Verify no other services use the port
|
||||
|
||||
### Debug Routing
|
||||
|
||||
Enable debug logging:
|
||||
```bash
|
||||
logwisp --router --log-level debug
|
||||
```
|
||||
|
||||
Look for routing decisions:
|
||||
```
|
||||
Router request method=GET path=/app/stream remote_addr=127.0.0.1:54321
|
||||
Routing request to stream stream=app original_path=/app/stream remaining_path=/stream
|
||||
```
|
||||
|
||||
### Performance Impact
|
||||
|
||||
Router mode adds minimal overhead:
|
||||
- ~100-200ns per request for path matching
|
||||
- Negligible memory overhead
|
||||
- No impact on streaming performance
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Naming Conventions
|
||||
|
||||
Use clear, consistent stream names:
|
||||
```toml
|
||||
# Good: Clear purpose
|
||||
name = "frontend-prod"
|
||||
name = "backend-staging"
|
||||
name = "worker-payments"
|
||||
|
||||
# Bad: Ambiguous
|
||||
name = "logs1"
|
||||
name = "stream2"
|
||||
name = "test"
|
||||
```
|
||||
|
||||
### Path Organization
|
||||
|
||||
Group related streams:
|
||||
```
|
||||
/prod/frontend/stream
|
||||
/prod/backend/stream
|
||||
/staging/frontend/stream
|
||||
/staging/backend/stream
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
Document your routing structure:
|
||||
```toml
|
||||
# Stream for production API logs
|
||||
# Access: https://logs.example.com/api-prod/stream
|
||||
[[streams]]
|
||||
name = "api-prod"
|
||||
```
|
||||
|
||||
### Monitoring
|
||||
|
||||
Use global status for overview:
|
||||
```bash
|
||||
# Monitor all streams
|
||||
watch -n 5 'curl -s localhost:8080/status | jq .streams'
|
||||
|
||||
# Check specific stream
|
||||
curl -s localhost:8080/status | jq '.streams.app'
|
||||
```
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From Standalone to Router
|
||||
|
||||
1. **Update configuration** - ensure consistent ports:
|
||||
```toml
|
||||
# Change from different ports
|
||||
[streams.httpserver]
|
||||
port = 8080 # Was 8081, 8082, etc.
|
||||
```
|
||||
|
||||
2. **Start with router flag**:
|
||||
```bash
|
||||
logwisp --router --config existing.toml
|
||||
```
|
||||
|
||||
3. **Update client URLs**:
|
||||
```bash
|
||||
# Old: http://localhost:8081/stream
|
||||
# New: http://localhost:8080/streamname/stream
|
||||
```
|
||||
|
||||
4. **Update monitoring**:
|
||||
```bash
|
||||
# Global status now available
|
||||
curl http://localhost:8080/status
|
||||
```
|
||||
|
||||
### Gradual Migration
|
||||
|
||||
Run both modes during transition:
|
||||
```bash
|
||||
# Week 1: Run standalone (current)
|
||||
logwisp --config prod.toml
|
||||
|
||||
# Week 2: Run both
|
||||
logwisp --config prod.toml & # Standalone
|
||||
logwisp --router --config prod-router.toml & # Router
|
||||
|
||||
# Week 3: Router only
|
||||
logwisp --router --config prod.toml
|
||||
```
|
||||
|
||||
## See Also
|
||||
|
||||
- [Configuration Guide](configuration.md) - Stream configuration
|
||||
- [HTTP Streaming](api.md#http-sse) - SSE protocol details
|
||||
- [Load Balancing](integrations.md#load-balancers) - Integration patterns
|
||||
- [Security Best Practices](security.md) - Securing router deployments
|
||||
location /logs/ {
|
||||
proxy_pass http://logwisp/;
|
||||
proxy_buffering off;
|
||||
}
|
||||
```
|
||||
186
doc/status.md
Normal file
186
doc/status.md
Normal file
@ -0,0 +1,186 @@
|
||||
# Status Monitoring
|
||||
|
||||
LogWisp provides comprehensive monitoring through status endpoints and operational logs.
|
||||
|
||||
## Status Endpoints
|
||||
|
||||
### Pipeline Status
|
||||
|
||||
```bash
|
||||
# Standalone mode
|
||||
curl http://localhost:8080/status
|
||||
|
||||
# Router mode
|
||||
curl http://localhost:8080/pipelinename/status
|
||||
```
|
||||
|
||||
Example response:
|
||||
```json
|
||||
{
|
||||
"service": "LogWisp",
|
||||
"version": "1.0.0",
|
||||
"server": {
|
||||
"type": "http",
|
||||
"port": 8080,
|
||||
"active_clients": 5,
|
||||
"buffer_size": 1000,
|
||||
"uptime_seconds": 3600
|
||||
},
|
||||
"sources": [{
|
||||
"type": "directory",
|
||||
"total_entries": 152341,
|
||||
"dropped_entries": 12,
|
||||
"active_watchers": 3
|
||||
}],
|
||||
"filters": {
|
||||
"filter_count": 2,
|
||||
"total_processed": 152341,
|
||||
"total_passed": 48234
|
||||
},
|
||||
"sinks": [{
|
||||
"type": "http",
|
||||
"total_processed": 48234,
|
||||
"active_connections": 5
|
||||
}]
|
||||
}
|
||||
```
|
||||
|
||||
## Key Metrics
|
||||
|
||||
### Source Metrics
|
||||
| Metric | Description | Healthy Range |
|
||||
|--------|-------------|---------------|
|
||||
| `active_watchers` | Files being watched | 1-1000 |
|
||||
| `total_entries` | Entries processed | Increasing |
|
||||
| `dropped_entries` | Buffer overflows | < 1% of total |
|
||||
|
||||
### Sink Metrics
|
||||
| Metric | Description | Warning Signs |
|
||||
|--------|-------------|---------------|
|
||||
| `active_connections` | Current clients | Near limit |
|
||||
| `total_processed` | Entries sent | Should match filter output |
|
||||
|
||||
### Filter Metrics
|
||||
| Metric | Description | Notes |
|
||||
|--------|-------------|-------|
|
||||
| `total_processed` | Entries checked | All entries |
|
||||
| `total_passed` | Passed filters | Check if too low/high |
|
||||
|
||||
## Operational Logging
|
||||
|
||||
### Log Levels
|
||||
```toml
|
||||
[logging]
|
||||
level = "info" # debug, info, warn, error
|
||||
```
|
||||
|
||||
### Important Messages
|
||||
|
||||
**Startup**:
|
||||
```
|
||||
LogWisp starting version=1.0.0
|
||||
Pipeline created successfully pipeline=app
|
||||
HTTP server started port=8080
|
||||
```
|
||||
|
||||
**Connections**:
|
||||
```
|
||||
HTTP client connected remote_addr=192.168.1.100 active_clients=6
|
||||
TCP connection opened active_connections=3
|
||||
```
|
||||
|
||||
**Errors**:
|
||||
```
|
||||
Failed to open file path=/var/log/app.log error=permission denied
|
||||
Request rate limited ip=192.168.1.100
|
||||
```
|
||||
|
||||
## Health Checks
|
||||
|
||||
### Basic Check
|
||||
```bash
|
||||
#!/bin/bash
|
||||
if curl -s -f http://localhost:8080/status > /dev/null; then
|
||||
echo "Healthy"
|
||||
else
|
||||
echo "Unhealthy"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Advanced Check
|
||||
```bash
|
||||
#!/bin/bash
|
||||
STATUS=$(curl -s http://localhost:8080/status)
|
||||
DROPPED=$(echo "$STATUS" | jq '.sources[0].dropped_entries')
|
||||
TOTAL=$(echo "$STATUS" | jq '.sources[0].total_entries')
|
||||
|
||||
if [ $((DROPPED * 100 / TOTAL)) -gt 5 ]; then
|
||||
echo "High drop rate"
|
||||
exit 1
|
||||
fi
|
||||
```
|
||||
|
||||
### Docker
|
||||
```dockerfile
|
||||
HEALTHCHECK --interval=30s --timeout=3s \
|
||||
CMD curl -f http://localhost:8080/status || exit 1
|
||||
```
|
||||
|
||||
## Integration
|
||||
|
||||
### Prometheus Export
|
||||
```bash
|
||||
#!/bin/bash
|
||||
STATUS=$(curl -s http://localhost:8080/status)
|
||||
|
||||
cat << EOF
|
||||
# HELP logwisp_active_clients Active streaming clients
|
||||
# TYPE logwisp_active_clients gauge
|
||||
logwisp_active_clients $(echo "$STATUS" | jq '.server.active_clients')
|
||||
|
||||
# HELP logwisp_total_entries Total log entries
|
||||
# TYPE logwisp_total_entries counter
|
||||
logwisp_total_entries $(echo "$STATUS" | jq '.sources[0].total_entries')
|
||||
EOF
|
||||
```
|
||||
|
||||
### Alerts
|
||||
|
||||
| Alert | Condition | Severity |
|
||||
|-------|-----------|----------|
|
||||
| Service Down | Status fails | Critical |
|
||||
| High Drops | >10% dropped | Warning |
|
||||
| No Activity | 0 entries/min | Warning |
|
||||
| Rate Limited | >20% blocked | Warning |
|
||||
|
||||
## Performance Monitoring
|
||||
|
||||
### CPU Usage
|
||||
```bash
|
||||
top -p $(pgrep logwisp)
|
||||
```
|
||||
|
||||
### Memory Usage
|
||||
```bash
|
||||
ps aux | grep logwisp
|
||||
```
|
||||
|
||||
### Connections
|
||||
```bash
|
||||
ss -tan | grep :8080 | wc -l
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Enable debug logging:
|
||||
```bash
|
||||
logwisp --log-level debug --log-output stderr
|
||||
```
|
||||
|
||||
Check specific components:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.sources'
|
||||
curl -s http://localhost:8080/status | jq '.filters'
|
||||
curl -s http://localhost:8080/status | jq '.sinks'
|
||||
```
|
||||
@ -1,537 +0,0 @@
|
||||
# Troubleshooting Guide
|
||||
|
||||
This guide helps diagnose and resolve common issues with LogWisp.
|
||||
|
||||
## Diagnostic Tools
|
||||
|
||||
### Enable Debug Logging
|
||||
|
||||
The first step in troubleshooting is enabling debug logs:
|
||||
|
||||
```bash
|
||||
# Via command line
|
||||
logwisp --log-level debug --log-output stderr
|
||||
|
||||
# Via environment
|
||||
export LOGWISP_LOGGING_LEVEL=debug
|
||||
logwisp
|
||||
|
||||
# Via config
|
||||
[logging]
|
||||
level = "debug"
|
||||
output = "stderr"
|
||||
```
|
||||
|
||||
### Check Status Endpoint
|
||||
|
||||
Verify LogWisp is running and processing:
|
||||
|
||||
```bash
|
||||
# Basic check
|
||||
curl http://localhost:8080/status
|
||||
|
||||
# Pretty print
|
||||
curl -s http://localhost:8080/status | jq .
|
||||
|
||||
# Check specific metrics
|
||||
curl -s http://localhost:8080/status | jq '.monitor'
|
||||
```
|
||||
|
||||
### Test Log Streaming
|
||||
|
||||
Verify streams are working:
|
||||
|
||||
```bash
|
||||
# Test SSE stream (should show heartbeats if enabled)
|
||||
curl -N http://localhost:8080/stream
|
||||
|
||||
# Test with timeout
|
||||
timeout 5 curl -N http://localhost:8080/stream
|
||||
|
||||
# Test TCP stream
|
||||
nc localhost 9090
|
||||
```
|
||||
|
||||
## Common Issues
|
||||
|
||||
### No Logs Appearing
|
||||
|
||||
**Symptoms:**
|
||||
- Stream connects but no log entries appear
|
||||
- Status shows `total_entries: 0`
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check monitor configuration:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.monitor'
|
||||
```
|
||||
|
||||
2. Verify file paths exist:
|
||||
```bash
|
||||
# Check your configured paths
|
||||
ls -la /var/log/myapp/
|
||||
```
|
||||
|
||||
3. Check file permissions:
|
||||
```bash
|
||||
# LogWisp user must have read access
|
||||
sudo -u logwisp ls /var/log/myapp/
|
||||
```
|
||||
|
||||
4. Verify files match pattern:
|
||||
```bash
|
||||
# If pattern is "*.log"
|
||||
ls /var/log/myapp/*.log
|
||||
```
|
||||
|
||||
5. Check if files are being updated:
|
||||
```bash
|
||||
# Should show recent timestamps
|
||||
ls -la /var/log/myapp/*.log
|
||||
tail -f /var/log/myapp/app.log
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Fix file permissions:
|
||||
```bash
|
||||
sudo chmod 644 /var/log/myapp/*.log
|
||||
sudo usermod -a -G adm logwisp # Add to log group
|
||||
```
|
||||
|
||||
- Correct path configuration:
|
||||
```toml
|
||||
targets = [
|
||||
{ path = "/correct/path/to/logs", pattern = "*.log" }
|
||||
]
|
||||
```
|
||||
|
||||
- Use absolute paths:
|
||||
```toml
|
||||
# Bad: Relative path
|
||||
targets = [{ path = "./logs", pattern = "*.log" }]
|
||||
|
||||
# Good: Absolute path
|
||||
targets = [{ path = "/var/log/app", pattern = "*.log" }]
|
||||
```
|
||||
|
||||
### High CPU Usage
|
||||
|
||||
**Symptoms:**
|
||||
- LogWisp process using excessive CPU
|
||||
- System slowdown
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check process CPU:
|
||||
```bash
|
||||
top -p $(pgrep logwisp)
|
||||
```
|
||||
|
||||
2. Review check intervals:
|
||||
```bash
|
||||
grep check_interval /etc/logwisp/logwisp.toml
|
||||
```
|
||||
|
||||
3. Count active watchers:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.monitor.active_watchers'
|
||||
```
|
||||
|
||||
4. Check filter complexity:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.filters'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Increase check interval:
|
||||
```toml
|
||||
[streams.monitor]
|
||||
check_interval_ms = 1000 # Was 50ms
|
||||
```
|
||||
|
||||
- Reduce watched files:
|
||||
```toml
|
||||
# Instead of watching entire directory
|
||||
targets = [
|
||||
{ path = "/var/log/specific-app.log", is_file = true }
|
||||
]
|
||||
```
|
||||
|
||||
- Simplify filter patterns:
|
||||
```toml
|
||||
# Complex regex (slow)
|
||||
patterns = ["^\\[\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\]\\s+\\[(ERROR|WARN)\\]"]
|
||||
|
||||
# Simple patterns (fast)
|
||||
patterns = ["ERROR", "WARN"]
|
||||
```
|
||||
|
||||
### Memory Growth
|
||||
|
||||
**Symptoms:**
|
||||
- Increasing memory usage over time
|
||||
- Eventually runs out of memory
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Monitor memory usage:
|
||||
```bash
|
||||
watch -n 10 'ps aux | grep logwisp'
|
||||
```
|
||||
|
||||
2. Check connection count:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.server.active_clients'
|
||||
```
|
||||
|
||||
3. Check for dropped entries:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.monitor.dropped_entries'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Limit connections:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
max_connections_per_ip = 5
|
||||
max_total_connections = 100
|
||||
```
|
||||
|
||||
- Reduce buffer sizes:
|
||||
```toml
|
||||
[streams.httpserver]
|
||||
buffer_size = 500 # Was 5000
|
||||
```
|
||||
|
||||
- Enable rate limiting:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
enabled = true
|
||||
requests_per_second = 10.0
|
||||
```
|
||||
|
||||
### Connection Refused
|
||||
|
||||
**Symptoms:**
|
||||
- Cannot connect to LogWisp
|
||||
- `curl: (7) Failed to connect`
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check if LogWisp is running:
|
||||
```bash
|
||||
ps aux | grep logwisp
|
||||
systemctl status logwisp
|
||||
```
|
||||
|
||||
2. Verify listening ports:
|
||||
```bash
|
||||
sudo netstat -tlnp | grep logwisp
|
||||
# or
|
||||
sudo ss -tlnp | grep logwisp
|
||||
```
|
||||
|
||||
3. Check firewall:
|
||||
```bash
|
||||
sudo iptables -L -n | grep 8080
|
||||
sudo ufw status
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Start the service:
|
||||
```bash
|
||||
sudo systemctl start logwisp
|
||||
```
|
||||
|
||||
- Fix port configuration:
|
||||
```toml
|
||||
[streams.httpserver]
|
||||
enabled = true # Must be true
|
||||
port = 8080 # Correct port
|
||||
```
|
||||
|
||||
- Open firewall:
|
||||
```bash
|
||||
sudo ufw allow 8080/tcp
|
||||
```
|
||||
|
||||
### Rate Limit Errors
|
||||
|
||||
**Symptoms:**
|
||||
- HTTP 429 responses
|
||||
- "Rate limit exceeded" errors
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check rate limit stats:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.features.rate_limit'
|
||||
```
|
||||
|
||||
2. Test rate limits:
|
||||
```bash
|
||||
# Rapid requests
|
||||
for i in {1..20}; do curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/status; done
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Increase rate limits:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
requests_per_second = 50.0 # Was 10.0
|
||||
burst_size = 100 # Was 20
|
||||
```
|
||||
|
||||
- Use per-IP limiting:
|
||||
```toml
|
||||
limit_by = "ip" # Instead of "global"
|
||||
```
|
||||
|
||||
- Disable for internal use:
|
||||
```toml
|
||||
enabled = false
|
||||
```
|
||||
|
||||
### Filter Not Working
|
||||
|
||||
**Symptoms:**
|
||||
- Unwanted logs still appearing
|
||||
- Wanted logs being filtered out
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check filter configuration:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '.filters'
|
||||
```
|
||||
|
||||
2. Test patterns:
|
||||
```bash
|
||||
# Test regex pattern
|
||||
echo "ERROR: test message" | grep -E "your-pattern"
|
||||
```
|
||||
|
||||
3. Enable debug logging to see filter decisions:
|
||||
```bash
|
||||
logwisp --log-level debug 2>&1 | grep filter
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Fix pattern syntax:
|
||||
```toml
|
||||
# Word boundaries
|
||||
patterns = ["\\bERROR\\b"] # Not "ERROR" which matches "TERROR"
|
||||
|
||||
# Case insensitive
|
||||
patterns = ["(?i)error"]
|
||||
```
|
||||
|
||||
- Check filter order:
|
||||
```toml
|
||||
# Include filters run first
|
||||
[[streams.filters]]
|
||||
type = "include"
|
||||
patterns = ["ERROR", "WARN"]
|
||||
|
||||
# Then exclude filters
|
||||
[[streams.filters]]
|
||||
type = "exclude"
|
||||
patterns = ["IGNORE_THIS"]
|
||||
```
|
||||
|
||||
- Use correct logic:
|
||||
```toml
|
||||
logic = "or" # Match ANY pattern
|
||||
# not
|
||||
logic = "and" # Match ALL patterns
|
||||
```
|
||||
|
||||
### Logs Dropping
|
||||
|
||||
**Symptoms:**
|
||||
- `dropped_entries` counter increasing
|
||||
- Missing log entries in stream
|
||||
|
||||
**Diagnosis:**
|
||||
|
||||
1. Check drop statistics:
|
||||
```bash
|
||||
curl -s http://localhost:8080/status | jq '{
|
||||
dropped: .monitor.dropped_entries,
|
||||
total: .monitor.total_entries,
|
||||
percent: (.monitor.dropped_entries / .monitor.total_entries * 100)
|
||||
}'
|
||||
```
|
||||
|
||||
2. Monitor drop rate:
|
||||
```bash
|
||||
watch -n 5 'curl -s http://localhost:8080/status | jq .monitor.dropped_entries'
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
|
||||
- Increase buffer sizes:
|
||||
```toml
|
||||
[streams.httpserver]
|
||||
buffer_size = 5000 # Was 1000
|
||||
```
|
||||
|
||||
- Add flow control:
|
||||
```toml
|
||||
[streams.monitor]
|
||||
check_interval_ms = 500 # Slow down reading
|
||||
```
|
||||
|
||||
- Reduce clients:
|
||||
```toml
|
||||
[streams.httpserver.rate_limit]
|
||||
max_total_connections = 50
|
||||
```
|
||||
|
||||
## Performance Issues
|
||||
|
||||
### Slow Response Times
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Measure response time
|
||||
time curl -s http://localhost:8080/status > /dev/null
|
||||
|
||||
# Check system load
|
||||
uptime
|
||||
top
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Reduce concurrent operations
|
||||
- Increase system resources
|
||||
- Use TCP instead of HTTP for high volume
|
||||
|
||||
### Network Bandwidth
|
||||
|
||||
**Diagnosis:**
|
||||
```bash
|
||||
# Monitor network usage
|
||||
iftop -i eth0 -f "port 8080"
|
||||
|
||||
# Check connection count
|
||||
ss -tan | grep :8080 | wc -l
|
||||
```
|
||||
|
||||
**Solutions:**
|
||||
- Enable compression (future feature)
|
||||
- Filter more aggressively
|
||||
- Use TCP for local connections
|
||||
|
||||
## Debug Commands
|
||||
|
||||
### System Information
|
||||
|
||||
```bash
|
||||
# LogWisp version
|
||||
logwisp --version
|
||||
|
||||
# System resources
|
||||
free -h
|
||||
df -h
|
||||
ulimit -a
|
||||
|
||||
# Network state
|
||||
ss -tlnp
|
||||
netstat -anp | grep logwisp
|
||||
```
|
||||
|
||||
### Process Inspection
|
||||
|
||||
```bash
|
||||
# Process details
|
||||
ps aux | grep logwisp
|
||||
|
||||
# Open files
|
||||
lsof -p $(pgrep logwisp)
|
||||
|
||||
# System calls (Linux)
|
||||
strace -p $(pgrep logwisp) -e trace=open,read,write
|
||||
|
||||
# File system activity
|
||||
inotifywait -m /var/log/myapp/
|
||||
```
|
||||
|
||||
### Configuration Validation
|
||||
|
||||
```bash
|
||||
# Test configuration
|
||||
logwisp --config test.toml --log-level debug --log-output stderr
|
||||
|
||||
# Check file syntax
|
||||
cat /etc/logwisp/logwisp.toml | grep -E "^\s*\["
|
||||
|
||||
# Validate TOML
|
||||
python3 -m pip install toml
|
||||
python3 -c "import toml; toml.load('/etc/logwisp/logwisp.toml'); print('Valid')"
|
||||
```
|
||||
|
||||
## Getting Help
|
||||
|
||||
### Collect Diagnostic Information
|
||||
|
||||
Create a diagnostic bundle:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# diagnostic.sh
|
||||
|
||||
DIAG_DIR="logwisp-diag-$(date +%Y%m%d-%H%M%S)"
|
||||
mkdir -p "$DIAG_DIR"
|
||||
|
||||
# Version
|
||||
logwisp --version > "$DIAG_DIR/version.txt" 2>&1
|
||||
|
||||
# Configuration (sanitized)
|
||||
grep -v "password\|secret\|token" /etc/logwisp/logwisp.toml > "$DIAG_DIR/config.toml"
|
||||
|
||||
# Status
|
||||
curl -s http://localhost:8080/status > "$DIAG_DIR/status.json"
|
||||
|
||||
# System info
|
||||
uname -a > "$DIAG_DIR/system.txt"
|
||||
free -h >> "$DIAG_DIR/system.txt"
|
||||
df -h >> "$DIAG_DIR/system.txt"
|
||||
|
||||
# Process info
|
||||
ps aux | grep logwisp > "$DIAG_DIR/process.txt"
|
||||
lsof -p $(pgrep logwisp) > "$DIAG_DIR/files.txt" 2>&1
|
||||
|
||||
# Recent logs
|
||||
journalctl -u logwisp -n 1000 > "$DIAG_DIR/logs.txt" 2>&1
|
||||
|
||||
# Create archive
|
||||
tar -czf "$DIAG_DIR.tar.gz" "$DIAG_DIR"
|
||||
rm -rf "$DIAG_DIR"
|
||||
|
||||
echo "Diagnostic bundle created: $DIAG_DIR.tar.gz"
|
||||
```
|
||||
|
||||
### Report Issues
|
||||
|
||||
When reporting issues, include:
|
||||
1. LogWisp version
|
||||
2. Configuration (sanitized)
|
||||
3. Error messages
|
||||
4. Steps to reproduce
|
||||
5. Diagnostic bundle
|
||||
|
||||
## See Also
|
||||
|
||||
- [Monitoring Guide](monitoring.md) - Status and metrics
|
||||
- [Performance Tuning](performance.md) - Optimization
|
||||
- [Configuration Guide](configuration.md) - Settings reference
|
||||
- [FAQ](faq.md) - Frequently asked questions
|
||||
Reference in New Issue
Block a user