172 lines
5.1 KiB
Markdown
172 lines
5.1 KiB
Markdown
# Architecture Overview
|
|
|
|
LogWisp implements a pipeline-based architecture for flexible log processing and distribution.
|
|
|
|
## Core Concepts
|
|
|
|
### Pipeline Model
|
|
|
|
Each pipeline operates independently with a source → filter → format → sink flow. Multiple pipelines can run concurrently within a single LogWisp instance, each processing different log streams with unique configurations.
|
|
|
|
### Component Hierarchy
|
|
|
|
```
|
|
Service (Main Process)
|
|
├── Pipeline 1
|
|
│ ├── Plugin Sources (1 or more)
|
|
│ ├── Flow
|
|
│ │ ├── Heartbeat Generator (optional)
|
|
│ │ ├── Rate Limiter (optional)
|
|
│ │ ├── Filter Chain (optional)
|
|
│ │ └── Formatter (optional)
|
|
│ └── Plugin Sinks (1 or more)
|
|
├── Pipeline 2
|
|
│ └── [Similar structure]
|
|
└── Status Reporter (optional)
|
|
```
|
|
|
|
## Data Flow
|
|
|
|
### Processing Stages
|
|
|
|
1. **Source Stage**: Plugin sources monitor inputs and generate log entries
|
|
2. **Flow - Rate Limiting**: Optional pipeline-level rate control
|
|
3. **Flow - Filtering**: Pattern-based inclusion/exclusion
|
|
4. **Flow - Formatting**: Transform entries to desired output format with sanitization
|
|
5. **Distribution**: Fan-out to multiple plugin sinks
|
|
|
|
### Entry Lifecycle
|
|
|
|
Log entries flow through the pipeline as `core.LogEntry` structures containing:
|
|
- **Time**: Entry timestamp
|
|
- **Level**: Log level (DEBUG, INFO, WARN, ERROR)
|
|
- **Source**: Origin identifier
|
|
- **Message**: Log content
|
|
- **Fields**: Additional metadata (JSON)
|
|
- **RawSize**: Original entry size
|
|
|
|
### Buffering Strategy
|
|
|
|
Each component maintains internal buffers to handle burst traffic:
|
|
- Sources: Configurable buffer size (default 1000 entries)
|
|
- Sinks: Independent buffers per sink
|
|
- Network components: Additional TCP/HTTP buffers
|
|
|
|
*"Sink dispatch uses non-blocking sends. When a sink's input buffer is full, the event is dropped for that sink only and counted in pipeline statistics (`total_dropped_by_sink`). The policy is uniform and not configurable; a slow sink does not stall the pipeline or other sinks."*
|
|
|
|
## Component Types
|
|
|
|
### Sources (Input)
|
|
|
|
- **File Source**: File system directory monitoring with rotation detection
|
|
- **Console Source**: Standard input processing (stdin)
|
|
- **Random Source**: Generates random log entries for testing
|
|
- **Null Source**: Discards logs, used for testing
|
|
|
|
### Sinks (Output)
|
|
|
|
- **Console Sink**: stdout/stderr output
|
|
- **File Sink**: Rotating file writer
|
|
- **HTTP Sink**: Server-Sent Events (SSE) streaming
|
|
- **TCP Sink**: TCP server for client connections
|
|
- **Null Sink**: Discards all received events
|
|
|
|
### Processing Components
|
|
|
|
- **Rate Limiter**: Token bucket algorithm for flow control
|
|
- **Filter Chain**: Sequential pattern matching
|
|
- **Formatters**: Raw, JSON, or text transformation with sanitizer policies
|
|
|
|
## Concurrency Model
|
|
|
|
### Goroutine Architecture
|
|
|
|
- Each source runs in dedicated goroutines for monitoring
|
|
- Sinks operate independently with their own processing loops
|
|
- Network listeners use optimized event loops (gnet for TCP)
|
|
- Pipeline processing uses channel-based communication
|
|
|
|
### Synchronization
|
|
|
|
- Atomic counters for statistics
|
|
- Read-write mutexes for configuration access
|
|
- Context-based cancellation for graceful shutdown
|
|
- Wait groups for coordinated startup/shutdown
|
|
|
|
## Network Architecture
|
|
|
|
### Connection Patterns
|
|
|
|
**Chaining Design**:
|
|
- Future plan
|
|
|
|
**Monitoring Design**:
|
|
- TCP Sink: Debugging interface
|
|
- HTTP Sink: Browser-based live monitoring
|
|
|
|
### Protocol Support
|
|
|
|
- HTTP/1.1 and HTTP/2 for HTTP connections
|
|
- Raw TCP connections
|
|
|
|
## Resource Management
|
|
|
|
### Memory Management
|
|
|
|
- Bounded buffers prevent unbounded growth
|
|
- Automatic garbage collection via Go runtime
|
|
- Connection limits prevent resource exhaustion
|
|
|
|
### File Management
|
|
|
|
- Automatic rotation based on size thresholds
|
|
- Retention policies for old log files
|
|
- Minimum disk space checks before writing
|
|
|
|
### Connection Management
|
|
|
|
- HTTP sink silenty drops IPv6 connections (deliberate IPv4-only enforcement)
|
|
|
|
*Note:* Placeholder; below features are removed in restructuring and will be added in the future release.
|
|
|
|
- Per-IP connection limits
|
|
- Global connection caps
|
|
- Automatic reconnection with exponential backoff
|
|
- Keep-alive for persistent connections
|
|
|
|
## Reliability Features
|
|
|
|
### Fault Tolerance
|
|
|
|
- Panic recovery in pipeline processing
|
|
- Independent pipeline operation
|
|
- Sink failure isolation
|
|
|
|
### Data Integrity
|
|
|
|
- Entry validation at ingestion
|
|
- Size limits for entries and batches
|
|
- Duplicate detection in file monitoring
|
|
- Position tracking for file reads
|
|
|
|
## Performance Characteristics
|
|
|
|
### Throughput
|
|
|
|
- Pipeline rate limiting: Configurable (default 1000 entries/second)
|
|
- Network throughput: Limited by network and sink capacity
|
|
- File monitoring: Sub-second detection (default 100ms interval)
|
|
|
|
### Latency
|
|
|
|
- Entry processing: Sub-millisecond in-memory
|
|
- Network forwarding: Depends on batch configuration
|
|
- File detection: Configurable check interval
|
|
|
|
### Scalability
|
|
|
|
- Horizontal: Multiple LogWisp instances with different configurations
|
|
- Vertical: Multiple pipelines per instance
|
|
- Fan-out: Multiple sinks per pipeline
|
|
- Fan-in: Multiple sources per pipeline
|