v0.13.0 doc update, refactor, http/tcp chain source and sink added

This commit is contained in:
2026-07-17 05:58:44 -04:00
parent ebb5aa3bfe
commit 87e57784da
35 changed files with 2211 additions and 1292 deletions
+27 -24
View File
@@ -13,13 +13,15 @@ Each pipeline operates independently with a source → filter → format → sin
```
Service (Main Process)
├── Pipeline 1
│ ├── Sources (1 or more)
│ ├── Rate Limiter (optional)
│ ├── Filter Chain (optional)
│ ├── Formatter (optional)
└── Sinks (1 or more)
│ ├── Plugin Sources (1 or more)
│ ├── Flow
│ ├── Heartbeat Generator (optional)
│ ├── Rate Limiter (optional)
│ ├── Filter Chain (optional)
│ │ └── Formatter (optional)
│ └── Plugin Sinks (1 or more)
├── Pipeline 2
│ └── [Same structure]
│ └── [Similar structure]
└── Status Reporter (optional)
```
@@ -27,11 +29,11 @@ Service (Main Process)
### Processing Stages
1. **Source Stage**: Sources monitor inputs and generate log entries
2. **Rate Limiting**: Optional pipeline-level rate control
3. **Filtering**: Pattern-based inclusion/exclusion
4. **Formatting**: Transform entries to desired output format
5. **Distribution**: Fan-out to multiple sinks
1. **Source Stage**: Plugin sources monitor inputs and generate log entries
2. **Flow - Rate Limiting**: Optional pipeline-level rate control
3. **Flow - Filtering**: Pattern-based inclusion/exclusion
4. **Flow - Formatting**: Transform entries to desired output format with sanitization
5. **Distribution**: Fan-out to multiple plugin sinks
### Entry Lifecycle
@@ -50,14 +52,16 @@ Each component maintains internal buffers to handle burst traffic:
- Sinks: Independent buffers per sink
- Network components: Additional TCP/HTTP buffers
*"Sink dispatch uses non-blocking sends. When a sink's input buffer is full, the event is dropped for that sink only and counted in pipeline statistics (`total_dropped_by_sink`). The policy is uniform and not configurable; a slow sink does not stall the pipeline or other sinks."*
## Component Types
### Sources (Input)
- **Directory Source**: File system monitoring with rotation detection
- **Stdin Source**: Standard input processing
- **HTTP Source**: REST endpoint for log ingestion
- **TCP Source**: Raw TCP socket listener
- **File Source**: File system directory monitoring with rotation detection
- **Console Source**: Standard input processing (stdin)
- **Random Source**: Generates random log entries for testing
- **Null Source**: Discards logs, used for testing
### Sinks (Output)
@@ -65,14 +69,13 @@ Each component maintains internal buffers to handle burst traffic:
- **File Sink**: Rotating file writer
- **HTTP Sink**: Server-Sent Events (SSE) streaming
- **TCP Sink**: TCP server for client connections
- **HTTP Client Sink**: Forward to remote HTTP endpoints
- **TCP Client Sink**: Forward to remote TCP servers
- **Null Sink**: Discards all received events
### Processing Components
- **Rate Limiter**: Token bucket algorithm for flow control
- **Filter Chain**: Sequential pattern matching
- **Formatters**: Raw, JSON, or template-based text transformation
- **Formatters**: Raw, JSON, or text transformation with sanitizer policies
## Concurrency Model
@@ -95,8 +98,7 @@ Each component maintains internal buffers to handle burst traffic:
### Connection Patterns
**Chaining Design**:
- TCP Client Sink → TCP Source: Direct TCP forwarding
- HTTP Client Sink → HTTP Source: HTTP-based forwarding
- Future plan
**Monitoring Design**:
- TCP Sink: Debugging interface
@@ -106,8 +108,6 @@ Each component maintains internal buffers to handle burst traffic:
- HTTP/1.1 and HTTP/2 for HTTP connections
- Raw TCP connections
- TLS 1.2/1.3 for HTTPS connections (HTTP only)
- Server-Sent Events for real-time streaming
## Resource Management
@@ -125,6 +125,10 @@ Each component maintains internal buffers to handle burst traffic:
### Connection Management
- HTTP sink silenty drops IPv6 connections (deliberate IPv4-only enforcement)
*Note:* Placeholder; below features are removed in restructuring and will be added in the future release.
- Per-IP connection limits
- Global connection caps
- Automatic reconnection with exponential backoff
@@ -136,7 +140,6 @@ Each component maintains internal buffers to handle burst traffic:
- Panic recovery in pipeline processing
- Independent pipeline operation
- Automatic source restart on failure
- Sink failure isolation
### Data Integrity
@@ -165,4 +168,4 @@ Each component maintains internal buffers to handle burst traffic:
- Horizontal: Multiple LogWisp instances with different configurations
- Vertical: Multiple pipelines per instance
- Fan-out: Multiple sinks per pipeline
- Fan-in: Multiple sources per pipeline
- Fan-in: Multiple sources per pipeline