lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.1.3 stream changed from net/http to fasthttp for http and gnet for tcp stream, heartbeat config added

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-01 23:43:51 -04:00
parent a3450a9589
commit a7595061ba
13 changed files with 1134 additions and 1474 deletions
Download Patch File Download Diff File Expand all files Collapse all files

625
README.md
View File

@ -2,41 +2,39 @@
<img src="assets/logo.svg" alt="LogWisp Logo" width="200"/>
</p>
# LogWisp - Simple Log Streaming
# LogWisp - Dual-Stack Log Streaming
A lightweight log streaming service that monitors files and streams updates via Server-Sent Events (SSE).
## Philosophy
LogWisp follows the Unix philosophy: do one thing and do it well. It monitors log files and streams them over HTTP/SSE. That's it.
A high-performance log streaming service with dual-stack architecture: raw TCP streaming via gnet and HTTP/SSE streaming via fasthttp.
## Features
- Monitors multiple files and directories simultaneously
- Streams log updates in real-time via SSE
- Supports both plain text and JSON formatted logs
- Automatic file rotation detection
- Configurable rate limiting
- Environment variable support
- Simple TOML configuration
- Atomic configuration management
- Optional ANSI color pass-through
- **Dual streaming modes**: TCP (gnet) and HTTP/SSE (fasthttp)
- **Fan-out architecture**: Multiple independent consumers
- **Real-time updates**: File monitoring with rotation detection
- **Zero dependencies**: Only gnet and fasthttp beyond stdlib
- **High performance**: Non-blocking I/O throughout
## Quick Start
1. Build:
```bash
./build.sh
```
# Build
go build -o logwisp ./src/cmd/logwisp
2. Run with defaults (monitors current directory):
```bash
# Run with HTTP only (default)
./logwisp
# Enable both TCP and HTTP
./logwisp --enable-tcp --tcp-port 9090
# Monitor specific paths
./logwisp /var/log:*.log /app/logs:error*.log
```
3. View logs:
```bash
curl -N http://localhost:8080/stream
## Architecture
```
Monitor (Publisher) → [Subscriber Channels] → TCP Server (default port 9090)
↘ HTTP Server (default port 8080)
```
## Command Line Options
@ -45,279 +43,87 @@ curl -N http://localhost:8080/stream
logwisp [OPTIONS] [TARGET...]
OPTIONS:
-c, --color Enable color pass-through for ANSI escape codes
--config FILE Config file path (default: ~/.config/logwisp.toml)
--port PORT HTTP port (default: 8080)
--buffer-size SIZE Stream buffer size (default: 1000)
--check-interval MS File check interval in ms (default: 100)
--rate-limit Enable rate limiting
--rate-requests N Rate limit requests/sec (default: 10)
--rate-burst N Rate limit burst size (default: 20)
--config FILE Config file path
--check-interval MS File check interval (default: 100)
# TCP Server
--enable-tcp Enable TCP server
--tcp-port PORT TCP port (default: 9090)
--tcp-buffer-size SIZE TCP buffer size (default: 1000)
# HTTP Server
--enable-http Enable HTTP server (default: true)
--http-port PORT HTTP port (default: 8080)
--http-buffer-size SIZE HTTP buffer size (default: 1000)
# Legacy compatibility
--port PORT Same as --http-port
--buffer-size SIZE Same as --http-buffer-size
TARGET:
path[:pattern[:isfile]] Path to monitor (file or directory)
pattern: glob pattern for directories (default: *.log)
isfile: true/false (auto-detected if omitted)
EXAMPLES:
# Monitor current directory for *.log files
logwisp
# Monitor specific file with color support
logwisp -c /var/log/app.log
# Monitor multiple locations
logwisp /var/log:*.log /app/logs:error*.log:false /tmp/debug.log::true
# Custom port with rate limiting
logwisp --port 9090 --rate-limit --rate-requests 100 --rate-burst 200
path[:pattern[:isfile]] Path to monitor
pattern: glob pattern for directories
isfile: true/false (auto-detected if omitted)
```
## Configuration
LogWisp uses a three-level configuration hierarchy:
1. **Command-line arguments** (highest priority)
2. **Environment variables**
3. **Configuration file** (~/.config/logwisp.toml)
4. **Default values** (lowest priority)
### Default Values
| Setting | Default | Description |
|---------|---------|-------------|
| `port` | 8080 | HTTP listen port |
| `monitor.check_interval_ms` | 100 | File check interval (milliseconds) |
| `monitor.targets` | [{"path": "./", "pattern": "*.log", "is_file": false}] | Paths to monitor |
| `stream.buffer_size` | 1000 | Per-client event buffer size |
| `stream.rate_limit.enabled` | false | Enable rate limiting |
| `stream.rate_limit.requests_per_second` | 10 | Sustained request rate |
| `stream.rate_limit.burst_size` | 20 | Maximum burst size |
| `stream.rate_limit.cleanup_interval_s` | 60 | Client cleanup interval |
### Configuration File Location
Default: `~/.config/logwisp.toml`
Override with environment variables:
- `LOGWISP_CONFIG_DIR` - Directory containing config file
- `LOGWISP_CONFIG_FILE` - Config filename (absolute or relative)
Examples:
```bash
# Use config from current directory
LOGWISP_CONFIG_DIR=. ./logwisp
# Use specific config file
LOGWISP_CONFIG_FILE=/etc/logwisp/prod.toml ./logwisp
# Use custom directory and filename
LOGWISP_CONFIG_DIR=/opt/configs LOGWISP_CONFIG_FILE=myapp.toml ./logwisp
```
### Environment Variables
All configuration values can be overridden via environment variables:
| Environment Variable | Config Path | Description |
|---------------------|-------------|-------------|
| `LOGWISP_PORT` | `port` | HTTP listen port |
| `LOGWISP_MONITOR_CHECK_INTERVAL_MS` | `monitor.check_interval_ms` | File check interval |
| `LOGWISP_MONITOR_TARGETS` | `monitor.targets` | Comma-separated targets |
| `LOGWISP_STREAM_BUFFER_SIZE` | `stream.buffer_size` | Client buffer size |
| `LOGWISP_STREAM_RATE_LIMIT_ENABLED` | `stream.rate_limit.enabled` | Enable rate limiting |
| `LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC` | `stream.rate_limit.requests_per_second` | Rate limit |
| `LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE` | `stream.rate_limit.burst_size` | Burst size |
| `LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL_S` | `stream.rate_limit.cleanup_interval_s` | Cleanup interval |
### Monitor Targets Format
The `LOGWISP_MONITOR_TARGETS` environment variable uses a special format:
```
path:pattern:isfile,path2:pattern2:isfile
```
Examples:
```bash
# Monitor directory and specific file
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp
# Multiple directories
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp
```
### Complete Configuration Example
Config file location: `~/.config/logwisp.toml`
```toml
# Port to listen on (default: 8080)
port = 8080
[monitor]
# How often to check for file changes in milliseconds (default: 100)
check_interval_ms = 100
# Paths to monitor
# Default: [{"path": "./", "pattern": "*.log", "is_file": false}]
# Monitor all .log files in current directory
[[monitor.targets]]
path = "./"
pattern = "*.log"
is_file = false
# Monitor specific file
[[monitor.targets]]
path = "/app/logs/app.log"
pattern = "" # Ignored for files
is_file = true
# Monitor with specific pattern
[[monitor.targets]]
path = "/var/log/nginx"
pattern = "access*.log"
is_file = false
[stream]
# Buffer size for each client connection (default: 1000)
# Controls how many log entries can be queued per client
[tcpserver]
enabled = false
port = 9090
buffer_size = 1000
[stream.rate_limit]
# Enable rate limiting (default: false)
enabled = false
# Requests per second per client (default: 10)
# This is the sustained rate
requests_per_second = 10
# Burst size - max requests at once (default: 20)
# Allows temporary bursts above the sustained rate
burst_size = 20
# How often to clean up old client limiters in seconds (default: 60)
# Clients inactive for 2x this duration are removed
cleanup_interval_s = 60
[httpserver]
enabled = true
port = 8080
buffer_size = 1000
```
## Color Support
LogWisp can pass through ANSI color escape codes from monitored logs to SSE clients using the `-c` flag.
## Clients
### TCP Stream
```bash
# Enable color pass-through
./logwisp -c
# Simple TCP client
nc localhost 9090
# Or via systemd
ExecStart=/opt/logwisp/bin/logwisp -c
# Using telnet
telnet localhost 9090
# Using socat
socat - TCP:localhost:9090
```
### How It Works
When color mode is enabled (`-c` flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream.
### Example Log with Colors
Original log file content:
```
\033[31mERROR\033[0m: Database connection failed
\033[33mWARN\033[0m: High memory usage detected
\033[32mINFO\033[0m: Service started successfully
```
SSE output with `-c`:
```json
{
"time": "2024-01-01T12:00:00.123456Z",
"source": "app.log",
"message": "\u001b[31mERROR\u001b[0m: Database connection failed"
}
```
### Client-Side Handling
#### Terminal Clients
For terminal-based clients (like curl), the escape codes will render as colors:
### HTTP/SSE Stream
```bash
# This will show colored output in terminals that support ANSI codes
curl -N http://localhost:8080/stream | jq -r '.message'
# Stream logs
curl -N http://localhost:8080/stream
# Check status
curl http://localhost:8080/status
```
#### Web Clients
## Environment Variables
For web-based clients, you'll need to convert ANSI codes to HTML:
All config values can be set via environment:
- `LOGWISP_MONITOR_CHECK_INTERVAL_MS`
- `LOGWISP_MONITOR_TARGETS` (format: "path:pattern:isfile,...")
- `LOGWISP_TCPSERVER_ENABLED`
- `LOGWISP_TCPSERVER_PORT`
- `LOGWISP_HTTPSERVER_ENABLED`
- `LOGWISP_HTTPSERVER_PORT`
```javascript
// Example using ansi-to-html library
const AnsiToHtml = require('ansi-to-html');
const convert = new AnsiToHtml();
eventSource.onmessage = (event) => {
const data = JSON.parse(event.data);
const html = convert.toHtml(data.message);
document.getElementById('log').innerHTML += html + '<br>';
};
```
#### Custom Processing
```python
# Python example with colorama
import json
import colorama
from colorama import init
init() # Initialize colorama for Windows support
# Process SSE stream
for line in stream:
if line.startswith('data: '):
data = json.loads(line[6:])
# Colorama will handle ANSI codes automatically
print(data['message'])
```
### Common ANSI Color Codes
| Code | Color/Style |
|------|-------------|
| `\033[0m` | Reset |
| `\033[1m` | Bold |
| `\033[31m` | Red |
| `\033[32m` | Green |
| `\033[33m` | Yellow |
| `\033[34m` | Blue |
| `\033[35m` | Magenta |
| `\033[36m` | Cyan |
### Limitations
1. **JSON Escaping**: ANSI codes are JSON-escaped in the stream (e.g., `\033` becomes `\u001b`)
2. **Client Support**: The client must support or convert ANSI codes
3. **Performance**: No significant impact, but slightly larger message sizes
### Security Note
Color codes are passed through as-is. Ensure monitored logs come from trusted sources to avoid terminal escape sequence attacks.
### Disabling Colors
To strip color codes instead of passing them through:
- Don't use the `-c` flag
- Or set up a preprocessing pipeline:
```bash
tail -f colored.log | sed 's/\x1b\[[0-9;]*m//g' > plain.log
```
## API
### Endpoints
- `GET /stream` - Server-Sent Events stream of log entries
- `GET /status` - Service status and configuration information
### Log Entry Format
## Log Entry Format
```json
{
@ -329,223 +135,156 @@ To strip color codes instead of passing them through:
}
```
### SSE Event Types
## API Endpoints
| Event | Description | Data Format |
|-------|-------------|-------------|
| `connected` | Initial connection | `{"client_id": "123456789"}` |
| `data` | Log entry | JSON log entry |
| `disconnected` | Client disconnected | `{"reason": "slow_client"}` |
| `timeout` | Client timeout | `{"reason": "client_timeout"}` |
| `:` | Heartbeat (comment) | ISO timestamp |
### TCP Protocol
- Raw JSON lines, one entry per line
- No headers or authentication
- Instant connection, streaming starts immediately
### Status Response Format
### HTTP Endpoints
- `GET /stream` - SSE stream of log entries
- `GET /status` - Service status JSON
### SSE Events
- `connected` - Initial connection with client_id
- `data` - Log entry JSON
- `:` - Heartbeat comment (30s interval)
## Heartbeat Configuration
LogWisp supports configurable heartbeat messages for both HTTP/SSE and TCP streams to detect stale connections and provide server statistics.
**HTTP/SSE Heartbeat:**
- **Format Options:**
- `comment`: SSE comment format (`: heartbeat ...`)
- `json`: Standard data message with JSON payload
- **Content Options:**
- `include_timestamp`: Add current UTC timestamp
- `include_stats`: Add active clients count and server uptime
**TCP Heartbeat:**
- Always uses JSON format
- Same content options as HTTP
- Useful for detecting disconnected clients
**Example Heartbeat Messages:**
HTTP Comment format:
```
: heartbeat 2024-01-01T12:00:00Z clients=5 uptime=3600s
```
JSON format:
```json
{
"service": "LogWisp",
"version": "2.0.0",
"port": 8080,
"color_mode": false,
"config": {
"monitor": {
"check_interval_ms": 100,
"targets_count": 2
},
"stream": {
"buffer_size": 1000,
"rate_limit": {
"enabled": true,
"requests_per_second": 10,
"burst_size": 20
}
}
},
"streamer": {
"active_clients": 5,
"buffer_size": 1000,
"color_mode": false,
"total_dropped": 42
},
"rate_limiter": "Active clients: 3"
}
{"type":"heartbeat","timestamp":"2024-01-01T12:00:00Z","active_clients":5,"uptime_seconds":3600}
```
## Usage Examples
### Basic Usage
```bash
# Start with defaults
./logwisp
# Monitor specific file
./logwisp /var/log/app.log
# View logs
curl -N http://localhost:8080/stream
**Configuration:**
```toml
[httpserver.heartbeat]
enabled = true
interval_seconds = 30
include_timestamp = true
include_stats = true
format = "json"
```
### With Environment Variables
```bash
# Change port and add rate limiting
LOGWISP_PORT=9090 \
LOGWISP_STREAM_RATE_LIMIT_ENABLED=true \
LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \
./logwisp
```
**Environment Variables:**
- `LOGWISP_HTTPSERVER_HEARTBEAT_ENABLED`
- `LOGWISP_HTTPSERVER_HEARTBEAT_INTERVAL_SECONDS`
- `LOGWISP_TCPSERVER_HEARTBEAT_ENABLED`
- `LOGWISP_TCPSERVER_HEARTBEAT_INTERVAL_SECONDS`
### Monitor Multiple Locations
```bash
# Via command line
./logwisp /var/log:*.log /app/logs:*.json /tmp/debug.log
## Summary
# Via environment variable
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \
./logwisp
**Fixed:**
- Removed duplicate `globToRegex` functions (never used)
- Added missing TCP heartbeat support
- Made HTTP heartbeat configurable
# Or via config file
cat > ~/.config/logwisp.toml << EOF
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false
**Enhanced:**
- Configurable heartbeat interval
- Multiple format options (comment/JSON)
- Optional timestamp and statistics
- Per-protocol configuration
[[monitor.targets]]
path = "/app/logs"
pattern = "*.json"
is_file = false
**⚠️ SECURITY:** Heartbeat statistics expose minimal server state (connection count, uptime). If this is sensitive in your environment, disable `include_stats`.
[[monitor.targets]]
path = "/tmp/debug.log"
is_file = true
EOF
```
### Production Deployment
Example systemd service with environment overrides:
## Deployment
### Systemd Service
```ini
[Unit]
Description=LogWisp Log Streaming Service
Description=LogWisp Log Streaming
After=network.target
[Service]
Type=simple
User=logwisp
ExecStart=/usr/local/bin/logwisp -c
ExecStart=/usr/local/bin/logwisp --enable-tcp --enable-http
Restart=always
RestartSec=5
# Environment overrides
Environment="LOGWISP_PORT=8080"
Environment="LOGWISP_STREAM_BUFFER_SIZE=5000"
Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true"
Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100"
Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false"
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/
ReadWritePaths=/var/log /opt/app/logs
Environment="LOGWISP_TCPSERVER_PORT=9090"
Environment="LOGWISP_HTTPSERVER_PORT=8080"
[Install]
WantedBy=multi-user.target
```
### Docker
```dockerfile
FROM golang:1.24 AS builder
WORKDIR /app
COPY . .
RUN go build -o logwisp ./src/cmd/logwisp
FROM debian:bookworm-slim
COPY --from=builder /app/logwisp /usr/local/bin/
EXPOSE 8080 9090
CMD ["logwisp", "--enable-tcp", "--enable-http"]
```
## Performance Tuning
### Buffer Size
- **Buffer Size**: Increase for burst traffic (5000+)
- **Check Interval**: Decrease for lower latency (10-50ms)
- **TCP**: Best for high-volume system consumers
- **HTTP**: Best for web browsers and REST clients
The `stream.buffer_size` setting controls how many log entries can be queued per client:
- **Small buffers (100-500)**: Lower memory usage, clients skip entries during bursts
- **Default (1000)**: Good balance for most use cases
- **Large buffers (5000+)**: Handle burst traffic better, higher memory usage
### Message Dropping and Client Behavior
When a client's buffer is full, new messages are skipped for that client until it catches up. The client remains connected and will receive future messages once buffer space is available.
LogWisp uses non-blocking message delivery to maintain system stability. When a client cannot keep up with the log stream, messages are dropped rather than blocking other clients or the monitor.
### Check Interval
**Common causes of dropped messages:**
- **Browser throttling**: Browsers may throttle background tabs, reducing JavaScript execution frequency
- **Network congestion**: Slow connections or high latency can cause client buffers to fill
- **Client processing**: Heavy client-side processing (parsing, rendering) can create backpressure
- **System resources**: CPU/memory constraints on client machines affect consumption rate
The `monitor.check_interval_ms` setting controls file polling frequency:
- **Fast (10-50ms)**: Near real-time updates, higher CPU usage
- **Default (100ms)**: Good balance
- **Slow (500ms+)**: Lower CPU usage, more latency
**TCP vs HTTP behavior:**
- **TCP**: Raw stream with kernel-level buffering. Drops occur when TCP send buffer fills
- **HTTP/SSE**: Application-level buffering. Each client has a dedicated channel (default: 1000 entries)
### Rate Limiting
**Mitigation strategies:**
1. Increase buffer sizes for burst tolerance: `--tcp-buffer-size 5000` or `--http-buffer-size 5000`
2. Implement client-side flow control (pause/resume based on queue depth)
3. Use TCP for high-volume consumers that need guaranteed delivery
4. Keep browser tabs in foreground for real-time monitoring
5. Consider log aggregation/filtering at source for high-volume scenarios
When to enable rate limiting:
- Internet-facing deployments
- Shared environments
- Protection against misbehaving clients
Rate limiting applies only to establishing SSE connections, not to individual messages. Once connected, clients receive all messages (subject to buffer capacity).
## Troubleshooting
### Client Missing Messages
If clients miss messages during bursts:
1. Check `total_dropped` and `clients_with_drops` in status endpoint
2. Increase `stream.buffer_size` to handle larger bursts
3. Messages are skipped when buffer is full, but clients stay connected
### High Memory Usage
If memory usage is high:
1. Reduce `stream.buffer_size`
2. Enable rate limiting to limit concurrent connections
3. Each client uses `buffer_size * avg_message_size` memory
### Browser Stops Receiving Updates
This shouldn't happen with the current implementation. If it does:
1. Check browser developer console for errors
2. Verify no proxy/firewall is timing out the connection
3. Ensure reverse proxy (if used) doesn't buffer SSE responses
## File Rotation Detection
LogWisp automatically detects log file rotation using multiple methods:
- Inode changes (Linux/Unix)
- File size decrease
- Modification time reset
- Read position beyond file size
When rotation is detected, LogWisp:
1. Logs a rotation event
2. Resets read position to beginning
3. Continues streaming from new file
## Security Notes
1. **No built-in authentication** - Use a reverse proxy for auth
2. **No TLS support** - Use a reverse proxy for HTTPS
3. **Path validation** - Only specified paths can be monitored
4. **Directory traversal protection** - Paths containing ".." are rejected
5. **Rate limiting** - Optional but recommended for public deployments
6. **ANSI escape sequences** - Only enable color mode for trusted log sources
## Design Decisions
- **Unix philosophy**: Single purpose - stream logs
- **SSE over WebSocket**: Simpler, works everywhere, built-in reconnect
- **No database**: Stateless operation, instant startup
- **Atomic config management**: Using LixenWraith/config package
- **Graceful shutdown**: Proper cleanup on SIGINT/SIGTERM
- **Platform agnostic**: POSIX-compliant where possible
**Monitoring drops:**
- HTTP: Check `/status` endpoint for drop statistics
- TCP: Monitor connection count and system TCP metrics
- Both: Watch for "channel full" indicators in client implementations
## Building from Source
Requirements:
- Go 1.23 or later
```bash
git clone https://github.com/yourusername/logwisp
cd logwisp
go mod download
go mod init logwisp
go get github.com/panjf2000/gnet/v2
go get github.com/valyala/fasthttp
go get github.com/lixenwraith/config
go build -o logwisp ./src/cmd/logwisp
```