lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.1.2 update readme and config, failed attempt to fix slow client

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-01 16:34:19 -04:00
parent bd13103a81
commit a3450a9589
6 changed files with 386 additions and 182 deletions
Download Patch File Download Diff File Expand all files Collapse all files

265
README.md
View File

@ -1,3 +1,7 @@
<p align="center">
<img src="assets/logo.svg" alt="LogWisp Logo" width="200"/>
</p>
# LogWisp - Simple Log Streaming
A lightweight log streaming service that monitors files and streams updates via Server-Sent Events (SSE).
@ -16,6 +20,7 @@ LogWisp follows the Unix philosophy: do one thing and do it well. It monitors lo
- Environment variable support
- Simple TOML configuration
- Atomic configuration management
- Optional ANSI color pass-through
## Quick Start
@ -34,13 +39,61 @@ LogWisp follows the Unix philosophy: do one thing and do it well. It monitors lo
curl -N http://localhost:8080/stream
```
## Command Line Options
```bash
logwisp [OPTIONS] [TARGET...]
OPTIONS:
-c, --color Enable color pass-through for ANSI escape codes
--config FILE Config file path (default: ~/.config/logwisp.toml)
--port PORT HTTP port (default: 8080)
--buffer-size SIZE Stream buffer size (default: 1000)
--check-interval MS File check interval in ms (default: 100)
--rate-limit Enable rate limiting
--rate-requests N Rate limit requests/sec (default: 10)
--rate-burst N Rate limit burst size (default: 20)
TARGET:
path[:pattern[:isfile]] Path to monitor (file or directory)
pattern: glob pattern for directories (default: *.log)
isfile: true/false (auto-detected if omitted)
EXAMPLES:
# Monitor current directory for *.log files
logwisp
# Monitor specific file with color support
logwisp -c /var/log/app.log
# Monitor multiple locations
logwisp /var/log:*.log /app/logs:error*.log:false /tmp/debug.log::true
# Custom port with rate limiting
logwisp --port 9090 --rate-limit --rate-requests 100 --rate-burst 200
```
## Configuration
LogWisp uses a three-level configuration hierarchy:
1. **Environment variables** (highest priority)
2. **Configuration file** (~/.config/logwisp.toml)
3. **Default values** (lowest priority)
1. **Command-line arguments** (highest priority)
2. **Environment variables**
3. **Configuration file** (~/.config/logwisp.toml)
4. **Default values** (lowest priority)
### Default Values
| Setting | Default | Description |
|---------|---------|-------------|
| `port` | 8080 | HTTP listen port |
| `monitor.check_interval_ms` | 100 | File check interval (milliseconds) |
| `monitor.targets` | [{"path": "./", "pattern": "*.log", "is_file": false}] | Paths to monitor |
| `stream.buffer_size` | 1000 | Per-client event buffer size |
| `stream.rate_limit.enabled` | false | Enable rate limiting |
| `stream.rate_limit.requests_per_second` | 10 | Sustained request rate |
| `stream.rate_limit.burst_size` | 20 | Maximum burst size |
| `stream.rate_limit.cleanup_interval_s` | 60 | Client cleanup interval |
### Configuration File Location
@ -75,7 +128,7 @@ All configuration values can be overridden via environment variables:
| `LOGWISP_STREAM_RATE_LIMIT_ENABLED` | `stream.rate_limit.enabled` | Enable rate limiting |
| `LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC` | `stream.rate_limit.requests_per_second` | Rate limit |
| `LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE` | `stream.rate_limit.burst_size` | Burst size |
| `LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL` | `stream.rate_limit.cleanup_interval_s` | Cleanup interval |
| `LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL_S` | `stream.rate_limit.cleanup_interval_s` | Cleanup interval |
### Monitor Targets Format
@ -93,17 +146,22 @@ LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp
```
### Example Configuration
### Complete Configuration Example
```toml
# Port to listen on (default: 8080)
port = 8080
[monitor]
# How often to check for file changes in milliseconds (default: 100)
check_interval_ms = 100
# Monitor directory (all .log files)
# Paths to monitor
# Default: [{"path": "./", "pattern": "*.log", "is_file": false}]
# Monitor all .log files in current directory
[[monitor.targets]]
path = "/var/log"
path = "./"
pattern = "*.log"
is_file = false
@ -120,12 +178,24 @@ pattern = "access*.log"
is_file = false
[stream]
# Buffer size for each client connection (default: 1000)
# Controls how many log entries can be queued per client
buffer_size = 1000
[stream.rate_limit]
enabled = true
# Enable rate limiting (default: false)
enabled = false
# Requests per second per client (default: 10)
# This is the sustained rate
requests_per_second = 10
# Burst size - max requests at once (default: 20)
# Allows temporary bursts above the sustained rate
burst_size = 20
# How often to clean up old client limiters in seconds (default: 60)
# Clients inactive for 2x this duration are removed
cleanup_interval_s = 60
```
@ -141,7 +211,7 @@ LogWisp can pass through ANSI color escape codes from monitored logs to SSE clie
ExecStart=/opt/logwisp/bin/logwisp -c
```
## How It Works
### How It Works
When color mode is enabled (`-c` flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream.
@ -163,9 +233,9 @@ SSE output with `-c`:
}
```
## Client-Side Handling
### Client-Side Handling
### Terminal Clients
#### Terminal Clients
For terminal-based clients (like curl), the escape codes will render as colors:
@ -174,7 +244,7 @@ For terminal-based clients (like curl), the escape codes will render as colors:
curl -N http://localhost:8080/stream | jq -r '.message'
```
### Web Clients
#### Web Clients
For web-based clients, you'll need to convert ANSI codes to HTML:
@ -190,7 +260,7 @@ eventSource.onmessage = (event) => {
};
```
### Custom Processing
#### Custom Processing
```python
# Python example with colorama
@ -245,13 +315,13 @@ To strip color codes instead of passing them through:
### Endpoints
- `GET /stream` - Server-Sent Events stream of log entries
- `GET /status` - Service status information
- `GET /status` - Service status and configuration information
### Log Entry Format
```json
{
"time": "2024-01-01T12:00:00Z",
"time": "2024-01-01T12:00:00.123456Z",
"source": "app.log",
"level": "error",
"message": "Something went wrong",
@ -259,6 +329,48 @@ To strip color codes instead of passing them through:
}
```
### SSE Event Types
| Event | Description | Data Format |
|-------|-------------|-------------|
| `connected` | Initial connection | `{"client_id": "123456789"}` |
| `data` | Log entry | JSON log entry |
| `disconnected` | Client disconnected | `{"reason": "slow_client"}` |
| `timeout` | Client timeout | `{"reason": "client_timeout"}` |
| `:` | Heartbeat (comment) | ISO timestamp |
### Status Response Format
```json
{
"service": "LogWisp",
"version": "2.0.0",
"port": 8080,
"color_mode": false,
"config": {
"monitor": {
"check_interval_ms": 100,
"targets_count": 2
},
"stream": {
"buffer_size": 1000,
"rate_limit": {
"enabled": true,
"requests_per_second": 10,
"burst_size": 20
}
}
},
"streamer": {
"active_clients": 5,
"buffer_size": 1000,
"color_mode": false,
"total_dropped": 42
},
"rate_limiter": "Active clients: 3"
}
```
## Usage Examples
### Basic Usage
@ -266,6 +378,9 @@ To strip color codes instead of passing them through:
# Start with defaults
./logwisp
# Monitor specific file
./logwisp /var/log/app.log
# View logs
curl -N http://localhost:8080/stream
```
@ -281,6 +396,9 @@ LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \
### Monitor Multiple Locations
```bash
# Via command line
./logwisp /var/log:*.log /app/logs:*.json /tmp/debug.log
# Via environment variable
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \
./logwisp
@ -315,11 +433,13 @@ After=network.target
[Service]
Type=simple
User=logwisp
ExecStart=/usr/local/bin/logwisp
ExecStart=/usr/local/bin/logwisp -c
Restart=always
RestartSec=5
# Environment overrides
Environment="LOGWISP_PORT=8080"
Environment="LOGWISP_STREAM_BUFFER_SIZE=5000"
Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true"
Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100"
Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false"
@ -330,25 +450,92 @@ PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/
ReadWritePaths=/var/log
ReadWritePaths=/var/log /opt/app/logs
[Install]
WantedBy=multi-user.target
```
## Rate Limiting
## Performance Tuning
When enabled, rate limiting is applied per client IP address:
### Buffer Size
```toml
[stream.rate_limit]
enabled = true
requests_per_second = 10 # Sustained rate
burst_size = 20 # Allow bursts up to this size
cleanup_interval_s = 60 # Clean old clients every minute
```
The `stream.buffer_size` setting controls how many log entries can be queued per client:
- **Small buffers (100-500)**: Lower memory usage, clients skip entries during bursts
- **Default (1000)**: Good balance for most use cases
- **Large buffers (5000+)**: Handle burst traffic better, higher memory usage
Rate limiting uses the `X-Forwarded-For` header if present, falling back to `RemoteAddr`.
When a client's buffer is full, new messages are skipped for that client until it catches up. The client remains connected and will receive future messages once buffer space is available.
### Check Interval
The `monitor.check_interval_ms` setting controls file polling frequency:
- **Fast (10-50ms)**: Near real-time updates, higher CPU usage
- **Default (100ms)**: Good balance
- **Slow (500ms+)**: Lower CPU usage, more latency
### Rate Limiting
When to enable rate limiting:
- Internet-facing deployments
- Shared environments
- Protection against misbehaving clients
Rate limiting applies only to establishing SSE connections, not to individual messages. Once connected, clients receive all messages (subject to buffer capacity).
## Troubleshooting
### Client Missing Messages
If clients miss messages during bursts:
1. Check `total_dropped` and `clients_with_drops` in status endpoint
2. Increase `stream.buffer_size` to handle larger bursts
3. Messages are skipped when buffer is full, but clients stay connected
### High Memory Usage
If memory usage is high:
1. Reduce `stream.buffer_size`
2. Enable rate limiting to limit concurrent connections
3. Each client uses `buffer_size * avg_message_size` memory
### Browser Stops Receiving Updates
This shouldn't happen with the current implementation. If it does:
1. Check browser developer console for errors
2. Verify no proxy/firewall is timing out the connection
3. Ensure reverse proxy (if used) doesn't buffer SSE responses
## File Rotation Detection
LogWisp automatically detects log file rotation using multiple methods:
- Inode changes (Linux/Unix)
- File size decrease
- Modification time reset
- Read position beyond file size
When rotation is detected, LogWisp:
1. Logs a rotation event
2. Resets read position to beginning
3. Continues streaming from new file
## Security Notes
1. **No built-in authentication** - Use a reverse proxy for auth
2. **No TLS support** - Use a reverse proxy for HTTPS
3. **Path validation** - Only specified paths can be monitored
4. **Directory traversal protection** - Paths containing ".." are rejected
5. **Rate limiting** - Optional but recommended for public deployments
6. **ANSI escape sequences** - Only enable color mode for trusted log sources
## Design Decisions
- **Unix philosophy**: Single purpose - stream logs
- **SSE over WebSocket**: Simpler, works everywhere, built-in reconnect
- **No database**: Stateless operation, instant startup
- **Atomic config management**: Using LixenWraith/config package
- **Graceful shutdown**: Proper cleanup on SIGINT/SIGTERM
- **Platform agnostic**: POSIX-compliant where possible
## Building from Source
@ -356,32 +543,12 @@ Requirements:
- Go 1.23 or later
```bash
git clone https://github.com/yourusername/logwisp
cd logwisp
go mod download
go build -o logwisp ./src/cmd/logwisp
```
## File Rotation Detection
LogWisp automatically detects log file rotation by:
- Monitoring file inode changes (Linux/Unix)
- Detecting file size decrease
- Resetting read position when rotation is detected
## Security Notes
1. **No built-in authentication** - Use a reverse proxy for auth
2. **No TLS support** - Use a reverse proxy for HTTPS
3. **Path validation** - Monitors only specified paths
4. **Rate limiting** - Optional but recommended for internet-facing deployments
## Design Decisions
- **Unix philosophy**: Single purpose - stream logs
- **No CLI arguments**: Configuration via file and environment only
- **SSE over WebSocket**: Simpler, works everywhere
- **Atomic config management**: Using LixenWraith/config package
- **Graceful shutdown**: Proper cleanup on SIGINT/SIGTERM
## License
BSD-3-Clause