lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.1.1 improved config, added ratelimiter (buggy), readme not fully updated

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-01 14:12:20 -04:00
parent 294771653c
commit bd13103a81
11 changed files with 1329 additions and 194 deletions
Download Patch File Download Diff File Expand all files Collapse all files

321
README.md
View File

@ -8,12 +8,14 @@ LogWisp follows the Unix philosophy: do one thing and do it well. It monitors lo
## Features
- Monitors multiple files and directories
- Monitors multiple files and directories simultaneously
- Streams log updates in real-time via SSE
- Supports both plain text and JSON formatted logs
- Automatic file rotation detection
- Configurable rate limiting
- Environment variable support
- Simple TOML configuration
- No authentication or complex features - use a reverse proxy if needed
- Atomic configuration management
## Quick Start
@ -34,32 +36,219 @@ curl -N http://localhost:8080/stream
## Configuration
LogWisp looks for configuration at `~/.config/logwisp.toml`. If not found, it uses sensible defaults.
LogWisp uses a three-level configuration hierarchy:
1. **Environment variables** (highest priority)
2. **Configuration file** (~/.config/logwisp.toml)
3. **Default values** (lowest priority)
### Configuration File Location
Default: `~/.config/logwisp.toml`
Override with environment variables:
- `LOGWISP_CONFIG_DIR` - Directory containing config file
- `LOGWISP_CONFIG_FILE` - Config filename (absolute or relative)
Examples:
```bash
# Use config from current directory
LOGWISP_CONFIG_DIR=. ./logwisp
# Use specific config file
LOGWISP_CONFIG_FILE=/etc/logwisp/prod.toml ./logwisp
# Use custom directory and filename
LOGWISP_CONFIG_DIR=/opt/configs LOGWISP_CONFIG_FILE=myapp.toml ./logwisp
```
### Environment Variables
All configuration values can be overridden via environment variables:
| Environment Variable | Config Path | Description |
|---------------------|-------------|-------------|
| `LOGWISP_PORT` | `port` | HTTP listen port |
| `LOGWISP_MONITOR_CHECK_INTERVAL_MS` | `monitor.check_interval_ms` | File check interval |
| `LOGWISP_MONITOR_TARGETS` | `monitor.targets` | Comma-separated targets |
| `LOGWISP_STREAM_BUFFER_SIZE` | `stream.buffer_size` | Client buffer size |
| `LOGWISP_STREAM_RATE_LIMIT_ENABLED` | `stream.rate_limit.enabled` | Enable rate limiting |
| `LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC` | `stream.rate_limit.requests_per_second` | Rate limit |
| `LOGWISP_STREAM_RATE_LIMIT_BURST_SIZE` | `stream.rate_limit.burst_size` | Burst size |
| `LOGWISP_STREAM_RATE_LIMIT_CLEANUP_INTERVAL` | `stream.rate_limit.cleanup_interval_s` | Cleanup interval |
### Monitor Targets Format
The `LOGWISP_MONITOR_TARGETS` environment variable uses a special format:
```
path:pattern:isfile,path2:pattern2:isfile
```
Examples:
```bash
# Monitor directory and specific file
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/app.log::true" ./logwisp
# Multiple directories
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/opt/app/logs:app-*.log:false" ./logwisp
```
### Example Configuration
Example configuration:
```toml
port = 8080
[monitor]
check_interval_ms = 100
# Monitor directory (all .log files)
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false
# Monitor specific file
[[monitor.targets]]
path = "/home/user/app/logs"
pattern = "app-*.log"
path = "/app/logs/app.log"
pattern = "" # Ignored for files
is_file = true
# Monitor with specific pattern
[[monitor.targets]]
path = "/var/log/nginx"
pattern = "access*.log"
is_file = false
[stream]
buffer_size = 1000
[stream.rate_limit]
enabled = true
requests_per_second = 10
burst_size = 20
cleanup_interval_s = 60
```
## Color Support
LogWisp can pass through ANSI color escape codes from monitored logs to SSE clients using the `-c` flag.
```bash
# Enable color pass-through
./logwisp -c
# Or via systemd
ExecStart=/opt/logwisp/bin/logwisp -c
```
## How It Works
When color mode is enabled (`-c` flag), LogWisp preserves ANSI escape codes in log messages. These are properly JSON-escaped in the SSE stream.
### Example Log with Colors
Original log file content:
```
\033[31mERROR\033[0m: Database connection failed
\033[33mWARN\033[0m: High memory usage detected
\033[32mINFO\033[0m: Service started successfully
```
SSE output with `-c`:
```json
{
"time": "2024-01-01T12:00:00.123456Z",
"source": "app.log",
"message": "\u001b[31mERROR\u001b[0m: Database connection failed"
}
```
## Client-Side Handling
### Terminal Clients
For terminal-based clients (like curl), the escape codes will render as colors:
```bash
# This will show colored output in terminals that support ANSI codes
curl -N http://localhost:8080/stream | jq -r '.message'
```
### Web Clients
For web-based clients, you'll need to convert ANSI codes to HTML:
```javascript
// Example using ansi-to-html library
const AnsiToHtml = require('ansi-to-html');
const convert = new AnsiToHtml();
eventSource.onmessage = (event) => {
const data = JSON.parse(event.data);
const html = convert.toHtml(data.message);
document.getElementById('log').innerHTML += html + '<br>';
};
```
### Custom Processing
```python
# Python example with colorama
import json
import colorama
from colorama import init
init() # Initialize colorama for Windows support
# Process SSE stream
for line in stream:
if line.startswith('data: '):
data = json.loads(line[6:])
# Colorama will handle ANSI codes automatically
print(data['message'])
```
### Common ANSI Color Codes
| Code | Color/Style |
|------|-------------|
| `\033[0m` | Reset |
| `\033[1m` | Bold |
| `\033[31m` | Red |
| `\033[32m` | Green |
| `\033[33m` | Yellow |
| `\033[34m` | Blue |
| `\033[35m` | Magenta |
| `\033[36m` | Cyan |
### Limitations
1. **JSON Escaping**: ANSI codes are JSON-escaped in the stream (e.g., `\033` becomes `\u001b`)
2. **Client Support**: The client must support or convert ANSI codes
3. **Performance**: No significant impact, but slightly larger message sizes
### Security Note
Color codes are passed through as-is. Ensure monitored logs come from trusted sources to avoid terminal escape sequence attacks.
### Disabling Colors
To strip color codes instead of passing them through:
- Don't use the `-c` flag
- Or set up a preprocessing pipeline:
```bash
tail -f colored.log | sed 's/\x1b\[[0-9;]*m//g' > plain.log
```
## API
- `GET /stream` - Server-Sent Events stream of log entries
### Endpoints
- `GET /stream` - Server-Sent Events stream of log entries
- `GET /status` - Service status information
### Log Entry Format
Log entry format:
```json
{
"time": "2024-01-01T12:00:00Z",
@ -70,52 +259,54 @@ Log entry format:
}
```
## Building from Source
Requirements:
- Go 1.23 or later
```bash
go mod download
go build -o logwisp ./src/cmd/logwisp
```
## Usage Examples
### Basic Usage
```bash
# Start LogWisp (monitors current directory by default)
# Start with defaults
./logwisp
# In another terminal, view the stream
# View logs
curl -N http://localhost:8080/stream
```
### With Custom Config
### With Environment Variables
```bash
# Create config
# Change port and add rate limiting
LOGWISP_PORT=9090 \
LOGWISP_STREAM_RATE_LIMIT_ENABLED=true \
LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=5 \
./logwisp
```
### Monitor Multiple Locations
```bash
# Via environment variable
LOGWISP_MONITOR_TARGETS="/var/log:*.log:false,/app/logs:*.json:false,/tmp/debug.log::true" \
./logwisp
# Or via config file
cat > ~/.config/logwisp.toml << EOF
port = 9090
[[monitor.targets]]
path = "/var/log"
pattern = "*.log"
is_file = false
[[monitor.targets]]
path = "/var/log/nginx"
pattern = "*.log"
EOF
path = "/app/logs"
pattern = "*.json"
is_file = false
# Run
./logwisp
[[monitor.targets]]
path = "/tmp/debug.log"
is_file = true
EOF
```
### Production Deployment
For production use, consider:
Example systemd service with environment overrides:
1. Run behind a reverse proxy (nginx, caddy) for SSL/TLS
2. Use systemd or similar for process management
3. Add authentication at the proxy level if needed
4. Set appropriate file permissions on monitored logs
Example systemd service:
```ini
[Unit]
Description=LogWisp Log Streaming Service
@ -127,17 +318,69 @@ User=logwisp
ExecStart=/usr/local/bin/logwisp
Restart=always
# Environment overrides
Environment="LOGWISP_PORT=8080"
Environment="LOGWISP_STREAM_RATE_LIMIT_ENABLED=true"
Environment="LOGWISP_STREAM_RATE_LIMIT_REQUESTS_PER_SEC=100"
Environment="LOGWISP_MONITOR_TARGETS=/var/log:*.log:false,/opt/app/logs:*.log:false"
# Security hardening
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ProtectHome=true
ReadOnlyPaths=/
ReadWritePaths=/var/log
[Install]
WantedBy=multi-user.target
```
## Rate Limiting
When enabled, rate limiting is applied per client IP address:
```toml
[stream.rate_limit]
enabled = true
requests_per_second = 10 # Sustained rate
burst_size = 20 # Allow bursts up to this size
cleanup_interval_s = 60 # Clean old clients every minute
```
Rate limiting uses the `X-Forwarded-For` header if present, falling back to `RemoteAddr`.
## Building from Source
Requirements:
- Go 1.23 or later
```bash
go mod download
go build -o logwisp ./src/cmd/logwisp
```
## File Rotation Detection
LogWisp automatically detects log file rotation by:
- Monitoring file inode changes (Linux/Unix)
- Detecting file size decrease
- Resetting read position when rotation is detected
## Security Notes
1. **No built-in authentication** - Use a reverse proxy for auth
2. **No TLS support** - Use a reverse proxy for HTTPS
3. **Path validation** - Monitors only specified paths
4. **Rate limiting** - Optional but recommended for internet-facing deployments
## Design Decisions
- **No built-in authentication**: Use a reverse proxy
- **No TLS**: Use a reverse proxy
- **No complex features**: Follows Unix philosophy
- **File-based configuration**: Simple, no CLI args needed
- **Unix philosophy**: Single purpose - stream logs
- **No CLI arguments**: Configuration via file and environment only
- **SSE over WebSocket**: Simpler, works everywhere
- **Atomic config management**: Using LixenWraith/config package
- **Graceful shutdown**: Proper cleanup on SIGINT/SIGTERM
## License