lixen/log
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

e3.0.0 Tests added, optimization, bug fixes, builder changed.

Browse Source
This commit is contained in:
Lixen Wraith
2025-07-20 18:11:03 -04:00
parent 97b85995e9
commit 98402cce37
43 changed files with 2469 additions and 1373 deletions
Download Patch File Download Diff File Expand all files Collapse all files

153
doc/disk-management.md
View File

@ -9,7 +9,7 @@ Comprehensive guide to log file rotation, retention policies, and disk space man
Log files are automatically rotated when they reach the configured size limit:
```go
logger.ApplyOverride(
logger.ApplyConfigString(
"max_size_mb=100", // Rotate at 100MB
)
```
@ -43,7 +43,7 @@ Components:
The logger enforces two types of space limits:
```go
logger.ApplyOverride(
logger.ApplyConfigString(
"max_total_size_mb=1000", // Total log directory size
"min_disk_free_mb=5000", // Minimum free disk space
)
@ -61,21 +61,21 @@ When limits are exceeded, the logger:
```go
// Conservative: Strict limits
logger.ApplyOverride(
logger.ApplyConfigString(
"max_size_mb=50", // 50MB files
"max_total_size_mb=500", // 500MB total
"min_disk_free_mb=1000", // 1GB free required
)
// Generous: Large files, external archival
logger.ApplyOverride(
logger.ApplyConfigString(
"max_size_mb=1000", // 1GB files
"max_total_size_mb=0", // No total limit
"min_disk_free_mb=100", // 100MB free required
)
// Balanced: Production defaults
logger.ApplyOverride(
logger.ApplyConfigString(
"max_size_mb=100", // 100MB files
"max_total_size_mb=5000", // 5GB total
"min_disk_free_mb=500", // 500MB free required
@ -89,7 +89,7 @@ logger.ApplyOverride(
Automatically delete logs older than a specified duration:
```go
logger.ApplyOverride(
logger.ApplyConfigString(
"retention_period_hrs=168", // Keep 7 days
"retention_check_mins=60", // Check hourly
)
@ -99,21 +99,21 @@ logger.ApplyOverride(
```go
// Daily logs, keep 30 days
logger.ApplyOverride(
logger.ApplyConfigString(
"retention_period_hrs=720", // 30 days
"retention_check_mins=60", // Check hourly
"max_size_mb=1000", // 1GB daily files
)
// High-frequency logs, keep 24 hours
logger.ApplyOverride(
logger.ApplyConfigString(
"retention_period_hrs=24", // 1 day
"retention_check_mins=15", // Check every 15 min
"max_size_mb=100", // 100MB files
)
// Compliance: Keep 90 days
logger.ApplyOverride(
logger.ApplyConfigString(
"retention_period_hrs=2160", // 90 days
"retention_check_mins=360", // Check every 6 hours
"max_total_size_mb=100000", // 100GB total
@ -134,7 +134,7 @@ When multiple policies conflict, cleanup priority is:
The logger adjusts disk check frequency based on logging volume:
```go
logger.ApplyOverride(
logger.ApplyConfigString(
"enable_adaptive_interval=true",
"disk_check_interval_ms=5000", // Base: 5 seconds
"min_check_interval_ms=100", // Minimum: 100ms
@ -153,7 +153,7 @@ logger.ApplyOverride(
Check disk-related heartbeat messages:
```go
logger.ApplyOverride(
logger.ApplyConfigString(
"heartbeat_level=2", // Enable disk stats
"heartbeat_interval_s=300", // Every 5 minutes
)
@ -164,9 +164,7 @@ Output:
2024-01-15T10:30:00Z DISK type="disk" sequence=1 rotated_files=5 deleted_files=2 total_log_size_mb="487.32" log_file_count=8 current_file_size_mb="23.45" disk_status_ok=true disk_free_mb="5234.67"
```
## Recovery Behavior
### Manual Intervention
## Manual Recovery
If automatic cleanup fails:
@ -184,133 +182,6 @@ ls -t /var/log/myapp/*.log | tail -n 20 | xargs rm
df -h /var/log
```
## Best Practices
### 1. Plan for Growth
Estimate log volume and set appropriate limits:
```go
// Calculate required space:
// - Average log entry: 200 bytes
// - Entries per second: 100
// - Daily volume: 200 * 100 * 86400 = 1.7GB
logger.ApplyOverride(
"max_size_mb=2000", // 2GB files (~ 1 day)
"max_total_size_mb=15000", // 15GB (~ 1 week)
"retention_period_hrs=168", // 7 days
)
```
### 2. External Archival
For long-term storage, implement external archival:
```go
// Configure for archival
logger.ApplyOverride(
"max_size_mb=1000", // 1GB files for easy transfer
"max_total_size_mb=10000", // 10GB local buffer
"retention_period_hrs=48", // 2 days local
)
// Archive completed files
func archiveCompletedLogs(archivePath string) error {
files, _ := filepath.Glob("/var/log/myapp/*.log")
for _, file := range files {
if !isCurrentLogFile(file) {
// Move to archive storage (S3, NFS, etc.)
if err := archiveFile(file, archivePath); err != nil {
return err
}
os.Remove(file)
}
}
return nil
}
```
### 3. Monitor Disk Health
Set up alerts for disk issues:
```go
// Parse heartbeat logs for monitoring
type DiskStats struct {
TotalSizeMB float64
FileCount int
DiskFreeMB float64
DiskStatusOK bool
}
func monitorDiskHealth(logLine string) {
if strings.Contains(logLine, "type=\"disk\"") {
stats := parseDiskHeartbeat(logLine)
if !stats.DiskStatusOK {
alert("Log disk unhealthy")
}
if stats.DiskFreeMB < 1000 {
alert("Low disk space: %.0fMB free", stats.DiskFreeMB)
}
if stats.FileCount > 100 {
alert("Too many log files: %d", stats.FileCount)
}
}
}
```
### 4. Separate Log Volumes
Use dedicated volumes for logs:
```bash
# Create dedicated log volume
mkdir -p /mnt/logs
mount /dev/sdb1 /mnt/logs
# Configure logger
logger.ApplyOverride(
"directory=/mnt/logs/myapp",
"max_total_size_mb=50000", # Use most of volume
"min_disk_free_mb=1000", # Leave 1GB free
)
```
### 5. Test Cleanup Behavior
Verify cleanup works before production:
```go
// Test configuration
func TestDiskCleanup(t *testing.T) {
logger := log.NewLogger()
logger.ApplyOverride(
"directory=./test_logs",
"max_size_mb=1", // Small files
"max_total_size_mb=5", // Low limit
"retention_period_hrs=0.01", // 36 seconds
"retention_check_mins=0.5", // 30 seconds
)
// Generate logs to trigger cleanup
for i := 0; i < 1000; i++ {
logger.Info(strings.Repeat("x", 1000))
}
time.Sleep(45 * time.Second)
// Verify cleanup occurred
files, _ := filepath.Glob("./test_logs/*.log")
if len(files) > 5 {
t.Errorf("Cleanup failed: %d files remain", len(files))
}
}
```
---
[← Logging Guide](logging-guide.md) | [← Back to README](../README.md) | [Heartbeat Monitoring →](heartbeat-monitoring.md)