log/doc/heartbeat.md

Heartbeat Monitoring

Guide to using heartbeat messages for operational monitoring and system health tracking.

Overview

Heartbeats are periodic log messages that provide operational statistics about the logger and system. They bypass normal log level filtering, ensuring visibility even when running at higher log levels.

Key Features

• Independent of level filtering: Heartbeats can still be dropped on queue overflow or output failure
• Multi-Level Detail: Choose from process, disk, or system statistics
• Production Monitoring: Track logger health without debug logs
• Metrics Source: Parse heartbeats for monitoring dashboards

Heartbeat Levels

Level 0: Disabled (Default)

No heartbeat messages are generated.

logger.ApplyConfigString(
    "heartbeat_level=0",  // No heartbeats
)

Level 1: Process Statistics (PROC)

Basic logger operation metrics:

logger.ApplyConfigString(
    "heartbeat_level=1",
    "heartbeat_interval_s=300",  // Every 5 minutes
)

Output:

2024-01-15T10:30:00Z PROC type=proc sequence=1 uptime_hours="24.50" processed_logs=1847293 total_dropped_logs=0

Fields:

• sequence: Incrementing counter
• uptime_hours: Logger uptime
• processed_logs: Application records processed successfully, excluding heartbeats
• total_dropped_logs: Cumulative queue and output-processing drops, including heartbeats
• dropped_since_last: Interval drops, emitted only when non-zero; reset at each PROC heartbeat attempt

Level 2: Process + Disk Statistics (DISK)

Includes file and disk usage information:

logger.ApplyConfigString(
    "heartbeat_level=2",
    "heartbeat_interval_s=300",
)

Additional Output:

2024-01-15T10:30:00Z DISK type=disk sequence=1 rotated_files=12 deleted_files=5 reopened_files=1 total_log_size_mb="487.32" log_file_count=8 current_file_size_mb="23.45" disk_status_ok=true disk_free_mb="5234.67"

Additional Fields:

• rotated_files: Total file rotations
• deleted_files: Files removed by cleanup
• reopened_files: Log paths opened again after an external rename or removal
• total_log_size_mb: Size of all log files
• log_file_count: Number of log files
• current_file_size_mb: Active file size
• disk_status_ok: Disk health status
• disk_free_mb: Available disk space

Level 3: Process + Disk + System Statistics (SYS)

Includes runtime and memory metrics:

logger.ApplyConfigString(
    "heartbeat_level=3",
    "heartbeat_interval_s=60",  // Every minute for detailed monitoring
)

Additional Output:

2024-01-15T10:30:00Z SYS type=sys sequence=1 alloc_mb="45.23" sys_mb="128.45" num_gc=1523 num_goroutine=42

Additional Fields:

• alloc_mb: Allocated memory
• sys_mb: System memory reserved
• num_gc: Garbage collection runs
• num_goroutine: Active goroutines

Configuration

Basic Configuration

logger.ApplyConfigString(
    "heartbeat_level=2",         // Process + Disk stats
    "heartbeat_interval_s=300",  // Every 5 minutes
)

Interval Recommendations

Environment	Level	Interval	Rationale
Development	3	30s	Detailed debugging info
Staging	2	300s	Balance detail vs noise
Production	1-2	300-600s	Minimize overhead
High-Load	1	600s	Reduce I/O impact

Dynamic Adjustment

// Start with basic monitoring
logger.ApplyConfigString(
    "heartbeat_level=1",
    "heartbeat_interval_s=600",
)

// During incident, increase detail
logger.ApplyConfigString(
    "heartbeat_level=3",
    "heartbeat_interval_s=60",
)

// After resolution, reduce back
logger.ApplyConfigString(
    "heartbeat_level=1",
    "heartbeat_interval_s=600",
)

Heartbeat Messages

JSON Format Example

With format=json, heartbeats are structured for easy parsing:

{
  "time": "2024-01-15T10:30:00.123456789Z",
  "level": "PROC",
  "fields": {
    "type": "proc",
    "sequence": 42,
    "uptime_hours": "24.50",
    "processed_logs": 1847293,
    "total_dropped_logs": 0
  }
}

Text Format Example

With format=txt, heartbeats are human-readable:

2024-01-15T10:30:00.123456789Z PROC type=proc sequence=42 uptime_hours="24.50" processed_logs=1847293 total_dropped_logs=0

PROC uptime and totals span the lifetime of the Logger instance, including stop/start cycles. DISK fields ending in _mb retain historical binary MiB units (1,048,576 bytes); SYS memory fields use decimal MB. Configuration size limits use decimal KB.