181 lines
3.9 KiB
Markdown
181 lines
3.9 KiB
Markdown
# Formatters
|
|
|
|
LogWisp formatters transform log entries before output to sinks.
|
|
|
|
## Formatter Types
|
|
|
|
### Raw Formatter
|
|
|
|
Outputs the log message as-is with optional newline.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "raw"
|
|
sanitizer_policy = "raw"
|
|
flags = 1
|
|
```
|
|
|
|
**Configuration Options:**
|
|
|
|
| Option | Type | Default | Description |
|
|
|--------|------|---------|-------------|
|
|
| `add_new_line` | bool | true | Append newline to messages |
|
|
| `type` | string | "raw" | raw, json, or txt |
|
|
| `flags` | int64 | 0 | log/formatter flags override |
|
|
| `sanitizer_policy` | string | | Sanitizer policy (e.g. "json", "raw", "txt", "shell") |
|
|
|
|
### JSON Formatter
|
|
|
|
Produces structured JSON output.
|
|
|
|
```toml
|
|
[pipelines.format]
|
|
type = "json"
|
|
|
|
+[pipelines.flow.format]
|
|
type = "json"
|
|
sanitizer_policy = "json"
|
|
```
|
|
|
|
**Output Structure:**
|
|
```json
|
|
{
|
|
"timestamp": "2024-01-01T12:00:00Z",
|
|
"level": "ERROR",
|
|
"source": "app",
|
|
"message": "Connection failed"
|
|
}
|
|
```
|
|
|
|
### Text Formatter
|
|
|
|
Template-based text formatting.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "txt"
|
|
sanitizer_policy = "txt"
|
|
timestamp_format = "2006-01-02T15:04:05.000Z07:00"
|
|
```
|
|
|
|
**Configuration Options:**
|
|
|
|
| Option | Type | Default | Description |
|
|
|--------|------|---------|-------------|
|
|
| `timestamp_format` | string | "" | Time format override |
|
|
|
|
**Default Template:**
|
|
```
|
|
[{{.Timestamp | FmtTime}}] [{{.Level | ToUpper}}] {{.Source}} - {{.Message}}{{ if .Fields }} {{.Fields}}{{ end }}
|
|
```
|
|
|
|
## Template Functions
|
|
|
|
Available functions in text templates:
|
|
|
|
| Function | Description | Example |
|
|
|----------|-------------|---------|
|
|
| `FmtTime` | Format timestamp | `{{.Timestamp \| FmtTime}}` |
|
|
| `ToUpper` | Convert to uppercase | `{{.Level \| ToUpper}}` |
|
|
| `ToLower` | Convert to lowercase | `{{.Source \| ToLower}}` |
|
|
| `TrimSpace` | Remove whitespace | `{{.Message \| TrimSpace}}` |
|
|
|
|
## Template Variables
|
|
|
|
Available variables in templates:
|
|
|
|
| Variable | Type | Description |
|
|
|----------|------|-------------|
|
|
| `.Timestamp` | time.Time | Entry timestamp |
|
|
| `.Level` | string | Log level |
|
|
| `.Source` | string | Source identifier |
|
|
| `.Message` | string | Log message |
|
|
| `.Fields` | string | Additional fields (JSON) |
|
|
|
|
## Time Format Strings
|
|
|
|
Common Go time format patterns:
|
|
|
|
| Pattern | Example Output |
|
|
|---------|---------------|
|
|
| `2006-01-02T15:04:05Z07:00` | 2024-01-02T15:04:05Z |
|
|
| `2006-01-02 15:04:05` | 2024-01-02 15:04:05 |
|
|
| `Jan 2 15:04:05` | Jan 2 15:04:05 |
|
|
| `15:04:05.000` | 15:04:05.123 |
|
|
| `2006/01/02` | 2024/01/02 |
|
|
|
|
## Format Selection
|
|
|
|
### Default Behavior
|
|
|
|
If no formatter specified:
|
|
- **HTTP/TCP sinks**: JSON format
|
|
- **Console/File sinks**: Raw format
|
|
- **Client sinks**: JSON format
|
|
|
|
### Per-Pipeline Configuration
|
|
|
|
Each pipeline can have its own formatter:
|
|
|
|
```toml
|
|
[[pipelines]]
|
|
name = "json-pipeline"
|
|
[pipelines.flow.format]
|
|
type = "json"
|
|
|
|
[[pipelines]]
|
|
name = "text-pipeline"
|
|
[pipelines.flow.format]
|
|
type = "txt"
|
|
```
|
|
|
|
## Message Processing
|
|
|
|
### JSON Message Handling
|
|
|
|
When using JSON formatter with JSON log messages:
|
|
1. Attempts to parse message as JSON
|
|
2. Merges fields with LogWisp metadata
|
|
3. LogWisp fields take precedence
|
|
4. Falls back to string if parsing fails
|
|
|
|
### Field Preservation
|
|
|
|
LogWisp metadata always includes:
|
|
- Timestamp (from source or current time)
|
|
- Level (detected or default)
|
|
- Source (origin identifier)
|
|
- Message (original content)
|
|
|
|
## Performance Characteristics
|
|
|
|
### Formatter Performance
|
|
|
|
Relative performance (fastest to slowest):
|
|
1. **Raw**: Direct passthrough
|
|
2. **Text**: Template execution
|
|
3. **JSON**: Serialization
|
|
4. **JSON (pretty)**: Formatted serialization
|
|
|
|
### Optimization Tips
|
|
|
|
- Use raw format for high throughput
|
|
- Cache template compilation (automatic)
|
|
- Minimize template complexity
|
|
- Avoid pretty JSON in production
|
|
|
|
## Common Configurations
|
|
|
|
### Structured Logging
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "json"
|
|
```
|
|
|
|
### Human-Readable Logs
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "txt"
|
|
timestamp_format = "15:04:05"
|
|
```
|