lixen/logwisp
SHA256
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.16.1 doc update

Browse Source
This commit is contained in:
lixen
2026-08-29 16:37:20 -04:00
parent 325b51d840
commit b2e36be53f
19 changed files with 3066 additions and 1323 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/formatters.md
+138 -143
View File
@@ -1,180 +1,175 @@
# Formatters
LogWisp formatters transform log entries before output to sinks.
The formatter is the last flow stage. It turns a `core.LogEntry` into the byte
payload that sinks write, applying a sanitizer policy on the way.
## Formatter Types
```toml
[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"
flags = 0
timestamp_format = ""
```
### Raw Formatter
One formatter serves the whole pipeline. Sinks receive an identical payload;
there is no per-sink formatting. When you need two shapes of the same data, run
two pipelines, or chain to a node that formats differently.
Outputs the log message as-is with optional newline.
Omitting `[pipelines.flow.format]` entirely selects `raw`.
## Options
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `type` | string | `raw` | `raw`, `txt` (alias `text`), or `json` |
| `sanitizer_policy` | string | derived from `type` | `raw`, `txt`, `json`, or `shell` |
| `flags` | int64 | `0` | Bitmask override; `0` selects a per-type default |
| `timestamp_format` | string | formatter default | Go reference layout, e.g. `"2006-01-02T15:04:05Z07:00"` |
## Types
### raw
Passthrough. `FlagRaw` bypasses both formatting and sanitization, so the
message reaches the sink exactly as the source produced it.
```toml
[pipelines.flow.format]
type = "raw"
sanitizer_policy = "raw"
flags = 1
```
**Configuration Options:**
Fastest option, and the right one when you are relaying text that is already in
its final form. Note that it also bypasses sanitization, so control characters
in the source data reach your sinks intact.
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `add_new_line` | bool | true | Append newline to messages |
| `type` | string | "raw" | raw, json, or txt |
| `flags` | int64 | 0 | log/formatter flags override |
| `sanitizer_policy` | string | | Sanitizer policy (e.g. "json", "raw", "txt", "shell") |
### txt
### JSON Formatter
Produces structured JSON output.
Human-readable line output with a timestamp and level.
```toml
[pipelines.format]
type = "json"
[pipelines.flow.format]
type = "txt"
sanitizer_policy = "txt"
timestamp_format = "2006-01-02 15:04:05"
```
+[pipelines.flow.format]
type = "json"
### json
Structured output, the natural choice for downstream ingestion.
```toml
[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"
```
**Output Structure:**
Output has the shape:
```json
{
"timestamp": "2024-01-01T12:00:00Z",
"level": "ERROR",
"source": "app",
"message": "Connection failed"
}
{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}
```
### Text Formatter
The exact key names and structure come from the `lixenwraith/log` formatter, not
from LogWisp; they are stable for a given dependency version but are not part of
LogWisp's own configuration surface.
Template-based text formatting.
## Flags
```toml
[pipelines.flow.format]
type = "txt"
sanitizer_policy = "txt"
timestamp_format = "2006-01-02T15:04:05.000Z07:00"
`flags` is a bitmask passed to the underlying formatter. Leave it at `0` unless
you need to override the defaults.
| Value | Name | Effect |
|-------|------|--------|
| `1` | Raw | Bypass formatting and sanitization entirely |
| `2` | ShowTimestamp | Emit the timestamp |
| `4` | ShowLevel | Emit the level |
| `8` | StructuredJSON | Render attached fields as a JSON object |
| `16` | NoTimestamp | Suppress the timestamp |
| `32` | NoLevel | Suppress the level |
With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
(timestamp + level) for every other type. `8` is added automatically whenever an
entry carries parseable `fields`.
Examples: `flags = 4` for level only, no timestamp; `flags = 2` for timestamp
only, no level.
## Sanitizer Policies
The sanitizer runs before serialization and neutralizes control characters that
would otherwise break framing or reach a terminal.
| Policy | Behaviour | Use with |
|--------|-----------|----------|
| `raw` | No-op passthrough | `type = "raw"` where you control the data |
| `txt` | Escapes non-printable characters | File and console sinks |
| `json` | Escapes control characters for safe JSON embedding | `type = "json"`, chain links |
| `shell` | Strips shell metacharacters, whitespace, and control characters | Data that will be passed to a command |
When `sanitizer_policy` is omitted, the policy is derived from `type`: `json`
for `json`, `txt` for `txt`/`text`, and `raw` for anything else — so the safe
pairing is the default.
> `shell` strips dangerous characters but is **not** sufficient to make a string
> safe for shell construction. Pass arguments through `exec` argv instead of
> building command lines.
To see a policy working, point a pipeline at the `random` source with
`special = true`, which injects control bytes and multi-byte Unicode into every
message.
## Node Identity in Output
Entries that arrived over a chain link carry a `Node` label. The formatter
renders it as a syslog-style prefix on the source field:
```
edge-01/app.log
```
**Configuration Options:**
Entries with no node label show the bare source. Node identity therefore appears
*inside* the source field rather than as a separate output key — worth knowing
when writing downstream parsers or grep patterns.
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `timestamp_format` | string | "" | Time format override |
## Structured Fields
**Default Template:**
```
[{{.Timestamp | FmtTime}}] [{{.Level | ToUpper}}] {{.Source}} - {{.Message}}{{ if .Fields }} {{.Fields}}{{ end }}
```
When an entry carries `Fields` (raw JSON), the formatter parses it and switches
to structured rendering by adding the `StructuredJSON` flag automatically.
Fields reach a pipeline in two ways: from the `file` source when a tailed line
parses as JSON with a `fields` key, and from the heartbeat generator when
`include_stats = true`.
## Template Functions
## Choosing a Configuration
Available functions in text templates:
| Goal | Configuration |
|------|---------------|
| Maximum throughput, data already formatted | `type = "raw"` |
| Human reading in a terminal or file | `type = "txt"`, `sanitizer_policy = "txt"` |
| Downstream ingestion (Loki, Elasticsearch, jq) | `type = "json"`, `sanitizer_policy = "json"` |
| Compact console output | `type = "txt"`, `flags = 4` |
| Untrusted log content | never `raw`; pick `txt` or `json` and set the matching policy |
| Function | Description | Example |
|----------|-------------|---------|
| `FmtTime` | Format timestamp | `{{.Timestamp \| FmtTime}}` |
| `ToUpper` | Convert to uppercase | `{{.Level \| ToUpper}}` |
| `ToLower` | Convert to lowercase | `{{.Source \| ToLower}}` |
| `TrimSpace` | Remove whitespace | `{{.Message \| TrimSpace}}` |
## Formatting and Chain Links
## Template Variables
Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload.
They re-serialize the structured entry into the canonical chain encoding, which
makes them independent of the local formatter.
Available variables in templates:
The practical consequence: setting `flow.format` on an edge node changes only
that node's own local sinks. The output shape seen by a human or a downstream
system is decided on the node that owns the sink they read.
| Variable | Type | Description |
|----------|------|-------------|
| `.Timestamp` | time.Time | Entry timestamp |
| `.Level` | string | Log level |
| `.Source` | string | Source identifier |
| `.Message` | string | Log message |
| `.Fields` | string | Additional fields (JSON) |
If an event ever reaches a chain sink without a structured entry, the sink wraps
the formatted payload into a synthetic entry and counts it in `synthesized`.
A non-zero `synthesized` count means something upstream lost structure.
## Time Format Strings
## Performance
Common Go time format patterns:
Relative cost, cheapest first: `raw` (passthrough) → `txt` (line assembly) →
`json` (serialization). Sanitization adds a scan of the message; the `raw`
policy skips it.
| Pattern | Example Output |
|---------|---------------|
| `2006-01-02T15:04:05Z07:00` | 2024-01-02T15:04:05Z |
| `2006-01-02 15:04:05` | 2024-01-02 15:04:05 |
| `Jan 2 15:04:05` | Jan 2 15:04:05 |
| `15:04:05.000` | 15:04:05.123 |
| `2006/01/02` | 2024/01/02 |
## Format Selection
### Default Behavior
If no formatter specified:
- **HTTP/TCP sinks**: JSON format
- **Console/File sinks**: Raw format
- **Client sinks**: JSON format
### Per-Pipeline Configuration
Each pipeline can have its own formatter:
```toml
[[pipelines]]
name = "json-pipeline"
[pipelines.flow.format]
type = "json"
[[pipelines]]
name = "text-pipeline"
[pipelines.flow.format]
type = "txt"
```
## Message Processing
### JSON Message Handling
When using JSON formatter with JSON log messages:
1. Attempts to parse message as JSON
2. Merges fields with LogWisp metadata
3. LogWisp fields take precedence
4. Falls back to string if parsing fails
### Field Preservation
LogWisp metadata always includes:
- Timestamp (from source or current time)
- Level (detected or default)
- Source (origin identifier)
- Message (original content)
## Performance Characteristics
### Formatter Performance
Relative performance (fastest to slowest):
1. **Raw**: Direct passthrough
2. **Text**: Template execution
3. **JSON**: Serialization
4. **JSON (pretty)**: Formatted serialization
### Optimization Tips
- Use raw format for high throughput
- Cache template compilation (automatic)
- Minimize template complexity
- Avoid pretty JSON in production
## Common Configurations
### Structured Logging
```toml
[pipelines.flow.format]
type = "json"
```
### Human-Readable Logs
```toml
[pipelines.flow.format]
type = "txt"
timestamp_format = "15:04:05"
```
The formatter holds a mutex because the underlying implementation reuses an
internal buffer and is not goroutine-safe. It is the only shared serialization
point in the hot path, and the reason a single pipeline formats entries one at
a time.