184 lines
6.8 KiB
Markdown
184 lines
6.8 KiB
Markdown
# Formatters
|
|
|
|
The formatter is the last flow stage. It turns a `core.LogEntry` into the byte
|
|
payload that sinks write, applying a sanitizer policy on the way.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "json"
|
|
sanitizer_policy = "json"
|
|
flags = 0
|
|
timestamp_format = ""
|
|
```
|
|
|
|
One formatter serves the whole pipeline. Sinks receive an identical payload;
|
|
there is no per-sink formatting. When you need two shapes of the same data, run
|
|
two pipelines, or chain to a node that formats differently.
|
|
|
|
Omitting `[pipelines.flow.format]` entirely selects `raw`.
|
|
|
|
## Options
|
|
|
|
| Option | Type | Default | Description |
|
|
|--------|------|---------|-------------|
|
|
| `type` | string | `raw` | `raw`, `txt` (alias `text`), or `json` |
|
|
| `sanitizer_policy` | string | derived from `type` | `raw`, `txt`, `json`, or `shell` |
|
|
| `flags` | int64 | `0` | Bitmask override; `0` selects a per-type default |
|
|
| `timestamp_format` | string | formatter default | Go reference layout, e.g. `"2006-01-02T15:04:05Z07:00"` |
|
|
|
|
## Types
|
|
|
|
### raw
|
|
|
|
Passthrough. `FlagRaw` bypasses formatting and sanitization: the message reaches
|
|
the sink exactly as the source produced it, with no timestamp, level or source
|
|
prefix added. An entry that also carries `fields` gets the fields JSON appended
|
|
verbatim after a single space — `raw` never drops data and never re-encodes it.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "raw"
|
|
```
|
|
|
|
Fastest option, and the right one when you are relaying text that is already in
|
|
its final form. Note that it also bypasses sanitization, so control characters
|
|
in the source data reach your sinks intact.
|
|
|
|
Byte-exact transport needs a source that does not split the line: the `console`
|
|
source, or the `file` source with `raw = true`. Both put the whole line —
|
|
newline included — in the message and leave `fields` empty. The `file` source's
|
|
JSON branch splits a line into message and fields, so `raw` reassembles it as
|
|
`<msg> <fields>` rather than reproducing the original object.
|
|
|
|
### txt
|
|
|
|
Human-readable line output with a timestamp and level.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "txt"
|
|
sanitizer_policy = "txt"
|
|
timestamp_format = "2006-01-02 15:04:05"
|
|
```
|
|
|
|
### json
|
|
|
|
Structured output, the natural choice for downstream ingestion.
|
|
|
|
```toml
|
|
[pipelines.flow.format]
|
|
type = "json"
|
|
sanitizer_policy = "json"
|
|
```
|
|
|
|
Output has the shape:
|
|
|
|
```json
|
|
{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}
|
|
```
|
|
|
|
The exact key names and structure come from the `lixenwraith/log` formatter, not
|
|
from LogWisp; they are stable for a given dependency version but are not part of
|
|
LogWisp's own configuration surface.
|
|
|
|
## Flags
|
|
|
|
`flags` is a bitmask passed to the underlying formatter. Leave it at `0` unless
|
|
you need to override the defaults.
|
|
|
|
| Value | Name | Effect |
|
|
|-------|------|--------|
|
|
| `1` | Raw | Bypass formatting and sanitization entirely |
|
|
| `2` | ShowTimestamp | Emit the timestamp |
|
|
| `4` | ShowLevel | Emit the level |
|
|
| `8` | StructuredJSON | Render attached fields as a JSON object |
|
|
| `16` | NoTimestamp | Suppress the timestamp |
|
|
| `32` | NoLevel | Suppress the level |
|
|
|
|
With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
|
|
(timestamp + level) for every other type. `8` is added automatically whenever an
|
|
entry carries parseable `fields` and `1` is not set; `1` always wins.
|
|
|
|
Examples: `flags = 4` for level only, no timestamp; `flags = 2` for timestamp
|
|
only, no level.
|
|
|
|
## Sanitizer Policies
|
|
|
|
The sanitizer runs before serialization and neutralizes control characters that
|
|
would otherwise break framing or reach a terminal.
|
|
|
|
| Policy | Behaviour | Use with |
|
|
|--------|-----------|----------|
|
|
| `raw` | No-op passthrough | `type = "raw"` where you control the data |
|
|
| `txt` | Escapes non-printable characters | File and console sinks |
|
|
| `json` | Escapes control characters for safe JSON embedding | `type = "json"`, chain links |
|
|
| `shell` | Strips shell metacharacters, whitespace, and control characters | Data that will be passed to a command |
|
|
|
|
When `sanitizer_policy` is omitted, the policy is derived from `type`: `json`
|
|
for `json`, `txt` for `txt`/`text`, and `raw` for anything else — so the safe
|
|
pairing is the default.
|
|
|
|
> `shell` strips dangerous characters but is **not** sufficient to make a string
|
|
> safe for shell construction. Pass arguments through `exec` argv instead of
|
|
> building command lines.
|
|
|
|
To see a policy working, point a pipeline at the `random` source with
|
|
`special = true`, which injects control bytes and multi-byte Unicode into every
|
|
message.
|
|
|
|
## Node Identity in Output
|
|
|
|
Entries that arrived over a chain link carry a `Node` label. The formatter
|
|
renders it as a syslog-style prefix on the source field:
|
|
|
|
```
|
|
edge-01/app.log
|
|
```
|
|
|
|
Entries with no node label show the bare source. Node identity therefore appears
|
|
*inside* the source field rather than as a separate output key — worth knowing
|
|
when writing downstream parsers or grep patterns.
|
|
|
|
## Structured Fields
|
|
|
|
When an entry carries `Fields` (raw JSON) and `FlagRaw` is not set, the
|
|
formatter parses it and switches to structured rendering by adding the
|
|
`StructuredJSON` flag automatically. Fields reach a pipeline in two ways: from
|
|
the `file` source when a tailed line parses as JSON with a `fields` key, and
|
|
from the heartbeat generator when `include_stats = true`.
|
|
|
|
## Choosing a Configuration
|
|
|
|
| Goal | Configuration |
|
|
|------|---------------|
|
|
| Maximum throughput, data already formatted | `type = "raw"` |
|
|
| Human reading in a terminal or file | `type = "txt"`, `sanitizer_policy = "txt"` |
|
|
| Downstream ingestion (Loki, Elasticsearch, jq) | `type = "json"`, `sanitizer_policy = "json"` |
|
|
| Compact console output | `type = "txt"`, `flags = 4` |
|
|
| Untrusted log content | never `raw`; pick `txt` or `json` and set the matching policy |
|
|
|
|
## Formatting and Chain Links
|
|
|
|
Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload.
|
|
They re-serialize the structured entry into the canonical chain encoding, which
|
|
makes them independent of the local formatter.
|
|
|
|
The practical consequence: setting `flow.format` on an edge node changes only
|
|
that node's own local sinks. The output shape seen by a human or a downstream
|
|
system is decided on the node that owns the sink they read.
|
|
|
|
If an event ever reaches a chain sink without a structured entry, the sink wraps
|
|
the formatted payload into a synthetic entry and counts it in `synthesized`.
|
|
A non-zero `synthesized` count means something upstream lost structure.
|
|
|
|
## Performance
|
|
|
|
Relative cost, cheapest first: `raw` (passthrough) → `txt` (line assembly) →
|
|
`json` (serialization). Sanitization adds a scan of the message; the `raw`
|
|
policy skips it.
|
|
|
|
The formatter holds a mutex because the underlying implementation reuses an
|
|
internal buffer and is not goroutine-safe. It is the only shared serialization
|
|
point in the hot path, and the reason a single pipeline formats entries one at
|
|
a time.
|