logwisp/doc/formatters.md

# Formatters

The formatter is the last flow stage. It turns a `core.LogEntry` into the byte
payload that sinks write, applying a sanitizer policy on the way.

```toml
[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"
flags            = 0
timestamp_format = ""
```

One formatter serves the whole pipeline. Sinks receive an identical payload,
one record ending in exactly one newline; there is no per-sink formatting. When you need two shapes of the same data, run
two pipelines, or chain to a node that formats differently.

Omitting `[pipelines.flow.format]` entirely selects `raw`.

## Options

Each as type and default:

- `type` (string, `raw`): `raw`, `txt`, or `json`.
- `sanitizer_policy` (string, derived from `type`): `raw`, `txt`, `json`, or
  `shell`.
- `flags` (int64, `0`): bitmask override; `0` selects a per-type default.
- `timestamp_format` (string, `time.RFC3339Nano`): Go reference layout, e.g.
  `"2006-01-02T15:04:05Z07:00"`.

## Types

### raw

Passthrough. `FlagRaw` bypasses formatting and sanitization: the message reaches
the sink as the source produced it, with no timestamp, level or source prefix
added, and a newline appended unless the message ends in one. An entry that also
carries `fields` gets the fields JSON appended verbatim after a single space —
`raw` never drops data and never re-encodes it.

```toml
[pipelines.flow.format]
type = "raw"
```

Fastest option, and the right one when you are relaying text that is already in
its final form. Note that it also bypasses sanitization, so control characters
in the source data reach your sinks intact.

Byte-exact transport needs a source that does not split the line: the `console`
source, or the `file` source with `raw = true`. Both put the whole line, its
terminator removed, in the message and leave `fields` empty; `raw` writes it
back as one `\n`. The `file` source's
JSON branch splits a line into message and fields, so `raw` reassembles it as
`<msg> <fields>` rather than reproducing the original object.

### txt

Human-readable line output with a timestamp and level.

```toml
[pipelines.flow.format]
type             = "txt"
sanitizer_policy = "txt"
timestamp_format = "2006-01-02 15:04:05"
```

### json

Structured output, the natural choice for downstream ingestion.

```toml
[pipelines.flow.format]
type             = "json"
sanitizer_policy = "json"
```

Output has the shape:

```json
{"time":"2026-01-02T15:04:05.123Z","level":"ERROR","trace":"edge-01/app.log","fields":["connection refused"]}
```

The exact key names and structure come from the `lixenwraith/log` formatter, not
from LogWisp; they are stable for a given dependency version but are not part of
LogWisp's own configuration surface.

## Flags

`flags` is a bitmask passed to the underlying formatter. Leave it at `0` unless
you need to override the defaults.

- `1` Raw: bypass formatting and sanitization entirely.
- `2` ShowTimestamp: emit the timestamp.
- `4` ShowLevel: emit the level.
- `8` StructuredJSON: render attached fields as a JSON object.
- `16` NoTimestamp: suppress the timestamp.
- `32` NoLevel: suppress the level.

With `flags = 0` the formatter selects `1` for `type = "raw"` and `6`
(timestamp + level) for every other type. `8` is added automatically whenever an
entry carries parseable `fields` and `1` is not set; `1` always wins.

`txt` and `json` show timestamp and level unless a suppress bit says otherwise
(suppress wins over show), so `2` and `4` alone change nothing. Examples:
`flags = 16` for level only, no timestamp; `flags = 32` for timestamp only, no
level.

## Sanitizer Policies

The sanitizer runs before serialization and neutralizes control characters that
would otherwise break framing or reach a terminal.

Each policy, then what to use it with:

- `raw`: no-op passthrough.
  - `type = "raw"` where you control the data.
- `txt`: escapes non-printable characters.
  - File and console sinks.
- `json`: escapes control characters for safe JSON embedding.
  - `type = "json"`.
- `shell`: strips shell metacharacters, whitespace, and control characters.
  - Data that will be passed to a command.

When `sanitizer_policy` is omitted, it is the policy of the same name as
`type`, so the safe pairing is the default.

> `shell` strips dangerous characters but is **not** sufficient to make a string
> safe for shell construction. Pass arguments through `exec` argv instead of
> building command lines.

To see a policy working, point a pipeline at the `random` source with
`special = true`, which injects control bytes and multi-byte Unicode into every
message.

## Node Identity in Output

Entries that arrived over a chain link carry a `Node` label. The formatter
renders it as a syslog-style prefix on the source field (`trace` in `json`
output):

```
edge-01/app.log
```

Entries with no node label show the bare source. Node identity therefore appears
*inside* the source field rather than as a separate output key — worth knowing
when writing downstream parsers or grep patterns.

## Structured Fields

When an entry carries `Fields` (raw JSON) and `FlagRaw` is not set, the
formatter parses it and switches to structured rendering by adding the
`StructuredJSON` flag automatically. Fields reach a pipeline in two ways: from
the `file` source when a tailed line parses as JSON with a `fields` key, and
from the heartbeat generator when `include_stats = true`.

## Choosing a Configuration

By goal:

- Maximum throughput, data already formatted: `type = "raw"`.
- Human reading in a terminal or file: `type = "txt"`,
  `sanitizer_policy = "txt"`.
- Downstream ingestion (Loki, Elasticsearch, jq): `type = "json"`,
  `sanitizer_policy = "json"`.
- Compact console output: `type = "txt"`, `flags = 16`.
- Untrusted log content: never `raw`; pick `txt` or `json` and set the
  matching policy.

## Formatting and Chain Links

Chain sinks (`tcp_chain`, `http_chain`) do **not** ship the formatted payload.
They re-serialize the structured entry into the canonical chain encoding, which
makes them independent of the local formatter.

The practical consequence: setting `flow.format` on an edge node changes only
that node's own local sinks. The output shape seen by a human or a downstream
system is decided on the node that owns the sink they read.

If an event ever reaches a chain sink without a structured entry, the sink wraps
the formatted payload into a synthetic entry and counts it in `synthesized`.
A non-zero `synthesized` count means something upstream lost structure.

## Performance

Relative cost, cheapest first: `raw` (passthrough) → `txt` (line assembly) →
`json` (serialization). Sanitization adds a scan of the message; the `raw`
policy skips it.

The formatter holds a mutex because the underlying implementation reuses an
internal buffer and is not goroutine-safe. It is the only shared serialization
point in the hot path, and the reason a single pipeline formats entries one at
a time.