logwisp/doc/chaining.md

# Chaining

Chaining links LogWisp nodes together. An edge node forwards its entries to a
relay or collector, which can filter, reformat, fan out, or forward them again.
Unlike the `tcp` and `http` sinks — which emit *formatted text* for humans and
generic clients — chain links carry the **structured entry**, so downstream
nodes can filter and reformat as if the entries were local.

## Topology

```
  edge-01                       relay                        consumers
  ┌───────────────┐             ┌────────────────────┐       ┌──────────────┐
  │ file source   │             │ tcp_chain source   │       │ browser (SSE)│
  │      ↓        │  TCP/TLS    │        ↓           │  ───► │ nc / telnet  │
  │ tcp_chain sink├────────────►│      flow          │       │ archive file │
  └───────────────┘   :15801    │        ↓           │       └──────────────┘
                                │ http sink, tcp sink│
  edge-02                       │ file sink          │
  ┌───────────────┐             │ http_chain sink ───┼──►  upstream collector
  │ file source   │  HTTP/TLS   │                    │
  │      ↓        ├────────────►│ http_chain source  │
  │ http_chain sink│   :15802   └────────────────────┘
  └───────────────┘
```

Both chain sources can feed a single pipeline (fan-in) whose sinks then fan the
merged stream out. `test/chain-test.sh` builds the two-independent-pipelines
variant; `test/chain-aggregate-test.sh` builds the fan-in variant.

## Node Identity

Chained entries carry a `node` label identifying where they originated.

- A chain **sink** stamps `node` on any entry that does not already have one.
  The label comes from the `node` option, defaulting to `os.Hostname()`.
- A chain **source** either honours the sender's label or overrides it,
  according to `trust_node`:
  - `true` (default): keep the label the sender declared; fall back to the
    remote address when absent.
  - `false`: always overwrite with the sender's remote address.

Relays preserve `node`, so a label survives any number of hops and identifies
the original producer rather than the last relay.

With an `auth` block the source can instead bind the label to the sender's
authenticated identity — its certificate identity under `mtls`, its username
under `scram`. `assert` and `force` take the connection label over from
`trust_node`; only `force` also overrides the per-entry labels.
`auth.node_binding`:

- `none`: `trust_node` governs the connection label and the per-entry `node`
  field.
- `assert`: the connection label must equal the identity, or the peer is
  rejected; `trust_node` governs the per-entry `node` field.
- `force` (default with an `auth` block): the identity is the connection label
  and overwrites every per-entry `node` field.

Pick `force` at an ingest boundary you do not trust — it is the only setting
where a compromised edge cannot mislabel its entries, including through the
per-entry `node` field. Pick `assert` on a relay-to-relay hop, where the relay
should prove its own identity but the origin labels it forwards must survive.
See [Security](security.md#node-binding).

Formatters render node identity as a syslog-style prefix on the source field:
`edge-01/app.log`. In JSON output the node therefore appears inside the source
field, not as a separate top-level key.

> `trust_node = true` with no `auth` block means any peer that can connect can
> claim **any** node label, including one belonging to another host. On an
> untrusted network set an `auth` block (`mtls` or `scram`) and keep
> `node_binding = "force"`; `trust_node = false` is the fallback when neither is
> an option.

## Wire Protocol

Protocol version: **1**. Both transports carry the same canonical entry
encoding, and differ only in how the preamble and framing are expressed.

### TCP transport

A persistent connection carrying newline-delimited JSON.

1. The dialer connects and, under TLS, completes the handshake.
2. The dialer immediately writes the hello preamble as one JSON line:

   ```json
   {"logwisp":1,"node":"edge-01"}
   ```

3. The listener reads that line within `hello_timeout_ms` and rejects the
   connection if it is malformed or declares a different protocol version.
4. Every subsequent line is one JSON-encoded `LogEntry`.

Line size is bounded at 1 MiB. An oversized line is a protocol violation and
terminates the connection, because the scanner cannot resynchronize afterwards.

Under `auth.type = "scram"` the hello also carries the SCRAM client-first
message, and three more lines precede the entries, each one JSON object of at
most 4 KiB:

```
dialer   {"logwisp":1,"node":"edge-01","scram":{"username":"edge-01","client_nonce":"…"}}
listener {"challenge":{"full_nonce":"…","salt":"…","argon_time":3,"argon_memory":65536,"argon_threads":4}}
dialer   {"proof":{"full_nonce":"…","client_proof":"…"},"binding":"<base64 SHA-256 of the server certificate>"}
listener {"final":{"server_signature":"…","username":"edge-01"}}
```

The whole exchange runs within `hello_timeout_ms`; the listener sends the final
line only after node binding agrees, and a refusal at any step is one
`{"error":"…"}` line before the connection closes. The field is additive, so
the protocol version stays `1`: a listener without `scram` answers a hello
carrying it with `{"error":"authentication not enabled"}`, while an older
binary ignores it and the dialer gives up after 10 s.

### HTTP transport

Batches of NDJSON delivered by `POST`, with the preamble expressed as headers.

- Request headers
  - `X-Logwisp-Protocol`: the protocol version; must be `1`.
  - `X-Logwisp-Node`: the origin node label.
  - `Content-Type`: `application/x-ndjson`.
  - `Authorization`: `Bearer <token>`, under `scram`.
- Response header
  - `X-Logwisp-Accepted`: the number of entries ingested.

Responses: `204` on success, `400` for a bad protocol version or a malformed
body, `413` when the body cap is exceeded, `405` for a non-`POST` method, `401`
for a missing or expired token, `403` for a refused certificate or node label.

Under `scram` the sink first logs in with two `POST /auth` requests carrying the
same JSON as the TCP lines: the hello (`200` and the challenge), then the proof
(`200` with the final message, `token` and `expires_in` in seconds). A refusal is
`{"error":"…"}`: `401` for a failed proof, `429` when throttled, `503` when the
listener is busy, `400` or `413` for a malformed or oversized body.

### Entry encoding

```json
{
  "time": "2026-01-02T15:04:05.123456789Z",
  "node": "edge-01",
  "source": "app.log",
  "level": "ERROR",
  "message": "connection refused",
  "fields": {"attempt": 3}
}
```

`node`, `level`, and `fields` are omitted when empty. A missing `time` is filled
in at ingest.

## Delivery Semantics

- `tcp_chain`: per line, each held across reconnects.
  - On failure it retries with exponential backoff plus ±20 % jitter until
    the line is written or the sink shuts down.
  - Back-pressure appears upstream as `total_dropped_by_sink`.
- `http_chain`: at least once per batch.
  - It retries transport errors, `408`, `429`, `5xx` and, under `scram`, failed
    logins and `401`.
  - It drops the batch on any other non-2xx, counted in `dropped_batches`.

`http_chain` batches can be delivered twice when a successful request's response
is lost. There is no de-duplication downstream; design your consumers to
tolerate it, or use `tcp_chain` where each line is written once per successful
write.

Neither transport persists anything to disk. Entries buffered in memory during
an outage are lost if the process exits.

## Worked Example

**Edge node** — tail files, forward over mTLS:

```toml
[[pipelines]]
name = "edge"

[[pipelines.plugin_sources]]
id = "app"
type = "file"
[pipelines.plugin_sources.config]
directory = "/var/log/myapp"
pattern = "*.log"

[[pipelines.plugin_sinks]]
id = "forward"
type = "tcp_chain"
[pipelines.plugin_sinks.config]
host = "relay.internal"
port = 15801
node = "edge-01"
[pipelines.plugin_sinks.config.tls]
enabled   = true
ca_file   = "/etc/logwisp/tls/ca.crt"
cert_file = "/etc/logwisp/tls/edge-01.crt"
key_file  = "/etc/logwisp/tls/edge-01.key"
[pipelines.plugin_sinks.config.auth]
type  = "mtls"
allow = ["relay.internal"]      # pin the relay, not just its hostname
```

**Relay** — ingest, keep errors only, archive and stream:

```toml
[[pipelines]]
name = "relay"

[[pipelines.flow.filters]]
type = "include"
patterns = ["ERROR", "FATAL"]

[pipelines.flow.format]
type = "json"
sanitizer_policy = "json"

[[pipelines.plugin_sources]]
id = "ingest"
type = "tcp_chain"
[pipelines.plugin_sources.config]
host = "0.0.0.0"
port = 15801
[pipelines.plugin_sources.config.tls]
enabled        = true
cert_file      = "/etc/logwisp/tls/relay.crt"
key_file       = "/etc/logwisp/tls/relay.key"
client_auth    = true
client_ca_file = "/etc/logwisp/tls/ca.crt"
[pipelines.plugin_sources.config.auth]
type         = "mtls"
allow        = ["edge-01", "edge-02"]
node_binding = "force"          # entries are labelled from the certificate

[[pipelines.plugin_sinks]]
id = "archive"
type = "file"
[pipelines.plugin_sinks.config]
directory = "/var/log/logwisp"
name = "errors"

[[pipelines.plugin_sinks]]
id = "live"
type = "http"
[pipelines.plugin_sinks.config]
host = "127.0.0.1"
port = 8080
```

Entries arriving on this relay are labelled `edge-01` or `edge-02` because that
is what their certificates say, regardless of the `node` each edge configured.
`test/mtls-chain-test.sh` builds exactly this shape against a throwaway PKI.

To use passwords instead of client certificates, replace both `auth` blocks:
`type = "scram"` with `username` and `password_file` on the edge, and with a
`credentials_file` on the relay. `client_auth` and the edge's certificate become
optional, and entries are labelled with the username. See
[Password Authentication](security.md#password-authentication-scram).

## Operational Notes

- **Formatting is a relay decision.** Because chain links carry structured
  entries, the edge node's `flow.format` affects only its own local sinks. Set
  the output shape on the node that owns the human-facing sink.
- **Filtering early saves bandwidth.** A filter on the edge drops entries before
  they cross the network; a filter on the relay is easier to change centrally.
- **Rate limits are per pipeline.** An edge limit protects the link; a relay
  limit protects the relay from a noisy edge.
- **Heartbeats traverse chain links** as ordinary structured entries and keep
  otherwise-idle links and their sessions warm.
- **Ports** used by the bundled test scripts: `15801` tcp_chain ingest, `15802`
  http_chain ingest, `15803` tcp sink, `15804` http sink.
- **Dial the listener's own literal** when testing locally: a listener on
  `127.0.0.1` takes IPv4 only, and `localhost` may resolve to `::1`.