logwisp/doc/networking.md

Networking

Everything LogWisp does over a socket, and the knobs that shape it. For certificates and trust see Security; for multi-node topologies see Chaining.

Address Family

Every listener and dialer keeps strictly to the family of its host:

• IPv4 literal (127.0.0.1, the default 0.0.0.0): IPv4 only (tcp4). A listener's empty host is the IPv4 wildcard, as 0.0.0.0.
• IPv6 literal (::1, ::, fe80::1%eth0): IPv6 only (tcp6).
  • :: takes no IPv4 connections, on Linux and FreeBSD alike, whatever the system's bindv6only default.
  • To serve both families, run two listeners on the same port, one on 0.0.0.0 and one on ::: neither holds the other's port.
• Hostname: resolved. A listener binds one address (IPv4 when there is one) in that address's family alone, a name for 0.0.0.0 or :: too, so name the address to be sure; a dialer tries each address in turn.

IPv6

• Notation
  • host takes the bare address: host = "::1". A bracketed host, one with a port, or an IPv4-mapped one (::ffff:10.0.0.1) fails at load.
  • Addresses and URLs bracket it: logs print [::1]:8443, and lw auth takes --addr [::1]:8443 and --url https://[::1]:8443. Older curl releases read the brackets as a glob; -g stops that and is harmless.
  • A link-local address carries its zone, fe80::1%eth0, escaped in URLs as https://[fe80::1%25eth0]:8443.
• TLS: a dialer verifies an IPv6 target against the certificate's IP SANs (subjectAltName = IP:::1), as it does an IPv4 one. The zone is not part of the name; server_name overrides as usual.
• Peers
  • Logs and sessions name an IPv6 peer by its address, a link-local one with its zone.
  • SCRAM throttling counts an IPv6 client by its /64, so the hosts on a link share one link-local budget per interface (zone): each can pick any fe80::/64 address.
  • A peer with a zone never matches auth.trusted_proxies or acl.proxy_from: put the proxy on loopback or a routed address. acl.allow and deny match it without its zone.

Network Plugins

Listeners:

• tcp sink: raw stream; broadcasts formatted payloads to clients.
• http sink: HTTP SSE; a browser viewer at /, the live stream and status JSON.
• tcp_chain source: chain v1; ingests a persistent NDJSON stream.
• http_chain source: chain v1; ingests NDJSON batches over POST.

Dialers:

• tcp_chain sink: chain v1; forwards entries over a persistent connection.
• http_chain sink: chain v1; forwards entries as batched POSTs.

There is no port registry and no default port: port is required on every network plugin. Two sinks on the same port pass lw --check and fail at bind when the service starts; lw exits 1:

ERROR msg "Failed to initialize service" error "failed to start service pipelines: failed to start pipeline <name>: failed to start sink <id>: tcp sink bind <host>:<port>: listen tcp4 <host>:<port>: bind: address already in use"

Timeouts

Every network plugin exposes the deadlines relevant to its role. 0 means no deadline where the default is 0 (none); elsewhere it selects the default.

• tcp sink
  • write_timeout_ms, default 5000: one write to one client; a miss disconnects that client.
• http sink
  • write_timeout_ms, default 0 (none): one SSE event write.
• tcp_chain source
  • hello_timeout_ms, default 10000: reading the protocol preamble, and under scram the whole login.
  • read_timeout_ms, default 0 (none): idle time between entries.
• http_chain source
  • read_timeout_ms, default 30000: reading a whole request, headers and body; also how long an idle keep-alive connection waits for its next request.
• tcp_chain sink
  • dial_timeout_ms, default 5000: TCP connect.
  • write_timeout_ms, default 5000: one line write.
• http_chain sink
  • request_timeout_ms, default 10000: dial plus write plus response, and a SCRAM login when one is due.

Fixed, non-configurable bounds:

• TLS handshake: 10 s, on all TLS listeners and dialers; on the http_chain source, read_timeout_ms when shorter.
• HTTP read-header timeout: 10 s, on the http sink and http_chain source.
• HTTP server shutdown grace: 2 s on the http_chain source and the http sink; on the http sink, write_timeout_ms instead when set below 2 s.
• Single entry line: at most 1 MiB, on chain listeners.
• SCRAM login: 10 s, on the tcp sink, chain sinks, lw auth, and each /auth request.
• Hello and SCRAM lines, /auth bodies: at most 4 KiB, on tcp_chain sources and on scram listeners and dialers.

The http sink deliberately leaves the server's WriteTimeout unset, since it would terminate long-lived SSE streams; per-event deadlines come from write_timeout_ms instead.

Connection Limits

max_connections caps concurrent connections on the tcp sink and the tcp_chain source, and concurrent streams on the http sink (other requests are not counted); 0 means unlimited.

• A connection or stream counts only after the check that admits it, so a burst can over-admit by several, more the larger the burst.
• On the tcp sink and tcp_chain source a connection counts before its TLS handshake, so the cap bounds handshakes in progress as well as sessions.
• Over the limit a TCP connection is closed and counted in rejected_conns; an http stream request is answered 503 and counted in rejected_clients, its connection kept.

The http_chain source has no connection cap; it bounds work with max_body_bytes and read_timeout_ms instead.

Per client, the acl block caps connections on every listener and requests on the HTTP ones (Security), and SCRAM logins are throttled (Security). flow.rate_limit is a pipeline-wide entry rate limit: it cannot distinguish or throttle a peer.

Keep-Alive

The tcp sink (accepted connections) and the tcp_chain sink (its outbound connection) configure TCP keep-alive:

keep_alive           = true    # false sends no probes
keep_alive_period_ms = 30000

Probes start after keep_alive_period_ms idle. Other listeners and the http_chain sink use Go's default TCP keep-alive (15 s); it is not configurable there. Keep-alive detects a dead peer but does not keep an application-level stream flowing: that is a heartbeat's job.

Heartbeats

A flow heartbeat keeps idle TCP clients and chain links alive through NAT and proxy timeouts; idle SSE streams already get a comment every 15 s. See Configuration.

Reconnection

Chain sinks reconnect on their own. Both use exponential backoff between backoff_min_ms and backoff_max_ms with ±20 % jitter, and both are interruptible by shutdown.

backoff_min_ms = 500
backoff_max_ms = 30000

The connection is established lazily, when the sink first has something to send, so an edge node starts cleanly even when its relay is down, and connects at its next attempt once the relay is up. The tcp_chain sink reports connected and reconnects; the http_chain sink reports request_errors and dropped_batches.

Server-side sinks (tcp, http) do not reconnect; clients are expected to retry. Browsers reconnect SSE streams automatically.

Protocol Details

HTTP sink (SSE): HTTP/1.1 in plaintext, HTTP/2 via ALPN under TLS; GET on the stream path, GET or HEAD on the status path. Framing and headers: Sinks. Under scram, POST /auth logs in and both paths take Authorization: Bearer <token>.

TCP sink: the formatted payloads, each ending in a newline, with no other framing; under json that is one entry per line, while raw passes a message's own line breaks through. Under scram the client's hello and login lines come first, as on a chain link.

Chain transports: see Chaining for the hello preamble, headers, and entry encoding.

Troubleshooting

unknown key "..." when the configuration loads

• The key is misspelled or belongs to another plugin type. A plugin's keys are checked at load (lw --check too), nested tables included, so a typo in tls or auth cannot leave a protection silently off.

Connection refused

• Confirm the pipeline started; a bind failure is logged at ERROR.
• Confirm the dialer uses the listener's family: [::1] does not reach a 127.0.0.1 or 0.0.0.0 listener, nor 127.0.0.1 a :: one. localhost may resolve to either.
• Check the port is not already bound by another pipeline in the same process.

Closed as soon as it connects, before TLS

• The listener's acl refused it: acl_denied or acl_limited rises, and its log has Connection refused by acl at WARN with a reason, once a minute at most. The dialer sees EOF or connection reset by peer (curl: exit 52 or 56; over TLS, 35).
  • address rules — allow or deny excludes the client: the one a PROXY header names (remote_addr, the proxy as peer_addr), else the socket peer. Behind the http sink's auth.trusted_proxies that is the proxy: allow it.
  • max_connections_per_client — the client holds that many already, or the listener tracks 65,536 clients; HTTP keep-alive connections count too.
  • no PROXY header — a peer in proxy_from sent none under proxy_protocol = "required": the proxy lacks proxy_protocol on (nginx) or send-proxy (HAProxy), or the proxy's host dialed directly.
  • PROXY header: … — malformed, too long, or incomplete after 10 s.
  • PROXY header from a peer outside proxy_from — at the first read, after the connection opened: list the proxy in proxy_from.
• acl: allow entry "...": IPv4, and the listener's host is IPv6 at startup, or the reverse: each family's listener takes its own. Behind proxy_from or trusted_proxies, allow and deny take both; proxy_from its own.

HTTP 429 Too Many Requests with Retry-After

• requests_per_second_per_client: acl_limited rises and the log has Request refused by acl, once a minute at most. A browser's page load takes several requests; raise the rate rather than retry faster. From /auth with too many attempts, it is SCRAM throttling (below).

TLS handshake failure

• client didn't provide a certificate — the listener has client_auth = true and the dialer has no cert_file/key_file.
• certificate signed by unknown authority — the dialer's ca_file does not contain the issuer of the server certificate, or the listener's client_ca_file does not contain the issuer of the client certificate.
• certificate is not valid for any names / SAN mismatch — the dialed host is not covered by the server certificate's SANs; set server_name.
• server key sha256//... matches no tls.pin_sha256 — the listener is not the one pinned, or a self-signed listener restarted and made a new key: take the pin from its startup warning, or move it to an issuer and ca_file.
• protocol version not supported — one side is pinned to min_version = "1.3" and the other cannot negotiate it.
• Listeners log handshake failures at WARN with the remote address (the tcp sink at DEBUG); the tcp sink and tcp_chain source count them in tls_handshake_errors.

Rejected after a successful handshake

• auth: refused: identity "<id>" is not allowed — the certificate is valid and chains to the CA, but the identity is not in auth.allow / auth.allow_patterns. On TCP the connection is closed; on HTTP the answer is 403.
• auth: refused: peer certificate carries no <mode> identity — auth.identity names a field the certificate does not populate, e.g. san_dns on a CN-only leaf.
• auth: node_binding "assert": declared node "<node>" does not match identity "<id>" — the sender's node option and its certificate disagree. Fix one, or use node_binding = "force" to let the certificate win silently.
• On a tcp_chain sink the same message inside Chain connect refused means the server was refused: its certificate identity is not in the sink's auth.allow.
• Allow-list and login rejections log WARN and count auth_rejected; a node-binding refusal logs Connection rejected by node binding (tcp_chain, rejected_conns) or Request rejected by node binding (http_chain, rejected_requests).

SCRAM login fails

• authentication failed — the dialer is not told why, by design. The listener's WARN line (Connection rejected by auth policy on TCP, Login rejected on HTTP) is: invalid credentials for a wrong password, an unknown user, or one removed or rotated on the listener (compare the password_file with the last add-user, and check the listener got SIGHUP); certificate cn "…" does not match user "…" for the identity binding.
• the client saw another server certificate: TLS interception or a terminating proxy in the listener's log, with auth_binding_mismatch rising — something between the peers terminates TLS. Pass TLS through to LogWisp; the login cannot work otherwise, except on an http sink in proxy mode (auth.trusted_proxies), where the client bound its proof to the proxy's certificate means a client that needs lw auth token --unbound, and the client sent an unbound proof the reverse. On an http_chain sink, server certificate differs from the one the SCRAM login was bound to means the certificate changed after the login: a rotation (the retry binds anew), several backends, or interception.
• 403 from an http sink in proxy mode, with not a trusted proxy or X-Forwarded-Proto in its WARN line — the request did not come from a listed proxy, or the proxy did not forward X-Forwarded-Proto: https and X-Forwarded-For. With Request refused by acl (once a minute at most) or none, the acl rules refuse the forwarded client (acl_denied).
• too many attempts — the address failed or abandoned logins faster than one per second beyond a burst of 10, or has 4 unfinished; it clears within seconds once the failing peer stops. Peers behind one NAT, or one passthrough proxy that sends no PROXY header, share the budget. busy — 4,096 logins in flight, or the listener is stopping.
• authentication not enabled — the dialer has scram; the listener's auth.type is none or mtls.
• no challenge within 10s (older logwisp, or not a scram listener) — the server read the hello and said nothing. Over HTTP the same case reads no auth endpoint at …/auth.
• Argon2 parameters below client minimum — the credentials file holds a cheaper Argon2 profile than the defaults; recreate it with lw auth.
• peer offered no credentials (authentication required on the wire) in the listener's log — a peer without scram reached a scram listener; a client that sends nothing logs read hello: … i/o timeout after 10 s instead. A tcp_chain sink without scram logs WARN Chain link refused with that reason and retries under growing backoff; what it wrote before reading the refusal is lost. An http_chain sink without scram gets 401 and drops every batch.

401 on HTTP under scram

• An http_chain sink renews its token before it expires, so a 401 means the listener reloaded (every reload revokes all tokens); the retry logs in again.
• Continuous 401 means the token never sticks: several LogWisp processes behind one address, a proxy that strips Authorization, or a token_lifetime_ms shorter than a request takes. A token used with curl dies with every listener reload.
• 403 is not the token: it is the identity binding, node binding or, in proxy mode, the proxy gate.

Behind an L4 proxy

• Every connection through the proxy fails, the listener logging the proxy's address and first record does not look like a TLS handshake (the tcp sink at DEBUG): the proxy sends a PROXY header (nginx stream with proxy_protocol on, HAProxy send-proxy) to a listener without acl.proxy_protocol. Set it, with the proxy in proxy_from: see PROXY protocol.
• All peers share one throttling budget and logs name the proxy: it sends no PROXY header, so every scram peer arrives from its address. One client failing logins exhausts everyone's, and a fifth login while 4 are unfinished (edges reconnecting together after a restart) is refused; both answer too many attempts (Throttling). Have the proxy send the header.

Entries not arriving over a chain link

• tcp_chain: check the sink's connected and reconnects; http_chain: its request_errors and dropped_batches.
• Check the source's auth_rejected — an allow-list miss or a failed login looks exactly like a network fault from the sender's side — and the sink's last_auth_error.
• A version skew: tcp_chain logs WARN Rejected chain connection (unsupported protocol version), counted only under scram, in auth_rejected; http_chain answers 400 and counts rejected_requests.
• On http_chain, remember entries wait up to flush_interval_ms before a batch is sent.

Entries arriving under an unexpected node label

• auth.node_binding defaults to force under mtls or scram, which relabels every entry with the sender's certificate identity or username. If a dashboard suddenly shows a different label, that is why. Use node_binding = "assert" to keep upstream origin labels on relay-to-relay hops, or "none" to leave trust_node in charge.

Clients connect but see nothing

• The pipeline may be filtering everything out; check flow.filters stats.
• The rate limiter may be dropping everything; check rate_limiter stats.
• Nothing may be arriving from the sources; check source total_entries.

Entries missing under load

• Compare dropped_writes (per-client queue full — raise client_buffer_size) against total_dropped_by_sink (sink input queue full — raise buffer_size or reduce sink latency).