logwisp/doc/networking.md

Networking

Everything LogWisp does over a socket, and the knobs that shape it. For certificates and trust see Security; for multi-node topologies see Chaining.

Address Family

Every listener and dialer keeps strictly to the family of its host:

• IPv4 literal (127.0.0.1, the default 0.0.0.0): IPv4 only (tcp4). An empty host is the IPv4 wildcard, as 0.0.0.0.
• IPv6 literal (::1, ::, fe80::1%eth0): IPv6 only (tcp6).
  • :: takes no IPv4 connections, on Linux and FreeBSD alike, whatever the system's bindv6only default.
  • To serve both families, run two listeners on the same port, one on 0.0.0.0 and one on ::: neither holds the other's port.
• Hostname: resolved. A listener binds one address (IPv4 when there is one) in that address's family alone, a name for 0.0.0.0 or :: too, so name the address to be sure; a dialer tries each address in turn.

IPv6

• Notation
  • host takes the bare address: host = "::1". A bracketed host, one with a port, or an IPv4-mapped one (::ffff:10.0.0.1) fails at load.
  • Addresses and URLs bracket it: logs print [::1]:8443, and lw auth takes --addr [::1]:8443 and --url https://[::1]:8443. Older curl releases read the brackets as a glob; -g stops that and is harmless.
  • A link-local address carries its zone, fe80::1%eth0, escaped in URLs as https://[fe80::1%25eth0]:8443.
• TLS: a dialer verifies an IPv6 target against the certificate's IP SANs (subjectAltName = IP:::1), as it does an IPv4 one. The zone is not part of the name; server_name overrides as usual.
• Peers
  • Logs and sessions name an IPv6 peer by its address, a link-local one with its zone.
  • SCRAM throttling counts an IPv6 client by its /64, so the hosts on a link share one link-local budget per interface (zone): each can pick any fe80::/64 address.
  • A peer with a zone never matches auth.trusted_proxies: put the proxy on loopback or a routed address.

Network Plugins

Listeners:

• tcp sink: raw stream; broadcasts formatted payloads to clients.
• http sink: HTTP SSE; a browser viewer at /, the live stream and status JSON.
• tcp_chain source: chain v1; ingests a persistent NDJSON stream.
• http_chain source: chain v1; ingests NDJSON batches over POST.

Dialers:

• tcp_chain sink: chain v1; forwards entries over a persistent connection.
• http_chain sink: chain v1; forwards entries as batched POSTs.

There is no port registry and no default port: port is required on every network plugin. Two sinks on the same port pass lw --check and fail at bind when the service starts; lw exits 1:

ERROR msg "Failed to initialize service" error "failed to start service pipelines: failed to start pipeline <name>: failed to start sink <id>: tcp sink bind <host>:<port>: listen tcp4 <host>:<port>: bind: address already in use"

Timeouts

Every network plugin exposes the deadlines relevant to its role. 0 means no deadline where the default is 0 (none); elsewhere it selects the default.

• tcp sink
  • write_timeout_ms, default 5000: one write to one client; a miss disconnects that client.
• http sink
  • write_timeout_ms, default 0 (none): one SSE event write.
• tcp_chain source
  • hello_timeout_ms, default 10000: reading the protocol preamble, and under scram the whole login.
  • read_timeout_ms, default 0 (none): idle time between entries.
• http_chain source
  • read_timeout_ms, default 30000: reading a whole request body.
• tcp_chain sink
  • dial_timeout_ms, default 5000: TCP connect.
  • write_timeout_ms, default 5000: one line write.
• http_chain sink
  • request_timeout_ms, default 10000: dial plus write plus response, and a SCRAM login when one is due.

Fixed, non-configurable bounds:

• TLS handshake: 10 s, on all TLS listeners and dialers.
• HTTP read-header timeout: 10 s, on the http sink and http_chain source.
• HTTP server shutdown grace: 2 s on the http_chain source and the http sink; on the http sink, write_timeout_ms instead when set below 2 s.
• Single entry line: at most 1 MiB, on chain listeners.
• SCRAM login: 10 s, on the tcp sink, chain sinks, lw auth, and each /auth request.
• SCRAM line or /auth body: at most 4 KiB, on scram listeners and dialers.

The http sink deliberately leaves the server's WriteTimeout unset, since it would terminate long-lived SSE streams; per-event deadlines come from write_timeout_ms instead.

Connection Limits

max_connections caps concurrent connections on the tcp sink and the tcp_chain source, and concurrent streams on the http sink (other requests are not counted); 0 means unlimited.

• Admission is a load-then-check, so a burst can over-admit by roughly one.
• On the tcp sink and tcp_chain source the count is taken at accept, so it bounds concurrent TLS handshakes as well as established sessions.
• Over the limit a TCP connection is closed and counted in rejected_conns; an http stream request is answered 503 and counted in rejected_clients, its connection kept.

The http_chain source has no connection cap; it bounds work with max_body_bytes and read_timeout_ms instead.

Per client, the acl block caps connections on every listener and requests on the HTTP ones (Security), and SCRAM logins are throttled (Security). flow.rate_limit is a pipeline-wide entry rate limit: it cannot distinguish or throttle a peer.

Keep-Alive

The tcp sink (accepted connections) and the tcp_chain sink (its outbound connection) configure TCP keep-alive:

keep_alive           = true    # false sends no probes
keep_alive_period_ms = 30000

Probes start after keep_alive_period_ms idle. Other listeners and the http_chain sink use Go's default TCP keep-alive (15 s); it is not configurable there. Keep-alive detects a dead peer but does not keep an application-level stream flowing: that is a heartbeat's job.

Heartbeats

A flow heartbeat keeps idle TCP clients and chain links alive through NAT and proxy timeouts; idle SSE streams already get a comment every 15 s. See Configuration.

Reconnection

Chain sinks reconnect on their own. Both use exponential backoff between backoff_min_ms and backoff_max_ms with ±20 % jitter, and both are interruptible by shutdown.

backoff_min_ms = 500
backoff_max_ms = 30000

The connection is established lazily, so an edge node starts cleanly even when its relay is down and connects as soon as the relay appears. The tcp_chain sink reports connected and reconnects; the http_chain sink reports request_errors and dropped_batches.

Server-side sinks (tcp, http) do not reconnect; clients are expected to retry. Browsers reconnect SSE streams automatically.

Protocol Details

HTTP sink (SSE): HTTP/1.1 in plaintext, HTTP/2 via ALPN under TLS; GET on the stream path, GET or HEAD on the status path. Framing and headers: Sinks. Under scram, POST /auth logs in and both paths take Authorization: Bearer <token>.

TCP sink: the formatted payloads, each ending in a newline, with no other framing; under json that is one entry per line, while raw passes a message's own line breaks through. Under scram the client's hello and login lines come first, as on a chain link.

Chain transports: see Chaining for the hello preamble, headers, and entry encoding.

Troubleshooting

unknown key "..." when the configuration loads

• The key is misspelled or belongs to another plugin type. A plugin's keys are checked at load (lw --check too), nested tables included, so a typo in tls or auth cannot leave a protection silently off.

Connection refused

• Confirm the pipeline started; a bind failure is logged at ERROR.
• Confirm the dialer uses the listener's family: [::1] does not reach a 127.0.0.1 or 0.0.0.0 listener, nor 127.0.0.1 a :: one. localhost may resolve to either.
• Check the port is not already bound by another pipeline in the same process.

Closed as soon as it connects, before TLS

• The listener's acl refused it: acl_denied or acl_limited rises, and its log has Connection refused by acl at WARN with a reason, once a minute at most. The dialer sees EOF or connection reset by peer (curl: exit 52 or 56).
  • address rules — allow or deny excludes the client: the one a PROXY header names (remote_addr, the proxy as peer_addr), else the socket peer. Behind the http sink's auth.trusted_proxies that is the proxy: allow it.
  • max_connections_per_client — the client holds that many already, or the listener tracks 65,536 clients; HTTP keep-alive connections count too.
  • no PROXY header — a peer in proxy_from sent none under proxy_protocol = "required": the proxy lacks proxy_protocol on (nginx) or send-proxy (HAProxy), or the proxy's host dialed directly.
  • PROXY header: … — malformed, too long, or incomplete after 10 s.
  • PROXY header from a peer outside proxy_from — at the first read, after the connection opened: list the proxy in proxy_from.
• acl: allow entry "...": IPv4, and the listener's host is IPv6 at startup, or the reverse: each family's listener takes its own. Behind proxy_from or trusted_proxies, allow and deny take both; proxy_from its own.

HTTP 429 Too Many Requests with Retry-After

• requests_per_second_per_client: acl_limited rises and the log has Request refused by acl, once a minute at most. A browser's page load takes several requests; raise the rate rather than retry faster. From /auth with too many attempts, it is SCRAM throttling (below).

TLS handshake failure

• client didn't provide a certificate — the listener has client_auth = true and the dialer has no cert_file/key_file.
• certificate signed by unknown authority — the dialer's ca_file does not contain the issuer of the server certificate, or the listener's client_ca_file does not contain the issuer of the client certificate.
• certificate is not valid for any names / SAN mismatch — the dialed host is not covered by the server certificate's SANs; set server_name.
• server key sha256//... matches no tls.pin_sha256 — the listener is not the one pinned, or a self-signed listener restarted and made a new key: take the pin from its startup warning, or move it to an issuer and ca_file.
• protocol version not supported — one side is pinned to min_version = "1.3" and the other cannot negotiate it.
• Listeners log handshake failures at WARN with the remote address (the tcp sink at DEBUG); the tcp sink and tcp_chain source count them in tls_handshake_errors.

Rejected after a successful handshake

• auth: refused: identity "<id>" is not allowed — the certificate is valid and chains to the CA, but the identity is not in auth.allow / auth.allow_patterns. On TCP the connection is closed; on HTTP the answer is 403.
• auth: refused: peer certificate carries no <mode> identity — auth.identity names a field the certificate does not populate, e.g. san_dns on a CN-only leaf.
• auth: node_binding "assert": declared node "<node>" does not match identity "<id>" — the sender's node option and its certificate disagree. Fix one, or use node_binding = "force" to let the certificate win silently.
• On a tcp_chain sink the same message inside Chain connect refused means the server was refused: its certificate identity is not in the sink's auth.allow.
• Allow-list and login rejections log WARN and count auth_rejected; a node-binding refusal logs Connection rejected by node binding (tcp_chain, rejected_conns) or Request rejected by node binding (http_chain, rejected_requests).

SCRAM login fails

• authentication failed — the dialer is not told why, by design. The listener's WARN line (Connection rejected by auth policy on TCP, Login rejected on HTTP) is: invalid credentials for a wrong password, an unknown user, or one removed or rotated on the listener (compare the password_file with the last add-user, and check the listener got SIGHUP); certificate cn "…" does not match user "…" for the identity binding.
• the client saw another server certificate: TLS interception or a terminating proxy in the listener's log, with auth_binding_mismatch rising — something between the peers terminates TLS. Pass TLS through to LogWisp; the login cannot work otherwise, except on an http sink in proxy mode (auth.trusted_proxies), where the client bound its proof to the proxy's certificate means a client that needs lw auth token --unbound, and the client sent an unbound proof the reverse. On an http_chain sink, server certificate differs from the one the SCRAM login was bound to means the certificate changed after the login: a rotation (the retry binds anew), several backends, or interception.
• 403 from an http sink in proxy mode, with not a trusted proxy or X-Forwarded-Proto in its WARN line — the request did not come from a listed proxy, or the proxy did not forward X-Forwarded-Proto: https and X-Forwarded-For. With Request refused by acl (once a minute at most) or none, the acl rules refuse the forwarded client (acl_denied).
• too many attempts — the address failed or abandoned logins faster than one per second beyond a burst of 10, or has 4 unfinished; it clears within seconds once the failing peer stops. Peers behind one NAT, or one passthrough proxy that sends no PROXY header, share the budget. busy — 4,096 logins in flight, or the listener is stopping.
• authentication not enabled — the dialer has scram; the listener's auth.type is none or mtls.
• no challenge within 10s (older logwisp, or not a scram listener) — the server read the hello and said nothing. Over HTTP the same case reads no auth endpoint at …/auth.
• Argon2 parameters below client minimum — the credentials file holds a cheaper Argon2 profile than the defaults; recreate it with lw auth.
• peer offered no credentials (authentication required on the wire) in the listener's log — a peer without scram reached a scram listener; a client that sends nothing logs read hello: … i/o timeout after 10 s instead. A tcp_chain sink without scram logs WARN Chain link refused with that reason and retries under growing backoff; what it wrote before reading the refusal is lost. An http_chain sink without scram gets 401 and drops every batch.

401 on HTTP under scram

• An http_chain sink renews its token before it expires, so a 401 means the listener reloaded (every reload revokes all tokens); the retry logs in again.
• Continuous 401 means the token never sticks: several LogWisp processes behind one address, a proxy that strips Authorization, or a token_lifetime_ms shorter than a request takes. A token used with curl dies with every listener reload.
• 403 is not the token: it is the identity binding, node binding or, in proxy mode, the proxy gate.

Behind an L4 proxy

• Every connection through the proxy fails, the listener logging the proxy's address and first record does not look like a TLS handshake (the tcp sink at DEBUG): the proxy sends a PROXY header (nginx stream with proxy_protocol on, HAProxy send-proxy) to a listener without acl.proxy_protocol. Set it, with the proxy in proxy_from: see PROXY protocol.
• All peers share one throttling budget and logs name the proxy: it sends no PROXY header, so every scram peer arrives from its address. One client failing logins exhausts everyone's, and a fifth login while 4 are unfinished (edges reconnecting together after a restart) is refused; both answer too many attempts (Throttling). Have the proxy send the header.

Entries not arriving over a chain link

• tcp_chain: check the sink's connected and reconnects; http_chain: its request_errors and dropped_batches.
• Check the source's auth_rejected — an allow-list miss or a failed login looks exactly like a network fault from the sender's side — and the sink's last_auth_error.
• A version skew: tcp_chain logs WARN Rejected chain connection (unsupported protocol version), counted only under scram, in auth_rejected; http_chain answers 400 and counts rejected_requests.
• On http_chain, remember entries wait up to flush_interval_ms before a batch is sent.

Entries arriving under an unexpected node label

• auth.node_binding defaults to force with any auth block, which relabels every entry with the sender's certificate identity or username. If a dashboard suddenly shows a different label, that is why. Use node_binding = "assert" to keep upstream origin labels on relay-to-relay hops, or "none" to leave trust_node in charge.

Clients connect but see nothing

• The pipeline may be filtering everything out; check flow.filters stats.
• The rate limiter may be dropping everything; check rate_limiter stats.
• Nothing may be arriving from the sources; check source total_entries.

Entries missing under load

• Compare dropped_writes (per-client queue full — raise client_buffer_size) against total_dropped_by_sink (sink input queue full — raise buffer_size or reduce sink latency).