logwisp/doc/installation.md

Installation Guide

Requirements

• Operating systems: Linux (kernel 6.10+), FreeBSD (14.0+)
• Architecture: amd64; arm64 builds but is untested
• To build: Go 1.27.2 or newer (FreeBSD: the go127 package), and GNU make or BSD make
• For make e2e: bash 5+, coreutils, curl, openssl, go (the proxy and acl tests build helpers) and nc (acl). Optional, else those checks SKIP: script(1) and setsid(1) (preset-test.sh's terminal checks), node with playwright and the Chromium builds playwright install chromium fetches (browser checks), an IPv6 loopback (ipv6-test.sh), docker as root with the logwisp:dev image (deploy-test.sh's container run)

Building from Source

git clone https://github.com/lixenwraith/logwisp.git
cd logwisp
make build              # bin/lw; plain `make` only lists the targets

The Makefile works with GNU make and BSD make alike. Targets:

• Build
  • make build: bin/lw, with version, commit and build time from git
  • make release: the same, static (CGO_ENABLED=0), -trimpath, stripped
  • make dev: built with the race detector
  • make wasm: bin/lwconf/, the configuration engine a website drops in: lwconf.wasm (version stamped), the toolchain's wasm_exec.js, lwconf.js that loads and calls it, schema.json (lw --schema) and lwconf.json, their manifest: version, commit, Go version and each file's sha256. Stamped with the commit's time, a commit's files are the same on every build. Its calls: Architecture
  • make version: the metadata a build would embed
  • make completion: regenerate the bash, zsh and fish completion in deploy/package/completion/ from cmd/lw's shorts and commands tables; go test ./cmd/lw fails while they differ
  • make clean: remove bin/
• Check
  • make test: the Go tests
  • make verify: tests, go vet, gofmt -l on the Go files changed since main, linux and freebsd cross builds for amd64 and arm64 and the js/wasm one, and, when node is installed, node --test for the web client and for cmd/lwconf (which runs make wasm)
  • make e2e: builds, then runs every test/*-test.sh --auto in turn and reports passes, failures and skips; E2E='test/scram-*-test.sh' runs a subset; see Test Scripts
• Container
  • make image, make image-check: see Container Image
• Install
  • make install, make uninstall, make deb, make arch: see Installing

Variables:

• GO: the toolchain. On FreeBSD the default is the versioned package that go.mod asks for (go127); make GO=/path/to/go overrides it, and check-go names the package to install when it is missing.
• GO_BUILDFLAGS, GO_LDFLAGS: extra go build and linker flags for packagers. They are distinct from GOFLAGS and LDFLAGS, which the go command and the C linker read themselves.
• VERSION, GIT_COMMIT, BUILD_TIME: what a build embeds, by default from git and the clock; a package sets them.
• SOURCE_DATE_EPOCH: when set, the embedded build time comes from it, so package builds are reproducible.

A plain go build -o bin/lw ./cmd/lw works too, as does go install github.com/lixenwraith/logwisp/cmd/lw@latest (or @vX.Y.Z). Without the Makefile's -ldflags, lw --version reports what Go stamps into the binary: the module version, or a pseudo-version with the commit in a checkout.

Installing

make install copies what a package ships and compiles nothing, so build first, or in one command: make release install. It honours DESTDIR, PREFIX, SYSCONFDIR, BINDIR (default $PREFIX/bin, the path the systemd unit and the rc.d script run lw from), MANDIR, DOCDIR and LICENSEDIR (under $PREFIX/share by default) and SERVICE; make help prints the values in effect:

• Linux defaults: PREFIX=/usr, SYSCONFDIR=/etc. Outside a package manager, prefer PREFIX=/usr/local; systemd reads units, sysusers and tmpfiles from there too.
• FreeBSD defaults: PREFIX=/usr/local, SYSCONFDIR=/usr/local/etc.
• A user who cannot write that layout's bin directory, unstaged, gets a per-user install instead: PREFIX=~/.local, SYSCONFDIR=~/.config, SERVICE=no (Without a Service). sudo make install installs system-wide.
• Before writing anything, install checks that it can write every directory it needs, and stops naming the first it cannot. It creates the missing ones and leaves the mode of existing ones alone.
• INSTALL_OS (default: uname -s) picks the layout, so INSTALL_OS=FreeBSD DESTDIR=... stages the FreeBSD one on another system.
• SERVICE: yes (default) adds the service and its configuration; no leaves them out (Without a Service).
• BASHCOMPDIR, ZSHCOMPDIR, FISHCOMPDIR: where the shell completion goes, by default $PREFIX/share/bash-completion/completions, $PREFIX/share/zsh/site-functions (/usr/share/zsh/vendor-completions for PREFIX=/usr on Debian and Ubuntu, whose zsh reads only that) and $PREFIX/share/fish/vendor_completions.d.

make release
sudo make install PREFIX=/usr/local

Build as yourself and install with sudo: under sudo the build would run with root's PATH and Go cache and leave a bin/ only root can replace.

Every install gets:

• $BINDIR/lw and the manual $MANDIR/man1/lw.1
• completion for lw, its commands, flags, files and preset names: lw, _lw and lw.fish in the three directories above
• $DOCDIR (this documentation) and $LICENSEDIR/LICENSE

A service install also gets the service files below and, on Linux and FreeBSD alike, $SYSCONFDIR/logwisp/logwisp.toml: the annotated sample configuration, placed when none is there and otherwise kept. As shipped it defines no pipeline, so the service would copy an empty stdin and exit: edit it before starting the service.

Linux (systemd)

• $PREFIX/lib/systemd/system/logwisp.service: runs lw -c $SYSCONFDIR/logwisp/logwisp.toml as the logwisp user, from /var/lib/logwisp; systemctl reload sends SIGHUP.
• $PREFIX/lib/sysusers.d/logwisp.conf: the logwisp account.
• $PREFIX/lib/tmpfiles.d/logwisp.conf: /var/lib/logwisp and /var/log/logwisp, owned by it.

sudo systemd-sysusers logwisp.conf
sudo systemd-tmpfiles --create logwisp.conf
sudo systemctl daemon-reload
sudo systemctl enable --now logwisp

The unit is hardened: no capabilities, a read-only view of the system with only its two directories writable, /home hidden, and IPv4, IPv6 and Unix sockets only. Widen it with a drop-in (systemctl edit logwisp) rather than editing the unit:

• A file sink or logging.file elsewhere: ReadWritePaths=/srv/logs
• A file source under /home: ProtectHome=read-only
• A port below 1024: CapabilityBoundingSet=CAP_NET_BIND_SERVICE and AmbientCapabilities=CAP_NET_BIND_SERVICE

The service must read every key, credentials and password file it names. Create a credentials file with its owner and mode first; lw auth keeps both:

sudo install -m 0640 -o root -g logwisp /dev/null /etc/logwisp/users.toml
sudo lw auth add-user --credentials /etc/logwisp/users.toml --user <name> \
  --password-file <file>
sudo systemctl reload logwisp

FreeBSD (rc.d)

$PREFIX/etc/rc.d/logwisp runs lw through daemon(8) as logwisp_user, with its output in syslog under the tag logwisp; service logwisp reload sends SIGHUP. The port in deploy/package/freebsd/ places the configuration the ports way, to the same effect: @sample installs logwisp.toml.sample and copies it to logwisp.toml when none exists.

sudo pw useradd logwisp -d /nonexistent -s /usr/sbin/nologin -c "LogWisp log transport"
sudo sysrc logwisp_enable=YES
sudo service logwisp start

rc.conf variables:

• logwisp_enable: YES to start at boot (default NO)
• logwisp_config: the configuration (default /usr/local/etc/logwisp/logwisp.toml)
• logwisp_user: the account (default logwisp)
• logwisp_chdir: the working directory (default /var/db/logwisp, created for logwisp_user when missing)
• logwisp_args: extra lw options, e.g. --logging.level=debug (default empty); logwisp_flags goes to daemon(8), not to lw

Debian and Ubuntu (.deb)

make deb builds a release and packages what make install stages, in Debian's layout, as bin/logwisp_VERSION_ARCH.deb, the architecture taken from the binary. A release tag gives the version; an untagged or modified tree's sorts below every release and by build time, so each build upgrades the last. DEB_MAINTAINER names the maintainer.

make deb
sudo apt install ./bin/logwisp_*.deb
sudo systemctl enable --now logwisp

• Installing creates the logwisp account and its directories where systemd-sysusers and systemd-tmpfiles exist; the service waits to be enabled, and an upgrade or reinstall restarts it if it runs.
• /etc/logwisp/logwisp.toml is a conffile: an upgrade or reinstall keeps your edits, and only a purge removes it.
• The same version again: sudo apt install --reinstall ./bin/logwisp_*.deb.
• Removing stops and disables the service. The account, its directories and any credentials in /etc/logwisp stay, as with make uninstall.

Arch Linux (makepkg)

make arch builds the PKGBUILD in deploy/package/arch/ from the working tree, committed or not, into bin/arch/, then prints the pacman -U line that installs the package. It needs base-devel and go, and runs as a user: makepkg refuses root.

make arch
sudo pacman -U bin/arch/logwisp-<version>-1-x86_64.pkg.tar.zst
sudo systemctl enable --now logwisp

• pacman's hooks create the logwisp account and its directories; the service waits to be enabled.
• /etc/logwisp/logwisp.toml is a backup file: an upgrade or reinstall keeps your edits, and a shipped version that changed lands beside it as logwisp.toml.pacnew. pacman -R keeps an edited one as logwisp.toml.pacsave.
• Reinstall: make arch and the same pacman -U. pacman does not restart the service: sudo systemctl restart logwisp.
• An earlier sudo make install conflicts with the package (PREFIX=/usr) or overrides it (PREFIX=/usr/local, whose unit and lw come first): sudo make uninstall with that PREFIX first. The configuration it keeps is no conflict: pacman keeps it, and when it was edited, the shipped one lands beside it as logwisp.toml.pacnew.

Reinstalling

Every method runs again over its own installation, for an upgrade or the same version, and keeps an edited configuration: make install leaves an existing $SYSCONFDIR/logwisp/logwisp.toml alone, and the packages treat it as above. Only the .deb restarts a running service; otherwise restart it (sudo systemctl restart logwisp, sudo service logwisp restart). To see what a release changed in the shipped configuration:

d=$(mktemp -d); lw config init --out "$d/logwisp.toml"; diff "$d/logwisp.toml" /etc/logwisp/logwisp.toml

Without a Service

SERVICE=no installs lw, its manual, completion and documentation, and nothing else: no service files, account or configuration. A user without root gets it under ~/.local by default:

make release install
lw config init

• ~/.local/bin goes on PATH (make install notes when it is not). man lw then finds the manual, and bash and fish the completion; zsh needs fpath=(~/.local/share/zsh/site-functions $fpath) before compinit.
• make uninstall, as the same user, removes it.
• Where that user can write the system's bin directory, name the layout: make install SERVICE=no PREFIX="$HOME/.local" SYSCONFDIR="$HOME/.config".

For every user: sudo make install SERVICE=no PREFIX=/usr/local.

lw config init writes the shipped, annotated configuration, never over an existing file, where you choose:

• lw config init: ~/.config/logwisp/logwisp.toml, which lw reads without -c unless a LOGWISP_CONFIG_* variable names another file. As shipped it defines no pipeline, so lw still copies stdin to stdout, though its own log on stderr is at the file's default info rather than warn; once it defines [[pipelines]], lw without pipeline options runs them, and pipeline options and presets still replace them.
• sudo lw config init --out /usr/local/etc/logwisp/logwisp.toml: a file for every user, named with -c, or without it by LOGWISP_CONFIG_DIR=/usr/local/etc/logwisp (File Location).

lw --check (with the same -c) then validates it.

Uninstall

sudo make uninstall (with the PREFIX used to install) removes everything make install wrote except $SYSCONFDIR/logwisp, which holds configuration and credentials, and any file lw config init wrote. Remove those, the working directories and the account by hand.

Container Image

The Dockerfile builds lw in a Go builder and copies it alone, static and stripped, into scratch: /lw running as UID 65532, plus the CA bundle that TLS dialers without a ca_file verify against. There is no shell and no configuration in the image.

make image                    # logwisp:dev; IMAGE=, IMAGE_TAG= rename it
make image-check

image-check runs the image the way it should run in production: read-only, --network none, --cap-drop ALL, no-new-privileges, as 65532. It prints --version, then runs --check on config/logwisp.toml (or IMAGE_CHECK_CONFIG) mounted read-only.

Building behind a proxy:

• A TLS-intercepting proxy: make image BUILD_CA=/path/to/ca-bundle.pem. The bundle reaches only the module download, as a BuildKit secret.
• An explicit proxy: pass Docker's predefined proxy build arguments, which never reach an image layer, e.g. IMAGE_BUILD_FLAGS='--network host --build-arg HTTPS_PROXY=http://127.0.0.1:<port>'.
• CONTAINER_ENGINE=podman works too.

Running it securely

docker run -d --name logwisp \
  --read-only --cap-drop ALL --security-opt no-new-privileges \
  -v /etc/logwisp:/etc/logwisp:ro \
  -v /srv/logwisp/secrets:/run/secrets:ro \
  -p 8443:8443 \
  -e LOGWISP_LOGGING_LEVEL=info \
  logwisp:<tag> -c /etc/logwisp/logwisp.toml

• Configuration: mount it read-only at /etc/logwisp and name it with -c. The files must be readable by UID 65532.

• Scalar settings: environment variables, LOGWISP_<PATH>.

• Pipelines without a file: the pipeline variables, in the --source and --sink syntax (see CLI); they replace the file's pipelines:

  -e LOGWISP_PIPELINE=relay \
  -e LOGWISP_SOURCE=tcp_chain:port=9000,tls.enabled=true,tls.cert_file=/run/secrets/tls.crt,tls.key_file=/run/secrets/tls.key,auth.type=scram,auth.credentials_file=/run/secrets/users.toml \
  -e LOGWISP_SINK=console
  

  Without a file lw logs less (CLI): add -e LOGWISP_LOGGING_LEVEL=info -e LOGWISP_STATUS_REPORTER=true; the image sets neither, as a variable would override a mounted file's value. Without -i, stdin is /dev/null, so a container needs a source other than console.

• Secrets: only as mounted files, at /run/secrets (Docker or Compose secrets, a Kubernetes secret volume), named by tls.key_file, auth.credentials_file and auth.password_file. Never put a password in the environment: docker inspect and /proc show it. Make each file readable by 65532 only (chown 65532 and mode 0400).

• Privileges: keep the image's non-root user, a read-only root filesystem, --cap-drop ALL and no-new-privileges. Without capabilities a listener needs a port of 1024 or above; publish it on any host port.

• Listeners bind 0.0.0.0 inside the container; 127.0.0.1 would be unreachable through -p.

• Writable data: a file sink or logging.file needs a volume writable by 65532; nothing else is written.

• Reload: docker kill -s HUP logwisp.

• lw auth runs from the image too, with directories writable by 65532. Keep the generated passwords out of the directory the server mounts: it needs only users.toml (verifiers) and the TLS files, while a .pass file is the client's plaintext password.

  docker run --rm --read-only --cap-drop ALL --network none \
    -v /srv/logwisp/secrets:/work -v /srv/logwisp/passwords:/pw logwisp:<tag> \
    auth add-user --credentials /work/users.toml --user <name> --password-file /pw/<name>.pass
  

Packaging Status

In place:

• make install with DESTDIR, PREFIX, SYSCONFDIR and SERVICE, the manual, shell completion and the service files in deploy/package/
• make deb, and make arch building the PKGBUILD from the working tree
• skeletons: deploy/package/arch/PKGBUILD and a sysutils/logwisp port in deploy/package/freebsd/
• .github/workflows/package.yml, on changes to them:
  • Arch container: make arch, installed over an edited configuration, reinstalled keeping it, removed
  • Ubuntu: the .deb installed, started, reinstalled keeping an edited configuration, removed and purged
  • Ubuntu, as a user without root: make release install into the home twice, without a service, then lw config init writing the file lw reads, once; an explicit PREFIX it cannot write fails
  • FreeBSD: BSD make building, installing twice keeping an edited configuration, and running the rc.d service
  • make image image-check

Still missing: release tags, which both skeletons download; submitting the AUR package and the port; a Debian source package for a PPA or the archive. To Do has the steps.

Verification

lw --version
lw -c /etc/logwisp/logwisp.toml --logging.level=debug --logging.output=stderr
sudo systemctl status logwisp      # Linux
sudo service logwisp status        # FreeBSD

Expect Created source instance, Created sink instance and Starting pipeline for each pipeline. To validate a configuration without starting it, run lw --check -c FILE; see Operations.

Test Scripts

The end-to-end scripts in test/ run against bin/lw and share test/lib.sh. make e2e runs each with --auto and reports it as PASS, FAIL or SKIP (exit 77, e.g. no IPv6 loopback); only a failure fails the target.

• Modes:
  • without flags: start the daemons, print a guide of the ports and one-line commands (run . test/run-X/env first for $LW, $CA and the token helper), and wait for Ctrl-C
  • --auto: run the checks and tear down; --keep leaves the daemons up after a pass
  • --follow: show only WARN, ERROR and key events from the daemons
• Daemons log only to test/run*/log/NAME.out, never to the terminal.
• Colour only on a terminal; NO_COLOR turns it off, FORCE_COLOR=1 on.
• Exit status: 0 all passed, 1 a failure, 77 skipped.
• Each script owns a port range and a gitignored run directory:

15801-15804  chain-test.sh, chain-aggregate-test.sh  test/run/
(no ports)   passthrough-test.sh                     test/run/passthrough/
(no ports)   pipe-test.sh                            test/run/pipe/
15811-15814  mtls-chain-test.sh                      test/run-mtls/
15821-15825  scram-chain-test.sh                     test/run-scram/
15831-15833  scram-proxy-test.sh                     test/run-proxy/
15851-15855  ipv6-test.sh                            test/run-ipv6/
15871-15879  deploy-test.sh                          test/run-deploy/
15881-15889  acl-test.sh                             test/run/acl/
15890-15898  preset-test.sh                          test/run/preset/

• chain-test.sh: two edges into a relay, one pipeline per chain transport
• chain-aggregate-test.sh: the same edges, with the relay fanning both into both sinks
• passthrough-test.sh: file source to file sink, a wide envelope byte for byte (raw = true) and, parsed, with no key dropped
• pipe-test.sh: lw without a configuration as a filter, stdin to stdout line for line without loss, quiet on stderr, exit 0 at the end of input
• mtls-chain-test.sh: mTLS chain edges and viewers, allow lists, node binding
• scram-chain-test.sh: SCRAM chain edges and viewers with lw auth credentials
• scram-proxy-test.sh: browser and CLI logins to an http sink behind a TLS-terminating proxy; the browser checks skip without node and playwright's Chromium: the headless shell for the scenarios with cookies, full Chromium for the profile that blocks them
• ipv6-test.sh: a relay on ::1 over both chain transports, IPv6 SANs, per-family listeners; skips (exit 77) without an IPv6 loopback
• deploy-test.sh: the configurations deploy/lw-deploy.sh writes, deployed and checked
• acl-test.sh: each listener admitting or refusing peers by address before TLS, with clients from 127.0.0.1 and 127.0.0.2, then by the client a PROXY header names behind a stub proxy on 127.0.0.11-13 (and nginx stream on 127.0.0.14 when installed) or an L7 proxy's X-Forwarded-For, with per-client connection caps, request rates and login throttling; skips (exit 77) where 127.0.0.2 cannot be bound
• preset-test.sh: colors, presets, --dump, lw tls, listeners that make their certificate, the serve preset's viewer, network sinks delivering a finite input whole, a SCRAM pipe whose password is on no disk; the terminal checks skip without script(1) or setsid(1)