v0.27.0 packaging update

doc/README.md

@@ -30,7 +30,7 @@ streams, or downstream LogWisp nodes.
 - [CLI](cli.md): flags, signals, exit codes, `lw auth`; also the `lw(1)`
   manual, [`lw.1`](lw.1)
 - [Operations](operations.md): running, monitoring, tuning, troubleshooting
-- [To Do](todo.md): planned work: packaging
+- [To Do](todo.md): planned work: packaging, `lw --tui` and its shared configuration engine
 
 A fully annotated configuration covering every option lives at
 [`config/logwisp.toml`](../config/logwisp.toml).

doc/installation.md

@@ -27,6 +27,9 @@ The Makefile works with GNU make and BSD make alike. Targets:
   - `make release`: the same, static (`CGO_ENABLED=0`), `-trimpath`, stripped
   - `make dev`: built with the race detector
   - `make version`: the metadata a build would embed
+  - `make completion`: regenerate the bash, zsh and fish completion in
+    `deploy/package/completion/` from `cmd/lw`'s `shorts` and `commands`
+    tables; `go test ./cmd/lw` fails while they differ
   - `make clean`: remove `bin/`
 - Check
   - `make test`: the Go tests
@@ -39,7 +42,7 @@ The Makefile works with GNU make and BSD make alike. Targets:
 - Container
   - `make image`, `make image-check`: see [Container Image](#container-image)
 - Install
-  - `make install`, `make uninstall`: see [Installing](#installing)
+  - `make install`, `make uninstall`, `make deb`: see [Installing](#installing)
 
 Variables:
 
@@ -70,6 +73,11 @@ first. It honours `DESTDIR`, `PREFIX`, `SYSCONFDIR` and `BINDIR` (default
 - FreeBSD defaults: `PREFIX=/usr/local`, `SYSCONFDIR=/usr/local/etc`.
 - `INSTALL_OS` (default: `uname -s`) picks the layout, so
   `INSTALL_OS=FreeBSD DESTDIR=...` stages the FreeBSD one on another system.
+- `BASHCOMPDIR`, `ZSHCOMPDIR`, `FISHCOMPDIR`: where the shell completion
+  goes, by default `$PREFIX/share/bash-completion/completions`,
+  `$PREFIX/share/zsh/site-functions` (`/usr/share/zsh/vendor-completions`
+  for `PREFIX=/usr` on Debian and Ubuntu, whose zsh reads only that) and
+  `$PREFIX/share/fish/vendor_completions.d`.
 
 ```bash
 make release
@@ -79,6 +87,8 @@ sudo make install PREFIX=/usr/local
 Every system gets:
 
 - `$BINDIR/lw` and the manual `$PREFIX/share/man/man1/lw.1`
+- completion for `lw`, its commands, flags, files and preset names:
+  `lw`, `_lw` and `lw.fish` in the three directories above
 - `$PREFIX/share/doc/logwisp/` (this documentation) and
   `$PREFIX/share/licenses/logwisp/LICENSE`
 
@@ -146,6 +156,28 @@ rc.conf variables:
 - `logwisp_args`: extra `lw` options, e.g. `--logging.level=debug` (default
   empty); `logwisp_flags` goes to `daemon(8)`, not to `lw`
 
+### Debian and Ubuntu (.deb)
+
+`make deb` builds a release and packages what `make install` stages, in
+Debian's layout, as `bin/logwisp_VERSION_ARCH.deb`, the architecture taken
+from the binary. A release tag gives the version; an untagged or modified
+tree's sorts below every release and by build time, so each build upgrades
+the last. `DEB_MAINTAINER` names the maintainer.
+
+```bash
+make deb
+sudo apt install ./bin/logwisp_*.deb
+sudo systemctl enable --now logwisp
+```
+
+- Installing creates the `logwisp` account and its directories where
+  `systemd-sysusers` and `systemd-tmpfiles` exist; the service waits to be
+  enabled, and an upgrade restarts it if it runs.
+- `/etc/logwisp/logwisp.toml` is a conffile: an upgrade keeps your edits,
+  and only a purge removes it.
+- Removing stops and disables the service. The account, its directories
+  and any credentials in `/etc/logwisp` stay, as with `make uninstall`.
+
 ### Uninstall
 
 `sudo make uninstall` (with the `PREFIX` used to install) removes everything
@@ -235,18 +267,20 @@ docker run -d --name logwisp \
 
 ## Packaging Status
 
-The foundation for distribution packages is in place:
+In place:
 
-- the `lw` binary name, and `make install` with `DESTDIR`, `PREFIX` and
-  `SYSCONFDIR`
-- the `doc/lw.1` manual
-- service files in `deploy/package/`: `logwisp.service`, `logwisp.sysusers`,
-  `logwisp.tmpfiles` and the FreeBSD `logwisp.rc`
+- `make install` with `DESTDIR`, `PREFIX` and `SYSCONFDIR`, the manual,
+  shell completion and the service files in `deploy/package/`
+- `make deb`
 - skeletons: `deploy/package/arch/PKGBUILD` and a `sysutils/logwisp` port in
   `deploy/package/freebsd/`
+- `.github/workflows/package.yml`, on changes to them: `makepkg` on the
+  PKGBUILD in an Arch container, the `.deb` installed, started, removed and
+  purged on Ubuntu, BSD make building, installing and running the rc.d
+  service on FreeBSD, and `make image image-check`
 
-Still missing: release tags, which both skeletons download; finishing and
-submitting the AUR package and the port; Debian packaging.
+Still missing: release tags, which both skeletons download; submitting the
+AUR package and the port; a Debian source package for a PPA or the archive.
 [To Do](todo.md) has the steps.
 
 ## Verification

doc/todo.md

@@ -11,14 +11,18 @@ The foundation exists:
   24.04 and 26.04, and the FreeBSD 15.1 ports tree.
 - The canonical module path `github.com/lixenwraith/logwisp`, which
   `go install`, FreeBSD's `USES=go:modules` and Debian's dh-golang expect.
-- `make install` with `DESTDIR`, `PREFIX` and `SYSCONFDIR`.
-- The `doc/lw.1` manual.
-- Service files in `deploy/package/`: systemd unit, sysusers, tmpfiles and the
-  FreeBSD rc.d script.
+- `make install` with `DESTDIR`, `PREFIX` and `SYSCONFDIR`: the binary, the
+  `doc/lw.1` manual, bash, zsh and fish completion generated from `cmd/lw`'s
+  `shorts` and `commands` tables, and the service files in `deploy/package/`
+  (systemd unit, sysusers, tmpfiles, FreeBSD rc.d script).
+- `make deb`, a binary package built with `dpkg-deb`.
 - Skeletons in `deploy/package/arch/` and `deploy/package/freebsd/`.
+- `.github/workflows/package.yml`: `makepkg` on the PKGBUILD, the `.deb`
+  installed, started and purged, BSD make and the rc.d service on FreeBSD
+  15.1, and `make image image-check`.
 
-What remains. Steps 1 to 3 are the maintainer's: they need push rights and an
-identity, and both skeletons download the tagged source.
+What remains is the maintainer's: it needs push rights, an identity or a
+release tag, and both skeletons download the tagged source.
 
 1. Tag the release on the merged main commit, `vX.Y.Z`:
    ```
@@ -52,11 +56,168 @@ identity, and both skeletons download the tagged source.
    - `portlint -AC`, `poudriere testport` on 14.x and 15.x jails, then a
      Bugzilla report with the port directory and the UIDs/GIDs diff.
 4. Debian:
-   - Debian policy wants every Go dependency packaged; the four `lixenwraith`
-     libraries are not.
-   - Either package them too (dh-golang), or start with an `.deb` built by the
-     Makefile `install` target (nfpm or `dpkg-deb`) and a PPA, and move to the
-     archive later.
-5. Shell completion for bash, zsh and fish, installed by `make install`.
-6. A packaging CI job: build the AUR package in an Arch container and the port
-   in a FreeBSD VM, run `lw --version` and `make image-check`.
+   - `make deb` serves direct installs. A PPA builds from a signed source
+     package without network access: a `debian/` directory (changelog,
+     control, rules) and the Go modules vendored into the source tarball.
+   - The archive wants every Go dependency packaged (dh-golang); the four
+     `lixenwraith` libraries are not.
+   - `lintian` on the `make deb` package reports `no-changelog` (a source
+     package's `debian/changelog` provides it), `statically-linked-binary`
+     (lw is static by design) and `maintainer-script-calls-systemctl` (dh's
+     scripts use `deb-systemd-helper` instead).
+5. With a tag, the packaging workflow's FreeBSD job can build the port itself
+   from a ports tree (`make stage check-plist` in a copy of
+   `deploy/package/freebsd/`), rather than BSD make on the checkout.
+
+## 2. `lw --tui` and the configuration engine it shares with lixen.com
+
+`lw --tui` composes pipelines in a full-screen terminal UI, then either runs
+them at once or prints one form of them, chosen on exit: a command line,
+environment variables, or a configuration file. Its logic is a pure engine that
+lixen.com's planned Configure tab (lixencom `doc/todo.md`) runs as WebAssembly,
+so no part of LogWisp is rewritten in JavaScript.
+
+What exists:
+- Option structs with toml tags for every plugin and flow stage
+  (`internal/config/config.go`), presets as rows with typed parameters
+  (`preset.go`), the command-line and environment grammar (`spec.go`),
+  `ValidateConfig`, `--check` and `--dump`.
+- `internal/config` already compiles for `GOOS=js`; it links no net/http,
+  crypto/tls or terminal code.
+- `github.com/lixenwraith/terminal` (pinned) and its `tui` package: regions,
+  layout, boxes, List, Tree, TabBar, StatusBar, TextField, Form, Modal,
+  ConfirmDialog, scrolling, mouse. `github.com/lixenwraith/color`: RGB,
+  blending, RGBTo256, RGBTo16.
+
+What is missing:
+- A machine-readable option schema. Defaults, enums and required checks live
+  inside the plugin constructors (about 50 `Default*` constants, 24 inline
+  validators); `doc/sources.md`, `doc/sinks.md`, `config/logwisp.toml` and the
+  site's catalogue copy them by hand, and the site's copy already lags.
+- Offline plugin validation: the constructors mix their checks with reading
+  certificates, credentials and files.
+- Typed command-line values: `lw --sink http,port=8080 --dump` prints
+  `port = "8080"`, a file `port = 8080`.
+- A host-independent preset: `fileSource` (`preset.go`) decides directory or
+  pattern with `os.Stat`, so off the target host `path=/var/log` expands to
+  directory `/var`, pattern `log`.
+- Emitters: only parsers exist for the three forms.
+
+Steps:
+1. Schema, in `internal/config`, as the one source of truth.
+   - Tags on the option structs: `default:`, `help:`, and `lw:` for required,
+     enum, min and max, what zero means (default, none, unlimited), hints
+     (path, regex, file) and roles (listener, dialer) on the shared tls, auth
+     and acl tables.
+   - One catalogue table, one row per plugin type: role, network role, single
+     instance, summary, option struct. `RegisterSource`/`RegisterSink` panic on
+     a type without a row.
+   - `Decode[T](role, type, map)` applies defaults, unknown-key and tag checks,
+     then the struct's pure `Check()`. Every constructor decodes through it and
+     keeps only I/O; the `Default*` constants and inline validators go.
+   - Pure cross-field checks become methods that tlsx, authz and netacl call
+     first: TLS role rules, auth bindings without file reads, ACL parsing, the
+     http sink's paths, filter regexes.
+   - `Coerce` converts command-line and environment strings, and JSON floats,
+     to the declared kinds; `spec.go` uses it, so a command line dumps typed.
+   - `ValidateConfig` adds unique plugin ids per pipeline and decodes every
+     plugin, naming the path (`pipelines[0].plugin_sinks[view].config.port`).
+     It opens nothing; `lw --check` stays the authority on files.
+   - Keep under test what moves: file sink `min_disk_free_mb` (negative means
+     100, 0 means none), a backoff maximum below the minimum, format `text`.
+2. Engine, `internal/compose`.
+   - Imports internal/config, internal/version and lixenwraith/toml only; a
+     test fails when `go list -deps` shows net, crypto/tls, os/exec, os/signal,
+     x/term or the auth, log and terminal modules.
+   - Starts: empty; a preset, with a `stat` function that is nil off the host
+     (the UI then asks "directory or file pattern"); a loaded configuration; a
+     pasted command line (wrapping `cliSpecs` and `buildPipelines`).
+   - Operations: add or remove a source, sink or filter; set or unset an
+     option; move a filter; validate.
+   - Emitters, one form per call:
+     - Command line: `--pipeline NAME`, `id=`, the rate limit policy and
+       heartbeat `enabled` always explicit (their command-line defaults differ
+       from the file's); `--key=value` when a value starts with `-`; `,` `=`
+       `\` escaped; POSIX-quoted.
+     - Environment: `LOGWISP_SOURCE_1` and the rest, single-quoted
+       `KEY='value'` lines; an error for more than one pipeline, which the
+       environment form cannot hold.
+     - File: the TOML `lw --dump` prints for those pipelines.
+3. `lw --schema`, a parser switch like `--help` (not a Config field, so no
+   file or environment can trigger it). It prints one JSON document: version;
+   settings with kind, default, enum, flag and variable; the pipeline flags;
+   flow stage order; the tls, auth and acl tables; sources, sinks, presets.
+   The site generates its catalogue from it.
+4. In `lixenwraith/terminal`, released and pinned before the TUI:
+   - Full-screen 8- and 16-color output (SGR 30-37, 90-97, 40-47) and the
+     default colors 39/49; it emits only 256-color and truecolor today.
+   - Detection as `inline` does it: 16 colors on text consoles, none under
+     `NO_COLOR`, refusal on `TERM=dumb`, and an override for LogWisp's `color`
+     key.
+   - Drawing on `/dev/tty`, so `lw --tui > logwisp.toml` and `cmd | lw --tui`
+     keep stdin and stdout as data.
+   - Bracketed paste into text fields.
+   - `tui` widgets the site's controls have: segmented choice, filterable
+     option list, toggle, typed fields (default as placeholder, required mark,
+     error and help line, folded groups), a focus ring; every widget styled
+     from one `tui.Theme`.
+   - In `color`, optional: export the 16 xterm reference shades. RGBTo16 maps
+     the site's hues badly (green to cyan, red to bright black), so the
+     8-color tier is chosen by hand.
+5. `lw --tui`.
+   - A parser switch, no short form. With `-c FILE`, pipeline flags or
+     `LOGWISP_SOURCE` it starts from that composition, and never rewrites FILE
+     (TOML encoding drops its comments); bare, it opens the preset menu (pipe,
+     tail, serve, edge, aggregator, empty).
+   - The pipeline as the site draws it: SOURCES, FLOW, SINKS; a node is a
+     tinted bar, its type, listens or dials, and a summary line; the four flow
+     stages in order inside a dashed box, the ones off dimmed; wires drawn from
+     per-cell connectivity so every junction is right. The inspector sits
+     below at 80 to 109 columns, beside at 110 and over; under 64 columns the
+     stages stack, as on the site.
+     ```
+      ● SOURCES              ● FLOW                       ● SINKS
+      ▌ file             ─┐  ╭╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╮  ┌─▌ http      listens
+      ▌ /var/log  *.log   │  ╎ ▌ rate_limit 400/s   ╎  │ ▌ :8080  /stream
+      █ tcp_chain listens─┴─►╎ ▌ filters -TRACE     ╎─►┤
+      █ :9000  tls  scram    ╎ ▌ format txt         ╎  └─▌ console
+                             ╎ ○ heartbeat off      ╎    ▌ stdout
+                             ╰╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╯
+     ── tcp_chain · source · id src-2 ─────────────────── ⏎ edit  tab next ──
+        host     0.0.0.0  default        format   raw [txt] json
+        port *   9000▌                   tls      ▸ on   (auth, acl folded)
+      ✓ valid   a add  d delete  ␣ on/off  J/K move  p preset  o output  ? help
+     ```
+   - Keys: arrows or hjkl move, Enter edits, `a` adds from the catalogue, `d`
+     deletes, Space switches a flow stage, `J`/`K` reorder filters, `p`
+     presets, `[` `]` switch pipelines, `c` lists errors, `o` output, `?`
+     help, `q` quits.
+   - Theme: the site's dark tokens as one role table in four tiers. Truecolor
+     and 256 colors paint the site's background; 8 colors and mono keep the
+     terminal's. Sources cyan, flow violet, sinks green, accent blue, muted
+     grey, errors red. A selection always changes a glyph (bar, `▸`,
+     underline), never only a shade. Glyphs: Unicode, a CP437 set on text
+     consoles, ASCII outside UTF-8.
+   - Output, `o`: Run hands `config.Args` built from the composition to the
+     usual `config.Load`, after the terminal is released and before anything
+     reads stdin; reloads keep the pipelines, and `--tui --check` and `--tui
+     --dump` follow. Or exactly one of command line, environment and file,
+     printed to stdout once the screen closes.
+6. WebAssembly: `cmd/lwconf` (`js && wasm`) exposes schema, preset, parse,
+   validate and emit to the page as JSON strings. `make wasm` builds
+   `bin/lwconf.wasm` with the version stamped, copies the toolchain's
+   `wasm_exec.js`, and prints both checksums. A prototype measured about 6 MB,
+   1.7 MB gzipped, ready in about 60 ms in Chromium; `wasm_exec.js` uses no
+   eval, so the page needs only `'wasm-unsafe-eval'` in its `script-src`.
+
+Verify:
+- A Go test in `cmd/lw`: for every preset, an edge and aggregator pair,
+  escaped values and two pipelines, the pipelines loaded from the emitted
+  command line, file and environment (one pipeline) are equal. It runs in an
+  empty directory with the `LOGWISP_` environment cleared, since a discovered
+  file would add settings.
+- A schema test: every option has a kind, every enum values, every catalogue
+  row a factory; `TestEveryPluginRejectsUnknownKeys` still passes.
+- CI builds `cmd/lwconf` for `GOOS=js` and runs a node test of its calls.
+- The diagram rendered into a cell buffer at 80x24 and 64x24, per tier; a
+  scripted session on a pseudo-terminal for Run and each output.