v0.37.1 match bytes to follow a file tail, preventing reset on edit in most cases

doc/sources.md

@@ -59,18 +59,36 @@ Options, each as type and default:
   written meanwhile is not read. `from = "start"` instead reads each file whole
   when its watcher is created, which a process writing beside LogWisp needs, at
   the cost of replaying files already on disk at every restart and reload.
-- Rotation is detected from size decrease, a modification time that went back
-  without growth, a position more than 1 KiB beyond end-of-file, or an inode
-  change. An inode change where the new file is non-empty and at least as large
-  as the recorded position is treated as an atomic save, not a rotation, and
-  the position is preserved. Each rotation also publishes an INFO entry
-  `Log rotation detected (#N): REASON` with the file's name as `Source`; it
-  reaches every sink, so exclude it with a filter (`Log rotation detected`)
-  where unwanted.
-- A rotation that renames in place — what a size-capped writer does — puts the
-  same inode back under a name `pattern` also matches. Its watcher resumes at
-  the position the original reached, so `from = "start"` reads the tail an
-  unfinished read left behind rather than the whole archive a second time.
+- A file is followed by its content, not its inode or size. lw keeps the last
+  8 lines read (1 KiB at most), matches them only whole (never a line still
+  being written), and assumes lines are mostly distinct, as timestamped logs
+  are; runs of identical lines can make it resume after a later copy.
+  - Appended to, it reads on.
+  - Rewritten, in place or by an editor's rename: reading resumes after the
+    newest of those lines still in the file's last 64 KiB, wherever lines were
+    deleted or inserted before it; inserted lines are not read. Nothing read
+    is replayed.
+  - None of those lines left (logrotate's `create` or `copytruncate`, a file
+    emptied and written again, an edit deleting them all), and no other
+    watcher having read the file (below), is a rotation: read from its start,
+    after an INFO entry
+    `Log rotation detected (#N): the file starts anew` with the file's name as
+    `Source`, which reaches every sink (filter `Log rotation detected` out
+    where unwanted). The NULs `copytruncate` leaves when a writer that does
+    not append goes on at its old offset are skipped, unless the file stayed
+    empty for a whole poll.
+  - While the line read last is missing and the file still changes between
+    polls, as when a slow save is rewriting it in place, lw holds that file,
+    for 2 s at most, and decides once it settles.
+  - Limits: a save appending more than 64 KiB after those lines loses them,
+    so it is read from its start. A rewrite still being written after 2 s is
+    judged as it stands: read from its start if it has not reached those
+    lines yet, else what it writes after that is replayed.
+- A rotation that renames in place (a size-capped writer, a numbered logrotate
+  chain shifting `app.log.1` to `app.log.2`) puts the same inode back under a
+  name `pattern` also matches. The watcher there, new or already following
+  that name, resumes where the file was read to: it reads only the tail an
+  unfinished read left behind, never the whole archive a second time.
 - A line is parsed as JSON only when it is a non-empty object whose top-level
   keys are all drawn from `time`, `level`, `msg` and `fields` (the four an
   entry can carry), with any `level` or `msg` a string. `time` is read as