v0.3.0 format a document, each comment with its owner; fewer allocations on the hot path

* Benchmarks for the parse, decode, encode and document paths

A command's payload from text to its struct, a settings file with comment
blocks, a scenario file of arrays of inline tables across lines, decoding
and encoding that scenario, and ParseDocument: the readings the changes
that follow are measured by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Fewer allocations on the parse, decode and encode paths

The parser's lexer drops what it never reads: a comment's text and a
number's literal, which the parser parses from the input bytes in place.
A string without escapes is one allocation, one with escapes is decoded
into a buffer sized to its source. The lexer scans ASCII bytewise, the
parser holds its lexer by value, keeps key paths on the stack, gathers an
array's elements in a reused buffer and allocates the array once at its
length, and holds the first few frozen inline tables in place; an inline
table that is an array element is not held, as no key can reach it.

Decode, Marshal and Unmapped read a struct's fields from one cache per
type instead of splitting every tag on every call. Decode reuses one key
and one element per map; Marshal sets a map's values into one array, and
writes keys, headers and numbers into its buffer without concatenating.

Tokens, positions, trees, error messages, encoded bytes and decoded
values are unchanged: a differential fuzz against the previous sources
ran 1.7M inputs over the lexer and parser and 9.5M over Marshal and
Decode without a difference.

benchstat, 8 runs each, Xeon 2.1 GHz:
  payload, parse and decode  3.31us -> 2.30us   31 -> 14 allocs  1.41 -> 1.09 KiB
  parse payload              2.40us -> 1.89us   24 -> 12 allocs
  parse settings            10.25us -> 6.98us   68 -> 32 allocs
  parse scenario             536us -> 391us   4959 -> 2970 allocs
  decode scenario            225us -> 110us   1863 -> 776 allocs
  marshal scenario           397us -> 252us   2326 -> 976 allocs
  ParseDocument settings    21.7us -> 16.5us   128 -> 85 allocs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Document: a comment goes with what fewer blank lines part it from

ParseDocument lays a document out as statements, runs of statements no
blank line parts, and the comment groups between them. A group belongs
to the side fewer blank lines part it from, the side below on a tie:
right above a statement it is that statement's own; right below a run, or
apart from it, it is the run's. Between two runs several groups split at
the widest gap. The file's start counts as no blank line and its end as
half of one, so a comment at the top with a blank line under it, or at
the end with one above it, is the file's, and one touching the last
statement stays with it.

Delete takes a statement with its own comments; a run going whole takes
its comments too and one blank run beside it, so no comment is left
orphaned and no gap doubles. Set puts a new table above the file's
closing comments and a first root key above the first table's comments,
a blank line apart. What v0.2.0 kept by position (the comment above a
dropped key, a dropped table's heading) now goes with its owner.

FuzzDocument checks that a delete leaves whole lines the document had,
in order, and that adding a key loses none; it ran 12.6M inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Format: one layout for a document, every comment kept with its owner

Format, and Formatter for its options, writes a document spaced as
Marshal writes one: one space around = and after a comma, [1, 2] and
{ a = 1 }, LF line ends, a value's lines a level deeper than the line its
bracket opens on, and a trailing comma exactly when the closing bracket
has a line of its own. Keys, scalars and comments keep their spelling.

Comments stay with what they belong to by the rule Document reads them
with, and the result's blank lines encode it again: a run of blank lines
becomes one, or two below a comment group that belongs above it, so a
second Format reads every comment as the first did and writes the same
bytes. Each table is written once, at its first header: a header opening
it again gives its keys there, its comments above the key they touched or
else below what is there before them, and a dotted key reaching a table
that has a header moves into it with its own comments. Tables of an array
of tables never merge.

Options: Indent (up to 8 tabs or spaces, a tab by default), IndentTables
(a level per enclosing table whose header comes first), Align (= and
comments of consecutive keys).

Hardening: Format refuses what Parse refuses, a comment holding a
bidirectional embedding, override or isolate or a line break other than
LF (U+202A-202E, U+2066-2069, U+0085, U+2028, U+2029), which can show a
reader text other than what parses, and a result that would not parse to
the same values. Such a rune in a string or quoted key is written as its
\u escape, by Format and by the encoder alike. A line indents at most 16
levels and aligns within 32 runes of key and 80 of line, so the result
stays within a fixed multiple of its source; table paths sit in a trie,
so placing a key costs its length, not its square.

Marshal writes { a = 1 } and no blank first line, so Format(Marshal(v))
is Marshal(v), which FuzzRoundTrip now checks. FuzzFormat checks, with
the options off and on, that the result reads as the source, formats to
itself, keeps every comment once, holds no CR or trailing blank and stays
within the size bound; CI runs it for 10 s. All 29 TOML files in vif
format idempotently to the same values; vif.toml and keymap.toml are
unchanged byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

---------

Co-authored-by: Claude <noreply@anthropic.com>

document_test.go

@@ -3,6 +3,7 @@ package toml
 import (
 	"fmt"
 	"slices"
+	"strings"
 	"testing"
 )
 
@@ -33,7 +34,7 @@ buffer_ms = 20
 // TestDocumentEditsOnlyWhatItNames: a set value keeps the rest of its line, a
 // missing key goes after its table's last key in that key's spelling, a missing
 // table is appended, a delete takes a key's lines or a table's header with every
-// key under it, and every other byte, comments and blank lines, stays.
+// key under it and the comments that are theirs, and every other byte stays.
 func TestDocumentEditsOnlyWhatItNames(t *testing.T) {
 	d, err := ParseDocument([]byte(settingsDoc))
 	if err != nil {
@@ -62,8 +63,6 @@ log = "logs" # beside the file
 mouse = false # keys
 sfx = false
 
-# about later
-
 [audio]
 buffer_ms = 20
 "buffer ms" = 30
@@ -110,6 +109,65 @@ func TestSetSpellsAKeyAsItsTableDoes(t *testing.T) {
 	}
 }
 
+// TestCommentsGoWithTheNearerSide: a comment group belongs to what fewer blank
+// lines part it from, below on a tie; the file's start and end count as sides, so
+// its opening and closing comments stay. A key takes its own comments; a run of
+// statements going whole takes its comments and one blank run beside it.
+func TestCommentsGoWithTheNearerSide(t *testing.T) {
+	for _, c := range []struct {
+		name, doc string
+		key       Key
+		want      string
+	}{
+		{"right above", "a = 1\n# about b\nb = 2\nc = 3\n", Key{"b"}, "a = 1\nc = 3\n"},
+		{"right below, run stays", "a = 1\nb = 2\n# after\n\n[t]\nk = 1\n", Key{"b"}, "a = 1\n# after\n\n[t]\nk = 1\n"},
+		{"right below, run goes", "[a]\nx = 1\n# end of a\n\n[b]\ny = 2\n", Key{"a"}, "[b]\ny = 2\n"},
+		{"tie goes below", "[a]\nx = 1\n\n# about b\n\n[b]\ny = 2\n\n[c]\nz = 3\n", Key{"b"}, "[a]\nx = 1\n\n[c]\nz = 3\n"},
+		{"nearer above", "[a]\nx = 1\n\n# about a\n\n\n[b]\ny = 2\n", Key{"b"}, "[a]\nx = 1\n\n# about a\n"},
+		{"widest gap splits", "[a]\nx = 1\n# a1\n\n# a2\n\n\n# b1\n\n# b2\n[b]\ny = 2\n", Key{"b"}, "[a]\nx = 1\n# a1\n\n# a2\n"},
+		{"file's opening", "# file\n\n[a]\nx = 1\n", Key{"a"}, "# file\n"},
+		{"file's closing", "[a]\nx = 1\n\n[b]\ny = 2\n\n# end\n", Key{"b"}, "[a]\nx = 1\n\n# end\n"},
+		{"touching the end", "[a]\nx = 1\n\n[b]\ny = 2\n# y's\n", Key{"b"}, "[a]\nx = 1\n"},
+		{"one blank run left", "a = 1\n\n[b]\nx = 1\n\n\n[c]\ny = 1\n", Key{"b"}, "a = 1\n\n[c]\ny = 1\n"},
+	} {
+		d, err := ParseDocument([]byte(c.doc))
+		if err != nil {
+			t.Fatalf("%s: %v", c.name, err)
+		}
+		if err := del(d, c.key); err != nil {
+			t.Fatalf("%s: %v", c.name, err)
+		}
+		if got := string(d.Bytes()); got != c.want {
+			t.Errorf("%s:\n%s\nwant:\n%s", c.name, got, c.want)
+		}
+	}
+}
+
+// TestSetLeavesCommentsTheirOwners: a root key added above the first table keeps
+// a blank line from that table's comments, and a new table goes above the file's
+// closing comments.
+func TestSetLeavesCommentsTheirOwners(t *testing.T) {
+	for _, c := range []struct {
+		doc  string
+		key  Key
+		want string
+	}{
+		{"# file\n\n# about a\n[a]\nx = 1\n", Key{"top"}, "# file\n\ntop = 1\n\n# about a\n[a]\nx = 1\n"},
+		{"[a]\nx = 1\n\n# end\n", Key{"b", "y"}, "[a]\nx = 1\n\n[b]\ny = 1\n\n# end\n"},
+	} {
+		d, err := ParseDocument([]byte(c.doc))
+		if err != nil {
+			t.Fatal(err)
+		}
+		if err := d.Set(c.key, 1); err != nil {
+			t.Fatal(err)
+		}
+		if got := string(d.Bytes()); got != c.want {
+			t.Errorf("set %s:\n%s\nwant:\n%s", c.key, got, c.want)
+		}
+	}
+}
+
 // TestDocumentRefusesAnEditItCannotPlace: under an array of tables, inside an
 // inline value, or over a table, an edit fails and the document stays as it was.
 func TestDocumentRefusesAnEditItCannotPlace(t *testing.T) {
@@ -200,14 +258,17 @@ func TestKeyStringQuotesWhatABareKeyCannotHold(t *testing.T) {
 }
 
 // FuzzDocument: a document that parses keeps its bytes; deleting any key it sets
-// leaves a shorter document that parses without that key, and setting one or a
-// key beside it, where the edit is taken, leaves it holding the value set.
+// leaves a shorter document that parses without that key, of whole lines it had
+// in their order, and setting one or a key beside it, where the edit is taken,
+// leaves it holding the value set, a new key keeping every line it had.
 func FuzzDocument(f *testing.F) {
 	for _, s := range []string{
 		settingsDoc,
 		"a = 1\n[t]\nk = [1,\n2] # c\n[[r]]\nx = 1\n[[r]]\nx = 2\n",
 		"t.a.b = { c = 1 }\n# end",
 		"\"q.k\" = 1\r\n[\"x y\".z]\r\nv = \"#\"\r\n",
+		"# file\n\n# a\n[a]\nx = 1 # x\n# end a\n\n\n# b\n\n[b]\ny = [\n# in\n1,\n]\n\n# foot\n",
+		"A={}",
 	} {
 		f.Add([]byte(s))
 	}
@@ -234,13 +295,21 @@ func FuzzDocument(f *testing.F) {
 			if len(c.Bytes()) >= len(data) {
 				t.Fatalf("delete of %s did not shorten the document", s.path())
 			}
+			if !keptLines(c.Bytes(), data) {
+				t.Fatalf("delete of %s left lines the document did not have:\n%s", s.path(), c.Bytes())
+			}
 		}
 		for _, s := range d.stmts {
 			for _, k := range []Key{s.path(), append(s.path()[:len(s.path())-1:len(s.path())-1], "added")} {
 				c, _ := ParseDocument(data)
-				if err := c.Set(k, int64(7)); err == nil && value(c.Tree(), k) != int64(7) {
+				added := value(c.Tree(), k) == nil
+				err := c.Set(k, int64(7))
+				if err == nil && value(c.Tree(), k) != int64(7) {
 					t.Fatalf("set %s took but reads %v:\n%s", k, value(c.Tree(), k), c.Bytes())
 				}
+				if err == nil && added && !keptLines(data, c.Bytes()) {
+					t.Fatalf("set of new %s lost a line:\n%s", k, c.Bytes())
+				}
 			}
 		}
 	})
@@ -248,6 +317,22 @@ func FuzzDocument(f *testing.F) {
 
 func has(tree map[string]any, k Key) bool { return value(tree, k) != nil }
 
+// keptLines reports every line of part is a line of whole, in whole's order; a
+// last line may gain its newline
+func keptLines(part, whole []byte) bool {
+	lines := strings.Split(string(whole), "\n")
+	for _, l := range strings.Split(string(part), "\n") {
+		for len(lines) > 0 && lines[0] != l {
+			lines = lines[1:]
+		}
+		if len(lines) == 0 {
+			return l == ""
+		}
+		lines = lines[1:]
+	}
+	return true
+}
+
 // value is what k holds in tree, in the last table of an array of tables
 func value(tree map[string]any, k Key) any {
 	var cur any = tree