v0.3.0 format a document, each comment with its owner; fewer allocations on the hot path

* Benchmarks for the parse, decode, encode and document paths

A command's payload from text to its struct, a settings file with comment
blocks, a scenario file of arrays of inline tables across lines, decoding
and encoding that scenario, and ParseDocument: the readings the changes
that follow are measured by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Fewer allocations on the parse, decode and encode paths

The parser's lexer drops what it never reads: a comment's text and a
number's literal, which the parser parses from the input bytes in place.
A string without escapes is one allocation, one with escapes is decoded
into a buffer sized to its source. The lexer scans ASCII bytewise, the
parser holds its lexer by value, keeps key paths on the stack, gathers an
array's elements in a reused buffer and allocates the array once at its
length, and holds the first few frozen inline tables in place; an inline
table that is an array element is not held, as no key can reach it.

Decode, Marshal and Unmapped read a struct's fields from one cache per
type instead of splitting every tag on every call. Decode reuses one key
and one element per map; Marshal sets a map's values into one array, and
writes keys, headers and numbers into its buffer without concatenating.

Tokens, positions, trees, error messages, encoded bytes and decoded
values are unchanged: a differential fuzz against the previous sources
ran 1.7M inputs over the lexer and parser and 9.5M over Marshal and
Decode without a difference.

benchstat, 8 runs each, Xeon 2.1 GHz:
  payload, parse and decode  3.31us -> 2.30us   31 -> 14 allocs  1.41 -> 1.09 KiB
  parse payload              2.40us -> 1.89us   24 -> 12 allocs
  parse settings            10.25us -> 6.98us   68 -> 32 allocs
  parse scenario             536us -> 391us   4959 -> 2970 allocs
  decode scenario            225us -> 110us   1863 -> 776 allocs
  marshal scenario           397us -> 252us   2326 -> 976 allocs
  ParseDocument settings    21.7us -> 16.5us   128 -> 85 allocs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Document: a comment goes with what fewer blank lines part it from

ParseDocument lays a document out as statements, runs of statements no
blank line parts, and the comment groups between them. A group belongs
to the side fewer blank lines part it from, the side below on a tie:
right above a statement it is that statement's own; right below a run, or
apart from it, it is the run's. Between two runs several groups split at
the widest gap. The file's start counts as no blank line and its end as
half of one, so a comment at the top with a blank line under it, or at
the end with one above it, is the file's, and one touching the last
statement stays with it.

Delete takes a statement with its own comments; a run going whole takes
its comments too and one blank run beside it, so no comment is left
orphaned and no gap doubles. Set puts a new table above the file's
closing comments and a first root key above the first table's comments,
a blank line apart. What v0.2.0 kept by position (the comment above a
dropped key, a dropped table's heading) now goes with its owner.

FuzzDocument checks that a delete leaves whole lines the document had,
in order, and that adding a key loses none; it ran 12.6M inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Format: one layout for a document, every comment kept with its owner

Format, and Formatter for its options, writes a document spaced as
Marshal writes one: one space around = and after a comma, [1, 2] and
{ a = 1 }, LF line ends, a value's lines a level deeper than the line its
bracket opens on, and a trailing comma exactly when the closing bracket
has a line of its own. Keys, scalars and comments keep their spelling.

Comments stay with what they belong to by the rule Document reads them
with, and the result's blank lines encode it again: a run of blank lines
becomes one, or two below a comment group that belongs above it, so a
second Format reads every comment as the first did and writes the same
bytes. Each table is written once, at its first header: a header opening
it again gives its keys there, its comments above the key they touched or
else below what is there before them, and a dotted key reaching a table
that has a header moves into it with its own comments. Tables of an array
of tables never merge.

Options: Indent (up to 8 tabs or spaces, a tab by default), IndentTables
(a level per enclosing table whose header comes first), Align (= and
comments of consecutive keys).

Hardening: Format refuses what Parse refuses, a comment holding a
bidirectional embedding, override or isolate or a line break other than
LF (U+202A-202E, U+2066-2069, U+0085, U+2028, U+2029), which can show a
reader text other than what parses, and a result that would not parse to
the same values. Such a rune in a string or quoted key is written as its
\u escape, by Format and by the encoder alike. A line indents at most 16
levels and aligns within 32 runes of key and 80 of line, so the result
stays within a fixed multiple of its source; table paths sit in a trie,
so placing a key costs its length, not its square.

Marshal writes { a = 1 } and no blank first line, so Format(Marshal(v))
is Marshal(v), which FuzzRoundTrip now checks. FuzzFormat checks, with
the options off and on, that the result reads as the source, formats to
itself, keeps every comment once, holds no CR or trailing blank and stays
within the size bound; CI runs it for 10 s. All 29 TOML files in vif
format idempotently to the same values; vif.toml and keymap.toml are
unchanged byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

---------

Co-authored-by: Claude <noreply@anthropic.com>

lexer.go

@@ -2,6 +2,7 @@ package toml
 
 import (
 	"fmt"
+	"strings"
 	"unicode/utf8"
 )
 
@@ -10,15 +11,13 @@ type Lexer struct {
 	pos   int
 	line  int
 	col   int
-	width int
+	// lean is a parser's lexer: a comment or number carries no Literal, the parser
+	// reading what it needs from Pos and End, so neither allocates
+	lean bool
 }
 
 func NewLexer(input []byte) *Lexer {
-	return &Lexer{
-		input: input,
-		line:  1,
-		col:   0,
-	}
+	return &Lexer{input: input, line: 1}
 }
 
 func (l *Lexer) NextToken() Token {
@@ -35,17 +34,12 @@ func (l *Lexer) next() Token {
 	}
 
 	ch := l.peek()
-
-	if ch == '\n' {
+	switch ch {
+	case '\n':
 		l.advance()
 		return l.newToken(TokenNewline, "\n")
-	}
-
-	if ch == '#' {
+	case '#':
 		return l.readComment()
-	}
-
-	switch ch {
 	case '=':
 		l.advance()
 		return l.newToken(TokenEqual, "=")
@@ -72,27 +66,16 @@ func (l *Lexer) next() Token {
 	}
 
 	// Number: digit or sign+digit
-	if isDigit(ch) {
+	if isDigit(ch) || (ch == '+' || ch == '-') && isDigit(rune(l.byteAt(l.pos+1))) {
 		return l.readNumber()
 	}
 	if ch == '+' {
-		if isDigit(l.peekAt(1)) {
-			return l.readNumber()
-		}
 		// Lone + is invalid TOML
 		l.advance()
 		return l.newToken(TokenError, "unexpected character: +")
 	}
-	if ch == '-' {
-		if isDigit(l.peekAt(1)) {
-			return l.readNumber()
-		}
-		// Bare key can start with hyphen
-		return l.readIdent()
-	}
-
-	// Identifier: alpha or underscore
-	if isAlpha(ch) || ch == '_' {
+	// A bare key may start with a hyphen, not a digit
+	if isAlpha(ch) || ch == '_' || ch == '-' {
 		return l.readIdent()
 	}
 
@@ -101,165 +84,96 @@ func (l *Lexer) next() Token {
 }
 
 func (l *Lexer) readNumber() Token {
-	start := l.pos
-	startCol := l.col
-
-	// Optional sign
-	if l.peek() == '+' || l.peek() == '-' {
-		l.advance()
+	start, startCol := l.pos, l.col
+	i := start
+	if c := l.input[i]; c == '+' || c == '-' {
+		i++
 	}
 
-	// Check radix prefix
-	if l.peek() == '0' {
-		next := l.peekAt(1)
-		switch next {
+	// Radix prefix
+	if l.byteAt(i) == '0' {
+		var digit func(rune) bool
+		var none string
+		switch l.byteAt(i + 1) {
 		case 'x', 'X':
-			l.advance() // '0'
-			l.advance() // 'x'
-			return l.readHex(start, startCol)
+			digit, none = isHexDigit, "invalid hex: no digits after prefix"
 		case 'o', 'O':
-			l.advance()
-			l.advance()
-			return l.readOctal(start, startCol)
+			digit, none = isOctalDigit, "invalid octal: no digits after prefix"
 		case 'b', 'B':
-			l.advance()
-			l.advance()
-			return l.readBinary(start, startCol)
+			digit, none = isBinaryDigit, "invalid binary: no digits after prefix"
+		}
+		if digit != nil {
+			i += 2
+			j := i
+			for j < len(l.input) && digit(rune(l.input[j])) {
+				j++
+			}
+			l.skipTo(j)
+			if j == i {
+				return Token{Type: TokenError, Literal: none, Line: l.line, Col: startCol}
+			}
+			return l.number(TokenInteger, start, startCol)
 		}
 	}
 
-	// Integer part
-	for isDigit(l.peek()) {
-		l.advance()
-	}
-
-	isFloat := false
-
+	typ := TokenInteger
+	i = l.digits(i)
 	// Fractional part: '.' followed by digit
-	if l.peek() == '.' && isDigit(l.peekAt(1)) {
-		isFloat = true
-		l.advance() // consume '.'
-
-		for isDigit(l.peek()) {
-			l.advance()
-		}
-
-		// Multi-dot: another '.' followed by digit is fatal
-		if l.peek() == '.' && isDigit(l.peekAt(1)) {
+	if l.byteAt(i) == '.' && isDigit(rune(l.byteAt(i+1))) {
+		typ, i = TokenFloat, l.digits(i+1)
+		if l.byteAt(i) == '.' && isDigit(rune(l.byteAt(i+1))) {
+			l.skipTo(i)
 			return Token{Type: TokenError, Literal: "invalid number: multiple decimal points", Line: l.line, Col: startCol}
 		}
 	}
-
 	// Exponent: e/E followed by optional sign and digits
-	if l.peek() == 'e' || l.peek() == 'E' {
-		next := l.peekAt(1)
-		validExp := isDigit(next) || ((next == '+' || next == '-') && isDigit(l.peekAt(2)))
-		if validExp {
-			isFloat = true
-			l.advance() // 'e'
-			if l.peek() == '+' || l.peek() == '-' {
-				l.advance()
-			}
-			for isDigit(l.peek()) {
-				l.advance()
-			}
+	if c := l.byteAt(i); c == 'e' || c == 'E' {
+		j := i + 1
+		if c := l.byteAt(j); c == '+' || c == '-' {
+			j++
+		}
+		if isDigit(rune(l.byteAt(j))) {
+			typ, i = TokenFloat, l.digits(j)
 		}
 	}
-
-	lit := string(l.input[start:l.pos])
-
-	// Validate float format: 1.e2 is invalid (no digit after dot before exponent)
-	if isFloat {
-		if err := validateFloat(lit); err != nil {
-			return Token{Type: TokenError, Literal: err.Error(), Line: l.line, Col: startCol}
-		}
-		return Token{Type: TokenFloat, Literal: lit, Line: l.line, Col: startCol}
-	}
-	return Token{Type: TokenInteger, Literal: lit, Line: l.line, Col: startCol}
+	l.skipTo(i)
+	return l.number(typ, start, startCol)
 }
 
-func validateFloat(lit string) error {
-	// Find dot position (if any)
-	dotIdx := -1
-	expIdx := -1
-	for i := 0; i < len(lit); i++ {
-		if lit[i] == '.' {
-			dotIdx = i
-		}
-		if lit[i] == 'e' || lit[i] == 'E' {
-			expIdx = i
-			break
-		}
+func (l *Lexer) number(typ TokenType, start, col int) Token {
+	t := Token{Type: typ, Line: l.line, Col: col}
+	if !l.lean {
+		t.Literal = string(l.input[start:l.pos])
 	}
-
-	if dotIdx >= 0 {
-		// Check digit before dot (ignoring sign)
-		start := 0
-		if lit[0] == '+' || lit[0] == '-' {
-			start = 1
-		}
-		if dotIdx == start {
-			return fmt.Errorf("invalid float %q: no digit before decimal point", lit)
-		}
-
-		// Check digit after dot
-		afterDot := dotIdx + 1
-		endFrac := len(lit)
-		if expIdx > 0 {
-			endFrac = expIdx
-		}
-		if afterDot >= endFrac {
-			return fmt.Errorf("invalid float %q: no digit after decimal point", lit)
-		}
-	}
-	return nil
+	return t
 }
 
-func (l *Lexer) readHex(start, startCol int) Token {
-	if !isHexDigit(l.peek()) {
-		return Token{Type: TokenError, Literal: "invalid hex: no digits after prefix", Line: l.line, Col: startCol}
+// digits is the offset past the decimal digits at i
+func (l *Lexer) digits(i int) int {
+	for i < len(l.input) && isDigit(rune(l.input[i])) {
+		i++
 	}
-	for isHexDigit(l.peek()) {
-		l.advance()
-	}
-	return Token{Type: TokenInteger, Literal: string(l.input[start:l.pos]), Line: l.line, Col: startCol}
-}
-
-func (l *Lexer) readOctal(start, startCol int) Token {
-	if !isOctalDigit(l.peek()) {
-		return Token{Type: TokenError, Literal: "invalid octal: no digits after prefix", Line: l.line, Col: startCol}
-	}
-	for isOctalDigit(l.peek()) {
-		l.advance()
-	}
-	return Token{Type: TokenInteger, Literal: string(l.input[start:l.pos]), Line: l.line, Col: startCol}
-}
-
-func (l *Lexer) readBinary(start, startCol int) Token {
-	if !isBinaryDigit(l.peek()) {
-		return Token{Type: TokenError, Literal: "invalid binary: no digits after prefix", Line: l.line, Col: startCol}
-	}
-	for isBinaryDigit(l.peek()) {
-		l.advance()
-	}
-	return Token{Type: TokenInteger, Literal: string(l.input[start:l.pos]), Line: l.line, Col: startCol}
+	return i
 }
 
 func (l *Lexer) readIdent() Token {
-	start := l.pos
-	for l.pos < len(l.input) {
-		ch := l.peek()
-		if isAlpha(ch) || isDigit(ch) || ch == '_' || ch == '-' {
-			l.advance()
+	start, i := l.pos, l.pos
+	for i < len(l.input) {
+		if c := rune(l.input[i]); isAlpha(c) || isDigit(c) || c == '_' || c == '-' {
+			i++
 		} else {
 			break
 		}
 	}
-	lit := string(l.input[start:l.pos])
-	if lit == "true" || lit == "false" {
-		return l.newToken(TokenBool, lit)
+	l.skipTo(i)
+	switch lit := l.input[start:i]; string(lit) {
+	case "true":
+		return l.newToken(TokenBool, "true")
+	case "false":
+		return l.newToken(TokenBool, "false")
+	default:
+		return l.newToken(TokenIdent, string(lit))
 	}
-	return l.newToken(TokenIdent, lit)
 }
 
 func (l *Lexer) newToken(typ TokenType, literal string) Token {
@@ -272,11 +186,12 @@ func (l *Lexer) newToken(typ TokenType, literal string) Token {
 
 func (l *Lexer) advance() rune {
 	if l.pos >= len(l.input) {
-		l.width = 0
 		return 0
 	}
-	r, w := utf8.DecodeRune(l.input[l.pos:])
-	l.width = w
+	r, w := rune(l.input[l.pos]), 1
+	if r >= utf8.RuneSelf {
+		r, w = utf8.DecodeRune(l.input[l.pos:])
+	}
 	l.pos += w
 	if r == '\n' {
 		l.line++
@@ -287,38 +202,43 @@ func (l *Lexer) advance() rune {
 	return r
 }
 
+// skipTo advances to i over single-byte runes on one line
+func (l *Lexer) skipTo(i int) {
+	l.col += i - l.pos
+	l.pos = i
+}
+
 func (l *Lexer) peek() rune {
 	if l.pos >= len(l.input) {
 		return 0
 	}
+	if c := l.input[l.pos]; c < utf8.RuneSelf {
+		return rune(c)
+	}
 	r, _ := utf8.DecodeRune(l.input[l.pos:])
 	return r
 }
 
-func (l *Lexer) peekAt(n int) rune {
-	pos := l.pos
-	for i := 0; i < n && pos < len(l.input); i++ {
-		_, w := utf8.DecodeRune(l.input[pos:])
-		pos += w
+// byteAt is the input byte at i, 0 past the end
+func (l *Lexer) byteAt(i int) byte {
+	if i < len(l.input) {
+		return l.input[i]
 	}
-	if pos >= len(l.input) {
-		return 0
-	}
-	r, _ := utf8.DecodeRune(l.input[pos:])
-	return r
+	return 0
 }
 
 // skipWhitespace takes a CR only as part of CRLF: a lone one would let a
 // terminal hide what precedes it on the line, so it lexes as an error.
 func (l *Lexer) skipWhitespace() {
-	for l.pos < len(l.input) {
-		ch := l.peek()
-		if ch == ' ' || ch == '\t' || ch == '\r' && l.peekAt(1) == '\n' {
-			l.advance()
+	i := l.pos
+	for i < len(l.input) {
+		if c := l.input[i]; c == ' ' || c == '\t' || c == '\r' && l.byteAt(i+1) == '\n' {
+			i++
 		} else {
 			break
 		}
 	}
+	l.skipTo(i)
 }
 
 // readComment refuses control characters but tab, as TOML does: an ESC or a
@@ -327,21 +247,49 @@ func (l *Lexer) readComment() Token {
 	l.advance() // '#'
 	start := l.pos
 	for l.pos < len(l.input) {
-		ch := l.peek()
-		if ch == '\n' || ch == '\r' && l.peekAt(1) == '\n' {
+		c := l.input[l.pos]
+		if c == '\n' || c == '\r' && l.byteAt(l.pos+1) == '\n' {
 			break
 		}
-		if ch < 0x20 && ch != '\t' || ch == 0x7f {
-			return l.newToken(TokenError, fmt.Sprintf("control character %q in comment", ch))
+		if c < 0x20 && c != '\t' || c == 0x7f {
+			return l.newToken(TokenError, fmt.Sprintf("control character %q in comment", rune(c)))
 		}
-		l.advance()
+		w := 1
+		if c >= utf8.RuneSelf {
+			_, w = utf8.DecodeRune(l.input[l.pos:])
+		}
+		l.pos += w
+		l.col++
+	}
+	if l.lean {
+		return l.newToken(TokenComment, "")
 	}
 	return l.newToken(TokenComment, string(l.input[start:l.pos]))
 }
 
+// readString takes a string without escapes in one allocation, the string itself
 func (l *Lexer) readString() Token {
 	l.advance() // opening quote
-	var result []byte
+	start := l.pos
+	for i := start; i < len(l.input); i++ {
+		c := l.input[i]
+		if c == '"' {
+			l.col += utf8.RuneCount(l.input[start:i]) + 1
+			l.pos = i + 1
+			return l.newToken(TokenString, string(l.input[start:i]))
+		}
+		if c == '\\' || c < 0x20 && c != '\t' || c == 0x7f {
+			break
+		}
+	}
+	return l.readEscaped()
+}
+
+// readEscaped decodes a string's escapes into one buffer sized to its source,
+// which an escape only shortens
+func (l *Lexer) readEscaped() Token {
+	var result strings.Builder
+	result.Grow(l.stringBound())
 	escaped := false
 
 	for l.pos < len(l.input) {
@@ -351,7 +299,7 @@ func (l *Lexer) readString() Token {
 		}
 		if ch == '"' && !escaped {
 			l.advance()
-			return l.newToken(TokenString, string(result))
+			return l.newToken(TokenString, result.String())
 		}
 		if ch == '\\' && !escaped {
 			escaped = true
@@ -361,21 +309,21 @@ func (l *Lexer) readString() Token {
 		if escaped {
 			switch ch {
 			case '"':
-				result = append(result, '"')
+				result.WriteByte('"')
 			case '\\':
-				result = append(result, '\\')
+				result.WriteByte('\\')
 			case 'n':
-				result = append(result, '\n')
+				result.WriteByte('\n')
 			case 't':
-				result = append(result, '\t')
+				result.WriteByte('\t')
 			case 'r':
-				result = append(result, '\r')
+				result.WriteByte('\r')
 			case 'b':
-				result = append(result, '\b')
+				result.WriteByte('\b')
 			case 'f':
-				result = append(result, '\f')
+				result.WriteByte('\f')
 			case 'e':
-				result = append(result, 0x1b)
+				result.WriteByte(0x1b)
 			case 'x', 'u', 'U':
 				n := 2
 				switch ch {
@@ -399,29 +347,38 @@ func (l *Lexer) readString() Token {
 				if !utf8.ValidRune(code) {
 					return l.newToken(TokenError, fmt.Sprintf("invalid unicode escape: U+%X is not a scalar value", code))
 				}
-				var buf [utf8.UTFMax]byte
-				w := utf8.EncodeRune(buf[:], code)
-				result = append(result, buf[:w]...)
+				result.WriteRune(code)
 				escaped = false
 				continue // hex digits already consumed; skip trailing l.advance()
 			default:
 				// Unknown escape, invalid in every TOML version: preserve backslash and full rune
-				result = append(result, '\\')
-				var buf [utf8.UTFMax]byte
-				n := utf8.EncodeRune(buf[:], ch)
-				result = append(result, buf[:n]...)
+				result.WriteByte('\\')
+				result.WriteRune(ch)
 			}
 			escaped = false
 		} else {
-			// Get actual width of current character, don't use stale l.width
 			_, w := utf8.DecodeRune(l.input[l.pos:])
-			result = append(result, l.input[l.pos:l.pos+w]...)
+			result.Write(l.input[l.pos : l.pos+w])
 		}
 		l.advance()
 	}
 	return l.newToken(TokenError, "unterminated string")
 }
 
+// stringBound is the source length of the string at pos, through its closing
+// quote, a line's end or the input's
+func (l *Lexer) stringBound() int {
+	for i := l.pos; i < len(l.input); i++ {
+		switch c := l.input[i]; {
+		case c == '\\':
+			i++
+		case c == '"', c < 0x20 && c != '\t', c == 0x7f:
+			return i - l.pos
+		}
+	}
+	return len(l.input) - l.pos
+}
+
 func isDigit(r rune) bool {
 	return r >= '0' && r <= '9'
 }