lixen/toml
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.3.0 format a document, each comment with its owner; fewer allocations on the hot path
Test / test (push) Canceled after 0s
Details

Browse Source 
* Benchmarks for the parse, decode, encode and document paths

A command's payload from text to its struct, a settings file with comment
blocks, a scenario file of arrays of inline tables across lines, decoding
and encoding that scenario, and ParseDocument: the readings the changes
that follow are measured by.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Fewer allocations on the parse, decode and encode paths

The parser's lexer drops what it never reads: a comment's text and a
number's literal, which the parser parses from the input bytes in place.
A string without escapes is one allocation, one with escapes is decoded
into a buffer sized to its source. The lexer scans ASCII bytewise, the
parser holds its lexer by value, keeps key paths on the stack, gathers an
array's elements in a reused buffer and allocates the array once at its
length, and holds the first few frozen inline tables in place; an inline
table that is an array element is not held, as no key can reach it.

Decode, Marshal and Unmapped read a struct's fields from one cache per
type instead of splitting every tag on every call. Decode reuses one key
and one element per map; Marshal sets a map's values into one array, and
writes keys, headers and numbers into its buffer without concatenating.

Tokens, positions, trees, error messages, encoded bytes and decoded
values are unchanged: a differential fuzz against the previous sources
ran 1.7M inputs over the lexer and parser and 9.5M over Marshal and
Decode without a difference.

benchstat, 8 runs each, Xeon 2.1 GHz:
  payload, parse and decode  3.31us -> 2.30us   31 -> 14 allocs  1.41 -> 1.09 KiB
  parse payload              2.40us -> 1.89us   24 -> 12 allocs
  parse settings            10.25us -> 6.98us   68 -> 32 allocs
  parse scenario             536us -> 391us   4959 -> 2970 allocs
  decode scenario            225us -> 110us   1863 -> 776 allocs
  marshal scenario           397us -> 252us   2326 -> 976 allocs
  ParseDocument settings    21.7us -> 16.5us   128 -> 85 allocs

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Document: a comment goes with what fewer blank lines part it from

ParseDocument lays a document out as statements, runs of statements no
blank line parts, and the comment groups between them. A group belongs
to the side fewer blank lines part it from, the side below on a tie:
right above a statement it is that statement's own; right below a run, or
apart from it, it is the run's. Between two runs several groups split at
the widest gap. The file's start counts as no blank line and its end as
half of one, so a comment at the top with a blank line under it, or at
the end with one above it, is the file's, and one touching the last
statement stays with it.

Delete takes a statement with its own comments; a run going whole takes
its comments too and one blank run beside it, so no comment is left
orphaned and no gap doubles. Set puts a new table above the file's
closing comments and a first root key above the first table's comments,
a blank line apart. What v0.2.0 kept by position (the comment above a
dropped key, a dropped table's heading) now goes with its owner.

FuzzDocument checks that a delete leaves whole lines the document had,
in order, and that adding a key loses none; it ran 12.6M inputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

* Format: one layout for a document, every comment kept with its owner

Format, and Formatter for its options, writes a document spaced as
Marshal writes one: one space around = and after a comma, [1, 2] and
{ a = 1 }, LF line ends, a value's lines a level deeper than the line its
bracket opens on, and a trailing comma exactly when the closing bracket
has a line of its own. Keys, scalars and comments keep their spelling.

Comments stay with what they belong to by the rule Document reads them
with, and the result's blank lines encode it again: a run of blank lines
becomes one, or two below a comment group that belongs above it, so a
second Format reads every comment as the first did and writes the same
bytes. Each table is written once, at its first header: a header opening
it again gives its keys there, its comments above the key they touched or
else below what is there before them, and a dotted key reaching a table
that has a header moves into it with its own comments. Tables of an array
of tables never merge.

Options: Indent (up to 8 tabs or spaces, a tab by default), IndentTables
(a level per enclosing table whose header comes first), Align (= and
comments of consecutive keys).

Hardening: Format refuses what Parse refuses, a comment holding a
bidirectional embedding, override or isolate or a line break other than
LF (U+202A-202E, U+2066-2069, U+0085, U+2028, U+2029), which can show a
reader text other than what parses, and a result that would not parse to
the same values. Such a rune in a string or quoted key is written as its
\u escape, by Format and by the encoder alike. A line indents at most 16
levels and aligns within 32 runes of key and 80 of line, so the result
stays within a fixed multiple of its source; table paths sit in a trie,
so placing a key costs its length, not its square.

Marshal writes { a = 1 } and no blank first line, so Format(Marshal(v))
is Marshal(v), which FuzzRoundTrip now checks. FuzzFormat checks, with
the options off and on, that the result reads as the source, formats to
itself, keeps every comment once, holds no CR or trailing blank and stays
within the size bound; CI runs it for 10 s. All 29 TOML files in vif
format idempotently to the same values; vif.toml and keymap.toml are
unchanged byte for byte.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc

---------

Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
LixenandClaude Opus 5.5 authored and GitHub committed 2026-10-25 04:49:22 -04:00
1 parent e2712e32a2
commit 170e71f10a
12 files changed
+1809 -440
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

document.go
+183 -50
View File
@@ -2,7 +2,6 @@ package toml
import (
"bytes"
"cmp"
"fmt"
"reflect"
"slices"
@@ -31,23 +30,35 @@ func (k Key) String() string {
func (k Key) under(p Key) bool { return len(k) >= len(p) && slices.Equal(k[:len(p)], p) }
// Document is a parsed document that keeps its bytes. An edit rewrites only the
// statements it names, so comments, blank lines, order and the spelling of every
// other value stay as written, and the result is parsed again before it stands.
// statements it names, with the comments that belong to them, so blank lines,
// order, every other comment and the spelling of every other value stay as
// written, and the result is parsed again before it stands.
type Document struct {
data []byte
tree map[string]any
stmts []statement
units []unit
pre int // where the comments heading the file end, 0 when none
foot int // where those closing it start, -1 when none
}
// statement is a table header or a key and its value, as the lines it spans: from
// the start of its line through its newline, a comment after it on that line
// included. A key is its path relative to scope, the header it sits under.
// included. lead is where the comment lines right above it, which are its own,
// start. A key is its path relative to scope, the header it sits under.
type statement struct {
start, end int
scope, key Key
header bool
array bool // an [[array]] header
value [2]int // a key's value bytes
start, end, lead int
scope, key Key
header bool
array bool // an [[array]] header
value [2]int // a key's value bytes
}
// unit is a run of statements no blank line parts, from the first comment line
// that belongs to the run through the last, whole lines: stmts[first:last]
type unit struct {
start, end int
first, last int
}
func (s statement) path() Key {
@@ -64,12 +75,94 @@ func ParseDocument(data []byte) (*Document, error) {
if err != nil {
return nil, err
}
return &Document{data: data, tree: tree, stmts: statements(data)}, nil
d := layout(data)
d.tree = tree
return &d, nil
}
// statements splits a document the parser accepted into its statements
func statements(data []byte) []statement {
lx := NewLexer(data)
// item is a statement, or a group of comment lines no blank line parts, with the
// blank lines between it and the item before it or the file's start
type item struct {
start, end int
stmt int // its index in stmts, -1 for comments
gap int
}
// layout splits a document the parser accepted into statements and units and
// gives each comment group to the side fewer blank lines part it from, below on
// a tie
func layout(data []byte) Document {
d := Document{data: data, foot: -1}
items := d.scan()
para := make([]int, len(items)) // a statement's unit
open := false // no blank line since the last unit's last item
for k, it := range items {
open = open && it.gap == 0
if it.stmt < 0 {
continue
}
if !open {
d.units = append(d.units, unit{start: it.start, first: it.stmt})
}
para[k], open = len(d.units)-1, true
d.units[para[k]].end, d.units[para[k]].last = it.end, it.stmt+1
}
for i := 0; i < len(items); {
if items[i].stmt >= 0 {
i++
continue
}
j := i
for j < len(items) && items[j].stmt < 0 {
j++
}
split := placeComments(items, i, j)
for _, c := range items[i:split] {
if i == 0 {
d.pre = c.end
} else {
d.units[para[i-1]].end = max(d.units[para[i-1]].end, c.end)
}
}
for _, c := range items[split:j] {
if j < len(items) {
d.units[para[j]].start = min(d.units[para[j]].start, c.start)
} else if d.foot < 0 {
d.foot = c.start
}
}
if split < j && j < len(items) && items[j].gap == 0 {
d.stmts[items[j].stmt].lead = items[j-1].start
}
i = j
}
return d
}
// placeComments splits the comment groups items[i:j] at the widest run of blank
// lines around them, the first of equals: those above it go with what is above,
// the rest with what is below. The file's start counts as no blank line and its
// end as half of one, so a comment touching a statement stays with it.
func placeComments(items []item, i, j int) int {
split, widest := i, -1
for k := i; k <= j; k++ {
w := 1 // the file's end
switch {
case k == 0:
w = 0
case k < len(items):
w = 2 * items[k].gap
}
if w > widest {
split, widest = k, w
}
}
return split
}
// scan lexes data into its items, appending d.stmts
func (d *Document) scan() (items []item) {
lx := Lexer{input: d.data, line: 1, lean: true}
t := lx.NextToken()
key := func() Key {
var k Key
@@ -93,12 +186,26 @@ func statements(data []byte) []statement {
return end
}
var (
out []statement
scope Key
blank int
)
for t.Type != TokenEOF {
s := statement{start: bytes.LastIndexByte(data[:t.Pos], '\n') + 1, scope: scope}
start := bytes.LastIndexByte(d.data[:t.Pos], '\n') + 1
s := statement{start: start, lead: start, scope: scope}
switch t.Type {
case TokenNewline:
blank++
t = lx.NextToken()
continue
case TokenComment:
end := lineEnd()
if n := len(items); n > 0 && items[n-1].stmt < 0 && blank == 0 {
items[n-1].end = end
} else {
items = append(items, item{start: start, end: end, stmt: -1, gap: blank})
}
blank = 0
continue
case TokenLBracket:
s.header, t = true, lx.NextToken()
if t.Type == TokenLBracket {
@@ -109,7 +216,7 @@ func statements(data []byte) []statement {
t = lx.NextToken()
}
s.scope, scope = nil, s.key
case TokenIdent, TokenString:
default: // a key
s.key = key()
t = lx.NextToken() // past =
s.value[0] = t.Pos
@@ -127,14 +234,13 @@ func statements(data []byte) []statement {
break
}
}
default: // a blank or comment line
t = lx.NextToken()
continue
}
s.end = lineEnd()
out = append(out, s)
items = append(items, item{start: s.start, end: s.end, stmt: len(d.stmts), gap: blank})
d.stmts = append(d.stmts, s)
blank = 0
}
return out
return items
}
// Bytes is the document as it now reads; the caller must not change it
@@ -174,13 +280,9 @@ func unmapped(data any, t reflect.Type, at Key, seen map[string]Key, depth int)
if !ok {
return
}
fields := make(map[string]reflect.Type, t.NumField())
for i := range t.NumField() {
f := t.Field(i)
name, _, _ := strings.Cut(f.Tag.Get("toml"), ",")
if f.IsExported() && name != "-" {
fields[cmp.Or(name, f.Name)] = f.Type
}
fields := make(map[string]reflect.Type)
for _, f := range fieldsOf(t) {
fields[f.key] = t.Field(f.index).Type
}
for k, v := range m {
key := append(slices.Clip(at), k)
@@ -210,25 +312,55 @@ func unmapped(data any, t reflect.Type, at Key, seen map[string]Key, depth int)
}
}
// Delete removes key and every key under it: each statement setting one and each
// table header naming one, in every table of an array of tables, every other line
// kept. It reports whether anything was removed; a key inside an inline value, a
// table or array written as a value, cannot be edited and fails.
// Delete removes key and every key under it, each statement setting one and each
// header naming one, in every table of an array of tables, with the comments that
// are theirs; a run of statements going whole takes its comments and one blank run
// beside it. It reports whether anything was removed; a key inside an inline value,
// a table or array written as a value, cannot be edited and fails.
func (d *Document) Delete(key Key) (bool, error) {
var out []byte
at := 0
for _, s := range d.stmts {
gone := make([]bool, len(d.stmts))
some := false
for i, s := range d.stmts {
p := s.path()
if !s.header && len(p) < len(key) && key.under(p) && d.inline(s) {
return false, fmt.Errorf("key %s is inside the value of %s", key, p)
}
if p.under(key) {
out, at = append(out, d.data[at:s.start]...), s.end
}
gone[i] = p.under(key)
some = some || gone[i]
}
if at == 0 {
if !some {
return false, nil
}
whole := func(u unit) bool { return !slices.Contains(gone[u.first:u.last], false) }
var out []byte
at := 0
cut := func(from, to int) { out, at = append(out, d.data[at:from]...), to }
for k := 0; k < len(d.units); k++ {
u := d.units[k]
if !whole(u) {
for i := u.first; i < u.last; i++ {
if gone[i] {
cut(d.stmts[i].lead, d.stmts[i].end)
}
}
continue
}
j := k
for j+1 < len(d.units) && whole(d.units[j+1]) {
j++
}
switch {
case j+1 < len(d.units):
cut(u.start, d.units[j+1].start)
case d.foot >= 0:
cut(u.start, d.foot)
case k > 0:
cut(d.units[k-1].end, len(d.data))
default:
cut(d.pre, len(d.data))
}
k = j
}
return true, d.replace(append(out, d.data[at:]...))
}
@@ -238,11 +370,11 @@ func (d *Document) inline(s statement) bool {
return c == '{' || c == '['
}
// Set gives key the value v, encoded as Marshal writes a value. A key the document
// sets has its value replaced, the rest of its line kept; a key it lacks goes after
// the last key of its table, else after the table's header, else in a table
// appended to the document. A key under an array of tables or inside an inline
// value fails, and so does a value the document then would not parse with.
// Set gives key the value v, encoded as Marshal writes one. A key the document sets
// has its value replaced, the rest of its line kept; one it lacks goes after the
// last key of its table, else after its header, else in a table appended above the
// file's closing comments, a first root key above the first table and its comments.
// A key under an array of tables or an inline value fails, as does an unparsable edit.
func (d *Document) Set(key Key, v any) error {
if len(key) == 0 || v == nil {
return fmt.Errorf("a key and a value are needed")
@@ -273,21 +405,22 @@ func (d *Document) Set(key Key, v any) error {
}
}
}
head := ""
head, foot := "", "\n"
switch {
case after >= 0:
case len(parent) == 0: // a root key goes before the first header
case len(parent) == 0 && len(d.units) > 0: // a blank line keeps the table's comments its own
after, foot = d.units[0].start, "\n\n"
case len(parent) == 0:
after = len(d.data)
if i := slices.IndexFunc(d.stmts, func(s statement) bool { return s.header }); i >= 0 {
after = d.stmts[i].start
}
case d.foot >= 0:
after, scope, head, foot = d.foot, parent, "["+parent.String()+"]\n", "\n\n"
default:
after, scope, head = len(d.data), parent, "["+parent.String()+"]\n"
if after > 0 {
head = "\n" + head
}
}
line := slices.Concat([]byte(head+key[len(scope):].String()+" = "), value, []byte("\n"))
line := slices.Concat([]byte(head+key[len(scope):].String()+" = "), value, []byte(foot))
if after > 0 && d.data[after-1] != '\n' {
line = slices.Concat([]byte("\n"), line)
}