v0.3.0 format a document, each comment with its owner; fewer allocations on the hot path
Test / test (push) Canceled after 0s
Test / test (push) Canceled after 0s
* Benchmarks for the parse, decode, encode and document paths A command's payload from text to its struct, a settings file with comment blocks, a scenario file of arrays of inline tables across lines, decoding and encoding that scenario, and ParseDocument: the readings the changes that follow are measured by. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc * Fewer allocations on the parse, decode and encode paths The parser's lexer drops what it never reads: a comment's text and a number's literal, which the parser parses from the input bytes in place. A string without escapes is one allocation, one with escapes is decoded into a buffer sized to its source. The lexer scans ASCII bytewise, the parser holds its lexer by value, keeps key paths on the stack, gathers an array's elements in a reused buffer and allocates the array once at its length, and holds the first few frozen inline tables in place; an inline table that is an array element is not held, as no key can reach it. Decode, Marshal and Unmapped read a struct's fields from one cache per type instead of splitting every tag on every call. Decode reuses one key and one element per map; Marshal sets a map's values into one array, and writes keys, headers and numbers into its buffer without concatenating. Tokens, positions, trees, error messages, encoded bytes and decoded values are unchanged: a differential fuzz against the previous sources ran 1.7M inputs over the lexer and parser and 9.5M over Marshal and Decode without a difference. benchstat, 8 runs each, Xeon 2.1 GHz: payload, parse and decode 3.31us -> 2.30us 31 -> 14 allocs 1.41 -> 1.09 KiB parse payload 2.40us -> 1.89us 24 -> 12 allocs parse settings 10.25us -> 6.98us 68 -> 32 allocs parse scenario 536us -> 391us 4959 -> 2970 allocs decode scenario 225us -> 110us 1863 -> 776 allocs marshal scenario 397us -> 252us 2326 -> 976 allocs ParseDocument settings 21.7us -> 16.5us 128 -> 85 allocs Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc * Document: a comment goes with what fewer blank lines part it from ParseDocument lays a document out as statements, runs of statements no blank line parts, and the comment groups between them. A group belongs to the side fewer blank lines part it from, the side below on a tie: right above a statement it is that statement's own; right below a run, or apart from it, it is the run's. Between two runs several groups split at the widest gap. The file's start counts as no blank line and its end as half of one, so a comment at the top with a blank line under it, or at the end with one above it, is the file's, and one touching the last statement stays with it. Delete takes a statement with its own comments; a run going whole takes its comments too and one blank run beside it, so no comment is left orphaned and no gap doubles. Set puts a new table above the file's closing comments and a first root key above the first table's comments, a blank line apart. What v0.2.0 kept by position (the comment above a dropped key, a dropped table's heading) now goes with its owner. FuzzDocument checks that a delete leaves whole lines the document had, in order, and that adding a key loses none; it ran 12.6M inputs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc * Format: one layout for a document, every comment kept with its owner Format, and Formatter for its options, writes a document spaced as Marshal writes one: one space around = and after a comma, [1, 2] and { a = 1 }, LF line ends, a value's lines a level deeper than the line its bracket opens on, and a trailing comma exactly when the closing bracket has a line of its own. Keys, scalars and comments keep their spelling. Comments stay with what they belong to by the rule Document reads them with, and the result's blank lines encode it again: a run of blank lines becomes one, or two below a comment group that belongs above it, so a second Format reads every comment as the first did and writes the same bytes. Each table is written once, at its first header: a header opening it again gives its keys there, its comments above the key they touched or else below what is there before them, and a dotted key reaching a table that has a header moves into it with its own comments. Tables of an array of tables never merge. Options: Indent (up to 8 tabs or spaces, a tab by default), IndentTables (a level per enclosing table whose header comes first), Align (= and comments of consecutive keys). Hardening: Format refuses what Parse refuses, a comment holding a bidirectional embedding, override or isolate or a line break other than LF (U+202A-202E, U+2066-2069, U+0085, U+2028, U+2029), which can show a reader text other than what parses, and a result that would not parse to the same values. Such a rune in a string or quoted key is written as its \u escape, by Format and by the encoder alike. A line indents at most 16 levels and aligns within 32 runes of key and 80 of line, so the result stays within a fixed multiple of its source; table paths sit in a trie, so placing a key costs its length, not its square. Marshal writes { a = 1 } and no blank first line, so Format(Marshal(v)) is Marshal(v), which FuzzRoundTrip now checks. FuzzFormat checks, with the options off and on, that the result reads as the source, formats to itself, keeps every comment once, holds no CR or trailing blank and stays within the size bound; CI runs it for 10 s. All 29 TOML files in vif format idempotently to the same values; vif.toml and keymap.toml are unchanged byte for byte. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01ThctvSRwt5kV1pVMKtsdWc --------- Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
LixenandClaude Opus 5.5 authored and GitHub committed
1 parent
e2712e32a2
commit
170e71f10a
12 files changed
+1809
-440
No files matched your search
+188
@@ -0,0 +1,188 @@
|
||||
package toml
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"fmt"
|
||||
"reflect"
|
||||
"slices"
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func formats(t *testing.T, f Formatter, in, want string) {
|
||||
t.Helper()
|
||||
got, err := f.Format([]byte(in))
|
||||
if err != nil {
|
||||
t.Fatalf("format %q: %v", in, err)
|
||||
}
|
||||
if string(got) != want {
|
||||
t.Fatalf("format of:\n%s\ngot:\n%s\nwant:\n%s", in, got, want)
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatSpacesAsMarshal: one space around = and after a comma, none inside
|
||||
// an array's brackets, one inside an inline table's; a value's lines a tab deeper
|
||||
// than the line it opens on, a trailing comma exactly when the closing bracket
|
||||
// has its own line; a blank line run kept as one; LF line ends, no trailing blank.
|
||||
func TestFormatSpacesAsMarshal(t *testing.T) {
|
||||
formats(t, Formatter{}, " a=[1,2, 3]\r\nb = {x=1,y = {z=[ ]}, w = { }}\r\n\r\n\r\n"+
|
||||
"c = [\n 1, # one\n 2\n]\nd = [ [1,\n2], 3, ] # note \n[ t . \"u v\" ]\n[[ arr ]]\nk = {\na = 1 }\n\n",
|
||||
"a = [1, 2, 3]\nb = { x = 1, y = { z = [] }, w = {} }\n\n"+
|
||||
"c = [\n\t1, # one\n\t2,\n]\nd = [[1,\n\t2], 3] # note\n[t.\"u v\"]\n[[arr]]\nk = {\n\ta = 1 }\n")
|
||||
}
|
||||
|
||||
// TestFormatKeepsCommentsWithTheirOwners: a comment stays with what fewer blank
|
||||
// lines part it from as a Document reads it, and the output's blank lines say so
|
||||
// again: two follow a group belonging to what is above it, after a blank line.
|
||||
// Comments inside a value keep their place among its elements.
|
||||
func TestFormatKeepsCommentsWithTheirOwners(t *testing.T) {
|
||||
// The request's example: a dotted key under [section2] is section2's
|
||||
formats(t, Formatter{}, "# comment s1\n[section1]\npart1=1 # comment s1p1\n\n#comment s2\n[section2]\n"+
|
||||
"# comment s2p2\npart1=1\n\n# comment s1p2\nsection1.part2 = 5\n",
|
||||
"# comment s1\n[section1]\npart1 = 1 # comment s1p1\n\n#comment s2\n[section2]\n"+
|
||||
"# comment s2p2\npart1 = 1\n\n# comment s1p2\nsection1.part2 = 5\n")
|
||||
formats(t, Formatter{}, "# file\n\n# also file\n\n\n[a]\nx = 1\n\n# about a\n\n\n# about b\n\n[b]\n"+
|
||||
"y = [ # open\n\n # first\n 1,\n\n # second\n 2 # two\n , 3,\n # last\n\n]\n\n# end\n",
|
||||
"# file\n\n# also file\n\n\n[a]\nx = 1\n\n# about a\n\n\n# about b\n\n[b]\n"+
|
||||
"y = [ # open\n\t# first\n\t1,\n\n\t# second\n\t2, # two\n\t3,\n\t# last\n]\n\n# end\n")
|
||||
}
|
||||
|
||||
// TestFormatWritesATableOnce: a header opening a table again gives its keys to
|
||||
// the first header, a blank line apart, its comments above them; a dotted key
|
||||
// reaching a table that has a header moves there, spelled from it, its own
|
||||
// comments along. Arrays of tables, and tables under them, stay as written.
|
||||
func TestFormatWritesATableOnce(t *testing.T) {
|
||||
formats(t, Formatter{}, "# comment s1\n[section1]\npart1=1 # comment s1p1\n\n#comment s2\n[section2]\n"+
|
||||
"# comment s2p2\npart1=1\n\n# comment s1p2\n[section1]\npart2 = 5\n",
|
||||
"# comment s1\n[section1]\npart1 = 1 # comment s1p1\n\n# comment s1p2\npart2 = 5\n\n"+
|
||||
"#comment s2\n[section2]\n# comment s2p2\npart1 = 1\n")
|
||||
formats(t, Formatter{}, "# about part2\nsection1.part2 = 5\na.b.c = 1\n\n[a.b]\nd = 2\n[section1]\npart1 = 1\n[a]\nb.e = 3\n",
|
||||
"[a.b]\nd = 2\nc = 1\ne = 3\n[section1]\npart1 = 1\n# about part2\npart2 = 5\n[a]\n")
|
||||
formats(t, Formatter{}, "[[s]]\nx = 1\n[s.t]\ny = 1\n[[s]]\nx = 2\n[s.t]\ny = 2\n",
|
||||
"[[s]]\nx = 1\n[s.t]\ny = 1\n[[s]]\nx = 2\n[s.t]\ny = 2\n")
|
||||
}
|
||||
|
||||
// TestFormatterOptions: Indent sets a level's characters, IndentTables indents a
|
||||
// table a level per table it is under whose header comes first, Align lines up
|
||||
// the = and the comments of keys on consecutive lines, a comment or blank line
|
||||
// ending the run.
|
||||
func TestFormatterOptions(t *testing.T) {
|
||||
formats(t, Formatter{Indent: " ", IndentTables: true},
|
||||
"[servers]\na = 1\n[servers.alpha]\nports = [\n80,\n]\n[servers.alpha.deep]\nx = 1\n[other.deep]\ny = 2\n",
|
||||
"[servers]\na = 1\n [servers.alpha]\n ports = [\n 80,\n ]\n [servers.alpha.deep]\n x = 1\n[other.deep]\ny = 2\n")
|
||||
formats(t, Formatter{Align: true}, "[play]\nsfx = true # on\nmusic = true\nsfx_volume = 0.5 # half\n# lead\nmouse = true # m\n\nhud = false\nwait = true # w\n",
|
||||
"[play]\nsfx = true # on\nmusic = true\nsfx_volume = 0.5 # half\n# lead\nmouse = true # m\n\nhud = false\nwait = true # w\n")
|
||||
}
|
||||
|
||||
// TestFormatGrowsBoundedly: past 16 levels a line indents no deeper, and a key or
|
||||
// line too long to align stands alone, so however deep or long its source, the
|
||||
// result stays within a fixed multiple of it: a two-byte line such as "]" takes
|
||||
// 16 tabs and a comma at most
|
||||
func TestFormatGrowsBoundedly(t *testing.T) {
|
||||
deep := "a = " + strings.Repeat("[\n", 999) + "1" + strings.Repeat("\n]", 999) + "\n"
|
||||
var long strings.Builder
|
||||
long.WriteString("\"" + strings.Repeat("k", 4096) + "\" = 1 # c\n")
|
||||
for i := range 1000 {
|
||||
fmt.Fprintf(&long, "k%d = %d # c\n", i, i)
|
||||
}
|
||||
for _, doc := range []string{deep, long.String()} {
|
||||
out, err := Formatter{Align: true}.Format([]byte(doc))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if len(out) > 10*len(doc) {
|
||||
t.Errorf("%d bytes grew to %d", len(doc), len(out))
|
||||
}
|
||||
if bytes.Contains(out, []byte(strings.Repeat("\t", 17))) {
|
||||
t.Error("a line indents past 16 levels")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// TestFormatRefusesWhatItCannotKeep: a document Parse refuses, a comment holding
|
||||
// a rune that can hide the text after it, and an indent that is not up to 8
|
||||
// blanks are refused; such a rune in a string or quoted key is written escaped.
|
||||
func TestFormatRefusesWhatItCannotKeep(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
f Formatter
|
||||
doc string
|
||||
}{
|
||||
{Formatter{}, "a = \n"},
|
||||
{Formatter{}, "a = 1 # \u202e reversed\n"},
|
||||
{Formatter{}, "# \u2066isolate\na = 1\n"},
|
||||
{Formatter{}, "# next\u2028line\na = 1\n"},
|
||||
{Formatter{Indent: "# "}, "a = [\n1,\n]\n"},
|
||||
{Formatter{Indent: strings.Repeat(" ", 9)}, "a = 1\n"},
|
||||
} {
|
||||
if out, err := c.f.Format([]byte(c.doc)); err == nil {
|
||||
t.Errorf("format of %q taken:\n%s", c.doc, out)
|
||||
}
|
||||
}
|
||||
formats(t, Formatter{}, "s = \"a\u202eb\"\n\"k\u2066\" = \"\\\u202e\"\n", "s = \"a\\u202Eb\"\n\"k\\u2066\" = \"\\\\\\u202E\"\n")
|
||||
}
|
||||
|
||||
// FuzzFormat: whatever Parse takes Format writes, unless a comment holds a rune
|
||||
// that can hide text, with every option: the result reads as the document does,
|
||||
// formats to itself, keeps every comment once and has no CR or trailing blank.
|
||||
func FuzzFormat(f *testing.F) {
|
||||
for _, s := range []string{
|
||||
settingsDoc,
|
||||
"# a\n\n# b\n\n\n[t]\nk = 1\n# t's\n\n\n# u\n[u]\nv = [ # o\n\n1, # one\n# c\n2\n,3]\n\n# end\n",
|
||||
"x.y = 1\n[x]\nz = 2\n[w]\n[x] # again\nq = { a.b = 1, c = [ {d = 1} ] }\n",
|
||||
"[[s]]\n[s.t]\na = 1\n[[s]]\n[s.t]\nb = 2\n",
|
||||
"k = \"\\u00e9 \\\\ \\\" \\t\"\r\n\"q k\" = \"\u202e\"\r\n",
|
||||
"a = [[[[1]]],\n[[2,\n3]]]\nb = {\nc = {\nd = 1,\n},\n}\n",
|
||||
"\n\n# only\n\n# comments\n\n",
|
||||
"a=1",
|
||||
"[a]\n[a.b]\n[a]\nc = 1\n",
|
||||
"[A]\n\n#\n\n[A]",
|
||||
"[a]\n[a.A]\n[a]#",
|
||||
"[a]\n[a.A]\n[a.A]#",
|
||||
"[\"a b\".c]\nx = 1\n[\"a b\"]\nc.y = 2\n",
|
||||
"[A]\n\n#\n\n\n[A0]\n[A]#",
|
||||
"# p\n\n[a]\nx = 1\n# t\n\n# u\n\n\n[b]\n# lb\n[a] # n\n# la\ny = 2\n\n# v\n\n\n[a.c]\n[a]\n\n# w\n",
|
||||
} {
|
||||
f.Add([]byte(s))
|
||||
}
|
||||
f.Fuzz(func(t *testing.T, data []byte) {
|
||||
tree, err := NewParser(data).Parse()
|
||||
if err != nil || hiddenInComments(data) != nil {
|
||||
return
|
||||
}
|
||||
for _, fm := range []Formatter{{}, {Indent: " ", IndentTables: true, Align: true}} {
|
||||
out, err := fm.Format(data)
|
||||
if err != nil {
|
||||
t.Fatalf("%+v refused %q: %v", fm, data, err)
|
||||
}
|
||||
if got, _ := NewParser(out).Parse(); !reflect.DeepEqual(got, tree) {
|
||||
t.Fatalf("%+v changed the values:\n%s", fm, out)
|
||||
}
|
||||
if again, err := fm.Format(out); err != nil || !bytes.Equal(again, out) {
|
||||
t.Fatalf("%+v does not format to itself (%v):\n%s\nthen:\n%s", fm, err, out, again)
|
||||
}
|
||||
if a, b := comments(data), comments(out); !slices.Equal(a, b) {
|
||||
t.Fatalf("%+v comments %q became %q:\n%s", fm, a, b, out)
|
||||
}
|
||||
if bytes.Contains(out, []byte("\r")) || bytes.Contains(out, []byte(" \n")) || bytes.Contains(out, []byte("\t\n")) ||
|
||||
bytes.Contains(out, []byte("\n\n\n\n")) || len(out) > 0 && out[len(out)-1] != '\n' {
|
||||
t.Fatalf("%+v wrote a stray blank:\n%q", fm, out)
|
||||
}
|
||||
if len(out) > 48*len(data)+64 {
|
||||
t.Fatalf("%+v grew %d bytes to %d", fm, len(data), len(out))
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// comments lists a document's comments, trailing blanks dropped, sorted
|
||||
func comments(data []byte) []string {
|
||||
var out []string
|
||||
lx := NewLexer(data)
|
||||
for t := lx.NextToken(); t.Type != TokenEOF; t = lx.NextToken() {
|
||||
if t.Type == TokenComment {
|
||||
out = append(out, strings.TrimRight(t.Literal, " \t"))
|
||||
}
|
||||
}
|
||||
slices.Sort(out)
|
||||
return out
|
||||
}
|
||||
Reference in new issue
Block a user