yaml

README

go-yaml

A YAML library for Go, forked from the excellent goccy/go-yaml.

[!WARNING] Early days. The module path has changed, the API will change, and there is no release yet. If you want a stable YAML library today, use goccy/go-yaml upstream — it is well maintained and this fork exists for reasons specific to go-openapi, not because anything is wrong with it.

Why fork?

go-openapi needs a YAML library that a tooling consumer can build on, and it needs three things that no Go YAML library currently offers together:

• Low-level access — a token and AST surface with accurate positions, not a Marshal/Unmarshal facade. We drive editor and TUI tooling (syntax colouring, diagnostics, JSON-pointer navigation) over OpenAPI documents, so we need to know where every construct is, not just what it means.
• Streaming, with a bounded memory footprint. OpenAPI documents get large. At the fork point the whole input was materialised as []rune, every token was retained, and nothing could be emitted before the entire document had been parsed — an AST cost roughly 32× the source. The []rune is gone and a parse now makes 84% fewer allocations, but the parser still reads a whole document; streaming is the work in progress.
• Conformance good enough to project YAML onto JSON semantics faithfully, measured against the YAML Test Suite rather than asserted.

goccy/go-yaml is the only Go YAML library whose architecture exposes the machinery to build that on. That is why we started from it rather than from anything else.

See ANALYSIS-go-openapi.md for the measurements behind all of the above.

Is it a hard fork?

Yes. It started as a soft fork and it is no longer one.

We set out to keep every fix that cost upstream nothing as an isolated commit, ready to send as a pull request. Four things ended that:

• The root API. 133 exported entries, with Path sitting at the top level and the encoder holding a *Path. Ours declares four functions. Everything else moved to codec, ast, parser, errors and expressions.
• Comment manipulation wired into the top-level API, where it belongs to the AST.
• The parser materializes the whole document. The AST cost roughly 32× the source, which makes streaming impossible and the memory churn structural. Fixing it means rewriting the scanner and the token, not tuning.
• The parser API only ever addresses a fully parsed document. There is no shape in it for reading a stream.

Alongside those, the correctness round moved the YAML Test Suite from 88.3% to 100% of scored cases, the whole tree was relinted to go-openapi standards, and the wasi playground was removed. Upstream is barely maintained, and the distance is now too large for any of this to be retrofitted.

Licensing is unchanged. This repository stays under goccy/go-yaml's MIT license (see LICENSE) and claims no separate copyright. Credit for almost all of the original code belongs upstream. Third-party components are recorded in NOTICE.

Relationship to go-yaml/yaml

None — and that is inherited from upstream. This library was written from scratch by @goccy, not ported from libyaml, which is precisely what makes its internals approachable enough to fork. If you are coming from gopkg.in/yaml.v3 or go.yaml.in/yaml/v3, the upstream README's rationale still applies:

• the source is written in Go style rather than transliterated from C
• higher coverage of the YAML Test Suite
• errors carry source positions, which makes validation diagnostics possible
• comments and anchors survive a round trip, so reversible transformation is achievable
• an API that exposes Scanner and Parser, not only Encoder/Decoder

Installation

go get github.com/go-openapi/go-yaml

Requires Go 1.25 or later. We support the two most recent stable Go minor versions.

Packages

The root package holds Marshal, Unmarshal, ToJSON and FromJSON — the four calls that take no option. Everything else lives a layer down. The imports run one way: no package in this table imports one listed below it.

package	what it holds	imports
token	a token and its position	—
parser/scanner	reads a source into tokens	token
ast	the document as a tree	token
printer	draws a document, or a line of it under an error	ast
errors	Error, the kind it carries, and FormatError	printer
parser	builds a tree from a token stream	parser/scanner, errors
codec	Encoder, Decoder, the 25 options, MapSlice, RawMessage, the comment types, the marshaler interfaces	parser
expressions	Path and PathString, to navigate a document by path	codec
github.com/go-openapi/go-yaml	Marshal, Unmarshal, ToJSON, FromJSON	codec

Synopsis

1. Simple Encode/Decode

An interface like go-yaml/yaml, using reflect:

var v struct {
	A int
	B string
}
v.A = 1
v.B = "hello"
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes)) // "a: 1\nb: hello\n"

	yml := `
%YAML 1.2
---
a: 1
b: c
`
var v struct {
	A int
	B string
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

To control marshal/unmarshal behavior, you can use the yaml tag:

	yml := `---
foo: 1
bar: c
`
var v struct {
	A int    `yaml:"foo"`
	B string `yaml:"bar"`
}
if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
	//...
}

For convenience, the json tag is also accepted. Note that not all options from the json tag have significance when parsing YAML documents. If both tags exist, the yaml tag takes precedence.

For custom marshal/unmarshaling, implement one of the two variants declared in codec. codec.Marshaler/codec.Unmarshaler return and take the YAML text as []byte, like encoding/json; codec.GoYAMLMarshaler/codec.GoYAMLUnmarshaler return and take another Go value, like gopkg.in/yaml.v2.

Semantically both are the same, but they differ in performance. Because indentation matters in YAML, a valid YAML fragment returned by a marshaler cannot simply be spliced into the parent container's serialized form — so when we receive []byte from a codec.Marshaler, we must decode it once to work out how to place it in context. With a codec.GoYAMLMarshaler, that decode is skipped. If you repeatedly marshal complex objects, the latter is always better; for a config file read once, the former is easier to write.

2. Reference elements declared in another file

Given a directory -- testdata here -- holding an anchor.yml file:

a: &a
  b: 1
  c: hello

If the codec.ReferenceDirs("testdata") option is passed to codec.Decoder, the decoder looks for anchor definitions in the YAML files under that directory:

buf := bytes.NewBufferString("a: *a\n")
dec := codec.NewDecoder(buf, codec.ReferenceDirs("testdata"))
var v struct {
	A struct {
		B int
		C string
	}
}
if err := dec.Decode(&v); err != nil {
	//...
}
fmt.Printf("%+v\n", v) // {A:{B:1 C:hello}}

3. Encode with Anchor and Alias

3.1. Explicitly declared anchor and alias names

Declare them as a struct tag. If the value specified for an anchor is a pointer and the same address is found again, the value is automatically emitted as an alias. If an explicit alias name is specified, an error is raised when its value differs from the value specified in the anchor.

type T struct {
  A int
  B string
}
var v struct {
  C *T `yaml:"c,anchor=x"`
  D *T `yaml:"d,alias=x"`
}
v.C = &T{A: 1, B: "hello"}
v.D = v.C
bytes, err := yaml.Marshal(v)
if err != nil {
  panic(err)
}
fmt.Println(string(bytes))
/*
c: &x
  a: 1
  b: hello
d: *x
*/

3.2. Implicitly declared anchor and alias names

Without an explicit anchor name, the default is strings.ToLower($FieldName).

type T struct {
	I int
	S string
}
var v struct {
	A *T `yaml:"a,anchor"`
	B *T `yaml:"b,anchor"`
	C *T `yaml:"c"`
	D *T `yaml:"d"`
}
v.A = &T{I: 1, S: "hello"}
v.B = &T{I: 2, S: "world"}
v.C = v.A // C has the same pointer address as A
v.D = v.B // D has the same pointer address as B
bytes, err := yaml.Marshal(v)
if err != nil {
	//...
}
fmt.Println(string(bytes))
/*
a: &a
  i: 1
  s: hello
b: &b
  i: 2
  s: world
c: *a
d: *b
*/

3.3 Merge key and alias

A merge key with an alias (<<: *alias) can be used by embedding a structure with the inline,alias tag.

type Person struct {
	*Person `yaml:",omitempty,inline,alias"` // embed Person type for default value
	Name    string `yaml:",omitempty"`
	Age     int    `yaml:",omitempty"`
}
defaultPerson := &Person{
	Name: "John Smith",
	Age:  20,
}
people := []*Person{
	{
		Person: defaultPerson, // assign default value
		Name:   "Ken",         // override Name property
		Age:    10,            // override Age property
	},
	{
		Person: defaultPerson, // assign default value only
	},
}
var doc struct {
	Default *Person   `yaml:"default,anchor"`
	People  []*Person `yaml:"people"`
}
doc.Default = defaultPerson
doc.People = people
bytes, err := yaml.Marshal(doc)
if err != nil {
	//...
}
fmt.Println(string(bytes))
/*
default: &default
  name: John Smith
  age: 20
people:
- <<: *default
  name: Ken
  age: 10
- <<: *default
*/

4. Pretty formatted errors

Errors produced during parsing carry the location of the problem in the source document, and can optionally be colorized. Use errors.FormatError from github.com/go-openapi/go-yaml/errors to control both, which accepts two boolean values.

5. Use YAMLPath

yml := `
store:
  book:
    - author: john
      price: 10
    - author: ken
      price: 12
  bicycle:
    color: red
    price: 19.95
`
path, err := expressions.PathString("$.store.book[*].author")
if err != nil {
  //...
}
var authors []string
if err := path.Read(strings.NewReader(yml), &authors); err != nil {
  //...
}
fmt.Println(authors)
// [john ken]

5.1 Print a customized error with the YAML source

package main

import (
  "fmt"

  "github.com/go-openapi/go-yaml"
  "github.com/go-openapi/go-yaml/expressions"
)

func main() {
  yml := `
a: 1
b: "hello"
`
  var v struct {
    A int
    B string
  }
  if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
    panic(err)
  }
  if v.A != 2 {
    // output error with YAML source
    path, err := expressions.PathString("$.a")
    if err != nil {
      panic(err)
    }
    source, err := path.AnnotateSource([]byte(yml), true)
    if err != nil {
      panic(err)
    }
    fmt.Printf("a value expected 2 but actual %d:\n%s\n", v.A, string(source))
  }
}

Playground

Upstream hosts a playground that visualizes how the library processes YAML text, which is useful for debugging and for filing issues: https://goccy.github.io/go-yaml

Note that it runs upstream's code, so it will not reflect changes made in this fork.

For developers

See .github/CONTRIBUTING.md.

The library itself has no runtime dependencies, and that is a property worth keeping: go-openapi/core depends on this module, so anything we add here propagates.

Tests use go-openapi/testify/v2, which is itself dependency-free — so the only entry in go.mod is a test dependency that never reaches your binary. Everything that needs more than that lives under internal/, in modules of its own listed in go.work:

internal/analysis	the reproducible measurements behind ANALYSIS-go-openapi.md
internal/benchmarks	comparisons against other YAML libraries
internal/testintegration	tests needing third-party libraries

go test ./...          # the library
go test work ./...     # the library and every module in the workspace

Credits

This library was created by Masaaki Goshima (@goccy) and is developed upstream at github.com/goccy/go-yaml. If this fork is useful to you, the credit for almost all of it belongs there — and upstream is looking for sponsors.

License

MIT — see LICENSE. Third-party components are recorded in NOTICE.

Documentation

Overview

Package yaml reads and writes YAML documents.

Marshal and Unmarshal cover the ordinary case, as in the standard library's encoding packages, and take no option. Anything else lives a layer down:

• codec holds Encoder and Decoder, the twenty-five options that steer them, MapSlice, RawMessage, and the comment types. Use codec.NewDecoder to read a stream and codec.UnmarshalWithOptions to pass an option.
• github.com/go-openapi/go-yaml/ast holds the document as a tree.
• github.com/go-openapi/go-yaml/parser builds that tree from a source, and github.com/go-openapi/go-yaml/parser/scanner hands it the tokens.
• github.com/go-openapi/go-yaml/errors declares the failure every one of them reports, with the position it happened at.
• github.com/go-openapi/go-yaml/expressions navigates a document by path.

Tags

A tag is read by the URI it names rather than by the shorthand it was written with, so "!!int", "!<tag:yaml.org,2002:int>" and "!e!int" under "%TAG !e! tag:yaml.org,2002:" are one tag. The expansion is on the node, at github.com/go-openapi/go-yaml/ast.TagNode.URI.

The seven tags of the YAML 1.2 core schema (§10.2) are resolved: !!null, !!bool, !!int, !!float, !!str, !!seq and !!map.

Five more come from the 1.1 type repository at https://yaml.org/type and are resolved as well, under either version:

• !!binary decodes base64 into []byte, and a text base64 cannot read is an error.
• !!merge is the "<<" key, whose mapping's entries are folded into the one holding it.
• !!omap has to stand on a sequence and decodes as one, in the order it was written. There is no ordered-map type behind it; use codec.UseOrderedMap to get codec.MapSlice for every mapping.
• !!set has to stand on a mapping and decodes as a map with nil values.
• !!timestamp decodes to a time.Time, and a text no format reads is an error. The formats are the ones yaml.org/type/timestamp.html spells.

!!timestamp takes two rules, since YAML 1.2 has no timestamp of its own and other libraries differ. An explicit !!timestamp resolves whatever version the document declares, because a tag names a URI and is not resolution. An untagged "2001-12-14" is a string in both versions, and only a Go field of type time.Time asks for the conversion -- go.yaml.in/yaml/v3 reads it as a time.Time and gopkg.in/yaml.v2 as a string. github.com/go-openapi/go-yaml/parser.WithYAMLVersion and a "%YAML" directive select what an untagged plain scalar resolves to, and neither changes what a tag means.

Three tags of the 1.1 repository are not resolved: !!pairs, !!value and !!yaml. They are parsed and carried on the node like any other tag, and the value under them stands as it was written. So does every tag outside these fifteen -- a local "!thing", a handle a "%TAG" line declared, another namespace -- with one rule: a tag nothing resolves leaves its scalar as text, digits and all, so "!thing 12" is the string "12".

Index

• func FromJSON(bytes []byte) ([]byte, error)
• func Marshal(v interface{}) ([]byte, error)
• func ToJSON(bytes []byte) ([]byte, error)
• func Unmarshal(data []byte, v interface{}) error

Examples

• Marshal
• Marshal (ExplicitAnchorAlias)
• Marshal (ImplicitAnchorAlias)
• Marshal (Node)
• Unmarshal (JSONTags)
• Unmarshal (YAMLTags)

Functions

func FromJSON

func FromJSON(bytes []byte) ([]byte, error)

FromJSON converts a JSON document to the YAML that holds the same values.

func Marshal

func Marshal(v interface{}) ([]byte, error)

Marshal serializes v into a YAML document.

See codec.Marshal for how a Go value is mapped onto YAML and what the struct tags mean.

Example

package main

import (
	"bytes"
	"fmt"
	"strconv"

	"github.com/go-openapi/go-yaml"
	"github.com/go-openapi/go-yaml/codec"
)

type SlowMarshaler struct {
	A string
	B int
}
type FastMarshaler struct {
	A string
	B int
}
type TextMarshaler int64
type TextMarshalerContainer struct {
	Field TextMarshaler `yaml:"field"`
}

func (v SlowMarshaler) MarshalYAML() ([]byte, error) {
	var buf bytes.Buffer
	buf.WriteString("tags:\n")
	buf.WriteString("- slow-marshaler\n")
	buf.WriteString("a: " + v.A + "\n")
	buf.WriteString("b: " + strconv.FormatInt(int64(v.B), 10) + "\n")
	return buf.Bytes(), nil
}

func (v FastMarshaler) MarshalYAML() (interface{}, error) {
	return codec.MapSlice{
		{Key: "tags", Value: []string{"fast-marshaler"}},
		{Key: "a", Value: v.A},
		{Key: "b", Value: v.B},
	}, nil
}

func (t TextMarshaler) MarshalText() ([]byte, error) {
	return []byte(strconv.FormatInt(int64(t), 8)), nil
}

func main() {
	var slow SlowMarshaler
	slow.A = "Hello slow poke"
	slow.B = 100
	buf, err := yaml.Marshal(slow)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(string(buf))

	var fast FastMarshaler
	fast.A = "Hello speed demon"
	fast.B = 100
	buf, err = yaml.Marshal(fast)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(string(buf))

	text := TextMarshalerContainer{
		Field: 11,
	}
	buf, err = yaml.Marshal(text)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(string(buf))
}

Output:
tags:
- slow-marshaler
a: Hello slow poke
b: 100

tags:
- fast-marshaler
a: Hello speed demon
b: 100

field: "13"

Example (ExplicitAnchorAlias)

package main

import (
	"fmt"

	"github.com/go-openapi/go-yaml"
)

func main() {
	type T struct {
		A int
		B string
	}
	var v struct {
		C *T `yaml:"c,anchor=x"`
		D *T `yaml:"d,alias=x"`
	}
	v.C = &T{A: 1, B: "hello"}
	v.D = v.C
	bytes, err := yaml.Marshal(v)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(bytes))
}

Output:
c: &x
  a: 1
  b: hello
d: *x

Example (ImplicitAnchorAlias)

package main

import (
	"fmt"

	"github.com/go-openapi/go-yaml"
)

func main() {
	type T struct {
		I int
		S string
	}
	var v struct {
		A *T `yaml:"a,anchor"`
		B *T `yaml:"b,anchor"`
		C *T `yaml:"c"`
		D *T `yaml:"d"`
	}
	v.A = &T{I: 1, S: "hello"}
	v.B = &T{I: 2, S: "world"}
	v.C = v.A // C has same pointer address to A
	v.D = v.B // D has same pointer address to B
	bytes, err := yaml.Marshal(v)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(bytes))
}

Output:
a: &a
  i: 1
  s: hello
b: &b
  i: 2
  s: world
c: *a
d: *b

Example (Node)

package main

import (
	"fmt"

	"github.com/go-openapi/go-yaml"
	"github.com/go-openapi/go-yaml/ast"
	"github.com/go-openapi/go-yaml/codec"
)

func main() {
	type T struct {
		Text ast.Node `yaml:"text"`
	}
	stringNode, err := codec.ValueToNode("node example")
	if err != nil {
		panic(err)
	}
	bytes, err := yaml.Marshal(T{Text: stringNode})
	if err != nil {
		panic(err)
	}
	fmt.Println(string(bytes))
}

Output:
text: node example

func ToJSON

func ToJSON(bytes []byte) ([]byte, error)

ToJSON converts a YAML document to the JSON that holds the same values.

func Unmarshal

func Unmarshal(data []byte, v interface{}) error

Unmarshal decodes the YAML document data into the value pointed to by v.

See codec.Unmarshal for how a YAML document is mapped onto a Go value.

Example (JSONTags)

package main

import (
	"fmt"
	"log"

	"github.com/go-openapi/go-yaml"
)

func main() {
	yml := `---
foo: 1
bar: c
`
	var v struct {
		A int    `json:"foo"`
		B string `json:"bar"`
	}
	if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
		log.Fatal(err)
	}
	fmt.Println(v.A)
	fmt.Println(v.B)
}

Output:
1
c

Example (YAMLTags)

package main

import (
	"fmt"
	"log"

	"github.com/go-openapi/go-yaml"
)

func main() {
	yml := `---
foo: 1
bar: c
A: 2
B: d
`
	var v struct {
		A int    `yaml:"foo" json:"A"`
		B string `yaml:"bar" json:"B"`
	}
	if err := yaml.Unmarshal([]byte(yml), &v); err != nil {
		log.Fatal(err)
	}
	fmt.Println(v.A)
	fmt.Println(v.B)
}

Output:
1
c