fyaml

README

fyaml

📖 Documentation • 🚀 Installation • 📚 Usage Guide • 🐛 Issues

fyaml compiles a directory tree of YAML or JSON files into a single deterministic document (with optional comment and key order preservation).

It exists to solve a common, recurring problem:

Some tools expect configuration to live in a single YAML file, even as that file grows to thousands of lines.

fyaml lets you work with structure and small files, while still producing the single file those tools expect.

Quick Example

Given a directory structure:

config/
  entities/
    item1.yml
    item2.yml

entities/item1.yml:

entity:
  id: example1
  attributes:
    name: sample name
    tags:
      - tag1
      - tag2

entities/item2.yml:

entity:
  id: example2
  attributes:
    name: another name
    tags: []

Run:

fyaml                    # Pack current directory
fyaml config/            # Pack specific directory
fyaml -o output.yml      # Pack current directory to file
fyaml -m preserve        # Preserve authored order and comments

Produces:

entities:
  item1:
    entity:
      id: example1
      attributes:
        name: sample name
        tags:
          - tag1
          - tag2
  item2:
    entity:
      id: example2
      attributes:
        name: another name
        tags: []

---

What problem this solves

Many systems are designed around a single YAML configuration file:

• CI/CD platforms
• API specifications (such as OpenAPI)
• Tools that do not support includes or composition

As configurations grow, this becomes difficult to manage:

• Files reach thousands of lines
• Merge conflicts become common
• Reviews get harder, not easier
• Structure is implied by indentation and comments
• Confidence in changes drops over time

fyaml solves this by separating how configuration is authored (as files and directories) from how it is consumed (as a single document).

You organize configuration as directories and files. fyaml compiles that structure into the single document the target system expects.

There is no logic, templating, or execution model involved.

---

What fyaml does

fyaml is intentionally limited in scope to keep output predictable and diffs trustworthy.

• Organize as you want - Split large configs into small, focused files organized in directories
• Predictable output - Deterministic output makes diffs meaningful. Choose between canonical mode (sorted keys, no comments) or preserve mode (authored order and comments). See Output Modes below.
• No surprises - Pure structure compilation with no logic, templating, or execution model
• Build-time tool - Runs as a build step, producing the single file your tools expect

For technical details on how directory structure maps to YAML, see How It Works.

Output Modes

fyaml supports two output modes:

• Canonical mode (default) - Keys are sorted alphabetically and comments are removed. Sorted keys make diffs more readable.
• Preserve mode - Maintains the authored key order and preserves comments. Useful when you want to keep documentation in comments or preserve the structure from source files.

Both modes are deterministic. Use --mode (or -m) to select a mode:

fyaml --mode canonical    # Default: sorted keys, no comments
fyaml -m preserve         # Preserve order and comments

For more details, see Usage Guide - Output Modes.

---

When to use this

Use fyaml when:

• You need to produce a single YAML or JSON file
• The configuration is large enough to benefit from structure
• Readable diffs and predictable output matter
• You want organization without adding logic

fyaml is not a good fit if you need:

• conditionals
• loops
• variable resolution
• runtime behavior

Those concerns are better handled by other tools.

Installation

Quick Install (Linux/macOS)

curl -sSL https://raw.githubusercontent.com/jksmth/fyaml/main/install.sh | bash

Note: This downloads and executes a script. For verification steps, see Verification below.

From Source (Go)

go install github.com/jksmth/fyaml@latest

Docker

Run directly:

docker run --rm -v $(pwd):/workspace ghcr.io/jksmth/fyaml:latest pack /workspace/examples/basic

For more installation options (pre-built binaries, Windows, multi-stage Docker), see the Installation Guide.

Verification

fyaml releases are signed with cosign using keyless signing.

For verification steps (binaries, Docker images, SBOMs), see Installation - Verification.

Usage

Basic Usage

# Pack current directory to stdout
fyaml

# Pack specific directory
fyaml /path/to/config

# Write to file
fyaml /path/to/config -o output.yml

# Output as JSON
fyaml /path/to/config --format json -o output.json

# Preserve authored order and comments
fyaml /path/to/config --mode preserve -o output.yml

# Check if output matches existing file
fyaml /path/to/config -o output.yml --check

# Verbose output (show files being processed)
fyaml -v /path/to/config

# Pack directory with conflicting name (e.g., directory named "pack")
fyaml --dir pack

# Show version
fyaml --version

Note: fyaml pack [DIR] is an alias for fyaml [DIR] and works identically for backward compatibility. Both fyaml version and fyaml --version (or fyaml -V) work to show version information.

More examples and patterns

• See the Usage Guide for:
  • Directory structure rules - Basic structure, root files, @ files and directories
  • File includes - Using !include, !include-text, and <<include()>>
  • Limitations - File content requirements, YAML anchors, multi-document files
• See the Examples and the examples/ directory for runnable examples.
• See the Command Reference for:
  • Commands - Main command, pack alias, version
  • Flags reference - All available flags and options including --dir for avoiding subcommand conflicts

Using as a Library

fyaml can be used as a Go library in your own projects. Import the package:

import "github.com/jksmth/fyaml"

Basic Example

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/jksmth/fyaml"
)

func main() {
    result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
        Dir: "./config",
    })
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(string(result))
}

With Options

result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
    Dir:             "./config",
    Format:          fyaml.FormatJSON,
    Mode:            fyaml.ModePreserve,
    MergeStrategy:   fyaml.MergeDeep,
    EnableIncludes:  true,
    EnableAnchors:   true,
    ConvertBooleans: true,
    Chroot:          ".", // Optional: allow includes from project root
    Indent:          4,
})

With Logging

import (
    "context"
    "os"
    "github.com/jksmth/fyaml"
)

logger := fyaml.NewLogger(os.Stderr, true)

result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
    Dir:    "./config",
    Logger: logger,
})

API Reference

See pkg.go.dev for complete API documentation.

Exit Codes

• 0 - Success
• 1 - Pack or IO error
• 2 - --check mismatch (output differs from expected)

About

fyaml implements the FYAML specification (also available at github.com/CircleCI-Public/fyaml).

It's a small, focused tool that:

• Works with any YAML-based system
• Produces deterministic output
• Has a minimal surface area focused on one task
• Does not implement templating, variables, or conditionals

Need templating or variable substitution? Use external tools like envsubst alongside fyaml. This keeps fyaml focused on structure compilation while allowing you to use specialized tools for templating. See Usage Guide - Integration with Templating for examples.

Extensions: fyaml includes optional extensions (like JSON support) that enhance functionality while maintaining spec compliance. See the Extensions section for details.

Implementation Note: In canonical mode (default), fyaml sorts all map keys alphabetically to ensure deterministic output. Preserve mode maintains authored key order and comments. Both modes are deterministic. The FYAML specification does not specify key ordering, so this is an implementation choice that provides reproducibility.

Extensions

fyaml includes the following extensions beyond the FYAML specification. These features are opt-in and do not affect spec-compliant behavior.

• JSON Support - Accept .json files and output JSON format. See Usage Guide - Output Format and Command Reference - --format/-f for details.
• File Includes - Use !include, !include-text, and <<include()>> directives to include content from other files. Use --chroot to share includes across multiple configuration directories. See Usage Guide - File Includes for complete documentation.
• Cross-File Anchors - Reference anchors defined in one file from another file. See Usage Guide - YAML Anchors and Aliases for complete documentation.
• Boolean Conversion - Convert YAML 1.1 booleans (on/off, yes/no) to YAML 1.2 (true/false). See Usage Guide - Converting on/off and yes/no to true/false for details.
• @ Directory Support - Directories starting with @ merge into parent map, similar to @ files. See Usage Guide - @ Directories for details.

For complete documentation on all extensions, see the Usage Guide and Command Reference.

Links

• Documentation
• Examples
• API Reference
• GitHub Repository

License

MIT License - see LICENSE for details.

Attribution

Portions of this code are derived from the CircleCI CLI, which is also licensed under the MIT License. See NOTICE for details.

Documentation

Overview

Package fyaml provides a programmatic API for compiling directory-structured YAML/JSON files into a single document.

fyaml compiles a directory tree of YAML or JSON files into a single deterministic document. Directory names become map keys, file names (without extension) become nested keys, and files starting with @ merge their contents into the parent directory.

Example:

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir: "./config",
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(result))
}

Error Handling:

The package defines sentinel errors for programmatic error handling:

• ErrDirectoryRequired
• ErrInvalidFormat
• ErrInvalidMode
• ErrInvalidMergeStrategy
• ErrInvalidIndent
• ErrCheckMismatch

Use errors.Is() to check for specific errors:

result, err := fyaml.Pack(ctx, opts)
if err != nil {
	if errors.Is(err, fyaml.ErrInvalidFormat) {
		// Handle invalid format
	}
}

err := fyaml.Check(generated, expected, fyaml.CheckOptions{Format: fyaml.FormatYAML})
if err != nil {
	if errors.Is(err, fyaml.ErrCheckMismatch) {
		// Handle output mismatch
	}
}

For more examples, see the examples in the test files.

Index

• Variables
• func Check(generated []byte, expected []byte, opts CheckOptions) error
• func Pack(ctx context.Context, opts PackOptions) ([]byte, error)
• type CheckOptions
• type Format
  • func ParseFormat(s string) (Format, error)
• type Logger
  • func NewLogger(w io.Writer, verbose bool) Logger
  • func NopLogger() Logger
• type MergeStrategy
  • func ParseMergeStrategy(s string) (MergeStrategy, error)
• type Mode
  • func ParseMode(s string) (Mode, error)
• type PackOptions

Examples

• Check
• Check (EmptyJSON)
• Check (Mismatch)
• Format
• MergeStrategy
• Mode
• Pack
• Pack (ErrorHandling)
• Pack (Minimal)
• Pack (WithLogger)
• Pack (WithOptions)
• ParseFormat
• ParseMergeStrategy
• ParseMode

Variables

var (
	// ErrDirectoryRequired is returned when Dir is empty or not provided.
	ErrDirectoryRequired = errors.New("directory is required")

	// ErrInvalidFormat is returned when Format is not FormatYAML or FormatJSON.
	ErrInvalidFormat = errors.New("invalid format")

	// ErrInvalidMode is returned when Mode is not ModeCanonical or ModePreserve.
	ErrInvalidMode = errors.New("invalid mode")

	// ErrInvalidMergeStrategy is returned when MergeStrategy is not MergeShallow or MergeDeep.
	ErrInvalidMergeStrategy = errors.New("invalid merge strategy")

	// ErrInvalidIndent is returned when Indent is less than 1.
	ErrInvalidIndent = errors.New("invalid indent")

	// ErrCheckMismatch is returned when Check() finds differences between
	// generated output and expected content.
	ErrCheckMismatch = errors.New("output mismatch")
)

Sentinel errors for programmatic error handling. Use errors.Is() to check for specific errors:

result, err := fyaml.Pack(ctx, opts)
if err != nil {
	if errors.Is(err, fyaml.ErrInvalidFormat) {
		// Handle invalid format error
	}
}

Functions

func Check

func Check(generated []byte, expected []byte, opts CheckOptions) error

Check compares generated output with expected content using exact byte comparison. Returns ErrCheckMismatch if contents don't match. Whitespace differences will be detected as mismatches. opts.Format is used to normalize empty expected content to match format-specific empty output. opts defaults to FormatYAML if Format is empty.

Example

package main

import (
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	generated := []byte("key: value\n")
	expected := []byte("key: value\n")

	err := fyaml.Check(generated, expected, fyaml.CheckOptions{Format: fyaml.FormatYAML})
	if err != nil {
		log.Fatal(err)
	}
	// Output matches
}

Example (EmptyJSON)

package main

import (
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	// Empty JSON output is normalized to "null\n"
	generated := []byte("null\n")
	expected := []byte{} // Empty

	err := fyaml.Check(generated, expected, fyaml.CheckOptions{Format: fyaml.FormatJSON})
	if err != nil {
		log.Fatal(err)
	}
	// Empty expected is normalized, so it matches
}

Example (Mismatch)

package main

import (
	"errors"
	"fmt"

	"github.com/jksmth/fyaml"
)

func main() {
	generated := []byte("key: value\n")
	expected := []byte("key: different\n")

	err := fyaml.Check(generated, expected, fyaml.CheckOptions{Format: fyaml.FormatYAML})
	if errors.Is(err, fyaml.ErrCheckMismatch) {
		fmt.Println("Output mismatch detected")
	}
}

Output:
Output mismatch detected

func Pack

func Pack(ctx context.Context, opts PackOptions) ([]byte, error)

Pack compiles a directory of YAML/JSON files into a single document.

Pack is safe for concurrent use by multiple goroutines, provided that each call uses a separate PackOptions instance. If multiple goroutines share the same Logger instance, log output may interleave (this does not affect correctness, only log formatting).

The context can be used to cancel the operation. If the context is canceled, Pack will return an error wrapping context.Canceled or context.DeadlineExceeded.

PackOptions.Dir is required. All other options have sensible defaults:

• Format defaults to FormatYAML
• Mode defaults to ModeCanonical
• MergeStrategy defaults to MergeShallow
• Indent defaults to 2
• Logger defaults to a no-op logger if nil

Returns the packed document as bytes, or an error if packing fails.

Example

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	// Pack a directory with default options (YAML output, canonical mode)
	result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir: "./config",
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(result))
}

Example (ErrorHandling)

package main

import (
	"context"
	"errors"
	"fmt"

	"github.com/jksmth/fyaml"
)

func main() {
	_, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir: "",
	})
	if err != nil {
		if errors.Is(err, fyaml.ErrDirectoryRequired) {
			// Handle directory required error
			fmt.Println("Directory is required")
		} else if errors.Is(err, fyaml.ErrInvalidFormat) {
			// Handle invalid format error
			fmt.Println("Invalid format specified")
		}
	}
}

Example (Minimal)

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	// Minimal usage - only directory required, all other options use defaults
	result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir: "./config",
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(result))
}

Example (WithLogger)

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	"github.com/jksmth/fyaml"
)

func main() {
	// Pack with a logger for verbose output
	logger := fyaml.NewLogger(os.Stderr, true)

	result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir:    "./config",
		Logger: logger,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(result))
}

Example (WithOptions)

package main

import (
	"context"
	"fmt"
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	// Pack with all options specified
	result, err := fyaml.Pack(context.Background(), fyaml.PackOptions{
		Dir:             "./config",
		Format:          fyaml.FormatJSON,
		Mode:            fyaml.ModePreserve,
		MergeStrategy:   fyaml.MergeDeep,
		EnableIncludes:  true,
		ConvertBooleans: true,
		Indent:          4,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(string(result))
}

Types

type CheckOptions

type CheckOptions struct {
	// Format specifies the format used for normalization.
	// Defaults to FormatYAML if empty.
	// Used to normalize empty expected content to match format-specific empty output.
	Format Format
}

CheckOptions configures how Check compares content. Zero value provides default behavior (exact byte comparison, YAML format).

type Format

type Format string

Format specifies the output format for the packed document.

Example

package main

import (
	"github.com/jksmth/fyaml"
)

func main() {
	// Use FormatYAML for YAML output (default)
	_ = fyaml.FormatYAML

	// Use FormatJSON for JSON output
	_ = fyaml.FormatJSON
}

const (
	// FormatYAML outputs YAML format (default).
	FormatYAML Format = "yaml"
	// FormatJSON outputs JSON format.
	FormatJSON Format = "json"
)

func ParseFormat

func ParseFormat(s string) (Format, error)

ParseFormat parses a format string and returns the corresponding Format. Returns an error if the format is invalid.

Example

package main

import (
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	format, err := fyaml.ParseFormat("yaml")
	if err != nil {
		log.Fatal(err)
	}
	_ = format // Use format
}

type Logger

type Logger interface {
	// Debugf logs verbose/debug information (shown when verbose enabled)
	Debugf(format string, args ...interface{})
	// Warnf logs warnings (always shown)
	Warnf(format string, args ...interface{})
}

Logger defines the logging interface for fyaml. All output is written to the configured io.Writer (typically os.Stderr).

func NewLogger

func NewLogger(w io.Writer, verbose bool) Logger

NewLogger creates a logger that writes to w. If verbose is true, Debugf messages are shown. Warnf messages are always shown.

func NopLogger

func NopLogger() Logger

NopLogger returns a no-op logger that discards all output.

type MergeStrategy

type MergeStrategy string

MergeStrategy controls how maps are merged when multiple files contribute to the same key.

Example

package main

import (
	"github.com/jksmth/fyaml"
)

func main() {
	// Use MergeShallow for "last wins" behavior (default)
	_ = fyaml.MergeShallow

	// Use MergeDeep for recursive nested map merging
	_ = fyaml.MergeDeep
}

const (
	// MergeShallow uses "last wins" behavior - later values completely replace earlier ones (default).
	MergeShallow MergeStrategy = "shallow"
	// MergeDeep recursively merges nested maps, only replacing values at the leaf level.
	MergeDeep MergeStrategy = "deep"
)

func ParseMergeStrategy

func ParseMergeStrategy(s string) (MergeStrategy, error)

ParseMergeStrategy parses a merge strategy string and returns the corresponding MergeStrategy. Returns an error if the strategy is invalid.

Example

package main

import (
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	strategy, err := fyaml.ParseMergeStrategy("deep")
	if err != nil {
		log.Fatal(err)
	}
	_ = strategy // Use strategy
}

type Mode

type Mode string

Mode controls the output behavior of the packed document.

Example

package main

import (
	"github.com/jksmth/fyaml"
)

func main() {
	// Use ModeCanonical for sorted keys, no comments (default)
	_ = fyaml.ModeCanonical

	// Use ModePreserve for authored order and comments
	_ = fyaml.ModePreserve
}

const (
	// ModeCanonical produces canonical output with sorted keys and no comments (default).
	ModeCanonical Mode = "canonical"
	// ModePreserve preserves authored key order and comments.
	ModePreserve Mode = "preserve"
)

func ParseMode

func ParseMode(s string) (Mode, error)

ParseMode parses a mode string and returns the corresponding Mode. Returns an error if the mode is invalid.

Example

package main

import (
	"log"

	"github.com/jksmth/fyaml"
)

func main() {
	mode, err := fyaml.ParseMode("preserve")
	if err != nil {
		log.Fatal(err)
	}
	_ = mode // Use mode
}

type PackOptions

type PackOptions struct {
	// Dir is the directory to pack (required).
	Dir string

	// Format specifies the output format. Defaults to FormatYAML if empty.
	Format Format

	// Mode controls output behavior. Defaults to ModeCanonical if empty.
	Mode Mode

	// MergeStrategy controls merge behavior. Defaults to MergeShallow if empty.
	MergeStrategy MergeStrategy

	// EnableIncludes processes !include, !include-text, and <<include()>> directives.
	EnableIncludes bool

	// ConvertBooleans converts unquoted YAML 1.1 booleans (on/off, yes/no) to YAML 1.2 (true/false).
	ConvertBooleans bool

	// EnableAnchors enables anchor references across different files.
	// When enabled, anchors defined in one file can be referenced as aliases in another file.
	EnableAnchors bool

	// Indent is the number of spaces for indentation. Defaults to 2 if zero.
	Indent int

	// Chroot is the security/confinement boundary (like chroot).
	// If empty, defaults to Dir (current behavior).
	// If set, allows includes and other operations from outside Dir but within Chroot.
	// Does NOT affect what gets packed - that's still controlled by Dir.
	// Does NOT affect path resolution - relative paths in !include tags remain
	// relative to the file containing the tag (for portability).
	Chroot string

	// Logger is an optional logger for verbose output. If nil, no logging is performed.
	Logger Logger
}

PackOptions configures how a directory is packed into a single document.