walle

package module
v0.1.11 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 9, 2026 License: MIT Imports: 8 Imported by: 0

README

walle

A Moonshot AI flavored Json schema validator.

Entry Point

The main entry point is walle.go, which provides two core APIs:

  • ParseSchema: Parses input JSON schema string and creates a walle schema instance
  • Schema.Validate: Validates the schema with optional configurations
    • Validation levels:
      • ultra / default: Most comprehensive validation including potentially "harmless" checks (e.g., no duplicate items).
      • strict: The most permissive level required by Moonshot AI server for efficient structured generation.
      • lite: Skips a subset of rules that strict enforces.
      • loose: Skips schema validation in Schema.Validate and relies more on model capabilities.

Usage

cli

go install github.com/moonshotai/walle/cmd/walle@latest

walle -schema '{"type": "object"}' -level strict
walle -schema-file your_schema.json
go package
import "github.com/moonshotai/walle"

// Define your JSON schema
schemaStr := `{
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"}
    },
    "required": ["name"]
}`

// Create a schema instance
schema, err := walle.ParseSchema(schemaStr)
...

// Validate the schema with default options
err = schema.Validate()
...

// Canonical JSON string
canonicalJSON, warnErr := schema.Canonical()

// Validate the schema with custom options
err = schema.Validate(
    walle.WithValidateLevel(walle.ValidateLevelStrict),
)
...
Python

The Python interface is packaged as walle. Release wheels include libwalle.so, so users can import it directly after installing the wheel.

For local development, build the shared library before building or installing the package:

cd python/c-shared
./build.sh
cd ../..
python -m pip wheel . -w dist --no-deps

Example:

from walle import WalleValidator

# Initialize validator
validator = WalleValidator()

# Validate a schema
schema = '''
{
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"}
    },
    "required": ["name"]
}
'''
validator.validate_schema(schema)

# Canonical schema
canonical_json, warning = validator.canonical_schema(schema)

# With custom configuration
config = {
    "validateLevel": "strict",
    "maxEnumItems": 3
}
validator.validate_schema(schema, config)

Tool-calling schema helpers:

from walle import ms_tool_req_cvt, ms_tool_req_simplify

internal_json = ms_tool_req_cvt(request)
simplified_json, warnings = ms_tool_req_simplify(request)

Multiprocessing demos(recommended: spawn) in python/c-shared/example.py: use_multiprocessing_spawn() then use_multiprocessing_fork().

Python + multiprocessing (fork safety)

libwalle.so embeds a Go runtime via CGO. After fork(2), child processes must not reuse CGO handles inherited from the parent; crashes often appear as segfaults inside cgofree / FreeErrString.

1. Prefer spawn over fork (only if you use multiprocessing)

If your program uses multiprocessing or concurrent.futures.ProcessPoolExecutor, note that Linux often defaults to fork. Prefer spawn: each worker starts a fresh Python interpreter and reloads native code, so it does not inherit the parent’s CGO / Go runtime state. Set it once at the main entry point (stdlib only; no walle helper):

import multiprocessing as mp

if __name__ == "__main__":
    try:
        mp.set_start_method("spawn", force=True)
    except RuntimeError:
        pass  # already set
    ...

If you never use process-based parallelism, you can ignore this item.

2. If you must use fork

Import walle before starting worker pools. The package registers os.register_at_fork(after_in_child=...) to drop cached ctypes.CDLL handles; the next access through WalleValidator.lib loads a fresh libwalle.so in the child. This is a safety net when you cannot control the process model (default fork, third-party pools, etc.).

3. WalleValidator lifetime and process boundaries

Rule: any process that calls into libwalle should construct its own WalleValidator() there. Do not use a validator instance that was created in the parent (or in another interpreter) inside a worker.

Python + threading (main thread only)

Call WalleValidator (and helpers that use it, e.g. ms_tool_req_simplify) only from the thread that loaded libwalle first — in typical scripts, the main thread.

Do not invoke validate_schema / canonical_schema from threading.Thread targets or ThreadPoolExecutor workers: that path can SIGSEGV inside CGO (FreeErrString). A Python threading.Lock does not fix this.

If you need more throughput, add processes (see Python + multiprocessing), not more threads into libwalle.so.

Python regression tests (fork + CGO)

After python/c-shared/./build.sh (installs libwalle.so under python/walle/lib/):

PYTHONPATH=python python -m unittest discover -s python/tests -v

tests/test_fork_cgo_regression.py covers the historical crash: parent initializes libwalle, then forked workers call canonical_schema / ms_tool_req_simplify.

Documentation

Index

Constants

View Source 
const (
	Type                 = "type"
	Properties           = "properties"
	AdditionalProperties = "additionalProperties"
	Items                = "items"
	Enum                 = "enum"
	Required             = "required"
	AnyOf                = "anyOf"
	Description          = "description"
	Defs                 = "$defs"
	Ref                  = "$ref"
	Title                = "title"
	Id                   = "$id"
	Default              = "default"
	Pattern              = "pattern"
	MaxLength            = "maxLength"
	MinLength            = "minLength"
	Maximum              = "maximum"
	Minimum              = "minimum"
	MaxItems             = "maxItems"
	MinItems             = "minItems"
	String               = "string"
	Number               = "number"
	Integer              = "integer"
	Boolean              = "boolean"
	Null                 = "null"
	Array                = "array"
	Object               = "object"
	Root                 = "root"
)

Variables

View Source 
var (
	// Valid types and supported types
	ValidTypes = map[string]bool{
		"string":  true,
		"number":  true,
		"integer": true,
		"boolean": true,
		"null":    true,
		"array":   true,
		"object":  true,
	}

	// Common keywords
	CommonKeywords = map[string]bool{
		"description": true,
		"title":       true,
	}

	// Allowed keywords for different types
	ObjectAllowedKeywords = mergeMaps(map[string]bool{
		"type":                 true,
		"properties":           true,
		"required":             true,
		"additionalProperties": true,
		"$ref":                 true,
		"anyOf":                true,
		"default":              true,
	}, CommonKeywords)

	ArrayAllowedKeywords = mergeMaps(map[string]bool{
		"type":     true,
		"items":    true,
		"minItems": true,
		"maxItems": true,
		"$ref":     true,
		"default":  true,
	}, CommonKeywords)

	StringAllowedKeywords = mergeMaps(map[string]bool{
		"type":      true,
		"minLength": true,
		"maxLength": true,
		"pattern":   true,
		"enum":      true,
		"default":   true,
	}, CommonKeywords)

	BooleanAllowedKeywords = mergeMaps(map[string]bool{
		"type":    true,
		"enum":    true,
		"default": true,
	}, CommonKeywords)

	NullAllowedKeywords = mergeMaps(map[string]bool{
		"type":    true,
		"enum":    true,
		"default": true,
	}, CommonKeywords)

	NumberAllowedKeywords = mergeMaps(map[string]bool{
		"type":    true,
		"minimum": true,
		"maximum": true,
		"enum":    true,
		"default": true,
	}, CommonKeywords)

	TopLevelOnlyKeywords = map[string]bool{
		"$defs": true,
		"$id":   true,
	}

	// Contexts where $ref is allowed, "properties", "$defs", "root" will be processed independently
	RefAllowedContexts = map[string]bool{
		"additionalProperties": true,
		"anyOf":                true,
		"items":                true,
	}

	// currently supported keywords
	SupportedKeywords = map[string]bool{
		"type":                 true,
		"properties":           true,
		"additionalProperties": true,
		"items":                true,
		"enum":                 true,
		"required":             true,
		"anyOf":                true,
		"description":          true,
		"$defs":                true,
		"$ref":                 true,
		"title":                true,
		"$id":                  true,
		"default":              true,
		"maxLength":            true,
		"minLength":            true,
		"maximum":              true,
		"minimum":              true,
		"maxItems":             true,
		"minItems":             true,
	}

	// will be supported in the future
	FutureKeywords = map[string]bool{
		"pattern": true,
	}

	InvalidPropertyNames = map[string]bool{
		"$defs":                true,
		"$ref":                 true,
		"anyOf":                true,
		"required":             true,
		"additionalProperties": true,
	}
)

Functions

func IsSchemaError 

func IsSchemaError(err error) bool

func IsUnmarshalError 

func IsUnmarshalError(err error) bool

func NewSchemaError 

func NewSchemaError(message string, path string, simplifyFunc SimplifyFunc) error

Types

type KeywordValidatorFunc 

type KeywordValidatorFunc func(any, *validationContext, schemaPath) error

type Schema 

type Schema map[string]any

func ParseSchema 

func ParseSchema(jsonStr string) (Schema, error)

func SimplifyDefault 

func SimplifyDefault(schema Schema, _ schemaPath) Schema

func SimplifyNegativeVal 

func SimplifyNegativeVal(schema Schema, path schemaPath) Schema

func SimplifyRemoveAdditionalProperties 

func SimplifyRemoveAdditionalProperties(schema Schema, path schemaPath) Schema

func SimplifyRemoveAnyOf 

func SimplifyRemoveAnyOf(schema Schema, path schemaPath) Schema

func SimplifyRemoveConstraints 

func SimplifyRemoveConstraints(schema Schema, path schemaPath) Schema

func SimplifyRemoveDefs 

func SimplifyRemoveDefs(schema Schema, path schemaPath) Schema

func SimplifyRemoveDefsEmptySubSchema 

func SimplifyRemoveDefsEmptySubSchema(schema Schema, path schemaPath) Schema

func SimplifyRemoveDescription 

func SimplifyRemoveDescription(schema Schema, path schemaPath) Schema

func SimplifyRemoveDuplicateType 

func SimplifyRemoveDuplicateType(schema Schema, path schemaPath) Schema

func SimplifyRemoveEnum 

func SimplifyRemoveEnum(schema Schema, path schemaPath) Schema

func SimplifyRemoveID 

func SimplifyRemoveID(schema Schema, path schemaPath) Schema

func SimplifyRemoveItems 

func SimplifyRemoveItems(schema Schema, path schemaPath) Schema

func SimplifyRemoveParentSchema 

func SimplifyRemoveParentSchema(schema Schema, path schemaPath) Schema

func SimplifyRemovePattern 

func SimplifyRemovePattern(schema Schema, path schemaPath) Schema

func SimplifyRemoveProperties 

func SimplifyRemoveProperties(schema Schema, path schemaPath) Schema

func SimplifyRemoveRef 

func SimplifyRemoveRef(schema Schema, path schemaPath) Schema

func SimplifyRemoveRequired 

func SimplifyRemoveRequired(schema Schema, path schemaPath) Schema

func SimplifyRemoveSubSchema 

func SimplifyRemoveSubSchema(schema Schema, path schemaPath) Schema

func SimplifyRemoveType 

func SimplifyRemoveType(schema Schema, path schemaPath) Schema

func (Schema) Canonical 

func (s Schema) Canonical() (string, error)

Canonical returns a schema representation that conforms to Moonshot AI server requirements. It uses `ultra` validation level, which is the most permissive level supported by the enforcer-server. If the original schema has issues, it returns a simplified schema.

func (Schema) Validate 

func (s Schema) Validate(options ...SchemaValidatorOption) error

type SchemaDict 

type SchemaDict = map[string]any

type SchemaError 

type SchemaError struct {
	Path         string       `json:"path,omitempty"`
	Message      string       `json:"message,omitempty"`
	SimplifyFunc SimplifyFunc `json:"-"`
}

func (*SchemaError) Error 

func (e *SchemaError) Error() string

type SchemaList 

type SchemaList = []any

type SchemaValidatorConfig 

type SchemaValidatorConfig struct {
	ValidateLevel               ValidateLevel `json:"validateLevel,omitempty"`
	MaxEnumItems                int           `json:"maxEnumItems,omitempty"`
	MaxEnumStringLength         int           `json:"maxEnumStringLength,omitempty"`
	MaxEnumStringCheckThreshold int           `json:"maxEnumStringCheckThreshold,omitempty"`
	MaxAnyOfItems               int           `json:"maxAnyOfItems,omitempty"`
	MaxSchemaDepth              int           `json:"maxSchemaDepth,omitempty"`
	MaxSchemaSize               int           `json:"maxSchemaSize,omitempty"`
	MaxTotalPropertiesKeysNum   int           `json:"maxTotalPropertiesKeysNum,omitempty"`
}

func DefaultValidatorConfig 

func DefaultValidatorConfig() SchemaValidatorConfig

func (*SchemaValidatorConfig) IsGreaterThanStrict added in v0.1.4 

func (c *SchemaValidatorConfig) IsGreaterThanStrict() bool

func (*SchemaValidatorConfig) IsLite 

func (c *SchemaValidatorConfig) IsLite() bool

func (*SchemaValidatorConfig) IsLoose 

func (c *SchemaValidatorConfig) IsLoose() bool

func (*SchemaValidatorConfig) IsStrict 

func (c *SchemaValidatorConfig) IsStrict() bool

func (*SchemaValidatorConfig) IsTest 

func (c *SchemaValidatorConfig) IsTest() bool

do not use in production

func (*SchemaValidatorConfig) IsUltra added in v0.1.3 

func (c *SchemaValidatorConfig) IsUltra() bool

type SchemaValidatorOption 

type SchemaValidatorOption func(*SchemaValidatorConfig)

func WithMaxAnyOfItems 

func WithMaxAnyOfItems(max int) SchemaValidatorOption

WithMaxAnyOfItems set max anyOf items

func WithMaxEnumItems 

func WithMaxEnumItems(max int) SchemaValidatorOption

WithMaxEnumItems set max enum items

func WithMaxEnumStringCheckThreshold 

func WithMaxEnumStringCheckThreshold(max int) SchemaValidatorOption

WithMaxEnumStringCheckThreshold set max enum string check threshold

func WithMaxEnumStringLength 

func WithMaxEnumStringLength(max int) SchemaValidatorOption

WithMaxEnumStringLength set max enum string length

func WithMaxSchemaDepth 

func WithMaxSchemaDepth(max int) SchemaValidatorOption

WithMaxSchemaDepth set max schema depth

func WithMaxSchemaSize 

func WithMaxSchemaSize(max int) SchemaValidatorOption

WithMaxSchemaSize set max schema size

func WithMaxTotalPropertiesKeysNum 

func WithMaxTotalPropertiesKeysNum(max int) SchemaValidatorOption

WithMaxTotalPropertiesKeysNum set max total properties keys num

func WithValidateLevel 

func WithValidateLevel(level ValidateLevel) SchemaValidatorOption

type SimplifyFunc 

type SimplifyFunc func(schema Schema, path schemaPath) Schema

func SimplifyRemoveSchemaKeys added in v0.1.8 

func SimplifyRemoveSchemaKeys(keys []string) SimplifyFunc

deletes the given keys from the schema at path

type UnmarshalError 

type UnmarshalError struct {
	Err error `json:"error,omitempty"`
}

func NewUnmarshalError 

func NewUnmarshalError(err error) *UnmarshalError

func (*UnmarshalError) Error 

func (e *UnmarshalError) Error() string

type ValidateLevel 

type ValidateLevel string
const (
	ValidateLevelDefault ValidateLevel = "default" // default == ultra
	ValidateLevelLoose   ValidateLevel = "loose"
	ValidateLevelLite    ValidateLevel = "lite"
	ValidateLevelStrict  ValidateLevel = "strict"
	ValidateLevelUltra   ValidateLevel = "ultra"
	ValidateLevelTest    ValidateLevel = "test" // Do not use in production!
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
walle command
python
c-shared command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.