xgparser

README

XGParser - Go Implementation

A Go library for parsing eXtremeGammon (.xg) backgammon match files, with both full and lightweight parsing capabilities.

Features

• Full Parser: Complete XG file structure parsing (xgstruct.go)
• Lightweight Parser: Essential data extraction for database integration (xglight.go)
• Flexible Input: Parse from files, HTTP uploads, memory, or any io.ReadSeeker
• JSON Export: All lightweight structures are JSON-serializable
• Database Ready: Optimized for SQL storage and statistical analysis

License

This library is licensed under the MIT License (see LICENSE).

Credits

The .xg / .xgp file format was first documented publicly by the Python xgdatatools library:

• xgdatatools: Michael Petch (Copyright © 2013-2014)

  • Email: mpetch@gnubg.org
  • GitHub: https://github.com/oysteijo/xgdatatools

• This library: Kévin Unger (Copyright © 2025)

Thanks to Michael Petch for describing the format.

Installation

go get github.com/kevung/xgparser

Quick Start

Lightweight Parser (Recommended)

package main

import (
    "fmt"
    "log"
    "github.com/kevung/xgparser/xgparser"
)

func main() {
    // Parse XG file
    match, err := xgparser.ParseXGFromFile("match.xg")
    if err != nil {
        log.Fatal(err)
    }

    // Access match data
    fmt.Printf("%s vs %s\n", 
        match.Metadata.Player1Name,
        match.Metadata.Player2Name)
    fmt.Printf("Event: %s\n", match.Metadata.Event)
    fmt.Printf("Games: %d\n", len(match.Games))
    
    // Export to JSON
    jsonData, _ := match.ToJSON()
    fmt.Println(string(jsonData))
}

Parse from HTTP Upload

func uploadHandler(w http.ResponseWriter, r *http.Request) {
    file, _, _ := r.FormFile("xgfile")
    defer file.Close()
    
    data, _ := io.ReadAll(file)
    reader := io.NewSectionReader(bytes.NewReader(data), 0, int64(len(data)))
    
    match, err := xgparser.ParseXGFromReader(reader)
    if err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }
    
    w.Header().Set("Content-Type", "application/json")
    jsonData, _ := match.ToJSON()
    w.Write(jsonData)
}

Documentation

• Lightweight Parser Guide - Complete API reference, examples, and database schemas

Command-Line Tools

Build all tools:

go build -o xglight ./cmd/xglight/
go build -o stats_example ./cmd/stats_example/
go build -o reader_example ./cmd/reader_example/
go build -o web_example ./cmd/web_example/

xglight - Parse to JSON

./xglight match.xg > match.json

stats_example - Extract Statistics

./stats_example match.xg

reader_example - Demonstrate Parsing Methods

./reader_example match.xg

web_example - Web Server with Upload

./web_example
# Visit http://localhost:8080

API Overview

Parsing Functions

// Parse from file (simple)
match, err := xgparser.ParseXGFromFile("match.xg")

// Parse from io.Reader (flexible)
match, err := xgparser.ParseXGFromReader(reader)

// Parse from segments (advanced)
match, err := xgparser.ParseXG(segments)

Key Structures

type Match struct {
    Metadata MatchMetadata
    Games    []Game
}

type Game struct {
    GameNumber   int32
    InitialScore [2]int32
    Moves        []Move
    Winner       int32
    PointsWon    int32
}

type Move struct {
    MoveType    string  // "checker" or "cube"
    CheckerMove *CheckerMove
    CubeMove    *CubeMove
}

See LIGHTWEIGHT_PARSER.md for complete API reference.

Project Status

Complete ✅

• xgutils.go: Utility functions (CRC32, UTF-16, datetime conversion)
• xgzarc.go: ZLib archive handling
• xgstruct.go: Full XG data structures
• xgimport.go: File import and segment extraction
• xglight.go: Lightweight parser for database integration
• cmd/: Command-line tools (xglight, stats_example, reader_example, web_example)

All parsers produce output that matches the original Python implementation exactly.

Use Cases

• Match Analysis: Extract statistics from tournament matches
• Database Storage: Import matches into SQL databases
• Web Applications: Parse uploaded XG files in HTTP handlers
• Batch Processing: Analyze large collections of matches
• Statistical Research: Extract equity loss, move quality metrics

Performance

• Parses typical 7-point matches in milliseconds
• JSON output ~10-20% size of full parser
• Suitable for real-time web applications
• No memory leaks, efficient allocation

Full Parser Usage

The original full parser is still available for complete XG file analysis:

# Build full parser
go build -o xgparser/xgparser ./cmd/xgparser

# Run full parser
./xgparser/xgparser tmp/test.xg

Repository

• GitHub: https://github.com/kevung/xgparser
• Branch: xg_light_parsing

Contributing

Contributions are welcome! Please ensure:

• All tests pass (./test_xglight.sh)
• Code follows Go conventions
• Documentation is updated
• Backward compatibility is maintained

## Changelog

### v1.5.0

- `MatchMetadata` carries the XG match header fields `Player1Elo`, `Player2Elo`,
  `Player1Experience`, `Player2Experience`, `Transcriber`, `Jacoby`, `Beaver`,
  `MatchHeaderComment` and `MatchFooterComment` (additive, no API break).
- Fix: `ProductVersion` no longer receives the GDF header title ("Played on <location>");
  `ProductVersion` is now always empty for `.xg`/`.xgp` files, which do not record the
  XG release; it is set only by the XGID text parsers.
- Files whose game file follows the GDF header as a single zlib stream, without the
  archive index and trailer, are now read instead of failing with
  "seek: invalid argument" or "archive CRC check failed".
- Fix: an empty comment in the comment segment keeps its index; the comments after it
  no longer shift onto the wrong move, cube decision or match header.
- Decompressed and stored segments are capped by `MaxDecompressedSize` (default 128 MiB,
  about x21 the largest segment of the BMAB europe corpus) against decompression bombs;
  beyond it, parsing fails with a `*SizeLimitError` (`errors.Is(err, ErrDecompressionLimit)`).

## Fixed Issues

### EngineStructBestMoveRecord - CubePos vs Cubepos ✅ COMPLETE
**Problem**: Python has a bug where it defines `CubePos` in defaults but sets `Cubepos` (lowercase 'p') from the stream. This means both fields exist in the output with different values.

**Solution**: Added both fields to the Go struct:
- `CubePos int32` - remains at default value 0 (not updated from stream)
- `Cubepos int32` - actual value read from stream

This maintains perfect compatibility with the Python bug. The stream value goes into field index 32 of the unpacked data, which Python assigns to `self.Cubepos` (lowercase).

### FooterGameEntry Padding ✅ COMPLETE
**Problem**: Fields like `Score1g`, `Score2g`, `PointsWon`, `Winner` had wrong values. Score1g showed 33554432 instead of 0, Score2g showed 0 instead of 2.

**Root Cause**: The Python struct format `'<9xxxxllBxxxlllxxxxdd7dl'` has `9xxxx` at the start. This means:
- `9x` = skip 9 bytes
- `xxxx` = skip 3 MORE bytes (each 'x' is separate)
- Total: skip **12 bytes**, not 13!

Python's `struct.calcsize('<9xxxx')` returns 12, not 13. The Go code was skipping 9+4=13 bytes.

**Solution**: Changed `padding1 [4]byte` to `padding1 [3]byte` in FooterGameEntry.FromStream().

**Verification**: After fix, Score1g=0, Score2g=2, PointsWon=2, Winner=-1 all match Python exactly.

### FooterMatchEntry Padding ✅ COMPLETE
**Problem**: Same padding issue as FooterGameEntry. Score1m showed 83886080 instead of 7, Score2m showed 16777216 instead of 5.

**Root Cause**: Python format `'<9xxxxlllddlld'` has the same `9xxxx` pattern = 12 bytes total skip, not 13.

**Solution**: Changed `padding1 [4]byte` to `padding1 [3]byte` in FooterMatchEntry.FromStream().

**Key Learning**: In Python struct formats like `'9xxxx'`:
- The number prefix (9) only applies to the immediately following character
- `9x` = skip 9 bytes
- Each additional `x` adds 1 byte
- So `9xxxx` = 9 + 1 + 1 + 1 + 1 = **13 characters** but only **12 bytes**!
- Use `struct.calcsize()` to verify the actual byte count

### EngineStructBestMoveRecord ✅ COMPLETE
**Problem**: The `Dice` field was declared as `[2]int8` causing all subsequent fields in MoveEntry to be misaligned, resulting in garbage values for AnalyzeL, AnalyzeM, CompChoice, and all evaluation data.

**Solution**: Changed `Dice [2]int8` to `Dice [2]int32`. The Python struct format `'<26bxx2ll2llllll'` clearly shows `2l` for Dice, meaning 2 int32s (8 bytes total), not 2 int8s (2 bytes).

**Impact**: This single 6-byte misalignment cascaded through the entire MoveEntry structure, affecting:
- All fields read after the EngineStructBestMoveRecord
- The Eval arrays (were all zeros, now show correct float values)
- The move analysis data (AnalyzeL, AnalyzeM, CompChoice)

### HeaderMatchEntry ✅ COMPLETE
**Problem**: Several fields had wrong values due to incorrect struct field types and misalignment
- `Version`: Was 536870912, now correctly reads **32**
- `Invert`: Changed from `int8` to `int32` (Python uses `'l'` format = 4 bytes)
- `Magic`: Now correctly reads **1229737284** (0x494C4D44)
- `CubeLimit`: Now **10** (was 184549375)
- `CommentHeaderMatch`: Now **-1** (was -16777216)
- `Transcribed`: Now **true** (was false)
- `isMoneyMatch`: Now **false** (was true)

**Root Cause**: The Python struct format `'<9x41B41BxllBBBBddlld129Bxxx...'` uses specific padding and field sizes. The Go code was reading `Invert` as `int8` when Python reads it as `'l'` (int32). This created a cascading misalignment for all subsequent fields.

### HeaderGameEntry ✅ COMPLETE
**Problem**: `GameNumber` was 0 instead of 1, `CommentFooterGame` was 16777215 instead of -1

**Solution**: Fixed padding from 4 bytes to 3 bytes. Python format `'<9xxxxllB26bxlBxxxlll'` means:
- `9x` = skip 9 bytes
- `xxx` = skip 3 MORE bytes (each `x` is one byte)
- Total skip = **12 bytes** (not 13!)

**Key Learning**: In Python struct format strings like `'9xxxx'`:
- The number prefix (9) only applies to the immediately following character
- `9x` means "skip 9 bytes"
- `xxx` means "skip 1 + skip 1 + skip 1" = skip 3 bytes
- So `9xxxx` = 9 + 3 = **12 bytes** total

### CubeEntry ✅ COMPLETE
**Problem**: Several fields had wrong values, DiceRolled was empty, numeric fields showed garbage
- `AnalyzeC`, `AnalyzeCR`, `CommentCube`: Were showing 65535 or 0 instead of -1
- `CompChoiceD`: Was showing -65536 instead of 0
- `DiceRolled`: Was empty instead of showing dice values like '43'
- `ErrCube`, `ErrBeaver`: Were showing garbage float values instead of -1000.0

**Root Cause**: The EngineStructDoubleAction struct had `Crawford` defined as `int32` when it should be `int16`. The Python struct format `'<26bxxl2llllhhhh7ffffhh7f'` shows:
- `l2lll` = 1 + 2 + 3 = **6 int32s** (Level, Score[2], Cube, CubePos, Jacoby)
- `hhhh` = **4 int16s** (Crawford, met, FlagDouble, isBeaver)

The Go code was reading Crawford as int32 (4 bytes) instead of int16 (2 bytes), causing a 2-byte misalignment for all subsequent fields in CubeEntry's second section.

**Solution**: Changed `Crawford int32` to `Crawford int16` in EngineStructDoubleAction struct.

**Key Learning**: In Python struct format strings, the number prefix only applies to the immediately following character. So `l2llll` is parsed as:
- `l` = 1 int32
- `2l` = 2 int32s  
- `lll` = 3 int32s
- Total: 6 int32s, NOT 7!

### MoveEntry ✅ COMPLETE
**Problem**: After reading EngineStructBestMoveRecord, all subsequent fields were misaligned:
- `AnalyzeL`: Was 262143, now correctly **3**
- `AnalyzeM`: Was -1, now correctly **3**
- `CompChoice`: Was -1449787392, now correctly **1**
- `CommentMove`: Was 0, now correctly **-1**
- `Dice` in DataMoves: Was (4, 0), now correctly **(4, 3)**
- All `Eval` float arrays: Were zeros, now show correct values

**Root Cause**: In EngineStructBestMoveRecord, the `Dice` field was declared as `[2]int8` (2 bytes) when it should have been `[2]int32` (8 bytes). The Python struct format `'<26bxx2ll2llllll'` clearly shows:
- `26b` = 26 bytes (Pos)
- `xx` = 2 bytes padding
- `2l` = **2 int32s** (Dice) = 8 bytes
- `l` = 1 int32 (Level)
- ... etc

This 6-byte deficit (read 2 bytes, should have read 8) caused all subsequent reads in MoveEntry to be offset by 6 bytes, corrupting every field value.

**Solution**: Changed `Dice [2]int8` to `Dice [2]int32` in EngineStructBestMoveRecord struct.

**Verification**: After the fix, a complete diff between Python and Go output shows ONLY cosmetic differences:
- Boolean formatting (`False` vs `false`)
- Float formatting (`0.0` vs `0` for zero values)
- UTF-8 byte string display
- All numerical values match exactly (within float32 precision)

## Critical Learning: Python struct.unpack Padding Rules

The main challenge in porting from Python to Go is understanding Python's struct format strings:

1. **Number prefixes apply ONLY to the next character**:
   - `'9x'` = skip 9 bytes
   - `'9xxxx'` = skip 9 bytes, then skip 1, then skip 1, then skip 1 = **12 bytes total** (NOT 13!)
   - `'2l'` = 2 int32s (8 bytes)
   - `'l2l'` = 1 int32, then 2 int32s = 3 int32s total (12 bytes)

2. **Each format character must be counted individually**:
   - `'xxxx'` = 4 separate skip operations = 4 bytes
   - `'4x'` = one skip operation of 4 bytes = 4 bytes (same result, different notation)

3. **Verify with `struct.calcsize()`**: Always test the format string to see actual byte size

4. **Test with synthetic data**: Create test arrays and parse to see which byte positions map to which values

5. **Common format characters**:
   - `b` / `B` = int8 / uint8 (1 byte)
   - `h` / `H` = int16 / uint16 (2 bytes)
   - `l` / `L` = int32 / uint32 (4 bytes)
   - `f` = float32 (4 bytes)
   - `d` = float64 (8 bytes)
   - `x` = padding (1 byte)

## What Works Now

All record types parse correctly and produce output that matches the Python implementation EXACTLY! ✅

**Perfect numerical match achieved!** After comparing 25,173 numerical field values:
- ✅ All integer values match exactly
- ✅ All float values match within float32 precision
- ✅ Zero significant differences found

The only remaining differences are purely cosmetic formatting:
- Boolean capitalization: `False`/`True` (Python) vs `false`/`true` (Go)
- Float zeros: `0.0` (Python) vs `0` (Go)
- UTF-8 byte string display formatting
- Float precision display (Python shows float64, Go shows float32 as per spec)

Successfully implemented:
- File decompression and archive extraction ✅
- Record type identification ✅  
- **HeaderMatchEntry**: All fields parse correctly ✅
  - Version, Magic, GameId, MatchLength, Date ✅
  - Player names, Event, Location (UTF-16 decoded) ✅
  - Boolean fields (Crawford, Jacoby, Transcribed) ✅
  - Monetary fields, ELO ratings ✅
- **HeaderGameEntry**: All fields parse correctly ✅
  - GameNumber, Scores, Position arrays ✅
  - Comment indices ✅
- **CubeEntry**: All fields parse correctly ✅
  - ActiveP, Double, Take, BeaverR ✅
  - Position array ✅
  - EngineStructDoubleAction with correct padding ✅
  - All error values, analysis levels, dice ✅
- **MoveEntry**: All fields parse correctly ✅
  - Position arrays, moves, dice ✅
  - EngineStructBestMoveRecord with correct field sizes ✅
  - Both CubePos and Cubepos fields (Python bug compatibility) ✅
  - All analysis levels (AnalyzeL, AnalyzeM) ✅
  - All evaluation arrays with correct float values ✅
  - CompChoice, error values, rollout indices ✅
- **FooterGameEntry**: All fields parse correctly ✅
  - Score1g, Score2g, PointsWon, Winner, Termination ✅
  - ErrResign, ErrTakeResign ✅
  - Eval array, EvalLevel ✅
- **FooterMatchEntry**: All fields parse correctly ✅
  - Score1m, Score2m, WinnerM ✅
  - Elo1m, Elo2m, Exp1m, Exp2m ✅
  - Datem (datetime conversion) ✅

## Output Comparison

The Go parser now produces output that matches the Python parser exactly! The only differences are cosmetic formatting:

1. **Boolean formatting**: Python uses `True`/`False`, Go uses `true`/`false`
2. **Float formatting**: Python shows `0.0` for zero floats, Go shows `0`
3. **Precision**: Python shows full float64 precision, Go shows float32 precision (as per XG file format spec)
4. **UTF-8 display**: Different byte string representations in output
5. **File paths**: Relative paths may differ based on where command is run

All numerical values are identical within the expected precision of float32 values.

## Debugging Methodology

The successful approach used to fix all parsing issues:

1. **Extract binary test data**: Save the raw binary records to files
2. **Parse in Python with struct.unpack**: Get correct reference values
3. **Examine bytes manually**: Use hex dumps to see exact byte positions
4. **Analyze format strings carefully**: Count each format character individually, remembering that number prefixes only apply to the immediately following character
5. **Fix field types and padding**: Match Python format string exactly in Go struct definitions
6. **Test incrementally**: Rebuild and compare output after each fix
7. **Verify with diff**: Use diff to compare complete outputs and verify only cosmetic differences remain

The key insight was that seemingly small type mismatches (like `int8` vs `int32`, or `int16` vs `int32`) create cascading misalignments that corrupt all subsequent fields. Every byte must be accounted for precisely.

## Lightweight Parsing for Database Integration

🆕 **New in this fork**: A lightweight parsing module (`xglight.go`) that extracts only essential match information suitable for database integration. See [LIGHT_PARSING.md](LIGHT_PARSING.md) for detailed documentation.

### Key Features of Light Parsing

- **Simplified data structures** - Only essential match, game, and move information
- **JSON serializable** - Easy integration with databases and APIs
- **Database-ready** - Designed for SQL storage with suggested schema
- **Focused analysis** - Only the most relevant engine analysis metrics
- **No bloat** - Omits rollouts, comments, thumbnails, and other detailed data

### Quick Start with Light Parser

```bash
# Build the light parser
go build -o xglight ./cmd/xglight

# Parse to JSON
./xglight match.xg > match.json

# View summary
./xglight match.xg 2>&1 | grep "==="

# Example statistics tool
go build -o stats_example ./examples/stats_example.go
./stats_example match.xg
```

### Light Parsing API

```go
import "github.com/kevung/xgparser/xgparser"

// Parse an XG file
match, err := xgparser.ParseXGLight("match.xg")
if err != nil {
    log.Fatal(err)
}

// Access data
fmt.Printf("%s vs %s\n", match.Metadata.Player1Name, match.Metadata.Player2Name)
fmt.Printf("Games: %d\n", len(match.Games))

// Export to JSON
jsonData, _ := match.ToJSON()
fmt.Println(string(jsonData))
```

## Full Parser Usage

The original full parser is still available for complete XG file analysis:

```bash
# Build full parser
go build -o xgparser/xgparser ./cmd/xgparser

# Run full parser
./xgparser/xgparser tmp/test.xg

# Compare with Python
cd tmp/xgdatatools
python3 extractxgdata.py ../test.xg > ../python_output.txt
cd ../..
./xgparser/xgparser tmp/test.xg > go_output.txt

# Diff will show only cosmetic differences (True/False vs true/false, etc)
diff tmp/python_output.txt go_output.txt
```

## Testing

To verify the Go implementation produces correct output:

```bash
# Generate both outputs
python3 tmp/xgdatatools/extractxgdata.py tmp/test.xg > /tmp/python_output.txt
./xgparser/xgparser tmp/test.xg > /tmp/go_output.txt

# Compare - should see only cosmetic differences
diff /tmp/python_output.txt /tmp/go_output.txt
```

Expected differences in diff output:
- File paths (`../test.xg` vs `tmp/test.xg`)
- Boolean capitalization (`False` vs `false`, `True` vs `true`)
- Float zero formatting (`0.0` vs `0`)
- UTF-8 byte string display
- Float precision (Python shows more decimal places for float64, Go shows float32 precision)

All numerical values should be identical (within float32 precision).

## Project Structure

```
xgparser/
├── cmd/                    # Command-line tools
│   ├── batch_xgid/        # Batch XGID file processor
│   ├── debug_moves/       # Move debugging utility
│   ├── reader_example/    # File reading example
│   ├── stats_example/     # Statistics example
│   ├── web_example/       # Web server example
│   ├── xgid_cube_parser/  # XGID cube decision parser
│   ├── xgid_parser/       # XGID file parser
│   ├── xglight/           # Lightweight parser CLI
│   └── xgparser/          # Full parser CLI
│
├── xgparser/              # Core library package
│   ├── xgid.go           # XGID parsing
│   ├── xgid_test.go      # XGID tests
│   ├── xgimport.go       # File import
│   ├── xglight.go        # Lightweight parser
│   ├── xgstruct.go       # Data structures
│   ├── xgutils.go        # Utility functions
│   └── xgzarc.go         # Archive handling
│
├── tools/                 # Development utilities
│   └── verify_all_xgid.go # XGID verification tool
│
├── docs/                  # Additional documentation
├── tmp/                   # Test data and examples
│   └── xgid/             # XGID test files (9 languages)
│
├── README.md             # This file
├── LICENSE               # MIT license
├── LANGUAGE_SUPPORT.md   # Multi-language parsing info
├── LIGHTWEIGHT_PARSER.md # Lightweight parser docs
├── XGID_PARSER.md        # XGID format documentation
└── go.mod                # Go module definition
```

## Building

Build all command-line tools:
```bash
# Build specific tool
go build -o bin/xgid_parser ./cmd/xgid_parser

# Build all tools
for cmd in cmd/*; do
  name=$(basename $cmd)
  go build -o bin/$name ./$cmd
done
```

## Architecture

The package mirrors the Python implementation:

1. **Import** - Reads XG file, extracts Game Data Format header
2. **Archive** - Decompresses embedded ZLib archive containing game data
3. **Segments** - Extracts file segments (temp.xg, temp.xgr, temp.xgc, temp.xgi)
4. **Records** - Parses game file records (Match header, game headers, moves, cube actions, etc.)
5. **Output** - Formats output to match Python pprint style

The main implementation challenge is matching Python's C-style struct packing exactly in Go, which requires careful manual control over byte offsets and understanding Python's struct format notation.