rc2sb

README

go-rc2sb

Go library for converting Resource Container (RC) repositories to Scripture Burrito (SB) format.

Installation

go get github.com/unfoldingWord/go-rc2sb

Usage

package main

import (
    "context"
    "fmt"
    "log"

    rc2sb "github.com/unfoldingWord/go-rc2sb"
)

func main() {
    ctx := context.Background()

    result, err := rc2sb.Convert(ctx, "/path/to/rc-repo", "/path/to/sb-output", rc2sb.Options{})
    if err != nil {
        log.Fatal(err)
    }

    fmt.Printf("Converted %s (%s) with %d ingredients\n",
        result.Subject, result.Identifier, result.Ingredients)
}

TWL Payload

For TSV Translation Words Links repos, a payload directory can be created from a Translation Words source. When a payload is available, the handler will:

1. Copy bible/* from the TW directory to ingredients/payload/ in the SB output
2. Rewrite rc://*/tw/dict/bible/... links in the TSV files to ./payload/... paths

There are two ways to provide the TW source:

Option 1: Explicit path via PayloadPath — Use this when the TW directory is stored separately from the TWL repo:

opts := rc2sb.Options{
    PayloadPath: "/path/to/en_tw",  // path to the <lang>_tw directory
}
result, err := rc2sb.Convert(ctx, "/path/to/en_twl", "/path/to/output", opts)

Option 2: Auto-detection — If the RC repo contains a <lang>_tw/ subdirectory (where <lang> matches the manifest's language identifier, e.g., en_tw/), it is detected automatically:

// The en_twl RC repo contains en_tw/ as a subdirectory — auto-detected
result, err := rc2sb.Convert(ctx, "/path/to/en_twl", "/path/to/output", rc2sb.Options{})

If neither PayloadPath is set nor a <lang>_tw/ subdirectory exists, the TSV files are copied as-is without payload or link rewriting.

Localized Book Names

Bible book names in the SB localizedNames are resolved using this priority:

1. USFM \toc1/\toc2/\toc3 markers — For Bible/USFM repos, these are read directly from the USFM files in the input. For TSV repos, use USFMPath to point to a USFM directory.
2. Manifest projects[].title — The title field from the RC manifest.yaml project entries.
3. English fallback — Hardcoded English names from books/books.go.

For non-English repos, this ensures book names like "उत्पत्ति" (Hindi for Genesis) appear in the metadata instead of only English names.

// Convert a Hindi TN repo with book names from a Hindi Bible USFM repo
opts := rc2sb.Options{
    USFMPath: "/path/to/hi_irv",  // directory containing NN-CODE.usfm files
}
result, err := rc2sb.Convert(ctx, "/path/to/hi_tn", "/path/to/output", opts)

CLI Tool

A simple CLI wrapper is available at cmd/rc2sb/:

go run ./cmd/rc2sb /path/to/rc-repo /path/to/sb-output

# With localized book names from a USFM directory (for TSV repos)
go run ./cmd/rc2sb --usfm /path/to/hi_irv /path/to/hi_tn /path/to/sb-output

# With TWL payload
go run ./cmd/rc2sb --payload /path/to/en_tw /path/to/en_twl /path/to/sb-output

API

Convert(ctx, inDir, outDir, opts) (Result, error)

Converts an RC repository to SB format.

• ctx - Context for cancellation
• inDir - Path to the RC repository (must contain manifest.yaml)
• outDir - Path where SB output will be written
• opts - Conversion options (see Options below)

Returns a Result with conversion metadata, or an error.

Options

type Options struct {
    // PayloadPath is the path to a Translation Words directory (e.g., "/path/to/en_tw").
    // Used for TWL conversion to create the ingredients/payload/ directory and
    // rewrite rc:// links in TSV files. If empty, auto-detects <lang>_tw/ inside inDir.
    PayloadPath string

    // USFMPath is the path to a directory containing USFM files for localized
    // Bible book names. Used by TSV handlers (TN, TQ, TWL) to extract
    // \toc1, \toc2, \toc3 markers. If empty, uses manifest project titles,
    // then English fallback.
    USFMPath string
}

Result

type Result struct {
    Subject     string // RC subject that was converted
    Identifier  string // RC identifier (e.g., "obs", "ult", "tn")
    InDir       string // Input RC directory
    OutDir      string // Output SB directory
    Ingredients int    // Number of ingredient files
}

Supported Subjects

Subject	SB Flavor Type	Notes
Open Bible Stories	gloss/textStories	Copies story files 01.md-50.md plus front.md/back.md; front/title.md becomes name/description
Aligned Bible	scripture/textTranslation	Strips numeric prefix from USFM filenames; abbreviation from RC identifier
Bible	scripture/textTranslation	Same as Aligned Bible (e.g., ULT, UST)
Hebrew Old Testament	scripture/textTranslation	Same as Aligned Bible (e.g., UHB)
Greek New Testament	scripture/textTranslation	Same as Aligned Bible (e.g., UGNT)
Translation Words	parascriptural/x-bcvarticles	Copies bible/ to ingredients/payload/; auto-detects <lang>_twl/ for TSV ingredients
Translation Academy	peripheral/x-peripheralArticles	Copies nested markdown hierarchy
TSV Translation Notes	parascriptural/x-bcvnotes	Strips tn_ prefix from TSV filenames
TSV Translation Questions	parascriptural/x-bcvquestions	Strips tq_ prefix from TSV filenames
TSV Study Notes	parascriptural/x-bcvnotes	Strips sn_ prefix from TSV filenames
TSV Study Questions	parascriptural/x-bcvquestions	Strips sq_ prefix from TSV filenames
TSV Translation Words Links	parascriptural/x-bcvarticles	Auto-detects <lang>_tw/ for payload; rewrites rc:// links
TSV OBS Study Notes	peripheral/x-obsnotes	Single TSV file conversion
TSV OBS Study Questions	peripheral/x-obsquestions	Single TSV file conversion
TSV OBS Translation Notes	peripheral/x-obsnotes	Single TSV file conversion
TSV OBS Translation Questions	peripheral/x-obsquestions	Single TSV file conversion
TSV OBS Translation Words Links	parascriptural/x-bcvarticles	TWL-style payload and rc:// rewriting; OBS as a single book

Error Handling

• Missing manifest.yaml returns an error indicating the directory is not a valid RC repo
• Unsupported subjects return an error listing all supported subjects
• Context cancellation is checked at key points during conversion
• File I/O errors are wrapped with context and returned

Building

go build ./...

Testing

# Run all tests
go test ./...

# Run tests with verbose output
go test ./... -v

# Run a specific test
go test -run TestConvertOpenBibleStories

# Run only unit tests (no samples needed)
go test ./rc/... ./sb/... ./books/...

# Run integration tests (requires samples/ directory)
go test -run TestConvert -v

Integration Tests

Integration tests use sample RC/SB pairs in the samples/ directory (gitignored). Each test:

1. Runs Convert() on the sample RC input
2. Verifies the output metadata structure matches the expected SB metadata
3. Verifies internal consistency (every ingredient in metadata.json exists on disk with correct MD5 and size)

Unit Tests

• rc/manifest_test.go - Manifest parsing (valid, invalid, missing)
• sb/ingredient_test.go - MD5/MIME/size computation
• sb/metadata_test.go - Metadata creation, serialization, round-trip
• books/books_test.go - Book lookups, localized names, sort order
• error_test.go - Error handling (missing manifest, unsupported subject, cancelled context)

Architecture

go-rc2sb/
+-- convert.go              # Public Convert() function
+-- options.go              # Options and Result types
+-- cmd/rc2sb/
|   +-- main.go             # CLI wrapper
+-- rc/
|   +-- manifest.go         # RC manifest.yaml parsing
+-- sb/
|   +-- metadata.go         # SB metadata.json types
|   +-- ingredient.go       # Ingredient computation (MD5, MIME, size)
+-- books/
|   +-- books.go            # Bible book data (66 books, localized names)
+-- handler/
|   +-- handler.go          # Handler interface
|   +-- registry.go         # Subject -> handler registry
|   +-- common.go           # Shared helpers (file copy, metadata building)
|   +-- obs.go              # Open Bible Stories
|   +-- aligned_bible.go    # Bible/USFM handler (Aligned Bible, Bible, Hebrew OT, Greek NT)
|   +-- tw.go               # Translation Words
|   +-- ta.go               # Translation Academy
|   +-- bible_tsv.go        # Bible TSV variants (TN, TQ, SN, SQ)
|   +-- twl.go              # TSV Translation Words Links (with payload)
|   +-- obs_tsv.go          # OBS TSV variants (4 types)
|   +-- obs_twl.go          # TSV OBS Translation Words Links
|   +-- subjects/
|       +-- register.go     # Registers all handlers

License

See LICENSE for details.

Documentation

Overview

Package rc2sb converts Resource Container (RC) repositories to Scripture Burrito (SB) format.

Index

• type Options
• type Result
  • func Convert(ctx context.Context, inDir string, outDir string, opts Options) (Result, error)

Types

type Options

type Options struct {
	// PayloadPath is the path to a Translation Words directory (e.g., "/path/to/en_tw")
	// used when converting TSV Translation Words Links repos.
	// If set, the bible/ subdirectory within this path is copied to ingredients/payload/
	// in the SB output, and rc:// links in the TWL TSV files are rewritten to
	// relative ./payload/ paths.
	//
	// If empty, the TWL handler auto-detects a <lang>_tw/ subdirectory inside
	// the input RC repo directory (where <lang> is the manifest's language identifier).
	// If neither is found, no payload is created and TSV files are copied as-is.
	PayloadPath string

	// TWLPath is the path to a TSV Translation Words Links directory (e.g., "/path/to/en_twl")
	// used when converting Translation Words repos.
	// If set, the twl_*.tsv files within this path are processed as the main ingredients,
	// rc:// links in the TSV files are rewritten to relative ./payload/ paths, and the
	// TW bible/ content is copied to ingredients/payload/.
	//
	// If empty, the TW handler auto-detects a <lang>_twl/ subdirectory inside
	// the input RC repo directory (where <lang> is the manifest's language identifier).
	// If neither is found, no TSV ingredients are created; only the payload is written.
	TWLPath string

	// USFMPath is the path to a directory containing USFM files for localized
	// Bible book names. This is used by TSV handlers (TN, TQ, TWL) to extract
	// \toc1, \toc2, \toc3 markers for book names in the target language.
	//
	// For Bible/USFM handlers, the USFM files in the input RC repo are used
	// directly, so this option is not needed.
	//
	// If empty, TSV handlers will use project titles from the manifest,
	// falling back to English names from the books package.
	USFMPath string

	// Owner is the repository owner (user or organization) on DCS, used to
	// build the burrito's primary identifier as "owner/repo" under the "dcs"
	// ID authority.
	//
	// If empty, the owner is detected from the git remote "origin" URL of the
	// input directory (when it is a git clone), falling back to the manifest's
	// dublin_core.publisher.
	Owner string

	// RepoName is the repository name on DCS (e.g., "en_tw"), used together
	// with Owner to build the burrito's primary identifier.
	//
	// If empty, the name is detected from the git remote "origin" URL of the
	// input directory, falling back to the DCS naming convention
	// "<language>_<identifier>" from the manifest.
	RepoName string

	// DCSURL is the base URL of the DCS instance that acts as the burrito's
	// ID authority (recorded in idAuthorities and referenced by
	// identification.primary).
	//
	// If empty, the URL is detected from the git remote "origin" URL of the
	// input directory, falling back to "https://git.door43.org".
	DCSURL string

	// Revision identifies the state of the source repository that was
	// converted (e.g., a DCS release number, tag, or commit SHA), recorded in
	// identification.primary.
	//
	// If empty, the HEAD commit SHA of the input directory is used (when it is
	// a git clone), falling back to the manifest's dublin_core.version (the
	// published release number), then "1".
	Revision string
}

Options configures the RC to SB conversion.

type Result

type Result struct {
	// Subject is the RC subject that was converted.
	Subject string

	// Identifier is the RC identifier (e.g., "obs", "ult", "tn").
	Identifier string

	// InDir is the input RC directory that was converted.
	InDir string

	// OutDir is the output SB directory that was created.
	OutDir string

	// Ingredients is the number of ingredient files in the SB output.
	Ingredients int
}

Result holds information about a completed conversion.

func Convert

func Convert(ctx context.Context, inDir string, outDir string, opts Options) (Result, error)

Convert converts an RC repository at inDir to SB format, writing output to outDir.