assetsdb

README

assetsdb

assetsdb builds and provides searchable access to an on-disk database of CC0-licensed game-art assets. It ingests curated asset packs (currently Kenney and KayKit) into a single portable directory containing a JSON catalog and the original source archives.

The full on-disk and JSON format is specified in SPEC.md. This README covers what exists and how to use it.

License: code vs. assets

This distinction matters and is easy to blur, so it's stated up front:

• The assetsdb source code (this repository) is licensed under the MIT License.
• The assets it ingests (Kenney and KayKit packs) are licensed CC0-1.0 (public domain) by their original authors. assetsdb does not relicense them; it records their CC0-1.0 license in the database and never mixes it up with the MIT license of assetsdb's own code.

Install / build

make build

This builds the assetsdb CLI to bin/assetsdb (./cmd/assetsdb). make setup-tools installs the pinned golangci-lint; make validate runs lint, test, and build together.

Usage

assetsdb build (implemented)

assetsdb build --manifest packs.yaml --out <db>

Reads a manifest (see packs.yaml for the starter curated list of Kenney and KayKit packs), fetches each listed pack over the network with the matching ingester, and writes a fresh database to <db> (a datapackage.json index plus one ZIP per source under <db>/sources).

assetsdb add (reserved, phase-2 — not yet implemented)

assetsdb add is reserved for a future phase that ingests locally-generated assets (e.g. AI-generated sprites) into the database's generated/ directory as loose files, with a provenance record per item. The on-disk format and Go types already accommodate this (see SPEC.md), but no code path in this repository writes to generated/ or implements the add command yet.

On-disk layout

<db>/
  datapackage.json          # the index (Frictionless Data Package v2 shaped)
  sources/<sourceid>.zip    # curated packs, stored verbatim as fetched
  generated/                # reserved for phase-2 locally-generated assets (empty in v1)

Format: adopted specs

datapackage.json is a hybrid of three established specs rather than a bespoke schema:

• Frictionless Data Package v2 — the on-disk shape itself; every asset is a standard Data Resource in the required resources[].
• W3C DCAT v3 — the semantic cross-reference (Catalog/Dataset/Distribution), so the file can be losslessly re-expressed as RDF later.
• SPDX — license identifiers (CC0-1.0, MIT, ...).

See SPEC.md for the full key-by-key schema and the Frictionless/DCAT crosswalk.

v1 sources

Two curated ingest sources, both confirmed CC0-1.0:

• Kenney — page-scrape of the pack page's direct ZIP download link, plus optional parsing of a Sparrow/Starling-style spritesheet XML atlas (<TextureAtlas><SubTexture>) when a pack ships one.
• KayKit — GitHub codeload download (codeload.github.com/.../zip/refs/heads/<branch>) of each pack repo, walking the archive for 3D models and textures.

Go package

The module root, github.com/jbeshir/assetsdb, contains the database API, schema types, catalog construction helpers, JSON serialization, asset classification and tokenization, and the ingest contract. For example:

import "github.com/jbeshir/assetsdb"

db, err := assetsdb.Read(os.Getenv("ASSETS_DB"))
if err != nil {
    // handle error
}

hits := db.Search("dungeon torch")
for _, item := range hits {
    lic := db.LicenseFor(item)
    rc, err := db.Open(item)
    // ...
}

sources := db.Sources() // stable source-ID order; safe for the caller to mutate
items := db.ItemsForSource(sources[0].Name) // stable item-ID order; also mutation-safe

archive, err := db.OpenSource(sources[0].Name) // raw registered source archive
// use and close archive

The path passed to Read should be a directory built by assetsdb build (one containing datapackage.json). Open and OpenSource use the archive path registered in each source and reject paths that escape the database root. The root package also exposes DataPackage, Encode, Decode, Classify, MediaType, Tokenize, PackSpec, and Ingester; see SPEC.md for the complete API.

Documentation

Overview

Package assetsdb defines the assets database format and provides tools to build, serialize, classify, tokenize, and read an assetsdb database directory.

Package assetsdb defines the on-disk shape of an assetsdb database: a Frictionless Data Package v2 hybrid JSON descriptor (datapackage.json), read/write access to it, extension-based asset classification, filename tokenization for search, and the Ingester interface implemented by each curated-pack importer. It contains no network code and no provider-specific logic.

Index

• Variables
• func Encode(w io.Writer, dataPackage *DataPackage) error
• func ExtNoDot(p string) string
• func Hash(contents []byte) string
• func ItemID(sourceID, pathInSource, sub string) string
• func MediaType(path string) string
• func Slugify(s string) string
• func Title(base string) string
• func Tokenize(p string) []string
• type DB
  • func Read(root string) (*DB, error)
  • func (db *DB) ItemsForSource(sourceID string) []Item
  • func (db *DB) LicenseFor(item Item) License
  • func (db *DB) Open(item Item) (io.ReadCloser, error)
  • func (db *DB) OpenSource(sourceID string) (io.ReadCloser, error)
  • func (db *DB) Search(query string) []Item
  • func (db *DB) SourceByID(id string) (Source, bool)
  • func (db *DB) Sources() []Source
• type DataPackage
  • func Decode(r io.Reader) (*DataPackage, error)
• type Ingester
• type Item
• type Kind
  • func Classify(path string) (kind Kind, ok bool)
• type License
  • func LicenseForSPDX(spdx string) License
• type NameAllocator
  • func NewNameAllocator() *NameAllocator
  • func (a *NameAllocator) Allocate(base string) string
• type PackSpec
• type Provenance
• type Region
• type Source
  • func CuratedSource(spec PackSpec, license License) Source

Variables

var ErrNotFound = errors.New("assetsdb: not found")

ErrNotFound identifies an unknown source, a missing registered archive, or a missing item.

Functions

func Encode

func Encode(w io.Writer, dataPackage *DataPackage) error

Encode writes a DataPackage as indented JSON.

func ExtNoDot

func ExtNoDot(p string) string

ExtNoDot returns p's lowercased file extension without its leading dot, or "" if p has none. It uses path (not path/filepath) because it operates on forward-slash in-zip paths.

func Hash

func Hash(contents []byte) string

Hash returns the content hash recorded on an Item: the SHA-256 of contents, hex-encoded and carried with the "sha256:" algorithm prefix Frictionless hashes use.

func ItemID

func ItemID(sourceID, pathInSource, sub string) string

ItemID builds an Item's canonical stable global id: "assetsdb:<sourceID>/<pathInSource>", with a "#<sub>" suffix appended when sub is non-empty (used for atlas sub-sprites named by their SubTexture). This is the single source of truth for the id layout documented on Item.ID.

func MediaType

func MediaType(path string) string

MediaType returns a best-effort IANA media type for path's extension, or "" if unknown.

func Slugify

func Slugify(s string) string

Slugify lowercases s, replaces every run of disallowed characters with a single hyphen, collapses repeated hyphens, and trims leading and trailing hyphens, yielding a valid Frictionless resource name matching ^[a-z0-9._-]+$.

func Title

func Title(base string) string

Title humanizes an extension-stripped filename stem into a Title Case string, e.g. "chair_A" -> "Chair A". It reuses the same delimiter and word splitting as Tokenize (via splitIdentifier), so an item's title and its search tokens are always derived by the same word-boundary rule.

func Tokenize

func Tokenize(p string) []string

Tokenize splits path into lowercase, order-preserving deduplicated search tokens: the filename (minus extension) is split on delimiters and camelCase/letter-digit boundaries, and the single immediate parent directory is included too, unless it is a generic bucket directory (see bucketDirs).

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is a read-only, in-memory index of an assetsdb database directory.

func Read

func Read(root string) (*DB, error)

Read loads and indexes the datapackage.json beneath root.

func (*DB) ItemsForSource

func (db *DB) ItemsForSource(sourceID string) []Item

ItemsForSource returns the source's items ordered by stable item ID. Unknown sources and sources with no items both return an empty slice.

func (*DB) LicenseFor

func (db *DB) LicenseFor(item Item) License

LicenseFor resolves an item's first source license, then the package's first license.

func (*DB) Open

func (db *DB) Open(item Item) (io.ReadCloser, error)

Open extracts item.Path from the archive registered by the item's source.

func (*DB) OpenSource

func (db *DB) OpenSource(sourceID string) (io.ReadCloser, error)

OpenSource returns the raw bytes of the archive registered under sourceID. The caller must close the returned reader.

func (*DB) Search

func (db *DB) Search(query string) []Item

Search returns matching items ordered by stable item ID.

func (*DB) SourceByID

func (db *DB) SourceByID(id string) (Source, bool)

SourceByID returns a mutation-safe copy of the registered source with id.

func (*DB) Sources

func (db *DB) Sources() []Source

Sources returns all registered sources ordered by source ID. The returned values do not share mutable slices with the database.

type DataPackage

type DataPackage struct {
	Name     string    `json:"name"`
	Title    string    `json:"title"`
	Version  string    `json:"version"`
	Created  string    `json:"created"`
	Licenses []License `json:"licenses"`
	// SchemaVersion is the on-disk format version; bump on breaking change.
	SchemaVersion int `json:"x_assetsdb:schemaVersion"`
	// Sources is the pack registry that Item.Source values link back into.
	Sources []Source `json:"x_assetsdb:sources"`
	// Resources holds every catalogued Item; REQUIRED by the Frictionless spec.
	Resources []Item `json:"resources"`
}

DataPackage is the root of datapackage.json: a Frictionless Data Package v2 descriptor whose REQUIRED resources[] holds every catalogued Item, plus a custom x_assetsdb:sources[] registry of the asset packs those items belong to.

func Decode

func Decode(r io.Reader) (*DataPackage, error)

Decode reads a DataPackage from its JSON representation.

type Ingester

type Ingester interface {
	// Name identifies this ingester, matched against PackSpec.Source (e.g. "kenney", "kaykit").
	Name() string

	// Ingest fetches the pack and returns its Source metadata, its Items, and the raw ZIP bytes
	// to persist under sources/<id>.zip. Items' Path fields are relative to the ZIP root.
	Ingest(ctx context.Context, spec PackSpec) (src Source, items []Item, zip []byte, err error)
}

Ingester is implemented by each curated-pack importer (Kenney, KayKit, ...). It fetches one pack described by a PackSpec and returns everything needed to catalog it: the pack's own Source metadata, its Items, and the raw archive bytes to persist under sources/<id>.zip.

type Item

type Item struct {
	Name      string `json:"name"`
	Path      string `json:"path"`
	Format    string `json:"format,omitempty"`
	MediaType string `json:"mediatype,omitempty"`
	Bytes     int64  `json:"bytes,omitempty"`
	Hash      string `json:"hash,omitempty"`
	Title     string `json:"title,omitempty"`
	// ID is the canonical stable global id: "assetsdb:<sourceid>/<path>", with a "#<subtexture>"
	// suffix for atlas sub-sprites.
	ID string `json:"x_assetsdb:id"`
	// Source is the owning Source's name, resolvable via DB.SourceByID.
	Source string   `json:"x_assetsdb:source"`
	Kind   Kind     `json:"x_assetsdb:kind"`
	Tokens []string `json:"x_assetsdb:tokens,omitempty"`
	// Region is set only for an atlas sub-sprite; nil for a whole-file item.
	Region *Region `json:"x_assetsdb:region,omitempty"`
	// Provenance is RESERVED for phase-2 generated sources; see the Provenance type doc. Always
	// nil for curated items.
	Provenance *Provenance `json:"x_assetsdb:provenance,omitempty"`
}

Item is a single catalogued asset file, stored as a Frictionless Data Resource in DataPackage.Resources. An Item never carries its own license: callers resolve the effective license via DB.LicenseFor, which walks the Frictionless inheritance chain from Source to package default.

type Kind

type Kind string

Kind enumerates the asset categories assetsdb classifies files into. There is deliberately no catch-all "other" kind: files with an unrecognized or ignored extension are skipped by Classify rather than bucketed, so the set of kinds stays exhaustive and meaningful.

const (
	// KindSprite2D is a raster or vector 2D image: a sprite, spritesheet, or UI texture.
	KindSprite2D Kind = "sprite2d"
	// KindModel3D is a 3D model or scene file (glTF/GLB, FBX, OBJ, and similar formats).
	KindModel3D Kind = "model3d"
	// KindAudio is a sound effect or music file.
	KindAudio Kind = "audio"
	// KindFont is a font file.
	KindFont Kind = "font"
)

func Classify

func Classify(path string) (kind Kind, ok bool)

Classify maps path's file extension to an asset Kind. ok is false for ignored or unrecognized extensions (license/readme sidecars, atlas/material descriptors, OS cruft, unknown formats) — callers skip those files rather than bucketing them into a catch-all kind.

type License

type License struct {
	Name  string `json:"name,omitempty"`
	Path  string `json:"path,omitempty"`
	Title string `json:"title,omitempty"`
}

License is a Frictionless Data Package license entry. At least one of Name or Path should be set; Name carries an SPDX identifier (e.g. "CC0-1.0") and Title is always optional.

func LicenseForSPDX

func LicenseForSPDX(spdx string) License

LicenseForSPDX returns the Frictionless license entry for an SPDX identifier. A known id (e.g. "CC0-1.0") resolves to its full name/path/title entry; any other non-empty id yields a name-only entry so the declared license is still recorded; the empty string yields the zero License.

type NameAllocator

type NameAllocator struct {
	// contains filtered or unexported fields
}

NameAllocator hands out deterministic, unique Frictionless resource names within one pack: repeated calls with the same base receive "-2", "-3", ... suffixes on collision.

func NewNameAllocator

func NewNameAllocator() *NameAllocator

NewNameAllocator returns an empty NameAllocator ready to Allocate names.

func (*NameAllocator) Allocate

func (a *NameAllocator) Allocate(base string) string

Allocate returns base unchanged the first time it is seen, and base suffixed with "-N" for the Nth occurrence thereafter (base-2, base-3, ...).

type PackSpec

type PackSpec struct {
	// ID is the pack's source id, used as its Source.Name and as sources/<ID>.zip.
	ID string
	// Name is the pack's human-readable title.
	Name string
	// Source selects which Ingester handles this pack (e.g. "kenney", "kaykit").
	Source string
	// URL is the upstream pack page or repository to fetch from.
	URL string
	// Theme is a free-text category used for tagging, e.g. "space", "dungeon".
	Theme string
	// Tags are additional search/category tags carried onto the resulting Source.
	Tags []string
	// License is the SPDX identifier to record for the pack (e.g. "CC0-1.0").
	License string
}

PackSpec is one entry in the packs manifest, describing a single curated asset pack to fetch and catalog.

type Provenance

type Provenance struct {
	Generator        string            `json:"generator,omitempty"`
	Prompt           string            `json:"prompt,omitempty"`
	Model            string            `json:"model,omitempty"`
	Params           map[string]string `json:"params,omitempty"`
	CreatedAt        string            `json:"createdAt,omitempty"`
	LicenseAssertion string            `json:"licenseAssertion,omitempty"`
}

Provenance is RESERVED for phase-2 locally-generated assets. No v1 code path constructs a non-nil Provenance: every curated item leaves Item.Provenance nil. The type is defined now so the JSON schema and Go API are stable ahead of that later phase.

type Region

type Region struct {
	X      int `json:"x"`
	Y      int `json:"y"`
	Width  int `json:"width"`
	Height int `json:"height"`
}

Region is an atlas sub-region within a larger spritesheet image, in pixel coordinates. It is set on an Item only when that item is a SubTexture cut from a Kenney-style texture atlas.

type Source

type Source struct {
	Name        string    `json:"name"`
	Title       string    `json:"title"`
	Description string    `json:"description,omitempty"`
	Path        string    `json:"path"`
	Licenses    []License `json:"licenses,omitempty"`
	// Type is "curated" or "generated".
	Type string `json:"x_assetsdb:type"`
	// Origin is the upstream URL or repository the pack was fetched from.
	Origin string `json:"x_assetsdb:origin,omitempty"`
	// Tags are theme/category tags carried over from the packs manifest.
	Tags []string `json:"x_assetsdb:tags,omitempty"`
}

Source is one curated or generated asset pack, registered in DataPackage's x_assetsdb:sources. It mirrors a DCAT dataset: a named, licensed collection of items whose bytes are stored at Path.

func CuratedSource

func CuratedSource(spec PackSpec, license License) Source

CuratedSource builds the Source metadata for a curated pack fetched from spec and stored at sources/<id>.zip. The license is recorded only when non-zero, and spec.Theme is folded to the front of spec.Tags (deduplicated if it already appears) so the theme is searchable as a tag.