filetree

README

DBBuilder

A tool for building cat photo databases using bbolt key-value storage with hierarchical file storage.

Architecture

DBBuilder creates a database with the following structure:

• meta: bbolt database file containing metadata

  • Bucket: cat_photos
  • Keys: 16-byte binary (cat_id + photo_id, big-endian)
  • Values: empty (reserved for future use)

• data/: Hierarchical directory structure for photo files

  • Path format: data/xx/filename
  • Filename: SHA256 hash of the cat_id,photo_id key (hex format)
  • xx: First 2 characters of filename

Usage

cd dbbuilder
go run . -src=<source_directory> -db=<database_directory>

Parameters

• -src: Source directory containing photo files (required)
• -db: Database directory path (default: ./catdb)

Example

go run . -src=../testphotos -db=../mydb

Photo Filename Format

Photos must follow the naming convention: <cat_id>_<photo_id>.jpg

Examples:

• 1_1.jpg → cat_id=1, photo_id=1
• 2_5.jpg → cat_id=2, photo_id=5
• 10_123.jpg → cat_id=10, photo_id=123

Files not matching this pattern will be skipped.

Output

The tool creates:

1. Database directory with:

   • meta file (bbolt database)
   • data/ directory tree with photo files

2. Progress logging showing:

   • Files processed/skipped
   • Photo storage locations
   • Final statistics

Example Output Structure

mydb/
├── meta                    # bbolt database
└── data/
    ├── 53/
    │   └── 532deabf...  # SHA256 filename
    ├── 8c/
    │   └── 8c7654ec...
    └── ...

Database Inspection

Use bbolt CLI to inspect the database:

# Install bbolt CLI
go install go.etcd.io/bbolt/cmd/bbolt@latest

# List buckets
~/go/bin/bbolt buckets mydb/meta

# List keys
~/go/bin/bbolt keys mydb/meta cat_photos

# Show statistics
~/go/bin/bbolt stats mydb/meta

Notes

• The tool processes files sequentially
• Database directory is created if it doesn't exist
• Existing photos are overwritten without warning

Documentation

Index

• type FileTreeDB
  • func New(dbDir string) (*FileTreeDB, error)
  • func NewReader(dbDir string) (*FileTreeDB, error)
  • func (w *FileTreeDB) AddPhoto(catID, photoID uint64, photoData []byte) error
  • func (w *FileTreeDB) AddPhotosBatch(photos []manul.PhotoItem) error
  • func (w *FileTreeDB) Close() error
  • func (w *FileTreeDB) GetAllCatIDs() ([]uint64, error)
  • func (w *FileTreeDB) GetPhotoData(catID, photoID uint64) ([]byte, error)
  • func (w *FileTreeDB) GetPhotoIDs(catID uint64) ([]uint64, error)

Types

type FileTreeDB

type FileTreeDB struct {
	// contains filtered or unexported fields
}

FileTreeDB implements DBWriter interface using bbolt for metadata and filesystem for photos

func New

func New(dbDir string) (*FileTreeDB, error)

New creates a new FileTreeDB for writing

func NewReader

func NewReader(dbDir string) (*FileTreeDB, error)

NewReader creates a new FileTreeDB for reading (read-only mode)

func (*FileTreeDB) AddPhoto

func (w *FileTreeDB) AddPhoto(catID, photoID uint64, photoData []byte) error

func (*FileTreeDB) AddPhotosBatch

func (w *FileTreeDB) AddPhotosBatch(photos []manul.PhotoItem) error

func (*FileTreeDB) Close

func (w *FileTreeDB) Close() error

func (*FileTreeDB) GetAllCatIDs

func (w *FileTreeDB) GetAllCatIDs() ([]uint64, error)

func (*FileTreeDB) GetPhotoData

func (w *FileTreeDB) GetPhotoData(catID, photoID uint64) ([]byte, error)

func (*FileTreeDB) GetPhotoIDs

func (w *FileTreeDB) GetPhotoIDs(catID uint64) ([]uint64, error)