media

package
v0.0.0-...-c7587b1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 7, 2026 License: MIT Imports: 17 Imported by: 0

Documentation

Overview

Package media provides utilities for handling media files including MIME type detection, extension mapping, and size limits.

Package media provides media processing and understanding capabilities.

Index

Constants

View Source 
const (
	MaxImageBytes    = 6 * 1024 * 1024   // 6MB
	MaxAudioBytes    = 16 * 1024 * 1024  // 16MB
	MaxVideoBytes    = 16 * 1024 * 1024  // 16MB
	MaxDocumentBytes = 100 * 1024 * 1024 // 100MB
)

Size limits for various media types.

View Source 
const (
	DefaultScreenshotMaxSide  = 2000
	DefaultScreenshotMaxBytes = 5 * 1024 * 1024 // 5MB
)

Default limits for browser screenshots

Variables

This section is empty.

Functions

func BuildMediaNote 

func BuildMediaNote(ctx MediaContext) string

BuildMediaNote constructs a media note from the given context. It handles arrays of paths/urls/types, single values, and suppressed indices. Returns empty string if no attachments are present or all are suppressed.

For a single attachment: "[media attached: path]" For multiple attachments: 

"[media attached: N files]
[media attached 1/N: path1]
[media attached 2/N: path2]
..."

func DetectMIME 

func DetectMIME(data []byte, filename string, headerMIME string) string

DetectMIME attempts to detect the MIME type from various sources. It prefers: sniffed content > extension mapping > header MIME type.

func ExtensionFromMIME 

func ExtensionFromMIME(mime string) string

ExtensionFromMIME returns the preferred file extension for a MIME type.

func FormatMediaAttachedLine 

func FormatMediaAttachedLine(path, mediaType, url string, index, total int) string

FormatMediaAttachedLine formats a single media attachment line. If index and total are provided (both > 0), it includes the index/total prefix. Format: "[media attached: path (type) | url]" or "[media attached 1/3: path (type) | url]"

func GetExtension 

func GetExtension(path string) string

GetExtension returns the file extension from a path or URL.

func ImageMIMEFromFormat 

func ImageMIMEFromFormat(format string) string

ImageMIMEFromFormat returns the MIME type for an image format name.

func IsAudioExtension 

func IsAudioExtension(ext string) bool

IsAudioExtension checks if the extension is for an audio file.

func IsAudioFile 

func IsAudioFile(path string) bool

IsAudioFile checks if a file path is an audio file.

func IsGIF 

func IsGIF(mime, filename string) bool

IsGIF checks if the content is a GIF based on MIME or filename.

func IsSupported 

func IsSupported(mimeType string) bool

IsSupported checks if a media type is supported for processing.

func MIMEFromExtension 

func MIMEFromExtension(ext string) string

MIMEFromExtension returns the MIME type for a file extension.

func MaxBytesForKind 

func MaxBytesForKind(kind Kind) int64

MaxBytesForKind returns the maximum size for a media kind.

func ValidateSize 

func ValidateSize(size int64, mime string) bool

ValidateSize checks if a file size is within limits for its MIME type.

Types

type AggregatedContent 

type AggregatedContent struct {
	// Images are base64-encoded images ready for vision models
	Images []Content `json:"images,omitempty"`

	// Text is aggregated text content (transcriptions, descriptions)
	Text string `json:"text,omitempty"`

	// Errors lists any processing errors
	Errors []string `json:"errors,omitempty"`

	// ProcessedCount is the number of successfully processed attachments
	ProcessedCount int `json:"processed_count"`

	// TotalCount is the total number of attachments
	TotalCount int `json:"total_count"`
}

AggregatedContent holds all processed content from attachments.

func (*AggregatedContent) HasContent 

func (c *AggregatedContent) HasContent() bool

HasContent checks if the aggregated content has any usable content.

func (*AggregatedContent) HasErrors 

func (c *AggregatedContent) HasErrors() bool

HasErrors checks if there were any processing errors.

type Aggregator 

type Aggregator struct {
	// contains filtered or unexported fields
}

Aggregator processes multiple attachments concurrently.

func NewAggregator 

func NewAggregator(processor Processor, logger *slog.Logger) *Aggregator

NewAggregator creates a new media aggregator.

func (*Aggregator) Aggregate 

func (a *Aggregator) Aggregate(ctx context.Context, attachments []*Attachment, opts ProcessingOptions) *AggregatedContent

Aggregate processes attachments and aggregates the results.

func (*Aggregator) ProcessAll 

func (a *Aggregator) ProcessAll(ctx context.Context, attachments []*Attachment, opts ProcessingOptions) []*ProcessingResult

ProcessAll processes multiple attachments and returns all results.

func (*Aggregator) SetConcurrency 

func (a *Aggregator) SetConcurrency(n int)

SetConcurrency sets the maximum concurrent processing operations.

type Attachment 

type Attachment struct {
	// ID is a unique identifier for this attachment
	ID string `json:"id"`

	// Type is the media category
	Type MediaType `json:"type"`

	// MimeType is the MIME type (e.g., "image/png")
	MimeType string `json:"mime_type"`

	// Filename is the original filename
	Filename string `json:"filename,omitempty"`

	// Size is the file size in bytes
	Size int64 `json:"size,omitempty"`

	// URL is the remote URL if available
	URL string `json:"url,omitempty"`

	// LocalPath is the local file path if downloaded
	LocalPath string `json:"local_path,omitempty"`

	// Data holds the raw bytes (for small attachments)
	Data []byte `json:"-"`

	// Width/Height for images and videos
	Width  int `json:"width,omitempty"`
	Height int `json:"height,omitempty"`

	// Duration for audio and video
	Duration time.Duration `json:"duration,omitempty"`

	// Metadata holds additional format-specific data
	Metadata map[string]any `json:"metadata,omitempty"`
}

Attachment represents a media attachment from a message.

type Content 

type Content struct {
	// Type identifies how this content should be used
	Type ContentType `json:"type"`

	// Text is text content (transcription, description, etc)
	Text string `json:"text,omitempty"`

	// ImageData is base64-encoded image data
	ImageData string `json:"image_data,omitempty"`

	// ImageMediaType is the MIME type of the image
	ImageMediaType string `json:"image_media_type,omitempty"`

	// Source identifies where this content came from
	Source string `json:"source,omitempty"`
}

Content represents processed media content for LLM consumption.

type ContentType 

type ContentType string

ContentType identifies how processed content should be used. 

const (
	ContentTypeText  ContentType = "text"
	ContentTypeImage ContentType = "image"
)

type DefaultProcessor 

type DefaultProcessor struct {
	// contains filtered or unexported fields
}

DefaultProcessor is the default media processor implementation.

func NewDefaultProcessor 

func NewDefaultProcessor(logger *slog.Logger) *DefaultProcessor

NewDefaultProcessor creates a new default processor.

func (*DefaultProcessor) Process 

func (p *DefaultProcessor) Process(attachment *Attachment, opts ProcessingOptions) (*ProcessingResult, error)

Process processes an attachment.

func (*DefaultProcessor) SetTranscriber 

func (p *DefaultProcessor) SetTranscriber(t Transcriber)

SetTranscriber sets the transcriber for audio processing.

func (*DefaultProcessor) SupportedTypes 

func (p *DefaultProcessor) SupportedTypes() []MediaType

SupportedTypes returns supported media types.

type ImageMetadata 

type ImageMetadata struct {
	Width  int
	Height int
	Format string
}

ImageMetadata contains basic image information

func GetImageMetadata 

func GetImageMetadata(data []byte) (*ImageMetadata, error)

GetImageMetadata extracts image dimensions without full decode

type ImageProcessor 

type ImageProcessor interface {
	// PrepareForVision prepares an image for vision model consumption.
	// Returns base64-encoded image data.
	PrepareForVision(image io.Reader, mimeType string, maxSize int) (string, string, error)
}

ImageProcessor processes images for vision models.

type Kind 

type Kind string

Kind represents the type of media. 

const (
	KindImage    Kind = "image"
	KindAudio    Kind = "audio"
	KindVideo    Kind = "video"
	KindDocument Kind = "document"
	KindUnknown  Kind = "unknown"
)

func KindFromMIME 

func KindFromMIME(mime string) Kind

KindFromMIME returns the media kind based on MIME type.

type MediaAttachment 

type MediaAttachment struct {
	// Path is the file path or identifier for the attachment
	Path string

	// Type is the media type (e.g., "image", "audio", "video")
	Type string

	// URL is the optional URL for the attachment
	URL string
}

MediaAttachment represents a single media attachment with path, type, and URL.

type MediaContext 

type MediaContext struct {
	// MediaPaths is a list of file paths for multiple attachments
	MediaPaths []string

	// MediaUrls is a list of URLs corresponding to MediaPaths
	MediaUrls []string

	// MediaTypes is a list of types corresponding to MediaPaths
	MediaTypes []string

	// MediaPath is a single file path (used when MediaPaths is empty)
	MediaPath string

	// MediaUrl is a single URL (used when MediaUrls is empty)
	MediaUrl string

	// MediaType is a single type (used when MediaTypes is empty)
	MediaType string

	// SuppressedIndices contains indices of attachments that should be excluded
	// (e.g., already processed via media understanding)
	SuppressedIndices map[int]bool
}

MediaContext holds the context for building media notes. It supports both array-based and single-value attachment specifications.

type MediaType 

type MediaType string

MediaType identifies the category of media. 

const (
	MediaTypeImage    MediaType = "image"
	MediaTypeAudio    MediaType = "audio"
	MediaTypeVideo    MediaType = "video"
	MediaTypeDocument MediaType = "document"
	MediaTypeUnknown  MediaType = "unknown"
)

func DetectMediaType 

func DetectMediaType(mimeType, filename string) MediaType

DetectMediaType determines the media type from MIME type or filename.

type ProcessingOptions 

type ProcessingOptions struct {
	// MaxImageSize limits image dimensions (will resize if larger)
	MaxImageSize int

	// MaxFileSize limits file size in bytes
	MaxFileSize int64

	// EnableTranscription enables audio/video transcription
	EnableTranscription bool

	// EnableVision enables vision model processing
	EnableVision bool

	// TranscriptionLanguage for audio transcription
	TranscriptionLanguage string

	// Quality for image processing (1-100)
	Quality int

	// Timeout for processing operations
	Timeout time.Duration

	// AllowedBasePath restricts local file reads to this directory.
	// If empty, local file reads are rejected for security.
	AllowedBasePath string
}

ProcessingOptions configures media processing.

func DefaultOptions 

func DefaultOptions() ProcessingOptions

DefaultOptions returns sensible default processing options.

type ProcessingResult 

type ProcessingResult struct {
	// Attachment is the original attachment
	Attachment *Attachment `json:"attachment"`

	// Contents are the processed contents
	Contents []Content `json:"contents"`

	// Description is a text description of the media
	Description string `json:"description,omitempty"`

	// Transcription is text from audio/video
	Transcription string `json:"transcription,omitempty"`

	// Error is set if processing failed
	Error string `json:"error,omitempty"`

	// ProcessedAt is when processing completed
	ProcessedAt time.Time `json:"processed_at"`

	// ProcessingDuration is how long processing took
	ProcessingDuration time.Duration `json:"processing_duration,omitempty"`
}

ProcessingResult holds the result of processing an attachment.

type Processor 

type Processor interface {
	// Process processes an attachment and returns content for LLM consumption.
	Process(attachment *Attachment, opts ProcessingOptions) (*ProcessingResult, error)

	// SupportedTypes returns the media types this processor handles.
	SupportedTypes() []MediaType
}

Processor handles media processing.

type ScreenshotOptions 

type ScreenshotOptions struct {
	MaxSide  int
	MaxBytes int
}

ScreenshotOptions for normalization

type ScreenshotResult 

type ScreenshotResult struct {
	Buffer      []byte
	ContentType string
	Width       int
	Height      int
	Resized     bool
}

ScreenshotResult from normalization

func NormalizeBrowserScreenshot 

func NormalizeBrowserScreenshot(data []byte, opts *ScreenshotOptions) (*ScreenshotResult, error)

NormalizeBrowserScreenshot resizes and compresses a screenshot to fit limits. It follows the clawdbot browser/screenshot.ts pattern, trying various combinations of size and quality to fit within the specified limits.

type Transcriber 

type Transcriber interface {
	// Transcribe converts audio to text.
	Transcribe(audio io.Reader, mimeType string, language string) (string, error)
}

Transcriber transcribes audio to text.

Source Files

View all Source files

Directories

Path Synopsis
Package transcribe provides audio transcription capabilities using various providers.
 Package transcribe provides audio transcription capabilities using various providers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.