metaobserve

README

MetaObserve

A unified Go library for LLM and ML observability. MetaObserve provides a vendor-agnostic abstraction layer that enables you to instrument your AI applications once and seamlessly switch between different observability backends without code changes.

Features

• 🔗 Unified Interface: Single API for tracing, evaluation, prompts, and datasets across all providers
• 🔄 Provider Agnostic: Switch between Opik, Langfuse, and Phoenix without changing your code
• 🔍 Full Tracing: Trace LLM calls with spans, token usage, and cost tracking
• 📊 Evaluation Support: Run metrics and add feedback scores to traces
• 📦 Dataset Management: Create and manage evaluation datasets
• 📝 Prompt Versioning: Store and version prompt templates (provider-dependent)
• 🔀 Context Propagation: Automatic trace/span context propagation via context.Context
• ⚙️ Functional Options: Clean, extensible configuration using the options pattern

Installation

go get github.com/grokify/metaobserve

Quick Start

package main

import (
    "context"
    "log"

    "github.com/grokify/metaobserve/llmops"
    _ "github.com/grokify/metaobserve/llmops/opik"  // Register Opik provider
)

func main() {
    // Open a provider
    provider, err := llmops.Open("opik",
        llmops.WithAPIKey("your-api-key"),
        llmops.WithProjectName("my-project"),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer provider.Close()

    ctx := context.Background()

    // Start a trace
    ctx, trace, err := provider.StartTrace(ctx, "chat-workflow",
        llmops.WithTraceInput(map[string]any{"query": "Hello, world!"}),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer trace.End()

    // Start a span for the LLM call
    ctx, span, err := provider.StartSpan(ctx, "gpt-4-completion",
        llmops.WithSpanType(llmops.SpanTypeLLM),
        llmops.WithModel("gpt-4"),
        llmops.WithProvider("openai"),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Record the LLM interaction
    span.SetInput(map[string]any{
        "messages": []map[string]string{
            {"role": "user", "content": "Hello!"},
        },
    })

    // ... call your LLM here ...

    span.SetOutput(map[string]any{
        "response": "Hello! How can I help you today?",
    })
    span.SetUsage(llmops.TokenUsage{
        PromptTokens:     10,
        CompletionTokens: 8,
        TotalTokens:      18,
    })

    span.End()
    trace.SetOutput(map[string]any{"response": "Hello! How can I help you today?"})
}

Supported Providers

Provider	Package	Description
Opik	llmops/opik	Comet Opik - Open-source, full-featured
Langfuse	llmops/langfuse	Cloud & self-hosted, batch ingestion
Phoenix	llmops/phoenix	Arize Phoenix - OpenTelemetry-based

Provider Capabilities

Feature	Opik	Langfuse	Phoenix
Tracing	✅	✅	✅
Evaluation	✅	✅	✅
Prompts	✅	Partial	❌
Datasets	✅	✅	Partial
Experiments	✅	✅	Partial
Streaming	✅	✅	Planned
Distributed Tracing	✅	❌	✅
Cost Tracking	✅	✅	❌
OpenTelemetry	❌	❌	✅

Architecture

metaobserve/
├── metaobserve.go       # Main package with re-exports
├── llmops/              # LLM observability interfaces
│   ├── llmops.go        # Core interfaces (Provider, Tracer, Evaluator, etc.)
│   ├── trace.go         # Trace and Span interfaces
│   ├── types.go         # Data types (EvalInput, Dataset, Prompt, etc.)
│   ├── options.go       # Functional options
│   ├── provider.go      # Provider registration system
│   ├── errors.go        # Error definitions
│   ├── opik/            # Opik provider adapter
│   ├── langfuse/        # Langfuse provider adapter
│   └── phoenix/         # Phoenix provider adapter
├── integrations/        # Integrations with LLM libraries
│   └── fluxllm/         # FluxLLM observability hook (separate module)
├── mlops/               # ML operations interfaces (experiments, model registry)
└── sdk/                 # Provider-specific SDKs
    ├── langfuse/        # Langfuse Go SDK
    └── phoenix/         # Phoenix Go SDK

Core Interfaces

Provider

The main interface that all observability backends implement:

type Provider interface {
    Tracer           // Trace/span operations
    Evaluator        // Evaluation and feedback
    PromptManager    // Prompt template management
    DatasetManager   // Test dataset management
    ProjectManager   // Project/workspace management
    io.Closer

    Name() string
}

Trace and Span

type Trace interface {
    ID() string
    Name() string
    StartSpan(ctx context.Context, name string, opts ...SpanOption) (context.Context, Span, error)
    SetInput(input any)
    SetOutput(output any)
    SetMetadata(metadata map[string]any)
    AddTag(key, value string)
    AddFeedbackScore(ctx context.Context, name string, score float64, opts ...FeedbackOption) error
    End(opts ...EndOption)
}

type Span interface {
    ID() string
    TraceID() string
    Name() string
    Type() SpanType
    StartSpan(ctx context.Context, name string, opts ...SpanOption) (context.Context, Span, error)
    SetInput(input any)
    SetOutput(output any)
    SetModel(model string)
    SetProvider(provider string)
    SetUsage(usage TokenUsage)
    End(opts ...EndOption)
}

Span Types

const (
    SpanTypeGeneral   SpanType = "general"
    SpanTypeLLM       SpanType = "llm"
    SpanTypeTool      SpanType = "tool"
    SpanTypeRetrieval SpanType = "retrieval"
    SpanTypeAgent     SpanType = "agent"
    SpanTypeChain     SpanType = "chain"
    SpanTypeGuardrail SpanType = "guardrail"
)

Usage Examples

Using Different Providers

// Opik
import _ "github.com/grokify/metaobserve/llmops/opik"
provider, _ := llmops.Open("opik", llmops.WithAPIKey("..."))

// Langfuse
import _ "github.com/grokify/metaobserve/llmops/langfuse"
provider, _ := llmops.Open("langfuse",
    llmops.WithAPIKey("sk-lf-..."),
    llmops.WithEndpoint("https://cloud.langfuse.com"),
)

// Phoenix
import _ "github.com/grokify/metaobserve/llmops/phoenix"
provider, _ := llmops.Open("phoenix",
    llmops.WithEndpoint("http://localhost:6006"),
)

Nested Spans

ctx, trace, _ := provider.StartTrace(ctx, "rag-pipeline")
defer trace.End()

// Retrieval span
ctx, retrievalSpan, _ := provider.StartSpan(ctx, "vector-search",
    llmops.WithSpanType(llmops.SpanTypeRetrieval),
)
// ... perform retrieval ...
retrievalSpan.SetOutput(documents)
retrievalSpan.End()

// LLM span
ctx, llmSpan, _ := provider.StartSpan(ctx, "generate-response",
    llmops.WithSpanType(llmops.SpanTypeLLM),
    llmops.WithModel("gpt-4"),
)
// ... call LLM ...
llmSpan.SetUsage(llmops.TokenUsage{
    PromptTokens:     150,
    CompletionTokens: 50,
    TotalTokens:      200,
})
llmSpan.End()

Adding Feedback Scores

// Add a score to a span
span.AddFeedbackScore(ctx, "relevance", 0.95,
    llmops.WithFeedbackReason("Response directly addressed the query"),
    llmops.WithFeedbackCategory("quality"),
)

// Add a score to a trace
trace.AddFeedbackScore(ctx, "user_satisfaction", 0.8)

Working with Datasets

// Create a dataset
dataset, _ := provider.CreateDataset(ctx, "test-cases",
    llmops.WithDatasetDescription("Test cases for RAG evaluation"),
)

// Add items
provider.AddDatasetItems(ctx, "test-cases", []llmops.DatasetItem{
    {
        Input:    map[string]any{"query": "What is Go?"},
        Expected: map[string]any{"answer": "Go is a programming language..."},
    },
})

Working with Prompts (Opik)

// Create a versioned prompt
prompt, _ := provider.CreatePrompt(ctx, "chat-template",
    `You are a helpful assistant. User: {{.query}}`,
    llmops.WithPromptDescription("Main chat template"),
)

// Get a prompt
prompt, _ := provider.GetPrompt(ctx, "chat-template")

// Render with variables
rendered := prompt.Render(map[string]any{"query": "Hello!"})

Configuration Options

Client Options

llmops.WithAPIKey("...")           // API key for authentication
llmops.WithEndpoint("...")         // Custom endpoint URL
llmops.WithWorkspace("...")        // Workspace/organization name
llmops.WithProjectName("...")      // Default project name
llmops.WithHTTPClient(client)      // Custom HTTP client
llmops.WithTimeout(30 * time.Second)
llmops.WithDisabled(true)          // Disable tracing (no-op mode)
llmops.WithDebug(true)             // Enable debug logging

Trace Options

llmops.WithTraceProject("...")
llmops.WithTraceInput(input)
llmops.WithTraceOutput(output)
llmops.WithTraceMetadata(map[string]any{...})
llmops.WithTraceTags(map[string]string{...})
llmops.WithThreadID("...")

Span Options

llmops.WithSpanType(llmops.SpanTypeLLM)
llmops.WithSpanInput(input)
llmops.WithSpanOutput(output)
llmops.WithSpanMetadata(map[string]any{...})
llmops.WithModel("gpt-4")
llmops.WithProvider("openai")
llmops.WithTokenUsage(usage)
llmops.WithParentSpan(parentSpan)

Error Handling

The library provides typed errors for common conditions:

if errors.Is(err, llmops.ErrMissingAPIKey) {
    // Handle missing API key
}

if llmops.IsNotFound(err) {
    // Handle not found
}

if llmops.IsRateLimited(err) {
    // Handle rate limiting
}

Direct SDK Access

For provider-specific features, you can use the underlying SDKs directly:

import "github.com/grokify/metaobserve/sdk/langfuse"
import "github.com/grokify/metaobserve/sdk/phoenix"

FluxLLM Integration

MetaObserve provides an integration with FluxLLM, a multi-LLM abstraction layer. This allows you to automatically instrument all LLM calls made through FluxLLM with any MetaObserve provider.

go get github.com/grokify/metaobserve/integrations/fluxllm

package main

import (
    "github.com/grokify/fluxllm"
    fluxllmhook "github.com/grokify/metaobserve/integrations/fluxllm"
    "github.com/grokify/metaobserve/llmops"
    _ "github.com/grokify/metaobserve/llmops/opik"
)

func main() {
    // Initialize a MetaObserve provider
    provider, _ := llmops.Open("opik",
        llmops.WithAPIKey("your-api-key"),
        llmops.WithProjectName("my-project"),
    )
    defer provider.Close()

    // Create the observability hook
    hook := fluxllmhook.NewHook(provider)

    // Attach to your FluxLLM client
    client := fluxllm.NewClient(
        fluxllm.WithObservabilityHook(hook),
    )

    // All LLM calls through this client are now automatically traced
}

The hook automatically captures:

• Model and provider information
• Input messages and output responses
• Token usage (prompt, completion, total)
• Streaming responses
• Errors

Requirements

• Go 1.24.5 or later

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

See LICENSE for details.

Documentation

Overview

Package metaobserve provides a unified interface for LLM and ML observability platforms.

This library abstracts common functionality across providers like:

• Comet Opik
• Arize Phoenix
• Langfuse
• Lunary
• MLflow
• Weights & Biases

Quick Start

Import the provider you want to use:

import (
	"github.com/grokify/metaobserve/llmops"
	_ "github.com/grokify/metaobserve/llmops/opik"    // Register Opik
	// or
	_ "github.com/grokify/metaobserve/llmops/langfuse" // Register Langfuse
	// or
	_ "github.com/grokify/metaobserve/llmops/phoenix"  // Register Phoenix
)

Then open a provider:

provider, err := llmops.Open("opik", llmops.WithAPIKey("..."))
if err != nil {
	log.Fatal(err)
}
defer provider.Close()

Start tracing:

ctx, trace, _ := provider.StartTrace(ctx, "my-workflow")
defer trace.End()

ctx, span, _ := provider.StartSpan(ctx, "llm-call",
	llmops.WithSpanType(llmops.SpanTypeLLM),
	llmops.WithModel("gpt-4"),
)
defer span.End()

Architecture

The library is organized into two main packages:

• llmops: LLM observability (traces, spans, evaluations, prompts)
• mlops: ML operations (experiments, model registry, artifacts)

Each package defines interfaces that providers implement. Provider-specific implementations are in subpackages (e.g., llmops/opik, llmops/langfuse).

Providers

Available LLM observability providers:

• opik: Comet Opik (open-source, self-hosted)
• langfuse: Langfuse (open-source, cloud & self-hosted)
• phoenix: Arize Phoenix (open-source, uses OpenTelemetry)

Features

Common features across providers:

• Trace/span creation and context propagation
• Input/output capture
• Token usage and cost tracking
• Feedback scores and evaluations
• Dataset management
• Prompt versioning (provider-dependent)

SDK Access

For provider-specific features, you can use the underlying SDKs directly:

import "github.com/grokify/metaobserve/sdk/langfuse"
import "github.com/grokify/metaobserve/sdk/phoenix"

Or use the existing Opik SDK:

import "github.com/grokify/go-comet-ml-opik"

Index

• Constants
• Variables
• func OpenLLMOps(name string, opts ...llmops.ClientOption) (llmops.Provider, error)
• func Providers() []string
• type EvalInput
• type EvalResult
• type Experiment
• type MLProvider
• type Model
• type Provider
• type Run
• type Span
• type SpanType
• type TokenUsage
• type Trace

Constants

const (
	SpanTypeGeneral   = llmops.SpanTypeGeneral
	SpanTypeLLM       = llmops.SpanTypeLLM
	SpanTypeTool      = llmops.SpanTypeTool
	SpanTypeRetrieval = llmops.SpanTypeRetrieval
	SpanTypeAgent     = llmops.SpanTypeAgent
	SpanTypeChain     = llmops.SpanTypeChain
	SpanTypeGuardrail = llmops.SpanTypeGuardrail
)

Span type constants.

const Version = "0.1.0"

Version is the library version.

Variables

var (
	// Client options
	WithAPIKey      = llmops.WithAPIKey
	WithEndpoint    = llmops.WithEndpoint
	WithWorkspace   = llmops.WithWorkspace
	WithProjectName = llmops.WithProjectName
	WithHTTPClient  = llmops.WithHTTPClient
	WithTimeout     = llmops.WithTimeout
	WithDisabled    = llmops.WithDisabled
	WithDebug       = llmops.WithDebug

	// Trace options
	WithTraceProject  = llmops.WithTraceProject
	WithTraceInput    = llmops.WithTraceInput
	WithTraceOutput   = llmops.WithTraceOutput
	WithTraceMetadata = llmops.WithTraceMetadata
	WithTraceTags     = llmops.WithTraceTags
	WithThreadID      = llmops.WithThreadID

	// Span options
	WithSpanType     = llmops.WithSpanType
	WithSpanInput    = llmops.WithSpanInput
	WithSpanOutput   = llmops.WithSpanOutput
	WithSpanMetadata = llmops.WithSpanMetadata
	WithSpanTags     = llmops.WithSpanTags
	WithModel        = llmops.WithModel
	WithProvider     = llmops.WithProvider
	WithTokenUsage   = llmops.WithTokenUsage

	// End options
	WithEndOutput   = llmops.WithEndOutput
	WithEndMetadata = llmops.WithEndMetadata
	WithEndError    = llmops.WithEndError
)

Re-export option functions for convenience.

Functions

func OpenLLMOps

func OpenLLMOps(name string, opts ...llmops.ClientOption) (llmops.Provider, error)

OpenLLMOps opens an LLM observability provider. This is a convenience function that wraps llmops.Open.

func Providers

func Providers() []string

Providers returns the names of registered LLM providers.

Types

type EvalInput

type EvalInput = llmops.EvalInput

EvalInput is an alias for llmops.EvalInput.

type EvalResult

type EvalResult = llmops.EvalResult

EvalResult is an alias for llmops.EvalResult.

type Experiment

type Experiment = mlops.Experiment

Experiment is an alias for mlops.Experiment.

type MLProvider

type MLProvider = mlops.Provider

MLProvider is an alias for mlops.Provider.

type Model

type Model = mlops.Model

Model is an alias for mlops.Model.

type Provider

type Provider = llmops.Provider

Provider is an alias for llmops.Provider.

type Run

type Run = mlops.Run

Run is an alias for mlops.Run.

type Span

type Span = llmops.Span

Span is an alias for llmops.Span.

type SpanType

type SpanType = llmops.SpanType

SpanType is an alias for llmops.SpanType.

type TokenUsage

type TokenUsage = llmops.TokenUsage

TokenUsage is an alias for llmops.TokenUsage.

type Trace

type Trace = llmops.Trace

Trace is an alias for llmops.Trace.