yclogslog

README

yclogslog

A Go slog.Handler that sends logs directly to Yandex Cloud Logging via gRPC.

Requirements

• Go 1.25+
• Yandex Cloud credentials with the logging.writer role (the Quick Start example uses InstanceServiceAccount, but any SDK-supported auth method works)

Installation

go get github.com/davidmz/yclogslog

Quick Start

package main

import (
    "context"
    "log/slog"
    "os"
    "os/signal"

    "github.com/davidmz/yclogslog"
    ycsdk "github.com/yandex-cloud/go-sdk/v2"
    "github.com/yandex-cloud/go-sdk/v2/credentials"
    "github.com/yandex-cloud/go-sdk/v2/pkg/options"
)

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    sdk, err := ycsdk.Build(ctx,
        options.WithCredentials(credentials.InstanceServiceAccount()),
    )
    if err != nil {
        panic(err)
    }
    defer sdk.Shutdown(ctx)

    handler, err := yclogslog.NewHandler(ctx, sdk, yclogslog.Options{
        FolderID: "b1gxxxxxxxxxx", // or LogGroupID: "e23xxxxxxxxxx"
        Service:  "my-service",
    })
    if err != nil {
        panic(err)
    }
    defer handler.Close(context.Background())

    slog.SetDefault(slog.New(handler))

    slog.Info("server started", "port", 8080)
    slog.Warn("high memory", "percent", 92.5)

    logger := slog.Default().WithGroup("db")
    logger.Info("query", "table", "users", "rows", 42)
}

Destination

You must set exactly one of:

• FolderID — writes to the default log group of the folder
• LogGroupID — writes to a specific log group

Configuration

All fields except FolderID/LogGroupID and Service have sensible defaults:

Option	Default	Description
MinLevel	slog.LevelInfo	Minimum log level
QueueSize	10000	Max entries buffered in memory
BatchMaxEntries	100	Max entries per batch
FlushInterval	1s	Max time before sending a partial batch
ShutdownFlushTimeout	5s	Max time to flush on Close
RetryMaxElapsed	30s	Max retry duration per batch
RetryInitialBackoff	100ms	Initial retry delay
RetryMaxBackoff	5s	Max retry delay
GRPCTimeout	10s	Timeout per Write RPC
DropPolicy	DropNewest	What to drop when queue is full
DebugLog	no-op logger	*slog.Logger for internal diagnostics
IngestionEndpoint	ingester.logging.yandexcloud.net:443	gRPC endpoint for LogIngestionService

Field Mapping

slog	Cloud Logging
record.Message	IncomingLogEntry.Message
record.Time	IncomingLogEntry.Timestamp
slog.LevelDebug	LogLevel_DEBUG
slog.LevelInfo	LogLevel_INFO
slog.LevelWarn	LogLevel_WARN
slog.LevelError	LogLevel_ERROR
slog attributes	IncomingLogEntry.JsonPayload (as google.protobuf.Struct)

Attributes from WithAttrs and WithGroup are nested correctly in json_payload.

Observability

stats := handler.Stats()
fmt.Printf("sent=%d dropped=%d failed=%d retried=%d\n",
    stats.SentEntries, stats.DroppedEntries,
    stats.FailedBatches, stats.RetriedBatches)

Limitations

• Log loss is possible when the internal queue overflows (controlled by DropPolicy)
• During prolonged Cloud Logging unavailability, batches will be retried up to RetryMaxElapsed and then dropped
• The handler does not close the SDK — the caller is responsible for SDK lifecycle
• DropOldest policy is defined but not yet implemented (rejected at validation)
• int64/uint64 precision: values > 2^53 lose precision when stored in json_payload (limitation of protobuf Struct / JSON numbers)

Documentation

Overview

Package yclogslog provides a log/slog.Handler implementation that sends log entries to Yandex Cloud Logging via the gRPC LogIngestionService/Write API.

The handler buffers entries in memory, batches them, and sends asynchronously with retry on transient errors. It is designed for long-running services running on Yandex Cloud Compute / COI instances.

Basic usage:

sdk, err := ycsdk.Build(ctx,
    options.WithCredentials(credentials.InstanceServiceAccount()),
)
// handle err

handler, err := yclogslog.NewHandler(ctx, sdk, yclogslog.Options{
    FolderID: "b1gxxxxxxxxxx",
    Service:  "my-service",
})
// handle err
defer handler.Close(context.Background())

slog.SetDefault(slog.New(handler))
slog.Info("server started", "port", 8080)

Index

• type DropPolicy
• type Handler
  • func NewHandler(ctx context.Context, sdk *ycsdk.SDK, opts Options) (*Handler, error)
  • func (h *Handler) Close(ctx context.Context) error
  • func (h *Handler) Enabled(_ context.Context, level slog.Level) bool
  • func (h *Handler) Handle(_ context.Context, record slog.Record) error
  • func (h *Handler) Stats() StatsSnapshot
  • func (h *Handler) WithAttrs(attrs []slog.Attr) slog.Handler
  • func (h *Handler) WithGroup(name string) slog.Handler
• type Options
• type Stats
  • func (s *Stats) Snapshot() StatsSnapshot
• type StatsSnapshot

Types

type DropPolicy

type DropPolicy int

DropPolicy defines the behavior when the internal queue is full.

const (
	// DropNewest discards the incoming entry when the queue is full.
	DropNewest DropPolicy = iota
	// DropOldest discards the oldest entry in the queue to make room for the new one.
	DropOldest
)

type Handler

type Handler struct {
	// contains filtered or unexported fields
}

Handler implements slog.Handler and sends log entries to Yandex Cloud Logging.

func NewHandler

func NewHandler(ctx context.Context, sdk *ycsdk.SDK, opts Options) (*Handler, error)

NewHandler creates a new Handler that sends logs to Yandex Cloud Logging using the provided SDK instance. The caller owns the SDK and is responsible for closing it. The context is used to resolve the gRPC connection.

func (*Handler) Close

func (h *Handler) Close(ctx context.Context) error

Close flushes pending entries, stops the background worker, and closes the gRPC connection. The provided context controls the maximum time to wait for the flush. Close does NOT close the underlying SDK.

func (*Handler) Enabled

func (h *Handler) Enabled(_ context.Context, level slog.Level) bool

Enabled reports whether the handler handles records at the given level.

func (*Handler) Handle

func (h *Handler) Handle(_ context.Context, record slog.Record) error

Handle processes the log record: converts it and enqueues for sending.

func (*Handler) Stats

func (h *Handler) Stats() StatsSnapshot

Stats returns a snapshot of the handler's internal counters.

func (*Handler) WithAttrs

func (h *Handler) WithAttrs(attrs []slog.Attr) slog.Handler

WithAttrs returns a new Handler with the given attributes pre-appended.

func (*Handler) WithGroup

func (h *Handler) WithGroup(name string) slog.Handler

WithGroup returns a new Handler with the given group name appended.

type Options

type Options struct {
	// Destination: exactly one of FolderID or LogGroupID must be set.
	FolderID   string
	LogGroupID string

	// Service is a required label identifying the application.
	Service string

	// InstanceID is an optional identifier for the running instance.
	InstanceID string

	// DefaultPayload is a set of key-value pairs merged into json_payload of every log entry.
	// Conflicts are resolved in favor of the entry's own attributes.
	DefaultPayload map[string]string

	// MinLevel sets the minimum log level. Entries below this level are discarded.
	// Default: slog.LevelInfo (zero value).
	MinLevel slog.Level

	// QueueSize is the maximum number of entries buffered in memory.
	// Default: 10000.
	QueueSize int

	// BatchMaxEntries is the maximum number of entries per batch sent to Cloud Logging.
	// Default: 100.
	BatchMaxEntries int

	// FlushInterval is the maximum time to wait before sending a non-full batch.
	// Default: 1s.
	FlushInterval time.Duration

	// ShutdownFlushTimeout is the maximum time to wait for pending entries to be sent on Close.
	// Default: 5s.
	ShutdownFlushTimeout time.Duration

	// RetryMaxElapsed is the maximum total time spent retrying a single batch.
	// Default: 30s.
	RetryMaxElapsed time.Duration

	// RetryInitialBackoff is the initial delay before the first retry.
	// Default: 100ms.
	RetryInitialBackoff time.Duration

	// RetryMaxBackoff is the maximum delay between retries.
	// Default: 5s.
	RetryMaxBackoff time.Duration

	// DropPolicy defines the behavior when the queue is full.
	// Default: DropNewest.
	DropPolicy DropPolicy

	// GRPCTimeout is the timeout for a single Write RPC call.
	// Default: 10s.
	GRPCTimeout time.Duration

	// DebugLog is an optional logger for internal diagnostics (send errors,
	// retries, dropped entries, etc.). If nil, a no-op logger is used.
	DebugLog *slog.Logger

	// IngestionEndpoint overrides the gRPC endpoint for LogIngestionService.
	// Default: "ingester.logging.yandexcloud.net:443".
	IngestionEndpoint string
}

Options configures the Handler.

type Stats

type Stats struct {
	SentEntries    atomic.Int64
	DroppedEntries atomic.Int64
	FailedBatches  atomic.Int64
	RetriedBatches atomic.Int64
}

Stats holds atomic counters for observability.

func (*Stats) Snapshot

func (s *Stats) Snapshot() StatsSnapshot

Snapshot returns a point-in-time copy of the stats.

type StatsSnapshot

type StatsSnapshot struct {
	SentEntries    int64
	DroppedEntries int64
	FailedBatches  int64
	RetriedBatches int64
}

StatsSnapshot is a point-in-time copy of Stats values.