s3

package
v0.0.0-...-82bc909 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 26, 2026 License: MIT Imports: 24 Imported by: 0

README

S3 Tools

eino tools for browsing and analyzing AWS S3 and S3-compatible object storage (MinIO, Cloudian, etc.).

Design

  • Multi-instance — configure multiple named S3 bucket instances via a Configs map, each with its own endpoint, credentials, and description.
  • AWS-compatible — works with AWS S3, MinIO, Cloudian, and any S3-compatible service via custom endpoint configuration.
  • TLS — supports custom CA certificates via CACert and TLS verification skip via TLSSkipVerify.
  • Sorting — list tools support sorting by name (alphanumeric), size (largest first), or last modified date (most recent first).
  • Context descriptions — each bucket instance includes a Description field exposed to LLM agents via s3_list_buckets.
  • Read-only — all tools are read-only.

TLS Configuration

  • TLSSkipVerify disables TLS certificate verification. Use only for local development or trusted internal endpoints. For production, prefer supplying the CA certificate via CACert.
  • CACert is a PEM-encoded CA certificate used to validate the endpoint's TLS certificate. Use it when the endpoint uses a private/internal CA. It is not serialized to JSON.
  • PathStyle forces path-style addressing. Enable it for MinIO/Cloudian or other S3-compatible services that do not support virtual-hosted style. Leave it false (the default) for AWS S3.

Configuration

import "github.com/webcenter-fr/eino-ext/components/tool/s3"

configs := s3.Configs{
    "prod-logs": s3.Config{
        Endpoint:    "https://s3.amazonaws.com",
        BucketName:  "my-logs-bucket",
        AccessKey:   os.Getenv("AWS_ACCESS_KEY_ID"),
        SecretKey:   os.Getenv("AWS_SECRET_ACCESS_KEY"),
        Region:      "us-east-1",
        Description: "Production application logs",
    },
    "minio-backups": s3.Config{
        Endpoint:      "http://minio:9000",
        BucketName:    "backups",
        AccessKey:     "minioadmin",
        SecretKey:     "minioadmin",
        Region:        "us-east-1",
        TLSSkipVerify: true,
        PathStyle:     true,
        Description:   "Backup storage on MinIO",
    },
}

Available Tools

Tool Name Description
s3_list_buckets List all configured bucket instances with names, endpoints, and descriptions
s3_list_objects List objects and directories in a bucket with sorting and filtering
s3_get_usage Compute total storage usage (size + object count) in human-readable units
s3_list_objects_with_size List objects with detailed size information, sorted by size by default
s3_get_lifecycle Retrieve lifecycle configuration to understand data retention/cleanup policies

Factory Functions

// All tools
tools, err := s3.NewAllTools(ctx, configs)

// Read-only tools
tools, err := s3.NewReadOnlyTools(ctx, configs)

// All tools with safety middleware
tools, mw, err := s3.NewAllToolsWithSafety(ctx, configs, safetyCfg)

Tool Details

s3_list_buckets

Lists all configured S3 bucket instances with their descriptions.

No parameters required.

s3_list_objects

Lists directories and/or files in a bucket.

Parameter Required Description
instance Yes S3 bucket instance name
prefix No List only objects with this path prefix
delimiter No Use / to group into directories
max_keys No Max results (1–1000, default 200)
sort_by No alphanumeric, size, or last_modified
filter No Go RE2 regex filter on result JSON
s3_get_usage

Computes total storage usage for a bucket.

Parameter Required Description
instance Yes S3 bucket instance name

Returns total_objects, total_size_bytes, and total_size_human.

s3_list_objects_with_size

Lists objects with detailed size info, sorted by size descending by default.

Parameter Required Description
instance Yes S3 bucket instance name
prefix No List only objects with this path prefix
max_keys No Max results (1–1000, default 200)
sort_by No alphanumeric, size (default), or last_modified
filter No Go RE2 regex filter on result JSON
s3_get_lifecycle

Retrieves lifecycle configuration to check for automatic data expiration or storage tier transitions.

Parameter Required Description
instance Yes S3 bucket instance name

Documentation

Overview

Package s3 provides eino tools for browsing and analyzing AWS S3 and S3-compatible object storage buckets.

Supports AWS S3, MinIO, Cloudian, and other S3-compatible services.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func BuildClients 

func BuildClients(ctx context.Context, configs Configs) (map[string]Client, error)

BuildClients creates S3 Clients for all configurations in the Configs map.

func Check 

func Check(ctx context.Context, configs Configs) checkup.Results

Check runs health checks on each configured S3 instance.

func ExtractWriteToolNames 

func ExtractWriteToolNames(ctx context.Context, configs Configs) ([]string, error)

ExtractWriteToolNames dynamically extracts write tool names. Returns nil since there are no write tools.

func NewAllTools 

func NewAllTools(ctx context.Context, configs Configs) ([]tool.InvokableTool, error)

NewAllTools creates all S3 tools (all read-only).

func NewAllToolsWithSafety 

func NewAllToolsWithSafety(ctx context.Context, configs Configs, safetyCfg *safety.Config) ([]tool.InvokableTool, *safety.Middleware, error)

NewAllToolsWithSafety creates all S3 tools with safety middleware.

func NewReadOnlyTools 

func NewReadOnlyTools(ctx context.Context, configs Configs) ([]tool.InvokableTool, error)

NewReadOnlyTools creates only the read-only S3 tools.

func WriteToolNames 

func WriteToolNames() []string

WriteToolNames returns the names of write tools. All S3 tools are read-only.

Types

type BucketListParams 

type BucketListParams struct{}

BucketListParams holds the parameters for BucketListTool (none required).

type BucketListTool 

type BucketListTool struct {
	tool.InvokableTool
	// contains filtered or unexported fields
}

BucketListTool lists all configured S3 bucket instances.

func NewBucketListTool 

func NewBucketListTool(ctx context.Context, configs Configs) (*BucketListTool, error)

NewBucketListTool creates a new BucketListTool for the given configs.

func (*BucketListTool) Invoke 

func (t *BucketListTool) Invoke(ctx context.Context, params *BucketListParams) (string, error)

Invoke returns the configured bucket instances as a JSON string array.

type Client 

type Client interface {
	ListObjectsV2(ctx context.Context, params *s3sdk.ListObjectsV2Input, optFns ...func(*s3sdk.Options)) (*s3sdk.ListObjectsV2Output, error)
	GetBucketLifecycleConfiguration(ctx context.Context, params *s3sdk.GetBucketLifecycleConfigurationInput, optFns ...func(*s3sdk.Options)) (*s3sdk.GetBucketLifecycleConfigurationOutput, error)
}

Client is the interface for S3 operations used by the tools. It abstracts the AWS SDK to allow mocking in tests.

func NewClient 

func NewClient(ctx context.Context, cfg Config) (Client, error)

NewClient creates a new S3 Client from the given configuration.

type Config 

type Config struct {
	// Endpoint is the S3 server URL (e.g. "https://s3.amazonaws.com", "http://minio:9000").
	// For AWS S3, this can be left empty to use the default regional endpoint.
	Endpoint string `` /* 168-byte string literal not displayed */

	// BucketName is the actual bucket name on the S3 server.
	BucketName string `validate:"required" jsonschema:"description=Actual S3 bucket name, e.g. my-logs-bucket"`

	// AccessKey is the S3 access key ID.
	AccessKey string `json:"-" validate:"required"`

	// SecretKey is the S3 secret access key.
	SecretKey string `json:"-" validate:"required"`

	// Region is the AWS region (e.g. "us-east-1"). For S3-compatible services,
	// this can often be set to "us-east-1" or any value accepted by the server.
	Region string `` /* 136-byte string literal not displayed */

	// PathStyle forces path-style addressing (s3.amazonaws.com/bucket/key) instead
	// of virtual-hosted style (bucket.s3.amazonaws.com/key). Required for MinIO,
	// Cloudian, and other S3-compatible services. Leave false for AWS S3.
	PathStyle bool `` /* 128-byte string literal not displayed */

	// Description provides context about this bucket for LLM agents.
	// This is exposed by the s3_list_buckets tool.
	Description string `jsonschema:"description=Human-readable description of this bucket to help LLM agents understand its purpose."`

	// TLSSkipVerify disables TLS certificate verification.
	TLSSkipVerify bool

	// CACert is a PEM-encoded CA certificate used to validate the endpoint's TLS certificate.
	CACert string `json:"-"`
}

Config holds the connection and identity configuration for a single S3 bucket.

type Configs 

type Configs map[string]Config

Configs is a map of S3 bucket instance configurations, where the key is the logical instance name (exposed to the LLM via s3_list_buckets).

func (Configs) GetConfig 

func (c Configs) GetConfig(instanceName string) Config

GetConfig retrieves the configuration for a given instance name.

func (Configs) GetInstanceNames 

func (c Configs) GetInstanceNames() []string

GetInstanceNames returns a sorted slice of all instance names in the Configs map.

type GetLifecycleParams 

type GetLifecycleParams struct {
	Instance string `json:"instance" validate:"required" jsonschema:"(required) The S3 bucket instance to query."`
}

GetLifecycleParams holds the parameters for GetLifecycleTool.

type GetLifecycleTool 

type GetLifecycleTool struct {
	tool.InvokableTool
	// contains filtered or unexported fields
}

GetLifecycleTool retrieves lifecycle configuration for an S3 bucket.

func NewGetLifecycleTool 

func NewGetLifecycleTool(ctx context.Context, configs Configs) (*GetLifecycleTool, error)

NewGetLifecycleTool creates a new GetLifecycleTool for the given configs.

func (*GetLifecycleTool) Invoke 

func (t *GetLifecycleTool) Invoke(ctx context.Context, params *GetLifecycleParams) (string, error)

Invoke retrieves the lifecycle configuration for the bucket.

type GetUsageParams 

type GetUsageParams struct {
	Instance string `json:"instance" validate:"required" jsonschema:"(required) The S3 bucket instance to query."`
}

GetUsageParams holds the parameters for GetUsageTool.

type GetUsageTool 

type GetUsageTool struct {
	tool.InvokableTool
	// contains filtered or unexported fields
}

GetUsageTool computes total usage for an S3 bucket.

func NewGetUsageTool 

func NewGetUsageTool(ctx context.Context, configs Configs) (*GetUsageTool, error)

NewGetUsageTool creates a new GetUsageTool for the given configs.

func (*GetUsageTool) Invoke 

func (t *GetUsageTool) Invoke(ctx context.Context, params *GetUsageParams) (string, error)

Invoke computes the total storage usage for the bucket.

type ListObjectsParams 

type ListObjectsParams struct {
	Instance  string `json:"instance" validate:"required" jsonschema:"(required) The S3 bucket instance to list."`
	Prefix    string `` /* 127-byte string literal not displayed */
	Delimiter string `` /* 173-byte string literal not displayed */
	MaxKeys   int    `` /* 145-byte string literal not displayed */
	SortBy    string `` /* 229-byte string literal not displayed */
	Filter    string `json:"filter,omitempty" jsonschema:"(optional) A Go RE2 regex applied on each result JSON. Keep only results that match."`
}

ListObjectsParams holds the parameters for ListObjectsTool.

type ListObjectsTool 

type ListObjectsTool struct {
	tool.InvokableTool
	// contains filtered or unexported fields
}

ListObjectsTool lists objects and directories in an S3 bucket.

func NewListObjectsTool 

func NewListObjectsTool(ctx context.Context, configs Configs) (*ListObjectsTool, error)

NewListObjectsTool creates a new ListObjectsTool for the given configs.

func (*ListObjectsTool) Invoke 

func (t *ListObjectsTool) Invoke(ctx context.Context, params *ListObjectsParams) (string, error)

Invoke lists objects in the bucket.

type ListObjectsWithSizeParams 

type ListObjectsWithSizeParams struct {
	Instance string `json:"instance" validate:"required" jsonschema:"(required) The S3 bucket instance to list."`
	Prefix   string `json:"prefix,omitempty" jsonschema:"(optional) List only objects with this prefix (acts as a directory path)."`
	MaxKeys  int    `` /* 145-byte string literal not displayed */
	SortBy   string `` /* 230-byte string literal not displayed */
	Filter   string `json:"filter,omitempty" jsonschema:"(optional) A Go RE2 regex applied on each result JSON. Keep only results that match."`
}

ListObjectsWithSizeParams holds the parameters for ListObjectsWithSizeTool.

type ListObjectsWithSizeTool 

type ListObjectsWithSizeTool struct {
	tool.InvokableTool
	// contains filtered or unexported fields
}

ListObjectsWithSizeTool lists objects with detailed size information.

func NewListObjectsWithSizeTool 

func NewListObjectsWithSizeTool(ctx context.Context, configs Configs) (*ListObjectsWithSizeTool, error)

NewListObjectsWithSizeTool creates a new ListObjectsWithSizeTool for the given configs.

func (*ListObjectsWithSizeTool) Invoke 

func (t *ListObjectsWithSizeTool) Invoke(ctx context.Context, params *ListObjectsWithSizeParams) (string, error)

Invoke lists objects with size details, sorted by size descending by default.

type SortOrder 

type SortOrder string

SortOrder defines how directory/list results are ordered. 

const (
	// SortAlphanumeric sorts entries alphabetically by key.
	SortAlphanumeric SortOrder = "alphanumeric"
	// SortSize sorts entries by size, descending.
	SortSize SortOrder = "size"
	// SortLastModified sorts entries by last modification time, descending.
	SortLastModified SortOrder = "last_modified"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.