quickwit

README

A Go ingestion client for Quickwit

A Quickwit push client for Go. See slog-quickwit for a slog handler implementation.

If you're looking for a search library or Quickwit management interface, check the official library.

🚀 Install

go get github.com/samber/go-quickwit

This library is v0 and follows SemVer strictly. Some breaking changes might be made to exported APIs before v1.0.0.

💡 Spec

GoDoc: https://pkg.go.dev/github.com/samber/go-quickwit

type Config struct {
	URL    string
	Client http.Client

	IndexID string

	BatchWait  time.Duration
	BatchBytes int
	Commit     CommitMode   // either quickwit.Auto, quickwit.WaitFor or quickwit.Force

	BackoffConfig BackoffConfig
	Timeout       time.Duration
}

type BackoffConfig struct {
	// start backoff at this level
	MinBackoff time.Duration
	// increase exponentially to this level
	MaxBackoff time.Duration
	// give up after this many; zero means infinite retries
	MaxRetries int
}

Example

First, start Quickwit:

docker-compose up -d
curl -X POST \
    'http://localhost:7280/api/v1/indexes' \
    -H 'Content-Type: application/yaml' \
    --data-binary @test-config.yaml

Then push logs:

import "github.com/samber/go-quickwit"

func main() {
	client := quickwit.NewWithDefault("http://localhost:7280", "my-index")
	defer client.Stop() // flush and stop

	for i := 0; i < 10; i++ {
		client.Push(map[string]any{
			"timestamp": time.Now().Unix(),
			"message":   fmt.Sprintf("hello %d", i),
		})
		time.Sleep(1 * time.Second)
	}
}

🤝 Contributing

• Ping me on Twitter @samuelberthe (DMs, mentions, whatever :))
• Fork the project
• Fix open issues or request new features

Don't hesitate ;)

# start quickwit
docker-compose up -d
curl -X POST \
    'http://localhost:7280/api/v1/indexes' \
    -H 'Content-Type: application/yaml' \
    --data-binary @test-config.yaml

# Install some dev dependencies
make tools

# Run tests
make test
# or
make watch-test

👤 Contributors

💫 Show your support

Give a ⭐️ if this project helped you!

📝 License

Copyright © 2024 Samuel Berthe.

This project is MIT licensed.

Documentation

Index

• Constants
• type BackoffConfig
• type Client
  • func New(cfg Config) *Client
  • func NewWithDefault(url string, indexID string) *Client
  • func (c *Client) Push(record record)
  • func (c *Client) Stop()
• type CommitMode
• type Config
  • func NewDefaultConfig(url string, indexID string) Config

Constants

const (
	DefaultBatchWait  time.Duration = 3 * time.Second
	DefaultBatchBytes int           = 10 * 1024 * 1024 // 10 MB
	DefautCommitMode                = Auto

	DefaultMinBackoff time.Duration = 500 * time.Millisecond
	DefaultMaxBackoff time.Duration = 5 * time.Minute
	DefaultMaxRetries int           = 5

	DefaultTimeout time.Duration = 10 * time.Second
)

Types

type BackoffConfig

type BackoffConfig struct {
	// start backoff at this level
	MinBackoff time.Duration
	// increase exponentially to this level
	MaxBackoff time.Duration
	// give up after this many; zero means infinite retries
	MaxRetries int
}

type Client

type Client struct {
	// contains filtered or unexported fields
}

func New

func New(cfg Config) *Client

New makes a new Client from config

func NewWithDefault

func NewWithDefault(url string, indexID string) *Client

NewWithDefault creates a new client with default configuration.

func (*Client) Push

func (c *Client) Push(record record)

Push adds a new record to the next batch. Delivery is async.

func (*Client) Stop

func (c *Client) Stop()

Stop the client.

type CommitMode

type CommitMode string

const (
	Auto    CommitMode = "auto"
	WaitFor CommitMode = "wait_for"
	Force   CommitMode = "force"
)

type Config

type Config struct {
	URL    string
	Client http.Client

	IndexID string

	BatchWait  time.Duration
	BatchBytes int
	Commit     CommitMode

	BackoffConfig BackoffConfig
	Timeout       time.Duration
}

Config describes configuration for a HTTP pusher client.

func NewDefaultConfig

func NewDefaultConfig(url string, indexID string) Config

NewDefaultConfig creates a default configuration for a given Quickwit cluster.