Documentation
¶
Overview ¶
Package crdbpgx provides a CockroachDB-aware wrapper around github.com/jackc/pgx/v5/pgxpool.
It is intended for applications that connect to CockroachDB with pgx and want pool-level behavior that is specific to CockroachDB clusters: automatic retry handling for retryable transaction errors, connection reset and reacquisition for node-level failures, node health tracking, and optional connection rebalancing across CockroachDB gateway nodes.
The package is inspired by the mechanisms described in AuthZed's article "Maximizing CockroachDB Performance: Our Journey to 1 Million QPS": https://authzed.com/blog/maximizing-cockroachdb-performance
Pool lifecycle ¶
Create a Pool with NewPool from a pgxpool.Config. The context passed to NewPool is used only while the underlying pgx pool is created. Runtime background work started by this package, such as the default health tracker and the optional node connection balancer, is owned by the Pool and is stopped by Pool.Close.
Always close the pool when it is no longer needed:
defer pool.Close()
Pool.Close starts shutdown once and waits for internal goroutines and the underlying pgx pool to finish closing.
Context propagation ¶
Database operation contexts can affect connection health in pgx: canceling a context while a query is active can leave the connection unusable. crdbpgx exposes WithContextPropagationBehaviour to control how caller contexts are propagated to connection acquisition and database operations.
The default, ContextPropagationDetach, detaches both acquisition and database operations from caller cancellation. ContextPropagationOnAcquire keeps caller cancellation for acquiring a connection, but detaches the actual database operation. ContextPropagationPassthrough passes caller contexts directly to pgx.
Example: create a pool ¶
cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
return fmt.Errorf("parse database URL: %w", err)
}
pool, err := crdbpgx.NewPool(
context.Background(),
cfg,
crdbpgx.WithName("orders"),
crdbpgx.WithContextPropagationBehaviour(crdbpgx.ContextPropagationOnAcquire),
crdbpgx.WithBalancer(crdbpgx.NewNodeConnectionBalancer(
crdbpgx.WithBalancingInterval(30*time.Second),
crdbpgx.WithBalancingTimeout(10*time.Second),
)),
)
if err != nil {
return fmt.Errorf("create pool: %w", err)
}
defer pool.Close()
Example: run a transaction with automatic retries ¶
err := pool.InTx(ctx, pgx.TxOptions{}, func(ctx context.Context, db crdbpgx.Runner) error {
return db.ExecFunc(ctx, func(_ context.Context, _ pgconn.CommandTag, err error) error {
return err
}, "UPDATE accounts SET balance = balance - $1 WHERE id = $2", amount, accountID)
})
if err != nil {
return fmt.Errorf("transfer funds: %w", err)
}
Index ¶
- Constants
- Variables
- func IsResettableError(ctx context.Context, err error) bool
- func IsRetryableError(ctx context.Context, err error) bool
- func NewNodeHealthTracker(cfg *pgx.ConnConfig, opts ...NodeHealthTrackerOption) (_ *nodeHealthTracker, err error)
- func RetriesDisabled(ctx context.Context) bool
- func WithDisableRetries(ctx context.Context) context.Context
- type ContextPropagationBehaviour
- type Jitter
- type JitteredTicker
- type MaxRetryError
- type NodeBalancer
- type NodeConnectionBalancerOption
- type NodeHealthTracker
- type NodeHealthTrackerOption
- type Option
- func WithBalancer(b NodeBalancer) Option
- func WithConnectRate(rate time.Duration) Option
- func WithContextPropagationBehaviour(behavior ContextPropagationBehaviour) Option
- func WithHealthTracker(h NodeHealthTracker) Option
- func WithLogger(l *slog.Logger) Option
- func WithMaxRetries(retries int) Option
- func WithName(name string) Option
- type Pool
- func (p *Pool) Acquire(ctx context.Context) (*pgxpool.Conn, error)
- func (p *Pool) AcquireAllIdle(ctx context.Context) []*pgxpool.Conn
- func (p *Pool) AcquireFunc(ctx context.Context, f func(*pgxpool.Conn) error) error
- func (p *Pool) Close()
- func (p *Pool) Config() *pgxpool.Config
- func (p *Pool) CopyFrom(ctx context.Context, tableName pgx.Identifier, columnNames []string, ...) (int64, error)
- func (p *Pool) ExecFunc(ctx context.Context, ...) error
- func (p *Pool) GCMark(conn *pgx.Conn)
- func (p *Pool) InTx(ctx context.Context, txOptions pgx.TxOptions, ...) error
- func (p *Pool) InTxFunc(ctx context.Context, txOptions pgx.TxOptions, txFunc func(Tx) error) error
- func (p *Pool) MaxConns() int
- func (p *Pool) MinConns() int
- func (p *Pool) Name() string
- func (p *Pool) NodeID(conn *pgx.Conn) uint32
- func (p *Pool) Ping(ctx context.Context) error
- func (p *Pool) QueryFunc(ctx context.Context, rowsFunc func(ctx context.Context, rows pgx.Rows) error, ...) error
- func (p *Pool) QueryRowFunc(ctx context.Context, rowFunc func(ctx context.Context, row pgx.Row) error, ...) error
- func (p *Pool) SendBatch(ctx context.Context, b *pgx.Batch) pgx.BatchResults
- func (p *Pool) Stat() *pgxpool.Stat
- type ResettableError
- type RetryableError
- type Runner
- type Tx
- type UniformJitter
Examples ¶
Constants ¶
const ( // DefaultMaxRetries gives the number of transaction retries for a single transaction. // This is the same value as its used in the official cockroach-go library. // https://github.com/cockroachdb/cockroach-go/blob/master/crdb/tx.go#L114 DefaultMaxRetries = 50 // DefaultConnectRate limits how often the pool opens new database connections. DefaultConnectRate = 100 * time.Millisecond // DefaultContextPropagationBehaviour is the default behavior for context propagation. DefaultContextPropagationBehaviour = ContextPropagationDetach )
const ( // CrdbRetryErrCode is the PostgreSQL error code for transaction retry errors in CockroachDB. // https://www.cockroachlabs.com/docs/stable/common-errors.html#restart-transaction CrdbRetryErrCode = "40001" // CrdbAmbiguousErrorCode is the PostgreSQL error code for ambiguous result errors in CockroachDB. // https://www.cockroachlabs.com/docs/stable/common-errors.html#result-is-ambiguous CrdbAmbiguousErrorCode = "40003" // CrdbServerNotAcceptingClients is the PostgreSQL error code when a CockroachDB node is shutting down. // https://www.cockroachlabs.com/docs/stable/node-shutdown.html#connection-retry-loop CrdbServerNotAcceptingClients = "57P01" // CrdbUnknownSQLState is returned when the SQL state cannot be determined. CrdbUnknownSQLState = "XXUUU" // CrdbClockSkewMessage is the error message when CockroachDB nodes have large clock skew. CrdbClockSkewMessage = "cannot specify timestamp in the future" )
Variables ¶
var CtxDisableRetries = ctxDisableRetries{}
CtxDisableRetries is a context key used to disable automatic retries.
Functions ¶
func IsResettableError ¶
IsResettableError returns true if the error indicates that the connection should be reset and a retry attempted on a different node.
Resettable errors typically indicate node failures, network issues, or ambiguous transaction results. The connection is no longer usable and should be replaced.
func IsRetryableError ¶
IsRetryableError returns true if the error is a CockroachDB retryable error.
Retryable errors are serialization conflicts that can be resolved by retrying the transaction. The connection remains valid and can be reused.
func NewNodeHealthTracker ¶
func NewNodeHealthTracker(cfg *pgx.ConnConfig, opts ...NodeHealthTrackerOption) (_ *nodeHealthTracker, err error)
NewNodeHealthTracker creates a new [nodeHealthTracker].
The tracker periodically probes for healthy nodes in the CockroachDB cluster and tracks their health.
func RetriesDisabled ¶
RetriesDisabled returns true if automatic retries are disabled in the context.
Types ¶
type ContextPropagationBehaviour ¶
type ContextPropagationBehaviour int
ContextPropagationBehaviour controls how contexts are propagated from callers to database operations. This affects context cancellation behavior and is important for preventing unwanted connection closures during operations.
const ( // ContextPropagationPassthrough propagates the caller's context directly to // database operations without modification. This is the standard behavior but // may cause connection closures if contexts are canceled during operations. ContextPropagationPassthrough ContextPropagationBehaviour = iota // ContextPropagationOnAcquire propagates the caller's context when acquiring // a connection from the pool, but detaches it for the actual database operation. // This prevents context cancellation from affecting ongoing operations while // still respecting cancellation during connection acquisition. ContextPropagationOnAcquire // ContextPropagationDetach detaches the caller's context completely, creating // a new background context for both connection acquisition and database operations. // Tracing information is preserved. This provides maximum isolation from context // cancellation but operations cannot be canceled by the caller. ContextPropagationDetach )
type Jitter ¶
type Jitter interface {
// Jitter consumes an interval from a ticker and returns a jittered duration.
Jitter(time.Duration) time.Duration
}
Jitter applies randomness to durations to prevent thundering herd problems.
type JitteredTicker ¶
JitteredTicker is a ticker with an additional jitter applied to the interval. Similar to time.Ticker but with randomized intervals.
func NewJitteredTicker ¶
func NewJitteredTicker(d time.Duration, j Jitter) *JitteredTicker
NewJitteredTicker creates a new jittered ticker that ticks at intervals around the specified duration, modified by the provided jitter.
func (*JitteredTicker) Stop ¶
func (t *JitteredTicker) Stop()
Stop stops the jittered ticker. After Stop, no more ticks will be sent.
type MaxRetryError ¶
MaxRetryError is returned when the maximum number of retries has been exceeded.
func (*MaxRetryError) Error ¶
func (e *MaxRetryError) Error() string
func (*MaxRetryError) GRPCStatus ¶
func (e *MaxRetryError) GRPCStatus() *status.Status
func (*MaxRetryError) Unwrap ¶
func (e *MaxRetryError) Unwrap() error
type NodeBalancer ¶
type NodeBalancer interface {
// Run starts the periodic balancing loop and blocks until ctx is canceled.
Run(ctx context.Context, pool *Pool)
// Rebalance performs one balancing operation.
Rebalance(ctx context.Context, pool *Pool)
}
NodeBalancer balances pooled connections across CockroachDB nodes.
func NewNodeConnectionBalancer ¶
func NewNodeConnectionBalancer(opts ...NodeConnectionBalancerOption) NodeBalancer
NewNodeConnectionBalancer creates a connection balancer that distributes connections across CockroachDB nodes.
The balancer ensures an even distribution of connections across healthy nodes, pruning excess connections from over-utilized nodes and marking unhealthy node connections for garbage collection.
Pass the returned balancer to WithBalancer to run it with a Pool, or call NodeBalancer.Run yourself if you need to manage its lifecycle directly.
If no interval is specified via WithBalancingInterval, the default is 30 seconds.
type NodeConnectionBalancerOption ¶
type NodeConnectionBalancerOption func(*nodeConnectionBalancer)
NodeConnectionBalancerOption is a configuration option for the connection balancer.
func WithBalancingInterval ¶
func WithBalancingInterval(interval time.Duration) NodeConnectionBalancerOption
WithBalancingInterval sets how often to perform connection rebalancing. Default: 30 seconds.
func WithBalancingTimeout ¶
func WithBalancingTimeout(timeout time.Duration) NodeConnectionBalancerOption
WithBalancingTimeout sets the timeout for an individual balancing operation. Default: 10 seconds.
func WithTopologyChangeBalancingInterval ¶
func WithTopologyChangeBalancingInterval(interval, window time.Duration) NodeConnectionBalancerOption
WithTopologyChangeBalancingInterval configures a temporary faster balancing interval used after the tracker detects a topology change.
The faster interval is active for window. Defaults: 5 seconds for 5 minutes.
type NodeHealthTracker ¶
type NodeHealthTracker interface {
SetStatus(nodeID uint32, healthy bool)
IsHealthy(nodeID uint32) bool
NumHealthy() int
}
NodeHealthTracker is responsible for tracking health of a nodes in a CockroachDB cluster.
type NodeHealthTrackerOption ¶
type NodeHealthTrackerOption func(*nodeHealthTrackerConfig)
NodeHealthTrackerOption is a configuration option for the [nodeHealthTracker].
func WithPollInterval ¶
func WithPollInterval(d time.Duration) NodeHealthTrackerOption
WithPollInterval configures the interval at which the node health tracker attempts to connect to a random node in the cluster and assert its health.
func WithTopologyChangePollInterval ¶
func WithTopologyChangePollInterval(interval, window time.Duration) NodeHealthTrackerOption
WithTopologyChangePollInterval configures a temporary faster polling interval used after the tracker detects a topology change.
The faster interval is active for window. Defaults: 5 seconds for 5 minutes.
type Option ¶
type Option func(*poolCfg)
Option is a configuration option for a Pool.
func WithBalancer ¶
func WithBalancer(b NodeBalancer) Option
WithBalancer configures the node connection balancer used by the Pool.
The pool starts the balancer after the underlying pgx pool is created and stops it when Pool.Close is called. If no balancer is configured, connection rebalancing is disabled.
func WithConnectRate ¶
WithConnectRate configures the rate limit for establishing new connections to the database. This helps prevent connection storms. The default is DefaultConnectRate.
func WithContextPropagationBehaviour ¶
func WithContextPropagationBehaviour(behavior ContextPropagationBehaviour) Option
WithContextPropagationBehaviour configures how contexts are propagated from callers to database operations. See ContextPropagationBehaviour for available options. The default is DefaultContextPropagationBehaviour.
func WithHealthTracker ¶
func WithHealthTracker(h NodeHealthTracker) Option
WithHealthTracker configures the health tracker used by the Pool.
You might want to initialize and pass the health tracker to the pool yourself if you use multiple pools for the same database inside one application in which case the pools can share the health tracker.
If no custom health tracker is provided a default per-pool one is used.
func WithLogger ¶
WithLogger configures the logger for the Pool. By default, no logger is configured and debug logging is disabled.
func WithMaxRetries ¶
WithMaxRetries configures the maximum number of retries per database operation when encountering retryable errors. The default is DefaultMaxRetries.
type Pool ¶
type Pool struct {
// contains filtered or unexported fields
}
Pool is a CockroachDB-aware connection pool that wraps pgxpool.Pool with automatic retry logic, node health tracking, and optional connection rebalancing capabilities.
Pool provides the same interface as pgxpool.Pool but adds resilience features specifically designed for CockroachDB clusters, including automatic handling of retryable and resettable errors, and optional connection management across nodes.
func NewPool ¶
func NewPool( ctx context.Context, pgxConfig *pgxpool.Config, opts ...Option, ) (_ *Pool, err error)
NewPool creates a new CockroachDB-aware connection pool.
The pool wraps pgxpool.Pool and adds CockroachDB-specific features including automatic retry logic, node health tracking, and optional connection rebalancing.
Parameters:
- ctx: Context for pool initialization
- pgxConfig: pgxpool configuration (a copy will be made internally)
- healthTracker: Node health tracker for monitoring CockroachDB node status
- opts: Optional configuration options (see Option functions)
Returns an error if the configuration is invalid or pool creation fails.
Example ¶
package main
import (
"context"
"log"
"os"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/sumup/crdbpgx"
)
func main() {
cfg, err := pgxpool.ParseConfig(os.Getenv("DATABASE_URL"))
if err != nil {
log.Fatal(err)
}
pool, err := crdbpgx.NewPool(
context.Background(),
cfg,
crdbpgx.WithName("orders"),
crdbpgx.WithContextPropagationBehaviour(crdbpgx.ContextPropagationOnAcquire),
crdbpgx.WithBalancer(crdbpgx.NewNodeConnectionBalancer(
crdbpgx.WithBalancingInterval(30*time.Second),
crdbpgx.WithBalancingTimeout(10*time.Second),
)),
)
if err != nil {
log.Fatal(err)
}
defer pool.Close()
}
Output:
func (*Pool) Acquire ¶
Acquire returns a connection from the pool. The connection must be released back to the pool by calling pgxpool.Conn.Release on the returned connection.
func (*Pool) AcquireAllIdle ¶
AcquireAllIdle atomically acquires all currently idle connections from the pool. This is primarily used by the node connection balancer.
func (*Pool) AcquireFunc ¶
AcquireFunc acquires a connection, calls the provided function with it, and releases the connection when the function returns.
func (*Pool) Close ¶
func (p *Pool) Close()
Close closes all connections in the pool and prevents new connections from being acquired.
func (*Pool) CopyFrom ¶
func (p *Pool) CopyFrom(ctx context.Context, tableName pgx.Identifier, columnNames []string, rowSrc pgx.CopyFromSource) (int64, error)
CopyFrom efficiently copies data from rowSrc to the specified table using the PostgreSQL COPY protocol. Note that this operation does not have automatic retry logic.
func (*Pool) ExecFunc ¶
func (p *Pool) ExecFunc(ctx context.Context, tagFunc func(ctx context.Context, tag pgconn.CommandTag, err error) error, sql string, arguments ...any) error
ExecFunc executes a SQL command and calls tagFunc with the command result. The operation includes automatic retry logic for retryable and resettable errors.
func (*Pool) GCMark ¶
GCMark marks a connection to be destroyed on the next pool operation (acquire/release). This is used internally for connection rebalancing and when recovering from node failures.
func (*Pool) InTx ¶
func (p *Pool) InTx(ctx context.Context, txOptions pgx.TxOptions, txFunc func(context.Context, Runner) error) error
InTx executes a function within a transaction and passes a Runner that should be used for all operations in the transactional unit.
The transaction is automatically committed if txFunc returns nil, or rolled back if txFunc returns an error. In case of retryable errors, the entire transaction callback is re-executed.
The callback receives the context that should be used for transaction-bound database operations.
Example ¶
package main
import (
"context"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/sumup/crdbpgx"
)
func main() {
transferFunds := func(ctx context.Context, pool *crdbpgx.Pool, accountID string, amount int64) error {
return pool.InTx(ctx, pgx.TxOptions{}, func(ctx context.Context, db crdbpgx.Runner) error {
return db.ExecFunc(
ctx,
func(_ context.Context, _ pgconn.CommandTag, err error) error {
return err
},
"UPDATE accounts SET balance = balance - $1 WHERE id = $2",
amount,
accountID,
)
})
}
_ = transferFunds
}
Output:
func (*Pool) InTxFunc ¶
InTxFunc executes a function within a transaction with automatic retry logic.
This is a replacement for pgxpool.BeginTxFunc that provides CockroachDB-specific error handling. It resets the connection on resettable errors (e.g., node failures) and retries the transaction on retryable errors (e.g., serialization failures).
The transaction is automatically committed if txFunc returns nil, or rolled back if txFunc returns an error. In case of retryable errors, the entire transaction (including txFunc) is re-executed.
func (*Pool) NodeID ¶
NodeID returns the CockroachDB node ID for the given connection. Returns 0 if the connection is not tracked.
func (*Pool) QueryFunc ¶
func (p *Pool) QueryFunc(ctx context.Context, rowsFunc func(ctx context.Context, rows pgx.Rows) error, sql string, optionsAndArgs ...any) error
QueryFunc executes a SQL query and calls rowsFunc with the resulting rows. The operation includes automatic retry logic for retryable and resettable errors. rowsFunc must close the rows when done.
func (*Pool) QueryRowFunc ¶
func (p *Pool) QueryRowFunc(ctx context.Context, rowFunc func(ctx context.Context, row pgx.Row) error, sql string, optionsAndArgs ...any) error
QueryRowFunc executes a SQL query that returns at most one row and calls rowFunc with the result. The operation includes automatic retry logic for retryable and resettable errors.
type ResettableError ¶
type ResettableError struct {
Err error
}
ResettableError wraps an error that requires connection reset and retry on a different node.
func (*ResettableError) Error ¶
func (e *ResettableError) Error() string
func (*ResettableError) GRPCStatus ¶
func (e *ResettableError) GRPCStatus() *status.Status
func (*ResettableError) Unwrap ¶
func (e *ResettableError) Unwrap() error
type RetryableError ¶
type RetryableError struct {
Err error
}
RetryableError wraps an error that can be retried on the same connection.
func (*RetryableError) Error ¶
func (e *RetryableError) Error() string
func (*RetryableError) Unwrap ¶
func (e *RetryableError) Unwrap() error
type Runner ¶
type Runner interface {
ExecFunc(ctx context.Context, tagFunc func(context.Context, pgconn.CommandTag, error) error, sql string, arguments ...any) error
QueryFunc(ctx context.Context, rowsFunc func(context.Context, pgx.Rows) error, sql string, optionsAndArgs ...any) error
QueryRowFunc(ctx context.Context, rowFunc func(context.Context, pgx.Row) error, sql string, optionsAndArgs ...any) error
// InTxFunc ignores [pgx.TxOptions] when invoked as a nested (pseudo) transaction.
InTxFunc(context.Context, pgx.TxOptions, func(Tx) error) error
}
Runner is the shared query execution interface implemented by both Pool and Tx.
This is the main abstraction for application code that should work both inside and outside of a transaction without relying on context-carried transaction state.
type Tx ¶
Tx represents a CockroachDB database transaction.
Tx can be used in place of pgx.Tx and provides additional methods that are also defined on Pool such that users can use the same set of functions both outside and inside a transaction (ExecFunc, QueryFunc, QueryRowFunc, InTxFunc).
type UniformJitter ¶
UniformJitter implements uniform jitter distribution.
The jittered duration is uniformly distributed between Min and the provided duration. This helps distribute load more evenly and prevents synchronized retries across clients.