rds

package module
v0.11.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 10, 2023 License: Apache-2.0 Imports: 16 Imported by: 0

README

go-rds-driver

A golang sql Driver for the Amazon Aurora Serverless data api.

Note: The serverless data api only supports named query parameters, not ordinal ones. We perform a simple ordinal variable replacement in the driver, however we strongly recommend you used named parameters as a general rule.

Getting Started

The dsn used in this driver follows the standard URL pattern, however given the complexity of the ARN input parameters required and reserved URI characters, we're forced to put all parameters - even the required ones - in the query parameters.

rds://?resource_arn=...&secret_arn=...&database=...&aws_region=...

This complex string may be generated using the Config type and its ToDSN method.

conf := &rds.Config{
    ResourceArn: "...",
    SecretArn:   "...",
    Database:    "...",
    AWSRegion:   "...",
	SplitMulti:  false,
	ParseTime:   true,
}
dsn, err := conf.ToDSN()

db.ConnPool, err = sql.Open(rds.DRIVERNAME, dsn)

Data mappings

The nature of our data translation - from DB to HTTP to Go - makes converting database types somewhat tricky. In most cases, we've done our best to match the behavior of a commonly used driver, so swapping from Data API to Driver can be done quickly and easily. Even so, there are some unusual behaviors of the RDS Data API that we call out below:

MySQL

The RDS MySQL version supported is 5.7. Driver parity is tested using github.com/go-sql-driver/mysql

  • Unsigned integers are not natively supported by the AWS SDK's Data API, and are all converted to the int64 type. As such large integer values may be lossy.
  • The BIT column type is returned from RDS as a Boolean, preventing the full use of BIT(M). Until (if ever) this is fixed, only BIT(1) column values are supported.
  • Declaring a TINYINT(1) in your table will cause the Data API to return a Boolean instead of an integer. Numeric values are only returned by TINYINT(2) or greater.
  • The BOOLEAN column type is converted into a BIT column by RDS.
  • Boolean marshalling and unmarshalling via sql.*, because of the above issues, only works reliably with the TINYINT(2) column type. Do not use BOOLEAN, BIT, or TINYINT(1) due to the above behavior.
Postgresql

The RDS Postgres version supported is 10.14. Driver parity is tested using github.com/jackc/pgx/v4

  • Unsigned integers are not natively supported by the AWS SDK's Data API, and are all converted to the int64 type. As such large integer values may be lossy.
  • Postgres complex types - in short anything in section 8.8 and after, is not supported. If you'd like us to support that, pull requests are relatively easy to submit.

Options

This driver supports a variety of configuration options in the DSN, as follows:

  • parse_time: Instead of returning the default string value of a date or time type, the driver will convert it into time.Time
  • split_multi: This option will automatically split all SQL statements by the default delimiter ; and submit them to the API as separate requests. Enable this for uses with large migration statements.

Using your own RDS Client

golang's sql package interfaces provide a challenge, as it's quite difficult to capture all the configuration options available in an AWS Configuration instance in a DSN. For this, please construct your own client and create a Connector, then use that connector with the sql.OpenDB() method.

rdsDriver := rds.NewDriver()
rdsConfig := rds.NewConfig(...)
rdsAWSClient := rdsdata.NewFromConfig(...)
rdsConnector := rds.NewConnector(rdsDriver, rdsAWSClient, rdsConfig)

db := sql.OpenDB(rdsConnector)

Usage with Gorm

The above caveat with the Serverless Data API makes usage of gorm tricky. While you can easily use named parameters in your own code, the current implementation of the Gorm Migrators (as of Aug 1, 2021) exclusively uses ordinal parameters. Please be careful when using this driver.

// RDS using the MySQL Dialector
conf := mysql.Config{
    DriverName: rds.DRIVERNAME,
    DSN:        dsn,
}
dialector := mysql.New(conf)


// RDS using the Postgres Dialector
conf := postgres.Config{
    DriverName: rds.DRIVERNAME,
    DSN:        dsn,
}
dialector := postgres.New(conf)

Running the tests

The intent of this driver is to reach type conversion parity with a database instance that's directly available- in other words, that the types returned from the respective drivers are identical. For that purpose we require that you provision a locally run instance of mysql and postgres, as well as an RDS instance of each. The outputs of each are compared during a test run.

Creating locally run test databases

Locally run databases can be started using docker compose up.

Creating RDS Test databases

This project includes a `./terraform/ directory which provisions the necessary resources on RDS. To create them:

// Choose an AWS profile
export AWS_PROFILE=your_aws_profile_name

cd ./terraform
terraform init
terraform apply

To dispose of these resources once you're done:

cd ./terraform
terraform destroy

Once created, the output values will be defined in a local file named terraform.tfstate. These are parsed by our makefile to ensure the correct values are used in the test suite, however for your local IDE it might be useful to set them directly:

export AWS_PROFILE = "your_aws_profile"
export RDS_MYSQL_DB_NAME = "go_rds_driver_mysql"
export RDS_MYSQL_ARN = "arn:aws:rds:us-west-2:1234567890:cluster:mysql"
export RDS_POSTGRES_DB_NAME = "go_rds_driver_postgresql"
export RDS_POSTGRES_ARN = "arn:aws:rds:us-west-2:1234567890:cluster:postgresql"
export RDS_SECRET_ARN = "arn:aws:secretsmanager:us-west-2:1234567890:secret:aurora_password"
export AWS_REGION=us-west-2
Executing checks

Executing tests and generating reports can be done via the provided makefile.

make clean checks

Why does this even exist?

Not everyone has the capital to pay for the VPC resources necessary to access Aurora Serverless directly. In the author's case, he likes to keep his personal projects as cheap as possible, and paying for all VPC service gateways, just so he can access an RDBMS, crossed the threshold of affordability. If you're looking to run a personal project and don't want to break the bank with "overhead" expenses such as VPN service mappings, this driver's the way to go.

Acknowledgments

This implementation inspired by what came before.

Documentation

Index

Constants

View Source 
const DRIVERNAME = "rds"

DRIVERNAME is used when configuring your dialector

Variables

View Source 
var ErrClosed = fmt.Errorf("this connection is closed")

ErrClosed indicates that the connection is closed

View Source 
var ErrInvalidDSNScheme = fmt.Errorf("this driver requires a DSN scheme of rds://")

ErrInvalidDSNScheme for when the dsn doesn't match rds://

View Source 
var ErrNoMixedParams = fmt.Errorf("please do not mix ordinal and named parameters")

ErrNoMixedParams is thrown if parameters are mixed

Functions

func ConvertDefaults added in v0.5.1 

func ConvertDefaults() func(field types.Field) (value interface{}, err error)

ConvertDefaults handles all types that can be returned directly without additional conversion.

func ConvertNamedValue 

func ConvertNamedValue(arg driver.NamedValue) (value types.SqlParameter, err error)

ConvertNamedValue from a NamedValue to an SqlParameter

func ConvertNamedValues 

func ConvertNamedValues(args []driver.NamedValue) ([]types.SqlParameter, error)

ConvertNamedValues converts passed driver.NamedValue instances into RDS SQLParameters

func NewConnection 

func NewConnection(ctx context.Context, rds AWSClientInterface, conf *Config, dialect Dialect) driver.Conn

NewConnection that can make transaction and statement requests against RDS

func NewResult 

func NewResult(results []*rdsdata.ExecuteStatementOutput) driver.Result

NewResult for the executed statement

func NewRows 

func NewRows(dialect Dialect, results []*rdsdata.ExecuteStatementOutput) driver.Rows

NewRows instance for the provided statement output

func NewTx 

func NewTx(transactionID *string, conn *Connection) driver.Tx

NewTx creates a new transaction

Types

type AWSClientInterface added in v0.5.0 

type AWSClientInterface interface {
	ExecuteStatement(ctx context.Context, e *rdsdata.ExecuteStatementInput, optFns ...func(*rdsdata.Options)) (*rdsdata.ExecuteStatementOutput, error)
	BeginTransaction(ctx context.Context, b *rdsdata.BeginTransactionInput, optFns ...func(*rdsdata.Options)) (*rdsdata.BeginTransactionOutput, error)
	CommitTransaction(ctx context.Context, c *rdsdata.CommitTransactionInput, optFns ...func(*rdsdata.Options)) (*rdsdata.CommitTransactionOutput, error)
	RollbackTransaction(ctx context.Context, r *rdsdata.RollbackTransactionInput, optFns ...func(*rdsdata.Options)) (*rdsdata.RollbackTransactionOutput, error)
}

AWSClientInterface interface that captures methods required by the driver. In this case, replicating the RDS API

type Config 

type Config struct {
	ResourceArn string
	SecretArn   string
	Database    string
	AWSRegion   string
	ParseTime   bool
	SplitMulti  bool
	Custom      map[string][]string
}

Config struct used to provide AWS Configuration Credentials

func NewConfig 

func NewConfig(resourceARN string, secretARN string, database string, awsRegion string) (conf *Config)

NewConfig with basic values.

func NewConfigFromDSN 

func NewConfigFromDSN(dsn string) (conf *Config, err error)

NewConfigFromDSN assumes that the DSN is a JSON-encoded string

func (*Config) ToDSN 

func (o *Config) ToDSN() string

ToDSN converts the config to a DSN string

type Connection 

type Connection struct {
	// contains filtered or unexported fields
}

Connection to RDS's Aurora Serverless Data API

func (*Connection) Begin 

func (r *Connection) Begin() (driver.Tx, error)

Begin starts and returns a new transaction.

func (*Connection) BeginTx 

func (r *Connection) BeginTx(ctx context.Context, opts driver.TxOptions) (driver.Tx, error)

BeginTx starts and returns a new transaction.

func (*Connection) Close 

func (r *Connection) Close() error

Close the connection

func (*Connection) ExecContext 

func (r *Connection) ExecContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Result, error)

ExecContext executes a queries that would normally not return a result.

func (*Connection) IsValid 

func (r *Connection) IsValid() bool

IsValid is called prior to placing the connection into the connection pool. The connection will be discarded if false is returned.

func (*Connection) Ping 

func (r *Connection) Ping(ctx context.Context) (err error)

Ping the database

func (*Connection) Prepare 

func (r *Connection) Prepare(query string) (driver.Stmt, error)

Prepare returns a prepared statement, bound to this connection.

func (*Connection) PrepareContext 

func (r *Connection) PrepareContext(ctx context.Context, query string) (driver.Stmt, error)

PrepareContext returns a prepared statement, bound to this connection.

func (*Connection) QueryContext 

func (r *Connection) QueryContext(ctx context.Context, query string, args []driver.NamedValue) (driver.Rows, error)

QueryContext executes a statement that would return some kind of result.

func (*Connection) ResetSession 

func (r *Connection) ResetSession(_ context.Context) error

ResetSession is called prior to executing a queries on the connection if the connection has been used before. If the driver returns ErrBadConn the connection is discarded.

type Connector 

type Connector struct {
	// contains filtered or unexported fields
}

Connector spits out connections to our database.

func NewConnector 

func NewConnector(d driver.Driver, client AWSClientInterface, conf *Config) *Connector

NewConnector from the provided configuration fields

func (*Connector) Connect 

func (r *Connector) Connect(ctx context.Context) (driver.Conn, error)

Connect returns a connection to the database.

func (*Connector) Driver 

func (r *Connector) Driver() driver.Driver

Driver returns the underlying Driver of the Connector, mainly to maintain compatibility with the Driver method on sql.DB.

func (*Connector) Wakeup added in v0.3.0 

func (r *Connector) Wakeup() (dialect Dialect, err error)

Wakeup the cluster if it's dormant

type Dialect 

type Dialect interface {
	// MigrateQuery from the dialect to RDS
	MigrateQuery(string, []driver.NamedValue) (*rdsdata.ExecuteStatementInput, error)
	// GetFieldConverter for a given ColumnMetadata.TypeName field.
	GetFieldConverter(columnType string) FieldConverter
	// IsIsolationLevelSupported for this dialect?
	IsIsolationLevelSupported(level driver.IsolationLevel) bool
}

Dialect is an interface that encapsulates a particular languages' eccentricities

func NewMySQL added in v0.3.0 

func NewMySQL(config *Config) Dialect

NewMySQL dialect from our configuration

func NewPostgres added in v0.3.0 

func NewPostgres(config *Config) Dialect

NewPostgres dialect from our configuration

type DialectMySQL 

type DialectMySQL struct {
	// contains filtered or unexported fields
}

DialectMySQL for version 5.7

func (*DialectMySQL) GetFieldConverter added in v0.2.0 

func (d *DialectMySQL) GetFieldConverter(columnType string) FieldConverter

GetFieldConverter knows how to parse column results.

func (*DialectMySQL) IsIsolationLevelSupported added in v0.2.0 

func (d *DialectMySQL) IsIsolationLevelSupported(level driver.IsolationLevel) bool

IsIsolationLevelSupported for mysql?

func (*DialectMySQL) MigrateQuery added in v0.2.0 

func (d *DialectMySQL) MigrateQuery(query string, args []driver.NamedValue) (*rdsdata.ExecuteStatementInput, error)

MigrateQuery converts a mysql queries into an RDS stateement.

type DialectPostgres 

type DialectPostgres struct {
	// contains filtered or unexported fields
}

DialectPostgres is for postgres 10.14 as supported by aurora serverless

func (*DialectPostgres) GetFieldConverter added in v0.2.0 

func (d *DialectPostgres) GetFieldConverter(columnType string) FieldConverter

GetFieldConverter knows how to parse response data.

func (*DialectPostgres) IsIsolationLevelSupported added in v0.2.0 

func (d *DialectPostgres) IsIsolationLevelSupported(level driver.IsolationLevel) bool

IsIsolationLevelSupported for postgres?

func (*DialectPostgres) MigrateQuery added in v0.2.0 

func (d *DialectPostgres) MigrateQuery(query string, args []driver.NamedValue) (*rdsdata.ExecuteStatementInput, error)

MigrateQuery from Postgres to RDS.

type Driver 

type Driver struct{}

Driver implements the driver.Driver interface for RDS

func NewDriver 

func NewDriver() *Driver

NewDriver creates a new driver instance for RDS

func (*Driver) Open 

func (r *Driver) Open(name string) (driver.Conn, error)

Open returns a new connection to the database.

func (*Driver) OpenConnector 

func (r *Driver) OpenConnector(dsn string) (driver.Connector, error)

OpenConnector must parse the name in the same format that Driver.Open parses the name parameter.

type FieldConverter added in v0.2.0 

type FieldConverter func(field types.Field) (interface{}, error)

FieldConverter is a function that converts the passed result row field into the expected type.

type Result 

type Result struct {
	// contains filtered or unexported fields
}

Result from a queries

func (*Result) LastInsertId 

func (r *Result) LastInsertId() (int64, error)

LastInsertId from the executed statements.

func (*Result) RowsAffected 

func (r *Result) RowsAffected() (int64, error)

RowsAffected count

type Rows 

type Rows struct {
	// contains filtered or unexported fields
}

Rows implementation for the RDS Driver

func (*Rows) Close 

func (r *Rows) Close() error

Close the result set

func (*Rows) Columns 

func (r *Rows) Columns() []string

Columns returns the column names in order

func (*Rows) HasNextResultSet added in v0.8.0 

func (r *Rows) HasNextResultSet() bool

HasNextResultSet returns true if there's another result set.

func (*Rows) Next 

func (r *Rows) Next(dest []driver.Value) error

Next row in the result set

func (*Rows) NextResultSet added in v0.8.0 

func (r *Rows) NextResultSet() error

NextResultSet moves the result to the next result set.

type Statement 

type Statement struct {
	// contains filtered or unexported fields
}

Statement encapsulates a single RDS queries statement

func NewStatement 

func NewStatement(_ context.Context, connection *Connection, sql []string) *Statement

NewStatement for the provided connection

func (*Statement) Close 

func (s *Statement) Close() error

Close closes the statement.

func (*Statement) ConvertOrdinal added in v0.2.0 

func (s *Statement) ConvertOrdinal(values []driver.Value) []driver.NamedValue

ConvertOrdinal converts a list of Values to Ordinal NamedValues

func (*Statement) Exec 

func (s *Statement) Exec(values []driver.Value) (driver.Result, error)

Exec executes a queries that doesn't return rows, such as an INSERT or UPDATE.

func (*Statement) ExecContext 

func (s *Statement) ExecContext(ctx context.Context, args []driver.NamedValue) (driver.Result, error)

ExecContext executes a queries that doesn't return rows, such as an INSERT or UPDATE.

func (*Statement) NumInput 

func (s *Statement) NumInput() int

NumInput returns the number of placeholder parameters.

func (*Statement) Query 

func (s *Statement) Query(values []driver.Value) (driver.Rows, error)

Query executes a queries that may return rows, such as a SELECT.

func (*Statement) QueryContext 

func (s *Statement) QueryContext(ctx context.Context, args []driver.NamedValue) (driver.Rows, error)

QueryContext executes a queries that may return rows, such as a SELECT.

type Tx 

type Tx struct {
	Done          bool
	TransactionID *string
	// contains filtered or unexported fields
}

Tx is a transaction

func (*Tx) Commit 

func (r *Tx) Commit() error

Commit the transaction

func (*Tx) Rollback 

func (r *Tx) Rollback() error

Rollback the transaction

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.