impala

README

Golang Apache Impala Driver

The actively supported Apache Impala driver for Go's database/sql package

This driver started as a fork of github.com/bippio/go-impala, which hasn't been updated in over four years and appears to be abandoned. Several issues have been fixed since then — some quite severe. The original codebase also didn't support Go modules.

Install

Add impala-go to your Go module:

go get github.com/sclgo/impala-go

Alternatively, see below how to use it as a CLI. impala-go does not use CGO.

Connection Parameters and DSN

The data source name (DSN; connection string) uses a URL format: impala://username:password@host:port?param1=value&param2=value

Driver name is impala.

Parameters:

• auth - string. Authentication mode. Supported values: noauth, ldap.
• tls - boolean. Enable TLS
• ca-cert - The file that contains the public key certificate of the CA that signed the Impala certificate
• batch-size - integer value (default: 1024). Maximum number of rows fetched per request
• buffer-size- in bytes (default: 4096); Buffer size for the Thrift transport
• mem-limit - string value (example: 3m); Memory limit for query

A string of this format can be constructed using the URL type in the net/url package.

  query := url.Values{}
  query.Add("auth", "ldap")

  u := &url.URL{
      Scheme:   "impala",
      User:     url.UserPassword(username, password),
      Host:     net.JoinHostPort(hostname, port),
      RawQuery: query.Encode(),
  }
  db, err := sql.Open("impala", u.String())

Also, you can bypass the string-based data source name by using sql.OpenDB:

  opts := impala.DefaultOptions
  opts.Host = hostname
  opts.UseLDAP = true
  opts.Username = username
  opts.Password = password

  connector := impala.NewConnector(&opts)
  db, err := sql.OpenDB(connector)

CLI

impala-go is included in xo/usql - the universal SQL CLI, inspired by psql.

Install usql, start it, then on its prompt, run:

\connect impala DSN

where DSN is a data source name in the format above. Review the usql documentation for other options.

The latest version of usql typically comes with the latest version of impala-go but if you need to use a different one, you can prepare a custom build using usqlgen. For example, the following command builds a usql binary in the working directory using impala-go from master branch:

go run github.com/sclgo/usqlgen@latest build --get github.com/sclgo/impala-go@master -- -tags impala

usql with impala-go is arguably a better CLI for Impala than the official impala-shell. For one, usql is much easier to install.

Example Go code

package main

// Simple program to list databases and the tables

import (
	"context"
	"database/sql"
	"log"

	"github.com/sclgo/impala-go"
)

func main() {
	opts := impala.DefaultOptions

	opts.Host = "localhost" // impala host
	opts.Port = "21050"

	// enable LDAP authentication:
	//opts.UseLDAP = true
	//opts.Username = "<ldap username>"
	//opts.Password = "<ldap password>"
	//
	// enable TLS
	//opts.UseTLS = true
	//opts.CACertPath = "/path/to/cacert"

	connector := impala.NewConnector(&opts)
	db := sql.OpenDB(connector)
	defer func() {
		_ = db.Close()
	}()

	ctx := context.Background()

	rows, err := db.QueryContext(ctx, "SHOW DATABASES")
	if err != nil {
		log.Fatal(err)
	}

	var name, comment string
	databases := make([]string, 0) // databases will contain all the DBs to enumerate later
	for rows.Next() {
		if err := rows.Scan(&name, &comment); err != nil {
			log.Fatal(err)
		}
		databases = append(databases, name)
	}
	if err := rows.Err(); err != nil {
		log.Fatal(err)
	}
	log.Println("List of Databases", databases)

	tables, err := impala.NewMetadata(db).GetTables(ctx, "%", "%")
	if err != nil {
		log.Fatal(err)
	}
	log.Println("List of Tables", tables)
}

Check out also an open data end-to-end demo.

Data types

Impala data types are mapped to Go types as expected, with the following exceptions:

• "Complex" types - MAP, STRUCT, ARRAY - are not supported. Impala itself has limited support for those. As a workaround, select individual fields or flatten such values within select statements.
• Decimals are converted to strings by the Impala server API. Either parse the decimal value after Rows.Scan, or use a custom sql.Scanner implementation in Row(s).Scan e.g. Decimal from github.com/cockroachdb/apd. Note that the processing of sql.Scanner within Row(s).Scan is a feature of the database/sql package, and not the driver. The ScanType of such values is string, while the DatabaseTypeName is DECIMAL. Retrieving precision and scale using the DecimalSize API is supported.

Context support

The driver methods recognize Context and support early cancellation in most cases. As expected, the Query methods return early before all rows are retrieved. Exec methods return after the operation completes (this may be configurable in the future). Exec methods can still be stopped early by cancelling the context from another goroutine.

It is also supported to use a Query method for a DDL/DML statement if you need the method to return before the statement completes. In that case, calling Rows.Next will wait for the statement to complete and then return false.

Compatibility and Support

The library is actively tested with Impala 4.4 and 3.4. All 3.x and 4.x minor versions should work well. 2.x is also supported on a best-effort basis.

File any issues that you encounter as GitHub issues.

The library is not compatible with TinyGo because Thrift for Go requires tls.Listen which is not implemented by TinyGo at this time.

Copyright and acknowledgements

This library started as a fork of github.com/bippio/go-impala, under the MIT license. This library retains the same license.

The project logo combines the Golang Gopher from github.com/golang-samples/gopher-vector with the Apache Impala logo, licensed under the Apache 2 license.

Documentation

Index

• Variables
• func NewConnector(opts *Options) driver.Connector
• type AuthError
• type ColumnName
• type ConnRawAccess
• type Driver
  • func (d *Driver) Open(dsn string) (driver.Conn, error)
  • func (d *Driver) OpenConnector(name string) (driver.Connector, error)
• type Metadata
  • func NewMetadata(db *sql.DB) *Metadata
  • func NewMetadataFromConn(conn ConnRawAccess) *Metadata
  • func (m Metadata) GetColumns(ctx context.Context, schemaPattern string, tableNamePattern string, ...) ([]ColumnName, error)
  • func (m Metadata) GetSchemas(ctx context.Context, schemaPattern string) ([]string, error)
  • func (m Metadata) GetTables(ctx context.Context, schemaPattern string, tableNamePattern string) ([]TableName, error)
• type Options
• type TableName

Variables

var (
	// DefaultOptions for impala driver
	DefaultOptions = Options{BatchSize: 1024, BufferSize: 4096, Port: "21050", LogOut: io.Discard}
)

var (
	// ErrNotSupported means the driver does not support this operation
	ErrNotSupported = isql.ErrNotSupported
)

Functions

func NewConnector

func NewConnector(opts *Options) driver.Connector

NewConnector creates connector with specified options

Types

type AuthError

type AuthError = sasl.AuthError

AuthError indicates that there was an authentication or authorization failure. The error message documents the username used, if any. errors.Unwrap() returns the underlying error interpreted as auth. failure, if any. This error will not be top-level in the chain - earlier errors in the chain reflect the process during which the auth. error happened.

type ColumnName

type ColumnName = hive.ColumnName

ColumnName contains all attributes that identify a columns

type ConnRawAccess

type ConnRawAccess interface {
	Raw(func(driverConn any) error) error
}

ConnRawAccess exposes the Raw method of sql.Conn

type Driver

type Driver struct{}

Driver to impala

func (*Driver) Open

func (d *Driver) Open(dsn string) (driver.Conn, error)

Open creates new connection to impala using the given data source name. Implements driver.Driver. Returned error wraps any errors coming from thrift or stdlib - typically crypto or net packages. If TLS is requested, and the server certificate fails validation, the error chain includes *tls.CertificateVerificationError If there was an authentication error, an error chain includes one of the exported auth. errors in this package.

func (*Driver) OpenConnector

func (d *Driver) OpenConnector(name string) (driver.Connector, error)

OpenConnector parses name as a DSN (data source name) and returns connector with fixed options Implements driver.DriverContext

type Metadata

type Metadata struct {
	// contains filtered or unexported fields
}

Metadata exposes the schema and other metadata in an Impala instance

func NewMetadata

func NewMetadata(db *sql.DB) *Metadata

NewMetadata creates Metadata instance with the given Impala DB as data source. A new connection will be retrieved for any call. If that's an issue, use NewMetadataFromConn

func NewMetadataFromConn

func NewMetadataFromConn(conn ConnRawAccess) *Metadata

NewMetadataFromConn creates Metadata instance with the given Impala connection as data source *sql.Conn implements ConnRawAccess

func (Metadata) GetColumns

func (m Metadata) GetColumns(ctx context.Context, schemaPattern string, tableNamePattern string, columnNamePattern string) ([]ColumnName, error)

GetColumns retrieves columns that match the provided LIKE patterns

func (Metadata) GetSchemas

func (m Metadata) GetSchemas(ctx context.Context, schemaPattern string) ([]string, error)

GetSchemas retrieves schemas that match the provided LIKE pattern

func (Metadata) GetTables

func (m Metadata) GetTables(ctx context.Context, schemaPattern string, tableNamePattern string) ([]TableName, error)

GetTables retrieves tables and views that match the provided LIKE patterns

type Options

type Options struct {
	Host     string
	Port     string
	Username string
	Password string

	UseLDAP      bool
	UseTLS       bool
	CACertPath   string
	BufferSize   int
	BatchSize    int
	MemoryLimit  string
	QueryTimeout int

	LogOut io.Writer
}

Options for impala driver connection

type TableName

type TableName = hive.TableName

TableName contains all attributes that identify a table