encryption

Documentation

Overview

Package encryption defines interfaces and utilities for Iceberg table encryption. It provides:

• EncryptionManager — the central coordination interface for per-file envelope encryption and decryption.
• KeyManagementClient — the interface for KMS backends (wrap/unwrap DEKs).
• EncryptedInputFile / EncryptedOutputFile — file wrappers that carry per-file key metadata alongside the data stream.
• NativeEncryptionInputFile / NativeEncryptionOutputFile — extensions for format-native encryption (e.g., Parquet column-level encryption).
• PlaintextEncryptionManager — a no-op implementation suitable for unencrypted tables.

Index

• Constants
• Variables
• func GetRegisteredKMSNames() []string
• func RegisterKMS(name string, factory KMSFactory)
• func UnregisterKMS(name string)
• type EncryptedInputFile
• type EncryptedOutputFile
• type EncryptionKeyMetadata
• type EncryptionManager
• type InMemoryKeyManagementClient
  • func NewInMemoryKeyManagementClient() *InMemoryKeyManagementClient
  • func (c *InMemoryKeyManagementClient) AddKey(keyID string, masterKey []byte) error
  • func (c *InMemoryKeyManagementClient) GenerateKey(ctx context.Context, keyID string, length int) (plaintext, wrapped []byte, err error)
  • func (c *InMemoryKeyManagementClient) SupportsKeyGeneration() bool
  • func (c *InMemoryKeyManagementClient) UnwrapKey(_ context.Context, keyID string, wrappedKey []byte) ([]byte, error)
  • func (c *InMemoryKeyManagementClient) WrapKey(_ context.Context, keyID string, plaintextKey []byte) ([]byte, error)
• type KMSFactory
• type KeyManagementClient
  • func LoadKeyManagementClient(props map[string]string) (KeyManagementClient, error)
• type NativeEncryptionInputFile
• type NativeEncryptionOutputFile
• type PlaintextEncryptionManager
  • func (PlaintextEncryptionManager) NewDecryptedInputFile(_ context.Context, file icebergio.File, keyMetadata EncryptionKeyMetadata) (EncryptedInputFile, error)
  • func (PlaintextEncryptionManager) NewEncryptedOutputFile(_ context.Context, writer icebergio.FileWriter, keyID string) (EncryptedOutputFile, error)

Constants

const KMSTypeKey = "kms-type"

KMSTypeKey is the catalog property that selects a KeyManagementClient implementation registered via RegisterKMS (e.g. "memory"). It is a registered short name, not a fully-qualified class name.

KMSTypeKey ("kms-type") is an iceberg-go-specific mechanism for wiring up a KMS client at runtime. It is not a standardized Iceberg spec property and has no cross-implementation meaning. It will not be recognized by, or portable to, other Iceberg implementations.

Variables

var (
	// ErrKeyIDNotSupported is returned by [PlaintextEncryptionManager.NewEncryptedOutputFile]
	// when a caller supplies a non-empty keyID. Accepting the keyID but writing
	// plaintext would fail open: the caller believes the file is encrypted, but
	// no encryption is actually performed.
	ErrKeyIDNotSupported = errors.New("encryption: PlaintextEncryptionManager does not support a non-empty keyID; configure a real EncryptionManager")

	// ErrKeyMetadataNotSupported is returned by [PlaintextEncryptionManager.NewDecryptedInputFile]
	// when a caller supplies non-empty key metadata. Ignoring the metadata and
	// returning the raw bytes would fail open: the caller would silently read
	// encrypted bytes as if they were plaintext.
	ErrKeyMetadataNotSupported = errors.New("encryption: PlaintextEncryptionManager does not support non-empty key metadata; configure a real EncryptionManager")
)

var (
	// ErrUnknownKeyID is returned when a KMS client is asked to wrap or unwrap
	// with a key ID it does not know about.
	ErrUnknownKeyID = errors.New("encryption: unknown key ID")

	// ErrInvalidKeyLength is returned when a key or requested key length is not
	// valid for the underlying cipher (for example, a KEK that is not 16, 24,
	// or 32 bytes for AES, or a DEK request of length <= 0).
	ErrInvalidKeyLength = errors.New("encryption: invalid key length")

	// ErrCiphertextTooShort is returned when a wrapped key or encrypted payload
	// is smaller than the minimum required (e.g. the AES-GCM nonce prefix).
	ErrCiphertextTooShort = errors.New("encryption: ciphertext too short")

	// ErrAuthenticationFailed is returned when an authenticated decryption
	// primitive rejects its input (tampering, wrong key, or corruption).
	ErrAuthenticationFailed = errors.New("encryption: authentication failed")
)

Sentinel errors returned by KeyManagementClient implementations. Callers should test with errors.Is rather than string matching.

var ErrKMSTypeNotFound = errors.New("encryption: kms type not found")

ErrKMSTypeNotFound is returned by LoadKeyManagementClient when the requested kmsType has not been registered via RegisterKMS.

Functions

func GetRegisteredKMSNames

func GetRegisteredKMSNames() []string

GetRegisteredKMSNames returns the names of all currently registered KMS factories.

func RegisterKMS

func RegisterKMS(name string, factory KMSFactory)

RegisterKMS adds a new named KMSFactory to the registry so it can later be selected via the KMSTypeKey catalog property. It panics if factory is nil or if name is already registered.

func UnregisterKMS

func UnregisterKMS(name string)

UnregisterKMS removes the requested named factory from the registry.

Types

type EncryptedInputFile

type EncryptedInputFile interface {
	icebergio.File
	// KeyMetadata returns the opaque per-file encryption key metadata.
	KeyMetadata() EncryptionKeyMetadata
}

EncryptedInputFile is a readable file that carries the per-file encryption key metadata required by an EncryptionManager to derive the decryption key.

type EncryptedOutputFile

type EncryptedOutputFile interface {
	icebergio.FileWriter
	// KeyMetadata returns the finalized per-file encryption key metadata.
	// The returned value is only guaranteed to be populated after Close has
	// been called.
	KeyMetadata() EncryptionKeyMetadata
}

EncryptedOutputFile is a writable file whose plaintext content is encrypted by the EncryptionManager. After the file has been closed, [KeyMetadata] returns the finalized per-file key metadata to embed in the corresponding manifest entry.

type EncryptionKeyMetadata

type EncryptionKeyMetadata []byte

EncryptionKeyMetadata is the opaque per-file key metadata blob embedded in manifest entries (DataFile.KeyMetadata, ManifestFile.KeyMetadata) and statistics files (StatisticsFile.KeyMetadata). The encoding is EncryptionManager-defined and typically envelopes the wrapped DEK together with any associated authenticated additional data (AAD).

type EncryptionManager

type EncryptionManager interface {
	// NewEncryptedOutputFile creates a new encrypted output file.
	// keyID is the table property encryption.key-id that identifies the key
	// encryption key (KEK) to use when wrapping the generated DEK.
	NewEncryptedOutputFile(ctx context.Context, writer icebergio.FileWriter, keyID string) (EncryptedOutputFile, error)

	// NewDecryptedInputFile wraps an existing file for transparent decryption.
	// keyMetadata is the per-file opaque blob stored in the manifest entry
	// (DataFile.KeyMetadata or ManifestFile.KeyMetadata).
	NewDecryptedInputFile(ctx context.Context, file icebergio.File, keyMetadata EncryptionKeyMetadata) (EncryptedInputFile, error)
}

EncryptionManager handles envelope key derivation, file-level encryption, and decryption. It is the central coordination point between the KeyManagementClient (key encryption key operations) and the per-file data encryption keys (DEKs).

A PlaintextEncryptionManager is provided for tables that do not use encryption. Vendors and users can supply custom implementations to integrate with their KMS solutions.

type InMemoryKeyManagementClient

type InMemoryKeyManagementClient struct {
	// contains filtered or unexported fields
}

InMemoryKeyManagementClient is a KeyManagementClient backed by an in-memory map of master (key-encryption) keys. Key wrapping and unwrapping use AES-GCM.

Warning

InMemoryKeyManagementClient is intended for testing only. All keys are held in plaintext in process memory with no persistence, access control, or audit logging. Do not use it in production.

func NewInMemoryKeyManagementClient

func NewInMemoryKeyManagementClient() *InMemoryKeyManagementClient

NewInMemoryKeyManagementClient creates a new InMemoryKeyManagementClient with no pre-loaded keys. Register KEKs with [AddKey] before use.

func (*InMemoryKeyManagementClient) AddKey

func (c *InMemoryKeyManagementClient) AddKey(keyID string, masterKey []byte) error

AddKey registers a master key (KEK) under keyID. masterKey must be 16, 24, or 32 bytes for AES-128, AES-192, or AES-256 respectively. If keyID is already registered, its KEK is silently replaced.

func (*InMemoryKeyManagementClient) GenerateKey

func (c *InMemoryKeyManagementClient) GenerateKey(ctx context.Context, keyID string, length int) (plaintext, wrapped []byte, err error)

GenerateKey generates a cryptographically random DEK of the given byte length, wraps it with the KEK identified by keyID, and returns both the plaintext DEK and the wrapped form.

func (*InMemoryKeyManagementClient) SupportsKeyGeneration

func (c *InMemoryKeyManagementClient) SupportsKeyGeneration() bool

SupportsKeyGeneration returns true; the in-memory client generates DEKs locally using crypto/rand.

func (*InMemoryKeyManagementClient) UnwrapKey

func (c *InMemoryKeyManagementClient) UnwrapKey(_ context.Context, keyID string, wrappedKey []byte) ([]byte, error)

UnwrapKey decrypts wrappedKey using the KEK identified by keyID. The wrappedKey must have been produced by [WrapKey] or [GenerateKey].

func (*InMemoryKeyManagementClient) WrapKey

func (c *InMemoryKeyManagementClient) WrapKey(_ context.Context, keyID string, plaintextKey []byte) ([]byte, error)

WrapKey encrypts plaintextKey with the KEK identified by keyID using AES-GCM. The returned ciphertext layout is:

12-byte random nonce || AES-GCM ciphertext || 16-byte GCM authentication tag

type KMSFactory

type KMSFactory func(props map[string]string) (KeyManagementClient, error)

KMSFactory creates a KeyManagementClient from catalog/table properties. props is the full property map (typically catalog properties); factories should read whichever keys they need (e.g. key material, endpoints, credentials) directly from it.

type KeyManagementClient

type KeyManagementClient interface {
	// WrapKey encrypts (wraps) plaintextKey using the KEK identified by keyID.
	// The returned bytes are an opaque, KMS-specific wrapped key blob.
	WrapKey(ctx context.Context, keyID string, plaintextKey []byte) ([]byte, error)

	// UnwrapKey decrypts (unwraps) wrappedKey using the KEK identified by keyID.
	// It returns the original plaintext DEK.
	UnwrapKey(ctx context.Context, keyID string, wrappedKey []byte) ([]byte, error)

	// SupportsKeyGeneration reports whether this client can generate new DEKs
	// server-side. When false, callers should generate DEKs locally and use
	// [WrapKey] to protect them.
	SupportsKeyGeneration() bool

	// GenerateKey generates a new DEK of the given byte length, returning both
	// the plaintext DEK and its KMS-wrapped form. This is only valid when
	// [SupportsKeyGeneration] returns true.
	GenerateKey(ctx context.Context, keyID string, length int) (plaintext, wrapped []byte, err error)
}

KeyManagementClient is the interface for a Key Management Service (KMS) that wraps and unwraps data encryption keys (DEKs) using key encryption keys (KEKs) managed externally by the KMS.

In the standard envelope-encryption model:

• A KEK is identified by a key ID (the table property encryption.key-id).
• A fresh DEK is generated per-file (or per-rotation interval).
• The DEK is wrapped by the KMS and stored in the file's key_metadata field.
• On read, the wrapped DEK is unwrapped by the KMS to decrypt the file.

Implementations are responsible for authentication, authorization, and network communication with the KMS backend.

func LoadKeyManagementClient

func LoadKeyManagementClient(props map[string]string) (KeyManagementClient, error)

LoadKeyManagementClient resolves and constructs a KeyManagementClient from props. It looks up the KMS name under KMSTypeKey; it returns an error wrapping ErrKMSTypeNotFound if the property is unset or the named KMS has not been registered via RegisterKMS.

type NativeEncryptionInputFile

type NativeEncryptionInputFile interface {
	EncryptedInputFile
	// NativeDecryptionProperties returns format-native decryption properties.
	// The concrete type is format-specific; callers should type-assert as
	// needed (e.g., *parquet.FileDecryptionProperties for Parquet files).
	NativeDecryptionProperties(ctx context.Context) (any, error)
}

NativeEncryptionInputFile extends EncryptedInputFile with access to format-native decryption properties (e.g., Parquet FileDecryptionProperties). An EncryptionManager may return a NativeEncryptionInputFile from EncryptionManager.NewDecryptedInputFile when native-format encryption is in use; callers should type-assert to retrieve the concrete properties.

type NativeEncryptionOutputFile

type NativeEncryptionOutputFile interface {
	EncryptedOutputFile
	// NativeEncryptionProperties returns format-native encryption properties.
	// The concrete type is format-specific; callers should type-assert as
	// needed (e.g., *parquet.FileEncryptionProperties for Parquet files).
	NativeEncryptionProperties(ctx context.Context) (any, error)
}

NativeEncryptionOutputFile extends EncryptedOutputFile with access to format-native encryption properties (e.g., Parquet FileEncryptionProperties). An EncryptionManager may return a NativeEncryptionOutputFile from EncryptionManager.NewEncryptedOutputFile when native-format encryption is in use; callers should type-assert to retrieve the concrete properties.

type PlaintextEncryptionManager

type PlaintextEncryptionManager struct{}

PlaintextEncryptionManager is a no-op EncryptionManager that passes all data through without any encryption or decryption. It is the default manager used when no KMS is configured or when a table carries no encryption-keys metadata. It fails closed: any call that supplies a non-empty keyID or non-empty key metadata returns an error rather than silently reading or writing plaintext, since doing so could mask a misconfigured encryption setup.

func (PlaintextEncryptionManager) NewDecryptedInputFile

func (PlaintextEncryptionManager) NewDecryptedInputFile(_ context.Context, file icebergio.File, keyMetadata EncryptionKeyMetadata) (EncryptedInputFile, error)

NewDecryptedInputFile returns a pass-through EncryptedInputFile that reads data unmodified. It fails closed: non-empty keyMetadata indicates the underlying bytes were encrypted, and returning them unchanged would silently hand ciphertext to the caller as if it were plaintext, so it returns ErrKeyMetadataNotSupported rather than ignoring the metadata.

func (PlaintextEncryptionManager) NewEncryptedOutputFile

func (PlaintextEncryptionManager) NewEncryptedOutputFile(_ context.Context, writer icebergio.FileWriter, keyID string) (EncryptedOutputFile, error)

NewEncryptedOutputFile returns a pass-through EncryptedOutputFile that writes data unmodified and returns nil EncryptionKeyMetadata. It fails closed: a non-empty keyID indicates the caller expects the file to be encrypted, and returning nil here would silently write plaintext instead, so it returns ErrKeyIDNotSupported rather than ignoring the keyID.