Documentation
¶
Overview ¶
Package azure implements streaming connectors for Microsoft Azure services.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type BlobStorageObject ¶
type BlobStorageObject struct {
// Key is the object name including any subdirectories.
// For example, "directory/file.json".
Key string
// Data is an [io.ReadCloser] representing the binary content of the blob object.
Data io.ReadCloser
}
BlobStorageObject contains details of the Azure Blob storage object.
type BlobStorageSink ¶
type BlobStorageSink struct {
// contains filtered or unexported fields
}
BlobStorageSink represents the Azure Blob storage sink connector.
func NewBlobStorageSink ¶
func NewBlobStorageSink(ctx context.Context, client *azblob.Client, config *BlobStorageSinkConfig, logger *slog.Logger) *BlobStorageSink
NewBlobStorageSink returns a new BlobStorageSink. Incoming elements are expected to be of the BlobStorageObject type. These will be uploaded to the configured container using their key field as the path.
func (*BlobStorageSink) AwaitCompletion ¶
func (s *BlobStorageSink) AwaitCompletion()
AwaitCompletion blocks until the BlobStorageSink connector has completed processing all the received data.
func (*BlobStorageSink) In ¶
func (s *BlobStorageSink) In() chan<- any
In returns the input channel of the BlobStorageSink connector.
type BlobStorageSinkConfig ¶
type BlobStorageSinkConfig struct {
// The name of the Azure Blob storage container to write to.
Container string
// The number of concurrent workers to use when writing data to Azure Blob storage.
// The default is 1.
Parallelism int
// UploadOptions specifies set of configurations for the blob upload operation.
UploadOptions *blockblob.UploadStreamOptions
}
BlobStorageSinkConfig represents the configuration options for the Azure Blob storage sink connector.
type BlobStorageSource ¶
type BlobStorageSource struct {
// contains filtered or unexported fields
}
BlobStorageSource represents the Azure Blob storage source connector.
func NewBlobStorageSource ¶
func NewBlobStorageSource(ctx context.Context, client *azblob.Client, config *BlobStorageSourceConfig, logger *slog.Logger) *BlobStorageSource
NewBlobStorageSource returns a new BlobStorageSource. The connector reads all objects within the configured path and transmits them as a BlobStorageObject through the output channel.
func (*BlobStorageSource) Out ¶
func (s *BlobStorageSource) Out() <-chan any
Out returns the output channel of the BlobStorageSource connector.
func (*BlobStorageSource) Via ¶
func (s *BlobStorageSource) Via(operator streams.Flow) streams.Flow
Via asynchronously streams data to the given Flow and returns it.
type BlobStorageSourceConfig ¶
type BlobStorageSourceConfig struct {
// The name of the Azure Blob storage container to read from.
Container string
// The path within the container to use. If empty, the root of the container will be used.
Prefix string
// Indicates whether to ignore blob prefixes (virtual directories) in blob segments.
Flat bool
}
BlobStorageSourceConfig represents the configuration options for the Azure Blob storage source connector.
Source Files