arrowpb

README

arrowpb

Convert Apache Arrow records to Protocol Buffers

Introduction

arrowpb is a library designed to facilitate the conversion of Apache Arrow records into Protocol Buffers format. This allows for efficient serialization and deserialization of data structures, making it easier to work with data across different systems.

Feature

• Convert Arrow records to Protocol Buffers and vice versa.
• High performance and low memory overhead.
• Easy integration with existing Arrow and Protocol Buffers projects.

Author

arrowpb is developed by @TFMV.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Documentation

Index

• Variables
• func ArrowBatchToProto(reader array.RecordReader, protoDescriptor *descriptorpb.DescriptorProto) [][]byte
• func ArrowRecordToProto(record arrow.Record, descriptorProto *descriptorpb.DescriptorProto) ([]byte, error)
• func ArrowRowToProto(record arrow.Record, descriptorProto *descriptorpb.DescriptorProto, ...) ([]byte, error)
• func ArrowSchemaToProto(schema *arrow.Schema) *descriptorpb.DescriptorProto
• func CreateArrowRecord() (array.RecordReader, error)
• func CreateNestedMessage(fieldType *arrow.StructType, messageName string) *descriptorpb.DescriptorProto
• func ExtractValue(col arrow.Array, rowIndex int) any
• func FormatArrowJSON(reader array.RecordReader, output io.Writer) error
• func GenerateUniqueName(prefix string) string

Variables

var TypeMappings = map[arrow.DataType]descriptorpb.FieldDescriptorProto_Type{
	arrow.BinaryTypes.Binary:          descriptorpb.FieldDescriptorProto_TYPE_BYTES,
	arrow.FixedWidthTypes.Boolean:     descriptorpb.FieldDescriptorProto_TYPE_BOOL,
	arrow.PrimitiveTypes.Float64:      descriptorpb.FieldDescriptorProto_TYPE_DOUBLE,
	arrow.PrimitiveTypes.Int64:        descriptorpb.FieldDescriptorProto_TYPE_INT64,
	arrow.BinaryTypes.String:          descriptorpb.FieldDescriptorProto_TYPE_STRING,
	arrow.FixedWidthTypes.Date32:      descriptorpb.FieldDescriptorProto_TYPE_STRING,
	arrow.FixedWidthTypes.Timestamp_s: descriptorpb.FieldDescriptorProto_TYPE_STRING,
}

TypeMappings maps Apache Arrow types to Protocol Buffers field types.

Functions

func ArrowBatchToProto

func ArrowBatchToProto(reader array.RecordReader, protoDescriptor *descriptorpb.DescriptorProto) [][]byte

ArrowBatchToProto converts an Arrow RecordReader into serialized ProtoBuf messages.

func ArrowRecordToProto

func ArrowRecordToProto(record arrow.Record, descriptorProto *descriptorpb.DescriptorProto) ([]byte, error)

ArrowRecordToProto converts an Arrow RecordBatch into serialized ProtoBuf messages.

func ArrowRowToProto

func ArrowRowToProto(record arrow.Record, descriptorProto *descriptorpb.DescriptorProto, rowIndex int) ([]byte, error)

ArrowRowToProto converts a single row in an Arrow RecordBatch to a Protobuf message.

func ArrowSchemaToProto

func ArrowSchemaToProto(schema *arrow.Schema) *descriptorpb.DescriptorProto

ArrowSchemaToProto converts an Arrow schema into a Protocol Buffers descriptor.

func CreateArrowRecord

func CreateArrowRecord() (array.RecordReader, error)

CreateArrowRecord creates a sample Arrow RecordBatch for testing.

func CreateNestedMessage

func CreateNestedMessage(fieldType *arrow.StructType, messageName string) *descriptorpb.DescriptorProto

CreateNestedMessage constructs a nested ProtoBuf message descriptor.

func ExtractValue

func ExtractValue(col arrow.Array, rowIndex int) any

ExtractValue retrieves a value from an Arrow column at a specific row index.

func FormatArrowJSON

func FormatArrowJSON(reader array.RecordReader, output io.Writer) error

FormatArrowJSON formats Arrow records as pretty-printed JSON.

func GenerateUniqueName

func GenerateUniqueName(prefix string) string

GenerateUniqueName creates a unique name for the ProtoBuf message.