Documentation
¶
Overview ¶
Package arrowio reads and writes frames as Apache Arrow data: the Arrow IPC file format (Feather v2) and the Arrow IPC streaming format, and converts frames to and from Arrow record batches and tables; package parquetio builds Parquet on these conversions. It is built on github.com/apache/arrow-go, the Apache Arrow project's own pure-Go implementation, and needs no cgo.
Files written here read back in pandas (through pyarrow) as the same frame, index included, and files pandas writes read here with their index: both sides use the "pandas" schema metadata that records which columns hold the index levels, or the range a default index spans.
Type mapping ¶
dtype written as read from float64 double double float32 float float int8..64 int8..int64 int8..int64 uint8..64 uint8..uint64 uint8..uint64 bool bool bool string utf8 (large_utf8 past 2 GiB) utf8, large_utf8, null datetime timestamp[ns, tz=UTC] timestamp[s|ms|us|ns], any zone category dictionary<int32, utf8> dictionary<any int, utf8|large_utf8> period utf8, its text form - complex ErrUnsupported -
Missing values travel in the Arrow validity bitmaps both ways. A float column has NaN as its missing value, as in pandas: NaN is written as null (pandas writes it so too), and a null float reads as NaN. Arrow keeps a non-null NaN apart from null; here both read as a missing value. A null in any other column becomes a masked value. In the "pandas" metadata, an integer or bool column with missing values is described as pandas' nullable Int64/boolean dtype, so pandas reads it back without turning it into floats.
A datetime column holds UTC instants. It is written with the zone "UTC" (WithTimeZone changes the zone, or drops it to write naive timestamps); on read the zone is ignored, since Arrow timestamps are UTC instants whatever their zone, and a naive timestamp is read as if it were UTC. Units coarser than nanoseconds are scaled, and a value outside the nanosecond range (years 1677-2262) is an ErrOutOfRange.
A period column is written as its text, what Series.Strings gives: Arrow has no calendar-period type, and pandas itself stores periods through a pyarrow extension type that other Arrow readers do not know. It reads back as a string column.
The index ¶
A default range index is written as metadata only (a range descriptor), as pandas does. Any other index is written as one column per level, after the data columns, named after the level, or __index_level_k__ when the level is unnamed. On read the metadata is honoured: a range descriptor starting at 0 with step 1 gives a range index; any other range gives an int64 index of the same labels; index columns become the levels of a (possibly multi-level) index. Without metadata the frame gets a range index and every column is a data column.
Index ¶
- Variables
- func FromRecord(rec arrow.RecordBatch) (*dataframe.DataFrame, error)
- func FromRecords(schema *arrow.Schema, recs []arrow.RecordBatch) (*dataframe.DataFrame, error)
- func FromTable(tbl arrow.Table) (*dataframe.DataFrame, error)
- func ReadFeather(path string, opts ...Option) (*dataframe.DataFrame, error)
- func ReadIPCFile(r ReadAtSeeker, opts ...Option) (*dataframe.DataFrame, error)
- func ReadIPCStream(r io.Reader, opts ...Option) (*dataframe.DataFrame, error)
- func ToRecord(df *dataframe.DataFrame, opts ...Option) (arrow.RecordBatch, error)
- func WriteFeather(path string, df *dataframe.DataFrame, opts ...Option) error
- func WriteIPCFile(w io.Writer, df *dataframe.DataFrame, opts ...Option) error
- func WriteIPCStream(w io.Writer, df *dataframe.DataFrame, opts ...Option) error
- type Compression
- type Option
- type ReadAtSeeker
Constants ¶
This section is empty.
Variables ¶
var ( // ErrUnsupported reports a dtype or an Arrow type with no mapping, or an // option a format does not offer. ErrUnsupported = errors.New("arrowio: unsupported") // ErrOutOfRange reports a value that does not fit the dtype it maps to, // such as a timestamp in seconds beyond the nanosecond range. ErrOutOfRange = errors.New("arrowio: value out of range") // ErrMetadata reports "pandas" schema metadata that does not parse or // names columns the data lacks. ErrMetadata = errors.New("arrowio: bad pandas metadata") )
Errors returned by the package, wrapped with detail; test with errors.Is. Errors from the Arrow decoders themselves (a corrupt file, a failing writer) are returned as they come.
Functions ¶
func FromRecord ¶
func FromRecord(rec arrow.RecordBatch) (*dataframe.DataFrame, error)
FromRecord converts an Arrow record batch to a frame: one column per field, the index restored from "pandas" schema metadata when there is some (a range index otherwise). The values are copied: the frame does not keep the record alive, and the caller still owns it.
func FromRecords ¶
FromRecords converts a sequence of record batches sharing one schema (an Arrow IPC stream, say) to one frame, as FromRecord does. Category columns whose dictionary changes from one batch to the next are unified.
func FromTable ¶
FromTable converts an Arrow table, whose columns may each be split in several chunks, to a frame, as FromRecord does.
func ReadFeather ¶
ReadFeather reads the Arrow IPC file (Feather v2) at path.
func ReadIPCFile ¶
func ReadIPCFile(r ReadAtSeeker, opts ...Option) (*dataframe.DataFrame, error)
ReadIPCFile reads a frame from the Arrow IPC file format (Feather v2, as pandas' to_feather writes it), every record batch in order. The only option it takes is WithAllocator.
func ReadIPCStream ¶
ReadIPCStream reads a frame from the Arrow IPC streaming format, every record batch in order. The only option it takes is WithAllocator.
func ToRecord ¶
ToRecord converts a frame to an Arrow record batch: one field per column, in order, then (unless WithoutIndex) one field per level of a labelled index, and "pandas" schema metadata describing the index and the dtypes, so that pandas reads the data back as the same frame. Numeric and datetime columns share the frame's storage rather than copying it. The caller owns the record and must Release it.
func WriteFeather ¶
WriteFeather writes a frame to path in the Arrow IPC file format (Feather v2), as WriteIPCFile does.
func WriteIPCFile ¶
WriteIPCFile writes a frame in the Arrow IPC file format (Feather v2), as ToRecord lays it out. The options are WithCompression (LZ4 or Zstd; none by default), WithTimeZone, WithoutIndex and WithAllocator.
Types ¶
type Compression ¶
type Compression int
Compression selects the codec of an Arrow IPC file or stream. The Arrow IPC formats offer only LZ4 (the LZ4 frame format) and Zstd; Parquet has its own codecs, in package parquetio.
const ( // Uncompressed is the default. Uncompressed Compression = iota LZ4 Zstd )
The codecs.
type Option ¶
type Option func(*config)
Option configures a write or a read.
func WithAllocator ¶
WithAllocator sets the Arrow memory allocator used to read and write (default memory.DefaultAllocator).
func WithCompression ¶
func WithCompression(c Compression) Option
WithCompression sets the codec of a written IPC file or stream.
func WithTimeZone ¶
WithTimeZone sets the time zone written with datetime columns (default "UTC"). The values are UTC instants whatever the zone: it only tells a reader how to show them. The empty zone writes naive timestamps, which pandas reads as datetime64[ns] without a zone.
func WithoutIndex ¶
func WithoutIndex() Option
WithoutIndex leaves the row labels out of a written file (pandas' index=False): no index columns and no index metadata, so a reader sees a default range index.
Source Files