internal

package
v1.0.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2025 License: Apache-2.0 Imports: 24 Imported by: 0

Documentation

Index

Constants

View Source 
const ApiServerEnvVarKey = "CHALK_API_SERVER"
View Source 
const ClientIdEnvVarKey = "CHALK_CLIENT_ID"
View Source 
const ClientSecretEnvVarKey = "CHALK_CLIENT_SECRET"
View Source 
const EnvironmentEnvVarKey = "CHALK_ACTIVE_ENVIRONMENT"

Variables

View Source 
var ChalkTag = "chalk"
View Source 
var CodecMemo = &CodecMemoT{}
View Source 
var FieldNotFoundError = errors.New("field not found")
View Source 
var NameTag = "name"
View Source 
var NamespaceMemos = &NamespaceMemosT{}
View Source 
var NowTimeFormat = "2006-01-02T15:04:05.000000-07:00"
View Source 
var TableReaderChunkSize = defaultTableReaderChunkSize
View Source 
var WindowBucketDurationRe = regexp.MustCompile(`^(\d+)([smhdw])$`)
View Source 
var WindowsTag = "windows"

Functions

func ChalkMarshal added in v0.3.13 

func ChalkMarshal(attrs map[string]any) ([]byte, error)

ChalkMarshal converts a map to a byte array. Follows the byte-packing format as described in the Chalk Python repo's `byte_transmit.serialize()` function.

func ChalkUnmarshal added in v0.3.6 

func ChalkUnmarshal(body []byte) (map[string]any, error)

func ChalkpySnakeCase added in v0.8.2 

func ChalkpySnakeCase(s string) string

ChalkpySnakeCase aims to be in parity with our Python implementation of snake_case. We are supposed to use this in all places over LegacySnakeCase, but that's a breaking change because our CLI generates the `name` struct tag if and only if the snake case generated using LegacySnakeCase does not match the actual feature name. If we switch to using ChalkpySnakeCase, that would make new codegen code incompatible with old chalk-go versions. In short, we want to keep using LegacySnakeCase in chalk-go to snake-case field names, and ChalkpySnakeCase to snake-case struct names.

func ColumnMapToRecord added in v0.7.0 

func ColumnMapToRecord(inputs map[string]any) (arrow.Record, error)

ColumnMapToRecord converts a map of column names to slices of values to an Arrow Record.

func ConvertBytesToTable added in v0.3.7 

func ConvertBytesToTable(byteArr []byte) (result arrow.Table, err error)

func ConvertIfHasManyMap added in v1.0.10 

func ConvertIfHasManyMap(value any) (any, error)

func CreateOnlineQueryBulkBody added in v0.3.6 

func CreateOnlineQueryBulkBody(inputs map[string]any, header FeatherRequestHeader) ([]byte, error)

func CreateUploadFeaturesBody added in v0.3.13 

func CreateUploadFeaturesBody(inputs map[string]any) ([]byte, error)

func ExpandTilde added in v0.3.3 

func ExpandTilde(path string) (string, error)

func ExtractFeaturesFromTable added in v0.7.0 

func ExtractFeaturesFromTable(
	table arrow.Table,
	timeAsString bool,
) ([]map[string]any, []map[string]FeatureMeta, error)

func FileExists 

func FileExists(path string) bool

func FormatBucketDuration 

func FormatBucketDuration(duration int) string

FormatBucketDuration takes an integer number of seconds and returns a string representation that satisfies these conditions:

  • the largest possible unit of time (e.g. "10m" instead of "600s")
  • a single unit of time (e.g. "601s" instead of "10m1s")

func GenerateGetEnumFunction 

func GenerateGetEnumFunction[K comparable](valueToEnum map[string]K, enumName string) func(string) (*K, error)

func GetBucketFromFqn added in v0.16.0 

func GetBucketFromFqn(fqn string) (string, error)

func GetContextWithTimeout added in v0.23.0 

func GetContextWithTimeout(requestCtx context.Context, clientLevelTimeout *time.Duration) (context.Context, context.CancelFunc)

func GetHeaderFromSerializedOnlineQueryBulkBody added in v0.5.1 

func GetHeaderFromSerializedOnlineQueryBulkBody(body []byte) (map[string]any, error)

func GetReflectValue added in v0.3.6 

func GetReflectValue(value any, typ reflect.Type, allMemo *NamespaceMemosT) (*reflect.Value, error)

GetReflectValue returns a reflect.Value of the given type from the given non-reflect value.

func GetValueFromArrowArray added in v0.7.0 

func GetValueFromArrowArray(a arrow.Array, idx int, timeAsString bool) (any, error)

func GetWindowBucketsFromStructTag added in v0.16.0 

func GetWindowBucketsFromStructTag(field reflect.StructField) ([]string, error)

func GetWindowBucketsSecondsFromStructTag added in v0.16.0 

func GetWindowBucketsSecondsFromStructTag(field reflect.StructField) ([]int, error)

func HasDontOmitTag added in v0.16.0 

func HasDontOmitTag(field reflect.StructField) bool

func InitRemoteFeatureMap added in v1.0.10 

func InitRemoteFeatureMap(
	remoteFeatureMap map[string][]reflect.Value,
	structValue reflect.Value,
	cumulativeFqn string,
	visited map[string]bool,
	scope *InitScope,
	allMemo *NamespaceMemosT,
	scopeToJustStructs bool,
) error

func InputsToArrowBytes added in v0.13.0 

func InputsToArrowBytes(inputs map[string]any) ([]byte, error)

InputsToArrowBytes converts map of FQNs to slice of values to an Arrow Record, serialized.

func Int64ToInt added in v0.18.27 

func Int64ToInt(value int64) (int, error)

func IsDataclass added in v0.3.6 

func IsDataclass(field reflect.Value) bool

func IsFeaturesClass added in v0.18.29 

func IsFeaturesClass(typ reflect.Type) bool

func IsStruct added in v0.16.0 

func IsStruct(typ reflect.Type) bool

func IsTypeDataclass added in v0.11.0 

func IsTypeDataclass(typ reflect.Type) bool

func KindMismatchError added in v0.8.4 

func KindMismatchError(expected reflect.Kind, actual reflect.Kind) error

func LegacySnakeCase added in v0.20.3 

func LegacySnakeCase(s string) string

LegacySnakeCase is how we turn our struct field names into feature names. We really should be using ChalkpySnakeCase, but we can't change it now because it would break existing codegen customers. Using this in our CLI to convert struct field names to snake case is still correct because if it does not correctly match the feature name, it generates the `name` struct tag. If we switch to using ChalkpySnakeCase, that would make new codegen code incompatible with old chalk-go versions. In short, we want to keep using LegacySnakeCase in chalk-go to snake-case field names, and ChalkpySnakeCase to snake-case struct names.

func ParseBucketDuration 

func ParseBucketDuration(durationStr string) (int, error)

ParseBucketDuration parses a bucket duration string and returns the duration in seconds. The input string must be of the form "Nunit" where N is a positive integer and unit is one of "s", "m", "h", "d", or "w".

func PopulateNamespaceMemos added in v1.0.10 

func PopulateNamespaceMemos(typ reflect.Type, visited map[reflect.Type]bool) error

PopulateNamespaceMemos populates a memo for each struct to make bulk-unmarshalling and has-many unmarshalling * efficient. i.e. Given:

  • type User struct {
  • Id *string
  • Transactions *[]Transactions `has_many:"id,user_id"`
  • Grade *int `versioned:"default(2)"`
  • GradeV1 *int `versioned:"true"`
  • GradeV2 *int `versioned:"true"`
  • }
  • type Transactions struct {
  • Id *string
  • UserId *string
  • Amount *float64
  • }
  • The namespace memo will be:
  • {
  • "User": {
  • ResolvedFieldNameToIndices: {
  • "id": [0],
  • "user.id": [0],
  • "grade@2": [2, 4],
  • "user.grade@2": [2, 4],
  • "grade": [3],
  • "user.grade": [3],
  • "transactions": [1],
  • "user.transactions": [1],
  • }
  • },
  • "Transactions": {
  • ResolvedFieldNameToIndices: {
  • "id": [0],
  • "transactions.id": [0],
  • "user_id": [1],
  • "transactions.user_id": [1],
  • "amount": [2],
  • "transactions.amount": [2],
  • }
  • }
  • }

func PreprocessIfStruct added in v0.19.0 

func PreprocessIfStruct(values any) (any, error)

func RecordToBytes added in v1.0.10 

func RecordToBytes(record arrow.Record) ([]byte, error)

func ReflectPtr added in v0.8.4 

func ReflectPtr(value reflect.Value) reflect.Value

func ResolveFeatureName added in v0.11.0 

func ResolveFeatureName(field reflect.StructField) (string, error)

func SetMapEntryValue added in v0.16.0 

func SetMapEntryValue(mapValue reflect.Value, key string, value any, allMemo *NamespaceMemosT) error

SetMapEntryValue exists as a separate special setter function because while all other fields are settable and can be passed into GetReflectValue to be set, map field values are not settable, and the entire map has to be passed instead.

func SingleInputsToBulkInputs added in v0.19.0 

func SingleInputsToBulkInputs(singleInputs map[string]any) (map[string]any, error)

func StringOrNil 

func StringOrNil(value string) *string

func ThinUnmarshalInto added in v1.0.10 

func ThinUnmarshalInto(
	structValue reflect.Value,
	fqnToValue map[string]any,
	namespace string,
	namespaceScope *InitScope,
	namespaceMemo *NamespaceMemo,
	allMemo *NamespaceMemosT,
) (returnErr error)

ThinUnmarshalInto is called per row. Any operation that can be done outside of this function must be done outside of this function.

func UnmarshalTableInto added in v1.0.10 

func UnmarshalTableInto(table arrow.Table, resultHolders any) (returnErr error)

Types

type BufferWriteSeeker added in v0.3.7 

type BufferWriteSeeker struct {
	// contains filtered or unexported fields
}

func (*BufferWriteSeeker) Bytes added in v0.3.7 

func (b *BufferWriteSeeker) Bytes() []byte

func (*BufferWriteSeeker) Seek added in v0.3.7 

func (b *BufferWriteSeeker) Seek(offset int64, whence int) (int64, error)

func (*BufferWriteSeeker) Write added in v0.3.7 

func (b *BufferWriteSeeker) Write(p []byte) (n int, err error)

type ChunkResult added in v0.18.27 

type ChunkResult struct {
	// contains filtered or unexported fields
}

type Codec added in v1.0.13 

type Codec func(structValue reflect.Value, arr arrow.Array, arrIdx int) error

type CodecMemoT added in v1.0.13 

type CodecMemoT = MemosT[string, Codec]

type FeatherRequestHeader added in v0.4.7 

type FeatherRequestHeader struct {
	Outputs          []string            `json:"outputs"`
	BranchId         *string             `json:"branch_id"`
	Explain          bool                `json:"explain"`
	Context          *OnlineQueryContext `json:"context"`
	Staleness        map[string]string   `json:"staleness"`
	Now              []string            `json:"now,omitempty"`
	IncludeMeta      bool                `json:"include_meta"`
	CorrelationId    *string             `json:"correlation_id"`
	QueryName        *string             `json:"query_name"`
	QueryNameVersion *string             `json:"query_name_version"`
	QueryContext     *map[string]any     `json:"query_context"`
	Meta             map[string]string   `json:"meta"`
	StorePlanStages  bool                `json:"store_plan_stages"`
	PlannerOptions   map[string]any      `json:"planner_options"`
}

type FeatureEncodingOptions added in v0.17.0 

type FeatureEncodingOptions struct {
	EncodeStructsAsObjects bool `json:"encode_structs_as_objects"`
}

type FeatureMeta added in v0.21.13 

type FeatureMeta struct {
	SourceType  *string
	SourceId    *string
	ResolverFqn *string
	Pkey        any
}

type GetValueFunc added in v1.0.13 

type GetValueFunc func(arr arrow.Array, arrIdx int) (reflect.Value, error)

type InitFeatureFunc added in v1.0.13 

type InitFeatureFunc func(structValue reflect.Value) (leafStructValue reflect.Value, err error)

type InitScope added in v1.0.10 

type InitScope struct {
	Children map[string]*InitScope
}

func BuildScope added in v1.0.10 

func BuildScope(fqns []string) (*InitScope, error)

type Memo added in v1.0.10 

type Memo[V any] struct {
	// contains filtered or unexported fields
}

type MemosT added in v1.0.10 

type MemosT[K, V any] sync.Map

func (*MemosT[K, V]) LoadOrStore added in v1.0.10 

func (m *MemosT[K, V]) LoadOrStore(key K, generateObject func() (*V, error)) (*V, error)

type NamespaceMemo added in v0.18.26 

type NamespaceMemo struct {
	// Rooted and non-rooted FQN as keys
	ResolvedFieldNameToIndices map[string][]int
	// Non-rooted FQN as keys only
	StructFieldsSet map[string]bool
}

type NamespaceMemosT added in v1.0.10 

type NamespaceMemosT MemosT[reflect.Type, NamespaceMemo]

func (*NamespaceMemosT) LoadOrStore added in v1.0.10 

func (m *NamespaceMemosT) LoadOrStore(structType reflect.Type) (*NamespaceMemo, error)

func (*NamespaceMemosT) LoadOrStoreWithVisited added in v1.0.10 

func (m *NamespaceMemosT) LoadOrStoreWithVisited(structType reflect.Type, visited map[reflect.Type]bool) (*NamespaceMemo, error)

type Numbers added in v0.3.6 

type Numbers interface {
	int | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64 | float32 | float64
}

type OfflineQueryInputSerialized added in v0.14.2 

type OfflineQueryInputSerialized struct {
	Columns []string `json:"columns"`
	Values  [][]any  `json:"values"`
}

type OfflineQueryRequestSerialized added in v0.14.2 

type OfflineQueryRequestSerialized struct {
	Input                OfflineQueryInputSerialized `json:"input"`
	Output               []string                    `json:"output"`
	RequiredOutput       []string                    `json:"required_output"`
	DatasetName          *string                     `json:"dataset_name"`
	Branch               *string                     `json:"branch"`
	MaxSamples           *int                        `json:"max_samples"`
	DestinationFormat    string                      `json:"destination_format"`
	JobId                *string                     `json:"job_id"`
	MaxCacheAge          *int                        `json:"max_cache_age_secs"`
	ObservedAtLowerBound *time.Time                  `json:"observed_at_lower_bound"`
	ObservedAtUpperBound *time.Time                  `json:"observed_at_upper_bound"`
	QueryContext         *map[string]any             `json:"query_context"`
	Tags                 []string                    `json:"tags"`
}

type OnlineQueryContext added in v0.14.2 

type OnlineQueryContext struct {
	Environment          *string  `json:"environment"`
	Tags                 []string `json:"tags"`
	RequiredResolverTags []string `json:"required_resolver_tags"`
}

type OnlineQueryRequestSerialized added in v0.14.2 

type OnlineQueryRequestSerialized struct {
	Inputs           map[string]any         `json:"inputs"`
	Outputs          []string               `json:"outputs"`
	Context          OnlineQueryContext     `json:"context"`
	Staleness        map[string]string      `json:"staleness"`
	IncludeMeta      bool                   `json:"include_meta"`
	DeploymentId     *string                `json:"deployment_id"`
	QueryName        *string                `json:"query_name"`
	CorrelationId    *string                `json:"correlation_id"`
	QueryContext     *map[string]any        `json:"query_context"`
	Meta             map[string]string      `json:"meta"`
	QueryNameVersion *string                `json:"query_name_version"`
	Now              *string                `json:"now"`
	Explain          bool                   `json:"explain"`
	StorePlanStages  bool                   `json:"store_plan_stages"`
	EncodingOptions  FeatureEncodingOptions `json:"encoding_options"`
	PlannerOptions   map[string]any         `json:"planner_options"`
}

type ResultMetadataSourceType added in v0.21.13 

type ResultMetadataSourceType string
const (
	SourceTypeOnlineStore ResultMetadataSourceType = "online_store"
)

type SetMapFunc added in v1.0.13 

type SetMapFunc func(
	mapVal reflect.Value,
	entryValue reflect.Value,

	isValid bool,
)

Source Files

View all Source files

Directories

Path Synopsis
tests

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.