plugin

package
v2.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 30, 2022 License: Apache-2.0 Imports: 36 Imported by: 2

Documentation

Index

Constants

View Source 
const (
	// Require values
	Required = "required"
	Optional = "optional"
	AnyOf    = "any_of"
)
View Source 
const (
	SchemaModeStatic  = "static"
	SchemaModeDynamic = "dynamic"
)
View Source 
const ContextColumnName = "_ctx"

Variables

This section is empty.

Functions

func DiagsToError 

func DiagsToError(prefix string, diags hcl.Diagnostics) error

DiagsToError converts hcl diags into an error

func GetMatrixItem 

func GetMatrixItem(ctx context.Context) map[string]interface{}

GetMatrixItem extracts the matrix item map with the given key from the context

func IsCancelled 

func IsCancelled(ctx context.Context) bool

IsCancelled is a helper function which returns whether the context has been cancelled

func Logger 

func Logger(ctx context.Context) hclog.Logger

Logger extracts the logger from the context

func RetryHydrate 

func RetryHydrate(ctx context.Context, d *QueryData, hydrateData *HydrateData, hydrateFunc HydrateFunc, retryConfig *RetryConfig) (interface{}, error)

RetryHydrate function invokes the hydrate function with retryable errors and retries the function until the maximum attemptes before throwing error

func Serve 

func Serve(opts *ServeOpts)

Types

type Column 

type Column struct {
	// column name
	Name string
	// column type
	Type proto.ColumnType
	// column description
	Description string
	// explicitly specify the function which populates this data
	// - this is only needed if any of the default hydrate functions wil NOT return this column
	Hydrate HydrateFunc
	// the default column value
	Default interface{}
	//  a list of transforms to generate the column value
	Transform *transform.ColumnTransforms
}

Column is a struct representing a column defintion it is not mutated and contains column data, in a format compatible with proto ColumnDefinition

type ConcurrencyManager 

type ConcurrencyManager struct {
	// contains filtered or unexported fields
}

ConcurrencyManager struct ensures that hydrate functions stay within concurrency limits

func (*ConcurrencyManager) Close 

func (c *ConcurrencyManager) Close()

Close executs when the query is complete and dumps out the concurrency stats

func (*ConcurrencyManager) DisplayConcurrencyStats 

func (c *ConcurrencyManager) DisplayConcurrencyStats()

DisplayConcurrencyStats displays the the summary of all the concurrent hydrate calls

func (*ConcurrencyManager) Finished 

func (c *ConcurrencyManager) Finished(name string)

Finished decrements the counter for the named function

func (*ConcurrencyManager) StartIfAllowed 

func (c *ConcurrencyManager) StartIfAllowed(name string, maxCallConcurrency int) (res bool)

StartIfAllowed checks whether the named hydrate call is permitted to start based on the number of running instances of that call, and the total calls in progress

type Connection 

type Connection struct {
	Name string
	// the connection config
	// NOTE: we always pass and store connection config BY VALUE
	Config interface{}
}

type ConnectionConfigInstanceFunc 

type ConnectionConfigInstanceFunc func() interface{}

type ConnectionConfigSchema 

type ConnectionConfigSchema struct {
	Schema map[string]*schema.Attribute
	// function which returns an instance of a connection config struct
	NewInstance ConnectionConfigInstanceFunc
}

ConnectionConfigSchema struct is used to define the connection config schema and store the config for each plugin connection

func NewConnectionConfigSchema 

func NewConnectionConfigSchema() *ConnectionConfigSchema

func (*ConnectionConfigSchema) Parse 

func (c *ConnectionConfigSchema) Parse(configString string) (config interface{}, err error)

Parse function parses the hcl string into a connection config struct. The schema and the struct to parse into are provided by the plugin

func (*ConnectionConfigSchema) Validate 

func (c *ConnectionConfigSchema) Validate() []string

Validate validates the connection config

type CreatePlugin 

type CreatePlugin func(context.Context, string) (*Plugin, error)

type DefaultConcurrencyConfig 

type DefaultConcurrencyConfig struct {
	// max number of ALL hydrate calls in progress
	TotalMaxConcurrency   int
	DefaultMaxConcurrency int
}

DefaultConcurrencyConfig contains plugin level config to define default hydrate concurrency - this is used if no HydrateConfig is specified for a specific call

type ErrorPredicate 

type ErrorPredicate func(error) bool

type GetConfig 

type GetConfig struct {
	// key or keys which are used to uniquely identify rows - used to determine whether  a query is a 'get' call
	KeyColumns KeyColumnSlice
	// the hydrate function which is called first when performing a 'get' call.
	// if this returns 'not found', no further hydrate functions are called
	Hydrate HydrateFunc
	// a function which will return whenther to ignore a given error
	ShouldIgnoreError ErrorPredicate
	RetryConfig       *RetryConfig
}

GetConfig is a struct used to define the configuration of the table 'Get' function. This is the function used to retrieve a single row by id The config defines the function, the columns which may be used as id (KeyColumns), and the error handling behaviour

type HydrateCall 

type HydrateCall struct {
	Func HydrateFunc
	// the dependencies expressed using function name
	Depends []string
	Config  *HydrateConfig
	Name    string
}

HydrateCall struct encapsulates a hydrate call, its config and dependencies

func (HydrateCall) CanStart 

func (h HydrateCall) CanStart(rowData *RowData, name string, concurrencyManager *ConcurrencyManager) bool

func (*HydrateCall) Start 

func (h *HydrateCall) Start(ctx context.Context, r *RowData, d *QueryData, concurrencyManager *ConcurrencyManager)

Start starts a hydrate call

type HydrateConfig 

type HydrateConfig struct {
	Func           HydrateFunc
	MaxConcurrency int
	RetryConfig    *RetryConfig

	ShouldIgnoreError ErrorPredicate
	Depends           []HydrateFunc
}

HydrateConfig defines the hydrate function configurations, Name, Maximum number of concurrent calls to be allowed, dependencies

func (*HydrateConfig) DefaultTo added in v2.2.0 

func (c *HydrateConfig) DefaultTo(defaultConfig *HydrateConfig)

func (*HydrateConfig) String added in v2.2.0 

func (c *HydrateConfig) String() interface{}

type HydrateData 

type HydrateData struct {
	// if there was a parent-child list call, store the parent list item
	ParentItem     interface{}
	Item           interface{}
	HydrateResults map[string]interface{}
}

HydrateData contains the input data passed to every hydrate function

type HydrateDependencies 

type HydrateDependencies struct {
	Func    HydrateFunc
	Depends []HydrateFunc
}

HydrateDependencies defines the hydrate function dependencies - other hydrate functions which must be run first Deprecated: used HydrateConfig

type HydrateFunc 

type HydrateFunc func(context.Context, *QueryData, *HydrateData) (interface{}, error)

HydrateFunc is a function which retrieves some or all row data for a single row item.

func WrapHydrate 

func WrapHydrate(hydrateFunc HydrateFunc, shouldIgnoreError ErrorPredicate) HydrateFunc

WrapHydrate is a higher order function which returns a HydrateFunc which handles Ignorable errors

func (HydrateFunc) WithCache 

func (hydrate HydrateFunc) WithCache(args ...HydrateFunc) HydrateFunc

WithCache is a chainable function which wraps a hydrate call with caching and checks for pending execution of the same function

type KeyColumn 

type KeyColumn struct {
	Name       string
	Operators  []string
	Require    string
	CacheMatch string
}

KeyColumn is a struct representing the definition of a column used to filter and Get and List calls.

At least one key column must be defined for a Get call. They are optional for List calls.

Operators

This property specifies the accepted operators (from a possible set: "=", "<>", "<", "<=", ">", ">=")

Require

This property determines whether the column is required or optional. Possible values:

"required"

The key column must be provided as a query qualifier (i.e. in a where clause in the query).

"optional"

The key column is optional but if provided it will be used to filter the results.

"any_of"

Any one of the given columns must be provided.

CacheMatch

This property determines the logic used by the query results cache to determine whether a cached value matches a given query. Possible values:

"subset" [default value]

A cached item is considered a match (i.e. a cache hit) if the qual for the query is a subset of the quals for the cached item.

For example, is the cached qual is "val < 100", and the query qual is "val < 50", this would be considered a qual subset so would lead to a cache match

"exact"

A cached item is considered a match ONLY if the qual for the cached item is the same as as the qual for the query.

This is used for columns which are only populated if the qual for that column is passed. A common pattern is to provide a "filter" column, which is populated using the qual value provided. This filter value is used when making the API call the fetch the data. If no filter qual is provided, then the filter column returned by the plugin is empty.

This breaks the subset logic as if there is a cached data with no qual for the filter column, this cached data would contain null values for the filter column. This data would be considered a superset of the data returned from a query which provides a filter qual, which is incorrect as the data returned if a filter qual is passed would include a non null filter column

The solution is to set CacheMatch="exact"

func (*KeyColumn) InitialiseOperators 

func (k *KeyColumn) InitialiseOperators()

InitialiseOperators adds a default '=' operator is no operators are set, and converts "!=" to "<>".

func (*KeyColumn) SingleEqualsQual 

func (k *KeyColumn) SingleEqualsQual() bool

SingleEqualsQual returns whether this key column has a single = operator.

func (KeyColumn) String 

func (k KeyColumn) String() string

func (*KeyColumn) ToProtobuf 

func (k *KeyColumn) ToProtobuf() *proto.KeyColumn

ToProtobuf converts the KeyColumn to a protobuf object.

func (*KeyColumn) Validate 

func (k *KeyColumn) Validate() []string

Validate ensures 'Operators' and 'Require' are valid. It returns a list of validation error strings.

type KeyColumnEqualsQualMap 

type KeyColumnEqualsQualMap map[string]*proto.QualValue

KeyColumnEqualsQualMap is a map of column name to qual value, used to represent a map of any equals quals

func (KeyColumnEqualsQualMap) GetListQualValues 

func (m KeyColumnEqualsQualMap) GetListQualValues() map[string]*proto.QualValueList

GetListQualValues returns a map of all qual values with a List value

func (KeyColumnEqualsQualMap) String 

func (m KeyColumnEqualsQualMap) String() string

type KeyColumnQualMap 

type KeyColumnQualMap map[string]*KeyColumnQuals

KeyColumnQualMap is a map of KeyColumnQuals keyed by column name

func NewKeyColumnQualValueMap 

func NewKeyColumnQualValueMap(qualMap map[string]*proto.Quals, keyColumns KeyColumnSlice) KeyColumnQualMap

NewKeyColumnQualValueMap creates a KeyColumnQualMap from a qual map and a KeyColumnSlice

func (KeyColumnQualMap) GetListQualValues added in v2.1.0 

func (m KeyColumnQualMap) GetListQualValues() quals.QualSlice

GetListQualValues returns a slice of any quals we have which have a list value

func (KeyColumnQualMap) GetUnsatisfiedKeyColumns 

func (m KeyColumnQualMap) GetUnsatisfiedKeyColumns(columns KeyColumnSlice) KeyColumnSlice

func (KeyColumnQualMap) String 

func (m KeyColumnQualMap) String() string

func (KeyColumnQualMap) ToEqualsQualValueMap 

func (m KeyColumnQualMap) ToEqualsQualValueMap() map[string]*proto.QualValue

ToEqualsQualValueMap converts a KeyColumnQualMap to a column-qual value map, including only the

func (KeyColumnQualMap) ToQualMap 

func (m KeyColumnQualMap) ToQualMap() map[string]quals.QualSlice

ToQualMap converts the map into a simpler map of column to []Quals this is needed to avoid the transform package needing to reference plugin

type KeyColumnQuals 

type KeyColumnQuals struct {
	Name  string
	Quals quals.QualSlice
}

KeyColumnQuals is a struct representing all quals for a specific column

func (KeyColumnQuals) SatisfiesKeyColumn 

func (k KeyColumnQuals) SatisfiesKeyColumn(keyColumn *KeyColumn) bool

func (KeyColumnQuals) SingleEqualsQual 

func (k KeyColumnQuals) SingleEqualsQual() bool

type KeyColumnSlice 

type KeyColumnSlice []*KeyColumn

func AllColumns 

func AllColumns(columns []string) KeyColumnSlice

AllColumns creates a KeyColumnSlice based on a slice of column names, each with a single equals operator and Require=Required

func AnyColumn 

func AnyColumn(columns []string) KeyColumnSlice

AnyColumn Columns creates a KeyColumnSlice based on a slice of column names, each with a single equals operator and Require=AnyOf

func NewEqualsKeyColumnSlice 

func NewEqualsKeyColumnSlice(columns []string, require string) KeyColumnSlice

NewEqualsKeyColumnSlice creates a KeyColumnSlice from a list of column names, each with a single equals operator

func OptionalColumns 

func OptionalColumns(columns []string) KeyColumnSlice

OptionalColumns Columns creates a KeyColumnSlice based on a slice of column names, with a single equals operator and Require=Optional

func SingleColumn 

func SingleColumn(column string) KeyColumnSlice

SingleColumn creates a KeyColumnSlice based on a column name The created slice has a single KeyColumn using a single equals operator and Require=Required

func (KeyColumnSlice) AllEquals 

func (k KeyColumnSlice) AllEquals() bool

AllEquals returns whether all KeyColumns only use equals operators

func (*KeyColumnSlice) Find added in v2.1.0 

func (k *KeyColumnSlice) Find(name string) *KeyColumn

Find looks for a key column with the given name and returns it if found

func (KeyColumnSlice) IsAnyOf 

func (k KeyColumnSlice) IsAnyOf() bool

IsAnyOf returns whether all key columns have Require == AnyOf

func (KeyColumnSlice) String 

func (k KeyColumnSlice) String() string

func (KeyColumnSlice) StringSlice 

func (k KeyColumnSlice) StringSlice() []string

StringSlice converts a KeyColumnSlice to a slice of strings

func (KeyColumnSlice) ToProtobuf 

func (k KeyColumnSlice) ToProtobuf() []*proto.KeyColumn

ToProtobuf converts the KeyColumnSlice to a slice of protobuf KeyColumns

func (KeyColumnSlice) Validate 

func (k KeyColumnSlice) Validate() []string

Validate validates all child columns

type ListConfig 

type ListConfig struct {
	KeyColumns KeyColumnSlice
	// the list function, this should stream the list results back using the QueryData object, and return nil
	Hydrate HydrateFunc
	// the parent list function - if we list items with a parent-child relationship, this will list the parent items
	ParentHydrate     HydrateFunc
	ShouldIgnoreError ErrorPredicate
	RetryConfig       *RetryConfig
}

ListConfig is a struct used to define the configuration of the table 'List' function. This is the function used to retrieve rows of sata The config defines the function, the columns which may be used to optimise the fetch (KeyColumns), and the error handling behaviour

type MatrixItemFunc 

type MatrixItemFunc func(context.Context, *Connection) []map[string]interface{}

type NewPluginOptions 

type NewPluginOptions struct {
	ConnectionName   string
	ConnectionConfig string
}

type Plugin 

type Plugin struct {
	Name   string
	Logger hclog.Logger
	// TableMap is a map of all the tables in the plugin, keyed by the table name
	TableMap map[string]*Table
	// TableMapFunc is a callback function which can be used to populate the table map
	// this con optionally be provided by the plugin, and allows the connection config to be used in the table creation
	// (connection config is not available at plugin creation time)
	TableMapFunc             func(ctx context.Context, p *Plugin) (map[string]*Table, error)
	DefaultTransform         *transform.ColumnTransforms
	DefaultGetConfig         *GetConfig
	DefaultConcurrency       *DefaultConcurrencyConfig
	DefaultRetryConfig       *RetryConfig
	DefaultShouldIgnoreError ErrorPredicate
	// every table must implement these columns
	RequiredColumns        []*Column
	ConnectionConfigSchema *ConnectionConfigSchema
	// connection this plugin is instantiated for
	Connection *Connection
	// object to handle caching of connection specific data
	ConnectionManager *connection_manager.Manager
	// is this a static or dynamic schema
	SchemaMode string
	Schema     map[string]*proto.TableSchema
	// contains filtered or unexported fields
}

Plugin is a struct used define the GRPC plugin.

This includes the plugin schema (i.e. the tables provided by the plugin), as well as config for the default error handling and concurrency behaviour.

By convention, the package name for your plugin should be the same name as your plugin, and go files for your plugin (except main.go) should reside in a folder with the same name.

func (*Plugin) Execute 

func (p *Plugin) Execute(req *proto.ExecuteRequest, stream proto.WrapperPlugin_ExecuteServer) (err error)

Execute executes a query and streams the results using the given GRPC stream.

func (*Plugin) GetSchema 

func (p *Plugin) GetSchema() (*grpc.PluginSchema, error)

GetSchema returns the plugin schema. Note: the connection config must be set before calling this function.

func (*Plugin) Initialise 

func (p *Plugin) Initialise()

Initialise creates the 'connection manager' (which provides caching), sets up the logger and sets the file limit.

func (*Plugin) SetConnectionConfig 

func (p *Plugin) SetConnectionConfig(connectionName, connectionConfigString string) (err error)

SetConnectionConfig parses the connection config string, and populate the connection data for this connection. It also calls the table creation factory function, if provided by the plugin. Note: SetConnectionConfig is always called before any other plugin function.

func (*Plugin) Validate 

func (p *Plugin) Validate() string

type PluginFunc 

type PluginFunc func(context.Context) *Plugin

type QueryColumn 

type QueryColumn struct {
	*Column
	// contains filtered or unexported fields
}

QueryColumn is struct storing column name and resolved hydrate name this is used in the query data when the hydrate funciton has been resolved

func NewQueryColumn 

func NewQueryColumn(column *Column, hydrateName string) *QueryColumn

type QueryContext 

type QueryContext struct {
	Columns     []string
	UnsafeQuals map[string]*proto.Quals
	Limit       *int64
}

func NewQueryContext 

func NewQueryContext(p *proto.QueryContext) *QueryContext

NewQueryContext maps from a proto.QueryContext to a plugin.QueryContext. the only difference is the representation of the limit (as protobuf does not support pointers)

type QueryData 

type QueryData struct {
	// The table this query is associated with
	Table *Table
	// if this is a get call this will be populated with the quals as a map of column name to quals
	//  (this will also be populated for a list call if list key columns are specified -
	//  however this usage is deprecated and provided for legacy reasons only)
	KeyColumnQuals KeyColumnEqualsQualMap
	// a map of all key column quals which were specified in the query
	Quals KeyColumnQualMap
	// columns which have a single equals qual
	// is this a 'get' or a 'list' call
	FetchType fetchType
	// query context data passed from postgres - this includes the requested columns and the quals
	QueryContext *QueryContext
	// the status of the in-progress query
	QueryStatus *QueryStatus
	// connection details - the connection name and any config declared in the connection config file
	Connection *Connection
	// Matrix is an array of parameter maps (MatrixItems)
	// the list/get calls with be executed for each element of this array
	Matrix []map[string]interface{}
	// object to handle caching of connection specific data
	ConnectionManager *connection_manager.Manager

	// streaming funcs
	StreamListItem func(ctx context.Context, item interface{})

	// deprecated - plugins should no longer call StreamLeafListItem directly and should just call StreamListItem
	// event for the child list of a parent child list call
	StreamLeafListItem func(ctx context.Context, item interface{})
	// contains filtered or unexported fields
}

func (*QueryData) KeyColumnQualString 

func (d *QueryData) KeyColumnQualString(key string) string

KeyColumnQualString looks for the specified key column quals and if it exists, return the value as a string

func (*QueryData) ShallowCopy 

func (d *QueryData) ShallowCopy() *QueryData

ShallowCopy creates a shallow copy of the QueryData this is used to pass different quals to multiple list/get calls, when an in() clause is specified

type QueryStatus 

type QueryStatus struct {
	// contains filtered or unexported fields
}

func (*QueryStatus) RowsRemaining 

func (s *QueryStatus) RowsRemaining(ctx context.Context) int

RowsRemaining returns how many rows are required to complete the query

  • if no limit has been parsed from the query, this will return math.MaxInt32 (meaning an unknown number of rows remain)
  • if there is a limit, it will return the number of rows required to reach this limit
  • if the context has been cancelled, it will return zero

type RetryConfig 

type RetryConfig struct {
	ShouldRetryError ErrorPredicate
}

func (RetryConfig) String added in v2.2.0 

func (c RetryConfig) String() interface{}

type RowData 

type RowData struct {
	// the output of the get/list call which is passed to all other hydrate calls
	Item interface{}
	// if there was a parent-child list call, store the parent list item
	ParentItem interface{}
	// contains filtered or unexported fields
}

RowData contains the row data

func (*RowData) GetColumnData 

func (r *RowData) GetColumnData(column *QueryColumn) (interface{}, error)

GetColumnData returns the root item, and, if this column has a hydrate function registered, the associated hydrate data

type ServeOpts 

type ServeOpts struct {
	PluginName string
	PluginFunc PluginFunc
}

ServeOpts are the configurations to serve a plugin.

type Table 

type Table struct {
	Name string
	// table description
	Description string
	// column definitions
	Columns []*Column
	// the function used to list table rows
	List *ListConfig
	// the function used to efficiently retrieve a row by id
	Get *GetConfig
	// the function used when retrieving data for multiple 'matrix items', e.g. regions
	GetMatrixItem MatrixItemFunc
	// default transform applied to all columns
	DefaultTransform *transform.ColumnTransforms
	// function controlling default error handling behaviour
	DefaultShouldIgnoreError ErrorPredicate
	DefaultRetryConfig       *RetryConfig
	// the parent plugin object
	Plugin *Plugin
	// Deprecated: used HydrateConfig
	HydrateDependencies []HydrateDependencies
	// Config for any required hydrate functions, including dependencies between hydrate functions,
	// error handling and concurrency behaviour
	HydrateConfig []HydrateConfig
	// contains filtered or unexported fields
}

Table is a struct representing a plugin table. It defines the table columns, the function used to list table results (List) as well as (optionally) the function used to retrieve a single result by key (Get) and additional the functions required to fetch specific columns (HydrateConfig).

func (Table) GetSchema 

func (t Table) GetSchema() (*proto.TableSchema, error)

GetSchema returns the table schema Note: an additional '_ctx' column is added to all table schemas. This contains Steampipe specific data. (currently this is populated with the connection name)

func (*Table) ValidateColumnsExist 

func (t *Table) ValidateColumnsExist(keyColumns KeyColumnSlice) []string

Source Files

View all Source files

Directories

Path Synopsis
Transform package provides the ability to transform data from APIs.
 Transform package provides the ability to transform data from APIs.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.