timestream

Documentation

Overview

Package timestream provides a comprehensive set of tools for interacting with AWS Timestream. It includes functionalities for schema definition and management, as well as efficient marshalling and unmarshalling of data for AWS Timestream. The package is designed to be generic, allowing for flexible data types and simplifying the process of preparing data for Timestream as well as retrieving and interpreting query results.

Index

• func BuildQuery(template string, params map[string]interface{}) (string, error)
• func Marshal(v any) ([]types.Record, error)
• func Unmarshal(queryOutput *timestreamquery.QueryOutput, v any) error
• type DatabaseName
• type MeasureName
• type PredefinedValues
• type Record
• type Schema
• type TSSchema
  • func NewTSSchema[T1, T2 comparable](schema Schema[T1, T2]) TSSchema[T1, T2]
  • func (t TSSchema[T1, T2]) GenerateDummyData(db string, time time.Time, predefinedValues PredefinedValues[T2]) WriteRecords
  • func (s TSSchema[T1, T2]) GetMeasureNameFor(metricName T2) (string, error)
  • func (s TSSchema[T1, T2]) GetTableNameFor(metricName T2) (string, error)
• type Table
• type TableName
• type WriteRecords
  • func (w WriteRecords) RecordsForMeasure(measureName string) *timestreamwrite.WriteRecordsInput

Functions

func BuildQuery

func BuildQuery(template string, params map[string]interface{}) (string, error)

BuildQuery constructs a SQL query by replacing named placeholders within the template string with the corresponding values from the params map.

This function supports several types for parameter values: string, time.Time, int, int64, and float64. The replacement process involves: - Surrounding string values with single quotes. - Formatting time.Time values as RFC3339 strings, also surrounded with single quotes. - Directly inserting int, int64, and float64 values without additional formatting.

Placeholders in the template should be prefixed with a colon and followed by the key name. For example, a placeholder for a "startTime" parameter should be written as ":startTime".

Parameters:

• template: A SQL query template string containing named placeholders.
• params: A map where each key corresponds to a placeholder in the template, and the value is what will be used to replace the placeholder. The key should not include the colon prefix.

Returns:

• A string representing the final SQL query with all placeholders replaced by their respective values.
• An error if any placeholder is not found in the template or if a parameter type is not supported.

Example:

query, err := BuildQuery("SELECT * FROM table WHERE date > :startDate AND date < :endDate",
                         map[string]interface{}{"startDate": time.Now(), "endDate": time.Now().AddDate(0, 1, 0)})
if err != nil {
  // Handle error
}

Note:

The function ensures basic SQL injection prevention by correctly formatting and escaping
the parameter values based on their types. However, it's recommended to further validate
and sanitise all input values as per your application's security requirements.

func Marshal

func Marshal(v any) ([]types.Record, error)

Marshal takes a struct as input and transforms it into a types.Record compatible with AWS Timestream. The struct fields should be annotated with 'timestream' tags to indicate how they map to the Timestream data model.

Supported tag options:

• "timestamp": Indicates the field representing the timestamp for the record. The field must be of type time.Time and non-zero.
• "measure": Represents the measure name. It must be a non-empty string.
• "dimension": Used for dimensions in Timestream. Multiple dimensions are supported. Optionally, a 'name' can be specified (e.g., `timestream:"dimension,name=customName"`).
• "attribute": Represents measure values. Multiple measure values are supported. The field can be of a primitive type (string, int, float). For `time.Time` fields, you can specify the unit of time (s for seconds, ms for milliseconds, ns for nanoseconds) to format the timestamp accordingly, e.g., `timestream:"attribute,name=timestamp,unit=ms"`.
• "omitempty": This tag can only be applied to string fields. Fields with this tag are omitted if they are empty strings. For non-string fields, this tag will cause an error during marshalling. It is intended to reduce data size and handle optional string fields gracefully.

The function returns an error if the input is not a struct, does not meet the tagging requirements, or if any fields are of unsupported types.

Examples of struct field tags and their meanings:

type MyData struct {
    Timestamp   time.Time `timestream:"timestamp"`
    SensorName  string    `timestream:"measureName"`
    Location    string    `timestream:"dimension,name=location"`
    Temperature float64   `timestream:"attribute,name=temperature,omitempty"`
	EventTime   time.Time `timestream:"attribute,name=eventTime,unit=ms"`
}

Note: This function uses reflection to inspect struct fields. Fields with unsupported types or incorrect tagging will result in an error.

The function is designed to handle common use cases efficiently, but complex structs with deeply nested structures or a large number of fields may impact performance. Limitations: - The function does not support encoding of channel, complex, function values, or cyclic data structures. Attempting to encode such values will result in an error. - The function currently only supports basic types and time.Time for measure values. Custom types or types implementing specific interfaces are not currently supported. - There is a limitation in the depth of struct traversal; only the first level of fields is considered. Nested structs or embedded structs are not recursively processed.

It's important to ensure that structs passed to Marshal are well-formed according to the expectations of AWS Timestream data model, particularly regarding the types and formatting of timestamps, measure names, dimensions, and attributes. Example usage:

data := MyData{
    Time:        time.Now(),
    SensorName:  "Sensor1",
    Location:    "Room1",
    Temperature: 23.5,
	EventTime:   time.Now(),
}
record, err := Marshal(data)
if err != nil {
    // handle error
}
// use record with AWS Timestream

This function is part of a package designed to simplify the interaction with AWS Timestream, making the process of data preparation more straightforward and less error-prone.

func Unmarshal

func Unmarshal(queryOutput *timestreamquery.QueryOutput, v any) error

Unmarshal decodes data from Timestream query output into a struct or a slice of structs.

The 'v' parameter must be a pointer to a struct or a pointer to a slice of structs. The struct fields should be annotated with 'timestream' tags that specify how to map Timestream column names to struct fields. Supported struct field types are string, int, float64, and time.Time.

The function supports unmarshalling into either a single struct (if the query output contains a single row of data) or a slice of structs (if multiple rows are present). Each struct field's tag should match the Timestream column name, e.g., `timestream:"name=column_name"` for regular columns or `timestream:"time"` for the special timestamp column.

Example usage:

type MyData struct {
    Timestamp time.Time `timestream:"time"`
    Name      string    `timestream:"name=dimension_name"`
    Energy    float64   `timestream:"name=modelled_generation"`
    Power     int       `timestream:"name=actual_pv_power"`
}

var myData MyData err := Unmarshal(queryOutput, &myData)

if err != nil {
    // handle error
}

var myDataSlice []MyData err = Unmarshal(queryOutput, &myDataSlice)

if err != nil {
    // handle error
}

This function will return an error if: - The 'v' parameter is not a pointer. - The 'v' parameter is not a pointer to a struct or a slice of structs. - The length of the slice does not match the number of rows in the query output (when unmarshaling into a slice). - There is a mismatch between the number of columns in the query output and the number of fields in the struct.

Note: It's important to ensure that the types of the struct fields are compatible with the data types in the Timestream query output. For example, Timestream timestamps should be mapped to time.Time fields, and integers or floats in Timestream should be mapped to int or float64 fields in the struct, respectively.

Types

type DatabaseName

type DatabaseName string

type MeasureName

type MeasureName string

type PredefinedValues

type PredefinedValues[T comparable] map[T]float64

type Record

type Record[T1 comparable, T2 comparable] struct {
	Dimensions  []T1
	MetricNames []T2
}

type Schema

type Schema[T1 comparable, T2 comparable] map[Table]map[MeasureName]Record[T1, T2]

Schema represents a mapping from table names to measure names and then to a slice of metric names. It uses a generic type T, which must be a comparable type, allowing for flexibility in defining metric names.

type TSSchema

type TSSchema[T1 comparable, T2 comparable] struct {
	Schema Schema[T1, T2]
	// contains filtered or unexported fields
}

TSSchema represents a Timestream schema. It provides methods to retrieve measure and table names for given metric names. It uses a generic type T for metric names, allowing the use of custom types as long as they are comparable.

func NewTSSchema

func NewTSSchema[T1, T2 comparable](schema Schema[T1, T2]) TSSchema[T1, T2]

NewTSSchema initialises a new TSSchema instance from the given Schema. The schema parameter is a mapping from table names to measure names and then to metric names of the generic type T.

func (TSSchema[T1, T2]) GenerateDummyData

func (t TSSchema[T1, T2]) GenerateDummyData(db string, time time.Time, predefinedValues PredefinedValues[T2]) WriteRecords

GenerateDummyData generates dummy data based on the schema structure.

func (TSSchema[T1, T2]) GetMeasureNameFor

func (s TSSchema[T1, T2]) GetMeasureNameFor(metricName T2) (string, error)

GetMeasureNameFor retrieves the measure name associated with the given metric name. If the metric name is not found, it returns an error.

func (TSSchema[T1, T2]) GetTableNameFor

func (s TSSchema[T1, T2]) GetTableNameFor(metricName T2) (string, error)

GetTableNameFor retrieves the table name where the given metric name is stored. If the metric name is not found, it returns an error.

type Table

type Table string

type TableName

type TableName string

type WriteRecords

type WriteRecords []*timestreamwrite.WriteRecordsInput

func (WriteRecords) RecordsForMeasure

func (w WriteRecords) RecordsForMeasure(measureName string) *timestreamwrite.WriteRecordsInput