Documentation

Overview

Package kusto provides a client for accessing Azure Data Explorer, also known as Kusto.

For details on the Azure Kusto service, see: https://azure.microsoft.com/en-us/services/data-explorer/

For general documentation on APIs and the Kusto query language, see: https://docs.microsoft.com/en-us/azure/data-explorer/

Creating an Authorizer and a Client

To begin using this package, create an Authorizer and a client targeting your Kusto endpoint:

// auth package is: "github.com/Azure/go-autorest/autorest/azure/auth"

authorizer := kusto.Authorization{
	Config: auth.NewClientCredentialsConfig(clientID, clientSecret, tenantID),
}

client, err := kusto.New(endpoint, authorizer)
if err != nil {
	panic("add error handling")
}

For more examples on ways to create an Authorization object, see the Authorization object documentation.

Querying for Rows

Kusto provides a single method for querying, Query(). Query uses a Stmt object to provides SQL-like injection protection and accepts only string constants for arguments.

// table package is: data/table

// Query our database table "systemNodes" for the CollectionTimes and the NodeIds.
iter, err := client.Query(ctx, "database", kusto.NewStmt("systemNodes | project CollectionTime, NodeId"))
if err != nil {
	panic("add error handling")
}
defer iter.Stop()

// .Do() will call the function for every row in the table.
err = iter.Do(
	func(row *table.Row) error {
		fmt.Println(row) // As a convenience, printing a *table.Row will output csv
		return nil
	},
)
if err != nil {
	panic("add error handling")
}

Querying Rows Into Structs

Keeping our query the same, instead of printing the Rows we will simply put them into a slice of structs

// NodeRec represents our Kusto data that will be returned.
type NodeRec struct {
	// ID is the table's NodeId. We use the field tag here to to instruct our client to convert NodeId to ID.
	ID int64 `kusto:"NodeId"`
	// CollectionTime is Go representation of the Kusto datetime type.
	CollectionTime time.Time
}

iter, err := client.Query(ctx, "database", kusto.NewStmt("systemNodes | project CollectionTime, NodeId"))
if err != nil {
	panic("add error handling")
}
defer iter.Stop()

recs := []NodeRec{}
err = iter.Do(
	func(row *table.Row) error {
		rec := NodeRec{}
		if err := row.ToStruct(&rec); err != nil {
			return err
		}
		recs = append(recs, rec)
		return nil
	},
)
if err != nil {
	panic("add error handling")
}

A struct object can use fields to store the Kusto values as normal Go values, pointers to Go values and as value.Kusto types. The value.Kusto types are useful when you need to distiguish between the zero value of a variable and the value not being set in Kusto.

All value.Kusto types have a .Value and .Valid field. .Value is the native Go value, .Valid is a bool which indicates if the value was set. More information can be found in the sub-package data/value.

The following is a conversion table from the Kusto column types to native Go values within a struct that are allowed:

From Kusto Type			To Go Kusto Type
==============================================================================
bool				value.Bool, bool, *bool
------------------------------------------------------------------------------
datetime			value.DateTime, time.Time, *time.Time
------------------------------------------------------------------------------
dynamic				value.Dynamic, string, *string
------------------------------------------------------------------------------
guid				value.GUID, uuid.UUID, *uuid.UUID
------------------------------------------------------------------------------
int				value.Int, int32, *int32
------------------------------------------------------------------------------
long				value.Long, int64, *int64
------------------------------------------------------------------------------
real				value.Real, float64, *float64
------------------------------------------------------------------------------
string				value.String, string, *string
------------------------------------------------------------------------------
timestamp			value.Timestamp, time.Duration, *time.Duration
------------------------------------------------------------------------------
decimal				value.Decimal, string, *string
==============================================================================

For more information on Kusto scalar types, see: https://docs.microsoft.com/en-us/azure/kusto/query/scalar-data-types/

Stmt

Every query is done using a Stmt. A Stmt is built with Go string constants and can do variable substitution using Kusto's Query Paramaters.

// rootStmt builds a query that will gather all nodes in the DB.
rootStmt := kusto.NewStmt("systemNodes | project CollectionTime, NodeId")

// singleNodeStmt creates a new Stmt based on rootStmt and adds a "where" clause to find a single node.
// We pass a definition that sets the word ParamNodeId to a variable that will be substituted for a
// Kusto Long type (which is a a Go int64).
singleNodeStmt := rootStmt.Add(" | where NodeId == ParamNodeId").MustDefinitions(
	kusto.NewDefinitions().Must(
		kusto.ParamTypes{
			"ParamNodeId": kusto.ParamType{Type: types.Long},
		},
	),
)

// Query using our singleNodeStmt, variable substituting for ParamNodeId
iter, err := client.Query(
	ctx,
	"database",
	singleNode.MustParameters(
		kusto.NewParameters().Must(
			kusto.QueryValues{"ParamNodeId": int64(100)},
		),
	),
)

Ingest

Support for Kusto ingestion from local files, Azure Blob Storage and streaming is supported in the sub-package ingest. See documentation in that package for more details

Mocking

To support mocking for this client in your code for hermetic testing purposes, this client supports mocking the data returned by our RowIterator object. Please see the MockRows documentation for code examples.

Package Examples

Below you will find a simple and complex example of doing Query() the represent compiled code:

Example (Complex)
Output:

Example (Simple)
Output:

Index

Examples

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Authorization

type Authorization struct {
	// Authorizer provides an authorizer to use when talking to Kusto. If this is set, the
	// Authorizer must have its Resource (also called Resource ID) set to the endpoint passed
	// to the New() constructor. This will be something like "https://somename.westus.kusto.windows.net".
	// This package will try to set that automatically for you.
	Authorizer autorest.Authorizer
	// Config provides the authorizer's config that can create the authorizer. We recommending setting
	// this instead of Authorizer, as we will automatically set the Resource ID with the endpoint passed.
	Config auth.AuthorizerConfig
}

    Authorization provides the ADAL authorizer needed to access the resource. You can set Authorizer or Config, but not both.

    Example (Config)
    Output:
    
    
    Example (Msi)
    Output:
    
    

    func (*Authorization) Validate

    func (a *Authorization) Validate(endpoint string) error

      Validate validates the Authorization object against the endpoint an preps it for use. For internal use only.

      type Client

      type Client struct {
      	// contains filtered or unexported fields
      }

        Client is a client to a Kusto instance.

        func New

        func New(endpoint string, auth Authorization, options ...Option) (*Client, error)

          New returns a new Client. endpoint is the Kusto endpoint to use, example: https://somename.westus.kusto.windows.net .

          func (*Client) Auth

          func (c *Client) Auth() Authorization

            Auth returns the Authorization passed to New().

            func (*Client) Endpoint

            func (c *Client) Endpoint() string

              Endpoint returns the endpoint passed to New().

              func (*Client) Mgmt

              func (c *Client) Mgmt(ctx context.Context, db string, query Stmt, options ...MgmtOption) (*RowIterator, error)

                Mgmt is used to do management queries to Kusto. Details can be found at: https://docs.microsoft.com/en-us/azure/kusto/management/ Mgmt accepts a Stmt, but that Stmt cannot have any query parameters attached at this time. Note that the server has a timeout of 10 minutes for a management call by default unless the context deadline is set. There is a maximum of 1 hour.

                func (*Client) Query

                func (c *Client) Query(ctx context.Context, db string, query Stmt, options ...QueryOption) (*RowIterator, error)

                  Query queries Kusto for data. context can set a timeout or cancel the query. query is a injection safe Stmt object. Queries cannot take longer than 5 minutes by default and have row/size limitations. Note that the server has a timeout of 4 minutes for a query by default unless the context deadline is set. Queries can take a maximum of 1 hour.

                  Example (Do)
                  Output:
                  
                  
                  Example (Rows)
                  Output:
                  
                  
                  Example (Struct)
                  Output:
                  
                  

                  type Definitions

                  type Definitions struct {
                  	// contains filtered or unexported fields
                  }

                    Definitions represents definitions of parameters that are substituted for variables in a Kusto Query. This provides both variable substitution in a Stmt and provides protection against SQL-like injection attacks. See https://docs.microsoft.com/en-us/azure/kusto/query/queryparametersstatement?pivots=azuredataexplorer for internals. This object is not thread-safe and passing it as an argument to a function will create a copy that will share the internal state with the original.

                    func NewDefinitions

                    func NewDefinitions() Definitions

                      NewDefinitions is the constructor for Definitions.

                      func (Definitions) IsZero

                      func (p Definitions) IsZero() bool

                        IsZero indicates if the Definitions object is the zero type.

                        func (Definitions) Must

                        func (p Definitions) Must(types ParamTypes) Definitions

                          Must is the same as With(), but it must succeed or it panics.

                          func (Definitions) String

                          func (p Definitions) String() string

                            String implements fmt.Stringer.

                            func (Definitions) With

                            func (p Definitions) With(types ParamTypes) (Definitions, error)

                              With returns a copy of the Definitions object with the parameters names and types defined in "types".

                              type MgmtOption

                              type MgmtOption func(m *mgmtOptions) error

                                MgmtOption is an option type for a call to Mgmt().

                                func AllowWrite

                                func AllowWrite() MgmtOption

                                  AllowWrite allows a query that attempts to modify data in a table.

                                  func IngestionEndpoint

                                  func IngestionEndpoint() MgmtOption

                                    IngestionEndpoint will instruct the Mgmt call to connect to the ingest-[endpoint] instead of [endpoint]. This is not often used by end users and can only be used with a Mgmt() call.

                                    type MockRows

                                    type MockRows struct {
                                    	// contains filtered or unexported fields
                                    }

                                      MockRows provides the abilty to provide mocked Row data that can be played back from a RowIterator. This allows for creating hermetic tests from mock data or creating mock data from a real data fetch.

                                      func NewMockRows

                                      func NewMockRows(columns table.Columns) (*MockRows, error)

                                        NewMockRows is the constructor for MockRows.

                                        func (*MockRows) Error

                                        func (m *MockRows) Error(err error) error

                                          Error adds an error into the result stream. Nothing else added to this stream will matter once this is called.

                                          func (*MockRows) Row

                                          func (m *MockRows) Row(row value.Values) error

                                            Row adds Row data that will be replayed in a RowIterator.

                                            func (*MockRows) Struct

                                            func (m *MockRows) Struct(p interface{}) error

                                              Struct adds Row data that will be replayed in a RowIterator by parsing the passed *struct into value.Values.

                                              type Option

                                              type Option func(c *Client)

                                                Option is an optional argument type for New().

                                                type ParamType

                                                type ParamType struct {
                                                	// Type is the type of Column type this QueryParam will represent.
                                                	Type types.Column
                                                	// Default is a default value to use if the query doesn't provide this value.
                                                	// The value that can be set is defined by the Type:
                                                	// CTBool must be a bool
                                                	// CTDateTime must be a time.Time
                                                	// CTDynamic cannot have a default value
                                                	// CTGuid must be an uuid.UUID
                                                	// CTInt must be an int32
                                                	// CTLong must be an int64
                                                	// CTReal must be an float64
                                                	// CTString must be a string
                                                	// CTTimespan must be a time.Duration
                                                	// CTDecimal must be a string or *big.Float representing a decimal value
                                                	Default interface{}
                                                	// contains filtered or unexported fields
                                                }

                                                  ParamType provides type and default value information about the query parameter

                                                  type ParamTypes

                                                  type ParamTypes map[string]ParamType

                                                    ParamTypes is a list of parameter types and corresponding type data.

                                                    type Parameters

                                                    type Parameters struct {
                                                    	// contains filtered or unexported fields
                                                    }

                                                      Parameters represents values that will be substituted for a Stmt's Parameter. Keys are the names of corresponding Parameters, values are the value to be used. Keys must exist in the Parameter and value must be a Go type that corresponds to the ParamType.

                                                      func NewParameters

                                                      func NewParameters() Parameters

                                                        NewParameters is the construtor for Parameters.

                                                        func (Parameters) IsZero

                                                        func (q Parameters) IsZero() bool

                                                          IsZero returns if Parameters is the zero value.

                                                          func (Parameters) Must

                                                          func (q Parameters) Must(values QueryValues) Parameters

                                                            Must is the same as With() except any error is a panic.

                                                            func (Parameters) With

                                                            func (q Parameters) With(values QueryValues) (Parameters, error)

                                                              With returns a Parameters set to "values". values' keys represents Definitions names that will substituted for and the values to be subsituted.

                                                              type QueryOption

                                                              type QueryOption func(q *queryOptions) error

                                                                QueryOption is an option type for a call to Query().

                                                                func ResultsProgressiveDisable

                                                                func ResultsProgressiveDisable() QueryOption

                                                                  ResultsProgressiveDisable disables the progressive query stream.

                                                                  type QueryValues

                                                                  type QueryValues map[string]interface{}

                                                                    QueryValues represents a set of values that are substituted in Parameters. Every QueryValue key must have a corresponding Parameter name. All values must be compatible with the Kusto Column type it will go into (int64 for a long, int32 for int, time.Time for datetime, ...)

                                                                    type RowIterator

                                                                    type RowIterator struct {
                                                                    
                                                                    	// RequestHeader is the http.Header sent in the request to the server.
                                                                    	RequestHeader http.Header
                                                                    	// ResponseHeader is the http.header sent in the response from the server.
                                                                    	ResponseHeader http.Header
                                                                    	// contains filtered or unexported fields
                                                                    }

                                                                      RowIterator is used to iterate over the returned Row objects returned by Kusto.

                                                                      func (*RowIterator) Do

                                                                      func (r *RowIterator) Do(f func(r *table.Row) error) error

                                                                        Do calls f for every row returned by the query. If f returns a non-nil error, iteration stops.

                                                                        func (*RowIterator) Mock

                                                                        func (r *RowIterator) Mock(m *MockRows) error

                                                                          Mock is used to tell the RowIterator to return specific data for tests. This is useful when building fakes of the client's Query() call for hermetic tests. This can only be called in a test or it will panic.

                                                                          func (*RowIterator) Next

                                                                          func (r *RowIterator) Next() (*table.Row, error)

                                                                            Next gets the next Row from the query. io.EOF is returned if there are no more entries in the output. Once Next() returns an error, all subsequent calls will return the same error.

                                                                            func (*RowIterator) Progress

                                                                            func (r *RowIterator) Progress() float64

                                                                              Progress returns the progress of the query, 0-100%. This is only valid on Progressive data returns.

                                                                              func (*RowIterator) Progressive

                                                                              func (r *RowIterator) Progressive() bool

                                                                                Progressive indicates if the RowIterator is unpacking progressive (streaming) frames.

                                                                                func (*RowIterator) Stop

                                                                                func (r *RowIterator) Stop()

                                                                                  Stop is called to stop any further iteration. Always defer a Stop() call after receiving a RowIterator.

                                                                                  type Stmt

                                                                                  type Stmt struct {
                                                                                  	// contains filtered or unexported fields
                                                                                  }

                                                                                    Stmt is a Kusto Query statement. A Stmt is thread-safe, but methods on the Stmt are not. All methods on a Stmt do not alter the statement, they return a new Stmt object with the changes. This includes a copy of the Definitions and Parameters objects, if provided. This allows a root Stmt object that can be built upon. You should not pass *Stmt objects.

                                                                                    Example
                                                                                    Output:
                                                                                    
                                                                                    All Nodes Statement:
                                                                                     set notruncation;dcmInventoryComponentSystem
                                                                                    | project NodeId, Version
                                                                                    Single Statement:
                                                                                     declare query_parameters(Node:string);
                                                                                    set notruncation;dcmInventoryComponentSystem
                                                                                    | project NodeId, Version
                                                                                    | where NodeId == Node
                                                                                    
                                                                                    Single Statement Parameters:
                                                                                     {"Node":"my_id"}
                                                                                    Single Statement(Condensed):
                                                                                     declare query_parameters(Node:string);
                                                                                    set notruncation;dcmInventoryComponentSystem
                                                                                    | project NodeId, Version
                                                                                    | where NodeId == Node
                                                                                    
                                                                                    Single Statement(Repeat):
                                                                                     declare query_parameters(Node:string);
                                                                                    set notruncation;dcmInventoryComponentSystem
                                                                                    | project NodeId, Version
                                                                                    | where NodeId == Node
                                                                                    

                                                                                    func NewStmt

                                                                                    func NewStmt(query stringConstant, options ...StmtOption) Stmt

                                                                                      NewStmt creates a Stmt from a string constant.

                                                                                      func (Stmt) Add

                                                                                      func (s Stmt) Add(query stringConstant) Stmt

                                                                                        Add will add more text to the Stmt. This is similar to the + operator on two strings, except it only can be done with string constants. This allows dynamically building of a query from a root Stmt.

                                                                                        func (Stmt) MustDefinitions

                                                                                        func (s Stmt) MustDefinitions(defs Definitions) Stmt

                                                                                          MustDefinitions is the same as WithDefinitions with the exceptions that an error causes a panic.

                                                                                          func (Stmt) MustParameters

                                                                                          func (s Stmt) MustParameters(params Parameters) Stmt

                                                                                            MustParameters is the same as WithParameters with the exceptions that an error causes a panic.

                                                                                            func (Stmt) String

                                                                                            func (s Stmt) String() string

                                                                                              String implements fmt.Stringer. This can be used to see what the query statement to the server will be for debugging purposes.

                                                                                              func (Stmt) UnsafeAdd

                                                                                              func (s Stmt) UnsafeAdd(query string) Stmt

                                                                                                UnsafeAdd provides a method to add strings that are not injection protected to the Stmt. To utilize this method, you must create the Stmt with the UnsafeStmt() option and pass the unsafe.Stmt with .Add set to true. If not set, THIS WILL PANIC!

                                                                                                func (Stmt) ValuesJSON

                                                                                                func (s Stmt) ValuesJSON() (string, error)

                                                                                                  ValuesJSON returns a string in JSON format representing the Kusto QueryOptions.Parameters value that will be passed to the server. These values are substitued for Definitions in the Stmt and are represented by the Parameters that was passed.

                                                                                                  func (Stmt) WithDefinitions

                                                                                                  func (s Stmt) WithDefinitions(defs Definitions) (Stmt, error)

                                                                                                    WithDefinitions will return a Stmt that can be used in a Query() with Kusto Parameters to protect against SQL-like injection attacks. These Parameters must align with the placeholders in the statement. The new Stmt object will have a copy of the Parameters passed, not the original.

                                                                                                    func (Stmt) WithParameters

                                                                                                    func (s Stmt) WithParameters(params Parameters) (Stmt, error)

                                                                                                      WithParameters returns a Stmt that has the Parameters that will be substituted for Definitions in the query. Must have supplied the appropriate Definitions using WithQueryParamaters().

                                                                                                      type StmtOption

                                                                                                      type StmtOption func(s *Stmt)

                                                                                                        StmtOption is an optional argument to NewStmt().

                                                                                                        func UnsafeStmt

                                                                                                        func UnsafeStmt(options unsafe.Stmt) StmtOption

                                                                                                          UnsafeStmt enables unsafe actions on a Stmt and all Stmts derived from that Stmt. This turns off safety features that could allow a service client to compromise your data store. USE AT YOUR OWN RISK!

                                                                                                          Directories

                                                                                                          Path Synopsis
                                                                                                          data
                                                                                                          errors
                                                                                                          Package errors provides the error package for Kusto.
                                                                                                          Package errors provides the error package for Kusto.
                                                                                                          table
                                                                                                          Package table contains types that represent the makeup of a Kusto table.
                                                                                                          Package table contains types that represent the makeup of a Kusto table.
                                                                                                          types
                                                                                                          Package types holds Kusto type information that is used to describe what type would be held in a cell based on the column's type setting.
                                                                                                          Package types holds Kusto type information that is used to describe what type would be held in a cell based on the column's type setting.
                                                                                                          value
                                                                                                          Package value holds Kusto data value representations.
                                                                                                          Package value holds Kusto data value representations.
                                                                                                          Package ingest provides data ingestion from various external sources into Kusto.
                                                                                                          Package ingest provides data ingestion from various external sources into Kusto.
                                                                                                          test
                                                                                                          Package unsafe provides methods and types that loosen the native protections of the Kusto package.
                                                                                                          Package unsafe provides methods and types that loosen the native protections of the Kusto package.
                                                                                                          internal
                                                                                                          frames/unmarshal
                                                                                                          Package unmarshal provides decoding of Kusto row data in a frame into []value.Values representing those rows.
                                                                                                          Package unmarshal provides decoding of Kusto row data in a frame into []value.Values representing those rows.
                                                                                                          frames/unmarshal/json
                                                                                                          Package json implements encoding and decoding of JSON as defined in RFC 7159.
                                                                                                          Package json implements encoding and decoding of JSON as defined in RFC 7159.
                                                                                                          frames/v1
                                                                                                          Package v1 holds framing information for the v1 REST API.
                                                                                                          Package v1 holds framing information for the v1 REST API.
                                                                                                          frames/v2
                                                                                                          Package v2 holds framing information for the v2 REST API.
                                                                                                          Package v2 holds framing information for the v2 REST API.
                                                                                                          log
                                                                                                          version
                                                                                                          Package version keeps the internal version number of the client.
                                                                                                          Package version keeps the internal version number of the client.