Documentation

Overview

    The simplestreams package supports locating, parsing, and filtering metadata in simplestreams format. See http://launchpad.net/simplestreams and in particular the doc/README file in that project for more information about the file formats.

    Users of this package provide an empty struct and a matching function to be able to query and return a list of typed values for a given criteria.

    Index

    Constants

    View Source
    const (
    
    	// EXISTING_CLOUD_DATA is the lowest in priority.
    	// It is mostly used in merge functions
    	// where existing data does not need to be ranked.
    	EXISTING_CLOUD_DATA = 0
    
    	// DEFAULT_CLOUD_DATA is used for common cloud data that
    	// is shared an is publicly available.
    	DEFAULT_CLOUD_DATA = 10
    
    	// SPECIFIC_CLOUD_DATA is used to rank cloud specific data
    	// above commonly available.
    	// For e.g., openstack's "keystone catalogue".
    	SPECIFIC_CLOUD_DATA = 20
    
    	// CUSTOM_CLOUD_DATA is the highest available ranking and
    	// is given to custom data.
    	CUSTOM_CLOUD_DATA = 50
    )
    View Source
    const (
    	MirrorFile = "streams/v1/cpc-mirrors.json"
    
    	SignedSuffix   = ".sjson"
    	UnsignedSuffix = ".json"
    
    	// These constants define the currently supported simplestreams data formats.
    	IndexFormat   = "index:1.0"
    	ProductFormat = "products:1.0"
    	MirrorFormat  = "mirrors:1.0"
    )
    View Source
    const SimplestreamsPublicKeyFile = "publicsimplestreamskey"

    Variables

    View Source
    var EmptyCloudSpec = CloudSpec{}

      EmptyCloudSpec is used when we want all records regardless of cloud to be loaded.

      View Source
      var PGPSignatureCheckFn = func(keyring openpgp.KeyRing, signed, signature io.Reader) (*openpgp.Entity, error) {
      	return openpgp.CheckDetachedSignature(keyring, signed, signature)
      }

        PGPPGPSignatureCheckFn can be overridden by tests to allow signatures from non-trusted sources to be verified.

        Functions

        func DecodeCheckSignature

        func DecodeCheckSignature(r io.Reader, armoredPublicKey string) ([]byte, error)

          DecodeCheckSignature parses the inline signed PGP text, checks the signature, and returns plain text if the signature matches.

          func Encode

          func Encode(r io.Reader, armoredPrivateKey, passphrase string) ([]byte, error)

            Encode signs the data returned by the reader and returns an inline signed copy.

            func GetLatestMetadata

            func GetLatestMetadata(metadata *CloudMetadata, cons LookupConstraint, source DataSource, filterFunc AppendMatchingFunc) ([]interface{}, error)

              GetLatestMetadata extracts and returns the metadata records matching the given criteria.

              func MirrorsPath

              func MirrorsPath(streamsVersion string) string

                MirrorsPath returns the mirrors path for streamsVersion.

                func RegisterStructTags

                func RegisterStructTags(vals ...interface{})

                  RegisterStructTags ensures the json tags for the given structs are able to be used when parsing the simplestreams metadata.

                  func UnsignedIndex

                  func UnsignedIndex(streamsVersion string, indexFileVersion int) string

                    UnsignedIndex returns an unsigned index file name for streamsVersion.

                    func UnsignedMirror

                    func UnsignedMirror(streamsVersion string) string

                      UnsignedMirror returns an unsigned mirror file name for streamsVersion.

                      func UserPublicSigningKey

                      func UserPublicSigningKey() (string, error)

                        UserPublicSigningKey returns the public signing key (if defined).

                        Types

                        type AppendMatchingFunc

                        type AppendMatchingFunc func(DataSource, []interface{}, map[string]interface{}, LookupConstraint) ([]interface{}, error)

                          AppendMatchingFunc is the filter function signature used in our simple streams parsing code. It collects matching items from a DataSource in the returned interface slice.

                          type CloudMetadata

                          type CloudMetadata struct {
                          	Products   map[string]MetadataCatalog    `json:"products"`
                          	Aliases    map[string]aliasesByAttribute `json:"_aliases,omitempty"`
                          	Updated    string                        `json:"updated"`
                          	Format     string                        `json:"format"`
                          	ContentId  string                        `json:"content_id"`
                          	RegionName string                        `json:"region,omitempty"`
                          	Endpoint   string                        `json:"endpoint,omitempty"`
                          	Storage    string                        `json:"root_store,omitempty"`
                          	VirtType   string                        `json:"virt,omitempty"`
                          }

                          func ParseCloudMetadata

                          func ParseCloudMetadata(data []byte, format, url string, valueTemplate interface{}) (*CloudMetadata, error)

                            ParseCloudMetadata parses the given bytes into simplestreams metadata.

                            type CloudSpec

                            type CloudSpec struct {
                            	Region   string `json:"region"`
                            	Endpoint string `json:"endpoint"`
                            }

                              CloudSpec uniquely defines a specific cloud deployment.

                              type Config

                              type Config struct {
                              	// Description of the datasource
                              	Description string
                              
                              	// BaseURL is the URL for this datasource.
                              	BaseURL string
                              
                              	// HostnameVerification indicates whether to use self-signed credentials
                              	// and not try to verify the hostname on the TLS/SSL certificates.
                              	//
                              	// TODO (hml) 2020-07-16
                              	// It would be lovely to get rid of the SSLHostnameVerification
                              	// type and use the model-config value more.
                              	HostnameVerification utils.SSLHostnameVerification
                              
                              	// PublicSigningKey is the public key used to validate signed metadata.
                              	PublicSigningKey string
                              
                              	// Priority is an importance factor for the datasource. Higher number means
                              	// higher priority. This is will facilitate sorting data sources in order of
                              	// importance.
                              	Priority int
                              
                              	// RequireSigned indicates whether this datasource requires signed data.
                              	RequireSigned bool
                              
                              	// CACertificates contains an optional list of Certificate
                              	// Authority certificates to be used to validate certificates
                              	// of cloud infrastructure components
                              	// The contents are Base64 encoded x.509 certs.
                              	CACertificates []string
                              }

                                Config has values to be used in constructing a datasource.

                                func (*Config) Validate

                                func (c *Config) Validate() error

                                  Validate checks that the baseURL is valid and the description is set.

                                  type DataSource

                                  type DataSource interface {
                                  	// Description describes the origin of this datasource.
                                  	// eg agent-metadata-url, cloud storage, keystone catalog etc.
                                  	Description() string
                                  
                                  	// Fetch loads the data at the specified relative path. It returns a reader from which
                                  	// the data can be retrieved as well as the full URL of the file. The full URL is typically
                                  	// used in log messages to help diagnose issues accessing the data.
                                  	Fetch(path string) (io.ReadCloser, string, error)
                                  
                                  	// URL returns the full URL of the path, as applicable to this datasource.
                                  	// This method is used primarily for logging purposes.
                                  	URL(path string) (string, error)
                                  
                                  	// PublicSigningKey returns the public key used to validate signed metadata.
                                  	PublicSigningKey() string
                                  
                                  	// SetAllowRetry sets the flag which determines if the datasource will retry fetching the metadata
                                  	// if it is not immediately available.
                                  	SetAllowRetry(allow bool)
                                  
                                  	// Priority is an importance factor for Data Source. Higher number means higher priority.
                                  	// This is will allow to sort data sources in order of importance.
                                  	Priority() int
                                  
                                  	// RequireSigned indicates whether this data source requires signed data.
                                  	RequireSigned() bool
                                  }

                                    A DataSource retrieves simplestreams metadata.

                                    func NewDataSource

                                    func NewDataSource(cfg Config) DataSource

                                      NewDataSource returns a new DataSource as defined by the given config.

                                      type GetMetadataParams

                                      type GetMetadataParams struct {
                                      	StreamsVersion   string
                                      	LookupConstraint LookupConstraint
                                      	ValueParams      ValueParams
                                      }

                                        GetMetadataParams defines parameters used to load simplestreams metadata.

                                        type HasRegion

                                        type HasRegion interface {
                                        	// Region returns the necessary attributes to uniquely identify this cloud instance.
                                        	// Currently these attributes are "region" and "endpoint" values.
                                        	Region() (CloudSpec, error)
                                        }

                                          HasRegion is implemented by instances which can provide a region to which they belong. A region is defined by region name and endpoint.

                                          type IndexMetadata

                                          type IndexMetadata struct {
                                          	Updated          string      `json:"updated"`
                                          	Format           string      `json:"format"`
                                          	DataType         string      `json:"datatype"`
                                          	CloudName        string      `json:"cloudname,omitempty"`
                                          	Clouds           []CloudSpec `json:"clouds,omitempty"`
                                          	ProductsFilePath string      `json:"path"`
                                          	ProductIds       []string    `json:"products"`
                                          }

                                          func (*IndexMetadata) String

                                          func (metadata *IndexMetadata) String() string

                                          type IndexMetadataSlice

                                          type IndexMetadataSlice []*IndexMetadata

                                          type IndexReference

                                          type IndexReference struct {
                                          	Indices
                                          	MirroredProductsPath string
                                          	Source               DataSource
                                          	// contains filtered or unexported fields
                                          }

                                            Exported for testing.

                                            func GetIndexWithFormat

                                            func GetIndexWithFormat(source DataSource, indexPath, indexFormat, mirrorsPath string, requireSigned bool,
                                            	cloudSpec CloudSpec, params ValueParams) (*IndexReference, error)

                                              GetIndexWithFormat returns a simplestreams index of the specified format. Exported for testing.

                                              func (*IndexReference) GetCloudMetadataWithFormat

                                              func (indexRef *IndexReference) GetCloudMetadataWithFormat(cons LookupConstraint, format string, requireSigned bool) (*CloudMetadata, error)

                                                GetCloudMetadataWithFormat loads the entire cloud metadata encoded using the specified format. Exported for testing.

                                                func (*IndexReference) GetProductsPath

                                                func (indexRef *IndexReference) GetProductsPath(cons LookupConstraint) (string, error)

                                                  GetProductsPath returns the path to the metadata file containing products for the specified constraint. Exported for testing.

                                                  type Indices

                                                  type Indices struct {
                                                  	Indexes map[string]*IndexMetadata `json:"index"`
                                                  	Updated string                    `json:"updated"`
                                                  	Format  string                    `json:"format"`
                                                  }

                                                  type ItemCollection

                                                  type ItemCollection struct {
                                                  	Items      map[string]interface{} `json:"items"`
                                                  	Arch       string                 `json:"arch,omitempty"`
                                                  	Series     string                 `json:"release,omitempty"`
                                                  	Version    string                 `json:"version,omitempty"`
                                                  	RegionName string                 `json:"region,omitempty"`
                                                  	Endpoint   string                 `json:"endpoint,omitempty"`
                                                  	Storage    string                 `json:"root_store,omitempty"`
                                                  	VirtType   string                 `json:"virt,omitempty"`
                                                  	// contains filtered or unexported fields
                                                  }

                                                  func (*ItemCollection) UnmarshalJSON

                                                  func (c *ItemCollection) UnmarshalJSON(b []byte) error

                                                    ItemsCollection.UnmarshalJSON unmarshals the ItemCollection, storing the raw bytes for each item. These can later be unmarshalled again into product-specific types.

                                                    type LookupConstraint

                                                    type LookupConstraint interface {
                                                    	// IndexIds generates a string array representing index ids formed similarly to an ISCSI qualified name (IQN).
                                                    	IndexIds() []string
                                                    	// ProductIds generates a string array representing product ids formed similarly to an ISCSI qualified name (IQN).
                                                    	ProductIds() ([]string, error)
                                                    	// Params returns the constraint parameters.
                                                    	Params() LookupParams
                                                    }

                                                    type LookupParams

                                                    type LookupParams struct {
                                                    	CloudSpec
                                                    	Series []string
                                                    	Arches []string
                                                    	// Stream can be "" or "released" for the default "released" stream,
                                                    	// or "daily" for daily images, or any other stream that the available
                                                    	// simplestreams metadata supports.
                                                    	Stream string
                                                    }

                                                      LookupParams defines criteria used to find a metadata record. Derived structs implement the IndexIds() and ProductIds() method.

                                                      func (LookupParams) Params

                                                      func (p LookupParams) Params() LookupParams

                                                      type MetadataCatalog

                                                      type MetadataCatalog struct {
                                                      	Series     string `json:"release,omitempty"`
                                                      	Version    string `json:"version,omitempty"`
                                                      	Arch       string `json:"arch,omitempty"`
                                                      	RegionName string `json:"region,omitempty"`
                                                      	Endpoint   string `json:"endpoint,omitempty"`
                                                      	Storage    string `json:"root_store,omitempty"`
                                                      	VirtType   string `json:"virt,omitempty"`
                                                      
                                                      	// Items is a mapping from version to an ItemCollection,
                                                      	// where the version is the date the items were produced,
                                                      	// in the format YYYYMMDD.
                                                      	Items map[string]*ItemCollection `json:"versions"`
                                                      }

                                                      type MetadataLookupParams

                                                      type MetadataLookupParams struct {
                                                      	Region        string
                                                      	Series        string
                                                      	Architectures []string
                                                      	Endpoint      string
                                                      	Sources       []DataSource
                                                      	Stream        string
                                                      }

                                                      type MetadataValidator

                                                      type MetadataValidator interface {
                                                      	MetadataLookupParams(region string) (*MetadataLookupParams, error)
                                                      }

                                                        MetadataValidator instances can provide parameters used to query simplestreams metadata to find information for the specified parameters. If region is "", then the implementation may use its own default region if it has one, or else returns an error.

                                                        type MirrorInfo

                                                        type MirrorInfo struct {
                                                        	Clouds    []CloudSpec `json:"clouds"`
                                                        	MirrorURL string      `json:"mirror"`
                                                        	Path      string      `json:"path"`
                                                        }

                                                        type MirrorInfoSlice

                                                        type MirrorInfoSlice []MirrorInfo

                                                        type MirrorMetadata

                                                        type MirrorMetadata struct {
                                                        	Updated string                  `json:"updated"`
                                                        	Format  string                  `json:"format"`
                                                        	Mirrors map[string][]MirrorInfo `json:"mirrors"`
                                                        }

                                                        func GetMirrorMetadataWithFormat

                                                        func GetMirrorMetadataWithFormat(source DataSource, mirrorPath, format string,
                                                        	requireSigned bool) (*MirrorMetadata, error)

                                                          GetMirrorMetadataWithFormat returns simplestreams mirror data of the specified format. Exported for testing.

                                                          type MirrorRefSlice

                                                          type MirrorRefSlice []MirrorReference

                                                          type MirrorReference

                                                          type MirrorReference struct {
                                                          	Updated  string      `json:"updated"`
                                                          	Format   string      `json:"format"`
                                                          	DataType string      `json:"datatype"`
                                                          	Path     string      `json:"path"`
                                                          	Clouds   []CloudSpec `json:"clouds,omitempty"`
                                                          }

                                                          type MirrorRefs

                                                          type MirrorRefs struct {
                                                          	Mirrors map[string][]MirrorReference `json:"mirrors"`
                                                          }

                                                          type NotPGPSignedError

                                                          type NotPGPSignedError struct{}

                                                            NotPGPSignedError is used when PGP text does not contain an inline signature.

                                                            func (*NotPGPSignedError) Error

                                                            func (*NotPGPSignedError) Error() string

                                                            type ResolveInfo

                                                            type ResolveInfo struct {
                                                            	Source    string `yaml:"source" json:"source"`
                                                            	Signed    bool   `yaml:"signed" json:"signed"`
                                                            	IndexURL  string `yaml:"indexURL" json:"indexURL"`
                                                            	MirrorURL string `yaml:"mirrorURL,omitempty" json:"mirrorURL,omitempty"`
                                                            }

                                                            func GetMetadata

                                                            func GetMetadata(sources []DataSource, params GetMetadataParams) (items []interface{}, resolveInfo *ResolveInfo, err error)

                                                              GetMetadata returns metadata records matching the specified constraint,looking in each source for signed metadata. If onlySigned is false and no signed metadata is found in a source, the source is used to look for unsigned metadata. Each source is tried in turn until at least one signed (or unsigned) match is found.

                                                              type ValueParams

                                                              type ValueParams struct {
                                                              	// The simplestreams data type key.
                                                              	DataType string
                                                              	// The key to use when looking for content mirrors.
                                                              	MirrorContentId string
                                                              	// A function used to filter and return records of a given type.
                                                              	FilterFunc AppendMatchingFunc
                                                              	// An struct representing the type of records to return.
                                                              	ValueTemplate interface{}
                                                              }

                                                                ValueParams contains the information required to pull out from the metadata structs of a particular type.

                                                                Directories

                                                                Path Synopsis