mason

README

mason

Introduction

Mason creates and update virtual resources from existing physical resources. As an example, in order to create a GKE cluster you need a GCP Project. Mason will construct virtual resources based on the configuration of a given resource type. Each configuration defines its physical resources requirement. Mason will acquire those resources from Boskos and pass them to the specific implementation that will turn them into a virtual resource. Mason only handles lease and release of physical resources.

Configuration

Each configuration is represented by a name. The name of confguration maps to the virtual resource type in Boskos.

Here is an example configuration:

configs:
- name: type2
  needs:
    type1: 1
  config:
    type: GCPResourceConfig
    content: |
      projectconfigs:
      - type: type1
        clusters:
        - machinetype: n1-standard-2
          numnodes: 4
          version: 1.7
          zone: us-central-1f
        vms:
        - machinetype: n1-standard-4
          sourceimage: projects/debian-cloud/global/images/debian-9-stretch-v20180105
          zone: us-central-1f
          tags:
          - http-server
          - https-server
          scopes:
          - https://www.googleapis.com/auth/cloud-platform

As you can see in this example. In order to create a virtual resource of type2 we need one resource of type1. Once we have acquired the right resources, we need Masonable interface of type GCPResourceConfig that will know how to parse this configuration.

Operation

Mason is only operating on a given set of resource types. In order to use Mason, one needs to register a Masonable interface using the RegisterConfigConverter function.

A simple binary would like that:

func main() {
  flag.Parse()
  logrus.SetFormatter(&logrus.JSONFormatter{})

  if *configPath == "" {
    logrus.Panic("--config must be set")
  }

  mason := mason.NewMasonFromFlags()

  // Registering Masonable Converters
  if err := mason.RegisterConfigConverter(gcp.ResourceConfigType, gcp.ConfigConverter); err != nil {
    logrus.WithError(err).Panicf("unable tp register config converter")
  }
  if err := mason.UpdateConfigs(*configPath); err != nil {
    logrus.WithError(err).Panicf("failed to update mason config")
  }
  go func() {
    for range time.NewTicker(defaultUpdatePeriod).C {
      if err := mason.UpdateConfigs(*configPath); err != nil {
        logrus.WithError(err).Warning("failed to update mason config")
      }
    }
  }()

  mason.Start()
  defer mason.Stop()
  stop := make(chan os.Signal, 1)
  signal.Notify(stop, syscall.SIGINT, syscall.SIGTERM)
  <-stop

Recycling Thread

The recycling thread is acquiring dirty virtual resources and releasing the associated physical resources as dirty such Janitor can clean them up as an example. It then put the virtual resource to the Fullfilling queue.

Fullfilling Thread

The fulfilling thread will look at the config resource needs, and will acquire the necessary resources. Once a resource is acquired it will update the virtual resource LEASED_RESOURCES user data. Once all resources are acquired, it will be put on the cleaning queue.

Cleaning Thread

The cleaning thread will construct the virtual resources from the leased physical resources and update the user data with the user data provided by the Masonable implementation

Freeing Thread

The freeing thread will release all resources. It will release the leased physical resources with a state that is equal to the name of virtual resource and release the virtual resource as free.

Mason Client

Mason comes with its own client to ease usage. The mason client takes care of acquiring and release all the right resources from the User Data information.

References

Istio Deployment

Design Doc

Documentation

Index

• Constants
• func ParseConfig(configPath string) ([]common.ResourcesConfig, error)
• func ValidateConfig(configs []common.ResourcesConfig, resources []common.Resource) error
• type Client
  • func NewClient(boskosClient boskosClient) *Client
  • func (c *Client) Acquire(rtype, state, dest string) (*common.Resource, error)
  • func (c *Client) ReleaseOne(name, dest string) (allErrors error)
  • func (c *Client) UpdateAll(state string) error
• type ConfigConverter
• type Mason
  • func NewMason(cleanerCount int, client boskosClient, waitPeriod, syncPeriod time.Duration) *Mason
  • func (m *Mason) RegisterConfigConverter(name string, fn ConfigConverter) error
  • func (m *Mason) Start()
  • func (m *Mason) Stop()
  • func (m *Mason) UpdateConfigs(storagePath string) error
• type Masonable
• type Storage
  • func (s *Storage) AddConfig(conf common.ResourcesConfig) error
  • func (s *Storage) DeleteConfig(name string) error
  • func (s *Storage) GetConfig(name string) (common.ResourcesConfig, error)
  • func (s *Storage) GetConfigs() ([]common.ResourcesConfig, error)
  • func (s *Storage) SyncConfigs(newConfigs []common.ResourcesConfig) error
  • func (s *Storage) UpdateConfig(conf common.ResourcesConfig) error

Constants

const (
	// LeasedResources is a common.UserData entry
	LeasedResources = "leasedResources"
)

Functions

func ParseConfig

func ParseConfig(configPath string) ([]common.ResourcesConfig, error)

ParseConfig reads data stored in given config path In: configPath - path to the config file Out: A list of ResourceConfig object on success, or nil on error.

func ValidateConfig

func ValidateConfig(configs []common.ResourcesConfig, resources []common.Resource) error

ValidateConfig validates config with existing resources In: configs - a list of resources configs

resources - a list of resources

Out: nil on success, error on failure

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client extends boskos client with support with resource with leased resources

func NewClient

func NewClient(boskosClient boskosClient) *Client

NewClient creates a new client from a boskosClient interface

func (*Client) Acquire

func (c *Client) Acquire(rtype, state, dest string) (*common.Resource, error)

Acquire gets a resource with associated leased resources

func (*Client) ReleaseOne

func (c *Client) ReleaseOne(name, dest string) (allErrors error)

ReleaseOne will release a resource as well as leased resources associated to it

func (*Client) UpdateAll

func (c *Client) UpdateAll(state string) error

UpdateAll updates all the acquired resources with a given state

type ConfigConverter

type ConfigConverter func(string) (Masonable, error)

ConfigConverter converts a string into a Masonable

type Mason

type Mason struct {
	// contains filtered or unexported fields
}

Mason uses config to convert dirty resources to usable one

func NewMason

func NewMason(cleanerCount int, client boskosClient, waitPeriod, syncPeriod time.Duration) *Mason

NewMason creates and initialized a new Mason object In: rtypes - A list of resource types to act on

cleanerCount      - Number of cleaning threads
client            - boskos client
waitPeriod        - time to wait before a new boskos operation (acquire mostly)
syncPeriod        - time to wait before syncing resource information to boskos

Out: A Pointer to a Mason Object

func (*Mason) RegisterConfigConverter

func (m *Mason) RegisterConfigConverter(name string, fn ConfigConverter) error

RegisterConfigConverter is used to register a new Masonable interface In: name - identifier for Masonable implementation

fn   - function that will parse the configuration string and return a Masonable interface

Out: nil on success, error otherwise

func (*Mason) Start

func (m *Mason) Start()

Start Mason

func (*Mason) Stop

func (m *Mason) Stop()

Stop Mason

func (*Mason) UpdateConfigs

func (m *Mason) UpdateConfigs(storagePath string) error

UpdateConfigs updates configs from storage path In: storagePath - the path to read the config file from Out: nil on success error otherwise

type Masonable

type Masonable interface {
	Construct(context.Context, common.Resource, common.TypeToResources) (*common.UserData, error)
}

Masonable should be implemented by all configurations

type Storage

type Storage struct {
	// contains filtered or unexported fields
}

Storage stores configuration information

func (*Storage) AddConfig

func (s *Storage) AddConfig(conf common.ResourcesConfig) error

AddConfig adds a new config

func (*Storage) DeleteConfig

func (s *Storage) DeleteConfig(name string) error

DeleteConfig deletes an existing config if it exists or fail otherwise

func (*Storage) GetConfig

func (s *Storage) GetConfig(name string) (common.ResourcesConfig, error)

GetConfig returns an existing if it exists errors out otherwise

func (*Storage) GetConfigs

func (s *Storage) GetConfigs() ([]common.ResourcesConfig, error)

GetConfigs returns all configs

func (*Storage) SyncConfigs

func (s *Storage) SyncConfigs(newConfigs []common.ResourcesConfig) error

SyncConfigs syncs new configs

func (*Storage) UpdateConfig

func (s *Storage) UpdateConfig(conf common.ResourcesConfig) error

UpdateConfig updates a given if it exists or fail otherwise