ccachemanager

Documentation

Index

• Constants
• type CCacheArchive
• type CCacheConfiguration
• type CCacheGroupConfig
• type CCacheManager
  • func CreateManager(rootDir string, configFileName string) (m *CCacheManager, err error)
  • func (m *CCacheManager) DownloadPkgGroupCCache() (err error)
  • func (m *CCacheManager) SetCurrentPkgGroup(basePackageName string, arch string) (err error)
  • func (m *CCacheManager) UploadMultiPkgGroupCCaches() (err error)
  • func (m *CCacheManager) UploadPkgGroupCCache() (err error)
• type CCachePkgGroup
  • func (g *CCachePkgGroup) UpdateTagsPaths(remoteStoreConfig *RemoteStoreConfig, localDownloadsDir string, ...)
  • func (g *CCachePkgGroup) UpdateTarPaths(remoteStoreConfig *RemoteStoreConfig, localDownloadsDir string, ...)
• type RemoteStoreConfig

Constants

const (
	CCacheTarSuffix = "-ccache.tar.gz"
	CCacheTagSuffix = "-latest-build.txt"
	// This are just place holders when constructing a new manager object.
	UninitializedGroupName         = "unknown"
	UninitializedGroupSize         = 0
	UninitializedGroupArchitecture = "unknown"
)

Types

type CCacheArchive

type CCacheArchive struct {
	LocalSourcePath  string
	RemoteSourcePath string
	LocalTargetPath  string
	RemoteTargetPath string
}

Note that the design separate the artifacts we download (source) from those that we upload (target). What we start with (source), has a remote path on the remote storage, and is downloaded to disk at the local path). What the generate (target), has a local path where we create the archive, and uploaded to the remote store at the remote target path.

type CCacheConfiguration

type CCacheConfiguration struct {
	RemoteStoreConfig *RemoteStoreConfig  `json:"remoteStore"`
	Groups            []CCacheGroupConfig `json:"groups"`
}

type CCacheGroupConfig

type CCacheGroupConfig struct {
	Name         string   `json:"name"`
	Comment      string   `json:"comment"`
	Enabled      bool     `json:"enabled"`
	PackageNames []string `json:"packageNames"`
}

CCacheGroupConfig is where package groups are defined. A package group is a group of packages that can share the same ccache artifacts. This is typical for packages like kernel and kernel-hci, for example. A package group can have an arbitrary name, and a list of package names associated with it. A package group can also be disabled if the ccache breaks its build. This is usually a bug - and would need to be investigated further. The field 'comment' can be used to clarify any configuration related to this package family.

type CCacheManager

type CCacheManager struct {
	// Full path to the ccache json configuration file.
	ConfigFileName string

	// The in-memory representation of the ConfigFile contents.
	Configuration *CCacheConfiguration

	// ccache root folder as specified by build pipelines.
	RootWorkDir string

	// Working folder where CCacheManager will download artifacts.
	LocalDownloadsDir string

	// Working folder where CCacheManager will create archives in preparation
	// for uploading them.
	LocalUploadsDir string

	// Pointer to the current active pkg group state/configuration.
	CurrentPkgGroup *CCachePkgGroup

	// A utility helper for downloading/uploading archives from/to Azure blob
	// storage.
	AzureBlobStorage *azureblobstorage.AzureBlobStorage
}

CCacheManager is the main object...

func CreateManager

func CreateManager(rootDir string, configFileName string) (m *CCacheManager, err error)

func (*CCacheManager) DownloadPkgGroupCCache

func (m *CCacheManager) DownloadPkgGroupCCache() (err error)

func (*CCacheManager) SetCurrentPkgGroup

func (m *CCacheManager) SetCurrentPkgGroup(basePackageName string, arch string) (err error)

SetCurrentPkgGroup() is called once per package.

func (*CCacheManager) UploadMultiPkgGroupCCaches

func (m *CCacheManager) UploadMultiPkgGroupCCaches() (err error)

After building a package or more, the ccache folder is expected to look as follows:

<rootDir> (i.e. /ccache)

<m.LocalDownloadsDir>
<m.LocalUploadsDir>
<m.RootWorkDir>
  x86_64
    <groupName-1>
    <groupName-2>
  noarch
    <groupName-3>
    <groupName-4>

This function is typically called at the end of the build - after all packages have completed building.

At that point, there is not per package information about the group name or the architecture.

We use this directory structure to encode the per package group information at build time, so we can use them now.

func (*CCacheManager) UploadPkgGroupCCache

func (m *CCacheManager) UploadPkgGroupCCache() (err error)

type CCachePkgGroup

type CCachePkgGroup struct {
	Name      string
	Enabled   bool
	Size      int
	Arch      string
	CCacheDir string

	TarFile *CCacheArchive
	TagFile *CCacheArchive
}

CCachePkgGroup is calculated for each package as we encounter it during the build. It is derived from the CCacheGroupConfig + runtime parameters.

func (*CCachePkgGroup) UpdateTagsPaths

func (g *CCachePkgGroup) UpdateTagsPaths(remoteStoreConfig *RemoteStoreConfig, localDownloadsDir string, localUploadsDir string)

func (*CCachePkgGroup) UpdateTarPaths

func (g *CCachePkgGroup) UpdateTarPaths(remoteStoreConfig *RemoteStoreConfig, localDownloadsDir string, localUploadsDir string)

type RemoteStoreConfig

type RemoteStoreConfig struct {
	// The remote store type. Currently, there is only one type support;
	// Azure blob storage.
	Type string `json:"type"`

	// Azure subscription tenant id.
	TenantId string `json:"tenantId"`

	// Service principal client id with write-permissions to the Azure blob
	// storage. This can be left empty if upload is disabled.
	UserName string `json:"userName"`

	// Service principal secret with write-permissions to the Azure blob
	// storage. This can be left empty if upload is disabled.
	Password string `json:"password"`

	// Azure storage account name.
	StorageAccount string `json:"storageAccount"`

	// Azure storage container name.
	ContainerName string `json:"containerName"`

	// Tags folder is the location where the files holding information about
	// the latest folders are kept.
	TagsFolder string `json:"tagsFolder"`

	// If true, the build will download ccache artifacts from the remote store
	// (before the package family builds).
	DownloadEnabled bool `json:"downloadEnabled"`

	// If true, the build will determine the latest build and download its
	// artifacts. If true, DownloadFolder does not need to be set.
	DownloadLatest bool `json:"downloadLatest"`

	// The folder on the remote store where the ccache artifacts to use are.
	// There should be a folder for each build.
	// If DownloadLatest is true, this does not need to be set.
	DownloadFolder string `json:"downloadFolder"`

	// If true, the build will upload ccache artifacts to the remote store
	// after the package family builds).
	UploadEnabled bool `json:"uploadEnabled"`

	// The folder on the remote store where the ccache artifacts are to be
	// uploaded.
	UploadFolder string `json:"uploadFolder"`

	// If true, the tags specifying the latest artifacts will be updated to
	// point to the current upload.
	UpdateLatest bool `json:"updateLatest"`

	// If true, previous 'latest' ccache artifacts will be deleted from the
	// remote store.
	KeepLatestOnly bool `json:"keepLatestOnly"`
}

RemoveStoreConfig holds the following: - The remote store end-point and the necessary credentials. - The behavior of the download from the remote store. - The behavior of the upload to the remote store. - The clean-up policy of the remote store.