extdata

Documentation

Overview

Package extdata implements the external data file mechanism.

Index

• func RunDownloads(ctx context.Context, dataDir string, jobs []*DownloadJob, cl devserver.Client)
• type DownloadJob
• type LinkData
• type LinkType
• type Manager
  • func NewManager(ctx context.Context, dataDir, artifactsURL string) (*Manager, error)
  • func (m *Manager) PrepareDownloads(ctx context.Context, entities []*protocol.Entity) (jobs []*DownloadJob, release func())
  • func (m *Manager) Purgeable() []string

Functions

func RunDownloads

func RunDownloads(ctx context.Context, dataDir string, jobs []*DownloadJob, cl devserver.Client)

RunDownloads downloads required external data files in parallel.

dataDir is the path to the base directory containing external data link files (typically "/usr/local/share/tast/data" on DUT). jobs are typically obtained by calling PrepareDownloads.

This function does not return errors; instead it tries to download files as far as possible and logs encountered errors with ctx so that a single download error does not cause all tests to fail.

Types

type DownloadJob

type DownloadJob struct {
	// contains filtered or unexported fields
}

DownloadJob represents a job to download an external data file and make hard links at several file paths.

type LinkData

type LinkData struct {
	// Type declares the type of the external data link.
	Type LinkType `json:"type"`

	// StaticURL is the URL of the static external data file on Google Cloud Storage.
	// This field is valid for static external data links only.
	StaticURL string `json:"url"`

	// Size is the size of the external data file in bytes.
	// This field is valid for static external data links only.
	Size int64 `json:"size"`

	// Size is SHA256 hash of the external data file.
	// This field is valid for static external data links only.
	SHA256Sum string `json:"sha256sum"`

	// Name is the file name of a build artifact.
	// This field is valid for build artifact external data links only.
	Name string `json:"name"`

	// Executable specifies whether the external data file is executable.
	// If this is true, executable permission is given to the downloaded file.
	Executable bool `json:"executable"`
}

LinkData defines the schema of external data link files.

type LinkType

type LinkType string

LinkType represents a type of an external data link.

const (
	// TypeStatic is for a link to a file on web with fixed URL and content.
	TypeStatic LinkType = ""

	// TypeArtifact is for a link to a file in ChromeOS build artifacts
	// corresponding to the DUT image version.
	TypeArtifact LinkType = "artifact"
)

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages operations for external data files.

func NewManager

func NewManager(ctx context.Context, dataDir, artifactsURL string) (*Manager, error)

NewManager creates a new Manager.

dataDir is the path to the base directory containing external data link files (typically "/usr/local/share/tast/data" on DUT). artifactURL is the URL of Google Cloud Storage directory, ending with a slash, containing build artifacts for the current ChromeOS image.

func (*Manager) PrepareDownloads

func (m *Manager) PrepareDownloads(ctx context.Context, entities []*protocol.Entity) (jobs []*DownloadJob, release func())

PrepareDownloads computes a list of external data files that need to be downloaded for entities.

PrepareDownloads also removes stale files so they are never used even if we fail to download them later. When it encounters errors, *.external-error files are saved so that they can be read and reported by bundles later.

PrepareDownloads returns a list of download job specifications that can be passed to RunDownloads to perform actual downloads.

release must be called after entities finish.

func (*Manager) Purgeable

func (m *Manager) Purgeable() []string

Purgeable returns a list of external data file paths not needed by the currently running entities. They can be deleted if the disk space is low.