sourcebundle

package
v0.15.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 16, 2024 License: MPL-2.0 Imports: 22 Imported by: 2

Documentation

Overview

Package sourcebundle deals with the construction of and later consumption of "source bundles", which are in some sense "meta-slugs" that capture a variety of different source packages together into a single working directory, which can optionally be bundled up into an archive for insertion into a blob storage system.

Whereas single slugs (as implemented in the parent package) have very little predefined structure aside from the possibility of a .terraformignore file, source bundles have a more prescriptive structure that allows callers to use a source bundle as a direct substitute for fetching the individual source packages it was built from.

NOTE WELL: Everything in this package is currently experimental and subject to breaking changes even in patch releases. We will make stronger commitments to backward-compatibility once we have more experience using this functionality in real contexts.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type BuildTracer 

type BuildTracer struct {
	// The RegistryPackageVersions... callbacks frame any requests to
	// fetch the list of available versions for a module registry package.
	RegistryPackageVersionsStart   func(ctx context.Context, pkgAddr regaddr.ModulePackage) context.Context
	RegistryPackageVersionsSuccess func(ctx context.Context, pkgAddr regaddr.ModulePackage, versions versions.List)
	RegistryPackageVersionsFailure func(ctx context.Context, pkgAddr regaddr.ModulePackage, err error)
	RegistryPackageVersionsAlready func(ctx context.Context, pkgAddr regaddr.ModulePackage, versions versions.List)

	// The RegistryPackageSource... callbacks frame any requests to fetch
	// the real underlying source address for a selected registry package
	// version.
	RegistryPackageSourceStart   func(ctx context.Context, pkgAddr regaddr.ModulePackage, version versions.Version) context.Context
	RegistryPackageSourceSuccess func(ctx context.Context, pkgAddr regaddr.ModulePackage, version versions.Version, sourceAddr sourceaddrs.RemoteSource)
	RegistryPackageSourceFailure func(ctx context.Context, pkgAddr regaddr.ModulePackage, version versions.Version, err error)
	RegistryPackageSourceAlready func(ctx context.Context, pkgAddr regaddr.ModulePackage, version versions.Version, sourceAddr sourceaddrs.RemoteSource)

	// The RemotePackageDownload... callbacks frame any requests to download
	// remote source packages.
	RemotePackageDownloadStart   func(ctx context.Context, pkgAddr sourceaddrs.RemotePackage) context.Context
	RemotePackageDownloadSuccess func(ctx context.Context, pkgAddr sourceaddrs.RemotePackage)
	RemotePackageDownloadFailure func(ctx context.Context, pkgAddr sourceaddrs.RemotePackage, err error)
	RemotePackageDownloadAlready func(ctx context.Context, pkgAddr sourceaddrs.RemotePackage)

	// Diagnostics will be called for any diagnostics that describe problems
	// that aren't also reported by calling one of the "Failure" callbacks
	// above. A recipient that is going to report the errors itself using
	// the Failure callbacks anyway should consume diagnostics from this
	// event, rather than from the return values of the [Builder] methods,
	// to avoid redundantly reporting the same errors twice.
	//
	// Diagnostics might be called multiple times during an operation. Callers
	// should consider each new call to represent additional diagnostics,
	// not replacing any previously returned.
	Diagnostics func(ctx context.Context, diags Diagnostics)
}

BuildTracer contains a set of callbacks that a caller can optionally provide to Builder methods via their context.Context arguments to be notified when various long-running events are starting and stopping, to allow both for debugging and for UI feedback about progress.

Any or all of the callbacks may be left as nil, in which case no event will be delivered for the corresponding event.

The context.Context passed to each trace function is guaranteed to be a child of the one passed to whatever Builder method caused the event to occur, and so it can carry cross-cutting information such as distributed tracing clients.

The "Start"-suffixed methods all allow returning a new context which will then be passed to the corresponding "Success"-suffixed or "Failure"-suffixed function, and also used for outgoing requests within the scope of that operation. This allows carrying values such as tracing spans between the start and end, so they can properly bracket the operation in question. If your tracer doesn't need this then just return the given context.

func (*BuildTracer) OnContext 

func (bt *BuildTracer) OnContext(ctx context.Context) context.Context

OnContext takes a context and returns a derived context which has everything the given context already had plus also the receiving BuildTrace object, so that passing the resulting context to methods of Builder will cause the trace object's callbacks to be called.

Each context can have only one tracer, so if the given context already has a tracer then it will be overridden by the new one.

type Builder 

type Builder struct {
	// contains filtered or unexported fields
}

Builder deals with the process of gathering source code

func NewBuilder 

func NewBuilder(targetDir string, fetcher PackageFetcher, registryClient RegistryClient) (*Builder, error)

NewBuilder creates a new builder that will construct a source bundle in the given target directory, which must already exist and be empty before any work begins.

During the lifetime of a builder the target directory must not be modified or moved by anything other than the builder, including other concurrent processes running on the system. The target directory is not a valid source bundle until a call to Builder.Close returns successfully; the directory may be apepar in an inconsistent state while the builder is working.

func (*Builder) AddFinalRegistrySource 

func (b *Builder) AddFinalRegistrySource(ctx context.Context, addr sourceaddrs.RegistrySourceFinal, depFinder DependencyFinder) Diagnostics

AddFinalRegistrySource is a variant of Builder.AddRegistrySource which takes an already-selected version of a registry source, instead of taking a version constraint and then selecting the latest available version matching that constraint.

This function still asks the registry for its set of available versions for the unversioned package first, to ensure that the results from installing from a final source will always be consistent with those from installing from a not-yet-resolved registry source.

func (*Builder) AddRegistrySource 

func (b *Builder) AddRegistrySource(ctx context.Context, addr sourceaddrs.RegistrySource, allowedVersions versions.Set, depFinder DependencyFinder) Diagnostics

AddRegistrySource incorporates the registry metadata for the given address and the package associated with the latest version in allowedVersions into the bundle, and then analyzes the new artifact for dependencies using the given dependency finder.

If you have already selected a specific version to install, consider using Builder.AddFinalRegistrySource instead.

If the returned diagnostics contains errors then the bundle is left in an inconsistent state and must not be used for any other calls.

func (*Builder) AddRemoteSource 

func (b *Builder) AddRemoteSource(ctx context.Context, addr sourceaddrs.RemoteSource, depFinder DependencyFinder) Diagnostics

AddRemoteSource incorporates the package containing the given remote source into the bundle, and then analyzes the source artifact for dependencies using the given dependency finder.

If the returned diagnostics contains errors then the bundle is left in an inconsistent state and must not be used for any other calls.

func (*Builder) Close 

func (b *Builder) Close() (*Bundle, error)

Close ensures that the target directory is in a valid and consistent state to be used as a source bundle and then returns an object providing the read-only API for that bundle.

After calling Close the receiving builder becomes invalid and must not be used any further.

type Bundle 

type Bundle struct {
	// contains filtered or unexported fields
}

func ExtractArchive 

func ExtractArchive(r io.Reader, targetDir string) (*Bundle, error)

ExtractArchive reads a source bundle archive from the given reader and extracts it into the given target directory, which must already exist and must be empty.

If successful, it returns a Bundle value representing the created bundle, as if the given target directory were passed to OpenDir.

func OpenDir 

func OpenDir(baseDir string) (*Bundle, error)

OpenDir opens a bundle rooted at the given base directory.

If OpenDir succeeds then nothing else (inside or outside the calling program) may modify anything under the given base directory for the lifetime of the returned Bundle object. If the bundle directory is modified while the object is still alive then behavior is undefined.

func (*Bundle) ChecksumV1 

func (b *Bundle) ChecksumV1() (string, error)

ChecksumV1 returns a checksum of the contents of the source bundle that can be used to determine if another source bundle is equivalent to this one.

"Equivalent" means that it contains all of the same source packages with identical content each.

A successful result is a string with the prefix "h1:" to indicate that it was built with checksum algorithm version 1. Future versions may introduce other checksum formats.

func (*Bundle) LocalPathForFinalRegistrySource 

func (b *Bundle) LocalPathForFinalRegistrySource(addr sourceaddrs.RegistrySourceFinal) (string, error)

LocalPathForFinalRegistrySource is a variant of Bundle.LocalPathForRegistrySource which passes the source address and selected version together as a single address value.

func (*Bundle) LocalPathForRegistrySource 

func (b *Bundle) LocalPathForRegistrySource(addr sourceaddrs.RegistrySource, version versions.Version) (string, error)

LocalPathForRegistrySource returns the local path within the bundle that corresponds with the given registry address and version, or an error if the source address is within a source package not included in the bundle.

func (*Bundle) LocalPathForRemoteSource 

func (b *Bundle) LocalPathForRemoteSource(addr sourceaddrs.RemoteSource) (string, error)

LocalPathForRemoteSource returns the local path within the bundle that corresponds with the given source address, or an error if the source address is within a source package not included in the bundle.

func (*Bundle) LocalPathForSource 

func (b *Bundle) LocalPathForSource(addr sourceaddrs.FinalSource) (string, error)

LocalPathForSource takes either a remote or registry final source address and returns the local path within the bundle that corresponds with it.

It doesn't make sense to pass a sourceaddrs.LocalSource to this function because a source bundle cannot contain anything other than remote packages, but as a concession to convenience this function will return a filepath-shaped relative path in that case, assuming that the source was intended to be a local filesystem path relative to the current working directory. The result will therefore not necessarily be a subdirectory of the recieving bundle in that case.

func (*Bundle) RegistryPackageSourceAddr 

func (b *Bundle) RegistryPackageSourceAddr(pkgAddr regaddr.ModulePackage, version versions.Version) (sourceaddrs.RemoteSource, bool)

RegistryPackageSourceAddr returns the remote source address corresponding to the given version of the given module package, or sets its second return value to false if no such version is included in the bundle.

func (*Bundle) RegistryPackageVersions 

func (b *Bundle) RegistryPackageVersions(pkgAddr regaddr.ModulePackage) versions.List

RegistryPackageVersions returns a list of all of the versions of the given module registry package that this bundle has package content for.

This result can be used as a substitute for asking the remote registry which versions are available in any situation where a caller is interested only in what's bundled, and will not consider installing anything new from the origin registry.

The result is guaranteed to be sorted with lower-precedence version numbers placed earlier in the list.

func (*Bundle) RegistryPackages 

func (b *Bundle) RegistryPackages() []regaddr.ModulePackage

RegistryPackages returns a list of all of the distinct registry packages that contributed to this bundle.

The result is in a consistent but unspecified sorted order.

func (*Bundle) RemotePackageMeta 

func (b *Bundle) RemotePackageMeta(pkgAddr sourceaddrs.RemotePackage) *PackageMeta

RemotePackageMeta returns the package metadata for the given package address, or nil if there is no metadata for that package tracked in the bundle.

func (*Bundle) RemotePackages 

func (b *Bundle) RemotePackages() []sourceaddrs.RemotePackage

RemotePackages returns a slice of all of the remote source packages that contributed to this source bundle.

The result is sorted into a consistent but unspecified order.

func (*Bundle) SourceForLocalPath 

func (b *Bundle) SourceForLocalPath(p string) (sourceaddrs.FinalSource, error)

SourceForLocalPath is the inverse of Bundle.LocalPathForSource, translating a local path beneath the bundle's base directory back into a source address that it's a snapshot of.

Returns an error if the given directory is not within the bundle's base directory, or is not within one of the subdirectories of the bundle that represents a source package. A caller using this to present more user-friendly file paths in error messages etc could reasonably choose to just retain the source string if this function returns an error, and not show the error to the user.

The Bundle implementation is optimized for forward lookups from source address to local path rather than the other way around, so this function may be considerably more expensive than the forward lookup and is intended primarily for reporting friendly source locations in diagnostic messages instead of exposing the opaque internal directory names from the source bundle. This function should not typically be used in performance-sensitive portions of the happy path.

func (*Bundle) WriteArchive 

func (b *Bundle) WriteArchive(w io.Writer) error

WriteArchive writes a source bundle archive containing the same contents as the bundle to the given writer.

A source bundle archive is a gzip-compressed tar stream that can then be extracted in some other location to produce an equivalent source bundle directory.

type Dependencies 

type Dependencies struct {
	// contains filtered or unexported fields
}

Dependencies is part of the callback API for DependencyFinder. Dependency finders use the methods of this type to report the dependencies they find in the source artifact being analyzed.

func (*Dependencies) AddLocalSource 

func (d *Dependencies) AddLocalSource(source sourceaddrs.LocalSource, depFinder DependencyFinder)

func (*Dependencies) AddRegistrySource 

func (d *Dependencies) AddRegistrySource(source sourceaddrs.RegistrySource, allowedVersions versions.Set, depFinder DependencyFinder)

func (*Dependencies) AddRemoteSource 

func (d *Dependencies) AddRemoteSource(source sourceaddrs.RemoteSource, depFinder DependencyFinder)

type DependencyFinder 

type DependencyFinder interface {
	// FindDependencies should analyze the file or directory at the given
	// sub-path of the given filesystem and then call the given callback
	// once for each detected dependency, providing both its source
	// address and the appropriate [DependencyFinder] for whatever kind
	// of source artifact is expected at that source address.
	//
	// The same source address can potentially contain artifacts of multiple
	// different types. The calling [Builder] will visit each distinct
	// (source, finder) pair only once for analysis, and will also aim to
	// avoid redundantly re-fetching the same source package more than once.
	//
	// If an implementer sends a local source address to the callback function,
	// the calling [Builder] will automatically resolve that relative to
	// the source address being analyzed. Implementers should typically first
	// validate that the local address does not traverse up (with "..") more
	// levels than are included in subPath, because implementers can return
	// higher-quality error diagnostics (with source location information)
	// than the calling Builder can.
	//
	// If the implementer emits diagnostics with source location information
	// then the filenames in the source ranges must be strings that would
	// pass [fs.ValidPath] describing a path from the root of the given fs
	// to the file containing the error. The builder will then translate those
	// paths into remote source address strings within the containing package.
	FindDependencies(fsys fs.FS, subPath string, deps *Dependencies) Diagnostics
}

A DependencyFinder analyzes a file or directory inside a source package and reports any dependencies described in that location.

The same location could potentially be analyzed by multiple different DependencyFinder implementations if e.g. it's a directory containing a mixture of different kinds of artifact where each artifact has a disjoint set of relevant files.

All DependencyFinder implementations must be comparable in the sense of supporting the == operator without panicking, and should typically be singletons, because Builder will use values of this type as part of the unique key for tracking whether a particular dependency has already been analyzed. A typical DependencyFinder implementation is an empty struct type with the FindDependency method implemented on it.

type DiagDescription 

type DiagDescription struct {
	Summary string
	Detail  string
}

type DiagSeverity 

type DiagSeverity rune
const (
	DiagError   DiagSeverity = 'E'
	DiagWarning DiagSeverity = 'W'
)

type DiagSource 

type DiagSource struct {
	Subject *SourceRange
	Context *SourceRange
}

type Diagnostic 

type Diagnostic interface {
	Severity() DiagSeverity
	Description() DiagDescription
	Source() DiagSource

	// ExtraInfo returns the raw extra information value. This is a low-level
	// API which requires some work on the part of the caller to properly
	// access associated information. This convention comes from HCL and
	// Terraform and this is here primarily for their benefit; sourcebundle
	// passes through these values verbatim without trying to interpret them.
	ExtraInfo() interface{}
}

Diagnostics represents a single problem (error or warning) that has occurred during an operation.

This interface has no concrete implementations in this package. Implementors of DependencyFinder will need to implement this interface to report any problems they find while analyzing the designated source artifact. For example, a DependencyFinder that uses the HCL library to analyze an HCL-based language would probably implement this interface in terms of HCL's Diagnostic type.

type Diagnostics 

type Diagnostics []Diagnostic

Diagnostics is a collection of problems (errors and warnings) that occurred during an operation.

func (Diagnostics) Append 

func (diags Diagnostics) Append(more ...interface{}) Diagnostics

func (Diagnostics) HasErrors 

func (diags Diagnostics) HasErrors() bool

type FetchSourcePackageResponse added in v0.14.0 

type FetchSourcePackageResponse struct {
	PackageMeta *PackageMeta
}

FetchSourcePackageResponse is a structure which represents metadata about the fetch operation. This type may grow to add more data over time in later minor releases.

type ModulePackageSourceAddrResponse added in v0.14.0 

type ModulePackageSourceAddrResponse struct {
	SourceAddr sourceaddrs.RemoteSource
}

ModulePackageSourceAddrResponse is an opaque type which represents the result of the source address client operation. This type may grow to add more functionality over time in later minor releases.

type ModulePackageVersionsResponse added in v0.14.0 

type ModulePackageVersionsResponse struct {
	Versions versions.List
}

ModulePackageVersionsResponse is an opaque type which represents the result of the package versions client operation. This type may grow to add more functionality over time in later minor releases.

type PackageFetcher 

type PackageFetcher interface {
	// FetchSourcePackage retrieves the a source package from the given
	// location and extracts it into the given local filesystem directory.
	//
	// A package fetcher is responsible for ensuring that nothing gets written
	// outside of the given target directory. However, a fetcher can assume that
	// nothing should be modifying or moving targetDir and or any of its contents
	// concurrently with the fetcher running.
	//
	// If the function returns with a nil error then the target directory must be
	// a complete copy of the designated remote package, ready for further analysis.
	//
	// Package fetchers should respond to cancellation of the given
	// [context.Context] to a reasonable extent, so that the source bundle build
	// process can be interrupted relatively promptly. Return a non-nil error when
	// cancelled to allow the caller to detect that the target directory might not
	// be in a consistent state.
	//
	// PackageFetchers should not have any persistent mutable state: each call
	// should be independent of all past, concurrent, and future calls. In
	// particular, a fetcher should not attempt to implement any caching behavior,
	// because it's [Builder]'s responsibility to handle caching and request
	// coalescing during bundle construction to ensure that it will happen
	// consistently across different fetcher implementations.
	FetchSourcePackage(ctx context.Context, sourceType string, url *url.URL, targetDir string) (FetchSourcePackageResponse, error)
}

A PackageFetcher knows how to fetch remote source packages into a local filesystem directory.

type PackageMeta 

type PackageMeta struct {
	// contains filtered or unexported fields
}

PackageMeta is a collection of metadata about how the content of a particular remote package was derived.

A nil value of this type represents no metadata. A non-nil value will typically omit some or all of the fields if they are not relevant.

func PackageMetaWithGitMetadata added in v0.15.0 

func PackageMetaWithGitMetadata(
	commitID string,
	commitMessage string,
) *PackageMeta

PackageMetaWithGitMetadata returns a PackageMeta object with a Git Commit ID and message tracked.

The given commit ID must be a fully-qualified ID, and never an abbreviated commit ID, the name of a ref, or anything other proxy-for-commit identifier.

func (*PackageMeta) GitCommitID 

func (m *PackageMeta) GitCommitID() string

If the content of this package was derived from a particular commit from a Git repository, GitCommitID returns the fully-qualified ID of that commit. This is never an abbreviated commit ID, the name of a ref, or anything else that could serve as a proxy for a commit ID.

If there is no relevant commit ID for this package, returns an empty string.

func (*PackageMeta) GitCommitMessage added in v0.15.0 

func (m *PackageMeta) GitCommitMessage() string

GitCommitMessage returns a commit message for the commit this package was derived from.

type RegistryClient 

type RegistryClient interface {
	// ModulePackageVersions fetches all of the known exact versions
	// available for the given package in its module registry.
	ModulePackageVersions(ctx context.Context, pkgAddr regaddr.ModulePackage) (ModulePackageVersionsResponse, error)

	// ModulePackageSourceAddr fetches the real remote source address for the
	// given version of the given module registry package.
	ModulePackageSourceAddr(ctx context.Context, pkgAddr regaddr.ModulePackage, version versions.Version) (ModulePackageSourceAddrResponse, error)
}

RegistryClient provides a minimal client for the Terraform module registry protocol, sufficient to find the available versions for a particular registry entry and then to find the real remote package for a particular version.

An implementation should not itself attempt to cache the direct results of the client methods, but it can (and probably should) cache prerequisite information such as the results of performing service discovery against the hostname in a module package address.

type SourcePos 

type SourcePos struct {
	Line, Column, Byte int
}

type SourceRange 

type SourceRange struct {
	// Filename is a human-oriented label for the file that the range belongs
	// to. This is often the string representation of a source address, but
	// isn't guaranteed to be.
	Filename   string
	Start, End SourcePos
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.