schedulerutils

package
v0.0.0-...-4cb4cf9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2024 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// The highest possible freshness value, used to force unbounded cascading rebuilds.
	NodeFreshnessAbsoluteMax uint64 = math.MaxUint64
)

Variables

This section is empty.

Functions

func BuildNodeWorker 

func BuildNodeWorker(channels *BuildChannels, agent buildagents.BuildAgent, graphMutex *sync.RWMutex, buildAttempts int, checkAttempts int, ignoredPackages, ignoredTests []*pkgjson.PackageVer)

BuildNodeWorker process all build requests, can be run concurrently with multiple instances.

func CanSubGraph 

func CanSubGraph(pkgGraph *pkggraph.PkgGraph, node *pkggraph.PkgNode, useCachedImplicit bool) bool

CanSubGraph returns true if a node can be subgraphed without any unresolved dynamic dependencies. Used to optimize graph solving.

func FindUnblockedNodesFromResult 

func FindUnblockedNodesFromResult(res *BuildResult, pkgGraph *pkggraph.PkgGraph, graphMutex *sync.RWMutex, buildState *GraphBuildState) (unblockedNodes []*pkggraph.PkgNode)

FindUnblockedNodesFromResult takes a package build result and returns a list of nodes that are now unblocked for building.

func InitializeGraphFromFile 

func InitializeGraphFromFile(inputFile string, packagesToBuild, testsToRun []*pkgjson.PackageVer, canUseCachedImplicit bool, extraLayers int) (isOptimized bool, pkgGraph *pkggraph.PkgGraph, goalNode *pkggraph.PkgNode, err error)

InitializeGraphFromFile initializes and prepares a graph dot file for building.

  • It will use an existing graph if provided (graph should not have been previously initialized, input file must be "")
  • It will load from disk if a path is provided (existing graph must be nil)
  • It will subgraph the graph to only contain the desired packages if possible.
  • If canUseCachedImplicit is true, it will use cached nodes to resolve implicit dependencies instead of waiting for them to be built in the graph (This can allow the graph to be optimized immediately instead of waiting for the implicit nodes to be resolved by an unknown package later in the build).

func InjectMissingImplicitProvides 

func InjectMissingImplicitProvides(res *BuildResult, pkgGraph *pkggraph.PkgGraph, useCachedImplicit bool) (didInjectAny bool, err error)

InjectMissingImplicitProvides will inject implicit provide nodes into the graph from a build result if they satisfy any unresolved nodes.

func IsReservedFile 

func IsReservedFile(rpmPath string, reservedRPMs []string) bool

IsReservedFile determines if a given file path or filename is found in a list of reserved RPMs. reservedRPMs may be a list of filenames or paths to reserved files. (e.g. 'foo-1.0.0-1.cm1.x86_64.rpm' or '/path/to/foo-1.0.0-1.cm1.x86_64.rpm').

func LeafNodes 

func LeafNodes(pkgGraph *pkggraph.PkgGraph, graphMutex *sync.RWMutex, goalNode *pkggraph.PkgNode, buildState *GraphBuildState, useCachedImplicit bool) (leafNodes []*pkggraph.PkgNode)

LeafNodes returns a slice of all leaf nodes in the graph.

func OptimizeGraph 

func OptimizeGraph(pkgGraph *pkggraph.PkgGraph, canUseCachedImplicit bool) (optimizedGraph *pkggraph.PkgGraph, goalNode *pkggraph.PkgNode, err error)

OptimizeGraph will attempt to create a solvable subgraph that satisfies the build goal node.

func ParseAndGeneratePackageBuildList 

func ParseAndGeneratePackageBuildList(dependencyGraph *pkggraph.PkgGraph, pkgsToBuild, pkgsToRebuild, pkgsToIgnore []string, imageConfig, baseDirPath string) (finalPackagesToBuild, packagesToRebuild, packagesToIgnore []*pkgjson.PackageVer, err error)

ParseAndGeneratePackageBuildList parses the common package request arguments and generates a list of packages to build based on the given dependency graph. - dependencyGraph: the dependency graph of all packages. Used to convert package/spec names to PackageVers. - pkgsToBuild: a list of package/spec names to build. If empty, all packages will be built. - pkgsToRebuild: a list of package/spec names to always rebuild. - pkgsToIgnore: a list of package/spec names to ignore. - imageConfig: the path to the image config file. Used to extract additional packages to build. - baseDirPath: the path to the base directory for the image. Used to resolve relative paths in the image config.

func ParseAndGeneratePackageTestList 

func ParseAndGeneratePackageTestList(dependencyGraph *pkggraph.PkgGraph, testsToRun, testsToRerun, testsToIgnore []string, imageConfig, baseDirPath string) (finalPackagesToBuild, packagesToRebuild, packagesToIgnore []*pkgjson.PackageVer, err error)

ParseAndGeneratePackageTestList parses the common package request arguments and generates a list of packages to test based on the given dependency graph. - dependencyGraph: the dependency graph of all packages. Used to convert package/spec names to PackageVers. - testsToRun: a list of package/spec names to test. If empty, all packages will be tested. - testsToRerun: a list of package/spec names to always test. - testsToIgnore: a list of package/spec names to ignore. - imageConfig: the path to the image config file. Used to extract additional packages to test. - baseDirPath: the path to the base directory for the image. Used to resolve relative paths in the image config.

func PrepareGraphForBuild 

func PrepareGraphForBuild(pkgGraph *pkggraph.PkgGraph, packagesToBuild, testsToRun []*pkgjson.PackageVer, canUseCachedImplicit bool, extraLayers int) (isOptimized bool, preparedGraph *pkggraph.PkgGraph, goalNode *pkggraph.PkgNode, err error)

PrepareGraphForBuild takes a graph and prepares it for package building.

  • It will subgraph the graph to only contain the desired packages if possible.
  • If canUseCachedImplicit is true, it will use cached nodes to resolve implicit dependencies instead of waiting for them to be built in the graph (This can allow the graph to be optimized immediately instead of waiting for the implicit nodes to be resolved by an unknown package later in the build).

func PrintBuildResult 

func PrintBuildResult(res *BuildResult)

PrintBuildResult prints a build result to the logger.

func PrintBuildSummary 

func PrintBuildSummary(pkgGraph *pkggraph.PkgGraph, graphMutex *sync.RWMutex, buildState *GraphBuildState, allowToolchainRebuilds bool)

PrintBuildSummary prints the summary of the entire build to the logger.

func ReadReservedFilesList 

func ReadReservedFilesList(path string) (reservedFiles []string, err error)

ReadReservedFilesList reads the list of reserved files (such as toolchain RPMs) from the manifest file passed in. Entries will be returned in the form '<rpm>-<version>-<release>.rpm' with any preceding path removed. If the file path is empty, an empty list will be returned.

func RecordBuildSummary 

func RecordBuildSummary(pkgGraph *pkggraph.PkgGraph, graphMutex *sync.RWMutex, buildState *GraphBuildState, outputPath string)

RecordBuildSummary stores the summary in to a csv.

Types

type BuildChannels 

type BuildChannels struct {
	Requests         <-chan *BuildRequest
	PriorityRequests <-chan *BuildRequest
	Results          chan<- *BuildResult
	Cancel           <-chan struct{}
	Done             <-chan struct{}
}

BuildChannels represents the communicate channels used by a build agent.

type BuildRequest 

type BuildRequest struct {
	Node           *pkggraph.PkgNode   // The main node being analyzed for the build.
	PkgGraph       *pkggraph.PkgGraph  // The graph of all packages.
	AncillaryNodes []*pkggraph.PkgNode // For SRPM builds: other nodes stemming from the same SRPM. Empty otherwise.
	ExpectedFiles  []string            // List of RPMs built by this node.
	UseCache       bool                // Can we use a cached copy of this package instead of building it.
	IsDelta        bool                // Is this a pre-downloaded RPM (not traditional cache) that we may be able to skip rebuilding.
	Freshness      uint                // The freshness of the node (used to determine if we can skip building future nodes).
}

BuildRequest represents a work-order to a build agent asking it to build a given node.

func ConvertNodesToRequests 

func ConvertNodesToRequests(pkgGraph *pkggraph.PkgGraph, graphMutex *sync.RWMutex, nodesToBuild []*pkggraph.PkgNode, packagesToRebuild, testsToRerun []*pkgjson.PackageVer, buildState *GraphBuildState, isCacheAllowed bool) (requests []*BuildRequest)

ConvertNodesToRequests converts a slice of nodes into a slice of build requests. - It will determine if the cache can be used for prebuilt nodes. - It will group similar build nodes together into AncillaryNodes.

Explanation of handling of the test nodes:

  1. The virtual B -> T edge guarantees the build node are unblocked and analyzed first.
  2. Once the build node is unblocked, analyze its partner test node in partnerTestNodesToRequest(). We remove the virtual edge and the test node either gets immediately queued or is blocked on some extra dependencies. Blocking is decided by canUseCacheForNode().
  3. If the test node ends up being blocked, it gets re-analyzed later once its dependencies are done. The test nodes unblocked this way end up inside the 'testNodes' list in ConvertNodesToRequests() and are queued for building in the testNodesToRequests() function. At this point the partner build nodes for these test nodes have either already finished building or are being built, thus the check for active and cached SRPMs inside testNodesToRequests().

type BuildResult 

type BuildResult struct {
	AncillaryNodes []*pkggraph.PkgNode // For SRPM builds: other nodes stemming from the same SRPM. Empty otherwise.
	BuiltFiles     []string            // List of RPMs built by this node.
	Err            error               // Error encountered during the build.
	LogFile        string              // Path to the log file from the build.
	Node           *pkggraph.PkgNode   // The main node being analyzed for the build.
	CheckFailed    bool                // Indicator if the package test failed but the build itself was correct.
	Ignored        bool                // Indicator if the build was ignored by user request.
	UsedCache      bool                // Indicator if we used the cached artifacts (external or earlier local build) instead of building the node.
	WasDelta       bool                // Indicator if we used a pre-built component from an external repository instead of building the node.
	Freshness      uint                // The freshness of the node (used to determine if we can skip building future nodes).
}

BuildResult represents the results of a build agent trying to build a given node.

type GraphBuildState 

type GraphBuildState struct {
	// contains filtered or unexported fields
}

GraphBuildState represents the build state of a graph.

func NewGraphBuildState 

func NewGraphBuildState(reservedFiles []string, maxFreshness uint) (g *GraphBuildState)

NewGraphBuildState returns a new GraphBuildState.

  • reservedFiles is a list of reserved files which should NOT be rebuilt. Any files that ARE rebuilt will be recorded.
  • maxFreshness is how fresh a newly rebuilt node is. Each dependant node will have a freshness of 'n-1', etc. until '0' where the subsequent nodes will no longer be rebuilt. 'maxFreshness < 0' will cause unbounded cascading rebuilds, while 'maxFreshness = 0' will cause no cascading rebuilds.

func (*GraphBuildState) ActiveBuildFromSRPM 

func (g *GraphBuildState) ActiveBuildFromSRPM(srpmFileName string) *BuildRequest

ActiveBuildFromSRPM returns a build request for the queried SRPM file or nil if the SRPM is not among the active builds.

func (*GraphBuildState) ActiveBuilds 

func (g *GraphBuildState) ActiveBuilds() map[int64]*BuildRequest

ActiveBuilds returns a map of Node IDs to BuildRequests that represents all outstanding builds.

func (*GraphBuildState) ActiveSRPMs 

func (g *GraphBuildState) ActiveSRPMs() (builtSRPMs []string)

ActiveSRPMs returns a list of all SRPMs, which are currently being built.

func (*GraphBuildState) ActiveTests 

func (g *GraphBuildState) ActiveTests() (testedSRPMs []string)

ActiveTests returns a list of all tests, which are currently being run.

func (*GraphBuildState) BuildFailures 

func (g *GraphBuildState) BuildFailures() []*BuildResult

BuildFailures returns a slice of all failed builds.

func (*GraphBuildState) ConflictingRPMs 

func (g *GraphBuildState) ConflictingRPMs() (rpms []string)

ConflictingRPMs will return a list of *.rpm files which should not have been rebuilt. This list is based on the manifest of pre-built toolchain rpms.

func (*GraphBuildState) ConflictingSRPMs 

func (g *GraphBuildState) ConflictingSRPMs() (srpms []string)

ConflictingSRPMs will return a list of *.src.rpm files which created rpms that should not have been rebuilt. This list is based on the manifest of pre-built toolchain rpms.

func (*GraphBuildState) DidNodeFail 

func (g *GraphBuildState) DidNodeFail(node *pkggraph.PkgNode) bool

DidNodeFail returns true if the requested node failed to be made available.

func (*GraphBuildState) GetFreshnessOfNode 

func (g *GraphBuildState) GetFreshnessOfNode(node *pkggraph.PkgNode) uint

GetFreshnessOfNode returns the freshness of a node.

func (*GraphBuildState) GetMaxFreshness 

func (g *GraphBuildState) GetMaxFreshness() uint

GetMaxFreshness returns the maximum freshness a node can have. (ie if a package is directly rebuilt due to user request, or missing files, it will have this freshness. Each dependant node will have a freshness of 'n-1', etc.

func (*GraphBuildState) IsNodeAvailable 

func (g *GraphBuildState) IsNodeAvailable(node *pkggraph.PkgNode) bool

IsNodeAvailable returns true if the requested node is available for other nodes to build with.

func (*GraphBuildState) IsNodeCached 

func (g *GraphBuildState) IsNodeCached(node *pkggraph.PkgNode) bool

IsNodeCached returns true if the requested node has been cached.

func (*GraphBuildState) IsNodeDelta 

func (g *GraphBuildState) IsNodeDelta(node *pkggraph.PkgNode) bool

IsNodeDelta returns true if the requested node was pre-downloaded as a delta package.

func (*GraphBuildState) IsNodeProcessed 

func (g *GraphBuildState) IsNodeProcessed(node *pkggraph.PkgNode) bool

IsNodeProcessed returns true if the requested node is has been processed already.

func (*GraphBuildState) IsSRPMCached 

func (g *GraphBuildState) IsSRPMCached(srpmFileName string) bool

IsSRPMCached returns true if the requested SRPM build used a cache.

func (*GraphBuildState) RecordBuildRequest 

func (g *GraphBuildState) RecordBuildRequest(req *BuildRequest)

RecordBuildRequest records a build request in the graph build state.

func (*GraphBuildState) RecordBuildResult 

func (g *GraphBuildState) RecordBuildResult(res *BuildResult, allowToolchainRebuilds bool) (err error)

RecordBuildResult records a build result in the graph build state. - It will record the result as a failure if applicable. - It will record all ancillary nodes of the result.

func (*GraphBuildState) RemoveBuildRequest 

func (g *GraphBuildState) RemoveBuildRequest(req *BuildRequest)

RemoveBuildRequest removes a build request from the graph build state.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.