pipeline

package
v1.4.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 19, 2026 License: Apache-2.0 Imports: 30 Imported by: 0

Documentation

Overview

Package pipeline provides the core visualization pipeline for Stacktower.

This package implements the complete parse → layout → render pipeline that can be used by CLI, API, and worker components. By centralizing this logic, we ensure consistent behavior across all entry points and avoid code duplication.

Architecture

The pipeline consists of three stages:

  1. Parse: Resolve dependencies from package registries or manifest files
  2. Layout: Compute visual positions for the dependency graph
  3. Render: Generate output in various formats (SVG, PNG, PDF, JSON)

Each stage can be run independently or as part of the complete pipeline.

Usage

Create a Runner and execute the pipeline: 

runner := pipeline.NewRunner(cache, nil, logger)
opts := pipeline.Options{
    Language: "python",
    Package:  "requests",
    VizType:  "tower",
    Formats:  []string{"svg"},
}
result, err := runner.Execute(ctx, opts)
if err != nil {
    log.Fatal(err)
}
svg := result.Artifacts["svg"]

Run individual stages: 

// Parse only
g, err := runner.Parse(ctx, parseOpts)

// Layout with existing graph
layout, err := runner.ComputeLayout(ctx, g, layoutOpts)

// Render with existing layout
artifacts, err := runner.Render(ctx, layout, g, renderOpts)

Index

Constants

View Source 
const (
	// DefaultMaxDepth is the maximum dependency traversal depth for the pipeline.
	// This is intentionally more conservative than deps.DefaultMaxDepth (50) to
	// provide better UX for CLI users and prevent excessively large graphs.
	// API users can override this by setting MaxDepth explicitly.
	DefaultMaxDepth = 10

	// DefaultMaxNodes is the maximum number of nodes to fetch.
	// This matches deps.DefaultMaxNodes (5000) to maintain consistency.
	DefaultMaxNodes = 5000

	// DefaultWidth is the default frame width in pixels.
	DefaultWidth = 800.0

	// DefaultHeight is the default frame height in pixels.
	DefaultHeight = 600.0

	// DefaultSeed is the default random seed for reproducibility.
	DefaultSeed = uint64(42)

	// DefaultOrdering is the default ordering algorithm.
	DefaultOrdering = "optimal"
)
View Source 
const (
	FormatSVG  = "svg"
	FormatPNG  = "png"
	FormatPDF  = "pdf"
	FormatJSON = "json"
)

Format constants for output formats.

View Source 
const (
	// PresetCLI applies defaults optimized for interactive CLI usage.
	// Enables: Randomize, Merge, Normalize, Popups, ShowVulns, ShowLicenses.
	PresetCLI = "cli"

	// PresetAPI applies defaults optimized for API/programmatic usage.
	// Uses minimal defaults suitable for embedding in web applications.
	PresetAPI = "api"

	// PresetWorker applies defaults optimized for background worker processing.
	// Similar to API but may include additional options for batch processing.
	PresetWorker = "worker"
)

Preset names for ApplyPreset.

View Source 
const DefaultStyle = graph.StyleHanddrawn

DefaultStyle is the default visual style.

View Source 
const DefaultVizType = graph.VizTypeTower

DefaultVizType is the default visualization type.

Variables

View Source 
var ValidFormats = map[string]bool{
	FormatSVG:  true,
	FormatPNG:  true,
	FormatPDF:  true,
	FormatJSON: true,
}

ValidFormats is the set of supported output formats.

View Source 
var ValidStyles = map[string]bool{
	graph.StyleSimple:    true,
	graph.StyleHanddrawn: true,
}

ValidStyles is the set of supported visual styles.

View Source 
var ValidVizTypes = map[string]bool{
	graph.VizTypeTower:    true,
	graph.VizTypeNodelink: true,
}

ValidVizTypes is the set of supported visualization types.

Functions

func GenerateLayout 

func GenerateLayout(g *dag.DAG, opts Options) (graph.Layout, error)

GenerateLayout generates a complete layout for any visualization type. This is the unified entry point for generating serializable layout data.

Both tower and nodelink layouts include:

  • Graph structure (nodes, edges, rows)
  • Nebraska rankings (maintainer data)
  • Visualization-specific data (blocks for tower, DOT for nodelink)

func Render 

func Render(l layout.Layout, g *dag.DAG, opts Options) (map[string][]byte, error)

Render generates output artifacts in the requested formats.

func RenderFromLayout 

func RenderFromLayout(graphLayout graph.Layout, g *dag.DAG, opts Options) (map[string][]byte, error)

RenderFromLayout renders output from a graph.Layout. This is the preferred entry point when you have a graph.Layout.

func RenderFromLayoutData 

func RenderFromLayoutData(layoutData []byte, g *dag.DAG, opts Options) (map[string][]byte, error)

RenderFromLayoutData renders output from serialized layout data. This is useful when the layout was computed elsewhere (e.g., cached). 

func RenderNodelink(layout graph.Layout, opts Options) (map[string][]byte, error)

RenderNodelink generates nodelink outputs from a layout. The layout must be a nodelink layout (VizType = "nodelink") with a DOT string.

func ValidateFormat 

func ValidateFormat(format string) error

ValidateFormat checks that a format is valid.

func ValidateFormats 

func ValidateFormats(formats []string) error

ValidateFormats checks that all formats are valid.

func ValidateListOptions 

func ValidateListOptions(opts ListOptions) error

ValidateListOptions validates options for the List operation.

func ValidateStyle 

func ValidateStyle(style string) error

ValidateStyle checks that a style is valid.

func ValidateVizType 

func ValidateVizType(vizType string) error

ValidateVizType checks that a visualization type is valid.

Types

type CacheInfo 

type CacheInfo struct {
	ParseHit  bool // Whether parse result came from cache
	LayoutHit bool // Whether layout result came from cache
	RenderHit bool // Whether all artifacts came from cache
}

CacheInfo tracks cache hits for each pipeline stage.

type ListOptions 

type ListOptions struct {
	// Language is the target ecosystem (e.g., "python", "rust").
	Language string

	// Package is the package name to list versions for.
	Package string

	// RuntimeVersion filters versions compatible with this runtime (e.g., "3.8" for Python).
	// Empty string means no filtering.
	RuntimeVersion string

	// IncludeConstraints includes runtime constraints in the result.
	// This may require additional API calls for some registries.
	IncludeConstraints bool

	// Refresh bypasses cache for fresh data.
	Refresh bool
}

ListOptions configures version listing behavior.

type ListResult 

type ListResult struct {
	// Package is the normalized package name.
	Package string

	// Language is the ecosystem name.
	Language string

	// Versions is the list of available versions, sorted newest first.
	Versions []string

	// RuntimeConstraints maps version -> runtime constraint (e.g., ">=3.8").
	// Only populated when ListOptions.IncludeConstraints is true.
	RuntimeConstraints map[string]string

	// LatestStable is the highest non-prerelease version.
	LatestStable string
}

ListResult contains the result of a version listing operation.

func ListVersions 

func ListVersions(ctx context.Context, c cache.Cache, opts ListOptions) (*ListResult, error)

ListVersions returns available versions for a package. This is a stateless function that can be called without a Runner.

type Options 

type Options struct {
	// Parse options
	Language          string `json:"language"`
	Package           string `json:"package,omitempty"`
	Version           string `json:"version,omitempty"` // Specific package version (e.g., "2.31.0")
	Manifest          string `json:"manifest,omitempty"`
	ManifestFilename  string `json:"manifest_filename,omitempty"`
	ManifestPath      string `json:"manifest_path,omitempty"` // Optional on-disk path used when parser needs workspace context
	Owner             string `json:"owner,omitempty"`         // GitHub owner (user/org)
	Repo              string `json:"repo,omitempty"`          // GitHub repository name
	Ref               string `json:"ref,omitempty"`           // Git ref (branch/tag)
	Path              string `json:"path,omitempty"`          // Path within repo
	RootName          string `json:"root_name,omitempty"`     // Custom name for root node (replaces __project__)
	MaxDepth          int    `json:"max_depth,omitempty"`
	MaxNodes          int    `json:"max_nodes,omitempty"`
	Workers           int    `json:"workers,omitempty"`            // Concurrent fetch workers (0 = default 20)
	SkipEnrich        bool   `json:"skip_enrich,omitempty"`        // Skip metadata enrichment (default: false = enrich)
	FetchContributors bool   `json:"fetch_contributors,omitempty"` // Fetch GitHub contributors (slower, enables Nebraska rankings)
	Refresh           bool   `json:"refresh,omitempty"`
	DependencyScope   string `json:"dependency_scope,omitempty"`   // Dependency scope policy: prod_only (default) or all
	IncludePrerelease bool   `json:"include_prerelease,omitempty"` // Include prerelease versions (alpha/beta/rc/dev/etc.)
	RuntimeVersion    string `json:"runtime_version,omitempty"`    // Target runtime version for marker evaluation (e.g., "3.11" for Python)

	// Layout options
	VizType   string  `json:"viz_type,omitempty"`
	Width     float64 `json:"width,omitempty"`
	Height    float64 `json:"height,omitempty"`
	Normalize bool    `json:"normalize,omitempty"` // Apply graph normalization during layout
	Ordering  string  `json:"ordering,omitempty"`
	Merge     bool    `json:"merge,omitempty"`
	Randomize bool    `json:"randomize,omitempty"`
	Seed      uint64  `json:"seed,omitempty"`

	// Render options
	Formats    []string `json:"formats,omitempty"`
	Style      string   `json:"style,omitempty"`
	ShowEdges  bool     `json:"show_edges,omitempty"`
	Nebraska   bool     `json:"nebraska,omitempty"` // Show Nebraska ranking panel in SVG (data is always computed)
	Popups     bool     `json:"popups,omitempty"`
	FlagsOnTop bool     `json:"flags_on_top,omitempty"` // Render security flags (license/vuln) on top of all blocks

	// Security options
	SecurityScan bool `json:"security_scan,omitempty"` // Run vulnerability scan during parse
	ShowVulns    bool `json:"show_vulns,omitempty"`    // Include vulnerability data in rendered output
	ShowLicenses bool `json:"show_licenses,omitempty"` // Analyze and show license compliance data in rendered output

	// Runtime options (not serialized)
	Logger      *log.Logger      `json:"-"`
	GitHubToken string           `json:"-"`
	Orderer     ordering.Orderer `json:"-"`
	// contains filtered or unexported fields
}

Options contains all configuration for the visualization pipeline. This struct supports JSON serialization for API requests.

func (*Options) ApplyPreset 

func (o *Options) ApplyPreset(preset string)

ApplyPreset applies consumer-specific defaults on top of base pipeline defaults. This should be called after setting Language/Package but before validation.

Presets:

  • "cli": Interactive CLI with rich visualizations (randomize, merge, popups, vulns, licenses)
  • "api": Minimal defaults for programmatic/web usage
  • "worker": Background processing defaults (similar to API)

Unknown presets are silently ignored (base defaults apply).

func (*Options) ArtifactKeyOpts 

func (o *Options) ArtifactKeyOpts(format string) cache.ArtifactKeyOpts

ArtifactKeyOpts returns cache key options for artifact rendering. 

func (o *Options) IsNodelink() bool

IsNodelink returns true if this is a nodelink visualization.

func (*Options) IsTower 

func (o *Options) IsTower() bool

IsTower returns true if this is a tower visualization.

func (*Options) LayoutKeyOpts 

func (o *Options) LayoutKeyOpts() cache.LayoutKeyOpts

LayoutKeyOpts returns cache key options for layout computation.

func (*Options) NeedsOptimalOrderer 

func (o *Options) NeedsOptimalOrderer() bool

NeedsOptimalOrderer returns true if the ordering algorithm requires the optimal orderer. This is true when ordering is "optimal" (the default) or empty.

func (*Options) SetLayoutDefaults 

func (o *Options) SetLayoutDefaults()

SetLayoutDefaults sets default values for layout computation.

func (*Options) SetRenderDefaults 

func (o *Options) SetRenderDefaults()

SetRenderDefaults sets default values for rendering.

func (*Options) ShouldEnrich 

func (o *Options) ShouldEnrich() bool

ShouldEnrich returns whether metadata enrichment should be performed.

func (*Options) ValidateAndSetDefaults 

func (o *Options) ValidateAndSetDefaults() error

ValidateAndSetDefaults checks required fields and applies defaults for the full pipeline. This method is idempotent - calling it multiple times has the same effect as calling it once.

func (*Options) ValidateForLayout 

func (o *Options) ValidateForLayout() error

ValidateForLayout validates and sets defaults for layout computation.

func (*Options) ValidateForParse 

func (o *Options) ValidateForParse() error

ValidateForParse checks required fields for parsing.

func (*Options) ValidateForRender 

func (o *Options) ValidateForRender() error

ValidateForRender validates and sets defaults for rendering.

type OrdererWithHooks 

type OrdererWithHooks struct {
	Timeout time.Duration
	// contains filtered or unexported fields
}

OrdererWithHooks wraps ordering.OptimalSearch with hooks-based progress reporting.

func (*OrdererWithHooks) OrderRows 

func (o *OrdererWithHooks) OrderRows(g *dag.DAG) map[int][]string

OrderRows implements ordering.Orderer.

type ParseResult 

type ParseResult struct {
	Graph          *dag.DAG
	RuntimeVersion string // Target runtime version used (e.g., "3.11")
	RuntimeSource  string // Where runtime came from: "cli", "manifest", "default"
}

ParseResult contains the parsed dependency graph and metadata.

func Parse 

func Parse(ctx context.Context, c cache.Cache, opts Options) (*ParseResult, error)

Parse resolves dependencies for a package or manifest.

type ParseResultWithCacheInfo 

type ParseResultWithCacheInfo struct {
	Graph          *dag.DAG
	CacheHit       bool
	RuntimeVersion string // Target runtime version used (e.g., "3.11")
	RuntimeSource  string // Where runtime came from: "cli", "manifest", "default"
}

ParseResultWithCacheInfo contains the parsed dependency graph plus metadata.

type ResolveDependency 

type ResolveDependency struct {
	Package    string   `json:"package"`
	Resolved   string   `json:"resolved"`
	Constraint string   `json:"constraint"`
	RequiredBy []string `json:"required_by,omitempty"`
}

ResolveDependency represents a single dependency in JSON output.

type ResolveEntry 

type ResolveEntry struct {
	Package     string
	Version     string   // resolved/pinned version
	Constraints []string // version constraints that applied
	RequiredBy  []string // packages that required this dependency
	IsDirect    bool     // true if directly required by root
}

ResolveEntry holds resolution details for a single package.

type ResolveJSON 

type ResolveJSON struct {
	Meta       ResolveMetaJSON     `json:"meta"`
	Root       string              `json:"root"`
	Direct     []ResolveDependency `json:"direct"`
	Transitive []ResolveDependency `json:"transitive"`
	Summary    ResolveSummaryJSON  `json:"summary"`
}

ResolveJSON is the JSON-serializable format for resolution output.

type ResolveMetaJSON 

type ResolveMetaJSON struct {
	RuntimeVersion    string `json:"runtime_version,omitempty"`
	RuntimeSource     string `json:"runtime_source,omitempty"`
	DependencyScope   string `json:"dependency_scope,omitempty"`
	IncludePrerelease bool   `json:"include_prerelease"`
}

ResolveMetaJSON holds metadata about the resolution.

type ResolveResult 

type ResolveResult struct {
	RootName  string
	Entries   []ResolveEntry
	DirectCnt int
	TransCnt  int
}

ResolveResult holds the complete resolution output.

func BuildResolveResult 

func BuildResolveResult(g *dag.DAG, rootID string) ResolveResult

BuildResolveResult extracts resolution details from a DAG.

func (ResolveResult) ToJSON 

func (r ResolveResult) ToJSON(meta ResolveMetaJSON) ResolveJSON

ToJSON converts a ResolveResult to its JSON-serializable form.

type ResolveSummaryJSON 

type ResolveSummaryJSON struct {
	Total      int `json:"total"`
	Direct     int `json:"direct"`
	Transitive int `json:"transitive"`
}

ResolveSummaryJSON holds summary statistics.

type Result 

type Result struct {
	// Graph is the parsed dependency graph.
	Graph *dag.DAG

	// GraphHash is the content hash of the graph.
	GraphHash string

	// Layout contains the layout data (positions, nebraska, etc).
	Layout graph.Layout

	// Artifacts contains rendered outputs keyed by format.
	Artifacts map[string][]byte

	// Stats contains timing and size information.
	Stats Stats

	// CacheInfo tracks which stages hit the cache.
	CacheInfo CacheInfo

	// RuntimeVersion is the target runtime version used for resolution.
	// For Python: "3.11", for Node.js: "20", etc.
	RuntimeVersion string

	// RuntimeSource indicates where the runtime version came from.
	// Values: "manifest" (detected from file), "cli" (user specified), "default" (language default)
	RuntimeSource string
}

Result contains the outputs of a pipeline run.

type Runner 

type Runner struct {
	Cache   cache.Cache
	Keyer   cache.Keyer
	Logger  *log.Logger
	Scanner security.Scanner // Optional vulnerability scanner (nil = scanning disabled)

	// Hooks provides optional per-runner observability hooks.
	// When set, these override the global hooks from the observability package.
	// This enables multi-tenant scenarios where different API requests need
	// different observability backends (e.g., per-request tracing).
	Hooks *RunnerHooks
}

Runner encapsulates pipeline execution with caching. Both CLI and API can use this to avoid duplicating caching logic.

The Runner is stateless except for the cache, logger, and scanner — it doesn't store pipeline results. Multiple goroutines can safely use the same Runner with different options.

func NewRunner 

func NewRunner(c cache.Cache, keyer cache.Keyer, logger *log.Logger) *Runner

NewRunner creates a runner with the given cache and keyer. If keyer is nil, a DefaultKeyer is used. If cache is nil, a NullCache is used (caching disabled).

func NewRunnerWithScanner 

func NewRunnerWithScanner(c cache.Cache, keyer cache.Keyer, logger *log.Logger, scanner security.Scanner) *Runner

NewRunnerWithScanner creates a runner with an optional security scanner. If scanner is nil, security scanning is unavailable even when opts.SecurityScan is true.

func (*Runner) Close 

func (r *Runner) Close() error

Close releases resources held by the runner (primarily the cache).

func (*Runner) Execute 

func (r *Runner) Execute(ctx context.Context, opts Options) (*Result, error)

Execute runs the complete parse → layout → render pipeline with caching.

func (*Runner) GenerateLayout 

func (r *Runner) GenerateLayout(ctx context.Context, g *dag.DAG, opts Options) (graph.Layout, error)

GenerateLayout is a convenience wrapper that calls GenerateLayoutWithCacheInfo and discards the cache hit info.

func (*Runner) GenerateLayoutWithCacheInfo 

func (r *Runner) GenerateLayoutWithCacheInfo(ctx context.Context, g *dag.DAG, opts Options) (graph.Layout, bool, error)

GenerateLayoutWithCacheInfo generates a layout with caching and returns cache hit info.

func (*Runner) ListVersions 

func (r *Runner) ListVersions(ctx context.Context, opts ListOptions) (*ListResult, error)

ListVersions is a convenience method on Runner that uses its cache.

func (*Runner) NewOptimalOrderer 

func (r *Runner) NewOptimalOrderer(timeout time.Duration) *OrdererWithHooks

NewOptimalOrderer creates an optimal search orderer with progress reporting wired to the runner's pipeline hooks. This is a convenience for consumers who want an orderer with progress callbacks that integrate with observability.

Example: 

orderer := runner.NewOptimalOrderer(60 * time.Second)
opts.Orderer = orderer
result, err := runner.Execute(ctx, opts)

func (*Runner) Parse 

func (r *Runner) Parse(ctx context.Context, opts Options) (*dag.DAG, error)

Parse is a convenience wrapper that calls ParseWithCacheInfo and discards the cache hit info.

func (*Runner) ParseWithCacheInfo 

func (r *Runner) ParseWithCacheInfo(ctx context.Context, opts Options) (*ParseResultWithCacheInfo, error)

func (*Runner) PrepareGraph 

func (r *Runner) PrepareGraph(g *dag.DAG, opts Options) (*dag.DAG, error)

PrepareGraph applies normalization and optionally strips vulnerability/license data. Returns the original graph if no transformations are needed.

When opts.ShowVulns is false, vulnerability metadata is removed from nodes so that downstream renderers do not colour nodes by severity. When opts.ShowLicenses is true, license risk analysis is run and annotated. When opts.ShowLicenses is false, any existing license risk metadata is stripped.

func (*Runner) Render 

func (r *Runner) Render(ctx context.Context, layout graph.Layout, g *dag.DAG, opts Options) (map[string][]byte, error)

Render is a convenience wrapper that calls RenderWithCacheInfo and discards the cache hit info.

func (*Runner) RenderWithCacheInfo 

func (r *Runner) RenderWithCacheInfo(ctx context.Context, layout graph.Layout, g *dag.DAG, opts Options) (map[string][]byte, bool, error)

RenderWithCacheInfo generates artifacts with caching and returns cache hit info.

type RunnerHooks 

type RunnerHooks struct {
	Pipeline observability.PipelineHooks
	Security observability.SecurityHooks
}

RunnerHooks contains optional observability hooks for a Runner. Any nil field falls back to the global hook from the observability package.

type Stats 

type Stats struct {
	NodeCount  int
	EdgeCount  int
	ParseTime  time.Duration
	LayoutTime time.Duration
	RenderTime time.Duration
}

Stats contains pipeline execution statistics.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.