Documentation

Overview

    Package migrator provides interfaces and tooling for migrating LUCI configuration files across all known LUCI projects.

    The tool revolves around the user providing a plugin to perform bulk analysis and optional fixes across all LUCI projects' configuration folders, from the point of view of the 'luci-config' service.

    The plugin currently has two main points of extension:

    * FindProblems - Analyze the LUCI project and config file contents to see
      if the project has migrated. This can inspect the project name, as well
      as the presence and contents of any config files.
    * ApplyFix - If FindProblems revealed issues, ApplyFix will be run in the
      context of a local sparse+shallow checkout containing the configuration
      files. This has the ability to run programs in the checkout, as well as
      stat/read/modify files.
    

    This package contains the interface definitions for the migrator plugin.

    Index

    Constants

    View Source
    const TieStderr = "2>&1"

      TieStderr is a special token for Shell.{Run,Retval,Stdout} to indicate that Stderr content should be redirected to Stdout.

      Variables

      This section is empty.

      Functions

      func NonActionable

      func NonActionable(r *Report)

        NonActionable is-a ReportOption which indicates that this Report cannot be fixed by ApplyFix. If there are no Actionable Reports for a given project in FindProblems, the checkout and ApplyFix phase will be skipped.

        Types

        type API

        type API interface {
        	// FindProblems allows you to report problems about a Project, or about
        	// certain configuration files within the project.
        	//
        	// If the method finds issues which warrant followup, it should use
        	// proj.Report and/or proj.ConfigFiles()["filename"].Report. Reporting one or
        	// more problems will cause the migrator tool to set up a checkout for this
        	// project.
        	//
        	// Logging is set up for this context, and will be diverted to a per-project
        	// logfile.
        	//
        	// `proj` is implemented with LUCI Config API calls; it reflects the state of
        	// the project as currently known by the luci-config service.
        	//
        	// This function should panic on error.
        	FindProblems(ctx context.Context, proj Project)
        
        	// ApplyFix allows you to attempt to automatically fix problems within a repo.
        	//
        	// Note that for real implementations you may want to keep details on the
        	// `impl` struct; this will let you carry over information from
        	// FindProblems.
        	//
        	// Logging is set up for this context, and will be diverted to a per-project
        	// logfile.
        	//
        	// `repo.Project()` is implemented with local filesystem interactions on the
        	// checked-out repo. This may differ from the current state of the luci-config
        	// service.
        	//
        	// This function should panic on error.
        	ApplyFix(ctx context.Context, repo Repo)
        }

          API is the implementation of a plugin, as returned by the plugin's InstantiateAPI function.

          One API instance will be created per LUCI project during 'scan', and `FindProblems` will be invoked before `ApplyFix`.

          type ConfigFile

          type ConfigFile interface {
          	Reportable
          
          	// Path returns the path relative to the repo's generated configuration
          	// directory.
          	Path() string
          
          	// RawData returns the data of this configuration file as a string.
          	//
          	// May do a (cached) network operation to retrieve the raw data.
          	RawData() string
          
          	// TextPb parses the raw data as a text proto with "heredoc" processing into
          	// `out`.
          	//
          	// May do a (cached) network operation to retrieve the raw data.
          	//
          	// If there are errors during parsing, this panics.
          	TextPb(out proto.Message)
          }

            ConfigFile encapsulates as single configuration file from a Project.

            type InstantiateAPI

            type InstantiateAPI func() API

              InstantiateAPI is the symbol that plugins must export.

              It should return a new instance of API.

              If this returns nil, it has the effect of a plugin which:

              FindProblems reports a generic problem "FindProblems not defined".
              ApplyFix does nothing.
              

              type Project

              type Project interface {
              	Reportable
              
              	ID() string
              
              	// ConfigFiles returns a mapping of path to config file.
              	//
              	// May do a (cached) network operation to retrieve the data.
              	ConfigFiles() map[string]ConfigFile
              }

                Project encapsulates the pertinent details of a single LUCI Project's configuration files.

                type Repo

                type Repo interface {
                	// ConfigRoot returns the path to the config file 'root'.
                	//
                	// This would be the directory containing `main.star`, if the repo has one,
                	// otherwise is identical to GeneratedConfigRoot.
                	//
                	// TODO: Have a less-heuristic way to find this path in the repo.
                	//
                	// This an 'absolute-style' path (see Shell).
                	ConfigRoot() string
                
                	// GeneratedConfigRoot returns the path to the generated config files (i.e.
                	// the ones seen by the luci-config service).
                	//
                	// This an 'absolute-style' path (see Shell).
                	GeneratedConfigRoot() string
                
                	// Project returns the LUCI Project associated with this repo.
                	//
                	// Files retrieved with ConfigFiles() will be based on the checked-out data.
                	Project() Project
                
                	// Shell returns a new shell object for this repo with its current working
                	// directory set to ConfigRoot().
                	Shell() Shell
                }

                  Repo represents a checked-out git repo on disk.

                  Use Shell to manipulate the repo in your plugin's ApplyFix method.

                  type Report

                  type Report struct {
                  	ReportID
                  
                  	Tag     string
                  	Problem string
                  
                  	// If true, indicates that this report can be fixed by ApplyFix.
                  	Actionable bool
                  
                  	Metadata map[string]stringset.Set
                  }

                    Report stores a single tagged problem (and metadata).

                    func NewReportFromCSVRow

                    func NewReportFromCSVRow(row []string) (ret *Report, err error)

                      NewReportFromCSVRow creates a new Report from a CSVRow written with ToCSVRow.

                      func (*Report) Clone

                      func (r *Report) Clone() *Report

                        Clone returns a deep copy of this Report.

                        func (*Report) ToCSVRow

                        func (r *Report) ToCSVRow() []string

                          ToCSVRow returns a CSV row:

                          Project, ConfigFile, Tag, Problem, Actionable, Metadata*
                          

                          Where Metadata* is one key:value entry per value in Metadata.

                          type ReportDump

                          type ReportDump struct {
                          	// contains filtered or unexported fields
                          }

                            ReportDump is a mapping of all reports, generated via DumpReports(ctx).

                            It maps the ReportID to a list of all Reports found for that ReportID.

                            func NewReportDumpFromCSV

                            func NewReportDumpFromCSV(raw io.Reader) (*ReportDump, error)

                              NewReportDumpFromCSV reads a raw CSV and returns the *ReportDump.

                              CSV must be a compatible schema version (i.e. produced with a similar version of ReportDump.WriteToCSV.

                              func (*ReportDump) Add

                              func (r *ReportDump) Add(reports ...*Report)

                                Add appends one or more reports to this ReportDump.

                                func (*ReportDump) Clone

                                func (r *ReportDump) Clone() *ReportDump

                                  Clone makes a deep copy of this ReportDump.

                                  func (*ReportDump) Empty

                                  func (r *ReportDump) Empty() bool

                                    Empty returns true iff this ReportDump has no entries.

                                    func (*ReportDump) Iterate

                                    func (r *ReportDump) Iterate(cb func(ReportID, []*Report) bool)

                                      Iterate invokes `cb` for each ReportID with all Reports from that ReportID.

                                      `cb` will be called in sorted order on ReportID. If it returns `false`, iteration will stop.

                                      func (*ReportDump) UpdateFrom

                                      func (r *ReportDump) UpdateFrom(other *ReportDump) int

                                        UpdateFrom appends `other` to this ReportDump.

                                        Returns the number of Report records in `other`.

                                        func (*ReportDump) WriteToCSV

                                        func (r *ReportDump) WriteToCSV(out io.Writer) error

                                          WriteToCSV writes this ReportDump out as CSV.

                                          type ReportID

                                          type ReportID struct {
                                          	Project    string
                                          	ConfigFile string
                                          }

                                            ReportID is a simple Project/ConfigFile tuple and identifies the object which generated the report.

                                            func (ReportID) ConfigSet

                                            func (r ReportID) ConfigSet() config.Set

                                              ConfigSet returns the luci-config "config.Set" for this report.

                                              e.g. "projects/${Project}"

                                              func (ReportID) String

                                              func (r ReportID) String() string

                                              type ReportOption

                                              type ReportOption func(*Report)

                                                ReportOption allows attaching additional optional data to reports.

                                                func MetadataOption

                                                func MetadataOption(key string, values ...string) ReportOption

                                                  MetadataOption returns a ReportOption which allows attaching a string-string multimap of metadatadata to a Report.

                                                  type Reportable

                                                  type Reportable interface {
                                                  	// Report logs a problem about this object.
                                                  	//
                                                  	// `tag` should be a CAPS_STRING which makes sense to your particular
                                                  	// application.
                                                  	//
                                                  	// `description` should be a human readable explanation of the problem.
                                                  	//
                                                  	// Example:
                                                  	//    tag: MISSING_BLARF
                                                  	//    description: "blarf.cfg file is missing"
                                                  	Report(tag, description string, opts ...ReportOption)
                                                  }

                                                    Reportable is an interface for reporting problems about an object.

                                                    Implemented by Project and ConfigFile.

                                                    type Shell

                                                    type Shell interface {
                                                    	// Cd changes the current directory.
                                                    	//
                                                    	// Absolute paths (i.e. beginning with '/') are interpreted as relative to the
                                                    	// repo root.
                                                    	//
                                                    	// Attempting to Cd out of the top of the repo is an error.
                                                    	Cd(path string)
                                                    
                                                    	// ModifyFile allows trivial modification of a file.
                                                    	//
                                                    	// This will call `modify` with the contents of the old file ("" if the file
                                                    	// didn't exist), and will write the returned string back to `path`, if the
                                                    	// returned string is different. This will create missing intermediate
                                                    	// directories.
                                                    	//
                                                    	// Allows supplying FileMode. If omitted, defaults to 0666 (i.e. a+rw) for new
                                                    	// files, and otherwise will keep the mode of the existing file.
                                                    	ModifyFile(path string, modify func(oldContents string) string, mode ...os.FileMode)
                                                    
                                                    	// Stat returns the FileInfo for the entity at `path`, or nil if no such
                                                    	// entity exists.
                                                    	Stat(path string) os.FileInfo
                                                    
                                                    	// Run executes a command `name` with arguments `args`.
                                                    	//
                                                    	// `name` may be relative to the Shell's cwd (e.g. `./main.star`), or to $PATH
                                                    	// (e.g. `git`).
                                                    	//
                                                    	// The command is expected to return exitcode 0.
                                                    	//
                                                    	// The command runs with the cwd of the Shell.
                                                    	//
                                                    	// Stdout+Stderr are redirected to logging (with Stdout logged at 'Info' level
                                                    	// and Stderr logged at 'Error' level).
                                                    	//
                                                    	// If the very last item in `args` is the TieStderr constant, both will be
                                                    	// tied together and all output will be logged at 'Info' level.
                                                    	Run(name string, args ...string)
                                                    
                                                    	// Same as `Run`, but returns the exitcode of the process (instead of
                                                    	// asserting an exit code of 0).
                                                    	Retval(name string, args ...string) int
                                                    
                                                    	// Same as `Run`, but returns the stdout of the process. If TieStderr is the
                                                    	// last argument, then this captures Stderr, too.
                                                    	Stdout(name string, args ...string) string
                                                    }

                                                    Shell is a basic interface for interacting with the Repo.

                                                    Paths

                                                    All `path` arguments to the Shell are either:

                                                    * start with '/' and are relative to the corresponding Repo's root.
                                                    * OR; are relative to the Shell's current working directory.
                                                    

                                                    It is not permitted to access a path outside the Repo's root.

                                                    A just-created Shell starts with the 'cwd' at the ConfigDir.

                                                    Errors

                                                    All functions of Shell will panic under error conditions. This is consistent with the API of the ApplyFix plugin method.

                                                    Directories

                                                    Path Synopsis
                                                    cmd
                                                    internal
                                                    plugsupport
                                                    Package plugsupport provides implementations for loading migrator plugins.
                                                    Package plugsupport provides implementations for loading migrator plugins.
                                                    plugsupport/templates
                                                    Package templates is generated by go.chromium.org/luci/tools/cmd/assets.
                                                    Package templates is generated by go.chromium.org/luci/tools/cmd/assets.