dirmd

package
Version: v0.0.0-...-e560ebb Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 13, 2021 License: BSD-3-Clause Imports: 27 Imported by: 0

README

Directory metadata

DIR_METADATA files are a source-focused mechanism by which owners can provide users of their code important information. They map a directory to external entities, including:

  • The Monorail component where bugs should be filed.
  • The team responsible for the code.
  • The OS type.

This infrastructure is overall generic. It has Chromium-specific proto fields, but their usage is not required, and overall this infra can be used by other projects as well.

DIR_METADATA files

DIR_METADATA files are structured text-protobuf files that are amenable to programmatic interaction. Example:

team_email: "team@chromium.org"
os: LINUX
monorail {
  project: "chromium"
  component: "Infra"
}

For file schema, see Metadata message.

Inheritance

DIR_METADATA files apply to all contents of a directory including its subdirectories. By default, individual fields are inherited by subdirectories. In the following, example, the value of monorail.project field in directory a/b is "chromium".

a/DIR_METADATA

monorail {
  project: "chromium"
  component: "Component"
}

a/b/DIR_METADATA

monorail { component: "Component>Foo" }

dirmd tool

dirmd is a tool to work with DIR_METADATA files. Features:

  • Gather all metadata in a given directory tree and export it to a single JSON file. Optionally remove all redundant metadata, or instead compute inherited metadata.
  • Compute inherited metadata for a given set of directories.
  • Convert from text proto to JSON which is easier to interpret by programs that don't have an easy access to the protobuf files.
  • Validate a given set of files. Used in PRESUBMIT.
  • Fall back to legacy OWNERS files, so that metadata can migrate off of OWNERS files smoothly.

The tool is deployed via depot_tools and is available as a CIPD package.

The tool also hosts implementation of the metadata-exporter builder.

Source code: ./cmd/dirmd.

Library

Go package infra/tools/dirmd can be used work with METADATA files programmatically. The dirmd executable is a thin wrapper around it.

Continuous export

metadata-exporter builder contunuously exports metadata from the src.git to gs://chrome-metadata/metadata_reduced.json and gs://chrome-metadata/metadata_computed.json. metadata_reduced.json is semantic equivalent of metadata_computed.json, but it has all redundant information removed (see MappingForm.REDUCED). As of July 2020, the update latency is up to 20 min.

Documentation

Overview

Package dirmd implements parsing and transformations of directory metadata.

Index

Constants

View Source
const Filename = "DIR_METADATA"

Filename is the standard name of the metadata file.

View Source
const OwnersFilename = "OWNERS"

OwnersFilename is the filename for OWNERS files.

Variables

This section is empty.

Functions

func Merge

func Merge(dst, src *dirmdpb.Metadata)

Merge merges metadata from src to dst. Does nothing if src is nil.

The current implementation is just proto.Merge, but it may change in the future.

func MigrateMetadata

func MigrateMetadata(dir string) error

MigrateMetadata moves metadata from legacy OWNERS of the given directory into DIR_METADATA file.

func ReadFile

func ReadFile(fileName string) (*dirmdpb.Metadata, error)

ReadFile reads metadata from a file.

func ReadMetadata

func ReadMetadata(dir string) (*dirmdpb.Metadata, error)

ReadMetadata reads metadata from a single directory. See also ReadMapping.

Returns (nil, nil) if the metadata is not defined.

func ReadOwners

func ReadOwners(dir string) (md *dirmdpb.Metadata, ignored []string, err error)

ReadOwners reads metadata from legacy OWNERS of the given directory and returns a dirmdpb.Metadata message along with the lines that were ignored. Returns (nil, nil, nil) if OWNERS file does not exist.

func ToLocationTags

func ToLocationTags(mapping *Mapping) (*sinkpb.LocationTags_Repo, error)

ToLocationTags converts all dir metadata to test location tags.

func ValidateFile

func ValidateFile(fileName string) error

ValidateFile returns a non-nil error if the metadata file is invalid.

A valid file has a base filename "DIR_METADATA" or "OWNERS". The format of its contents correspond to the base name.

Types

type Mapping

type Mapping dirmdpb.Mapping

Mapping is a mapping from a directory to its metadata.

It wraps the corresponding protobuf message and adds utility functions.

func NewMapping

func NewMapping(size int) *Mapping

NewMapping initializes an empty mapping.

func ReadMapping

func ReadMapping(ctx context.Context, form dirmdpb.MappingForm, dirs ...string) (*Mapping, error)

ReadMapping reads all metadata from files in git in the given directories.

Each directory must reside in a git checkout. One of the repos must be the root repo, while other repos must be its sub-repos. In other words, all git repos referred to by the directories must be subdirectories of one of the repos. The root dir of the root repo becomes the metadata root.

Unless the form is sparse, the returned mapping includes metadata of ancestors and descendants of the specified directories. In the sparse form, metadata of only the specified directories is returned, which is usually much faster.

Descendants of the specified directories are discovered using "git ls-files <dir>" and not FS walk. This means files outside of the repo are ignored, as well as files matched by .gitignore files. Note that when reading ancestors of the specified directories, the .gitignore files are not respected. This inconsistency should not make a difference in the vast majority of cases because it is confusing to have git-ignored DIR_METADATA in the middle of the ancestry chain. Such a case might indicate that DIR_METADATA files are used incorrectly. This behavior can be changed, but it would come with a performance penalty.

func (*Mapping) Clone

func (m *Mapping) Clone() *Mapping

Clone returns a deep copy of m.

func (*Mapping) Compute

func (m *Mapping) Compute(key string) (*dirmdpb.Metadata, error)

Compute computes metadata for the given directory key.

func (*Mapping) ComputeAll

func (m *Mapping) ComputeAll() error

ComputeAll computes full metadata for each dir.

func (*Mapping) Proto

func (m *Mapping) Proto() *dirmdpb.Mapping

Proto converts m back to the protobuf message.

func (*Mapping) Reduce

func (m *Mapping) Reduce() error

Reduce removes all redundant information.

Directories

Path Synopsis
cli
updater
Package updater computes inherited metadata and uploads it to GCS.
Package updater computes inherited metadata and uploads it to GCS.
cmd

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL