Documentation

Overview

    Package model contains core CIPD datastore entities.

    Index

    Constants

    View Source
    const (
    	// Hidden can be used in place of 'true' when working with Package.Hidden flag.
    	Hidden = true
    	// Visible can be used in place of 'false' when working with Package.Hidden flag.
    	Visible = false
    )
    View Source
    const NoInstance = "NONE"

      NoInstance is used in place of empty Instance field in Event entity, so that we can query for all events that are not associated with instances and only them (these are package-level events).

      Variables

      View Source
      var EventsEpoch = time.Date(2018, time.January, 1, 0, 0, 0, 0, time.UTC)

        EventsEpoch is used to calculate timestamps used to order entities in the datastore.

        time.Time is stored with millisecond precision in the datastore. This is not enough to preserve the order of cipd.Event protos (that use nanoseconds for ordering events within a transactional group).

        So instead we store (event.When-EventsEpoch).Nanos() as int64. The original timestamp is still available in cipd.Event proto. This is only for datastore index.

        Functions

        func AttachMetadata

        func AttachMetadata(ctx context.Context, inst *Instance, md []*api.InstanceMetadata) error

          AttachMetadata transactionally attaches metadata to an instance.

          Mutates `md` in place by calculating fingerprints and "guessing" content type if necessary.

          Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

          Returns gRPC-tagged errors:

          NotFound if there's no such instance or package.
          FailedPrecondition if some processors are still running.
          Aborted if some processors have failed.
          Internal on fingerprint collision.
          

          func AttachTags

          func AttachTags(c context.Context, inst *Instance, tags []*api.Tag) error

            AttachTags transactionally attaches a bunch of tags to an instance.

            Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

            Returns gRPC-tagged errors:

            NotFound if there's no such instance or package.
            FailedPrecondition if some processors are still running.
            Aborted if some processors have failed.
            Internal on tag ID collision.
            

            func CheckInstanceExists

            func CheckInstanceExists(c context.Context, inst *Instance) error

              CheckInstanceExists fetches the instance and verifies it exists.

              Can be called as part of a transaction. Updates 'inst' in place.

              Returns gRPC-tagged NotFound error if there's no such instance or package.

              func CheckInstanceReady

              func CheckInstanceReady(c context.Context, inst *Instance) error

                CheckInstanceReady fetches the instance and verifies it exists and has zero pending or failed processors.

                Can be called as part of a transaction. Updates 'inst' in place.

                Returns gRPC-tagged errors:

                NotFound if there's no such instance or package.
                FailedPrecondition if some processors are still running.
                Aborted if some processors have failed.
                

                func CheckPackageExists

                func CheckPackageExists(c context.Context, pkg string) error

                  CheckPackageExists verifies the package exists.

                  Returns gRPC-tagged NotFound error if there's no such package.

                  func CheckPackages

                  func CheckPackages(c context.Context, names []string, includeHidden bool) ([]string, error)

                    CheckPackages given a list of package names returns packages that exist, in the order they are listed in the list.

                    If includeHidden is false, omits hidden packages from the result.

                    Returns only transient errors.

                    func DeletePackage

                    func DeletePackage(c context.Context, pkg string) error

                      DeletePackage deletes all entities associated with a package.

                      Deleting everything under a single transaction is generally not feasible, since deleting packages with lots of metadata exceeds transaction limits.

                      Instead we delete most of the stuff non-transactionally first, and then delete the rest inside the transaction (to make sure nothing stays, even if it was created while we were deleting stuff).

                      Returns grpc-tagged errors (in particular NotFound if there's no such package).

                      func DeleteRef

                      func DeleteRef(c context.Context, pkg, ref string) error

                        DeleteRef removes the ref if it exists.

                        Does nothing if there's no such ref or package.

                        func DetachMetadata

                        func DetachMetadata(ctx context.Context, inst *Instance, md []*api.InstanceMetadata) error

                          DetachMetadata detaches a bunch of metadata entries from an instance.

                          Assumes inputs are already validated. If Fingerprint is populated, uses it to identifies entries to detach. Otherwise calculates it from Key and Value (which must be populated in this case).

                          Launches a transaction inside (and thus can't be a part of a transaction itself).

                          func DetachTags

                          func DetachTags(c context.Context, inst *Instance, tags []*api.Tag) error

                            DetachTags detaches a bunch of tags from an instance.

                            Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself).

                            'inst' is used only for its key.

                            func EmitEvent

                            func EmitEvent(c context.Context, e *api.Event) error

                              EmitEvent adds a single event to the event log.

                              Prefer using Events to add multiple events at once.

                              func EmitMetadataEvents

                              func EmitMetadataEvents(c context.Context, before, after *api.PrefixMetadata) error

                                EmitMetadataEvents compares two metadatum of a prefix and emits events for fields that have changed.

                                func FetchProcessors

                                func FetchProcessors(c context.Context, inst *Instance) ([]*api.Processor, error)

                                  FetchProcessors fetches results of all processors assigned to the instance and returns them as cipd.Processor proto messages (sorted by processor ID).

                                  func FlushEventsToBQ

                                  func FlushEventsToBQ(c context.Context) error

                                    FlushEventsToBQ sends all buffered events to BigQuery.

                                    It is fine to call FlushEventsToBQ concurrently from multiple request handlers, if necessary (it will effectively parallelize the flush).

                                    func IsTextContentType

                                    func IsTextContentType(ct string) bool

                                      IsTextContentType checks the metadata content type against a list of known-good values.

                                      func ListPackages

                                      func ListPackages(c context.Context, prefix string, includeHidden bool) (out []string, err error)

                                        ListPackages returns a list of names of packages under the given prefix.

                                        Lists all packages recursively. If there's package named as 'prefix' it is NOT included in the result. Only packaged under the prefix are included.

                                        The result is sorted by the package name. Returns only transient errors.

                                        func NewEventsQuery

                                        func NewEventsQuery() *datastore.Query

                                          NewEventsQuery returns an unfiltered query for Event entities.

                                          func PackageKey

                                          func PackageKey(c context.Context, pkg string) *datastore.Key

                                            PackageKey returns a datastore key of some package, given its name.

                                            func QueryEvents

                                            func QueryEvents(c context.Context, q *datastore.Query) ([]*api.Event, error)

                                              QueryEvents fetches and deserializes all events matching the query, sorting them by timestamp (most recent first).

                                              Start the query with NewEventsQuery.

                                              func ResolveTag

                                              func ResolveTag(c context.Context, pkg string, tag *api.Tag) (string, error)

                                                ResolveTag searches for a given tag among all instances of the package and returns an ID of the instance the tag is attached to.

                                                Assumes inputs are already validated. Doesn't double check the instance exists.

                                                Returns gRPC-tagged errors:

                                                NotFound if there's no such tag at all.
                                                FailedPrecondition if the tag resolves to multiple instances.
                                                

                                                func SetPackageHidden

                                                func SetPackageHidden(c context.Context, pkg string, hidden bool) error

                                                  SetPackageHidden updates Hidden field of the package.

                                                  If the package is missing returns datastore.ErrNoSuchEntity. All other errors are transient.

                                                  func SetRef

                                                  func SetRef(c context.Context, ref string, inst *Instance) error

                                                    SetRef moves or creates a ref.

                                                    Assumes inputs are already validated. Launches a transaction inside (and thus can't be a part of a transaction itself). Updates 'inst' in-place with the most recent instance state.

                                                    Returns gRPC-tagged errors:

                                                    NotFound if there's no such instance or package.
                                                    FailedPrecondition if some processors are still running.
                                                    Aborted if some processors have failed.
                                                    

                                                    func TagID

                                                    func TagID(t *api.Tag) string

                                                      TagID calculates Tag entity ID (SHA1 digest) from the given tag.

                                                      Panics if the tag is invalid.

                                                      func Txn

                                                      func Txn(c context.Context, name string, cb func(context.Context) error) error

                                                        Txn runs the callback in a datastore transaction, marking commit errors with transient tag.

                                                        The given name will be used for logging and error messages.

                                                        Types

                                                        type Event

                                                        type Event struct {
                                                        	ID      int64          `gae:"$id"`      // auto-generated
                                                        	Package *datastore.Key `gae:"$parent"`  // see PackageKey()
                                                        	Event   []byte         `gae:",noindex"` // serialized cipd.Event proto
                                                        
                                                        	// Fields extracted from cipd.Event for indexing.
                                                        	Kind          api.EventKind
                                                        	Who           string
                                                        	When          int64  // nanoseconds since EventsEpoch
                                                        	Instance      string // literal 'NONE' for package events, to be able to query them
                                                        	Ref           string
                                                        	Tag           string
                                                        	MdKey         string
                                                        	MdFingerprint string
                                                        	// contains filtered or unexported fields
                                                        }

                                                          Event in a global structured event log.

                                                          It exists in both BigQuery (for adhoc queries) and in Datastore (for showing in web UI, e.g. for "recent tags" feature).

                                                          ID as auto-generated. The parent entity is the corresponding Package entity (so that events can be committed transactionally with actual changes).

                                                          func (*Event) FromProto

                                                          func (e *Event) FromProto(c context.Context, p *api.Event) *Event

                                                            FromProto fills in the entity based on the proto message.

                                                            Panics if the proto can't be serialized. This should never happen.

                                                            Returns the entity itself for easier chaining.

                                                            type Events

                                                            type Events struct {
                                                            	// contains filtered or unexported fields
                                                            }

                                                              Events collects events emitted in some transaction and then flushes them.

                                                              func (*Events) Emit

                                                              func (t *Events) Emit(e *api.Event)

                                                                Emit adds an event to be flushed later in Flush.

                                                                'Who' and 'When' fields are populated in Flush using values from the context.

                                                                func (*Events) Flush

                                                                func (t *Events) Flush(c context.Context) error

                                                                  Flush sends all pending events to the datastore/bigquery.

                                                                  Returned errors are tagged as transient. On success, clears the pending events queue.

                                                                  type Instance

                                                                  type Instance struct {
                                                                  	InstanceID string         `gae:"$id"`     // see common.ObjectRefToInstanceID()
                                                                  	Package    *datastore.Key `gae:"$parent"` // see PackageKey()
                                                                  
                                                                  	RegisteredBy string    `gae:"registered_by"` // who registered it
                                                                  	RegisteredTs time.Time `gae:"registered_ts"` // when it was registered
                                                                  
                                                                  	// Names of currently running or scheduled processors.
                                                                  	ProcessorsPending []string `gae:"processors_pending"`
                                                                  	// Names of processors that successfully finished the processing.
                                                                  	ProcessorsSuccess []string `gae:"processors_success"`
                                                                  	// Names of processors that returned fatal errors.
                                                                  	ProcessorsFailure []string `gae:"processors_failure"`
                                                                  	// contains filtered or unexported fields
                                                                  }

                                                                    Instance represents a package instance as it is stored in the datastore.

                                                                    Contains some instance metadata, in particular a list of processors that scanned the instance (see below).

                                                                    The parent entity is the corresponding package entity. ID is derived from package instance file digest, see common.ObjectRefToInstanceID().

                                                                    Compatible with the python version of the backend.

                                                                    func ListInstances

                                                                    func ListInstances(c context.Context, pkg string, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

                                                                      ListInstances lists instances of a package, more recent first.

                                                                      Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). pageSize must be positive.

                                                                      func RegisterInstance

                                                                      func RegisterInstance(c context.Context, inst *Instance, cb func(context.Context, *Instance) error) (reg bool, out *Instance, err error)

                                                                        RegisterInstance transactionally registers an instance (and the corresponding package), if it isn't registered already.

                                                                        Calls the given callback (inside the transaction) if it is indeed registering a new instance. The callback may mutate the instance entity before it is stored. The callback may be called multiple times in case of retries (each time it will be given a fresh instance to be mutated).

                                                                        Returns (true, entity, nil) if the instance was registered just now or (false, entity, nil) if it existed before.

                                                                        In either case, it returns the entity that is stored now in the datastore. It is either the new instance, or something that existed there before.

                                                                        func ResolveVersion

                                                                        func ResolveVersion(c context.Context, pkg, version string) (*Instance, error)

                                                                          ResolveVersion takes a version identifier (an instance ID, a ref or a tag) and resolves it into a concrete package instance, ensuring it exists.

                                                                          Returns gRPC-tagged errors:

                                                                          InvalidArgument if the version string format is invalid.
                                                                          NotFound if there's no such package or such version.
                                                                          FailedPrecondition if the tag resolves to multiple instances.
                                                                          

                                                                          func SearchInstances

                                                                          func SearchInstances(c context.Context, pkg string, tags []*api.Tag, pageSize int32, cursor datastore.Cursor) (out []*Instance, nextCur datastore.Cursor, err error)

                                                                            SearchInstances lists instances of a package with all given tags attached.

                                                                            Only does a query over Instances entities. Doesn't check whether the Package entity exists. Returns up to pageSize entities, plus non-nil cursor (if there are more results). 'pageSize' must be positive.

                                                                            func (*Instance) FromProto

                                                                            func (e *Instance) FromProto(c context.Context, p *api.Instance) *Instance

                                                                              FromProto fills in the entity based on the proto message.

                                                                              Returns the entity itself for easier chaining.

                                                                              Doesn't touch output-only fields at all.

                                                                              func (*Instance) Proto

                                                                              func (e *Instance) Proto() *api.Instance

                                                                                Proto returns cipd.Instance proto with information from this entity.

                                                                                type InstanceMetadata

                                                                                type InstanceMetadata struct {
                                                                                	Fingerprint string         `gae:"$id"`     // see common.InstanceMetadataFingerprint
                                                                                	Instance    *datastore.Key `gae:"$parent"` // a key of the corresponding Instance entity
                                                                                
                                                                                	Key         string `gae:"key"`                  // the metadata key
                                                                                	Value       []byte `gae:"value,noindex"`        // the metadata payload, can be big
                                                                                	ContentType string `gae:"content_type,noindex"` // a content type (perhaps guessed)
                                                                                
                                                                                	AttachedBy string    `gae:"attached_by"` // who added this metadata
                                                                                	AttachedTs time.Time `gae:"attached_ts"` // when it was added
                                                                                	// contains filtered or unexported fields
                                                                                }

                                                                                  InstanceMetadata represents one instance metadata entry.

                                                                                  It is a key-value pair (along with some additional attributes).

                                                                                  The parent entity is the instance entity. ID is derived from the key-value pair, see common.InstanceMetadataFingerprint.

                                                                                  func ListMetadata

                                                                                  func ListMetadata(ctx context.Context, inst *Instance) ([]*InstanceMetadata, error)

                                                                                    ListMetadata lists all instance metadata.

                                                                                    The result is ordered by AttachedTs (the most recent first).

                                                                                    func ListMetadataWithKeys

                                                                                    func ListMetadataWithKeys(ctx context.Context, inst *Instance, keys []string) ([]*InstanceMetadata, error)

                                                                                      ListMetadataWithKeys lists instance metadata with any of the given keys.

                                                                                      The result is ordered by AttachedTs (the most recent first).

                                                                                      func (*InstanceMetadata) Proto

                                                                                      func (md *InstanceMetadata) Proto() *api.InstanceMetadata

                                                                                        Proto returns cipd.InstanceMetadata proto with information from this entity.

                                                                                        Assumes the entity is valid.

                                                                                        type Package

                                                                                        type Package struct {
                                                                                        	Name string `gae:"$id"` // e.g. "a/b/c"
                                                                                        
                                                                                        	RegisteredBy string    `gae:"registered_by"` // who registered it
                                                                                        	RegisteredTs time.Time `gae:"registered_ts"` // when it was registered
                                                                                        
                                                                                        	Hidden bool `gae:"hidden"` // if true, hide from the listings
                                                                                        	// contains filtered or unexported fields
                                                                                        }

                                                                                          Package represents a package as it is stored in the datastore.

                                                                                          It is mostly a marker that the package exists plus some minimal metadata about this specific package. Metadata for the package prefix is stored separately elsewhere (see 'metadata' package). Package instances, tags and refs are stored as child entities (see below).

                                                                                          Root entity. ID is the package name.

                                                                                          Compatible with the python version of the backend.

                                                                                          type ProcessingResult

                                                                                          type ProcessingResult struct {
                                                                                          	ProcID   string         `gae:"$id"`     // processor that generated the result
                                                                                          	Instance *datastore.Key `gae:"$parent"` // instance it was generated from
                                                                                          
                                                                                          	CreatedTs time.Time `gae:"created_ts"`     // when it was generated
                                                                                          	Success   bool      `gae:"success"`        // mostly for for indexing
                                                                                          	Error     string    `gae:"error,noindex"`  // for Success == false
                                                                                          	ResultRaw []byte    `gae:"result,noindex"` // for Success == true
                                                                                          	// contains filtered or unexported fields
                                                                                          }

                                                                                            ProcessingResult holds information extracted from the package instance file.

                                                                                            It is obtained during an asynchronous post processing step triggered after the instance is uploaded. Immutable.

                                                                                            Entity ID is a processor name used to extract it. Parent entity is PackageInstance the information was extracted from.

                                                                                            func (*ProcessingResult) Proto

                                                                                            func (p *ProcessingResult) Proto() (*api.Processor, error)

                                                                                              Proto returns cipd.Processor proto with information from this entity.

                                                                                              func (*ProcessingResult) ReadResult

                                                                                              func (p *ProcessingResult) ReadResult(r interface{}) error

                                                                                                ReadResult deserializes the result into the given variable.

                                                                                                Does nothing if there's no results stored.

                                                                                                func (*ProcessingResult) ReadResultIntoStruct

                                                                                                func (p *ProcessingResult) ReadResultIntoStruct(s *structpb.Struct) error

                                                                                                  ReadResultIntoStruct deserializes the result into the protobuf.Struct.

                                                                                                  Does nothing if there's no results stored.

                                                                                                  func (*ProcessingResult) WriteResult

                                                                                                  func (p *ProcessingResult) WriteResult(r interface{}) error

                                                                                                    WriteResult overwrites ResultRaw field with compressed JSON-serialized 'r'.

                                                                                                    'r' should serialize into a JSON object, e.g '{...}'.

                                                                                                    type Ref

                                                                                                    type Ref struct {
                                                                                                    	Name    string         `gae:"$id"`     // e.g. "latest"
                                                                                                    	Package *datastore.Key `gae:"$parent"` // see PackageKey()
                                                                                                    
                                                                                                    	InstanceID string    `gae:"instance_id"` // see common.ObjectRefToInstanceID()
                                                                                                    	ModifiedBy string    `gae:"modified_by"` // who moved it the last time
                                                                                                    	ModifiedTs time.Time `gae:"modified_ts"` // when it was moved the last time
                                                                                                    	// contains filtered or unexported fields
                                                                                                    }

                                                                                                      Ref represents a named pointer to some package instance.

                                                                                                      ID is a ref name, the parent entity is the corresponding Package.

                                                                                                      Compatible with the python version of the backend.

                                                                                                      func GetRef

                                                                                                      func GetRef(c context.Context, pkg, ref string) (*Ref, error)

                                                                                                        GetRef fetches the given ref.

                                                                                                        Returns gRPC-tagged NotFound error if there's no such ref.

                                                                                                        func ListInstanceRefs

                                                                                                        func ListInstanceRefs(c context.Context, inst *Instance) (out []*Ref, err error)

                                                                                                          ListInstanceRefs returns all refs that point to a particular instance, most recently modified first.

                                                                                                          This is a subset of the output of ListPackageRefs for the corresponding package.

                                                                                                          Assumes 'inst' is a valid Instance, panics otherwise.

                                                                                                          Returns an empty list if there's no such instance at all.

                                                                                                          func ListPackageRefs

                                                                                                          func ListPackageRefs(c context.Context, pkg string) (out []*Ref, err error)

                                                                                                            ListPackageRefs returns all refs in a package, most recently modified first.

                                                                                                            Returns an empty list if there's no such package at all.

                                                                                                            func (*Ref) Proto

                                                                                                            func (e *Ref) Proto() *api.Ref

                                                                                                              Proto returns cipd.Ref proto with information from this entity.

                                                                                                              type Tag

                                                                                                              type Tag struct {
                                                                                                              	ID       string         `gae:"$id"`     // see TagID()
                                                                                                              	Instance *datastore.Key `gae:"$parent"` // key of corresponding Instance entity
                                                                                                              
                                                                                                              	Tag          string    `gae:"tag"`           // the tag itself, as "k:v" pair
                                                                                                              	RegisteredBy string    `gae:"registered_by"` // who added this tag
                                                                                                              	RegisteredTs time.Time `gae:"registered_ts"` // when it was added
                                                                                                              	// contains filtered or unexported fields
                                                                                                              }

                                                                                                                Tag represents a "key:value" pair attached to some package instance.

                                                                                                                Tags exist as separate entities (as opposed to being a repeated property of Instance entity) to allow essentially unlimited number of them. Also we often do not want to fetch all tags when fetching Instance entity.

                                                                                                                ID is hex-encoded SHA1 of the tag. The parent entity is the corresponding Instance. The tag string can't be made a part of the key because the total key length (including entity kind names and all parent keys) is limited to 500 bytes and tags are allowed to be pretty long (up to 400 bytes).

                                                                                                                There's a possibility someone tries to abuse a collision in SHA1 hash used by TagID() to attach or detach some wrong tag. The chance is minuscule, and it's not a big deal if it happens now, but it may become important in the future once tags have their own ACLs. For the sake of paranoia, we double check 'Tag' field in all entities we touch.

                                                                                                                Compatible with the python version of the backend.

                                                                                                                func ListInstanceTags

                                                                                                                func ListInstanceTags(c context.Context, inst *Instance) (out []*Tag, err error)

                                                                                                                  ListInstanceTags returns all tags attached to an instance, sorting them by the tag key first, and then by the timestamp (most recent first).

                                                                                                                  Returns an empty list if there's no such instance at all.

                                                                                                                  func (*Tag) Proto

                                                                                                                  func (t *Tag) Proto() *api.Tag

                                                                                                                    Proto returns cipd.Tag proto with information from this entity.

                                                                                                                    Assumes the tag is valid.