remotecache

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package remotecache puts a shared, remote cache behind senro's local one, so two machines can reuse each other's work: a fresh CI runner starts warm instead of empty. The store is an S3-compatible bucket (Open) or an OCI registry repository (OpenOCI); both produce the same *Remote and differ only in naming, confined to RemoteObjects and docs.

The remote is a second tier behind the local one, never a replacement: reads try disk first, a remote hit is written through to disk, writes go to disk first. Both the CAS and the action cache are tiered; either alone is useless.

Two rules are not negotiable. Nothing is served without verifying it: every object goes through cas.DecodeVerify and every entry is checked against its key, because a store returning wrong bytes is ordinary and serving them would silently poison every downstream build. And a cache that is down never fails a run: unreachable, unauthenticated, slow or erroring all mean "no remote cache" and nothing more; the run says so loudly once, then stops trying.

Index

Constants

View Source 
const (
	// EnvTarget turns the shared cache on and says where it is, which is also
	// what chooses the backend: "s3://<bucket>[/<prefix>]" for a bucket, and
	// "oci://<registry>/<repository>" for a registry.
	EnvTarget = "SENRO_REMOTE_CACHE"

	// EnvEndpoint is the object store's URL.
	EnvEndpoint = "SENRO_REMOTE_CACHE_ENDPOINT"
	// EnvRegion scopes the request signature.
	EnvRegion = "SENRO_REMOTE_CACHE_REGION"
	// EnvPathStyle overrides the bucket addressing style.
	EnvPathStyle = "SENRO_REMOTE_CACHE_PATH_STYLE"

	// EnvUsername and EnvPassword are the credential presented to the
	// registry's token endpoint. senro's own names, unlike the AWS ones
	// below: no standard pair exists for a registry (Docker keeps its
	// credential in a config file senro deliberately does not read, and
	// every forge spells its own differently).
	EnvUsername = "SENRO_REMOTE_CACHE_USERNAME"
	EnvPassword = "SENRO_REMOTE_CACHE_PASSWORD"
	// EnvPlainHTTP talks to the registry over http rather than https, for one
	// on a trusted network that serves no certificate.
	EnvPlainHTTP = "SENRO_REMOTE_CACHE_PLAIN_HTTP"

	// EnvTimeout bounds one request, as a Go duration.
	EnvTimeout = "SENRO_REMOTE_CACHE_TIMEOUT"
	// EnvReadOnly makes a run read the shared cache and never write it.
	EnvReadOnly = "SENRO_REMOTE_CACHE_READ_ONLY"

	// The bucket's credential variables, deliberately the standard AWS names:
	// CI already sets these when a job assumes a role, and a senro-specific
	// spelling would mean every pipeline copying them across for no reason.
	EnvAccessKeyID     = "AWS_ACCESS_KEY_ID"
	EnvSecretAccessKey = "AWS_SECRET_ACCESS_KEY"
	EnvSessionToken    = "AWS_SESSION_TOKEN"
)

The environment variables a shared cache is configured from. They live here, not in the root package that parses them, so EnvNames can be one list; the root package re-exports each as public API.

View Source 
const DefaultArchiveGrace = 60 * time.Second

DefaultArchiveGrace is how long Close waits for the queue to drain. Generous, because losing logs at the last moment defeats archiving; finite, because a CI job that will not exit is worse.

View Source 
const DefaultPrefix = "senro"

DefaultPrefix is where senro's objects live in a bucket that may hold other things. Named rather than empty so that pointing senro at a shared bucket does not scatter opaque hex directories across its root.

Variables

This section is empty.

Functions

func ClearEnv 

func ClearEnv()

ClearEnv removes every one of them from this process's environment. For test binaries: a run with no explicit configuration reads these variables, so a developer exporting SENRO_REMOTE_CACHE would have the test suite writing into their team's bucket, and WithCacheDir isolates only the LOCAL cache root. Non-test code never calls this.

func EnvNames 

func EnvNames() []string

EnvNames is every variable above, in one list.

func OCITag 

func OCITag(d cas.Digest) string

OCITag is the tag one object is stored under. Returns "" for a malformed digest (digests arrive from logs, plans and command lines, untrusted). The layout version is in the tag because a tag is the only namespace a repository has, and a later layout must live beside this one.

Types

type Archiver 

type Archiver struct {
	// contains filtered or unexported fields
}

Archiver uploads a run's logs in the background. The rule it keeps: a step's execution never waits on an upload. Enqueuing is a non-blocking send; everything after happens on this type's own goroutines, and every failure there degrades rather than propagating.

One Archiver serves one run. Close drains it with a bounded grace: "a finished run has uploaded its logs" without "a run cannot finish until its uploads do".

func (*Archiver) Close 

func (a *Archiver) Close(grace time.Duration)

Close stops accepting work and waits, up to grace, for what is queued.

Zero or negative grace means DefaultArchiveGrace. It is idempotent, and safe on a nil Archiver.

func (*Archiver) Dropped 

func (a *Archiver) Dropped() int64

func (*Archiver) Ledger 

func (a *Archiver) Ledger(path string)

Ledger queues the run's event ledger for upload. Without it the archived logs are unreadable in practice: a store full of anonymous log files is not a record of a run.

func (*Archiver) Stream 

func (a *Archiver) Stream(step string, attempt int, stream, path string)

Stream queues one completed log stream for upload. Call it once the attempt's writers have closed: uploading a file still being written would archive a prefix of it, and the archive is the only copy that survives the runner.

func (*Archiver) Uploaded 

func (a *Archiver) Uploaded() int64

Uploaded and Dropped report what happened, for a test and for anything that wants to say so at the end of a run.

type Config 

type Config struct {
	// Endpoint is the object store's URL, such as
	// "https://s3.eu-west-1.amazonaws.com" or "http://minio.internal:9000".
	Endpoint string
	// Region scopes the request signature. Required, because it is signed
	// over, and a store that does not care about regions still has to be told
	// which one to expect.
	Region string
	Bucket string
	// Prefix is the key prefix inside the bucket. Empty means DefaultPrefix.
	Prefix string

	AccessKeyID     string
	SecretAccessKey string
	// SessionToken accompanies temporary credentials, which is what an
	// OIDC-assumed role in CI produces.
	SessionToken string

	// PathStyle selects bucket-in-path over bucket-in-host addressing. Nil
	// means "decide from the endpoint": Amazon expects bucket-in-host, and
	// essentially every other implementation needs bucket-in-path.
	PathStyle *bool

	// Timeout bounds one request. Zero means s3.DefaultTimeout.
	Timeout time.Duration

	// ReadOnly reads the shared cache and never writes to it: what a fork's
	// pull-request build should use, gaining the trunk-filled cache without
	// being able to put anything into a cache others trust.
	ReadOnly bool

	// Report receives every degradation. Optional: the report also goes to
	// ReportWriter regardless, so a caller that wires nothing still finds out.
	Report func(Degradation)

	// ReportWriter is where the human-readable degradation line goes. Zero
	// means os.Stderr: a run with no attached client and no configured sink
	// must still be able to say its cache went away.
	ReportWriter io.Writer

	// Transport is the HTTP round tripper requests go through. Zero means
	// the client's own. Exists for tests, for failures a real store will
	// not produce on demand (an upload refused while reads still work); it
	// wraps the suite's real store rather than replacing it.
	Transport http.RoundTripper
}

Config is everything needed to open one remote cache.

type Degradation 

type Degradation struct {
	// Store names the remote without naming its credentials, e.g.
	// "s3 bucket team-cache at s3.eu-west-1.amazonaws.com".
	Store string
	// Op is what was being attempted: "get", "put", "head", "lookup", "save".
	Op string
	// Err is why it failed, with any credential scrubbed out of it.
	Err error
	// Disabled reports whether the remote was switched off for the rest of
	// this run. False means one object was bad and the store is still in
	// use (one corrupt object says nothing about the rest); true means the
	// store itself stopped answering.
	Disabled bool
}

Degradation is one report that the remote cache did not do its job.

func (Degradation) String 

func (d Degradation) String() string

type Entries 

type Entries struct {
	// contains filtered or unexported fields
}

Entries is the shared action cache: the correctness-critical one, where a hit SKIPS THE STEP. Written once over docs: what to believe and what a failure means do not depend on the backend. The two layouts are: 

<prefix>action/entries/<aa>/<hex>       the entry, as JSON
<prefix>action/recent/<encoded step>    that step's latest key digest

senro-v1-action-sha256-<hex>            the entry, as JSON
senro-v1-recent-sha256-<hex>            that step's latest key digest

The entry document is byte-for-byte what the local backend writes, so all three are readable by the same code and diffable by a person.

func (*Entries) EntryKey 

func (e *Entries) EntryKey(k cas.Digest) string

EntryKey is the name the entry for a key digest is stored under: a key in a bucket, a tag in a repository.

func (*Entries) Forget 

func (e *Entries) Forget(context.Context, cache.Key) error

Forget does nothing to the shared cache, deliberately. A hit referencing missing content is a statement about the machine that noticed, not the shared cache; one pruned machine would otherwise delete an entry every other machine can still reproduce. The local half does forget, so this machine misses cleanly and re-saves. It also lets the cache run on credentials with no delete permission, the only policy worth having on a registry: pull and push are all senro asks for.

func (*Entries) Lookup 

func (e *Entries) Lookup(ctx context.Context, _ string, k cache.Key) (*cache.Result, bool, error)

Lookup returns the stored result for k, if the shared cache holds one.

A miss, an unreadable entry and a store that could not answer are all (nil, false, nil): an error return would let a cache problem fail a build, which is backwards (the local Lookup rules the same way). The degrader is what makes "could not answer" heard.

The entry is checked against the key it was asked for before it is believed: the most safety-critical check in the package, since a hit SKIPS THE STEP. An entry served under the wrong key produces a build that quietly did not do what it was told, on every machine sharing the cache.

func (*Entries) Previous 

func (e *Entries) Previous(ctx context.Context, step string) (*cache.Entry, bool, error)

Previous returns the most recent entry saved for a step, which is what `senro cache explain` diffs a miss against.

func (*Entries) RecentKey 

func (e *Entries) RecentKey(step string) string

RecentKey is the name a step's most recent key digest is recorded under.

func (*Entries) Save 

func (e *Entries) Save(ctx context.Context, step string, k cache.Key, r *cache.Result) error

Save writes the entry and records it as the step's most recent, in that order (matching the local backend): an entry with no pointer is still a valid hit, while a pointer with no entry is a dangling read Previous tolerates anyway.

Saving a key that already has an entry REPLACES it: the one mutable thing this half of the cache does, and the reason docs exists. Concurrent saves of one key both succeed and the later write stays; see ociDocs for the registry case.

type OCIConfig 

type OCIConfig struct {
	// Registry is the host and optional port: "ghcr.io",
	// "registry.internal:5000".
	Registry string
	// Repository is the path inside it that holds the cache, such as
	// "acme/senro-cache".
	Repository string

	// Username and Password are the credential presented to the registry's
	// token endpoint. See oci.Config for what senro does and does not do to
	// resolve one.
	Username string
	Password string

	// PlainHTTP talks to the registry over http rather than https, for a
	// registry on a trusted network that serves no certificate.
	PlainHTTP bool

	// Timeout bounds one request. Zero means oci.DefaultTimeout.
	Timeout time.Duration

	// ReadOnly reads the shared cache and never writes to it; see
	// Config.ReadOnly.
	ReadOnly bool

	// Report receives every degradation. Optional: the report also goes to
	// ReportWriter regardless, so a caller that wires nothing still finds out.
	Report func(Degradation)

	// ReportWriter is where the human-readable degradation line goes. Zero
	// means os.Stderr, for the same reason Config.ReportWriter does.
	ReportWriter io.Writer

	// Transport is the HTTP round tripper requests go through. Zero means the
	// client's own. It exists for the tests, and specifically for failures a
	// real registry will not produce on demand.
	Transport http.RoundTripper
}

OCIConfig is everything needed to open a shared object store held in an OCI registry. It exists beside Config rather than inside it because the two backends agree on nothing that could be shared: one struct holding both would be a struct where half the fields are always wrong.

type OCIObjects 

type OCIObjects struct {
	// contains filtered or unexported fields
}

OCIObjects is the content-addressed store held in a registry repository.

A registry addresses a blob by the digest of the bytes it holds; senro addresses an object by the digest of its PLAINTEXT while storing it compressed (internal/cas), so a blob cannot be found by asking directly. The bridge is a tag: each object is a tiny OCI artifact, tagged with the plaintext digest, whose single layer is the encoded blob: 

tag  senro-v1-sha256-<hex of the plaintext digest>
 └── manifest (application/vnd.oci.image.manifest.v1+json)
      └── layer  the object, in the encoding cas.NewEncoder produces

Storing plaintext instead would remove the manifest, and was rejected: it would send workspace snapshots uncompressed over the uplink and force every upload to decode a multi-gigabyte object. The manifest also keeps the blob referenced: a registry's garbage collector deletes loose blobs, and a retention policy needs something to act on.

func (*OCIObjects) Get 

func (o *OCIObjects) Get(ctx context.Context, d cas.Digest) (io.ReadCloser, error)

Get returns the object's plaintext, verified: the returned reader fails with cas.ErrCorrupt if what it decoded is not what d promised. A registry only verifies that a blob matches its own digest, which says nothing about whether the manifest pointing at it names the right one.

func (*OCIObjects) Has 

func (o *OCIObjects) Has(ctx context.Context, d cas.Digest) (bool, error)

Has reports whether the object is in the registry. One request, on the manifest: asking about the blob would mean knowing the encoded digest, which is what the manifest exists to record. Like the local Has, it cannot detect corruption; Has saying yes then Get saying ErrCorrupt is a legitimate sequence.

func (*OCIObjects) Name 

func (o *OCIObjects) Name(d cas.Digest) string

Name is the tag this object is stored under, which for a registry is OCITag. See RemoteObjects.Name.

func (*OCIObjects) Put 

func (o *OCIObjects) Put(ctx context.Context, r io.Reader) (cas.Digest, error)

Put stores everything r yields and returns its digest.

The body is spooled to a temp file: a blob is uploaded under a digest known before the bytes are sent, and a workspace tarball does not belong in memory. The tier above uploads straight from the local store and never comes through here; this path is for a caller using the registry on its own.

Concurrent Puts of the same content are safe: identical bytes to the same blob digest, then byte-identical manifests to the same tag.

func (*OCIObjects) UploadEncoded 

func (o *OCIObjects) UploadEncoded(ctx context.Context, d cas.Digest, path string) error

UploadEncoded stores a local object's file verbatim under its digest. path must be bytes in cas.NewEncoder's encoding; re-encoding would burn CPU to change nothing on the largest objects this cache moves.

The file is read once to derive the blob digest the registry requires up front (the one cost a registry has that a bucket does not) and once more to send it. Safe to read concurrently: content is immutable and completed files are renamed into place.

type Objects 

type Objects struct {
	// contains filtered or unexported fields
}

Objects is the content-addressed store held in the bucket.

Keys mirror the local directory backend's layout, hex fanout and all: <prefix>cas/sha256/<aa>/<bb>/<hex>. Object stores do not need the fanout, but a listing that reads the same on both sides beats two saved characters.

The stored bytes are exactly what the local backend writes (the cas.NewEncoder encoding), which lets the tier upload an object straight from disk without re-encoding a multi-gigabyte workspace.

func (*Objects) Get 

func (o *Objects) Get(ctx context.Context, d cas.Digest) (io.ReadCloser, error)

Get returns the object's plaintext, verified: the returned reader fails with cas.ErrCorrupt if what it decoded is not what d promised, through the same code the local backend uses.

func (*Objects) Has 

func (o *Objects) Has(ctx context.Context, d cas.Digest) (bool, error)

Has reports whether the object is in the bucket. Like the local Has, it cannot detect corruption (metadata only, no bytes); Has saying yes then Get saying ErrCorrupt is a legitimate sequence every caller survives.

func (*Objects) Name 

func (o *Objects) Name(d cas.Digest) string

Name is the key a digest is stored under in the bucket. It returns "" for a digest that is not well-formed: a digest reaches this package from event logs, plans and command-line arguments, and none of those are trusted.

func (*Objects) Put 

func (o *Objects) Put(ctx context.Context, r io.Reader) (cas.Digest, error)

Put stores everything r yields and returns its digest.

The body is spooled to a temp file: the request is signed over the exact bytes it will send, and a workspace tarball does not belong in memory. The tier above uploads straight from the local store and never comes through here; this path is for a caller using the remote on its own.

Concurrent Puts of the same content are safe: same key, same meaning, an atomic PUT, and the reader verifies whatever it gets.

func (*Objects) UploadEncoded 

func (o *Objects) UploadEncoded(ctx context.Context, d cas.Digest, path string) error

UploadEncoded stores a local object's file verbatim under its digest. path must be bytes in cas.NewEncoder's encoding (what the local backend writes): re-encoding would burn CPU to change nothing, on the largest objects this cache moves. Safe to read concurrently: content is immutable and the local backend renames completed files into place, so a path that exists is a complete object.

type Remote 

type Remote struct {
	// contains filtered or unexported fields
}

Remote is an opened shared cache, backed by a bucket or by a registry. One type for both: everything a run does with a shared cache is the same either way, and what differs is confined to RemoteObjects and docs. A second Remote type would mean a second engine branch and a second place for "a down cache never fails a run" to be got wrong.

func Open 

func Open(cfg Config) (*Remote, error)

Open validates the config and prepares the remote. No I/O: reachability is discovered on first use and answered by degrading. A configuration that cannot possibly work (no bucket, a non-URL endpoint) is an error here, at startup: an operator's mistake, not a network condition, and reporting it as "your cache is down" would mislead. See OpenOCI for the registry form.

func OpenOCI 

func OpenOCI(cfg OCIConfig) (*Remote, error)

OpenOCI validates the config and prepares a shared cache held in a registry repository. No I/O: reachability is discovered on first use and answered by degrading.

It returns the same *Remote that Open does, and that is the point: a registry is a place to keep the cache, not a different cache, so a run configured for one behaves identically to one configured for a bucket, including when it is down. A configuration that cannot possibly work is an error here, at startup, for the reason Open gives.

func (*Remote) Archive 

func (r *Remote) Archive(runID string) *Archiver

Archive starts an archiver for one run. It returns nil when there is no remote, so a caller writes no branch: every method below tolerates a nil receiver.

func (*Remote) Close 

func (r *Remote) Close() error

Close releases the connections the remote holds.

func (*Remote) Entries 

func (r *Remote) Entries() *Entries

Entries is the remote action cache, on its own. See Objects.

func (*Remote) Live 

func (r *Remote) Live() bool

Live reports whether the remote is still being used. False once it has degraded.

func (*Remote) Objects 

func (r *Remote) Objects() RemoteObjects

Objects is the shared content-addressed store, on its own. Callers in the engine want TierObjects instead; this is for tests and for a tool that deliberately wants to talk only to the shared store.

func (*Remote) Observe 

func (r *Remote) Observe(fn func(Degradation)) (stop func())

Observe redirects degradation reports to fn, in addition to Config.Report and the stderr line, and returns a function that stops doing so. Config.Report is wired before a run exists; the run's ledger exists only once the engine starts. Observe is how the engine subscribes for one run and unsubscribes after, so a Storage reused across runs never appends to a sealed ledger.

fn may be called from any goroutine, and never from inside a Sink's Emit.

func (*Remote) RunLogs 

func (r *Remote) RunLogs() *RunLogs

RunLogs is the archive of a run's ledger and step output. See its own doc for why archiving is per completed attempt rather than live.

func (*Remote) String 

func (r *Remote) String() string

String names the remote without naming its credentials.

func (*Remote) TierEntries 

func (r *Remote) TierEntries(local cache.ActionCache) *TieredEntries

TierEntries returns the action cache the engine should use: local first, remote behind it.

func (*Remote) TierObjects 

func (r *Remote) TierObjects(local *cas.Dir) *TieredObjects

TierObjects returns the store the engine should read and write objects through: local first, remote behind it.

type RemoteObjects 

type RemoteObjects interface {
	cas.Store
	UploadEncoded(ctx context.Context, d cas.Digest, path string) error
	// Name is where the object is kept: a key in a bucket, a tag in a
	// repository. It is what an error message quotes and what a test plants
	// bytes at, and it is "" for a digest that is not well-formed.
	Name(d cas.Digest) string
}

RemoteObjects is the remote half of an object tier, with two implementations: the bucket (Objects) and the registry (OCIObjects). What a remote failure means for a run is identical for both, so it is written once below.

UploadEncoded is on it rather than left to cas.Store because it is why the tier is fast: both remotes take the locally encoded file as it is instead of re-encoding a multi-gigabyte workspace.

type RunLogs 

type RunLogs struct {
	// contains filtered or unexported fields
}

RunLogs is a run's archived record in the shared store: its event ledger and the output of every step attempt.

Archival rather than live, deliberately: neither backend has append, so "live logs in S3" delivers no actual live reading (the attach server already serves that from the local file) while putting a network round trip, and so a shared-store outage, inside every log line's write path. Uploading each stream once it completes fully covers the case that motivates archiving: a CI runner destroyed when the job ends, including a run that crashed halfway.

The bytes go into the same content-addressed store the cache uses, and a small mutable pointer names the digest. In a bucket: 

<prefix>cas/sha256/<aa>/<bb>/<hex>                        the bytes
<prefix>runs/<run>/logs/<step>/<attempt>/<stream>         the digest
<prefix>runs/<run>/events                                 the ledger's digest

and in a registry, where every name is a tag and a tag has an alphabet: 

senro-v1-sha256-<hex>                                     the bytes
senro-v1-log-sha256-<hex>                                 the digest
senro-v1-run-sha256-<hex>                                 the ledger's digest

Content addressing makes the read path safe (a fetched log is verified by the same code that verifies a cached object), stores identical logs once, and lets two uploaders never conflict. The cost: a stream is fetched whole, not by byte range, the right trade for files of kilobytes to a few megabytes.

func (*RunLogs) Fetch 

func (r *RunLogs) Fetch(ctx context.Context, runID, dir string, streams []StreamRef) error

Fetch materializes an archived run back into dir, in the layout the run wrote it in: senro's readers of a finished run are all built on a directory of files, so restoring the directory makes every one of them work on an archived run with no second implementation. Every byte is verified against its digest on the way in.

The stream set comes from the ledger, never a bucket listing (see StreamsFromLedger); a caller that must read the ledger first to decide uses FetchLedger and FetchStreams directly.

func (*RunLogs) FetchLedger 

func (r *RunLogs) FetchLedger(ctx context.Context, runID, dir string) error

FetchLedger materializes a run's event ledger, and nothing else, into dir: Fetch's first half, exported because the ledger is what NAMES the streams, so "fetch the ledger, decide, fetch the rest" is the only possible order, and going through Fetch would download the ledger twice.

func (*RunLogs) FetchStreams 

func (r *RunLogs) FetchStreams(
	ctx context.Context, runID, dir string, streams []StreamRef,
) (missing []StreamRef, err error)

FetchStreams materializes the named streams into dir and reports the ones the archive does not hold. A missing stream is not an error (an unfinished upload, an expired object; the rest of the run is still worth having), but it is returned rather than silently skipped: the caller is the one who can say the log somebody came for is the absent one.

func (*RunLogs) Get 

func (r *RunLogs) Get(ctx context.Context, key string) (io.ReadCloser, error)

Get returns the archived bytes the pointer at key names, verified. A missing pointer, or one naming an object no longer there, is cas.ErrNotFound: a log expired by a lifecycle rule is absent, not broken.

func (*RunLogs) LedgerKey 

func (r *RunLogs) LedgerKey(runID string) string

LedgerKey is the name of the pointer naming a run's event ledger.

func (*RunLogs) PutFile 

func (r *RunLogs) PutFile(ctx context.Context, key, path string) error

PutFile uploads path's bytes and writes the pointer at key: bytes first, pointer second (the action cache's order, for the same reason: a pointer without content is a promise nothing can keep). A file that does not exist is not an error: a step that wrote nothing to stderr has no stderr file.

func (*RunLogs) StreamKey 

func (r *RunLogs) StreamKey(runID, step string, attempt int, stream string) string

StreamKey is the name of the pointer naming one stream of one attempt. In a bucket the step id is percent-encoded into a single path segment (ids contain slashes and brackets); in a registry the four parts are hashed into one tag, which holds neither those characters nor that length. See ociDocs.

type StreamRef 

type StreamRef struct {
	Step    string
	Attempt int
	Stream  string
}

StreamRef names one archived log stream.

func StreamsFromLedger 

func StreamsFromLedger(path string) ([]StreamRef, error)

StreamsFromLedger returns every archived stream the ledger at path names: the streams come from the run's own record, never a bucket listing, so reading an archive needs no permission beyond GetObject.

Two kinds of event name a stream, and both are needed. step.log.appended names one that actually produced output (live, retried, or replayed from a cache hit). A handler emits NO markers, so handler.started is the only thing that can name its output; both streams are claimed for it since the ledger does not say which one it wrote, and the silent one simply comes back from FetchStreams as missing, at the cost of one refused GET.

A torn final line is tolerated: the killed run is exactly what this feature exists for.

type TieredEntries 

type TieredEntries struct {
	// contains filtered or unexported fields
}

TieredEntries is the local action cache with the remote one behind it.

func (*TieredEntries) Forget 

func (t *TieredEntries) Forget(ctx context.Context, k cache.Key) error

Forget removes the entry from this machine only. See Entries.Forget for why the shared copy is left alone.

func (*TieredEntries) Lookup 

func (t *TieredEntries) Lookup(
	ctx context.Context, step string, k cache.Key,
) (*cache.Result, bool, error)

Lookup consults this machine first and the bucket second. A remote hit is written through to the local cache, best effort: failing to warm the local cache is no reason to discard a hit already in hand.

func (*TieredEntries) Previous 

func (t *TieredEntries) Previous(ctx context.Context, step string) (*cache.Entry, bool, error)

Previous returns this step's most recent entry, preferring this machine's own history: `senro cache explain` is answering "what changed since I last built this", and the local answer is the one the person asking means.

func (*TieredEntries) Save 

func (t *TieredEntries) Save(ctx context.Context, step string, k cache.Key, r *cache.Result) error

Save writes the entry locally and then shares it. The local write must succeed; failing to publish is only a degradation, since a build failed by an unreachable cache is worse than a build nobody else can reuse.

type TieredObjects 

type TieredObjects struct {
	// contains filtered or unexported fields
}

TieredObjects is the local content-addressed store with a remote one behind it: the type that enforces the rule the feature rests on. Every method treats a remote failure as "there is no remote cache", records it through the degrader, and returns what the local store alone would have.

func (*TieredObjects) Get 

func (t *TieredObjects) Get(ctx context.Context, d cas.Digest) (io.ReadCloser, error)

Get returns the object, from disk if it is there and from the bucket if not. A remote fetch is written through the local store, which is also what verifies it: bytes that are not what the digest promised fail before anything lands on disk.

func (*TieredObjects) Has 

func (t *TieredObjects) Has(ctx context.Context, d cas.Digest) (bool, error)

Has reports whether either tier holds the object. A remote that cannot answer reports false, not an error: the engine calls this to decide whether a hit can be reproduced, and an error would fail the step. False means "run the step", which is always safe.

func (*TieredObjects) Local 

func (t *TieredObjects) Local() *cas.Dir

Local is the store on this machine's disk.

func (*TieredObjects) Put 

func (t *TieredObjects) Put(ctx context.Context, r io.Reader) (cas.Digest, error)

Put stores the object locally and then uploads it. Local first, and the local result is what the caller gets; a failed upload is a degradation, never an error, so a step's output is never lost to an unreachable cache.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.