db

package
v0.0.0-...-03d6fc4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 23, 2019 License: BSD-3-Clause Imports: 21 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// Maximum number of simultaneous GetModifiedTasks users.
	MAX_MODIFIED_DATA_USERS = 20

	// Expiration for GetModifiedTasks users.
	MODIFIED_DATA_TIMEOUT = 30 * time.Minute

	// Retries attempted by Update*WithRetries.
	NUM_RETRIES = 5
)
View Source 
const (
	TS_RESOLUTION = time.Microsecond
)

Variables

View Source 
var (
	ErrAlreadyExists    = errors.New("Object already exists and modification not allowed.")
	ErrConcurrentUpdate = errors.New("Concurrent update")
	ErrNotFound         = errors.New("Task/Job with given ID does not exist")
	ErrTooManyUsers     = errors.New("Too many users")
	ErrUnknownId        = types.ErrUnknownId
)
View Source 
var AssertDeepEqual func(t testutils.TestingT, expected, actual interface{})

AssertDeepEqual does a deep equals comparison using the testutils.TestingT interface.

Callers of these tests utils should assign a value to AssertDeepEqual beforehand, e.g.: 

AssertDeepEqual = deepequal.AssertDeepEqual

This is necessary to break the hard linking of this file to the "testing" module.

Functions

func GetTasksFromWindow 

func GetTasksFromWindow(db TaskReader, w *window.Window, now time.Time) ([]*types.Task, error)

GetTasksFromWindow returns all tasks matching the given Window from the TaskReader.

func IsAlreadyExists 

func IsAlreadyExists(e error) bool

func IsConcurrentUpdate 

func IsConcurrentUpdate(e error) bool

func IsNotFound 

func IsNotFound(e error) bool

func IsTooManyUsers 

func IsTooManyUsers(e error) bool

func IsUnknownId 

func IsUnknownId(e error) bool

func SearchJobs 

func SearchJobs(db JobReader, p *JobSearchParams) ([]*types.Job, error)

SearchJobs returns Jobs in the given time range which match the given search parameters.

func SearchTasks 

func SearchTasks(db TaskReader, p *TaskSearchParams) ([]*types.Task, error)

SearchTasks returns Tasks in the given time range which match the given search parameters.

func TestCommentDB 

func TestCommentDB(t testutils.TestingT, db CommentDB)

TestCommentDB validates that db correctly implements the CommentDB interface.

func TestJobDB 

func TestJobDB(t testutils.TestingT, db JobDB)

TestJobDB performs basic tests on an implementation of JobDB.

func TestJobDBConcurrentUpdate 

func TestJobDBConcurrentUpdate(t testutils.TestingT, db JobDB)

Test that PutJob and PutJobs return ErrConcurrentUpdate when a cached Job has been updated in the DB.

func TestJobDBTooManyUsers 

func TestJobDBTooManyUsers(t testutils.TestingT, db JobDB)

Test that a JobDB properly tracks its maximum number of users.

func TestModifiedComments 

func TestModifiedComments(t testutils.TestingT, m ModifiedComments)

func TestModifiedJobs 

func TestModifiedJobs(t testutils.TestingT, m ModifiedJobs)

func TestModifiedTasks 

func TestModifiedTasks(t testutils.TestingT, m ModifiedTasks)

func TestMultipleCommentModifications 

func TestMultipleCommentModifications(t testutils.TestingT, m ModifiedComments)

func TestMultipleJobModifications 

func TestMultipleJobModifications(t testutils.TestingT, m ModifiedJobs)

func TestMultipleTaskModifications 

func TestMultipleTaskModifications(t testutils.TestingT, m ModifiedTasks)

Test that if a Task is modified multiple times, it only appears once in the result of GetModifiedTasks.

func TestTaskDB 

func TestTaskDB(t testutils.TestingT, db TaskDB)

TestTaskDB performs basic tests for an implementation of TaskDB.

func TestTaskDBConcurrentUpdate 

func TestTaskDBConcurrentUpdate(t testutils.TestingT, db TaskDB)

Test that PutTask and PutTasks return ErrConcurrentUpdate when a cached Task has been updated in the DB.

func TestTaskDBGetTasksFromDateRangeByRepo 

func TestTaskDBGetTasksFromDateRangeByRepo(t testutils.TestingT, db TaskDB)

func TestTaskDBGetTasksFromWindow 

func TestTaskDBGetTasksFromWindow(t testutils.TestingT, db TaskDB)

func TestUpdateDBFromSwarmingTask 

func TestUpdateDBFromSwarmingTask(t testutils.TestingT, db TaskDB)

func TestUpdateDBFromSwarmingTaskTryJob 

func TestUpdateDBFromSwarmingTaskTryJob(t testutils.TestingT, db TaskDB)

func TestUpdateJobsWithRetries 

func TestUpdateJobsWithRetries(t testutils.TestingT, db JobDB)

Test UpdateJobsWithRetries and UpdateJobWithRetries.

func TestUpdateTasksWithRetries 

func TestUpdateTasksWithRetries(t testutils.TestingT, db TaskDB)

Test UpdateTasksWithRetries and UpdateTaskWithRetries.

func UpdateDBFromSwarmingTask 

func UpdateDBFromSwarmingTask(db TaskDB, s *swarming_api.SwarmingRpcsTaskResult) error

UpdateDBFromSwarmingTask updates a task in db from data in s.

func UpdateJobWithRetries 

func UpdateJobWithRetries(db JobDB, id string, f func(*types.Job) error) (*types.Job, error)

UpdateJobWithRetries reads, updates, and writes a single Job in the DB. It:

  1. reads the job with the given id,
  2. calls f on that job, and
  3. calls db.PutJob() on the updated job
  4. repeats from step 1 as long as PutJobs returns ErrConcurrentUpdate and retries have not been exhausted.

Returns the updated job if it was successfully updated in the DB. Immediately returns ErrNotFound if db.GetJobById(id) returns nil. Immediately returns any error returned from f or from PutJobs (except ErrConcurrentUpdate). Returns ErrConcurrentUpdate if retries are exhausted. TODO(borenet): We probably don't need this; consider removing.

func UpdateJobsWithRetries 

func UpdateJobsWithRetries(db JobDB, f func() ([]*types.Job, error)) ([]*types.Job, error)

UpdateJobsWithRetries wraps a call to db.PutJobs with retries. It calls db.PutJobs(f()) repeatedly until one of the following happen:

  • f or db.PutJobs returns an error, which is then returned from UpdateJobsWithRetries;
  • PutJobs succeeds, in which case UpdateJobsWithRetries returns the updated Jobs returned by f;
  • retries are exhausted, in which case UpdateJobsWithRetries returns ErrConcurrentUpdate.

Within f, jobs should be refreshed from the DB, e.g. with db.GetModifiedJobs or db.GetJobById. TODO(borenet): We probably don't need this; consider removing.

func UpdateTaskWithRetries 

func UpdateTaskWithRetries(db TaskDB, id string, f func(*types.Task) error) (*types.Task, error)

UpdateTaskWithRetries reads, updates, and writes a single Task in the DB. It:

  1. reads the task with the given id,
  2. calls f on that task, and
  3. calls db.PutTask() on the updated task
  4. repeats from step 1 as long as PutTasks returns ErrConcurrentUpdate and retries have not been exhausted.

Returns the updated task if it was successfully updated in the DB. Immediately returns ErrNotFound if db.GetTaskById(id) returns nil. Immediately returns any error returned from f or from PutTasks (except ErrConcurrentUpdate). Returns ErrConcurrentUpdate if retries are exhausted.

func UpdateTasksWithRetries 

func UpdateTasksWithRetries(db TaskDB, f func() ([]*types.Task, error)) ([]*types.Task, error)

UpdateTasksWithRetries wraps a call to db.PutTasks with retries. It calls db.PutTasks(f()) repeatedly until one of the following happen:

  • f or db.PutTasks returns an error, which is then returned from UpdateTasksWithRetries;
  • PutTasks succeeds, in which case UpdateTasksWithRetries returns the updated Tasks returned by f;
  • retries are exhausted, in which case UpdateTasksWithRetries returns ErrConcurrentUpdate.

Within f, tasks should be refreshed from the DB, e.g. with db.GetModifiedTasks or db.GetTaskById.

Types

type BackupDBCloser 

type BackupDBCloser interface {
	DBCloser

	// WriteBackup writes a backup of the DB to the given io.Writer.
	WriteBackup(io.Writer) error

	// SetIncrementalBackupTime marks the given time as a checkpoint for
	// incremental backups.
	SetIncrementalBackupTime(time.Time) error
	// GetIncrementalBackupTime returns the most recent time provided to
	// SetIncrementalBackupTime. Any incremental backups taken after the returned
	// time should be reapplied to the DB.
	GetIncrementalBackupTime() (time.Time, error)
}

BackupDBCloser is a DBCloser that provides backups.

type CommentDB 

type CommentDB interface {
	ModifiedComments

	// GetComments returns all comments for the given repos.
	//
	// If from is specified, it is a hint that TaskComments and CommitComments
	// before this time will be ignored by the caller, thus they may be ommitted.
	GetCommentsForRepos(repos []string, from time.Time) ([]*types.RepoComments, error)

	// PutTaskComment inserts the TaskComment into the database. May return
	// ErrAlreadyExists.
	PutTaskComment(*types.TaskComment) error

	// DeleteTaskComment deletes the matching TaskComment from the database.
	// Non-ID fields of the argument are ignored.
	DeleteTaskComment(*types.TaskComment) error

	// PutTaskSpecComment inserts the TaskSpecComment into the database. May
	// return ErrAlreadyExists.
	PutTaskSpecComment(*types.TaskSpecComment) error

	// DeleteTaskSpecComment deletes the matching TaskSpecComment from the
	// database. Non-ID fields of the argument are ignored.
	DeleteTaskSpecComment(*types.TaskSpecComment) error

	// PutCommitComment inserts the CommitComment into the database. May return
	// ErrAlreadyExists.
	PutCommitComment(*types.CommitComment) error

	// DeleteCommitComment deletes the matching CommitComment from the database.
	// Non-ID fields of the argument are ignored.
	DeleteCommitComment(*types.CommitComment) error
}

CommentDB stores comments on Tasks, TaskSpecs, and commits.

Clients must be tolerant of comments that refer to nonexistent Tasks, TaskSpecs, or commits.

type DB 

type DB interface {
	TaskDB
	JobDB
	CommentDB
}

DB implements TaskDB, JobDB, and CommentDB.

func NewDB 

func NewDB(tdb TaskDB, jdb JobDB, cdb CommentDB) DB

NewDB returns a DB that delegates to independent TaskDB, JobDB, and CommentDB.

type DBCloser 

type DBCloser interface {
	io.Closer
	DB
}

DBCloser is a DB that must be closed when no longer in use.

type GetRevisionTimestamp 

type GetRevisionTimestamp func(repo, revision string) (time.Time, error)

GetRevisionTimestamp is a function signature that retrieves the timestamp of a revision. NewJobCache accepts this type rather than repograph.Map to aide testing.

func DummyGetRevisionTimestamp 

func DummyGetRevisionTimestamp(ts time.Time) GetRevisionTimestamp

type JobDB 

type JobDB interface {
	JobReader

	// PutJob inserts or updates the Job in the database. Job's Id field
	// must be empty if it is a new Job. PutJob will set Job.DbModified.
	PutJob(*types.Job) error

	// PutJobs inserts or updates the Jobs in the database. Each Jobs' Id
	// field must be empty if it is a new Job. Each Jobs' DbModified field
	// will be set.
	PutJobs([]*types.Job) error
}

JobDB is used by the task scheduler to store Jobs.

type JobReader 

type JobReader interface {
	ModifiedJobsReader

	// GetJobById returns the job with the given Id field. Returns nil, nil if
	// job is not found.
	GetJobById(string) (*types.Job, error)

	// GetJobsFromDateRange retrieves all jobs with Created in the given range.
	// The returned jobs are sorted by Created timestamp.
	GetJobsFromDateRange(time.Time, time.Time) ([]*types.Job, error)
}

JobReader is a read-only view of a JobDB.

type JobSearchParams 

type JobSearchParams struct {
	types.RepoState
	BuildbucketBuildId *int64          `json:"buildbucket_build_id,string,omitempty"`
	IsForce            *bool           `json:"is_force,omitempty"`
	Name               string          `json:"name"`
	Status             types.JobStatus `json:"status"`
	TimeStart          time.Time       `json:"time_start"`
	TimeEnd            time.Time       `json:"time_end"`
}

JobSearchParams are parameters on which Jobs may be searched. All fields are optional; if a field is not provided, the search will return Jobs with any value for that field. If either of TimeStart or TimeEnd is not provided, the search defaults to the last 24 hours.

type ModifiedComments 

type ModifiedComments interface {
	// GetModifiedComments returns all comments added or deleted since the
	// last time GetModifiedComments was run with the given id. The returned
	// comments are sorted by timestamp. If GetModifiedComments returns an
	// error, the caller should call StopTrackingModifiedComments and
	// StartTrackingModifiedComments again, and load all data from scratch
	// to be sure that no comments were missed.
	GetModifiedComments(string) ([]*types.TaskComment, []*types.TaskSpecComment, []*types.CommitComment, error)

	// StartTrackingModifiedComments initiates tracking of modified comments
	// for the current caller. Returns a unique ID which can be used by the
	// caller to retrieve comments which have been added or deleted since
	// the last query. The ID expires after a period of inactivity.
	StartTrackingModifiedComments() (string, error)

	// StopTrackingModifiedComments cancels tracking of modified comments
	// for the provided ID.
	StopTrackingModifiedComments(string)

	// TrackModifiedTaskComment indicates the given comment should be
	// returned from the next call to GetModifiedComments from each
	// subscriber.
	TrackModifiedTaskComment(*types.TaskComment)

	// TrackModifiedTaskSpecComment indicates the given comment should be
	// returned from the next call to GetModifiedComments from each
	// subscriber.
	TrackModifiedTaskSpecComment(*types.TaskSpecComment)

	// TrackModifiedCommitComment indicates the given comment should be
	// returned from the next call to GetModifiedComments from each
	// subscriber.
	TrackModifiedCommitComment(*types.CommitComment)
}

ModifiedCommentsReader tracks which comments have been added or deleted and returns results to subscribers based on what has changed since the last call to GetModifiedComments.

type ModifiedData 

type ModifiedData interface {
	ModifiedTasks
	ModifiedJobs
	ModifiedComments
}

ModifiedData combines ModifiedTasks, ModifiedJobs, and ModifiedComments.

func NewModifiedData 

func NewModifiedData(t ModifiedTasks, j ModifiedJobs, c ModifiedComments) ModifiedData

NewModifiedData returns a ModifiedData which combines the given ModifiedTasks, ModifiedJobs, and ModifiedComments.

type ModifiedJobs 

type ModifiedJobs interface {
	ModifiedJobsReader

	// TrackModifiedJob indicates the given Job should be returned from the next
	// call to GetModifiedJobs from each subscriber.
	TrackModifiedJob(*types.Job)

	// TrackModifiedJobsGOB is a batch, GOB version of TrackModifiedJob. Given a
	// map from Job.Id to GOB-encoded task, it is equivalent to GOB-decoding each
	// value of gobs as a Job and calling TrackModifiedJob on each one. Values of
	// gobs must not be modified after this call. The time parameter is the
	// DbModified timestamp of the jobs.
	TrackModifiedJobsGOB(time.Time, map[string][]byte)
}

ModifiedJobs tracks which tasks have been modified and returns results to subscribers based on what has changed since the last call to GetModifiedJobs.

type ModifiedJobsReader 

type ModifiedJobsReader interface {
	// GetModifiedJobs returns all jobs modified since the last time
	// GetModifiedJobs was run with the given id. The returned jobs are sorted by
	// Created timestamp. If GetModifiedJobs returns an error, the caller
	// should call StopTrackingModifiedJobs and StartTrackingModifiedJobs
	// again, and load all data from scratch to be sure that no jobs were
	// missed.
	GetModifiedJobs(string) ([]*types.Job, error)

	// GetModifiedJobsGOB returns the GOB-encoded results of GetModifiedJobs,
	// keyed by Job.Id. Callers should not modify the returned byte slices.
	GetModifiedJobsGOB(string) (map[string][]byte, error)

	// StartTrackingModifiedJobs initiates tracking of modified jobs for
	// the current caller. Returns a unique ID which can be used by the caller
	// to retrieve jobs which have been modified since the last query. The ID
	// expires after a period of inactivity.
	StartTrackingModifiedJobs() (string, error)

	// StopTrackingModifiedJobs cancels tracking of modified jobs for the
	// provided ID.
	StopTrackingModifiedJobs(string)
}

ModifiedJobsReader tracks which tasks have been modified and returns results to subscribers based on what has changed since the last call to GetModifiedJobs.

type ModifiedTasks 

type ModifiedTasks interface {
	ModifiedTasksReader

	// TrackModifiedTask indicates the given Task should be returned from the next
	// call to GetModifiedTasks from each subscriber.
	TrackModifiedTask(*types.Task)

	// TrackModifiedTasksGOB is a batch, GOB version of TrackModifiedTask. Given a
	// map from Task.Id to GOB-encoded task, it is equivalent to GOB-decoding each
	// value of gobs as a Task and calling TrackModifiedTask on each one. Values of
	// gobs must not be modified after this call. The time parameter is the
	// DbModified timestamp of the tasks.
	TrackModifiedTasksGOB(time.Time, map[string][]byte)
}

ModifiedTasks tracks which tasks have been modified and returns results to subscribers based on what has changed since the last call to GetModifiedTasks.

type ModifiedTasksReader 

type ModifiedTasksReader interface {
	// GetModifiedTasks returns all tasks modified since the last time
	// GetModifiedTasks was run with the given id. The returned tasks are sorted
	// by Created timestamp. If GetModifiedTasks returns an error, the caller
	// should call StopTrackingModifiedTasks and StartTrackingModifiedTasks
	// again, and load all data from scratch to be sure that no tasks were
	// missed.
	GetModifiedTasks(string) ([]*types.Task, error)

	// GetModifiedTasksGOB returns the GOB-encoded results of GetModifiedTasks,
	// keyed by Task.Id. Callers should not modify the returned byte slices.
	GetModifiedTasksGOB(string) (map[string][]byte, error)

	// StartTrackingModifiedTasks initiates tracking of modified tasks for
	// the current caller. Returns a unique ID which can be used by the caller
	// to retrieve tasks which have been modified since the last query. The ID
	// expires after a period of inactivity.
	StartTrackingModifiedTasks() (string, error)

	// StopTrackingModifiedTasks cancels tracking of modified tasks for the
	// provided ID.
	StopTrackingModifiedTasks(string)
}

ModifiedTasksReader tracks which tasks have been modified and returns results to subscribers based on what has changed since the last call to GetModifiedTasks.

type RemoteDB 

type RemoteDB interface {
	TaskReader
	JobReader
	CommentDB
}

RemoteDB allows retrieving tasks and jobs and full access to comments.

type TaskDB 

type TaskDB interface {
	TaskReader

	// AssignId sets the given task's Id field. Does not insert the task into the
	// database.
	AssignId(*types.Task) error

	// PutTask inserts or updates the Task in the database. Task's Id field must
	// be empty or set with AssignId. PutTask will set Task.DbModified.
	PutTask(*types.Task) error

	// PutTasks inserts or updates the Tasks in the database. Each Task's Id field
	// must be empty or set with AssignId. Each Task's DbModified field will be
	// set.
	PutTasks([]*types.Task) error
}

TaskDB is used by the task scheduler to store Tasks.

type TaskReader 

type TaskReader interface {
	ModifiedTasksReader

	// GetTaskById returns the task with the given Id field. Returns nil, nil if
	// task is not found.
	GetTaskById(string) (*types.Task, error)

	// GetTasksFromDateRange retrieves all tasks with Created in the given range.
	// The returned tasks are sorted by Created timestamp. The string field is
	// an optional repository; if provided, only return tasks associated with
	// that repo.
	GetTasksFromDateRange(time.Time, time.Time, string) ([]*types.Task, error)
}

TaskReader is a read-only view of a TaskDB.

type TaskSearchParams 

type TaskSearchParams struct {
	Attempt *int64           `json:"attempt,string,omitempty"`
	Status  types.TaskStatus `json:"status"`
	types.TaskKey
	TimeStart time.Time `json:"time_start"`
	TimeEnd   time.Time `json:"time_end"`
}

TaskSearchParams are parameters on which Tasks may be searched. All fields are optional; if a field is not provided, the search will return Tasks with any value for that field. If either of TimeStart or TimeEnd is not provided, the search defaults to the last 24 hours.

Source Files

View all Source files

Directories

Path Synopsis
busywork
busywork is an end-to-end test for local_db.
 busywork is an end-to-end test for local_db.
ts_local_db_viewer
Read data from a local task scheduler DB file and output as JSON.
 Read data from a local task scheduler DB file and output as JSON.
Implementation of backing up a DB to Google Cloud Storage (GCS).
 Implementation of backing up a DB to Google Cloud Storage (GCS).
gs_jobs_viewer
Read Job GOBs from GCS and write as JSON.
 Read Job GOBs from GCS and write as JSON.
remote_db provides a client/server pair for accessing a db.RemoteDB over HTTP.
 remote_db provides a client/server pair for accessing a db.RemoteDB over HTTP.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.