store

package
Version: v0.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: May 6, 2020 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright (c) 2016-2019 Uber Technologies, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RunDownload

func RunDownload(cads *CADownloadStore, d core.Digest, content []byte) error

RunDownload downloads content to cads.

Types

type CADownloadStore

type CADownloadStore struct {
	// contains filtered or unexported fields
}

CADownloadStore allows simultaneously downloading and uploading content-adddressable files.

func CADownloadStoreFixture

func CADownloadStoreFixture() (*CADownloadStore, func())

CADownloadStoreFixture returns a CADownloadStore for testing purposes.

func NewCADownloadStore

func NewCADownloadStore(config CADownloadStoreConfig, stats tally.Scope) (*CADownloadStore, error)

NewCADownloadStore creates a new CADownloadStore.

func (*CADownloadStore) Any

Any scopes the store to files in any state.

func (*CADownloadStore) Cache

Cache scopes the store to files in the cache state.

func (*CADownloadStore) Close

func (s *CADownloadStore) Close()

Close terminates all goroutines started by s.

func (*CADownloadStore) CreateDownloadFile

func (s *CADownloadStore) CreateDownloadFile(name string, length int64) error

CreateDownloadFile creates an empty download file initialized with length.

func (*CADownloadStore) Download

func (s *CADownloadStore) Download() *CADownloadStoreScope

Download scopes the store to files in the download state.

func (*CADownloadStore) GetCacheFileReader

func (s *CADownloadStore) GetCacheFileReader(name string) (FileReader, error)

GetCacheFileReader gets a cache file reader. Implemented for compatibility with other stores.

func (*CADownloadStore) GetCacheFileStat

func (s *CADownloadStore) GetCacheFileStat(name string) (os.FileInfo, error)

GetCacheFileStat stats a cache file. Implemented for compatibility with other stores.

func (*CADownloadStore) GetDownloadFileReadWriter

func (s *CADownloadStore) GetDownloadFileReadWriter(name string) (FileReadWriter, error)

GetDownloadFileReadWriter returns a FileReadWriter for name.

func (*CADownloadStore) InCacheError

func (s *CADownloadStore) InCacheError(err error) bool

InCacheError returns true for errors originating from file store operations which do not accept files in cache state.

func (*CADownloadStore) InDownloadError

func (s *CADownloadStore) InDownloadError(err error) bool

InDownloadError returns true for errors originating from file store operations which do not accept files in download state.

func (*CADownloadStore) MoveDownloadFileToCache

func (s *CADownloadStore) MoveDownloadFileToCache(name string) error

MoveDownloadFileToCache moves a download file to the cache.

type CADownloadStoreConfig

type CADownloadStoreConfig struct {
	DownloadDir     string        `yaml:"download_dir"`
	CacheDir        string        `yaml:"cache_dir"`
	DownloadCleanup CleanupConfig `yaml:"download_cleanup"`
	CacheCleanup    CleanupConfig `yaml:"cache_cleanup"`
}

CADownloadStoreConfig defines CADownloadStore configuration. TODO(evelynl94): rename

type CADownloadStoreScope

type CADownloadStoreScope struct {
	// contains filtered or unexported fields
}

CADownloadStoreScope scopes what states an operation may be accepted within. Should only be used for read / write operations which are acceptable in any state.

func (*CADownloadStoreScope) DeleteFile

func (a *CADownloadStoreScope) DeleteFile(name string) error

DeleteFile deletes name.

func (*CADownloadStoreScope) GetFileReader

func (a *CADownloadStoreScope) GetFileReader(name string) (FileReader, error)

GetFileReader returns a reader for name.

func (*CADownloadStoreScope) GetFileStat

func (a *CADownloadStoreScope) GetFileStat(name string) (os.FileInfo, error)

GetFileStat returns file info for name.

func (*CADownloadStoreScope) GetMetadata

func (a *CADownloadStoreScope) GetMetadata(name string, md metadata.Metadata) error

GetMetadata returns the metadata content of md for name.

func (*CADownloadStoreScope) GetOrSetMetadata

func (a *CADownloadStoreScope) GetOrSetMetadata(name string, md metadata.Metadata) error

GetOrSetMetadata returns the metadata content of md for name, or initializes the metadata content to b if not set.

func (*CADownloadStoreScope) SetMetadata

func (a *CADownloadStoreScope) SetMetadata(
	name string, md metadata.Metadata) (updated bool, err error)

SetMetadata writes b to metadata content of md for name.

func (*CADownloadStoreScope) SetMetadataAt

func (a *CADownloadStoreScope) SetMetadataAt(
	name string, md metadata.Metadata, b []byte, offset int64) (updated bool, err error)

SetMetadataAt writes b to metadata content of md starting at index i for name.

type CAStore

type CAStore struct {
	// contains filtered or unexported fields
}

CAStore allows uploading / caching content-addressable files.

func CAStoreFixture

func CAStoreFixture() (*CAStore, func())

CAStoreFixture returns a CAStore for testing purposes.

func NewCAStore

func NewCAStore(config CAStoreConfig, stats tally.Scope) (*CAStore, error)

NewCAStore creates a new CAStore.

func (*CAStore) Close

func (s *CAStore) Close()

Close terminates any goroutines started by s.

func (*CAStore) CreateCacheFile

func (s *CAStore) CreateCacheFile(name string, r io.Reader) error

CreateCacheFile initializes a cache file for name from r. name should be a raw hex sha256 digest, and the contents of r must hash to name.

func (CAStore) CreateUploadFile

func (s CAStore) CreateUploadFile(name string, length int64) error

func (CAStore) DeleteCacheFile

func (s CAStore) DeleteCacheFile(name string) error

func (CAStore) DeleteCacheFileMetadata

func (s CAStore) DeleteCacheFileMetadata(name string, md metadata.Metadata) error

func (CAStore) DeleteUploadFile

func (s CAStore) DeleteUploadFile(name string) error

func (CAStore) GetCacheFileMetadata

func (s CAStore) GetCacheFileMetadata(name string, md metadata.Metadata) error

func (CAStore) GetCacheFileReader

func (s CAStore) GetCacheFileReader(name string) (FileReader, error)

func (CAStore) GetCacheFileStat

func (s CAStore) GetCacheFileStat(name string) (os.FileInfo, error)

func (CAStore) GetOrSetCacheFileMetadata

func (s CAStore) GetOrSetCacheFileMetadata(name string, md metadata.Metadata) error

func (CAStore) GetUploadFileMetadata

func (s CAStore) GetUploadFileMetadata(name string, md metadata.Metadata) error

func (CAStore) GetUploadFileReadWriter

func (s CAStore) GetUploadFileReadWriter(name string) (FileReadWriter, error)

func (CAStore) GetUploadFileReader

func (s CAStore) GetUploadFileReader(name string) (FileReader, error)

func (CAStore) GetUploadFileStat

func (s CAStore) GetUploadFileStat(name string) (os.FileInfo, error)

func (CAStore) ListCacheFiles

func (s CAStore) ListCacheFiles() ([]string, error)

func (*CAStore) MoveUploadFileToCache

func (s *CAStore) MoveUploadFileToCache(uploadName, cacheName string) error

MoveUploadFileToCache commits uploadName as cacheName. Clients are expected to validate the content of the upload file matches the cacheName digest.

func (CAStore) RangeUploadMetadata

func (s CAStore) RangeUploadMetadata(name string, f func(metadata.Metadata) error) error

func (CAStore) SetCacheFileMetadata

func (s CAStore) SetCacheFileMetadata(name string, md metadata.Metadata) (bool, error)

func (CAStore) SetUploadFileMetadata

func (s CAStore) SetUploadFileMetadata(name string, md metadata.Metadata) error

func (*CAStore) WriteCacheFile

func (s *CAStore) WriteCacheFile(name string, write func(w FileReadWriter) error) error

WriteCacheFile initializes a cache file for name by passing a temporary upload file writer to the write function.

type CAStoreConfig

type CAStoreConfig struct {
	UploadDir     string        `yaml:"upload_dir"`
	CacheDir      string        `yaml:"cache_dir"`
	Volumes       []Volume      `yaml:"volumes"`
	Capacity      int           `yaml:"capacity"`
	UploadCleanup CleanupConfig `yaml:"upload_cleanup"`
	CacheCleanup  CleanupConfig `yaml:"cache_cleanup"`
}

CAStoreConfig defines CAStore configuration.

func CAStoreConfigFixture

func CAStoreConfigFixture() (CAStoreConfig, func())

CAStoreConfigFixture returns config for CAStore for testing purposes.

type CleanupConfig

type CleanupConfig struct {
	Disabled bool          `yaml:"disabled"`
	Interval time.Duration `yaml:"interval"` // How often cleanup runs.
	TTI      time.Duration `yaml:"tti"`      // Time to idle based on last access time.
	TTL      time.Duration `yaml:"ttl"`      // Time to live regardless of access. If 0, disables TTL.
}

CleanupConfig defines configuration for periodically cleaning up idle files.

type FileReadWriter

type FileReadWriter = base.FileReadWriter

FileReadWriter is a readable, writable file.

type FileReader

type FileReader = base.FileReader

FileReader is a read-only file.

func NewBufferFileReader

func NewBufferFileReader(b []byte) FileReader

NewBufferFileReader returns an in-memory FileReader backed by b.

type MockFileReadWriter

type MockFileReadWriter struct {
	*os.File
	Committed bool
}

MockFileReadWriter is a mock FileReadWriter that is backed by a physical file. This is preferred to a gomock struct because read/write operations are greatly simplified.

func NewMockFileReadWriter

func NewMockFileReadWriter(content []byte) (*MockFileReadWriter, func())

NewMockFileReadWriter returns a new MockFileReadWriter and a cleanup function.

func (*MockFileReadWriter) Cancel

func (f *MockFileReadWriter) Cancel() error

Cancel implements FileReadWriter.Cancel

func (*MockFileReadWriter) Commit

func (f *MockFileReadWriter) Commit() error

Commit implements FileReadWriter.Commit

func (*MockFileReadWriter) Size

func (f *MockFileReadWriter) Size() int64

Size implements FileReadWriter.Size

type SimpleStore

type SimpleStore struct {
	// contains filtered or unexported fields
}

SimpleStore allows uploading / caching raw files of any format.

func NewSimpleStore

func NewSimpleStore(config SimpleStoreConfig, stats tally.Scope) (*SimpleStore, error)

NewSimpleStore creates a new SimpleStore.

func SimpleStoreFixture

func SimpleStoreFixture() (*SimpleStore, func())

SimpleStoreFixture returns a SimpleStore for testing purposes.

func (*SimpleStore) Close

func (s *SimpleStore) Close()

Close terminates goroutines started by s.

func (*SimpleStore) CreateCacheFile

func (s *SimpleStore) CreateCacheFile(name string, r io.Reader) error

CreateCacheFile initializes a cache file for name from r.

func (SimpleStore) CreateUploadFile

func (s SimpleStore) CreateUploadFile(name string, length int64) error

func (SimpleStore) DeleteCacheFile

func (s SimpleStore) DeleteCacheFile(name string) error

func (SimpleStore) DeleteCacheFileMetadata

func (s SimpleStore) DeleteCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) DeleteUploadFile

func (s SimpleStore) DeleteUploadFile(name string) error

func (SimpleStore) GetCacheFileMetadata

func (s SimpleStore) GetCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetCacheFileReader

func (s SimpleStore) GetCacheFileReader(name string) (FileReader, error)

func (SimpleStore) GetCacheFileStat

func (s SimpleStore) GetCacheFileStat(name string) (os.FileInfo, error)

func (SimpleStore) GetOrSetCacheFileMetadata

func (s SimpleStore) GetOrSetCacheFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetUploadFileMetadata

func (s SimpleStore) GetUploadFileMetadata(name string, md metadata.Metadata) error

func (SimpleStore) GetUploadFileReadWriter

func (s SimpleStore) GetUploadFileReadWriter(name string) (FileReadWriter, error)

func (SimpleStore) GetUploadFileReader

func (s SimpleStore) GetUploadFileReader(name string) (FileReader, error)

func (SimpleStore) GetUploadFileStat

func (s SimpleStore) GetUploadFileStat(name string) (os.FileInfo, error)

func (SimpleStore) ListCacheFiles

func (s SimpleStore) ListCacheFiles() ([]string, error)

func (*SimpleStore) MoveUploadFileToCache

func (s *SimpleStore) MoveUploadFileToCache(uploadName, cacheName string) error

MoveUploadFileToCache commits uploadName as cacheName.

func (SimpleStore) RangeUploadMetadata

func (s SimpleStore) RangeUploadMetadata(name string, f func(metadata.Metadata) error) error

func (SimpleStore) SetCacheFileMetadata

func (s SimpleStore) SetCacheFileMetadata(name string, md metadata.Metadata) (bool, error)

func (SimpleStore) SetUploadFileMetadata

func (s SimpleStore) SetUploadFileMetadata(name string, md metadata.Metadata) error

type SimpleStoreConfig

type SimpleStoreConfig struct {
	UploadDir     string        `yaml:"upload_dir"`
	CacheDir      string        `yaml:"cache_dir"`
	UploadCleanup CleanupConfig `yaml:"upload_cleanup"`
	CacheCleanup  CleanupConfig `yaml:"cache_cleanup"`
}

SimpleStoreConfig defines SimpleStore configuration.

type Volume

type Volume struct {
	Location string
	Weight   int
}

Volume - if provided, volumes are used to store the actual files. Symlinks will be created under state directories. This configuration is needed on hosts with multiple disks.

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL