datastore

package module
Version: v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 9, 2018 License: MIT Imports: 16 Imported by: 111

README

Datastore Wrapper Go Documentation CircleCI

🚧 This package is unstable 🚧 github repo

(AppEngine | Cloud) Datastore wrapper for Go 👉

Simple. Happy. Respect standard library.

$ go get -u go.mercari.io/datastore

Feature

DO
  • Wrap google.golang.org/appengine/datastore and cloud.google.com/go/datastore
    • keep key behavior
    • align to cloud.google.com/go/datastore first
  • Re-implement datastore package
  • Re-implement datastore.SaveStruct & LoadStruct
    • Ignore unmapped property
    • Add PropertyTranslator interface
      • Convert types like mytime.Unix to time.Time and reverse it
      • Rename property like CreatedAt to createdAt or created_at and reverse it
  • Re-implement PropertyLoadSaver
    • Pass context.Context to Save & Load method
  • Add retry feature to each RPC
    • e.g. Retry AllocateID when it failed
  • Add middleware layer
    • About...
      • Local Cache
      • AE Memcache
      • Logging
      • Retry
      • etc...
    • Easy to ON/OFF switching
  • Add some useful methods
    • aedatastore/TransactionContext
DON'T
  • have utility functions
  • support firestore

Restriction

  • aedatastore package
    • When using slice of struct, MUST specified datastore:",flatten" option.
      • original (ae & cloud) datastore.SaveStruct have different behaviors.
      • see aeprodtest/main.go /api/test3

How To Use

TODO

Best Practice

TODO

Project Navigation

  • aedatastore
    • Datastore Wrapper implementation for AppEngine Datastore
  • aeprodtest
    • Test code that works in AppEngine production environment
  • boom
    • goon likes interface for this package
  • clouddatastore
    • Datastore Wrapper implementation for Cloud Datastore
  • dsmiddleware
    • Various processing can be intercepted
    • dsmiddleware/aememcache
    • dsmiddleware/chaosrpc
      • Randomly generate an RPC error!
    • dsmiddleware/dslog
      • Datastore operation logging in cache layer
    • dsmiddleware/fishbone
      • Replace query by KeysOnly query with Get ops
    • dsmiddleware/localcache
      • Entity caching with machine local memory
    • dsmiddleware/noop
      • noop middleware 😉
    • dsmiddleware/rediscache
      • Entity caching with Redis
    • dsmiddleware/rpcretry
      • Retry RPC when an error occurred
    • dsmiddleware/storagecache
      • Entity cacheing base code
  • internal
    • internal package
  • testbed
    • Testbed to check the behavior of AE Datastore and Cloud Datastore
  • testsuite

TODO

  • Write tests for namespace
  • Write more documents

Committers

Contribution

Please read the CLA below carefully before submitting your contribution.

https://www.mercari.com/cla/

Setup environment & Run tests
  • requirements
    • gcloud sdk
      • gcloud components install app-engine-go
      • gcloud components install beta cloud-datastore-emulator
    • dep
      • go get -u github.com/golang/dep/cmd/dep
  1. Testing in local
$ ./setup.sh # exec once
$ ./serve.sh # exec in background
$ ./test.sh
  1. Testing with Circle CI CLI
$ circleci build

License

Copyright 2017 Mercari, Inc.

Licensed under the MIT License.

Documentation

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrInvalidEntityType = errors.New("datastore: invalid entity type")
	ErrInvalidKey        = errors.New("datastore: invalid key")
	ErrNoSuchEntity      = errors.New("datastore: no such entity")
)
View Source
var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction")
View Source
var LoadEntity = loadEntity

TODO hide LoadEntity project outside

View Source
var SaveEntity = saveEntity
View Source
var SuppressErrFieldMismatch = true

Functions

func LoadStruct

func LoadStruct(ctx context.Context, dst interface{}, p []Property) error

LoadStruct loads the properties from p to dst. dst must be a struct pointer.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each LoadStruct call.

Types

type Batch

type Batch struct {
	Client Client
	// contains filtered or unexported fields
}

func (*Batch) Delete

func (b *Batch) Delete(key Key, h BatchErrHandler)

func (*Batch) Exec

func (b *Batch) Exec(ctx context.Context) error

func (*Batch) Get

func (b *Batch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*Batch) Put

func (b *Batch) Put(key Key, src interface{}, h BatchPutHandler)

type BatchErrHandler added in v0.9.0

type BatchErrHandler func(err error) error

type BatchPutHandler added in v0.9.0

type BatchPutHandler func(key Key, err error) error

type Client

type Client interface {
	Get(ctx context.Context, key Key, dst interface{}) error
	GetMulti(ctx context.Context, keys []Key, dst interface{}) error
	Put(ctx context.Context, key Key, src interface{}) (Key, error)
	PutMulti(ctx context.Context, keys []Key, src interface{}) ([]Key, error)
	Delete(ctx context.Context, key Key) error
	DeleteMulti(ctx context.Context, keys []Key) error

	NewTransaction(ctx context.Context) (Transaction, error)
	RunInTransaction(ctx context.Context, f func(tx Transaction) error) (Commit, error)
	Run(ctx context.Context, q Query) Iterator
	AllocatedIDs(ctx context.Context, keys []Key) ([]Key, error)
	Count(ctx context.Context, q Query) (int, error)
	GetAll(ctx context.Context, q Query, dst interface{}) ([]Key, error)

	IncompleteKey(kind string, parent Key) Key
	NameKey(kind, name string, parent Key) Key
	IDKey(kind string, id int64, parent Key) Key

	NewQuery(kind string) Query

	Close() error

	DecodeKey(encoded string) (Key, error)
	DecodeCursor(s string) (Cursor, error)

	Batch() *Batch
	AppendMiddleware(middleware Middleware) // NOTE First-In First-Apply
	RemoveMiddleware(middleware Middleware) bool
	Context() context.Context
	SetContext(ctx context.Context)
}

type ClientGenerator

type ClientGenerator func(ctx context.Context, opts ...ClientOption) (Client, error)
var FromContext ClientGenerator

type ClientOption

type ClientOption interface {
	Apply(*internal.ClientSettings)
}

func WithCredentialsFile

func WithCredentialsFile(filename string) ClientOption

WithCredentialsFile returns a ClientOption that authenticates API calls with the given service account or refresh token JSON credentials file.

func WithHTTPClient

func WithHTTPClient(client *http.Client) ClientOption

WithHTTPClient returns a ClientOption that specifies the HTTP client to use as the basis of communications. This option may only be used with services that support HTTP as their communication transport. When used, the WithHTTPClient option takes precedent over all other supplied options.

func WithProjectID

func WithProjectID(projectID string) ClientOption

func WithScopes

func WithScopes(scope ...string) ClientOption

WithScopes returns a ClientOption that overrides the default OAuth2 scopes to be used for a service.

func WithTokenSource

func WithTokenSource(s oauth2.TokenSource) ClientOption

WithTokenSource returns a ClientOption that specifies an OAuth2 token source to be used as the basis for authentication.

type Commit

type Commit interface {
	Key(p PendingKey) Key
}

type Cursor

type Cursor interface {
	String() string
}

type Entity

type Entity struct {
	Key        Key
	Properties []Property
}

type ErrFieldMismatch

type ErrFieldMismatch struct {
	StructType reflect.Type
	FieldName  string
	Reason     string
}

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. StructType is the type of the struct pointed to by the destination argument passed to Get or to Iterator.Next.

func (*ErrFieldMismatch) Error

func (e *ErrFieldMismatch) Error() string

type GeoPoint

type GeoPoint struct {
	Lat, Lng float64
}

type Iterator

type Iterator interface {
	Next(dst interface{}) (Key, error)
	Cursor() (Cursor, error)
}

type Key

type Key interface {
	Kind() string
	ID() int64
	Name() string
	ParentKey() Key
	Namespace() string

	String() string
	GobEncode() ([]byte, error)
	GobDecode(buf []byte) error
	MarshalJSON() ([]byte, error)
	UnmarshalJSON(buf []byte) error
	Encode() string
	Equal(o Key) bool
	Incomplete() bool
}

type KeyLoader

type KeyLoader interface {
	PropertyLoadSaver
	LoadKey(ctx context.Context, k Key) error
}

type Middleware added in v0.11.0

type Middleware interface {
	AllocateIDs(info *MiddlewareInfo, keys []Key) ([]Key, error)
	PutMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]Key, error)
	PutMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) ([]PendingKey, error)
	GetMultiWithoutTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	GetMultiWithTx(info *MiddlewareInfo, keys []Key, psList []PropertyList) error
	DeleteMultiWithoutTx(info *MiddlewareInfo, keys []Key) error
	DeleteMultiWithTx(info *MiddlewareInfo, keys []Key) error
	PostCommit(info *MiddlewareInfo, tx Transaction, commit Commit) error
	PostRollback(info *MiddlewareInfo, tx Transaction) error
	Run(info *MiddlewareInfo, q Query, qDump *QueryDump) Iterator
	GetAll(info *MiddlewareInfo, q Query, qDump *QueryDump, psList *[]PropertyList) ([]Key, error)
	Next(info *MiddlewareInfo, q Query, qDump *QueryDump, iter Iterator, ps *PropertyList) (Key, error)
	Count(info *MiddlewareInfo, q Query, qDump *QueryDump) (int, error)
}

type MiddlewareInfo added in v0.11.0

type MiddlewareInfo struct {
	Context     context.Context
	Client      Client
	Transaction Transaction
	Next        Middleware
}

type MultiError

type MultiError []error

MultiError is returned by batch operations when there are errors with particular elements. Errors will be in a one-to-one correspondence with the input elements; successful elements will have a nil entry.

func (MultiError) Error

func (m MultiError) Error() string

type PendingKey

type PendingKey interface {
	StoredContext() context.Context
}

type Property

type Property struct {
	Name    string
	Value   interface{}
	NoIndex bool
}

func SaveStruct

func SaveStruct(ctx context.Context, src interface{}) ([]Property, error)

SaveStruct returns the properties from src as a slice of Properties. src must be a struct pointer.

type PropertyList

type PropertyList []Property

func (*PropertyList) Load

func (l *PropertyList) Load(ctx context.Context, p []Property) error

Load loads all of the provided properties into l. It does not first reset *l to an empty slice.

func (*PropertyList) Save

func (l *PropertyList) Save(ctx context.Context) ([]Property, error)

Save saves all of l's properties as a slice of Properties.

type PropertyLoadSaver

type PropertyLoadSaver interface {
	Load(ctx context.Context, ps []Property) error
	Save(ctx context.Context) ([]Property, error)
}

type PropertyTranslator

type PropertyTranslator interface {
	ToPropertyValue(ctx context.Context) (interface{}, error)
	FromPropertyValue(ctx context.Context, p Property) (dst interface{}, err error)
}

type Query

type Query interface {
	Ancestor(ancestor Key) Query
	EventualConsistency() Query
	Namespace(ns string) Query
	Transaction(t Transaction) Query
	Filter(filterStr string, value interface{}) Query
	Order(fieldName string) Query
	Project(fieldNames ...string) Query
	Distinct() Query
	// NOT IMPLEMENTED ON APPENGINE DistinctOn(fieldNames ...string) *Query
	KeysOnly() Query
	Limit(limit int) Query
	Offset(offset int) Query
	Start(c Cursor) Query
	End(c Cursor) Query

	Dump() *QueryDump
}

type QueryDump added in v0.7.0

type QueryDump struct {
	Kind                string
	Ancestor            Key
	EventualConsistency bool
	Namespace           string
	Transaction         Transaction
	Filter              []*QueryFilterCondition
	Order               []string
	Project             []string
	Distinct            bool
	KeysOnly            bool
	Limit               int
	Offset              int
	Start               Cursor
	End                 Cursor
}

func (*QueryDump) String added in v0.7.0

func (dump *QueryDump) String() string

type QueryFilterCondition added in v0.7.0

type QueryFilterCondition struct {
	Filter string
	Value  interface{}
}

type Transaction

type Transaction interface {
	Get(key Key, dst interface{}) error
	GetMulti(keys []Key, dst interface{}) error
	Put(key Key, src interface{}) (PendingKey, error)
	PutMulti(keys []Key, src interface{}) ([]PendingKey, error)
	Delete(key Key) error
	DeleteMulti(keys []Key) error

	Commit() (Commit, error)
	Rollback() error

	Batch() *TransactionBatch
}

type TransactionBatch

type TransactionBatch struct {
	Transaction Transaction
	// contains filtered or unexported fields
}

func (*TransactionBatch) Delete

func (b *TransactionBatch) Delete(key Key, h BatchErrHandler)

func (*TransactionBatch) Exec

func (b *TransactionBatch) Exec() error

func (*TransactionBatch) Get

func (b *TransactionBatch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*TransactionBatch) Put

func (b *TransactionBatch) Put(key Key, src interface{}, h TxBatchPutHandler)

type TxBatchPutHandler added in v0.9.0

type TxBatchPutHandler func(pKey PendingKey, err error) error

Directories

Path Synopsis
dsmiddleware
c/atomiccache
Package atomiccache provides a map-based cache that supports very fast reads.
Package atomiccache provides a map-based cache that supports very fast reads.
c/fields
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
pb/memcache
Package memcache is a generated protocol buffer package.
Package memcache is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL