package module
v0.10.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Dec 7, 2017 License: MIT Imports: 16 Imported by: 125


Datastore Wrapper Go Documentation CircleCI

🚧 This package is unstable 🚧 github repo

(AppEngine | Cloud) Datastore wrapper for Go 👉

Simple. Happy. Respect standard library.

$ go get -u


  • Wrap and
    • keep key behavior
    • align to first
  • Re-implement datastore package
  • Re-implement datastore.SaveStruct & LoadStruct
    • Ignore unmapped property
    • Add PropertyTranslator interface
      • Convert types like mytime.Unix to time.Time and reverse it
      • Rename property like CreatedAt to createdAt or created_at and reverse it
  • Re-implement PropertyLoadSaver
    • Pass context.Context to Save & Load method
    • It can catch entity from the cache layers as well as from the datastore
  • Add retry feature to each RPC
    • e.g. Retry AllocateID when it failed
  • Add cache layer
    • About...
      • Local Cache
      • AE Memcache
      • etc...
    • Easy to ON/OFF switching
  • Add some useful methods
    • aedatastore/TransactionContext
  • have utility functions
  • support firestore


  • aedatastore package
    • When using slice of struct, MUST specified datastore:",flatten" option.
      • original (ae & cloud) datastore.SaveStruct have different behaviors.
      • see aeprodtest/main.go /api/test3

How To Use


Best Practice


Project Navigation

  • aedatastore
    • Datastore Wrapper implementation for AppEngine Datastore
  • aeprodtest
    • Test code that works in AppEngine production environment
  • boom
    • goon likes interface for this package
  • cache
    • Implementation of cache layer strategies
    • cache/aememcache
    • cache/dslog
      • Datastore operation logging in cache layer
    • cache/fishbone
      • Replace query by KeysOnly query with Get ops
    • cache/localcache
      • Entity caching with machine local memory
    • cache/storagecache
      • Entity cacheing base code
  • clouddatastore
    • Datastore Wrapper implementation for Cloud Datastore
  • internal
    • internal package
  • testbed
    • Testbed to check the behavior of AE Datastore and Cloud Datastore
  • testsuite


  • Write tests for namespace
  • Retry feature
  • Write more documents



Please read the CLA below carefully before submitting your contribution.

Setup environment & Run tests
  • requirements
    • gcloud sdk
      • gcloud components install app-engine-go
      • gcloud components install beta cloud-datastore-emulator
    • dep
      • go get -u
  1. Testing in local
$ ./ # exec once
$ ./ # exec in background
$ ./
  1. Testing with Docker
$ ./
  1. Testing with Circle CI CLI
$ circleci build


Copyright 2017 Mercari, Inc.

Licensed under the MIT License.




This section is empty.


View Source
var (
	ErrInvalidEntityType = errors.New("datastore: invalid entity type")
	ErrInvalidKey        = errors.New("datastore: invalid key")
	ErrNoSuchEntity      = errors.New("datastore: no such entity")
View Source
var ErrConcurrentTransaction = errors.New("datastore: concurrent transaction")
View Source
var LoadEntity = loadEntity

TODO hide LoadEntity project outside

View Source
var SaveEntity = saveEntity
View Source
var SuppressErrFieldMismatch = true


func LoadStruct

func LoadStruct(ctx context.Context, dst interface{}, p []Property) error

LoadStruct loads the properties from p to dst. dst must be a struct pointer.

The values of dst's unmatched struct fields are not modified, and matching slice-typed fields are not reset before appending to them. In particular, it is recommended to pass a pointer to a zero valued struct on each LoadStruct call.


type Batch

type Batch struct {
	Client Client
	// contains filtered or unexported fields

func (*Batch) Delete

func (b *Batch) Delete(key Key, h BatchErrHandler)

func (*Batch) Exec

func (b *Batch) Exec(ctx context.Context) error

func (*Batch) Get

func (b *Batch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*Batch) Put

func (b *Batch) Put(key Key, src interface{}, h BatchPutHandler)

type BatchErrHandler added in v0.9.0

type BatchErrHandler func(err error) error

type BatchPutHandler added in v0.9.0

type BatchPutHandler func(key Key, err error) error

type CacheInfo added in v0.7.0

type CacheInfo struct {
	Context     context.Context
	Client      Client
	Transaction Transaction
	Next        CacheStrategy

type CacheStrategy added in v0.7.0

type CacheStrategy interface {
	PutMultiWithoutTx(info *CacheInfo, keys []Key, psList []PropertyList) ([]Key, error)
	PutMultiWithTx(info *CacheInfo, keys []Key, psList []PropertyList) ([]PendingKey, error)
	GetMultiWithoutTx(info *CacheInfo, keys []Key, psList []PropertyList) error
	GetMultiWithTx(info *CacheInfo, keys []Key, psList []PropertyList) error
	DeleteMultiWithoutTx(info *CacheInfo, keys []Key) error
	DeleteMultiWithTx(info *CacheInfo, keys []Key) error
	PostCommit(info *CacheInfo, tx Transaction, commit Commit) error
	PostRollback(info *CacheInfo, tx Transaction) error
	Run(info *CacheInfo, q Query, qDump *QueryDump) Iterator
	GetAll(info *CacheInfo, q Query, qDump *QueryDump, psList *[]PropertyList) ([]Key, error)
	Next(info *CacheInfo, q Query, qDump *QueryDump, iter Iterator, ps *PropertyList) (Key, error)

type Client

type Client interface {
	Get(ctx context.Context, key Key, dst interface{}) error
	GetMulti(ctx context.Context, keys []Key, dst interface{}) error
	Put(ctx context.Context, key Key, src interface{}) (Key, error)
	PutMulti(ctx context.Context, keys []Key, src interface{}) ([]Key, error)
	Delete(ctx context.Context, key Key) error
	DeleteMulti(ctx context.Context, keys []Key) error

	NewTransaction(ctx context.Context) (Transaction, error)
	RunInTransaction(ctx context.Context, f func(tx Transaction) error) (Commit, error)
	Run(ctx context.Context, q Query) Iterator
	AllocatedIDs(ctx context.Context, keys []Key) ([]Key, error)
	Count(ctx context.Context, q Query) (int, error)
	GetAll(ctx context.Context, q Query, dst interface{}) ([]Key, error)

	IncompleteKey(kind string, parent Key) Key
	NameKey(kind, name string, parent Key) Key
	IDKey(kind string, id int64, parent Key) Key

	NewQuery(kind string) Query

	Close() error

	DecodeKey(encoded string) (Key, error)
	DecodeCursor(s string) (Cursor, error)

	Batch() *Batch
	AppendCacheStrategy(strategy CacheStrategy) // NOTE First-In Last-Apply
	RemoveCacheStrategy(strategy CacheStrategy) bool
	SwapContext(ctx context.Context) context.Context

type ClientGenerator

type ClientGenerator func(ctx context.Context, opts ...ClientOption) (Client, error)
var FromContext ClientGenerator

type ClientOption

type ClientOption interface {

func WithCredentialsFile

func WithCredentialsFile(filename string) ClientOption

WithCredentialsFile returns a ClientOption that authenticates API calls with the given service account or refresh token JSON credentials file.

func WithHTTPClient

func WithHTTPClient(client *http.Client) ClientOption

WithHTTPClient returns a ClientOption that specifies the HTTP client to use as the basis of communications. This option may only be used with services that support HTTP as their communication transport. When used, the WithHTTPClient option takes precedent over all other supplied options.

func WithProjectID

func WithProjectID(projectID string) ClientOption

func WithScopes

func WithScopes(scope ...string) ClientOption

WithScopes returns a ClientOption that overrides the default OAuth2 scopes to be used for a service.

func WithTokenSource

func WithTokenSource(s oauth2.TokenSource) ClientOption

WithTokenSource returns a ClientOption that specifies an OAuth2 token source to be used as the basis for authentication.

type Commit

type Commit interface {
	Key(p PendingKey) Key

type Cursor

type Cursor interface {
	String() string

type Entity

type Entity struct {
	Key        Key
	Properties []Property

type ErrFieldMismatch

type ErrFieldMismatch struct {
	StructType reflect.Type
	FieldName  string
	Reason     string

ErrFieldMismatch is returned when a field is to be loaded into a different type than the one it was stored from, or when a field is missing or unexported in the destination struct. StructType is the type of the struct pointed to by the destination argument passed to Get or to Iterator.Next.

func (*ErrFieldMismatch) Error

func (e *ErrFieldMismatch) Error() string

type GeoPoint

type GeoPoint struct {
	Lat, Lng float64

type Iterator

type Iterator interface {
	Next(dst interface{}) (Key, error)
	Cursor() (Cursor, error)

type Key

type Key interface {
	Kind() string
	ID() int64
	Name() string
	ParentKey() Key
	Namespace() string

	String() string
	GobEncode() ([]byte, error)
	GobDecode(buf []byte) error
	MarshalJSON() ([]byte, error)
	UnmarshalJSON(buf []byte) error
	Encode() string
	Equal(o Key) bool
	Incomplete() bool

type KeyLoader

type KeyLoader interface {
	LoadKey(ctx context.Context, k Key) error

type MultiError

type MultiError []error

MultiError is returned by batch operations when there are errors with particular elements. Errors will be in a one-to-one correspondence with the input elements; successful elements will have a nil entry.

func (MultiError) Error

func (m MultiError) Error() string

type PendingKey

type PendingKey interface {
	StoredContext() context.Context

type Property

type Property struct {
	Name    string
	Value   interface{}
	NoIndex bool

func SaveStruct

func SaveStruct(ctx context.Context, src interface{}) ([]Property, error)

SaveStruct returns the properties from src as a slice of Properties. src must be a struct pointer.

type PropertyList

type PropertyList []Property

func (*PropertyList) Load

func (l *PropertyList) Load(ctx context.Context, p []Property) error

Load loads all of the provided properties into l. It does not first reset *l to an empty slice.

func (*PropertyList) Save

func (l *PropertyList) Save(ctx context.Context) ([]Property, error)

Save saves all of l's properties as a slice of Properties.

type PropertyLoadSaver

type PropertyLoadSaver interface {
	Load(ctx context.Context, ps []Property) error
	Save(ctx context.Context) ([]Property, error)

type PropertyTranslator

type PropertyTranslator interface {
	ToPropertyValue(ctx context.Context) (interface{}, error)
	FromPropertyValue(ctx context.Context, p Property) (dst interface{}, err error)

type Query

type Query interface {
	Ancestor(ancestor Key) Query
	EventualConsistency() Query
	Namespace(ns string) Query
	Transaction(t Transaction) Query
	Filter(filterStr string, value interface{}) Query
	Order(fieldName string) Query
	Project(fieldNames ...string) Query
	Distinct() Query
	// NOT IMPLEMENTED ON APPENGINE DistinctOn(fieldNames ...string) *Query
	KeysOnly() Query
	Limit(limit int) Query
	Offset(offset int) Query
	Start(c Cursor) Query
	End(c Cursor) Query

	Dump() *QueryDump

type QueryDump added in v0.7.0

type QueryDump struct {
	Kind                string
	Ancestor            Key
	EventualConsistency bool
	Namespace           string
	Transaction         Transaction
	Filter              []*QueryFilterCondition
	Order               []string
	Project             []string
	Distinct            bool
	KeysOnly            bool
	Limit               int
	Offset              int
	Start               Cursor
	End                 Cursor

func (*QueryDump) String added in v0.7.0

func (dump *QueryDump) String() string

type QueryFilterCondition added in v0.7.0

type QueryFilterCondition struct {
	Filter string
	Value  interface{}

type Transaction

type Transaction interface {
	Get(key Key, dst interface{}) error
	GetMulti(keys []Key, dst interface{}) error
	Put(key Key, src interface{}) (PendingKey, error)
	PutMulti(keys []Key, src interface{}) ([]PendingKey, error)
	Delete(key Key) error
	DeleteMulti(keys []Key) error

	Commit() (Commit, error)
	Rollback() error

	Batch() *TransactionBatch

type TransactionBatch

type TransactionBatch struct {
	Transaction Transaction
	// contains filtered or unexported fields

func (*TransactionBatch) Delete

func (b *TransactionBatch) Delete(key Key, h BatchErrHandler)

func (*TransactionBatch) Exec

func (b *TransactionBatch) Exec() error

func (*TransactionBatch) Get

func (b *TransactionBatch) Get(key Key, dst interface{}, h BatchErrHandler)

func (*TransactionBatch) Put

func (b *TransactionBatch) Put(key Key, src interface{}, h TxBatchPutHandler)

type TxBatchPutHandler added in v0.9.0

type TxBatchPutHandler func(pKey PendingKey, err error) error


Path Synopsis
Package atomiccache provides a map-based cache that supports very fast reads.
Package atomiccache provides a map-based cache that supports very fast reads.
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
Package fields provides a view of the fields of a struct that follows the Go rules, amended to consider tags and case insensitivity.
Package memcache is a generated protocol buffer package.
Package memcache is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL