package module
v0.0.0-...-48a28c7 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jan 24, 2018 License: MIT Imports: 14 Imported by: 0



A package and AppEngine Standard service that can be used together to replicate Datastore Puts from one AppEngine project to another. The primary use case is to replicate data across regions.

Replication retains namespace and full parent keys.

There is no coordination to determine latest write when race conditions occur. Recommended only for use with Datastore entities that are write-once.




  • Google AppEngine (Standard)
  • Google Cloud Datastore
  • Google Cloud PubSub


Use the Makefile to setup the cross region PubSub topics and subscriptions. Use the package to publish Datastore Puts, and deploy the service to all regions where the data should be replicated.

1. Create the PubSub topics and subscriptions
cd service
make PROJECT_IDS="<project-id-1> <project-id-2> ..." create-pubsub
2. Publish Datastore Puts to other regions
import replicator ""
// Put the entity as you normally would.
err := datastore.Put(ctx, key, entity)
if err != nil {
	return err

// Publish the key and entity that was put.
err = replicator.Publish(ctx, "datastore-puts", key, entity)
if err != nil {
	return err

Recommendations and strategies:

  • Generate IDs for keys yourself and use complete keys, don't use the Datastore Allocator to generate IDs by putting entities with an incomplete key. If you use the allocator it may generate overlapping IDs in different projects causing loss of existing entities.
  • Write an entity once to a key, and write updates as new entities. Resolve conflicts when reading.
3. Subscribe to Datastore Puts and replicate them

Use the replicator service to receive pushes from the subscriptions setup in step 1. The replicator service will store the entities into its local project respecting the same namespace and key, including the parent keys.

cd service
make PROJECT_IDS="<project-id-1> <project-id-2> ..." deploy

Alternatively if you'd like to create your own service to store the entities, take a look at service/app.go for how to consume the PubSub request, unpack it, then put it to the local Datastore.


See example.




This section is empty.


This section is empty.


func Publish

func Publish(c context.Context, topic string, key *datastore.Key, src interface{}) error


type Source

type Source struct {
	AppID          string
	DatacenterID   string
	InstanceID     string
	ServiceName    string
	ServiceVersion string
	RequestID      string

func Unpack

func Unpack(c context.Context, r io.Reader) (source *Source, key *datastore.Key, props *datastore.PropertyList, err error)


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL