bundle

Documentation

Overview

Package bundle is a self-contained appengine module with aetools.

The bundle can be deployed using the Google Cloud SDK or the Google AppEngine SDK for Go:

gcloud preview app deploy --project=your-app-id --version=your-bundle-version app.yaml

If you want this to be outside de default module, you can change the app.yaml to specify the desired version.

Index

• func AdminHandler(w http.ResponseWriter, r *http.Request)
• func CreateTableHandler(w http.ResponseWriter, r *http.Request)
• func SchemaHandler(w http.ResponseWriter, r *http.Request)
• func SyncEntityHandler(w http.ResponseWriter, r *http.Request)
• func SyncKindHandler(w http.ResponseWriter, r *http.Request)

Functions

func AdminHandler

func AdminHandler(w http.ResponseWriter, r *http.Request)

AdminHandler displays the bundle administrative page.

func CreateTableHandler

func CreateTableHandler(w http.ResponseWriter, r *http.Request)

CreateTableHandler creates a new Bigquery table using the infered schema from the datastore statistics. This handler expects the parameters "project", "dataset" and "kind", and creates a table under "project:dataset.kind".

func SchemaHandler

func SchemaHandler(w http.ResponseWriter, r *http.Request)

SchemaHandler prints the JSON schema for a kind using the method bigquerysync.SchemaForKind.

func SyncEntityHandler

func SyncEntityHandler(w http.ResponseWriter, r *http.Request)

SyncEntityHandler synchronizes a range of entity keys. This handler expects the same parameters as SyncKindHandler, except for "kind". Instead of kind, you have to specify a mandatory "startKey", and optionally an "endKey".

If specified, both startKey and endKey must be URL Encoded complete datastore keys. The start is inclusive and all entities up to end (exclusive) will be synced. If end is empty, or an invalid key, all entities following start are synced, until there is no more entities. If start and end are equal, then only one entity is synced.

To optimize memory usage, this handler processes up to bigqueysync.BatchSize entities, and if the query is not done, reschedule itself with the last processed key as start. Also, in order to keep the payload sent to Bigquery under the URLFetch limit, the entities are processed in small chuncks of 9 entities.

func SyncKindHandler

func SyncKindHandler(w http.ResponseWriter, r *http.Request)

SyncKindHandler spawn task queues for paralell synchronization of all entities in a specific Kind. This handler requires the form parameters "project" and "dataset", as well as the "kind" parameter.

The following path synchronizes all entities with kind "Baz", into a table named "Baz", under the dataset "bar" in the "foo" project:

/bq/sync/kind?project=foo&dataset=bar&kind=Baz

This handler also supports the "exclude" optional parameter, that is a regular expression to exclude field names. For example, the following path will do the same as the previous one, and also will filter the properties starting with lowercase "a", "b" or that contains "foo" in their names:

/bq/sync/kind?project=foo&dataset=bar&kind=Baz&exclude=^a.*|^b.*|foo

This is particulary usefull on large entities, to skip long text fields and keep the request of each row under the Bigquery limit of 20Kb.

Finally, the "queue" parameter can be specified to target a specific task queue to run all sync jobs.