gdnotify

package module
v0.4.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 15, 2023 License: MIT Imports: 45 Imported by: 0

README

gdnotify

Documentation Latest GitHub release Github Actions test License

gdnotify is google drive change notifier for AWS

Changes that occur in Google Drive are notified through Amazon EventBridge.

Install

Homebrew (macOS and Linux)
$ brew install mashiike/tap/awstee
Binary packages

Releases

Usage with AWS Lambda (serverless)

gdnotify works with AWS Lambda, Amazon EventBridge and Amazon DynamoDB.

Lambda Function requires a webhook and a maintainer

sequenceDiagram
  autonumber
  Maintainer lambda function->>+Google API: GET /drive/v3/changes/startPageToken
  Google API-->>-Maintainer lambda function: PageToken
  Maintainer lambda function->>+Google API: POST /drive/v3/changes/watch
  Google API [push]--)Webhook lambda function:  sync request
  Google API-->>-Maintainer lambda function: response
  Maintainer lambda function->>+DynamoDB: put item
  DynamoDB-->>-Maintainer lambda function: response
  loop
    Google API [push]->>+Webhook lambda function: change request
    Webhook lambda function->>+Google API: GET /drive/v3/changes
    Google API-->>-Webhook lambda function: changes info
    Webhook lambda function->>+EventBridge: Put Events
    EventBridge-->>-Webhook lambda function: response
    Webhook lambda function->>+DynamoDB: update item
    DynamoDB-->>-Webhook lambda function: response
    Webhook lambda function->>+Google API [push]: Status Ok
  end

The basic configuration file is as follows:

required_version: ">=0.0.0"

webhook: "{{ env `WEBHOOK_LAMBDA_URL`}}" #webhook mode lambda function URL
expiration: 168h

# backend setting to get GOOGLE_APPLICATION_CREDENTIALS.
# Default is None, in which case https://cloud.google.com/docs/authentication/production 
# If you want to use AWS Secrets Manager, set the backend_type to SSMParameterStore and specify the parameter_name according to the following
https://docs.aws.amazon.com/systems-manager/latest/userguide/integration-ps-secretsmanager.html
credentials:
  backend_type: SSMParameterStore                               
  parameter_name: /gdnotify/GOOGLE_APPLICATION_CREDENTIALS #SSM Parameter Name
  base64encoding: false # If the Parameter Store value is base64encoded, set this value to true

# Storage settings for storing the status of notification channels.
# Default type is DynamoDB
storage:
  type: DynamoDB
  table_name: gdnotify # DynamoDB Table Name

# Set the recipients to be notified of detected changes
# Default type is EventBridge
notification:
  type: EventBridge
  event_bus: gdnotify # Event Bus Name. Although it is possible to use the `default`, it is recommended to create and use a custom event bus.

drives:
  - drive_id: __default__   # __default__ is a special setting, indicating a drive that is not tied to a specific Drive, 
                            # but can be sensed with the given permissions. (For example, files that reside in MyDrive)
  - drive_id: XXXXXXXXXXXXXXXXXXX  # Usually, you should specify the DriveID of the team drive

Let's solidify the Lambda package with the following configuration (runtime provided.al2)

lambda.zip
├── bootstrap    # build binary
└── config.yaml  # configuration file

A related document is https://docs.aws.amazon.com/lambda/latest/dg/runtimes-custom.html

Webhook lambda function

The Webhook's Lambda function receives notifications from the Google API. It then puts the Event into the EventBus of Amazon EventBridge.

Set the following environment variables:

  • GDNOTIFY_RUN_MODE=webhook
  • GDNOTIFY_CONFIG=config.yaml

The required IAM Role permissions are as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Webhook",
            "Effect": "Allow",
            "Action": [
                "events:PutEvents",
                "dynamodb:DescribeTable",
                "dynamodb:GetItem",
                "dynamodb:UpdateItem"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

It is necessary to set up a Lambda Function URL, or API gateway, ALB, etc. Here, the Lambda Function URL is assumed to be simple. lambda function URLs document is https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html Suppose you get the following URL: https://xxxxxxxxxxxx.lambda-url.ap-northeast-1.on.aws

Maintainer lambda function

Maintainer's Lambda function registers a notification channel with the Google API. It is assumed that this Lambda function will be called at regular intervals to perform maintenance tasks such as re-registering those newly added to the monitoring or those about to expire at that time.

Set the following environment variables:

The required IAM Role permissions are as follows.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Webhook",
            "Effect": "Allow",
            "Action": [
                "dynamodb:CreateTable",
                "dynamodb:PutItem",
                "dynamodb:DeleteItem",
                "dynamodb:Scan"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

EventBridge scheduling rules are required. The following tutorial will help you set it up. https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-run-lambda-schedule.html

Usage as CLI

gdnotify -config <config file> [options] [command]
version: v0.0.0

commands:
   list          list notification channels
   serve         serve webhook server
   register      register a new notification channel for a drive for which a notification channel has not yet been set
   maintenance   re-register expired notification channels or register new unregistered channels.
   cleanup       remove all notification channels

options:
  -config value
        config list
  -log-level string
        run mode (default "info")
  -port int
        webhook httpd port
  -run-mode string
        run mode (cli|webhook|maintainer) (default "cli")

For Local Development

required_version: ">=0.0.0"

webhook: "{{ env `HTTP_TUNNEL_URL` }}"
expiration: 168h

storage:
  type: File
  data_file: data/storage.gob

notification:
  type: File
  event_file: data/events.json

drives:
  - drive_id: __default__

It can be run locally using the cloudflare tunnel command or ngrok. For example

$ HTTP_TUNNEL="cloudflared tunnel --url localhost{{ .Address }}" GDNOTIFY_RUN_MODE=webhook go run cmd/gdnotify/main.go -config local.yaml -log-level debug

see tunnel.log, get tunnel url

$  HTTP_TUNNEL_URL="https://..." go run cmd/gdnotify/main.go -config my-local/local.yaml -log-level debug maintenance
$  HTTP_TUNNEL_URL="https://..." go run cmd/gdnotify/main.go -config my-local/local.yaml -log-level debug list
$  HTTP_TUNNEL_URL="https://..." go run cmd/gdnotify/main.go -config my-local/local.yaml -log-level debug cleanup

LICENSE

MIT License

Copyright (c) 2022 IKEDA Masashi

Documentation

Index

Constants

View Source 
const (
	DetailTypeFileRemoved  = "File Removed"
	DetailTypeFileTrashed  = "File Move to trash"
	DetailTypeFileChanged  = "File Changed"
	DetailTypeDriveRemoved = "Shared Drive Removed"
	DetailTypeDriveChanged = "Drive Status Changed"
)
View Source 
const (
	DefaultDriveID = "__default__"
)

Variables

This section is empty.

Functions

func CLICommandStrings 

func CLICommandStrings() []string

CLICommandStrings returns a slice of all String values of the enum

func CredentialsBackendTypeStrings added in v0.1.0 

func CredentialsBackendTypeStrings() []string

CredentialsBackendTypeStrings returns a slice of all String values of the enum

func GetAttributeValueAs 

func GetAttributeValueAs[T types.AttributeValue](key string, values map[string]types.AttributeValue) (T, bool)

func NotificationTypeStrings 

func NotificationTypeStrings() []string

NotificationTypeStrings returns a slice of all String values of the enum

func RunModeStrings 

func RunModeStrings() []string

RunModeStrings returns a slice of all String values of the enum

func StorageTypeStrings 

func StorageTypeStrings() []string

StorageTypeStrings returns a slice of all String values of the enum

func WithCLICommand 

func WithCLICommand(cmd string) func(*RunOptions) error

func WithLocalAddress 

func WithLocalAddress(addr string) func(*RunOptions) error

func WithRunMode 

func WithRunMode(mode string) func(*RunOptions) error

Types

type App 

type App struct {
	// contains filtered or unexported fields
}

func New 

func New(cfg *Config, gcpOpts ...option.ClientOption) (*App, error)

func (*App) ChangesList 

func (app *App) ChangesList(ctx context.Context, channelID string) ([]*drive.Change, *ChannelItem, error)

func (*App) Close 

func (app *App) Close() error

func (*App) CreateChannel 

func (app *App) CreateChannel(ctx context.Context, driveID string) error

func (*App) DeleteChannel 

func (app *App) DeleteChannel(ctx context.Context, item *ChannelItem) error

func (*App) DriveIDs added in v0.4.0 

func (app *App) DriveIDs(ctx context.Context) ([]string, error)

func (*App) RotateChannel 

func (app *App) RotateChannel(ctx context.Context, item *ChannelItem) error

func (*App) Run 

func (app *App) Run(optFns ...func(*RunOptions) error) error

func (*App) RunWithContext 

func (app *App) RunWithContext(ctx context.Context, optFns ...func(*RunOptions) error) error

func (*App) SendNotification added in v0.3.0 

func (app *App) SendNotification(ctx context.Context, item *ChannelItem, changes []*drive.Change) error

func (*App) ServeHTTP 

func (app *App) ServeHTTP(w http.ResponseWriter, r *http.Request)

type CLICommand 

type CLICommand int
const (
	CLICommandList CLICommand = iota
	CLICommandServe
	CLICommandRegister
	CLICommandMaintenance
	CLICommandCleanup
	CLICommandSync
)

func CLICommandString 

func CLICommandString(s string) (CLICommand, error)

CLICommandString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func CLICommandValues 

func CLICommandValues() []CLICommand

CLICommandValues returns all values of the enum

func (CLICommand) Description 

func (cmd CLICommand) Description() string

func (CLICommand) IsACLICommand 

func (i CLICommand) IsACLICommand() bool

IsACLICommand returns "true" if the value is listed in the enum definition. "false" otherwise

func (CLICommand) String 

func (i CLICommand) String() string

type ChangeEventDetail added in v0.2.0 

type ChangeEventDetail struct {
	Subject string        `json:"subject"`
	Entity  *TargetEntity `json:"entity"`
	Actor   *drive.User   `json:"actor"`
	Change  *drive.Change `json:"change"`
}

func (*ChangeEventDetail) DetailType added in v0.2.0 

func (e *ChangeEventDetail) DetailType() string

func (*ChangeEventDetail) MarshalJSON added in v0.2.0 

func (e *ChangeEventDetail) MarshalJSON() ([]byte, error)

func (*ChangeEventDetail) Source added in v0.2.0 

func (e *ChangeEventDetail) Source(sourcePrefix string) string

type ChannelAlreadyExists 

type ChannelAlreadyExists struct {
	ChannelID string
}

func (*ChannelAlreadyExists) Error 

func (err *ChannelAlreadyExists) Error() string

type ChannelItem 

type ChannelItem struct {
	ChannelID          string
	Expiration         time.Time
	PageToken          string
	ResourceID         string
	DriveID            string
	PageTokenFetchedAt time.Time
	CreatedAt          time.Time
	UpdatedAt          time.Time
}

func NewChannelItemWithDynamoDBAttributeValues 

func NewChannelItemWithDynamoDBAttributeValues(values map[string]types.AttributeValue) *ChannelItem

func (*ChannelItem) IsAboutToExpired 

func (item *ChannelItem) IsAboutToExpired(ctx context.Context, remaining time.Duration) bool

func (*ChannelItem) ToDynamoDBAttributeValues 

func (item *ChannelItem) ToDynamoDBAttributeValues() map[string]types.AttributeValue

type ChannelNotFound 

type ChannelNotFound struct {
	ChannelID string
}

func (*ChannelNotFound) Error 

func (err *ChannelNotFound) Error() string

type Config 

type Config struct {
	RequiredVersion string `yaml:"required_version,omitempty"`

	Webhook            string                    `yaml:"webhook,omitempty"`
	Credentials        *CredentialsBackendConfig `yaml:"credentials,omitempty"`
	Expiration         time.Duration             `yaml:"expiration,omitempty"`
	Storage            *StorageConfig            `yaml:"storage,omitempty"`
	Notification       *NotificationConfig       `yaml:"notification,omitempty"`
	Drives             []*DriveConfig            `yaml:"drives,omitempty"`
	WithinModifiedTime *time.Duration            `yaml:"within_modified_time,omitempty"`
	DrivesAutoDetect   *bool                     `yaml:"drives_auto_detect,omitempty"`
	// contains filtered or unexported fields
}

Config for App

func DefaultConfig 

func DefaultConfig() *Config

func (*Config) Load 

func (cfg *Config) Load(ctx context.Context, paths ...string) error

Load loads configuration file from file paths.

func (*Config) Restrict 

func (cfg *Config) Restrict() error

Restrict restricts a configuration.

func (*Config) ValidateVersion 

func (c *Config) ValidateVersion(version string) error

ValidateVersion validates a version satisfies required_version.

type CredentialsBackend added in v0.1.0 

type CredentialsBackend interface {
	WithCredentialsClientOption(context.Context, []option.ClientOption) ([]option.ClientOption, error)
}

func NewCredentialsBackend added in v0.1.0 

func NewCredentialsBackend(ctx context.Context, cfg *CredentialsBackendConfig, awsCfg aws.Config) (CredentialsBackend, error)

type CredentialsBackendConfig added in v0.1.0 

type CredentialsBackendConfig struct {
	BackendType    CredentialsBackendType `yaml:"backend_type,omitempty"`
	ParameterName  *string                `yaml:"parameter_name,omitempty"`
	Base64Encoding bool                   `yaml:"base64encoding,omitempty"`
}

func (*CredentialsBackendConfig) Restrict added in v0.1.0 

func (cfg *CredentialsBackendConfig) Restrict() error

Restrict restricts a configuration.

type CredentialsBackendType added in v0.1.0 

type CredentialsBackendType int
const (
	CredentialsBackendTypeNone CredentialsBackendType = iota
	CredentialsBackendTypeSSMParameterStore
)

func CredentialsBackendTypeString added in v0.1.0 

func CredentialsBackendTypeString(s string) (CredentialsBackendType, error)

CredentialsBackendTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func CredentialsBackendTypeValues added in v0.1.0 

func CredentialsBackendTypeValues() []CredentialsBackendType

CredentialsBackendTypeValues returns all values of the enum

func (CredentialsBackendType) IsACredentialsBackendType added in v0.1.0 

func (i CredentialsBackendType) IsACredentialsBackendType() bool

IsACredentialsBackendType returns "true" if the value is listed in the enum definition. "false" otherwise

func (CredentialsBackendType) MarshalYAML added in v0.1.0 

func (i CredentialsBackendType) MarshalYAML() (interface{}, error)

MarshalYAML implements a YAML Marshaler for CredentialsBackendType

func (CredentialsBackendType) String added in v0.1.0 

func (i CredentialsBackendType) String() string

func (*CredentialsBackendType) UnmarshalYAML added in v0.1.0 

func (i *CredentialsBackendType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements a YAML Unmarshaler for CredentialsBackendType

type DriveConfig 

type DriveConfig struct {
	DriveID string `yaml:"drive_id,omitempty"`
}

func (*DriveConfig) Restrict 

func (cfg *DriveConfig) Restrict() error

Restrict restricts a configuration.

type DynamoDBStorage 

type DynamoDBStorage struct {
	// contains filtered or unexported fields
}

func NewDynamoDBStorage 

func NewDynamoDBStorage(ctx context.Context, cfg *StorageConfig, awsCfg aws.Config) (*DynamoDBStorage, func() error, error)

func (*DynamoDBStorage) DeleteChannel 

func (s *DynamoDBStorage) DeleteChannel(ctx context.Context, target *ChannelItem) error

func (*DynamoDBStorage) FindAllChannels 

func (s *DynamoDBStorage) FindAllChannels(ctx context.Context) (<-chan []*ChannelItem, error)

func (*DynamoDBStorage) FindOneByChannelID 

func (s *DynamoDBStorage) FindOneByChannelID(ctx context.Context, channelID string) (*ChannelItem, error)

func (*DynamoDBStorage) SaveChannel 

func (s *DynamoDBStorage) SaveChannel(ctx context.Context, item *ChannelItem) error

func (*DynamoDBStorage) UpdatePageToken 

func (s *DynamoDBStorage) UpdatePageToken(ctx context.Context, target *ChannelItem) error

type EventBridgeClient 

type EventBridgeClient interface {
	PutEvents(ctx context.Context, params *eventbridge.PutEventsInput, optFns ...func(*eventbridge.Options)) (*eventbridge.PutEventsOutput, error)
}

type EventBridgeNotification 

type EventBridgeNotification struct {
	// contains filtered or unexported fields
}

func (*EventBridgeNotification) SendChanges 

func (n *EventBridgeNotification) SendChanges(ctx context.Context, item *ChannelItem, changes []*drive.Change) error

type FileNotification 

type FileNotification struct {
	// contains filtered or unexported fields
}

func NewFileNotification 

func NewFileNotification(ctx context.Context, cfg *NotificationConfig) (*FileNotification, func() error, error)

func (*FileNotification) SendChanges 

func (n *FileNotification) SendChanges(ctx context.Context, _ *ChannelItem, changes []*drive.Change) error

type FileStorage 

type FileStorage struct {
	Items []*ChannelItem

	LockFile string
	FilePath string
}

func NewFileStorage 

func NewFileStorage(ctx context.Context, cfg *StorageConfig) (*FileStorage, func() error, error)

func (*FileStorage) DeleteChannel 

func (s *FileStorage) DeleteChannel(ctx context.Context, target *ChannelItem) error

func (*FileStorage) FindAllChannels 

func (s *FileStorage) FindAllChannels(ctx context.Context) (<-chan []*ChannelItem, error)

func (*FileStorage) FindOneByChannelID 

func (s *FileStorage) FindOneByChannelID(ctx context.Context, channelID string) (*ChannelItem, error)

func (*FileStorage) SaveChannel 

func (s *FileStorage) SaveChannel(ctx context.Context, item *ChannelItem) error

func (*FileStorage) UpdatePageToken 

func (s *FileStorage) UpdatePageToken(ctx context.Context, target *ChannelItem) error

type NoneCredentialsBackend added in v0.1.0 

type NoneCredentialsBackend struct{}

func (*NoneCredentialsBackend) WithCredentialsClientOption added in v0.1.0 

func (b *NoneCredentialsBackend) WithCredentialsClientOption(_ context.Context, orig []option.ClientOption) ([]option.ClientOption, error)

type Notification 

type Notification interface {
	SendChanges(context.Context, *ChannelItem, []*drive.Change) error
}

func NewEventBridgeNotification 

func NewEventBridgeNotification(ctx context.Context, cfg *NotificationConfig, awsCfg aws.Config) (Notification, func() error, error)

func NewNotification 

func NewNotification(ctx context.Context, cfg *NotificationConfig, awsCfg aws.Config) (Notification, func() error, error)

type NotificationConfig 

type NotificationConfig struct {
	Type      NotificationType `yaml:"type,omitempty"`
	EventBus  *string          `yaml:"event_bus,omitempty"`
	EventFile *string          `yaml:"event_file,omitempty"`
}

func (*NotificationConfig) Restrict 

func (cfg *NotificationConfig) Restrict() error

Restrict restricts a configuration.

type NotificationType 

type NotificationType int
const (
	NotificationTypeEventBridge NotificationType = iota
	NotificationTypeFile
)

func NotificationTypeString 

func NotificationTypeString(s string) (NotificationType, error)

NotificationTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func NotificationTypeValues 

func NotificationTypeValues() []NotificationType

NotificationTypeValues returns all values of the enum

func (NotificationType) IsANotificationType 

func (i NotificationType) IsANotificationType() bool

IsANotificationType returns "true" if the value is listed in the enum definition. "false" otherwise

func (NotificationType) MarshalYAML 

func (i NotificationType) MarshalYAML() (interface{}, error)

MarshalYAML implements a YAML Marshaler for NotificationType

func (NotificationType) String 

func (i NotificationType) String() string

func (*NotificationType) UnmarshalYAML 

func (i *NotificationType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements a YAML Unmarshaler for NotificationType

type RunMode 

type RunMode int
const (
	RunModeCLI RunMode = iota
	RunModeWebhook
	RunModeMaintainer
	RunModeSyncer
)

func DefaultRunMode 

func DefaultRunMode() RunMode

func RunModeString 

func RunModeString(s string) (RunMode, error)

RunModeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func RunModeValues 

func RunModeValues() []RunMode

RunModeValues returns all values of the enum

func (RunMode) IsARunMode 

func (i RunMode) IsARunMode() bool

IsARunMode returns "true" if the value is listed in the enum definition. "false" otherwise

func (RunMode) String 

func (i RunMode) String() string

type RunOptions 

type RunOptions struct {
	Mode         RunMode
	LocalAddress string
	CLICommand   CLICommand
}

type SSMParameterStoreCredentialsBackend added in v0.1.0 

type SSMParameterStoreCredentialsBackend struct {
	// contains filtered or unexported fields
}

func NewSSMParameterStoreCredentialsBackend added in v0.1.0 

func NewSSMParameterStoreCredentialsBackend(ctx context.Context, cfg *CredentialsBackendConfig, awsCfg aws.Config) (*SSMParameterStoreCredentialsBackend, error)

func (*SSMParameterStoreCredentialsBackend) WithCredentialsClientOption added in v0.1.0 

func (cb *SSMParameterStoreCredentialsBackend) WithCredentialsClientOption(ctx context.Context, orig []option.ClientOption) ([]option.ClientOption, error)

type Storage 

type Storage interface {
	FindAllChannels(context.Context) (<-chan []*ChannelItem, error)
	FindOneByChannelID(context.Context, string) (*ChannelItem, error)
	UpdatePageToken(context.Context, *ChannelItem) error
	SaveChannel(context.Context, *ChannelItem) error
	DeleteChannel(context.Context, *ChannelItem) error
}

func NewStorage 

func NewStorage(ctx context.Context, cfg *StorageConfig, awsCfg aws.Config) (Storage, func() error, error)

type StorageConfig 

type StorageConfig struct {
	Type      StorageType `yaml:"type,omitempty"`
	TableName *string     `yaml:"table_name,omitempty"`
	DataFile  *string     `yaml:"data_file,omitempty"`
	LockFile  *string     `yaml:"lock_file,omitempty"`
}

func (*StorageConfig) Restrict 

func (cfg *StorageConfig) Restrict() error

Restrict restricts a configuration.

type StorageType 

type StorageType int
const (
	StorageTypeDynamoDB StorageType = iota
	StorageTypeFile
)

func StorageTypeString 

func StorageTypeString(s string) (StorageType, error)

StorageTypeString retrieves an enum value from the enum constants string name. Throws an error if the param is not part of the enum.

func StorageTypeValues 

func StorageTypeValues() []StorageType

StorageTypeValues returns all values of the enum

func (StorageType) IsAStorageType 

func (i StorageType) IsAStorageType() bool

IsAStorageType returns "true" if the value is listed in the enum definition. "false" otherwise

func (StorageType) MarshalYAML 

func (i StorageType) MarshalYAML() (interface{}, error)

MarshalYAML implements a YAML Marshaler for StorageType

func (StorageType) String 

func (i StorageType) String() string

func (*StorageType) UnmarshalYAML 

func (i *StorageType) UnmarshalYAML(unmarshal func(interface{}) error) error

UnmarshalYAML implements a YAML Unmarshaler for StorageType

type TargetEntity added in v0.2.0 

type TargetEntity struct {
	Id          string `json:"id"`
	Kind        string `json:"kind"`
	Name        string `json:"name"`
	CreatedTime string `json:"createdTime"`
}

Source Files

View all Source files

Directories

Path Synopsis
cmd
gdnotify

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.