gcs

README

google cloud storage

Usage

source:
  name: gcs
  config:
    project_id: google-project-id
    extract_blob: true
     # Only one of service_account_base64 / service_account_json is needed. 
    # If both are present, service_account_base64 takes precedence
    service_account_base64: _________BASE64_ENCODED_SERVICE_ACCOUNT_________________
    service_account_json:
      {
        "type": "service_account",
        "private_key_id": "xxxxxxx",
        "private_key": "xxxxxxx",
        "client_email": "xxxxxxx",
        "client_id": "xxxxxxx",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "xxxxxxx",
        "client_x509_cert_url": "xxxxxxx",
      }
    exclude: [bucket_a, bucket_b]

Inputs

Key	Value	Example	Description
project_id	string	my-project	Cloud Storage Project ID	required
extract_blob	boolean	true	Extract blob metadata inside a bucket	optional
service_account_base64	string	____BASE64_ENCODED_SERVICE_ACCOUNT____	Service Account in base64 encoded string. Takes precedence over service_account_json value	optional
service_account_json	string	{"private_key": .., "private_id": ...}	Service Account in JSON string	optional
exclude	[]string	["bucket_a","bucket_b"]	Array of bucket names to excluded from crawling	optional

Notes

Leaving service_account_base64 and service_account_json blank will default to Google's default authentication. It is recommended if Meteor instance runs inside the same Google Cloud environment as the Google Cloud Storage project.

Outputs

Field	Sample Value
resource.urn	project_id/bucket_name
resource.name	bucket_name
resource.service	gcs
location	ASIA
storage_type	STANDARD
labels	[]{key:value}
timestamps.created_at.seconds	1551082913
timestamps.created_at.nanos	1551082913

Blob

Field	Sample Value
urn	project_id/bucket_name/blob_path
name	blob_path
size	311
deleted_at.seconds	1551082913
expired_at.seconds	1551082913
labels	[]{key:value}
ownership.owners	[]{name:serviceaccountname@project.gserviceaccount.com}
timestamps.created_at.seconds	1551082913
timestamps.created_at.nanos	1551082913
timestamps.updated_at.seconds	1551082913
timestamps.updated_at.nanos	1551082913

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

• type Config
• type Extractor
  • func New(logger log.Logger, newClient NewClientFunc) *Extractor
  • func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)
  • func (e *Extractor) Init(ctx context.Context, config plugins.Config) error
• type NewClientFunc

Types

type Config

type Config struct {
	ProjectID string `json:"project_id" yaml:"project_id" mapstructure:"project_id" validate:"required"`
	// ServiceAccountBase64 takes precedence over ServiceAccountJSON field
	ServiceAccountBase64 string   `json:"service_account_base64" yaml:"service_account_base64" mapstructure:"service_account_base64"`
	ServiceAccountJSON   string   `json:"service_account_json" yaml:"service_account_json" mapstructure:"service_account_json"`
	ExtractBlob          bool     `json:"extract_blob" yaml:"extract_blob" mapstructure:"extract_blob"`
	Exclude              []string `json:"exclude" yaml:"exclude" mapstructure:"exclude"`
}

Config holds the set of configuration for the extractor

type Extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the extraction of data from the google cloud storage

func New

func New(logger log.Logger, newClient NewClientFunc) *Extractor

New returns a pointer to an initialized Extractor Object

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)

func (*Extractor) Init

func (e *Extractor) Init(ctx context.Context, config plugins.Config) error

Init initializes the extractor

type NewClientFunc

type NewClientFunc func(ctx context.Context, logger log.Logger, config Config) (*storage.Client, error)