gcs

README

google cloud storage

Usage

source:
  name: gcs
  config:
    project_id: google-project-id
    extract_blob: true
    credentials_json:
      {
        "type": "service_account",
        "private_key_id": "xxxxxxx",
        "private_key": "xxxxxxx",
        "client_email": "xxxxxxx",
        "client_id": "xxxxxxx",
        "auth_uri": "https://accounts.google.com/o/oauth2/auth",
        "token_uri": "https://oauth2.googleapis.com/token",
        "auth_provider_x509_cert_url": "xxxxxxx",
        "client_x509_cert_url": "xxxxxxx",
      }
    exclude: [bucket_a, bucket_b]

Inputs

Key	Value	Example	Description
project_id	string	my-project	Cloud Storage Project ID	required
extract_blob	boolean	true	Extract blob metadata inside a bucket	optional
credentials_json	string	{"private_key": .., "private_id": ...}	Service Account in JSON string	optional
exclude	[]string	["bucket_a","bucket_b"]	Array of bucket names to excluded from crawling	optional

Notes

Leaving credentials_json blank will default to Google's default authentication. It is recommended if Meteor instance runs inside the same Google Cloud environment as the Google Cloud Storage project.

Outputs

Field	Sample Value
resource.urn	project_id/bucket_name
resource.name	bucket_name
resource.service	gcs
location	ASIA
storage_type	STANDARD
labels	[]{key:value}
timestamps.created_at.seconds	1551082913
timestamps.created_at.nanos	1551082913

Blob

Field	Sample Value
urn	project_id/bucket_name/blob_path
name	blob_path
size	311
deleted_at.seconds	1551082913
expired_at.seconds	1551082913
labels	[]{key:value}
ownership.owners	[]{name:serviceaccountname@project.gserviceaccount.com}
timestamps.created_at.seconds	1551082913
timestamps.created_at.nanos	1551082913
timestamps.updated_at.seconds	1551082913
timestamps.updated_at.nanos	1551082913

Contributing

Refer to the contribution guidelines for information on contributing to this module.

Documentation

Index

• type Config
• type Extractor
  • func New(logger log.Logger) *Extractor
  • func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)
  • func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Types

type Config

type Config struct {
	ProjectID          string   `json:"project_id" yaml:"project_id" mapstructure:"project_id" validate:"required"`
	ServiceAccountJSON string   `json:"service_account_json" yaml:"service_account_json" mapstructure:"service_account_json"`
	ExtractBlob        bool     `json:"extract_blob" yaml:"extract_blob" mapstructure:"extract_blob"`
	Exclude            []string `json:"exclude" yaml:"exclude" mapstructure:"exclude"`
}

Config holds the set of configuration for the extractor

type Extractor

type Extractor struct {
	plugins.BaseExtractor
	// contains filtered or unexported fields
}

Extractor manages the extraction of data from the google cloud storage

func New

func New(logger log.Logger) *Extractor

New returns a pointer to an initialized Extractor Object

func (*Extractor) Extract

func (e *Extractor) Extract(ctx context.Context, emit plugins.Emit) (err error)

func (*Extractor) Init

func (e *Extractor) Init(ctx context.Context, config plugins.Config) (err error)

Init initializes the extractor