Documentation ¶
Index ¶
- Constants
- Variables
- func CrawlBucket(logger *log.Logger, qc *cque.Client, j *cque.Job, ...) error
- func Getlogsfroms3(bucket, key string, appconfig map[string]interface{}) ([]byte, error)
- func Getsubfolder(bucket, path string) ([]string, error)
- func Isinternal(ip string) bool
- func Listfileinfolder(bucket, path string, depth int) ([]string, error)
- func ParseS3Log(logger *log.Logger, qc *cque.Client, j *cque.Job, ...) error
- type Connection
- type CrawlBucketJob
- type JobFunc
- type JobFuncWrapper
- type ParseS3LogJob
Constants ¶
View Source
const (
KeyCrawlBucket = "crawl_bucket"
)
View Source
const (
KeyParseS3Log = "parse_s3_log"
)
Variables ¶
View Source
var ( ErrImmediateReschedule = errors.New("reschedule ASAP") ErrDidNotReschedule = errors.New("no need to reschedule, we are done") )
Functions ¶
func CrawlBucket ¶
func CrawlBucket(logger *log.Logger, qc *cque.Client, j *cque.Job, appconfig map[string]interface{}) error
Example - we can wrap the other function
func Getlogsfroms3 ¶
Getlogsfroms3 - Download flowlog from s3, decompress using gzip
func Getsubfolder ¶
func Isinternal ¶
Check If ip is not between internal ranges
10.0.0.0 – 10.255.255.255 172.16.0.0 – 172.31.255.255 192.168.0.0 – 192.168.255.255
Types ¶
type Connection ¶
func (Connection) String ¶
func (c Connection) String() string
type CrawlBucketJob ¶
type JobFunc ¶
type JobFunc func(logger *log.Logger, qc *que.Client, job *que.Job, appconfig map[string]interface{}) error
JobFunc should do a thing. Return either: nil => No error, move onto next job ErrImmediateReschedule => wrapper try it again immediately. ErrDidNotReschedule => wrapper will rely on queue lib to reschedule or retry. any other error => wrapper will rely on que to reschedule
type JobFuncWrapper ¶
type ParseS3LogJob ¶
Click to show internal directories.
Click to hide internal directories.