ga

Documentation

Index

• func DownloadFile(filepath string, url string) error
• func GUnzip(filepath string) error
• func GetDownloadLink(year, month, day, hour int) (constructURL string, filePath string)
• func IsValidDate(date string) bool
• func PushToMongo(filepath string)
• type GithubEvent
• type WriteCounter
  • func (wc WriteCounter) PrintProgress()
  • func (wc *WriteCounter) Write(p []byte) (int, error)

Functions

func DownloadFile

func DownloadFile(filepath string, url string) error

DownloadFile will download a url to a local file. It's efficient because it will write as it downloads and not load the whole file into memory.

func GUnzip

func GUnzip(filepath string) error

GUnzip unzip gunzip file

func GetDownloadLink

func GetDownloadLink(year, month, day, hour int) (constructURL string, filePath string)

GetDownloadLink returns string depending

func IsValidDate

func IsValidDate(date string) bool

IsValidDate returns true if matches ^[0-9]{10,10}$ vlidates 2021010101

func PushToMongo

func PushToMongo(filepath string)

PushToMongo goes into each line of file and sends json object to mongo

Types

type GithubEvent

type GithubEvent struct {
	ID    string `json:"id"`
	Type  string `json:"type"`
	Actor struct {
		ID           int    `json:"id"`
		Login        string `json:"login"`
		DisplayLogin string `json:"display_login"`
		GravatarID   string `json:"gravatar_id"`
		URL          string `json:"url"`
		AvatarURL    string `json:"avatar_url"`
	} `json:"actor"`
	Repo struct {
		ID   int    `json:"id"`
		Name string `json:"name"`
		URL  string `json:"url"`
	} `json:"repo"`
	Payload struct {
		Ref        string `json:"ref"`
		RefType    string `json:"ref_type"`
		PusherType string `json:"pusher_type"`
	} `json:"payload"`
	Public    bool      `json:"public"`
	CreatedAt time.Time `json:"created_at"`
}

GithubEvent struct to hold github events from json

type WriteCounter

type WriteCounter struct {
	Total uint64
}

WriteCounter counts the number of bytes written to it. It implements to the io.Writer interface and we can pass this into io.TeeReader() which will report progress on each write cycle.

func (WriteCounter) PrintProgress

func (wc WriteCounter) PrintProgress()

PrintProgress prints progress to stdout

func (*WriteCounter) Write

func (wc *WriteCounter) Write(p []byte) (int, error)