archivewrangler

Documentation

Index

• func ArchiveMetaCleaner()
• func DataCrawlerSingle(structArchive schema.Archive, archiveURL string, apiKey string, year string, ...)
• type CleanedCorpus
• type CleanerResponse

Functions

func ArchiveMetaCleaner

func ArchiveMetaCleaner()

ArchiveMetaCleaner - function to clean all corpus data

func DataCrawlerSingle

func DataCrawlerSingle(structArchive schema.Archive, archiveURL string, apiKey string, year string, month string)

DataCrawlerSingle - get data for single URL

Types

type CleanedCorpus

type CleanedCorpus struct {
	Abstract      string `json:"abstract"`
	WebURL        string `json:"web_url"`
	PublishedDate string `json:"pub_date"`
	Headline      string `json:"headline"`
}

CleanedCorpus - struct type for cleaned corpus

type CleanerResponse

type CleanerResponse struct {
	Response map[string]interface{} `json:"response"`
}

CleanerResponse - Struct to Unmarshal json byte data