commoncrawl

README

goCommonCrawl

goCommonCrawl extracts web data from Common Crawl Web archive, that is located on Amazon S3 storage, using their index API server.

Release

Compiled version available at https://github.com/karust/goCommonCrawl/releases/tag/1

Installation

go get -u github.com/karust/goCommonCrawl

Usage

If you need to fetch some URL without concurrency:

package main

import (
	"log"
	cc "github.com/karust/gocommoncrawl"
)

func main() {
    // Get information about `example.com/` URL from  `CC-MAIN-2019-22` archive
	pages, err := cc.GetPagesInfo("CC-MAIN-2019-22", "example.com/", 45)
	if err != nil {
		log.Fatalln(err)
	}

    // Parse retrieved pages information and save it in `./data`
	cc.SaveContent(pages, "./data", 45)
}

Concurrent way to do things:

func main() {
	// Create channel for results
	resChan := make(chan cc.Result)

	// Some URLs to fetch pages from
	sites := []string{"medium.com/", "example.com/", "tutorialspoint.com/"}

	// Make save folder and start goroutine for each URL
	for _, url := range sites {
		// Configure request
		commonConfig := cc.Config{
			ResultChan: resChan,
			Timeout:    30,
			// Version of archive
			CrawlDB: "CC-MAIN-2019-22",
			// Wait time between AWS S3 downloads in milliseconds
			WaitMS: 53,
			// Extensions to save
			Extensions: []string{".html", ".pdf", ".doc", ".txt"},
			// Max amount of files to save
			MaxAmount: 20,
		}

		saveFolder := "./data/" + cc.EscapeURL(url)
		go cc.FetchURLData(url, saveFolder, commonConfig)
	}

	// Listen for results from goroutines
	for r := range resChan {
		if r.Error != nil {
			fmt.Printf("Error occured: %v\n", r.Error)
		} else if r.Progress > 0 {
			fmt.Printf("Progress %v: %v/%v\n", r.URL, r.Progress, r.Total)
		}
	}
}

In the result, you should get folders with files (mostly HTML and robot.txt) that belong to given URLs.

Documentation

Index

• func ChangeIndexServer(server string)
• func EscapeURL(url string) string
• func ExtensionByContent(content []byte) string
• func FetchURLData(url string, saveto string, config Config)
• func IsExtensionExist(allowedExtensions []string, extension string) bool
• func SaveContent(pages []IndexAPI, saveTo string, timeout int) error
• type Config
• type IndexAPI
  • func GetPagesInfo(crawl string, url string, timeout int) ([]IndexAPI, error)
• type Result

Functions

func ChangeIndexServer

func ChangeIndexServer(server string)

ChangeIndexServer ... changes default address `http://index.commoncrawl.org/`

func EscapeURL

func EscapeURL(url string) string

EscapeURL ... Makes possible to create files with URL name by replacing system chars with %<char code>

func ExtensionByContent

func ExtensionByContent(content []byte) string

ExtensionByContent ... Returns extension of file by detecting its MIME type, `.none` returned if no MIME found

func FetchURLData

func FetchURLData(url string, saveto string, config Config)

FetchURLData ... Fetches pages located on the given URL from Common Crawl archive using Index API and saves them to the pointed location Set `crawlDB` argument empty if not sure what it is

func IsExtensionExist

func IsExtensionExist(allowedExtensions []string, extension string) bool

IsExtensionExist ...Return `true` if there is extension in allowed or false if there is no returns `true` if length of `allowedExtensions` == 0

func SaveContent

func SaveContent(pages []IndexAPI, saveTo string, timeout int) error

SaveContent ... Saves pages or text that were found in Common Crawl to choosen folder

pages: info about found web pages from `getIndex` function
saveTo: destination fodler, where save fetched web data
timeout: timeout in seconds, default 30

Types

type Config

type Config struct {
	ResultChan chan Result
	Timeout    int
	CrawlDB    string
	WaitMS     int
	Extensions []string
	MaxAmount  int
}

Config ... Holds configurations of crawler

type IndexAPI

type IndexAPI struct {
	Urlkey       string `json:"urlkey,omitempty"`
	Timestamp    string `json:"timestamp,omitempty"`
	Charset      string `json:"charset,omitempty"`
	Mime         string `json:"mime,omitempty"`
	Languages    string `json:"languages,omitempty"`
	MimeDetected string `json:"mimedetected,omitempty"`
	Digest       string `json:"digest,omitempty"`
	Offset       string `json:"offset,omitempty"`
	URL          string `json:"url,omitempty"`
	Length       string `json:"length,omitempty"`
	Status       string `json:"status,omitempty"`
	Filename     string `json:"filename,omitempty"`
}

IndexAPI ... API Response structure from `index.commoncrawl.org`

func GetPagesInfo

func GetPagesInfo(crawl string, url string, timeout int) ([]IndexAPI, error)

GetPagesInfo ... Makes request to commoncrawl index API to gather all offsets that contain pointed URL

crawl: Crawl a database which should be used, e.g 'CC-MAIN-2019-22';
url: URL of a site, offsets and other info of which should be returned.
timeout: timeout in seconds, default 30

Returns a list of JSON objects with information about each file offset and other data.

type Result

type Result struct {
	URL      string
	Progress int
	Total    int
	Error    error
	Done     bool
}

Result ... of `FetchURLData` function execution