goscraper

README

goscraper

Web Scraping library in GO

Documentation

Index

• type Field
• type Scrape
  • func (s *Scrape) ProcessURL(url string) (map[string]interface{}, error)

Types

type Field

type Field struct {
	// XPath selector
	Selector string
	// Callback is used to post process selected field
	Callback func(n *xml.Node) (interface{}, error)
}

Field defines how data should be selected and process

type Scrape

type Scrape struct {
	// definitions of fields that will be scraped
	Fields map[string]Field
	// HTTP headers used in query
	Headers map[string]string
}

Scrape type defines what to scrape and headers used in HTTP request, and allows to process data

func (*Scrape) ProcessURL

func (s *Scrape) ProcessURL(url string) (map[string]interface{}, error)

ProcessURL method is used to fetch website contents and process it's data