scraper

Documentation

Index

• type RedirectError
  • func (r RedirectError) Error() string
• type Scraper
  • func NewScraper() Scraper
  • func (scrap *Scraper) GetPart(URL string) (*models.Part, error)
  • func (scrap *Scraper) GetPartList(URL string) (*models.PartList, error)
  • func (scrap *Scraper) RandomizeUserAgent()
  • func (scrap *Scraper) SearchPCParts(searchTerm string, region string) ([]models.SearchPart, error)
  • func (scrap *Scraper) UpdateHeaders(site string, newHeaders map[string]string)

Types

type RedirectError

type RedirectError struct {
	URL string
}

func (RedirectError) Error

func (r RedirectError) Error() string

type Scraper

type Scraper struct {
	Collector *colly.Collector
	Headers   map[string]map[string]string
}

func NewScraper

func NewScraper() Scraper

NewScraper initializes a new instance of the Scraper type and returns it. It creates a new collector, sets the async and AllowURLRevisit properties to true, and initializes an empty Headers map with the "global" site.

func (*Scraper) GetPart

func (scrap *Scraper) GetPart(URL string) (*models.Part, error)

GetPart retrieves information about a specific part from the given URL. It returns a pointer to models.Part and an error. If the URL is invalid, it returns an error.

func (*Scraper) GetPartList

func (scrap *Scraper) GetPartList(URL string) (*models.PartList, error)

GetPartList retrieves a list of parts from the given PCPartPicker URL. It returns a pointer to models.PartList and an error. If the URL is invalid, it returns an error.

func (*Scraper) RandomizeUserAgent

func (scrap *Scraper) RandomizeUserAgent()

RandomizeUserAgent is a method of the Scraper struct. It sets a random User-Agent in the Collector headers and logs the chosen User-Agent for every request made by the collector.

func (*Scraper) SearchPCParts

func (scrap *Scraper) SearchPCParts(searchTerm string, region string) ([]models.SearchPart, error)

SearchPCParts retrieves a list of parts from the given search term and region. It returns a slice of models.SearchPart and an error. If the region is invalid, it returns an error.

func (*Scraper) UpdateHeaders

func (scrap *Scraper) UpdateHeaders(site string, newHeaders map[string]string)

UpdateHeaders updates the headers for the given site with the provided newHeaders map. It updates the headers for the "global" site as well. It also sets the headers for each request made by the Collector.