scraper

README

Prompt API - Scraper - Golang Package

PromptAPI struct is a simple golang wrapper for scraper api with few more extra cream and sugar.

Requirements

1. You need to signup for Prompt API
2. You need to subscribe scraper api, test drive is free!!!
3. You need to set PROMPTAPI_TOKEN environment variable after subscription.

then;

$ go get -u github.com/promptapi/scraper-go

Example Usage

// main.go

Development

Available rake tasks:

$ rake -T
rake default                    # Default task, show avaliable tasks
rake release:check              # Do release check
rake release:publish[revision]  # Publish project with revision: major,minor,patch, default: patch
rake serve_doc[port]            # Run doc server
rake test[verbose]              # Run tests

• Run tests: rake test or rake test[-v]
• Run doc server: rake serve_doc or rake serve_doc[9000]

Release package (if you have write access):

1. Commit your changes
2. Run rake release:check
3. If all goes ok, run rake release:publish

---

---

License

This project is licensed under MIT

---

Contributer(s)

• Prompt API - Creator, maintainer

---

Contribute

All PR’s are welcome!

1. fork (https://github.com/promptapi/scraper-go/fork)
2. Create your branch (git checkout -b my-feature)
3. commit yours (git commit -am 'Add awesome features...')
4. push your branch (git push origin my-feature)
5. Than create a new Pull Request!

This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the code of conduct.

---

Documentation

Index

• type ErrorResponse
• type Params
• type PromptAPI
  • func (pa PromptAPI) Save(filename string, result *Result) (n int, err error)
  • func (pa PromptAPI) Scrape(params *Params, result *Result) error
• type Result

Types

type ErrorResponse

type ErrorResponse struct {
	Message string `json:"message"`
}

ErrorResponse represents response errors from Prompt API - Scraper

type Params

type Params struct {
	URL          string
	AuthPassword string
	AuthUsername string
	Cookie       string
	Country      string
	Referer      string
	Selector     string
}

Params represents arguments for Scrape method

type PromptAPI

type PromptAPI struct{}

PromptAPI represents type

func (PromptAPI) Save

func (pa PromptAPI) Save(filename string, result *Result) (n int, err error)

Save saves fetched data to given file

func (PromptAPI) Scrape

func (pa PromptAPI) Scrape(params *Params, result *Result) error

Scrape makes API call to Prompt API - Scraper and returns result

type Result

type Result struct {
	Data         string                 `json:"data,omitempty"`
	DataSelector []string               `json:"data-selector,omitempty"`
	URL          string                 `json:"url"`
	Headers      map[string]interface{} `json:"headers"`
}

Result represents incoming JSON data from Prompt API - Scraper