netsurfer

README

netsurfer

netsurfer is a very lightweight scraping framework.
This has only the minimum function necessary for scraping.

What can be done

• You can know the ranking of the web site.
• You can know the URL displayed when searching with the google engine.
• You can get HTML.

Installation

$ go get -u github.com/nakabonne/netsurfer

Usage

If you want to know more, please read examples.

import (
	"fmt"
	"net/url"
	"github.com/ryonakao/netsurfer"
)
// If you want to know the rank of a page, please write the following code.
u, _ := url.Parse("https://qiita.com/ryonakao")
rank, _ := netsurfer.GetRank(u, "ryonakao", 2)
fmt.Println("Rank is ", rank)

// If you want to know the search result URL, please write following
urls, _ := netsurfer.OrganicSearch("ruby", 3)

for _, url := range urls {
	// If you want to know the title, please write the following
	title, _ := netsurfer.GetTitle(url.String())
	fmt.Println("Title is ", title)
}

dependencies

• goquery

License

netsurfer source code is available under the MIT License.

Documentation

Index

• func GetHTML(url string) (html string, err error)
• func GetRank(targetURL *url.URL, keyword string, depth int) (rank int, err error)
• func GetTitle(baseURL string) (title string, err error)
• func OrganicSearch(keyword string, depth int) (urls []*url.URL, err error)

Functions

func GetHTML

func GetHTML(url string) (html string, err error)

GetHTML returns the response HTML when requesting for the given URL

func GetRank

func GetRank(targetURL *url.URL, keyword string, depth int) (rank int, err error)

GetRank returns the rank of the specified page when you search with the specified keyword. If you want to search up to 3 pages, please give 3 to depth.

func GetTitle

func GetTitle(baseURL string) (title string, err error)

GetTitle returns the title tag of the HTML file indicated by the given URL

func OrganicSearch

func OrganicSearch(keyword string, depth int) (urls []*url.URL, err error)

OrganicSearch returns the page URL of organic search. If you want to search up to 3 pages, please give 3 to depth.