serp

README

Google Search Results GoLang API

This Golang package enables to scrape and parse Google results using SERP API. Feel free to fork this repository to add more backends.

This project is an implementation of Serp API in Golang 1.12 There is no dependency for this project.

This Go module is meant to scrape and parse Google results using SerpApi. The following services are provided:

• Search API
• Location API
• Search Archive API
• Account API

Serp API provides a script builder to get you started quickly.

An implementation example is provided here.

demo/demo.go

You take a look to our test.

test/google_seach_results_test.go

The full documentation is available here.

Feel free to fork this repository to add more backends.

Installation

Go 1.12 must be already installed.

go get -u github.com/serpapi/google_search_results_golang

Quick start

package main

import (
	"fmt"
	g "github.com/serpapi/google-search-results-golang"
)

parameter := map[string]string{
    "q":            "Coffee",
    "location":     "Portland"
}

query := newGoogleSearch(parameter)
results, err := query.GetJSON()
organicResult := results["organic_results"].([]interface{})
first_result := organicResult[0].(map[string]interface{})
fmt.Println(first_result["title"].(string))

This example runs a search about "coffee" using your secret api key.

The Serp API service (backend)

• searches on Google using the client: q = "coffee"
• parses the messy HTML responses
• return a standardizes JSON response The Ruby class GoogleSearchResults
• Format the request to Serp API server
• Execute GET http request
• Parse JSON into Ruby Hash using JSON standard library provided by Ruby Et voila..

Example

• How to set SERP API key
• Search API capability
• Example by specification
• Location API
• Search Archive API
• Account API

How to set SERP API key

The Serp API key can be set globally using a singleton pattern.

GoogleSearchResults.serp_api_key_default = "Your Private Key"
client = GoogleSearchResults(parameter)

The Serp API key can be provided for each client.

client = GoogleSearchResults(parameter, "Your Private Key")

Search API capability

parameter = {
  "q": "query",
  "google_domain": "Google Domain",
  "location": "Location Requested",
  "device": device,
  "hl": "Google UI Language",
  "gl": "Google Country",
  "safe": "Safe Search Flag",
  "num": "Number of Results",
  "start": "Pagination Offset",
  "serp_api_key": "Your SERP API Key",
  "tbm": "nws|isch|shop",
  "tbs": "custom to be search criteria",
  "async": true|false, # allow async 
  "output": "json|html" # output format
}

# define the search client
client := newGoogleSearch(parameter)

# override an existing parameter
client.parameter["location"] = "Portland,Oregon,United States"

# search format return as raw html
data, err := client.GetHTML()

# search format returns a json
data, err := client.GetJSON()

(the full documentation)[https://serpapi.com/search-api]

see below for more hands on examples.

Example by specification

We love true open source, continuous integration and Test Drive Development (TDD). We are using RSpec to test our infrastructure around the clock to achieve the best QoS (Quality Of Service).

The directory test/ includes specification/examples.

Set your api key.

export API_KEY="your secret key"

make test

Location API

var locationList SerpResponseArray
var err error
locationList, err = GetLocation("Austin", 3)

if err != nil {
  log.Println(err)
}
log.Println(locationList)

rsp contains the first 3 location matching Austin (Texas, Texas, Rochester)

Search Archive API

Run a search then get search result from the archive using the search archive API.

parameter := map[string]string{
  "api_key":  "your user key",
  "q":        "Coffee",
  "location": "Portland"
  }

client := NewGoogleSearch(parameter)
rsp, err := client.GetJSON()

if err != nil {
  log.Println("unexpected error", err)
}

searchID := rsp["search_metadata"].(map[string]interface{})["id"].(string)
searchArchive, err := client.GetSearchArchive(searchID)
if err != nil {
  log.Println(err)
  return
}

searchIDArchive := searchArchive["search_metadata"].(map[string]interface{})["id"].(string)
if searchIDArchive != searchID {
  log.Println("search_metadata.id do not match", searchIDArchive, searchID)
}

log.Println(searchIDArchive)

it prints the search ID from the archive.

Account API

var data SerpResponse
var err error
data, err = GetAccount()

if err != nil {
  log.Println(err)
  return
}
log.Println(data)

data contains the account information.

Change log

• Export NewGoogleSearch outside of the package.

Conclusion

Serp API supports Google Images, News, Shopping and more.. To enable a type of search, the field tbm (to be matched) must be set to:

• isch: Google Images API.
• nws: Google News API.
• shop: Google Shopping API.
• any other Google service should work out of the box.
• (no tbm parameter): regular Google client.

The field tbs allows to customize the search even more.

The full documentation is available here.

Contributing

Contributions are welcome, feel to submit a pull request!

To run the tests:

make test

Documentation

Index

• type Info
• type Meta
• type OrganicResult
• type Params
• type Query
  • func NewGoogleSearch(parameter map[string]string) Query
  • func (sq *Query) GetHTML() (*string, error)
  • func (sq *Query) GetJSON() (Results, error)
  • func (sq *Query) GetSearchArchive(searchID string) (Results, error)
• type Response
• type ResponseArray
  • func GetLocation(q string, limit int) (ResponseArray, error)
• type Results
• type ShoppingResult

Types

type Info

type Info struct {
	QueryDisplayed string `json:"query_displayed"`
}

Info is search information

type Meta

type Meta struct {
	CreatedAt      string  `json:"created_at"`
	GoogleURL      string  `json:"google_url"`
	ID             string  `json:"id"`
	JSONEndpoint   string  `json:"json_endpoint"`
	ProcessedAt    string  `json:"processed_at"`
	RawHTMLFile    string  `json:"raw_html_file"`
	Status         string  `json:"status"`
	TotalTimeTaken float64 `json:"total_time_taken"`
}

Meta ...

type OrganicResult

type OrganicResult struct {
	Title          string  `json:"title,omitempty"`
	CachedPageLink string  `json:"cached_page_link,omitempty"`
	DisplayedLink  string  `json:"displayed_link,omitempty"`
	Link           string  `json:"link,omitempty"`
	Source         string  `json:"source"`
	Snippet        string  `json:"snippet,omitempty"`
	Price          string  `json:"price,omitempty"`
	ExtractedPrice float64 `json:"extracted_price"`
}

OrganicResult ...

type Params

type Params struct {
	Device            string `json:"device"`
	Engine            string `json:"engine"`
	Filter            string `json:"filter"`
	Gl                string `json:"gl"`
	GoogleDomain      string `json:"google_domain"`
	Hl                string `json:"hl"`
	LocationRequested string `json:"location_requested"`
	LocationUsed      string `json:"location_used"`
	Q                 string `json:"q"`
	Tbm               string `json:"tbm"`
}

Params are search parameters

type Query

type Query struct {
	// contains filtered or unexported fields
}

Query hold query parameter

func NewGoogleSearch

func NewGoogleSearch(parameter map[string]string) Query

NewGoogleSearch initialize the query

func (*Query) GetHTML

func (sq *Query) GetHTML() (*string, error)

GetHTML returns html as a string

func (*Query) GetJSON

func (sq *Query) GetJSON() (Results, error)

GetJSON returns SerpResponse containing

func (*Query) GetSearchArchive

func (sq *Query) GetSearchArchive(searchID string) (Results, error)

GetSearchArchive retrieve search from the archive using the Search Archive API

type Response

type Response map[string]interface{}

Response hold response

type ResponseArray

type ResponseArray []interface{}

ResponseArray hold response array

func GetLocation

func GetLocation(q string, limit int) (ResponseArray, error)

GetLocation returns closest location

type Results

type Results struct {
	OrganicResults    []OrganicResult  `json:"organic_results"`
	SearchInformation Info             `json:"search_information"`
	SearchMetadata    Meta             `json:"search_metadata"`
	SearchParameters  Params           `json:"search_parameters"`
	SerpapiPagination interface{}      `json:"serpapi_pagination"`
	ShoppingResults   []ShoppingResult `json:"shopping_results"`
	ErrorMessage      string           `json:"error"`
}

Results ...

type ShoppingResult

type ShoppingResult struct {
	Extensions        []string `json:"extensions"`
	Link              string   `json:"link"`
	Position          int      `json:"position"`
	ProductID         string   `json:"product_id"`
	Rating            int      `json:"rating"`
	Reviews           int      `json:"reviews"`
	SerpapiProductAPI string   `json:"serpapi_product_api"`
	Source            string   `json:"source"`
	Snippet           string   `json:"snippet"`
	Thumbnail         string   `json:"thumbnail"`
	Title             string   `json:"title"`
	Price             string   `json:"price"`
	ExtractedPrice    float64  `json:"extracted_price"`
}

ShoppingResult ...