webrisk

README

Reference Implementation for the Usage of Google Cloud WebRisk APIs (Beta)

The webrisk Go package can be used with the Google Cloud WebRisk APIs (Beta) to access the Google Cloud WebRisk lists of unsafe web resources. Inside the cmd sub-directory, you can find two programs: wrlookup and wrserver. The wrserver program creates a proxy local server to check URLs and a URL redirector to redirect users to a warning page for unsafe URLs. The wrlookup program is a command line service that can also be used to check URLs.

This README.md is a quickstart guide on how to build, deploy, and use the WebRisk Go package. It can be used out-of-the-box. The GoDoc and API documentation provide more details on fine tuning the parameters if desired.

How to Build

To download and install from the source, run the following command:

go get github.com/onthegit/webrisk

The programs below execute from your $GOPATH/bin folder. Add that to your $PATH for convenience:

export PATH=$PATH:$GOPATH/bin

The program expects an API key as a parameter, export it with the following command for later use:

export APIKEY=Your Api Key

Proxy Server

The wrserver server binary runs a WebRisk API lookup proxy that allows users to check URLs via a simple JSON API.

1. Once the Go environment is setup, run the following command with your API key:

   go get github.com/onthegit/webrisk/cmd/wrserver
   wrserver -apikey $APIKEY
   

   With the default settings this will start a local server at 127.0.0.1:8080.

2. The server also uses an URL redirector (listening on /r) to show an interstitial for anything marked unsafe.
   If the URL is safe, the client is automatically redirected to the target. Else, an interstitial warning page is shown as recommended by Web Risk.
   Try these URLs:

   127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/
   127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/SOCIAL_ENGINEERING/URL/
   127.0.0.1:8080/r?url=http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/UNWANTED_SOFTWARE/URL/
   127.0.0.1:8080/r?url=http://www.google.com/
   

3. The server also has a lightweight implementation of the API v4 threatMatches endpoint.
   To use the local proxy server to check a URL, send a POST request to 127.0.0.1:8080/v1beta1/uris:search with the following JSON body:

   {
         "uri":"http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/",
         "threatTypes":[
           "MALWARE"
         ]
       }
   

Command-Line Lookup

The wrlookup command-line binary is another example of how the Go Safe Browsing library can be used to protect users from unsafe URLs. This command-line tool filters unsafe URLs piped via STDIN. Example usage:

$ go get github.com/onthegit/webrisk/cmd/wrlookup
$ echo "http://testsafebrowsing.appspot.com/apiv4/ANY_PLATFORM/MALWARE/URL/" | wrlookup -apikey=$APIKEY

WebRisk System Test

To perform an end-to-end test on the package with the WebRisk backend, run the following command after exporting your API key:

go test github.com/onthegit/webrisk -v -run TestWebriskClient

Documentation

Overview

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2019 Google LLC

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Package webrisk implements a client for the Web Risk API v4. API v4 emphasizes efficient usage of the network for bandwidth-constrained applications such as mobile devices. It achieves this by maintaining a small portion of the server state locally such that some queries can be answered immediately without any network requests. Thus, fewer API calls made, means less bandwidth is used.

At a high-level, the implementation does the following:

            hash(query)
                 |
            _____V_____
           |           | No
           | Database  |-----+
           |___________|     |
                 |           |
                 | Maybe?    |
            _____V_____      |
       Yes |           | No  V
     +-----|   Cache   |---->+
     |     |___________|     |
     |           |           |
     |           | Maybe?    |
     |      _____V_____      |
     V Yes |           | No  V
     +<----|    API    |---->+
     |     |___________|     |
     V                       V
(Yes, unsafe)            (No, safe)

Essentially the query is presented to three major components: The database, the cache, and the API. Each of these may satisfy the query immediately, or may say that it does not know and that the query should be satisfied by the next component. The goal of the database and cache is to satisfy as many queries as possible to avoid using the API.

Starting with a user query, a hash of the query is performed to preserve privacy regarded the exact nature of the query. For example, if the query was for a URL, then this would be the SHA256 hash of the URL in question.

Given a query hash, we first check the local database (which is periodically synced with the global Web Risk API servers). This database will either tell us that the query is definitely safe, or that it does not have enough information.

If we are unsure about the query, we check the local cache, which can be used to satisfy queries immediately if the same query had been made recently. The cache will tell us that the query is either safe, unsafe, or unknown (because the it's not in the cache or the entry expired).

If we are still unsure about the query, then we finally query the API server, which is guaranteed to return to us an authoritative answer, assuming no networking failures.

Index

• Constants
• Variables
• func ValidURL(url string) bool
• type Config
• type Stats
• type ThreatType
  • func (tt ThreatType) String() string
• type URLThreat
• type WebriskClient
  • func NewWebriskClient(conf Config) (*WebriskClient, error)
  • func (wr *WebriskClient) Close() error
  • func (wr *WebriskClient) LookupURLs(urls []string) (threats [][]URLThreat, err error)
  • func (wr *WebriskClient) LookupURLsContext(ctx context.Context, urls []string) (threats [][]URLThreat, err error)
  • func (wr *WebriskClient) Status() (Stats, error)
  • func (wr *WebriskClient) WaitUntilReady(ctx context.Context) error

Constants

const (
	// DefaultServerURL is the default URL for the Web Risk API.
	DefaultServerURL = "webrisk.googleapis.com"

	// DefaultUpdatePeriod is the default period for how often WebriskClient will
	// reload its blocklist database.
	DefaultUpdatePeriod = 30 * time.Minute

	// DefaultID and DefaultVersion are the default client ID and Version
	// strings to send with every API call.
	DefaultID      = "GoWebriskClient"
	DefaultVersion = "1.0.0"

	// DefaultRequestTimeout is the default amount of time a single
	// api request can take.
	DefaultRequestTimeout = time.Minute
)

const (
	ThreatTypeUnspecified       = ThreatType(pb.ThreatType_THREAT_TYPE_UNSPECIFIED)
	ThreatTypeMalware           = ThreatType(pb.ThreatType_MALWARE)
	ThreatTypeSocialEngineering = ThreatType(pb.ThreatType_SOCIAL_ENGINEERING)
	ThreatTypeUnwantedSoftware  = ThreatType(pb.ThreatType_UNWANTED_SOFTWARE)
)

List of ThreatType constants.

Variables

var DefaultThreatLists = []ThreatType{
	ThreatTypeMalware,
	ThreatTypeSocialEngineering,
	ThreatTypeUnwantedSoftware,
}

DefaultThreatLists is the default list of threat lists that WebriskClient will maintain. Do not modify this variable.

Functions

func ValidURL

func ValidURL(url string) bool

ValidURL parses the given string and returns true if it is a Web Risk compatible URL.

In general, clients can (and should) just call LookupURLs, which performs the same checks internally. This method can be useful when checking a batch of URLs, as the first parse failure will cause LookupURLs to stop processing the request and return an error.

Types

type Config

type Config struct {
	//SkipLookupAPI if set to true it will skip querying the remote API
	SkipLookupAPI bool

	// ServerURL is the URL for the Web Risk API server.
	// If empty, it defaults to DefaultServerURL.
	ServerURL string

	// ProxyURL is the URL of the proxy to use for all requests.
	// If empty, the underlying library uses $HTTP_PROXY environment variable.
	ProxyURL string

	// APIKey is the key used to authenticate with the Web Risk API
	// service. This field is required.
	APIKey string

	// ID and Version are client metadata associated with each API request to
	// identify the specific implementation of the client.
	// They are similar in usage to the "User-Agent" in an HTTP request.
	// If empty, these default to DefaultID and DefaultVersion, respectively.
	ID      string
	Version string

	// DBPath is a path to a persistent database file.
	// If empty, WebriskClient operates in a non-persistent manner.
	// This means that blocklist results will not be cached beyond the lifetime
	// of the WebriskClient object.
	DBPath string

	// UpdatePeriod determines how often we update the internal list database.
	// If zero value, it defaults to DefaultUpdatePeriod.
	UpdatePeriod time.Duration

	// ThreatLists determines which threat lists that WebriskClient should
	// subscribe to. The threats reported by LookupURLs will only be ones that
	// are specified by this list.
	// If empty, it defaults to DefaultThreatLists.
	ThreatLists []ThreatType

	// RequestTimeout determines the timeout value for the http client.
	RequestTimeout time.Duration

	// Logger is an io.Writer that allows WebriskClient to write debug information
	// intended for human consumption.
	// If empty, no logs will be written.
	Logger io.Writer
	// contains filtered or unexported fields
}

Config sets up the WebriskClient object.

type Stats

type Stats struct {
	QueriesByDatabase int64         // Number of queries satisfied by the database alone
	QueriesByCache    int64         // Number of queries satisfied by the cache alone
	QueriesByAPI      int64         // Number of queries satisfied by an API call
	QueriesFail       int64         // Number of queries that could not be satisfied
	DatabaseUpdateLag time.Duration // Duration since last *missed* update. 0 if next update is in the future.
}

Stats records statistics regarding WebriskClient's operation.

type ThreatType

type ThreatType uint16

ThreatType is an enumeration type for threats classes. Examples of threat classes are malware, social engineering, etc.

func (ThreatType) String

func (tt ThreatType) String() string

type URLThreat

type URLThreat struct {
	Pattern string
	ThreatType
}

A URLThreat is a specialized ThreatType for the URL threat entry type.

type WebriskClient

type WebriskClient struct {
	// contains filtered or unexported fields
}

WebriskClient is a client implementation of API v4.

It provides a set of lookup methods that allows the user to query whether certain entries are considered a threat. The implementation manages all of local database and caching that would normally be needed to interact with the API server.

func NewWebriskClient

func NewWebriskClient(conf Config) (*WebriskClient, error)

NewWebriskClient creates a new WebriskClient.

The conf struct allows the user to configure many aspects of the WebriskClient's operation.

func (*WebriskClient) Close

func (wr *WebriskClient) Close() error

Close cleans up all resources. This method must not be called concurrently with other lookup methods.

func (*WebriskClient) LookupURLs

func (wr *WebriskClient) LookupURLs(urls []string) (threats [][]URLThreat, err error)

LookupURLs looks up the provided URLs. It returns a list of threats, one for every URL requested, and an error if any occurred. It is safe to call this method concurrently.

The outer dimension is across all URLs requested, and will always have the same length as urls regardless of whether an error occurs or not. The inner dimension is across every fragment that a given URL produces. For some URL at index i, one can check for a hit on any blocklist by checking if len(threats[i]) > 0. The ThreatEntryType field in the inner ThreatType will be set to ThreatEntryType_URL as this is a URL lookup.

If an error occurs, the caller should treat the threats list returned as a best-effort response to the query. The results may be stale or be partial.

func (*WebriskClient) LookupURLsContext

func (wr *WebriskClient) LookupURLsContext(ctx context.Context, urls []string) (threats [][]URLThreat, err error)

LookupURLsContext looks up the provided URLs. The request will be canceled if the provided Context is canceled, or if Config.RequestTimeout has elapsed. It is safe to call this method concurrently.

See LookupURLs for details on the returned results.

func (*WebriskClient) Status

func (wr *WebriskClient) Status() (Stats, error)

Status reports the status of WebriskClient. It returns some statistics regarding the operation, and an error representing the status of its internal state. Most errors are transient and will recover themselves after some period.

func (*WebriskClient) WaitUntilReady

func (wr *WebriskClient) WaitUntilReady(ctx context.Context) error

WaitUntilReady blocks until the database is not in an error state. Returns nil when the database is ready. Returns an error if the provided context is canceled or if the WebriskClient instance is Closed.