topolib

package
v1.0.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 25, 2021 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

This package provides a set of structs and functions which are used to geolocate given IP addresses.

topolib is core of the topographer project. You can treat the rest of the application as an _example_ on how to use this library: how to pass parameters from HTTP requests, how to generate responses, how to implement providers.

Topographer is a main entity of the topolib. This struct contains all logic related to IP geolocation: how to resolve IP adddresses, how to use worker pools, how to gather and track usage statistics.

Topographer accepts ip address and returns ResolveResult: enriched and consolidated output of providers with chosen country/city.

Index

Examples

Constants

View Source 
const (
	// FsTargetDirPrefix is a prefix which is used to mark 'active'
	// directory with databases for the provider. All other
	// directories/files are ok to be removed at any given moment in
	// time.
	//
	// If there are many target directories, topographer uses a random
	// one.
	//
	// Suffix is generated based on a contents of the directory.
	// You can think about simplified merkle tree hash here.
	FsTargetDirPrefix = "target_"

	// FsTempDirPrefix defines a prefix for temporary directories populated
	// during update of the offline databases.
	FsTempDirPrefix = "tmp_"
)
View Source 
const (
	// A size of the worker pool to use in Topographer.
	//
	// Topographer is using worker pool to access providers. It is
	// done to prevent overloading and overusing of them especially if
	// provider accesses some external resource.
	//
	// A worker task is a single IP lookup for _all_ providers. So, if
	// you have a worker pool size of 100, you can concurrenly resolve
	// only 100 IPs. This is not quite a granular but useful on practice
	// if you plan capacity.
	//
	// Usually you want to have this number of workers in pool.
	DefaultWorkerPoolSize = 4096
)

Variables

View Source 
var (
	// ErrTopographerShutdown returns if you've tried to resolve ips via
	// topographer but instance was shutdown.
	ErrTopographerShutdown = errors.New("topographer instance was shutdown")

	// ErrContextIsClosed returns if context is closed during execution
	// of the method.
	ErrContextIsClosed = errors.New("context is closed")

	// ErrCircuitBreakerOpened returns by http client if circuit breaker
	// is opened.
	ErrCircuitBreakerOpened = errors.New("circuit breaker is opened")

	// ErrCircuitBreakerIgnore should be returned if it is necessary to
	// ignore circuit breaker error in http client.
	ErrCircuitBreakerIgnore = errors.New("this error should be ignores by circuit breaker")
)

Functions

func NormalizeAlpha2Code 

func NormalizeAlpha2Code(alpha2 string) string

NormalizeAlpha2Code returns a normalized 2-letter ISO3166 code. Normalized code is uppercased with some additional mapping. For example, some databases return ZZ as 'unknown' country. This function returns "" instead. Some databases still map Serbia to YU. This correctly maps YU to CS.

So, whenever you want to use 2-letter ISO3166 code and it is coming from unknown source, it is recommended to normalize it with this function.

Example 
package main

import (
	"fmt"

	"github.com/9seconds/topographer/topolib"
)

func main() {
	fmt.Println(topolib.NormalizeAlpha2Code("ru"))
}

Output:

RU
Example (Yugoslavia) 
package main

import (
	"fmt"

	"github.com/9seconds/topographer/topolib"
)

func main() {
	fmt.Println(topolib.NormalizeAlpha2Code("YU"))
}

Output:

CS

Types

type CountryCode 

type CountryCode uint8

CountryCode represents a code of the country. Actually it is possible to use strings but this custom type is more convenient because it is jsonable, stringified and you can query to get more detailed data if necessary.

func Alpha2ToCountryCode 

func Alpha2ToCountryCode(alpha2 string) CountryCode

Alpha2ToCountryCode maps 2-letter string of ISO3166 to CountryCode type.

Example 
package main

import (
	"fmt"

	"github.com/9seconds/topographer/topolib"
)

func main() {
	code := topolib.Alpha2ToCountryCode("uk")

	fmt.Println(code.String())
	fmt.Println(code.Details().Name.BaseLang.Common)
}

Output:

GB
United Kingdom

func Alpha3ToCountryCode 

func Alpha3ToCountryCode(alpha3 string) CountryCode

Alpha3ToCountryCode maps 3-letter string of ISO3166 to CountryCode type.

Example 
package main

import (
	"fmt"

	"github.com/9seconds/topographer/topolib"
)

func main() {
	code := topolib.Alpha3ToCountryCode("ita")

	fmt.Println(code.String())
	fmt.Println(code.Details().Name.BaseLang.Common)
}

Output:

IT
Italy

func (CountryCode) Details 

func (c CountryCode) Details() gountries.Country

Details returns a details for the country taken from gountries.

func (CountryCode) Known 

func (c CountryCode) Known() bool

Checks if country code is known. You can think about 'known' as 'empty' value. Actually we can name it as empty but it is hard for me to remember that notion. Known is more natural.

func (CountryCode) MarshalJSON 

func (c CountryCode) MarshalJSON() ([]byte, error)

MarshalJSON is to conform json.Marshaller interface.

func (CountryCode) String 

func (c CountryCode) String() string

String returns 2-letter ISO3166 country code. For example, for USA it is going to be US. For UK - GB.

type HTTPClient 

type HTTPClient interface {
	Do(*http.Request) (*http.Response, error)
}

HTTPClient is something which can do HTTP requests and returns HTTP responses.

For example, topolib enrich default http.Client with rate limiters and circuit breakers. If you want, you can build your own client.

It is also important that topographer is designed in a way that each provider has to use its own HTTP client. There should be no sharing of such clients between different instances.

func NewHTTPClient 

func NewHTTPClient(client *http.Client,
	userAgent string,
	rateLimiterInterval time.Duration,
	rateLimitBurst int,
	circuitBreakerOpenThreshold uint32,
	circuitBreakerHalfOpenTimeout, circuitBreakerResetFailuresTimeout time.Duration) HTTPClient

NewHTTPClient prepares a new HTTP client, wraps it with rate limiter, circuit breaker, sets a user agent etc.

Please see https://pkg.go.dev/golang.org/x/time/rate to get a meaning of rate limiter parameters.

A meaning of circuit breaker parameters:

circuitBreakerOpenThreshold - this is a threshold of failures when circuit breaker becomes OPEN. So, if you pass 3 here, then after 3 failures, circuit breaker switches into OPEN state and blocks access to a target.

circuitBreakerResetFailuresTimeout - is tightly coupled with circuitBreakerOpenThreshold. Each time period when circuit breaker is closed, we try to reset a failure counter. So, if you pass 10 here, make 2 errors then after 10 seconds this counter is going to be reset.

circuitBreakerHalfOpenTimeout - when circuit breaker is closed, we open it after this time perios and it goes into HALF_OPEN state. Within this state we allow 1 attempt. If this attempt fails, then it goes into OPEN state again. If succeed - goes to CLOSED.

type Logger 

type Logger interface {
	// LookupError is a method which logs different errors
	// related to IP address lookup.
	LookupError(ip net.IP, name string, err error)

	// UpdateInfo notifies that provider has updated a database with
	// no errors.
	UpdateInfo(name string)

	// UpdateError notifies that there was an error in updated database.
	UpdateError(name string, err error)
}

Logger is a logger interface used by Topographer.

Each method accepts name parameter. name is a name of the provider.

type OfflineProvider 

type OfflineProvider interface {
	Provider

	// Shutdown shuts down an offline provider. Once it is shutdown,
	// topographer won't call any methods of this provider except of
	// Name.
	Shutdown()

	// UpdateEvery returns a periodicity which should be used to update
	// databases.
	UpdateEvery() time.Duration

	// BaseDirectory return a base directory for this offline provider.
	// A base directory is that one which has target, tmps and so on.
	// Provider should not write anywhere outside of this directory.
	BaseDirectory() string

	// Open ingests a new database update from FS. It is expected
	// that on failures all old handles are alive. There should be no
	// situation that database is updated and this update breaks a
	// database.
	//
	// There is an open question if we have to keep old working db or
	// not. Right now topographer deletes old database so it is possible
	// that random restart of the service will make Provider broken.
	// This is intentional. If you want to do a validation, please do it
	// in Download method.
	Open(string) error

	// Download takes a directory and creates a file structure which is
	// suitable for Open method. Also, it is a place where you can do a
	// validation like checksum match etc. If download returns no error,
	// it means that database is working and should be used.
	Download(context.Context, string) error
}

OfflineProvider is a special version of Provider which works with offline databases.

A major difference is that these databases should be downloaded opened and updated on periodic basis. This all is responsibility of Topographer instance. It pass correct directories to Open and Download method and ensures they are writable/readable.

func NewCachingOfflineProvider 

func NewCachingOfflineProvider(provider OfflineProvider, itemsCount uint, ttl time.Duration) OfflineProvider

NewCachingOfflineProvider is a version of caching provider for OfflineProvider.

type Provider 

type Provider interface {
	// Name returns a unique identifier of the provider.
	// Usually a name.
	Name() string

	// Lookup resolves a location of IP address. If geolocation fails
	// for some reason, it has to return an error. There is no need to
	// fill each field of ProviderLookupResult but it is expected that
	// at least CountryCode is present.
	Lookup(context.Context, net.IP) (ProviderLookupResult, error)
}

Provider represents an entity which can resolve geolocation of IP addresses.

Each provider should work with its own service. For example, if you use MaxMind, then there should be a dedicated provider for MaxMind.

func NewCachingProvider 

func NewCachingProvider(provider Provider, itemsCount uint, ttl time.Duration) Provider

NewCachingProvider returns a wrapper for a given provider. Caching provider caches responses of Lookup calls in some LRU cache with TTL timestamp.

type ProviderLookupResult 

type ProviderLookupResult struct {
	// CountryCode is a code of the chosen country.
	CountryCode CountryCode

	// City is the name of the city.
	City string
}

ProviderLookupResult is a strucutre which is returned by Provider interface. It is not the same as ResolveResultDetail. A latter one is used in consolidated responses while ProviderLookupResult should be used by those who want to implement their own providers.

type ResolveResult 

type ResolveResult struct {
	// IP is IP address which we resolve.
	IP net.IP `json:"ip"`

	// Country is a set of details on a choosen country.
	Country struct {
		// Alpha2Code is 2-letter ISO3166 code. For example, for Russia
		// it is going to be RU.
		Alpha2Code string `json:"alpha2_code"`

		// Alpha3Code is 3-letter ISO3166 code. For example, for Russia
		// it is going to be RUS.
		Alpha3Code string `json:"alpha3_code"`

		// CommonName is a name of the country which we use in real life.
		// Like, Russia.
		CommonName string `json:"common_name"`

		// OfficialName is a name of the country which we use in
		// official papers. Like Russian Federation.
		OfficialName string `json:"official_name"`
	} `json:"country"`

	// City is a name of the city this IP belongs to.
	City string `json:"city"`

	// Details is a list of 'raw' results generated by Providers. If you
	// are interested in why Topographer choose this country or what was
	// the choise of ipinfo.io, this is a field to go.
	Details []ResolveResultDetail `json:"details"`
}

ResolveResult is a consolidated verdict on IP geolocation made by Topographer.

Consolidation means that there is a process of 'voting' for the location. Currently it is done in a following way:

1. All providers give their results

2. Topographer choose the most 'popular country'

3. It choose the most 'popular city' among these from step 2.

4. This is a verdict.

func (*ResolveResult) OK 

func (r *ResolveResult) OK() bool

OK checks if response has some data. For example, it is possible that we have no city or country.

type ResolveResultDetail 

type ResolveResultDetail struct {
	// ProviderName is a name of the provider which made
	// this choice.
	ProviderName string `json:"provider_name"`

	// CountryCode is a code of the chosen country.
	CountryCode CountryCode `json:"country_code"`

	// City is a name of the city.
	City string `json:"city"`
}

ResolveResultDetail is a result generated by Provider.

type Topographer 

type Topographer struct {
	// contains filtered or unexported fields
}

Topographer is a main entity of topolib. It is responsible for provider management, background updates and IP lookups. It also contains an instance of worker pool to use.

func NewTopographer 

func NewTopographer(providers []Provider, logger Logger, workerPoolSize int) (*Topographer, error)

NewTopographer creates a new instance of topographer.

func (*Topographer) Resolve 

func (t *Topographer) Resolve(ctx context.Context,
	ip net.IP,
	providers []string) (ResolveResult, error)

Resolve geolocation of the single IP.

'providers' argument contains names of the providers to use. If you want to use all providers, simply pass nil here.

func (*Topographer) ResolveAll 

func (t *Topographer) ResolveAll(ctx context.Context,
	ips []net.IP,
	providers []string) ([]ResolveResult, error)

ResolveAll concurrently resolves IP geolocation of the batch of ip addresses.

'providers' argument contains names of the providers to use. If you want to use all providers, simply pass nil here.

func (*Topographer) ServeHTTP 

func (t *Topographer) ServeHTTP(w http.ResponseWriter, req *http.Request)

ServeHTTP is to conform http.Handler interface.

func (*Topographer) Shutdown 

func (t *Topographer) Shutdown()

func (*Topographer) UsageStats 

func (t *Topographer) UsageStats() []*UsageStats

UsageStats returns an array with stats.

type UsageStats 

type UsageStats struct {
	// A name of the provider.
	Name string
	// contains filtered or unexported fields
}

UsageStats collects different counters which could be useful to detect providers which are in used or stale.

Currently we write a timestamp of the last usage, timestamp of last update (sucessful, only for offline providers); counters for a number of success and failed lookups.

func (*UsageStats) Failures 

func (u *UsageStats) Failures() uint64

Failures returns a count of failed lookups.

func (*UsageStats) LastUpdated 

func (u *UsageStats) LastUpdated() time.Time

LastUpdated returns a timestamp when provider was updated last time.

func (*UsageStats) LastUsed 

func (u *UsageStats) LastUsed() time.Time

LastUpdated returns a timestamp when provider was used last time.

func (*UsageStats) MarshalJSON 

func (u *UsageStats) MarshalJSON() ([]byte, error)

MarshalJSON to conform json.Marshaller interface.

func (*UsageStats) Successes 

func (u *UsageStats) Successes() uint64

Successes returns a count of successful lookups.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.