probeservices

package
v0.0.0-...-bd88772 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 28, 2021 License: BSD-3-Clause Imports: 14 Imported by: 0

README

Package github.com/ooni/probe-engine/probeservices

This package contains code to contact OONI probe services.

The probe services are HTTPS endpoints distributed across a bunch of data centres implementing a bunch of OONI APIs. When started, OONI will benchmark the available probe services and select the fastest one. Eventually all the possible OONI APIs will run as probe services.

Documentation

Overview

Package probeservices contains code to contact OONI probe services.

The probe services are HTTPS endpoints distributed across a bunch of data centres implementing a bunch of OONI APIs. When started, OONI will benchmark the available probe services and select the fastest one. Eventually all the possible OONI APIs will run as probe services.

This package implements the following APIs:

1. v2.0.0 of the OONI bouncer specification defined in https://github.com/ooni/spec/blob/master/backends/bk-004-bouncer;

2. v2.0.0 of the OONI collector specification defined in https://github.com/ooni/spec/blob/master/backends/bk-003-collector.md;

3. most of the OONI orchestra API: login, register, fetch URLs for the Web Connectivity experiment, input for Tor and Psiphon.

Orchestra is a set of OONI APIs for probe orchestration. We currently mainly using it for fetching inputs for the tor, psiphon, and web experiments.

In addition, this package also contains code to benchmark the available probe services, discard non working ones, select the fastest.

Index

Constants

View Source 
const (
	// DefaultDataFormatVersion is the default data format version.
	//
	// See https://github.com/ooni/spec/tree/master/data-formats#history.
	DefaultDataFormatVersion = "0.2.0"

	// DefaultFormat is the default format
	DefaultFormat = "json"
)

Variables

View Source 
var (
	// ErrUnsupportedDataFormatVersion indicates that the user provided
	// in input a data format version that we do not support.
	ErrUnsupportedDataFormatVersion = errors.New("Unsupported data format version")

	// ErrUnsupportedFormat indicates that the format is not supported.
	ErrUnsupportedFormat = errors.New("Unsupported format")

	// ErrJSONFormatNotSupported indicates that the collector we're using
	// does not support the JSON report format.
	ErrJSONFormatNotSupported = errors.New("JSON format not supported")
)
View Source 
var (
	// ErrUnsupportedEndpoint indicates that we don't support this endpoint type.
	ErrUnsupportedEndpoint = errors.New("probe services: unsupported endpoint type")

	// ErrUnsupportedCloudFrontAddress indicates that we don't support this
	// cloudfront address (e.g. wrong scheme, presence of port).
	ErrUnsupportedCloudFrontAddress = errors.New(
		"probe services: unsupported cloud front address",
	)

	// ErrNotRegistered indicates that the probe is not registered
	// with the OONI orchestra backend.
	ErrNotRegistered = errors.New("not registered")

	// ErrNotLoggedIn indicates that we are not logged in
	ErrNotLoggedIn = errors.New("not logged in")

	// ErrInvalidMetadata indicates that the metadata is not valid
	ErrInvalidMetadata = errors.New("invalid metadata")
)

Functions

func Default 

func Default() []model.Service

Default returns the default probe services

func OnlyFallbacks 

func OnlyFallbacks(in []model.Service) (out []model.Service)

OnlyFallbacks returns the fallback endpoints only.

func OnlyHTTPS 

func OnlyHTTPS(in []model.Service) (out []model.Service)

OnlyHTTPS returns the HTTPS endpoints only.

func SortEndpoints 

func SortEndpoints(in []model.Service) (out []model.Service)

SortEndpoints gives priority to https, then cloudfronted, then onion.

Types

type Candidate 

type Candidate struct {
	// Duration is the time it took to access the service.
	Duration time.Duration

	// Err indicates whether the service works.
	Err error

	// Endpoint is the service endpoint.
	Endpoint model.Service

	// TestHelpers contains the data returned by the endpoint.
	TestHelpers map[string][]model.Service
}

Candidate is a candidate probe service.

func SelectBest 

func SelectBest(candidates []*Candidate) (selected *Candidate)

SelectBest selects the best among the candidates. If there is no suitable candidate, then this function returns nil.

func TryAll 

func TryAll(ctx context.Context, sess Session, in []model.Service) (out []*Candidate)

TryAll tries all the input services using the provided context and session. It returns a list containing information on each candidate that was tried. We will try all the HTTPS candidates first. So, the beginning of the list will contain all of them, and for each of them you will know whether it worked (by checking the Err field) and how fast it was (by checking the Duration field). You should pick the fastest one that worked. If none of them works, then TryAll will subsequently attempt with all the available fallbacks, and return at the first success. In such case, you will see a list of N failing HTTPS candidates, followed by a single successful fallback candidate (e.g. cloudfronted). If all candidates fail, you see in output a list containing all entries where Err is not nil.

type Client 

type Client struct {
	httpx.Client
	LoginCalls    *atomicx.Int64
	RegisterCalls *atomicx.Int64
	StateFile     StateFile
}

Client is a client for the OONI probe services API.

func NewClient 

func NewClient(sess Session, endpoint model.Service) (*Client, error)

NewClient creates a new client for the specified probe services endpoint. This function fails, e.g., we don't support the specified endpoint.

func (Client) CheckIn 

func (c Client) CheckIn(ctx context.Context, config model.CheckInConfig) (*model.CheckInInfo, error)

CheckIn function is called by probes asking if there are tests to be run The config argument contains the mandatory settings. Returns the list of tests to run and the URLs, on success, or an explanatory error, in case of failure.

func (Client) CheckReportID 

func (c Client) CheckReportID(ctx context.Context, reportID string) (bool, error)

CheckReportID checks whether the given ReportID exists.

func (Client) FetchPsiphonConfig 

func (c Client) FetchPsiphonConfig(ctx context.Context) ([]byte, error)

FetchPsiphonConfig fetches psiphon config from authenticated OONI orchestra.

func (Client) FetchTorTargets 

func (c Client) FetchTorTargets(ctx context.Context, cc string) (result map[string]model.TorTarget, err error)

FetchTorTargets returns the targets for the tor experiment.

func (Client) FetchURLList 

func (c Client) FetchURLList(ctx context.Context, config model.URLListConfig) ([]model.URLInfo, error)

FetchURLList fetches the list of URLs used by WebConnectivity. The config argument contains the optional settings. Returns the list of URLs, on success, or an explanatory error, in case of failure.

func (Client) GetCredsAndAuth 

func (c Client) GetCredsAndAuth() (*LoginCredentials, *LoginAuth, error)

GetCredsAndAuth is an utility function that returns the credentials with which we are registered and the token with which we're logged in. If we're not registered or not logged in, an error is returned instead.

func (Client) GetMeasurementMeta 

func (c Client) GetMeasurementMeta(
	ctx context.Context, config MeasurementMetaConfig) (*MeasurementMeta, error)

GetMeasurementMeta returns meta information about a measurement.

func (Client) GetTestHelpers 

func (c Client) GetTestHelpers(
	ctx context.Context) (output map[string][]model.Service, err error)

GetTestHelpers is like GetCollectors but for test helpers.

func (Client) MaybeLogin 

func (c Client) MaybeLogin(ctx context.Context) error

MaybeLogin performs login if necessary

func (Client) MaybeRegister 

func (c Client) MaybeRegister(ctx context.Context, metadata Metadata) error

MaybeRegister registers this client if not already registered

func (Client) OpenReport 

func (c Client) OpenReport(ctx context.Context, rt ReportTemplate) (ReportChannel, error)

OpenReport opens a new report.

type LoginAuth 

type LoginAuth struct {
	Expire time.Time `json:"expire"`
	Token  string    `json:"token"`
}

LoginAuth contains authentication info

type LoginCredentials 

type LoginCredentials struct {
	ClientID string `json:"username"`
	Password string `json:"password"`
}

LoginCredentials contains the login credentials

type MeasurementMeta 

type MeasurementMeta struct {
	// Fields returned by the API server whenever we are
	// calling /api/v1/measurement_meta.
	Anomaly              bool      `json:"anomaly"`
	CategoryCode         string    `json:"category_code"`
	Confirmed            bool      `json:"confirmed"`
	Failure              bool      `json:"failure"`
	Input                *string   `json:"input"`
	MeasurementStartTime time.Time `json:"measurement_start_time"`
	ProbeASN             int64     `json:"probe_asn"`
	ProbeCC              string    `json:"probe_cc"`
	ReportID             string    `json:"report_id"`
	Scores               string    `json:"scores"`
	TestName             string    `json:"test_name"`
	TestStartTime        time.Time `json:"test_start_time"`

	// This field is only included if the user has specified
	// the config.Full option, otherwise it's empty.
	RawMeasurement string `json:"raw_measurement"`
}

MeasurementMeta contains measurement metadata.

type MeasurementMetaConfig 

type MeasurementMetaConfig struct {
	// ReportID is the mandatory report ID.
	ReportID string

	// Full indicates whether we also want the full measurement body.
	Full bool

	// Input is the optional input.
	Input string
}

MeasurementMetaConfig contains configuration for GetMeasurementMeta.

type Metadata 

type Metadata struct {
	AvailableBandwidth string   `json:"available_bandwidth,omitempty"`
	DeviceToken        string   `json:"device_token,omitempty"`
	Language           string   `json:"language,omitempty"`
	NetworkType        string   `json:"network_type,omitempty"`
	Platform           string   `json:"platform"`
	ProbeASN           string   `json:"probe_asn"`
	ProbeCC            string   `json:"probe_cc"`
	ProbeFamily        string   `json:"probe_family,omitempty"`
	ProbeTimezone      string   `json:"probe_timezone,omitempty"`
	SoftwareName       string   `json:"software_name"`
	SoftwareVersion    string   `json:"software_version"`
	SupportedTests     []string `json:"supported_tests"`
}

Metadata contains metadata about a probe. This message is included into a bunch of messages sent to orchestra.

func (Metadata) Valid 

func (m Metadata) Valid() bool

Valid returns true if metadata is valid, false otherwise. Metadata is considered valid if all the mandatory fields are not empty. If a field is marked `json:",omitempty"` in the structure definition, then it's for sure mandatory. The "device_token" field is mandatory only if the platform is "ios" or "android", because there's currently no device token that we know of for desktop devices.

type ReportChannel 

type ReportChannel interface {
	CanSubmit(m *model.Measurement) bool
	ReportID() string
	SubmitMeasurement(ctx context.Context, m *model.Measurement) error
}

ReportChannel is a channel through which one could submit measurements belonging to the same report. The Report struct belongs to this interface.

type ReportOpener 

type ReportOpener interface {
	OpenReport(ctx context.Context, rt ReportTemplate) (ReportChannel, error)
}

ReportOpener is any struct that is able to open a new ReportChannel. The Client struct belongs to this interface.

type ReportTemplate 

type ReportTemplate struct {
	// DataFormatVersion is unconditionally set to DefaultDataFormatVersion
	// and you don't need to be concerned about it.
	DataFormatVersion string `json:"data_format_version"`

	// Format is unconditionally set to `json` and you don't need
	// to be concerned about it.
	Format string `json:"format"`

	// ProbeASN is the probe's autonomous system number (e.g. `AS1234`)
	ProbeASN string `json:"probe_asn"`

	// ProbeCC is the probe's country code (e.g. `IT`)
	ProbeCC string `json:"probe_cc"`

	// SoftwareName is the app name (e.g. `measurement-kit`)
	SoftwareName string `json:"software_name"`

	// SoftwareVersion is the app version (e.g. `0.9.1`)
	SoftwareVersion string `json:"software_version"`

	// TestName is the test name (e.g. `ndt`)
	TestName string `json:"test_name"`

	// TestStartTime contains the test start time
	TestStartTime string `json:"test_start_time"`

	// TestVersion is the test version (e.g. `1.0.1`)
	TestVersion string `json:"test_version"`
}

ReportTemplate is the template for opening a report

func NewReportTemplate 

func NewReportTemplate(m *model.Measurement) ReportTemplate

NewReportTemplate creates a new ReportTemplate from a Measurement.

type Session 

type Session interface {
	DefaultHTTPClient() *http.Client
	KeyValueStore() model.KeyValueStore
	Logger() model.Logger
	ProxyURL() *url.URL
	UserAgent() string
}

Session is how this package sees a Session.

type State 

type State struct {
	ClientID string
	Expire   time.Time
	Password string
	Token    string
}

State is the state stored inside the state file

func (State) Auth 

func (s State) Auth() *LoginAuth

Auth returns an authentication structure, if possible, otherwise it returns nil, meaning that you should login again.

func (State) Credentials 

func (s State) Credentials() *LoginCredentials

Credentials returns login credentials, if possible, otherwise it returns nil, meaning that you should create an account.

type StateFile 

type StateFile struct {
	Store model.KeyValueStore
	// contains filtered or unexported fields
}

StateFile is the orchestra state file. It is backed by a generic key-value store configured by the user.

func NewStateFile 

func NewStateFile(kvstore model.KeyValueStore) StateFile

NewStateFile creates a new state file backed by a key-value store

func (StateFile) Get 

func (sf StateFile) Get() (state State)

Get returns the current state. In case of any error with the underlying key-value store, we return an empty state.

func (StateFile) GetMockable 

func (sf StateFile) GetMockable(sfget func(string) ([]byte, error),
	unmarshal func([]byte, interface{}) error) (State, error)

GetMockable is a mockable version of Get

func (StateFile) Set 

func (sf StateFile) Set(s State) error

Set saves the current state on the key-value store.

func (StateFile) SetMockable 

func (sf StateFile) SetMockable(s State, mf func(interface{}) ([]byte, error)) error

SetMockable is a mockable version of Set

type Submitter 

type Submitter struct {
	// contains filtered or unexported fields
}

Submitter is an abstraction allowing you to submit arbitrary measurements to a given OONI backend. This implementation will take care of opening reports when needed as well as of closing reports when needed. Nonetheless you need to remember to call its Close method when done, because there is likely an open report that has not been closed yet.

func NewSubmitter 

func NewSubmitter(opener ReportOpener, logger model.Logger) *Submitter

NewSubmitter creates a new Submitter instance.

func (*Submitter) Submit 

func (sub *Submitter) Submit(ctx context.Context, m *model.Measurement) error

Submit submits the current measurement to the OONI backend created using the ReportOpener passed to the constructor.

Source Files

View all Source files

Directories

Path Synopsis
Package testorchestra helps with testing the OONI orchestra API.
 Package testorchestra helps with testing the OONI orchestra API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.