probe-cli

module
v3.21.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 5, 2024 License: GPL-3.0

README

OONI Probe Android

GoDoc Coverage Status Slack

OONI Probe Measurement Engine and CLI

Install instructions »

User guide · API docs · Report bug · Request feature · Tutorials

Table of Contents
  1. About this project
  2. Install instructions
  3. Nightly Builds
  4. Build instructions
  5. Contributing
  6. License
  7. Updating dependencies
  8. Releasing
  9. Semantic versioning policy

About this project

The Open Observatory of Network Interference (OONI) is a non-profit free software project that aims to empower decentralized efforts in documenting Internet censorship around the world.

This repository contains the following Go packages:

  1. the ooniprobe command line client (cmd/ooniprobe);

  2. the test helper server (internal/cmd/oohelperd);

  3. the mobile library (pkg/oonimkall);

  4. the measurement-engine library (internal);

  5. the miniooni experimental command line client (internal/cmd/miniooni).

Every top-level directory in this repository contains an explanatory README file.

Install instructions

Follow the instructions at ooni.org/install/cli to install ooniprobe precompiled binaries for Windows, macOS, and Debian/Ubuntu. Once ooniprobe is installed, refer to the user guide.

Nightly builds

We publish nightly builds using the rolling release tag. These builds use the latest commit of the master branch.

Developer instructions

To setup development for this repository you need Go >= 1.15. The ./script/go.bash script will automatically download the expected version of Go mentioned in the GOVERSION file (i.e., go1.20.14) and use it for building.

You can also bypass ./script/go.bash and build ooniprobe manually using go build ... but, in such a case, note that:

  1. using an older version that the one mentioned in GOVERSION is definitely not recommended and may not even compile;

  2. using later versions should work as intended for core functionality but extra functionality may be disabled or not working as intended.

Here's why: we rely on packages forked from the standard library; so, it is more robust to use the same version of Go from which we forked those packages from.

You will also need a C compiler. On Linux and other Unix systems both GCC and Clang will work. If you're using Windows, we recommend installing Ubuntu or Debian on the Windows Subsystem for Linux. If you're targeting Windows, you should also install the mingw-w64 cross-compiler.

Debian developer setup

The following commands show how to setup a development environment using Debian 12 ("bookworm"). The same instructions should also work for Ubuntu 22.04 LTS.

# install the compilers, git, and the root CA
sudo apt install golang build-essential ca-certificates git

# [optional] install mingw-w64 if you're targeting windows
sudo apt install mingw-w64

Fedora developer setup

The following commands show how to setup a development environment using Fedora, as long as your Fedora uses Go >= 1.15.

# install the compilers and git
sudo dnf install golang make gcc gcc-c++ git

# [optional] install mingw-w64 if you're targeting windows
sudo dnf install mingw64-gcc mingw64-gcc-c++

macOS developer setup

The following commands show how to setup a development environment using macOS. We assume you have already installed Homebrew, which should also install the Xcode command line tools.

Then, you need to follow these instructions:

# install the compiler
brew install go

The ./script/go.bash script

The ./script/go.bash script requires Go >= 1.15 and automates installing and using the correct version of Go. Running this script as follows:

./script/go.bash build -v -ldflags '-s -w' ./internal/cmd/miniooni

Is equivalent to running these commands:

go install -v golang.org/dl/go1.20.14@latest
$HOME/go/bin/go1.20.14 download
$HOME/sdk/go1.20.14/bin/go build -v -ldflags '-s -w' ./internal/cmd/miniooni

Common build targets

This section shows how to build using ./script/go.bash. If you want to bypass using this script, just run go instead of ./script/go.bash.

You can compile ooniprobe using:

./script/go.bash build -v -ldflags '-s -w' ./cmd/ooniprobe

This command will generate a stripped binary called ooniprobe in the toplevel directory.

Likewise, you can compile miniooni using:

./script/go.bash build -v -ldflags '-s -w' ./internal/cmd/miniooni

This command will generate a stripped binary called miniooni in the toplevel directory.

And oohelperd using:

./script/go.bash build -v -ldflags '-s -w' ./internal/cmd/oohelperd

This command will generate a stripped binary called oohelperd in the toplevel directory.

Contributing

Please, see CONTRIBUTING.md.

License

SPDX-License-Identifier: GPL-3.0-or-later

Updating dependencies

go get -t -u -v ./... && go mod tidy

Releasing

Create an issue according to the routine release template and perform any item inside the check-list.

We build releases using Makefile, which requires GNU make. Run make help for detailed usage.

Semantic versioning policy

The mobile library is a public package for technical reasons. Go mobile tools require a public package to build from. Yet, we don't consider API breakages happening in such a package to be sufficient to bump our major version number. For us, the mobile library is just a mean to implement OONI Probe Android and OONI Probe iOS. We'll only bump the major version number if we change ./cmd/ooniprobe's CLI.

Directories

Path Synopsis
cmd
ooniprobe/internal/autorun
Package autorun contains code to manage automatic runs
Package autorun contains code to manage automatic runs
ooniprobe/internal/log/handlers/cli/progress
Package progress provides a simple terminal progress bar.
Package progress provides a simple terminal progress bar.
ooniprobe/internal/log/handlers/syslog
Package syslog contains a syslog handler.
Package syslog contains a syslog handler.
ooniprobe/internal/oonitest
Package oonitest contains code used for testing.
Package oonitest contains code used for testing.
internal
bytecounter
Package bytecounter contains code to track the number of bytes sent and received by a probe.
Package bytecounter contains code to track the number of bytes sent and received by a probe.
checkincache
Package checkincache contains an on-disk cache for check-in responses.
Package checkincache contains an on-disk cache for check-in responses.
cmd/apitool
Command apitool is a simple tool to fetch individual OONI measurements.
Command apitool is a simple tool to fetch individual OONI measurements.
cmd/buildtool/internal/buildtoolmodel
Package buildtoolmodel contains the model for buildtool.
Package buildtoolmodel contains the model for buildtool.
cmd/buildtool/internal/buildtooltest
Package buildtooltest contains code for testing buildtool.
Package buildtooltest contains code for testing buildtool.
cmd/gardener
Command gardener helps with test-lists management.
Command gardener helps with test-lists management.
cmd/gardener/internal/aggregationapi
Package aggregationapi allows one to query the OONI aggregation API.
Package aggregationapi allows one to query the OONI aggregation API.
cmd/gardener/internal/dnsfix
Package dnsfix implements the dnsfix command.
Package dnsfix implements the dnsfix command.
cmd/gardener/internal/dnsreport
Package dnsreport implements the dnsreport subcommand.
Package dnsreport implements the dnsreport subcommand.
cmd/gardener/internal/sync
Package sync implements the sync subcommand.
Package sync implements the sync subcommand.
cmd/gardener/internal/testlists
Package testlists contains code to read/rewrite the test lists.
Package testlists contains code to read/rewrite the test lists.
cmd/getresources
Command getresources downloads the resources
Command getresources downloads the resources
cmd/ghgen
Command ghgen regenerates selected GitHub actions.
Command ghgen regenerates selected GitHub actions.
cmd/miniooni
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x.
Command miniooni is a simple binary for research and QA purposes with a CLI interface similar to MK and OONI Probe v2.x.
cmd/oohelper
Command oohelper contains a simple command line client for the Web Connectivity test helper.
Command oohelper contains a simple command line client for the Web Connectivity test helper.
cmd/oohelperd
Command oohelperd implements the Web Connectivity test helper.
Command oohelperd implements the Web Connectivity test helper.
cmd/oonireport
Command oonireport uploads reports stored on disk to the OONI collector.
Command oonireport uploads reports stored on disk to the OONI collector.
cmd/ooporthelper
command ooporthelper implements the Port Filtering test helper
command ooporthelper implements the Port Filtering test helper
cmd/printversion
Command printversion prints the current version of this repository.
Command printversion prints the current version of this repository.
cmd/tinyjafar
Command tinyjafar implements a subset of the CLI flags of the original jafar tool.
Command tinyjafar implements a subset of the CLI flags of the original jafar tool.
engine
Package engine contains the engine API.
Package engine contains the engine API.
enginelocate
Package enginelocate implements IP lookup, resolver lookup, and geolocation.
Package enginelocate implements IP lookup, resolver lookup, and geolocation.
enginenetx
Package enginenetx contains engine-specific network-extensions.
Package enginenetx contains engine-specific network-extensions.
engineresolver
Package engineresolver contains the resolver used by the OONI engine.
Package engineresolver contains the resolver used by the OONI engine.
experiment/dash
Package dash implements the DASH network experiment.
Package dash implements the DASH network experiment.
experiment/dnscheck
Package dnscheck contains the DNS check experiment.
Package dnscheck contains the DNS check experiment.
experiment/dnsping
Package dnsping is the experimental dnsping experiment.
Package dnsping is the experimental dnsping experiment.
experiment/echcheck
Package echcheck contains the ECH blocking network experiment.
Package echcheck contains the ECH blocking network experiment.
experiment/example
Package example contains a simple example experiment.
Package example contains a simple example experiment.
experiment/fbmessenger
Package fbmessenger contains the Facebook Messenger network experiment.
Package fbmessenger contains the Facebook Messenger network experiment.
experiment/hhfm
Package hhfm contains the HTTP Header Field Manipulation network experiment.
Package hhfm contains the HTTP Header Field Manipulation network experiment.
experiment/hirl
Package hirl contains the HTTP Invalid Request Line network experiment.
Package hirl contains the HTTP Invalid Request Line network experiment.
experiment/httphostheader
Package httphostheader contains the HTTP host header network experiment.
Package httphostheader contains the HTTP host header network experiment.
experiment/ndt7
Package ndt7 contains the ndt7 network experiment.
Package ndt7 contains the ndt7 network experiment.
experiment/portfiltering
Package portfiltering implements the portfiltering experiment
Package portfiltering implements the portfiltering experiment
experiment/psiphon
Package psiphon implements the psiphon network experiment.
Package psiphon implements the psiphon network experiment.
experiment/quicping
Package quicping implements the quicping network experiment.
Package quicping implements the quicping network experiment.
experiment/riseupvpn
Package riseupvpn contains the RiseupVPN network experiment.
Package riseupvpn contains the RiseupVPN network experiment.
experiment/signal
Package signal contains the Signal network experiment.
Package signal contains the Signal network experiment.
experiment/simplequicping
Package simplequicping is the experimental simplequicping experiment.
Package simplequicping is the experimental simplequicping experiment.
experiment/sniblocking
Package sniblocking contains the SNI blocking network experiment.
Package sniblocking contains the SNI blocking network experiment.
experiment/stunreachability
Package stunreachability contains the STUN reachability experiment.
Package stunreachability contains the STUN reachability experiment.
experiment/tcpping
Package tcpping is the experimental tcpping experiment.
Package tcpping is the experimental tcpping experiment.
experiment/telegram
Package telegram contains the Telegram network experiment.
Package telegram contains the Telegram network experiment.
experiment/tlsmiddlebox
Package tlsmiddlebox implements the tlsmiddlebox experiment
Package tlsmiddlebox implements the tlsmiddlebox experiment
experiment/tlsping
Package tlsping is the experimental tlsping experiment.
Package tlsping is the experimental tlsping experiment.
experiment/tlstool
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments.
Package tlstool contains a TLS tool that we are currently using for running quick and dirty experiments.
experiment/tlstool/internal
Package internal contains the implementation of tlstool.
Package internal contains the implementation of tlstool.
experiment/tor
Package tor contains the tor experiment.
Package tor contains the tor experiment.
experiment/torsf
Package torsf contains the torsf experiment.
Package torsf contains the torsf experiment.
experiment/urlgetter
Package urlgetter implements a nettest that fetches a URL.
Package urlgetter implements a nettest that fetches a URL.
experiment/vanillator
Package vanillator contains the vanilla_tor experiment.
Package vanillator contains the vanilla_tor experiment.
experiment/webconnectivity
Package webconnectivity implements OONI's Web Connectivity experiment.
Package webconnectivity implements OONI's Web Connectivity experiment.
experiment/webconnectivity/internal
Package internal contains internal code.
Package internal contains internal code.
experiment/webconnectivitylte
Package webconnectivitylte implements the web_connectivity experiment.
Package webconnectivitylte implements the web_connectivity experiment.
experiment/whatsapp
Package whatsapp contains the WhatsApp network experiment.
Package whatsapp contains the WhatsApp network experiment.
feature/psiphonfeat
Package psiphonfeat implements the psiphon feature.
Package psiphonfeat implements the psiphon feature.
flagx
Package flagx contains extensions for the standard library flag package.
Package flagx contains extensions for the standard library flag package.
fsx
Package fsx contains io/fs extensions.
Package fsx contains io/fs extensions.
geoipx
Package geoipx contains code to use the embedded MaxMind-like databases.
Package geoipx contains code to use the embedded MaxMind-like databases.
httpapi
Package httpapi contains code for calling HTTP APIs.
Package httpapi contains code for calling HTTP APIs.
httpx
Package httpx contains http extensions.
Package httpx contains http extensions.
hujsonx
Package hujsonx contains github.com/tailscale/hujson extensions.
Package hujsonx contains github.com/tailscale/hujson extensions.
humanize
Package humanize is like dustin/go-humanize.
Package humanize is like dustin/go-humanize.
idnax
Package idnax contains IDNA extensions.
Package idnax contains IDNA extensions.
inputparser
Package inputparser contains code to parse experiments input.
Package inputparser contains code to parse experiments input.
kvstore
Package kvstore implements model.KeyValueStore.
Package kvstore implements model.KeyValueStore.
legacy/assetsdir
Package assetsdir contains code to cleanup the assets dir.
Package assetsdir contains code to cleanup the assets dir.
legacy/kvstore2dir
Package kvstore2dir migrates $OONI_HOME/kvstore2 to $OONI_HOME/engine.
Package kvstore2dir migrates $OONI_HOME/kvstore2 to $OONI_HOME/engine.
legacy/legacymodel
Package legacymodel contains legacy content that used to be in internal/model
Package legacymodel contains legacy content that used to be in internal/model
legacy/measurex
Package measurex contains measurement extensions.
Package measurex contains measurement extensions.
legacy/mockable
Package mockable contains mockable objects
Package mockable contains mockable objects
legacy/netx
Package netx contains code to perform network measurements.
Package netx contains code to perform network measurements.
legacy/tracex
Package tracex performs measurements using tracing.
Package tracex performs measurements using tracing.
logmodel
Package logmodel contains the core log model.
Package logmodel contains the core log model.
logx
Package logx contains logging extensions.
Package logx contains logging extensions.
measurexlite
Package measurexlite contains measurement extensions.
Package measurexlite contains measurement extensions.
memoryless
Package memoryless helps repeated calls to a function be distributed across time in a memoryless fashion.
Package memoryless helps repeated calls to a function be distributed across time in a memoryless fashion.
minipipeline
Package minipipeline implements a minimal data processing pipeline used to analyze local measurements collected by OONI Probe.
Package minipipeline implements a minimal data processing pipeline used to analyze local measurements collected by OONI Probe.
mlablocatev2
Package mlablocatev2 implements m-lab locate services API v2.
Package mlablocatev2 implements m-lab locate services API v2.
mocks
Package mocks contains mocks for implementing unit tests.
Package mocks contains mocks for implementing unit tests.
model
Package model contains the shared interfaces and data structures.
Package model contains the shared interfaces and data structures.
multierror
Package multierror contains code to manage multiple errors.
Package multierror contains code to manage multiple errors.
must
Package must contains functions that panic on error.
Package must contains functions that panic on error.
netemx
Package netemx contains code to run integration tests using github.com/ooni/netem.
Package netemx contains code to run integration tests using github.com/ooni/netem.
netxlite
Package netxlite contains network extensions.
Package netxlite contains network extensions.
ooapi
Package ooapi describes how the OONI API works.
Package ooapi describes how the OONI API works.
oonirun
Package oonirun contains code to run OONI experiments.
Package oonirun contains code to run OONI experiments.
optional
Package optional contains safer code to handle optional values.
Package optional contains safer code to handle optional values.
platform
Package platform allows you to obtain the platform name.
Package platform allows you to obtain the platform name.
probeservices
Package probeservices contains code to contact OONI probe services.
Package probeservices contains code to contact OONI probe services.
progress
Package progress contains utilities to emit progress.
Package progress contains utilities to emit progress.
ptx
Package ptx contains code to use pluggable transports.
Package ptx contains code to use pluggable transports.
randx
Package randx contains math/rand extensions.
Package randx contains math/rand extensions.
registry
Package registry contains a registry of all the available experiments.
Package registry contains a registry of all the available experiments.
runtimex
Package runtimex contains runtime extensions.
Package runtimex contains runtime extensions.
shellx
Package shellx helps to write shell-like Go code.
Package shellx helps to write shell-like Go code.
shellx/shellxtesting
Package shellxtesting supports shellx testing.
Package shellxtesting supports shellx testing.
strcasex
Package strcasex converts strings to various cases.
Package strcasex converts strings to various cases.
stuninput
Package stuninput contains stun targets as well as code to format such targets according to various conventions.
Package stuninput contains stun targets as well as code to format such targets according to various conventions.
testingproxy
Package testingproxy contains shared test cases for the proxies.
Package testingproxy contains shared test cases for the proxies.
testingquic
Package testingquic allows to retrieve the domain and endpoint to use for all the integration tests that use QUIC.
Package testingquic allows to retrieve the domain and endpoint to use for all the integration tests that use QUIC.
testingsocks5
Package testingsock5 is a netem-aware fork of https://github.com/armon/go-socks5.
Package testingsock5 is a netem-aware fork of https://github.com/armon/go-socks5.
testingx
Package testingx contains code useful for testing.
Package testingx contains code useful for testing.
throttling
Package throttling wraps connections to measure throttling.
Package throttling wraps connections to measure throttling.
torlogs
Package torlogs contains code to read tor logs.
Package torlogs contains code to read tor logs.
tunnel
Package tunnel allows to create tunnels to speak with OONI backends and other services.
Package tunnel allows to create tunnels to speak with OONI backends and other services.
tunnel/mocks
Package mocks contains mocks for tunnel.
Package mocks contains mocks for tunnel.
tutorial/dslx/chapter02
## Getting started
## Getting started
tutorial/experiment/torsf/chapter01
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/experiment/torsf/chapter02
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/generator
Command generator generates or re-generates the tutorial chapters.
Command generator generates or re-generates the tutorial chapters.
tutorial/measurex/chapter01
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter02
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter03
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter04
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter05
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter06
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter07
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter08
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter09
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter10
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter11
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter12
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter13
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/measurex/chapter14
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter01
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter02
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter03
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter04
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter05
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter06
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter07
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
tutorial/netxlite/chapter08
-=-=- StartHere -=-=-
-=-=- StartHere -=-=-
version
Package version contains version information.
Package version contains version information.
webconnectivityalgo
Package webconnectivityalgo contains Web Connectivity algorithms.
Package webconnectivityalgo contains Web Connectivity algorithms.
webconnectivityqa
Package webconnectivityqa contains code to perform Web Connectivity QA.
Package webconnectivityqa contains code to perform Web Connectivity QA.
x
Package x contains highly experimental packages.
Package x contains highly experimental packages.
x/dsljavascript
Package dsljavascript allows running experiments written in JavaScript.
Package dsljavascript allows running experiments written in JavaScript.
x/dsljson
Package dsljson allows expressing the measurement DSL using JSON.
Package dsljson allows expressing the measurement DSL using JSON.
x/dslvm
Package dslvm contains low-level code for implementing the measurements DSL.
Package dslvm contains low-level code for implementing the measurements DSL.
x/dslx
Package dslx implements a domain specific language for writing network experiments.
Package dslx implements a domain specific language for writing network experiments.
pkg
oonimkall
Package oonimkall implements APIs used by OONI mobile apps.
Package oonimkall implements APIs used by OONI mobile apps.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL