osl

package
v2.0.16+incompatible Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 15, 2021 License: GPL-3.0 Imports: 25 Imported by: 50

README

Psiphon Tunnel Obfuscated Server Lists

Design

Obfuscated server lists (OSLs) is a mechanism to distribute Psiphon servers to select users. Like standard server lists, these server lists are publicly available at locations encoded in the Psiphon client. Unlike standard server lists, these lists are encrypted. Only clients that have the required keys can access the servers within these lists.

One scheme for issuing server list encryption keys (SLOKs) is for the Psiphon servers to release the keys to clients that meet certain criteria for data transfer. The criteria is designed to distinguish regular, typical users from bots, scripts, and opportunistic adversaries. The data transfer SLOK scheme uses thresholds and groups of criteria to avoid being too strict. This serves two purposes: users don't need to meet all targets to earn a SLOK; and knowing that a user has earned a SLOK does not precisely reveal the user's activity.

Parameters include:

  • The base time period for each SLOK. For example, with a base time period of one week, any sufficient network activity within a given week would earn the SLOKs. Possession of the SLOK does not reveal when, within the week the network, activity took place.
  • A set of network destinations for each SLOK, and a threshold. For example, sufficient data transfer to any 2 of 5 of a group of different network destinations, which could be defined to include typical "search", "social media", "messaging", "news", "e-commerce" destinations. Possession of the SLOK does not reveal which of the groups were visited; nor exactly which sites within the groups. Furthermore, the definition of network destinations admits all sites hosted on shared infrastructure, introducing further ambiguity.
  • A time threshold on top of SLOKs. For example, any 2 of 4 weekly SLOKs are required to decrypt a given OSL. Decrypting the OSL and using its servers does not reveal exactly which SLOKs were used.
  • A propagation channel ID for each SLOK. Each propagation channel has its own SLOK key space, and an adversary in possession of a SLOK from a different propagation channel cannot use that to determine the properties of SLOKs from other propagation channels.

All parameters and criteria are operational secrets stored only in Psiphon servers/automation and not publicly revealed.

Client progress towards a SLOK is stored in Psiphon server volatile memory (and, potentially, swap).

When a SLOK is earned by a client, the Psiphon server sends the SLOK to the client through the secure Psiphon SSH tunnel. No logs are recorded for individual SLOK events.

The client stores all its SLOKs in its local database. SLOKs are random values. An adversary that compromises a client may obtain the SLOKs, but does not have access to the parameters which generated the SLOK. And even if the parameters are compromised (or the adversary reverse engineers partial parameter information by earning its own SLOKs via random activity) the thresholds and groupings within the parameters mean a single SLOK does not demonstrate browsing a particular site or browsing at a particular time.

When an OSL is downloaded, this event is logged. This is how we monitor the mechanism. This log is a standard Psiphon log, which does not record the client IP address but does record GeoIP information. A log that an OSL has been unlocked indirectly reveals that a client has sufficient SLOKs, earned through the various criteria. But this is much less specific than, for example, a domain bytes transferred log.

Documentation

Overview

Package osl implements the Obfuscated Server List (OSL) mechanism. This mechanism is a method of distributing server lists only to clients that demonstrate certain behavioral traits. Clients are seeded with Server List Obfuscation Keys (SLOKs) as they meet the configured criteria. These keys are stored and later combined to assemble keys to decrypt out-of-band distributed OSL files that contain server lists.

This package contains the core routines used in psiphond (to track client traits and issue SLOKs), clients (to manage SLOKs and decrypt OSLs), and automation (to create OSLs for distribution).

Index

Constants

View Source 
const (
	KEY_LENGTH_BYTES    = 32
	REGISTRY_FILENAME   = "osl-registry"
	OSL_FILENAME_FORMAT = "osl-%s"
)

Variables

This section is empty.

Functions

func GetOSLFileURL 

func GetOSLFileURL(baseURL string, oslID []byte) string

GetOSLFileURL returns the URL for an OSL file. Once the client has determined, from GetSeededOSLIDs, which OSLs it has sufficiently seeded, it calls this to fetch the OSLs for download and decryption. Clients are responsible for tracking whether the remote file has changed or not before downloading.

func GetOSLFilename 

func GetOSLFilename(baseDirectory string, oslID []byte) string

GetOSLFilename returns an appropriate filename for the resumable download destination for the OSL file.

func GetOSLRegistryFilename 

func GetOSLRegistryFilename(baseDirectory string) string

GetOSLRegistryFilename returns an appropriate filename for the resumable download destination for the OSL registry.

func GetOSLRegistryURL 

func GetOSLRegistryURL(baseURL string) string

GetOSLRegistryURL returns the URL for an OSL registry. Clients call this when fetching the registry from out-of-band distribution sites. Clients are responsible for tracking whether the remote file has changed or not before downloading.

func NewOSLReader added in v1.0.5 

func NewOSLReader(
	oslFileContent io.ReadSeeker,
	fileSpec *OSLFileSpec,
	lookup SLOKLookup,
	signingPublicKey string) (io.Reader, error)

NewOSLReader decrypts, authenticates and streams an OSL payload.

Types

type ClientSeedPortForward 

type ClientSeedPortForward struct {
	// contains filtered or unexported fields
}

ClientSeedPortForward map a client port forward, which is relaying traffic to a specific upstream address, to all seed state progress counters for SeedSpecs with subnets containing the upstream address. As traffic is relayed through the port forwards, the bytes transferred and duration count towards the progress of these SeedSpecs and associated SLOKs.

func (*ClientSeedPortForward) UpdateProgress 

func (portForward *ClientSeedPortForward) UpdateProgress(
	bytesRead, bytesWritten int64, durationNanoseconds int64)

UpdateProgress adds port forward bytes transferred and duration to all seed spec progresses associated with the port forward. If UpdateProgress is invoked after the SLOK time period has rolled over, any pending seeded SLOKs are issued and all progress is reset. UpdateProgress may be invoked concurrently by many psiphond port relay goroutines. The implementation of UpdateProgress prioritizes not blocking port forward relaying; a consequence of this lock-free design is that progress reported at the exact time of SLOK time period rollover may be dropped.

type ClientSeedProgress 

type ClientSeedProgress struct {
	// contains filtered or unexported fields
}

ClientSeedProgress tracks client progress towards seeding SLOKs for a particular scheme.

type ClientSeedState 

type ClientSeedState struct {
	// contains filtered or unexported fields
}

ClientSeedState tracks the progress of a client towards seeding SLOKs across all schemes the client qualifies for.

func (*ClientSeedState) ClearSeedPayload 

func (state *ClientSeedState) ClearSeedPayload()

ClearSeedPayload resets the accumulated SLOK payload (but not SLOK progress). psiphond calls this after the client has acknowledged receipt of a payload.

func (*ClientSeedState) GetSeedPayload 

func (state *ClientSeedState) GetSeedPayload() *SeedPayload

GetSeedPayload issues any pending SLOKs and returns the accumulated SLOKs for a given client. psiphond will calls this when it receives signalIssueSLOKs which is the trigger to check for new SLOKs. Note: caller must not modify the SLOKs in SeedPayload.SLOKs as these are shared data.

func (*ClientSeedState) Hibernate 

func (state *ClientSeedState) Hibernate()

Hibernate clears references to short-lived objects (currently, signalIssueSLOKs) so that a ClientSeedState can be stored for later resumption without blocking garbage collection of the short-lived objects.

The ClientSeedState will still hold references to its Config; the caller is responsible for discarding hibernated seed states when the config changes.

The caller should ensure that all ClientSeedPortForwards associated with this ClientSeedState are closed before hibernation.

func (*ClientSeedState) NewClientSeedPortForward 

func (state *ClientSeedState) NewClientSeedPortForward(
	upstreamIPAddress net.IP) *ClientSeedPortForward

NewClientSeedPortForward creates a new client port forward traffic progress tracker. Port forward progress reported to the ClientSeedPortForward is added to seed state progress for all seed specs containing upstreamIPAddress in their subnets. The return value will be nil when activity for upstreamIPAddress does not count towards any progress. NewClientSeedPortForward may be invoked concurrently by many psiphond port forward establishment goroutines.

func (*ClientSeedState) Resume 

func (state *ClientSeedState) Resume(
	signalIssueSLOKs chan struct{})

Resume resumes a hibernated ClientSeedState by resetting the required objects (currently, signalIssueSLOKs) cleared by Hibernate.

type Config 

type Config struct {
	common.ReloadableFile

	Schemes []*Scheme
}

Config is an OSL configuration, which consists of a list of schemes. The Reload function supports hot reloading of rules data while the process is running.

func LoadConfig 

func LoadConfig(configJSON []byte) (*Config, error)

LoadConfig loads, validates, and initializes a JSON encoded OSL configuration.

func NewConfig 

func NewConfig(filename string) (*Config, error)

NewConfig initializes a Config with the settings in the specified file.

func (*Config) CurrentOSLIDs added in v1.0.5 

func (config *Config) CurrentOSLIDs(schemeIndex int) (map[string]string, error)

CurrentOSLIDs returns a mapping from each propagation channel ID in the specified scheme to the corresponding current time period, hex-encoded OSL ID.

func (*Config) NewClientSeedState 

func (config *Config) NewClientSeedState(
	clientRegion, propagationChannelID string,
	signalIssueSLOKs chan struct{}) *ClientSeedState

NewClientSeedState creates a new client seed state to track client progress towards seeding SLOKs. psiphond maintains one ClientSeedState for each connected client.

A signal is sent on signalIssueSLOKs when sufficient progress has been made that a new SLOK *may* be issued. psiphond will receive the signal and then call GetClientSeedPayload/IssueSLOKs to issue SLOKs, generate payload, and send to the client. The sender will not block sending to signalIssueSLOKs; the channel should be appropriately buffered.

func (*Config) Pave 

func (config *Config) Pave(
	startTime time.Time,
	endTime time.Time,
	propagationChannelID string,
	signingPublicKey string,
	signingPrivateKey string,
	paveServerEntries map[string][]string,
	omitMD5SumsSchemes []int,
	omitEmptyOSLsSchemes []int,
	logCallback func(*PaveLogInfo)) ([]*PaveFile, error)

Pave creates the full set of OSL files, for all schemes in the configuration, to be dropped in an out-of-band distribution site. Only OSLs for the propagation channel ID associated with the distribution site are paved. This function is used by automation.

The Name component of each file relates to the values returned by the client functions GetRegistryURL and GetOSLFileURL.

Pave returns a pave file for the entire registry of all OSLs from epoch to endTime, and a pave file for each OSL. paveServerEntries is a map from hex-encoded OSL IDs to server entries to pave into that OSL. When entries are found, OSL will contain those entries, newline separated. Otherwise the OSL will still be issued, but be empty (unless the scheme is in omitEmptyOSLsSchemes). The server entries are paved in string value sort order, ensuring that the OSL content remains constant as long as the same _set_ of server entries is input.

If startTime is specified and is after epoch, the pave file will contain OSLs for the first period at or after startTime.

As OSLs outside the epoch-endTime range will no longer appear in the registry, Pave is intended to be used to create the full set of OSLs for a distribution site; i.e., not incrementally.

Automation is responsible for consistently distributing server entries to OSLs in the case where OSLs are repaved in subsequent calls.

type KeyShares 

type KeyShares struct {
	Threshold   int
	BoxedShares [][]byte
	SLOKIDs     [][]byte
	KeyShares   []*KeyShares
}

KeyShares is a tree data structure which describes the key splits used to divide a secret key. BoxedShares are encrypted shares of the key, and #Threshold amount of decrypted BoxedShares are required to reconstruct the secret key. The keys for BoxedShares are either SLOKs (referenced by SLOK ID) or random keys that are themselves split as described in child KeyShares.

type KeySplit 

type KeySplit struct {
	Total     int
	Threshold int
}

KeySplit defines a secret key splitting scheme where the secret is split into n (total) shares and any K (threshold) of N shares must be known to recostruct the split secret.

type OSLFileSpec 

type OSLFileSpec struct {
	ID        []byte
	KeyShares *KeyShares
	MD5Sum    []byte
}

An OSLFileSpec includes an ID which is used to reference the OSL file and describes the key splits used to divide the OSL file key along with the SLOKs required to reassemble those keys.

The MD5Sum field is a checksum of the contents of the OSL file to be used to skip redownloading previously downloaded files. MD5 is not cryptographically secure and this checksum is not relied upon for OSL verification. MD5 is used for compatibility with out-of-band distribution hosts.

type PaveFile 

type PaveFile struct {
	Name     string
	Contents []byte
}

PaveFile describes an OSL data file to be paved to an out-of-band distribution drop site. There are two types of files: a registry, which describes how to assemble keys for OSLs, and the encrypted OSL files.

type PaveLogInfo 

type PaveLogInfo struct {
	FileName             string
	SchemeIndex          int
	PropagationChannelID string
	OSLID                string
	OSLTime              time.Time
	OSLDuration          time.Duration
	ServerEntryCount     int
}

type Registry 

type Registry struct {
	FileSpecs []*OSLFileSpec
}

Registry describes a set of OSL files.

type RegistryStreamer added in v1.0.5 

type RegistryStreamer struct {
	// contains filtered or unexported fields
}

RegistryStreamer authenticates and processes a JSON encoded OSL registry. The streamer processes the registry without loading the entire file into memory, parsing each OSL file spec in turn and returning those OSL file specs for which the client has sufficient SLOKs to reassemble the OSL key and decrypt.

At this stage, SLOK reassembly simply does SLOK ID lookups and threshold counting and does not derive keys for every OSL. This allows the client to defer key derivation until NewOSLReader for cases where it has not already imported the OSL.

The client's propagation channel ID is used implicitly: it determines the base URL used to download the registry and OSL files. If the client has seeded SLOKs from a propagation channel ID different than the one associated with its present base URL, they will not appear in the registry and not be used.

func NewRegistryStreamer added in v1.0.5 

func NewRegistryStreamer(
	registryFileContent io.ReadSeeker,
	signingPublicKey string,
	lookup SLOKLookup) (*RegistryStreamer, error)

NewRegistryStreamer creates a new RegistryStreamer.

func (*RegistryStreamer) Next added in v1.0.5 

func (s *RegistryStreamer) Next() (*OSLFileSpec, error)

Next returns the next OSL file spec that the client has sufficient SLOKs to decrypt. The client calls NewOSLReader with the file spec to process that OSL. Next returns nil at EOF.

type SLOK 

type SLOK struct {
	ID  []byte
	Key []byte
}

SLOK is a seeded SLOK issued to a client. The client will store the SLOK in its local database; look it up by ID when checking which OSLs it can reassemble keys for; and use the key material to reassemble OSL file keys.

type SLOKLookup 

type SLOKLookup func([]byte) []byte

SLOKLookup is a callback to lookup SLOK keys by ID.

type Scheme 

type Scheme struct {

	// Epoch is the start time of the scheme, the start time of the
	// first OSL and when SLOKs will first be issued. It must be
	// specified in UTC and must be a multiple of SeedPeriodNanoseconds.
	Epoch string

	// Regions is a list of client country codes this scheme applies to.
	// If empty, the scheme applies to all regions.
	Regions []string

	// PropagationChannelIDs is a list of client propagtion channel IDs
	// this scheme applies to. Propagation channel IDs are an input
	// to SLOK key derivation.
	PropagationChannelIDs []string

	// MasterKey is the base random key used for SLOK key derivation. It
	// must be unique for each scheme. It must be 32 random bytes, base64
	// encoded.
	MasterKey []byte

	// SeedSpecs is the set of different client network activity patterns
	// that will result in issuing SLOKs. For a given time period, a distinct
	// SLOK is issued for each SeedSpec.
	// Duplicate subnets may appear in multiple SeedSpecs.
	SeedSpecs []*SeedSpec

	// SeedSpecThreshold is the threshold scheme for combining SLOKs to
	// decrypt an OSL. For any fixed time period, at least K (threshold) of
	// N (total) SLOKs from the N SeedSpecs must be seeded for a client to be
	// able to reassemble the OSL key.
	// Limitation: thresholds must be at least 2.
	SeedSpecThreshold int

	// SeedPeriodNanoseconds is the time period granularity of SLOKs.
	// New SLOKs are issued every SeedPeriodNanoseconds. Client progress
	// towards activity levels is reset at the end of each period.
	SeedPeriodNanoseconds int64

	// KeySplits is the time period threshold scheme layered on top of the
	// SeedSpecThreshold scheme for combining SLOKs to decrypt an OSL.
	// There must be at least one level. For one level, any K (threshold) of
	// N (total) SeedSpec SLOK groups must be sufficiently seeded for a client
	// to be able to reassemble the OSL key. When an additional level is
	// specified, then K' of N' groups of N of K SeedSpec SLOK groups must be
	// sufficiently seeded. And so on. The first level in the list is the
	// lowest level. The time period for OSLs is determined by the totals in
	// the KeySplits.
	//
	// Example:
	//
	//   SeedSpecs = <3 specs>
	//   SeedSpecThreshold = 2
	//   SeedPeriodNanoseconds = 100,000,000 = 100 milliseconds
	//   SeedPeriodKeySplits = [{10, 7}, {60, 5}]
	//
	//   In these scheme, up to 3 distinct SLOKs, one per spec, are issued
	//   every 100 milliseconds.
	//
	//   Distinct OSLs are paved for every minute (60 seconds). Each OSL
	//   key is split such that, for those 60 seconds, a client must seed
	//   2/3 spec SLOKs for 7 of 10 consecutive 100 ms. time periods within
	//   a second, for any 5 of 60 seconds within the minute.
	//
	SeedPeriodKeySplits []KeySplit
	// contains filtered or unexported fields
}

Scheme defines a OSL seeding and distribution strategy. SLOKs to decrypt OSLs are issued based on client network activity -- defined in the SeedSpecs -- and time. OSLs are created for periods of time and can be decrypted by clients that are seeded with a sufficient selection of SLOKs for that time period. Distribution of server entries to OSLs is delegated to automation.

func (*Scheme) GetOSLDuration 

func (scheme *Scheme) GetOSLDuration() time.Duration

GetOSLDuration returns the total time duration of an OSL, which is a function of the scheme's SeedPeriodNanoSeconds, the duration of a single SLOK, and the scheme's SeedPeriodKeySplits, the number of SLOKs associated with an OSL.

type SeedPayload 

type SeedPayload struct {
	SLOKs []*SLOK
}

SeedPayload is the list of seeded SLOKs sent to a client.

type SeedSpec 

type SeedSpec struct {
	Description     string
	ID              []byte
	UpstreamSubnets []string
	Targets         TrafficValues
}

SeedSpec defines a client traffic pattern that results in a seeded SLOK. For each time period, a unique SLOK is issued to a client that meets the traffic levels specified in Targets. All upstream port forward traffic to UpstreamSubnets is counted towards the targets.

ID is a SLOK key derivation component and must be 32 random bytes, base64 encoded. UpstreamSubnets is a list of CIDRs. Description is not used; it's for JSON config file comments.

type TrafficValues 

type TrafficValues struct {
	BytesRead                      int64
	BytesWritten                   int64
	PortForwardDurationNanoseconds int64
}

TrafficValues defines a client traffic level that seeds a SLOK. BytesRead and BytesWritten are the minimum bytes transferred counts to seed a SLOK. Both UDP and TCP data will be counted towards these totals. PortForwardDurationNanoseconds is the duration that a TCP or UDP port forward is active (not connected, in the UDP case). All threshold settings must be met to seed a SLOK; any threshold may be set to 0 to be trivially satisfied.

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.