README

iprepd

iprepd is a centralized reputation daemon that can be used to store reputation information for various objects such as IP addresses and retrieve reputation scores for the objects.

The project initially focused on managing reputation information for only IP addresses, but has since been expanded to allow reputation tracking for other types of values such as account names or email addresses.

The daemon provides an HTTP API for requests, and uses a Redis server as the backend storage mechanism. Multiple instances of the daemon can be deployed using the same Redis backend.

Note: Support for legacy endpoints, such as GET /127.0.0.1 has been discontinued. Typed endpoints must now be used. For more information see the API documentation below. Some minor modification to client code is required to convert from the old endpoints.

Configuration

Configuration is done through the configuration file, by default ./iprepd.yaml. The location can be overridden with the -c flag.

See iprepd.yaml.sample for an example configuration.

Building the Docker image

./write_version_json.sh
docker build -t iprepd:latest .

Docker images are also published.

docker pull mozilla/iprepd:latest
docker run -ti --rm -v `pwd`/iprepd.yaml:/app/iprepd.yaml mozilla/iprepd:latest

API

Authentication

iprepd supports two forms of authentication. Clients can authenticate to the service using either standard API keys, or by using Hawk authentication.

Standard API key authentication can be configured in the configuration file in the apikey (for read/write) and ROapikey (for a read-only user) sections under the auth section in the configuration file. To use API key authentication, clients should send an Authorization header that is of format APIKey <apikey>.

Similarly, Hawk authentication is configured under the hawk and ROhawk sections in the configuration file. To use Hawk authentication clients need to include the hawk authentication header in the Authorization header when making a request.

Endpoints
GET /type/ip/10.0.0.1

Request the reputation for an object of a given type. Responds with 200 and a JSON document describing the reputation if found. Responds with a 404 if the object is unknown to iprepd, or is in the exceptions list.

The current supported object types are ip for an IP address and email for an email address.

The response body may include a decayafter element if the reputation for the address was changed with a recovery suppression applied. If the timestamp is present, it indicates the time after which the reputation for the address will begin to recover.

Response body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"reputation": 75,
	"reviewed": false,
	"lastupdated": "2018-04-23T18:25:43.511Z"
}
DELETE /type/ip/10.0.0.1

Deletes the reputation entry for the requested object of the specified type.

PUT /type/ip/10.0.0.1

Sets a reputation score for the specified object of the specified type. A reputation JSON document must be provided with the request body. The reputation field must be provided in the document. The reviewed field can be included and set to true to toggle the reviewed field for a given reputation entry.

Note that if the reputation decays back to 100, if the reviewed field is set on the entry it will toggle back to false.

The reputation will begin to decay back to 100 immediately for the address based on the decay settings in the configuration file. If it is desired that the reputation should not decay for a period of time, the decayafter field can be set with a timestamp to indicate when the reputation decay logic should begin to be applied for the entry.

Request body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"reputation": 75
}
PUT /violations/type/ip/10.0.0.1

Applies a violation penalty to the specified object of the specified type.

If an unknown violation penalty is submitted, this endpoint will still return 200, but the error will be logged.

If desired, suppress_recovery can be included in the request body and set to an integer which indicates the number of seconds that must elapse before the reputation for this entry will begin to decay back to 100. If this setting is not included, the reputation will begin to decay immediately. If the violation is being applied to an existing entry, the suppress_recovery field will only be applied if the existing entry has no current recovery suppression, or the specified recovery suppression time frame would result in a time in the future beyond which the entry currently has. If suppress_recovery is included it must be less than 1209600 (14 days).

Request body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"violation": "violation1"
}
PUT /violations/type/ip

Applies a violation penalty to a multiple objects of a given type.

If an unknown violation penalty is submitted, this endpoint will still return 200, but the error will be logged.

Request body 
[
	{"object": "10.0.0.1", "type": "ip", "violation": "violation1"},
	{"object": "10.0.0.2", "type": "ip", "violation": "violation1"},
	{"object": "10.0.0.3", "type": "ip", "violation": "violation2"}
]
GET /violations

Returns violations configured in iprepd in a JSON document.

Response body 
[
	{"name": "violation1", "penalty": 5, "decreaselimit": 50},
	{"name": "violation2", "penalty": 25, "decreaselimit": 0},
]
GET /dump

Returns all reputation entries.

Note: This makes use of the KEYS redis command, which is known to be very slow. Use with care.

Response body 
[
  {"ip": "10.0.0.1", "reputation": 75, "reviewed": false, "lastupdated": "2018-04-23T18:25:43.511Z"},
  {"ip": "10.0.0.2", "reputation": 50, "reviewed": false, "lastupdated": "2018-04-23T18:31:27.457Z"},
  {"ip": "10.0.20.2", "reputation": 25, "reviewed": false, "lastupdated": "2018-04-23T17:22:42.230Z"},
]
GET /__heartbeat__

Service heartbeat endpoint.

GET /__lbheartbeat__

Service heartbeat endpoint.

GET /__version__

Return version data.

Acknowledgements

The API design and overall concept for this project are based on work done in Tigerblood.

Expand ▾ Collapse ▴

Documentation

Index

Constants

View Source 
const (
	// TypeIP is the object type for IP addresses
	TypeIP = "ip"

	// TypeEmail is the object type for email addresses
	TypeEmail = "email"
)

Variables

This section is empty.

Functions

func StartDaemon 

func StartDaemon(confpath string)

StartDaemon starts a new instance of iprepd using configuration file confpath.

Types

type Client 

type Client struct {
	// contains filtered or unexported fields
}

Client is the iprepd service client

func NewClient 

func NewClient(url, token string, httpClient *http.Client) (*Client, error)

NewClient is the default constructor for the client

func (*Client) ApplyViolation 

func (c *Client) ApplyViolation(vr *ViolationRequest) error

ApplyViolation submits a ViolationRequest to iprepd

func (*Client) BatchApplyViolation 

func (c *Client) BatchApplyViolation(typ string, vrs []ViolationRequest) error

BatchApplyViolation submits a batch of ViolationRequests to iprepd

func (*Client) DeleteReputation 

func (c *Client) DeleteReputation(objectType, object string) error

DeleteReputation deletes the reputation of a given object and type

func (*Client) Dump 

func (c *Client) Dump() ([]Reputation, error)

Dump retrieves all reputation entries

func (*Client) GetReputation 

func (c *Client) GetReputation(objectType, object string) (*Reputation, error)

GetReputation fetches the reputation of a given object and type

func (*Client) GetViolations 

func (c *Client) GetViolations() ([]Violation, error)

GetViolations gets all existing violations on the server

func (*Client) Heartbeat 

func (c *Client) Heartbeat() (bool, error)

Heartbeat checks whether an IPrepd deployment is healthy / reachable

func (*Client) LBHeartbeat 

func (c *Client) LBHeartbeat() (bool, error)

LBHeartbeat checks whether an IPrepd LB is healthy / reachable

func (*Client) SetReputation 

func (c *Client) SetReputation(r *Reputation) error

SetReputation updates the reputation of a given object and type to a given score

func (*Client) Version 

func (c *Client) Version() (*VersionResponse, error)

Version retrieves the version of the IPrepd deployment

type Reputation 

type Reputation struct {
	// Object is the object associated with the reputation entry. For example
	// if the type is "ip", object will be an IP address.
	Object string `json:"object"`

	// Type describes the type of object the reputation entry is for
	Type string `json:"type"`

	// Reputation is the reputation score for the object, ranging from 0 to
	// 100 where 100 indicates no violations have been applied to it.
	Reputation int `json:"reputation"`

	// Reviewed is true if the entry has been manually reviewed, this flag indicates
	// a firm confidence in the entry.
	Reviewed bool `json:"reviewed"`

	// LastUpdated indicates when a reputation was last either set manually or via
	// a violation on this entry
	LastUpdated time.Time `json:"lastupdated"`

	// DecayAfter is used to temporarily stop reputation recovery until after the
	// current time has passed the time indicated by DecayAfter. This can be used
	// to for example enforce a mandatory minimum reputation decrease for an object
	// for a set period of time.
	DecayAfter time.Time `json:"decayafter,omitempty"`
}

Reputation stores information related to the reputation of a given object

func (*Reputation) Validate 

func (r *Reputation) Validate() error

Validate performs validation of a Reputation type.

type VersionResponse 

type VersionResponse struct {
	Commit  string `json:"commit"`
	Version string `json:"version"`
	Source  string `json:"source"`
	Build   string `json:"build"`
}

VersionResponse is the response payload from the /__version__ endpoint

type Violation 

type Violation struct {
	// Name of violation as specified in iprepd cfg
	Name string `json:"name"`

	// Penalty is how many points a reputation will be decreased by if this
	// violation is submitted for an object
	Penalty int `json:"penalty"`

	// DecreaseLimit is the lowest possible value this violation will decrease a
	// reputation to. Since the same violation can be applied multiple times to
	// the same object, this can be used to place a lower bound on the total decrease.
	DecreaseLimit int `json:"decreaselimit"`
}

Violation describes a violation penalty that can be applied to an object.

type ViolationRequest 

type ViolationRequest struct {
	// The violation name to be applied
	Violation string `json:"violation,omitempty"`

	// The object the violation should be applied to.
	Object string `json:"object,omitempty"`

	// The type of object (e.g., ip).
	Type string `json:"type,omitempty"`

	// An optional recovery suppression value in seconds. If set, it indicates the
	// number of seconds which must pass before the reputation for the object will
	// begin to recover.
	SuppressRecovery int `json:"suppress_recovery,omitempty"`

	// The IP field supports reverse compatibility with older clients. It is essentially
	// the same thing as passing an IP address in the object field, with a type set to
	// ip.
	IP string `json:"ip,omitempty"`
}

ViolationRequest represents the structure used to apply a violation to a given object. This structure is used as the basis for unmarshaling requests to violation handlers in the API.

func (*ViolationRequest) Fixup 

func (v *ViolationRequest) Fixup(typestr string)

Fixup is used to convert legacy format violations

func (*ViolationRequest) Validate 

func (v *ViolationRequest) Validate() error

Validate performs validation of a ViolationRequest type

Source Files

View all

Directories

Path Synopsis
cmd/iprepd