README

iprepd

iprepd is a centralized reputation daemon that can be used to store reputation information for various objects such as IP addresses and retrieve reputation scores for the objects.

The project initially focused on managing reputation information for only IP addresses, but has since been expanded to allow reputation tracking for other types of values such as account names or email addresses.

The daemon provides an HTTP API for requests, and uses a Redis server as the backend storage mechanism. Multiple instances of the daemon can be deployed using the same Redis backend.

Note: Support for legacy endpoints, such as GET /127.0.0.1 has been discontinued. Typed endpoints must now be used. For more information see the API documentation below. Some minor modification to client code is required to convert from the old endpoints.

Configuration

Configuration is done through the configuration file, by default ./iprepd.yaml. The location can be overridden with the -c flag.

See iprepd.yaml.sample for an example configuration.

Building the Docker image

./write_version_json.sh
docker build -t iprepd:latest .

Docker images are also published.

docker pull mozilla/iprepd:latest
docker run -ti --rm -v `pwd`/iprepd.yaml:/app/iprepd.yaml mozilla/iprepd:latest

API

Authentication

iprepd supports two forms of authentication. Clients can authenticate to the service using either standard API keys, or by using Hawk authentication.

Standard API key authentication can be configured in the configuration file in the apikey (for read/write) and ROapikey (for a read-only user) sections under the auth section in the configuration file. To use API key authentication, clients should send an Authorization header that is of format APIKey <apikey>.

Similarly, Hawk authentication is configured under the hawk and ROhawk sections in the configuration file. To use Hawk authentication clients need to include the hawk authentication header in the Authorization header when making a request.

Endpoints
GET /type/ip/10.0.0.1

Request the reputation for an object of a given type. Responds with 200 and a JSON document describing the reputation if found. Responds with a 404 if the object is unknown to iprepd, or is in the exceptions list.

The current supported object types are ip for an IP address and email for an email address.

The response body may include a decayafter element if the reputation for the address was changed with a recovery suppression applied. If the timestamp is present, it indicates the time after which the reputation for the address will begin to recover.

Response body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"reputation": 75,
	"reviewed": false,
	"lastupdated": "2018-04-23T18:25:43.511Z"
}
DELETE /type/ip/10.0.0.1

Deletes the reputation entry for the requested object of the specified type.

PUT /type/ip/10.0.0.1

Sets a reputation score for the specified object of the specified type. A reputation JSON document must be provided with the request body. The reputation field must be provided in the document. The reviewed field can be included and set to true to toggle the reviewed field for a given reputation entry.

Note that if the reputation decays back to 100, if the reviewed field is set on the entry it will toggle back to false.

The reputation will begin to decay back to 100 immediately for the address based on the decay settings in the configuration file. If it is desired that the reputation should not decay for a period of time, the decayafter field can be set with a timestamp to indicate when the reputation decay logic should begin to be applied for the entry.

Request body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"reputation": 75
}
PUT /violations/type/ip/10.0.0.1

Applies a violation penalty to the specified object of the specified type.

If an unknown violation penalty is submitted, this endpoint will still return 200, but the error will be logged.

If desired, suppress_recovery can be included in the request body and set to an integer which indicates the number of seconds that must elapse before the reputation for this entry will begin to decay back to 100. If this setting is not included, the reputation will begin to decay immediately. If the violation is being applied to an existing entry, the suppress_recovery field will only be applied if the existing entry has no current recovery suppression, or the specified recovery suppression time frame would result in a time in the future beyond which the entry currently has. If suppress_recovery is included it must be less than 1209600 (14 days).

Request body 
{
	"object": "10.0.0.1",
        "type": "ip",
	"violation": "violation1"
}
PUT /violations/type/ip

Applies a violation penalty to a multiple objects of a given type.

If an unknown violation penalty is submitted, this endpoint will still return 200, but the error will be logged.

Request body 
[
	{"object": "10.0.0.1", "type": "ip", "violation": "violation1"},
	{"object": "10.0.0.2", "type": "ip", "violation": "violation1"},
	{"object": "10.0.0.3", "type": "ip", "violation": "violation2"}
]
GET /violations

Returns violations configured in iprepd in a JSON document.

Response body 
[
	{"name": "violation1", "penalty": 5, "decreaselimit": 50},
	{"name": "violation2", "penalty": 25, "decreaselimit": 0},
]
GET /dump

Returns all reputation entries.

Note: This makes use of the KEYS redis command, which is known to be very slow. Use with care.

Response body 
[
  {"ip": "10.0.0.1", "reputation": 75, "reviewed": false, "lastupdated": "2018-04-23T18:25:43.511Z"},
  {"ip": "10.0.0.2", "reputation": 50, "reviewed": false, "lastupdated": "2018-04-23T18:31:27.457Z"},
  {"ip": "10.0.20.2", "reputation": 25, "reviewed": false, "lastupdated": "2018-04-23T17:22:42.230Z"},
]
GET /__heartbeat__

Service heartbeat endpoint.

GET /__lbheartbeat__

Service heartbeat endpoint.

GET /__version__

Return version data.

Acknowledgements

The API design and overall concept for this project are based on work done in Tigerblood.

Expand ▾ Collapse ▴

Documentation

Index

Constants

View Source 
const (
	// TypeIP is the object type for IP addresses
	TypeIP = "ip"

	// TypeEmail is the object type for email addresses
	TypeEmail = "email"
)

Variables

This section is empty.

Functions

func StartDaemon 

func StartDaemon(confpath string)

    StartDaemon starts a new instance of iprepd using configuration file confpath.

    Types

    type Client 

    type Client struct {
	// contains filtered or unexported fields
}

      Client is the iprepd service client

      func NewClient 

      func NewClient(url, token string, httpClient *http.Client) (*Client, error)

        NewClient is the default constructor for the client

        func (*Client) ApplyViolation 

        func (c *Client) ApplyViolation(vr *ViolationRequest) error

          ApplyViolation submits a ViolationRequest to iprepd

          func (*Client) BatchApplyViolation 

          func (c *Client) BatchApplyViolation(typ string, vrs []ViolationRequest) error

            BatchApplyViolation submits a batch of ViolationRequests to iprepd

            func (*Client) DeleteReputation 

            func (c *Client) DeleteReputation(objectType, object string) error

              DeleteReputation deletes the reputation of a given object and type

              func (*Client) Dump 

              func (c *Client) Dump() ([]Reputation, error)

                Dump retrieves all reputation entries

                func (*Client) GetReputation 

                func (c *Client) GetReputation(objectType, object string) (*Reputation, error)

                  GetReputation fetches the reputation of a given object and type

                  func (*Client) GetViolations 

                  func (c *Client) GetViolations() ([]Violation, error)

                    GetViolations gets all existing violations on the server

                    func (*Client) Heartbeat 

                    func (c *Client) Heartbeat() (bool, error)

                      Heartbeat checks whether an IPrepd deployment is healthy / reachable

                      func (*Client) LBHeartbeat 

                      func (c *Client) LBHeartbeat() (bool, error)

                        LBHeartbeat checks whether an IPrepd LB is healthy / reachable

                        func (*Client) SetReputation 

                        func (c *Client) SetReputation(r *Reputation) error

                          SetReputation updates the reputation of a given object and type to a given score

                          func (*Client) Version 

                          func (c *Client) Version() (*VersionResponse, error)

                            Version retrieves the version of the IPrepd deployment

                            type Reputation 

                            type Reputation struct {
	// Object is the object associated with the reputation entry. For example
	// if the type is "ip", object will be an IP address.
	Object string `json:"object"`

	// Type describes the type of object the reputation entry is for
	Type string `json:"type"`

	// Reputation is the reputation score for the object, ranging from 0 to
	// 100 where 100 indicates no violations have been applied to it.
	Reputation int `json:"reputation"`

	// Reviewed is true if the entry has been manually reviewed, this flag indicates
	// a firm confidence in the entry.
	Reviewed bool `json:"reviewed"`

	// LastUpdated indicates when a reputation was last either set manually or via
	// a violation on this entry
	LastUpdated time.Time `json:"lastupdated"`

	// DecayAfter is used to temporarily stop reputation recovery until after the
	// current time has passed the time indicated by DecayAfter. This can be used
	// to for example enforce a mandatory minimum reputation decrease for an object
	// for a set period of time.
	DecayAfter time.Time `json:"decayafter,omitempty"`
}

                              Reputation stores information related to the reputation of a given object

                              func (*Reputation) Validate 

                              func (r *Reputation) Validate() error

                                Validate performs validation of a Reputation type.

                                type VersionResponse 

                                type VersionResponse struct {
	Commit  string `json:"commit"`
	Version string `json:"version"`
	Source  string `json:"source"`
	Build   string `json:"build"`
}

                                  VersionResponse is the response payload from the /__version__ endpoint

                                  type Violation 

                                  type Violation struct {
	// Name of violation as specified in iprepd cfg
	Name string `json:"name"`

	// Penalty is how many points a reputation will be decreased by if this
	// violation is submitted for an object
	Penalty int `json:"penalty"`

	// DecreaseLimit is the lowest possible value this violation will decrease a
	// reputation to. Since the same violation can be applied multiple times to
	// the same object, this can be used to place a lower bound on the total decrease.
	DecreaseLimit int `json:"decreaselimit"`
}

                                    Violation describes a violation penalty that can be applied to an object.

                                    type ViolationRequest 

                                    type ViolationRequest struct {
	// The violation name to be applied
	Violation string `json:"violation,omitempty"`

	// The object the violation should be applied to.
	Object string `json:"object,omitempty"`

	// The type of object (e.g., ip).
	Type string `json:"type,omitempty"`

	// An optional recovery suppression value in seconds. If set, it indicates the
	// number of seconds which must pass before the reputation for the object will
	// begin to recover.
	SuppressRecovery int `json:"suppress_recovery,omitempty"`

	// The IP field supports reverse compatibility with older clients. It is essentially
	// the same thing as passing an IP address in the object field, with a type set to
	// ip.
	IP string `json:"ip,omitempty"`
}

                                      ViolationRequest represents the structure used to apply a violation to a given object. This structure is used as the basis for unmarshaling requests to violation handlers in the API.

                                      func (*ViolationRequest) Fixup 

                                      func (v *ViolationRequest) Fixup(typestr string)

                                        Fixup is used to convert legacy format violations

                                        func (*ViolationRequest) Validate 

                                        func (v *ViolationRequest) Validate() error

                                          Validate performs validation of a ViolationRequest type

                                          Source Files

                                          View all

                                          Directories

                                          Path Synopsis
                                          cmd
                                          iprepd