README

usermetadb

The User Metadata Database (usermetadb) stores long-term information about user access patterns in order to detect anomalous behavior and implement other safety checks. It strives to do so while respecting the anonymity of the users, focusing on information that is actually useful to them.

In practical terms, it stores the following information on every successful login:

  • timestamps, quantized/fuzzed to a sufficiently large amount (1h?) to make correlation difficult
  • location, stored at the country level based on user IP
  • device information based on long-term cookies

The idea is that this is enough information to provide users with meaningful summaries such as "you have just logged in from a new/unknown device" and "you logged in earlier today with Chrome on a mobile Android device", without storing de-anonymizing information on the server side.

The cookie-based device detection might present an issue from this point of view, because it allows to establish a forensic link between a specific device and an account if one is in possession of the server-side log database (only partially mitigated by the fact that the cookie is encrypted).

usermetadb also stores last-login information for internal infrastructure maintenance. The idea is to retain the minimal amount of information to perform tasks such as "disable accounts that have not been active for more than N years".

API

The server exports an API over HTTP/HTTPS, all requests should be made using the POST method and an application/json Content-Type. The request body should contain a JSON-encoded request object. Responses will similarly be JSON-encoded.

The API is split into two conceptually separate sets, the log API and the analysis API.

Log API

/api/add_log (AddLogRequest)

Stores a new log entry for a user in the database. The request must be a LogEntry object. The method returns an empty response. If the log entry contains device information, the list of devices for the specified user is updated with that information.

/api/get_user_logs (GetUserLogsRequest) -> GetUserLogsResponse

Returns recent logs for a specific user.

/api/get_user_devices (GetUserDevicesRequest) -> GetUserDevicesResponse

Returns the list of known devices for a user.

Analysis API

/api/check_device (CheckDeviceRequest) -> CheckDeviceResponse

Returns information about a device, whether we have seen it before, if the localization information matches the historical trend, etc.

Last-login API

/api/set_last_login (SetLastLoginRequest)

Stores the last login of a user in the database. The request must be a LastLoginEntry object. The method returns an empty response. The service name must be specified in the last login entry.

/api/get_last_login (GetLastLoginRequest) -> GetLastLoginResponse

Returns the last login of a given user. If the service name is specified it returns the last login for that specific service, otherwise return last login for all services.

/api/get_unused_accounts (GetUnusedAccountsRequest) -> GetUnusedAccountsResponse

Returns accounts that have not been used in a specified amount of time.

Configuration

The configuration is contained in a YAML-encoded file, normally /etc/user-meta-server.yml (this can be changed using the --config command-line flag).

The known attributes are:

  • db_uri is the database URI. As currently only the sqlite3 driver is supported, this is just the path to the database file.
  • http_server specifies standard parameters for the HTTP server:
    • tls contains the server-side TLS configuration:
      • cert is the path to the server certificate
      • key is the path to the server's private key
      • ca is the path to the CA used to validate clients
      • acl specifies TLS-based access controls, a list of entries with the following attributes:
        • path is a regular expression to match the request URL path
        • cn is a regular expression that must match the CommonName part of the subject of the client certificate
Expand ▾ Collapse ▴

Documentation

Index

Constants

View Source
const (
	LogTypeLogin          = "login"
	LogTypeLogout         = "logout"
	LogTypePasswordReset  = "password_reset"
	LogTypePasswordChange = "password_change"
	LogTypeOTPEnabled     = "otp_enabled"
	LogTypeOTPDisabled    = "otp_disabled"
)
View Source
const (
	LoginMethodPassword = "password"
	LoginMethodOTP      = "otp"
	LoginMethodU2F      = "u2f"
)

Variables

This section is empty.

Functions

This section is empty.

Types

type AddLogRequest

type AddLogRequest struct {
	Log *LogEntry `json:"log"`
}

type AddLogResponse

type AddLogResponse struct{}

type CheckDeviceRequest

type CheckDeviceRequest struct {
	Username   string      `json:"username"`
	DeviceInfo *DeviceInfo `json:"device_info"`
}

type CheckDeviceResponse

type CheckDeviceResponse struct {
	Seen bool `json:"seen"`
}

type DeviceInfo

type DeviceInfo struct {
	ID         string `json:"id"`
	RemoteAddr string `json:"remote_addr"`
	RemoteZone string `json:"remote_zone"`
	UserAgent  string `json:"user_agent"`
	Browser    string `json:"browser"`
	OS         string `json:"os"`
	Mobile     bool   `json:"mobile"`
}

    DeviceInfo holds information about the client device. We use a simple persistent cookie to track the same client device across multiple session.

    func DecodeDeviceInfoFromMap

    func DecodeDeviceInfoFromMap(m map[string]string, prefix string) *DeviceInfo

    func (*DeviceInfo) EncodeToMap

    func (d *DeviceInfo) EncodeToMap(m map[string]string, prefix string)

    type GetLastLoginRequest

    type GetLastLoginRequest struct {
    	Username string `json:"username"`
    	Service  string `json:"service,omitempty"`
    }

    type GetLastLoginResponse

    type GetLastLoginResponse struct {
    	Results []*LastLoginEntry `json:"result"`
    }

    type GetUnusedAccountsRequest

    type GetUnusedAccountsRequest struct {
    	Usernames []string `json:"usernames"`
    	Days      int      `json:"days"`
    }

    type GetUnusedAccountsResponse

    type GetUnusedAccountsResponse struct {
    	UnusedUsernames []string `json:"unused_usernames"`
    }

    type GetUserDevicesRequest

    type GetUserDevicesRequest struct {
    	Username string `json:"username"`
    }

    type GetUserDevicesResponse

    type GetUserDevicesResponse struct {
    	Devices []*MetaDeviceInfo `json:"devices"`
    }

    type GetUserLogsRequest

    type GetUserLogsRequest struct {
    	Username string `json:"username"`
    	MaxDays  int    `json:"max_days"`
    	Limit    int    `json:"limit"`
    }

    type GetUserLogsResponse

    type GetUserLogsResponse struct {
    	Results []*LogEntry `json:"result"`
    }

    type LastLoginEntry

    type LastLoginEntry struct {
    	Timestamp time.Time `json:"timestamp"`
    	Username  string    `json:"username"`
    	Service   string    `json:"service"`
    }

    func (*LastLoginEntry) Validate

    func (e *LastLoginEntry) Validate() error

    type LogEntry

    type LogEntry struct {
    	Timestamp   time.Time   `json:"timestamp"`
    	Username    string      `json:"username"`
    	Type        string      `json:"log_type"`
    	Message     string      `json:"message,omitempty"`
    	Service     string      `json:"service,omitempty"`
    	LoginMethod string      `json:"login_method,omitempty"`
    	DeviceInfo  *DeviceInfo `json:"device_info,omitempty"`
    }

      LogEntry represents an authentication event in the user-specific log.

      func (*LogEntry) Validate

      func (e *LogEntry) Validate() error

      type MetaDeviceInfo

      type MetaDeviceInfo struct {
      	DeviceInfo *DeviceInfo `json:"device_info"`
      	FirstSeen  time.Time   `json:"first_seen"`
      	LastSeen   time.Time   `json:"last_seen"`
      	NumLogins  int         `json:"num_logins"`
      }

      type SetLastLoginRequest

      type SetLastLoginRequest struct {
      	LastLogin *LastLoginEntry `json:"last_login"`
      }

      type SetLastLoginResponse

      type SetLastLoginResponse struct{}

      Source Files

      Directories

      Path Synopsis
      cmd