usermetadb

README

usermetadb

The User Metadata Database (usermetadb) stores long-term information about user access patterns in order to detect anomalous behavior and implement other safety checks. It strives to do so while respecting the anonymity of the users, focusing on information that is actually useful to them.

In practical terms, it stores the following information on every successful login:

• timestamps, quantized/fuzzed to a sufficiently large amount (1h?) to make correlation difficult
• location, stored at the country level based on user IP
• device information based on long-term cookies

The idea is that this is enough information to provide users with meaningful summaries such as "you have just logged in from a new/unknown device" and "you logged in earlier today with Chrome on a mobile Android device", without storing de-anonymizing information on the server side.

The cookie-based device detection might present an issue from this point of view, because it allows to establish a forensic link between a specific device and an account if one is in possession of the server-side log database (only partially mitigated by the fact that the cookie is encrypted).

usermetadb also stores last-login information for internal infrastructure maintenance. The idea is to retain the minimal amount of information to perform tasks such as "disable accounts that have not been active for more than N years".

API

The server exports an API over HTTP/HTTPS, all requests should be made using the POST method and an application/json Content-Type. The request body should contain a JSON-encoded request object. Responses will similarly be JSON-encoded.

The API is split into two conceptually separate sets, the log API and the analysis API.

Log API

/api/add_log (AddLogRequest)

Stores a new log entry for a user in the database. The request must be a LogEntry object. The method returns an empty response. If the log entry contains device information, the list of devices for the specified user is updated with that information.

/api/get_user_logs (GetUserLogsRequest) -> GetUserLogsResponse

Returns recent logs for a specific user.

/api/get_user_devices (GetUserDevicesRequest) -> GetUserDevicesResponse

Returns the list of known devices for a user.

Analysis API

/api/check_device (CheckDeviceRequest) -> CheckDeviceResponse

Returns information about a device, whether we have seen it before, if the localization information matches the historical trend, etc.

Last-login API

/api/set_last_login (SetLastLoginRequest)

Stores the last login of a user in the database. The request must be a LastLoginEntry object. The method returns an empty response. The service name must be specified in the last login entry.

/api/get_last_login (GetLastLoginRequest) -> GetLastLoginResponse

Returns the last login of a given user. If the service name is specified it returns the last login for that specific service, otherwise return last login for all services.

/api/get_unused_accounts (GetUnusedAccountsRequest) -> GetUnusedAccountsResponse

Returns accounts that have not been used in a specified amount of time.

Configuration

The configuration is contained in a YAML-encoded file, normally /etc/user-meta-server.yml (this can be changed using the --config command-line flag).

The known attributes are:

• db_uri is the database URI. As currently only the sqlite3 driver is supported, this is just the path to the database file.
• http_server specifies standard parameters for the HTTP server:
  • tls contains the server-side TLS configuration:
    • cert is the path to the server certificate
    • key is the path to the server's private key
    • ca is the path to the CA used to validate clients
    • acl specifies TLS-based access controls, a list of entries with the following attributes:
      • path is a regular expression to match the request URL path
      • cn is a regular expression that must match the CommonName part of the subject of the client certificate

Documentation

Index

• Constants
• type AddLogRequest
• type AddLogResponse
• type CheckDeviceRequest
• type CheckDeviceResponse
• type DeviceInfo
  • func DecodeDeviceInfoFromMap(m map[string]string, prefix string) *DeviceInfo
  • func (d *DeviceInfo) EncodeToMap(m map[string]string, prefix string)
• type GetLastLoginRequest
• type GetLastLoginResponse
• type GetUnusedAccountsRequest
• type GetUnusedAccountsResponse
• type GetUserDevicesRequest
• type GetUserDevicesResponse
• type GetUserLogsRequest
• type GetUserLogsResponse
• type LastLoginEntry
  • func (e *LastLoginEntry) Validate() error
• type LogEntry
  • func (e *LogEntry) Validate() error
• type MetaDeviceInfo
• type SetLastLoginRequest
• type SetLastLoginResponse

Constants

const (
	LogTypeLogin          = "login"
	LogTypeLogout         = "logout"
	LogTypePasswordReset  = "password_reset"
	LogTypePasswordChange = "password_change"
	LogTypeOTPEnabled     = "otp_enabled"
	LogTypeOTPDisabled    = "otp_disabled"
)

Types

type AddLogRequest

type AddLogRequest struct {
	Log *LogEntry `json:"log"`
}

type AddLogResponse

type AddLogResponse struct{}

type CheckDeviceRequest

type CheckDeviceRequest struct {
	Username   string      `json:"username"`
	DeviceInfo *DeviceInfo `json:"device_info"`
}

type CheckDeviceResponse

type CheckDeviceResponse struct {
	Seen bool `json:"seen"`
}

type DeviceInfo

type DeviceInfo struct {
	ID         string `json:"id"`
	RemoteAddr string `json:"remote_addr"`
	RemoteZone string `json:"remote_zone"`
	UserAgent  string `json:"user_agent"`
	Browser    string `json:"browser"`
	OS         string `json:"os"`
	Mobile     bool   `json:"mobile"`
}

DeviceInfo holds information about the client device. We use a simple persistent cookie to track the same client device across multiple session.

func DecodeDeviceInfoFromMap

func DecodeDeviceInfoFromMap(m map[string]string, prefix string) *DeviceInfo

func (*DeviceInfo) EncodeToMap

func (d *DeviceInfo) EncodeToMap(m map[string]string, prefix string)

type GetLastLoginRequest

type GetLastLoginRequest struct {
	Username string `json:"username"`
	Service  string `json:"service,omitempty"`
}

type GetLastLoginResponse

type GetLastLoginResponse struct {
	Results []*LastLoginEntry `json:"result"`
}

type GetUnusedAccountsRequest

type GetUnusedAccountsRequest struct {
	Usernames []string `json:"usernames"`
	Days      int      `json:"days"`
}

type GetUnusedAccountsResponse

type GetUnusedAccountsResponse struct {
	UnusedUsernames []string `json:"unused_usernames"`
}

type GetUserDevicesRequest

type GetUserDevicesRequest struct {
	Username string `json:"username"`
}

type GetUserDevicesResponse

type GetUserDevicesResponse struct {
	Devices []*MetaDeviceInfo `json:"devices"`
}

type GetUserLogsRequest

type GetUserLogsRequest struct {
	Username string `json:"username"`
	MaxDays  int    `json:"max_days"`
	Limit    int    `json:"limit"`
}

type GetUserLogsResponse

type GetUserLogsResponse struct {
	Results []*LogEntry `json:"result"`
}

type LastLoginEntry

type LastLoginEntry struct {
	Timestamp time.Time `json:"timestamp"`
	Username  string    `json:"username"`
	Service   string    `json:"service"`
}

func (*LastLoginEntry) Validate

func (e *LastLoginEntry) Validate() error

type LogEntry

type LogEntry struct {
	Timestamp            time.Time   `json:"timestamp"`
	Username             string      `json:"username"`
	Type                 string      `json:"log_type"`
	Message              string      `json:"message,omitempty"`
	Service              string      `json:"service,omitempty"`
	LoginMethod          string      `json:"login_method,omitempty"`
	LoginAuthenticatorID string      `json:"login_authenticator_id,omitempty"`
	DeviceInfo           *DeviceInfo `json:"device_info,omitempty"`
}

LogEntry represents an authentication event in the user-specific log.

func (*LogEntry) Validate

func (e *LogEntry) Validate() error

type MetaDeviceInfo

type MetaDeviceInfo struct {
	DeviceInfo *DeviceInfo `json:"device_info"`
	FirstSeen  time.Time   `json:"first_seen"`
	LastSeen   time.Time   `json:"last_seen"`
	NumLogins  int         `json:"num_logins"`
}

type SetLastLoginRequest

type SetLastLoginRequest struct {
	LastLogin *LastLoginEntry `json:"last_login"`
}

type SetLastLoginResponse

type SetLastLoginResponse struct{}