The highest tagged major version is
README
¶
Go Redis IP Ranges (goripr)
goripr is an eficient way to store IP ranges in a redis database and mapping those ranges to specific strings.
This package wraps the widely used redis Go client and extends its feature set with a storage efficient mapping of IPv4 ranges to specific strings called reasons.
I intend to use this package in my VPN Detection, that's why the term "reason" is used. The term refers to a ban reason that is given when a player using a VPN (they usually do that with malicious intent) gets banned. The string can be used in any other way needed, especially containing JSON formatted data.
Idea
The general approach is to save the beginning and the end of a range into the database.
The beginning boundary has the property called LowerBound set to true and the last IP in a given range is called an upper boundary with the property UpperBound set to true.
Based on these properties it is possible to determine, how to cut existing boundaries, when new IP ranges are inserted into the database.
Problem it solves
The VPN detection and especially the ban server used to save all IPs from the given ranges with their corresponding reasons into the database. That is the trivial approach, but proved to be inefficient when having more than 100 million individual IPs stored in the Redis database. At it's peak the database needed ~7GB of RAM, which is not a feasible solution, especially when the source files containing the actual ranges in their respective masked shorthand notation (x.x.x.x/24) needed less than one MB of storage space.
Gains over the trivial approach
On the other hand, iterating over ~50k such range strings was also not a feasible solution, especially when the ban server should react within ~1 second. The compromise should be a slower reaction time compared to the triavial approach, but way less of a RAM overhead. I guess that the reduction of RAM usage by a factor of about 240x should also improve the response time significantly, as the ~7GB approach was burdening even high performance servers rather heavily. The current RAM that is being used is about 30MB, which is acceptable.
Input format of the package
# custom IP range
84.141.32.1 - 84.141.32.255
# single IP
84.141.32.1
# subnet mask
84.141.32.1/24
Example
package main
import (
"bufio"
"errors"
"flag"
"log"
"os"
"regexp"
"github.com/jxsl13/goripr"
)
var (
rdb *goripr.Client
splitRegex = regexp.MustCompile(`([0-9.\-\s/]+)#?\s*(.*)\s*$`)
defaultReason = "VPN - https://website.com"
addFile = ""
findIP = ""
)
func init() {
flag.StringVar(&addFile, "add", "", "-add filename.txt")
flag.StringVar(&findIP, "find", "", "-find 123.0.0.1")
flag.Parse()
c, err := goripr.NewClient(goripr.Options{
Addr: "localhost:6379",
DB: 0,
})
rdb = c
if err != nil {
panic(err)
}
}
func parseLine(line string) (ip, reason string, err error) {
if matches := splitRegex.FindStringSubmatch(line); len(matches) > 0 {
return matches[1], matches[2], nil
}
return "", "", errors.New("empty")
}
func addIPsToDatabase(filename string) error {
file, err := os.Open(filename)
if err != nil {
return err
}
scanner := bufio.NewScanner(file)
for scanner.Scan() {
ip, reason, err := parseLine(scanner.Text())
if err != nil {
continue
}
if reason == "" {
reason = defaultReason
}
err = rdb.Insert(ip, reason)
if err != nil {
if !errors.Is(err, goripr.ErrInvalidRange) {
log.Println(err, "Input:", ip)
}
continue
}
}
return nil
}
func main() {
defer rdb.Close()
if addFile != "" {
err := addIPsToDatabase(addFile)
if err != nil {
log.Printf("exit: %v", err)
return
}
}
if findIP != "" {
reason, err := rdb.Find(findIP)
if err != nil {
log.Println("error:", err)
return
}
log.Println("IP:", findIP, "Reason:", reason)
return
}
}
Example text file
84.141.32.1 - 84.141.32.255 # any range where the first IP is smaller than the second
2.56.92.0/22 # VPN subnet masking
# without a reason (uses default reason)
2.56.140.0/24
TODO
- Optional Cache of requested IPs for like 24 hours in order to improve response time for recurring requests (rejoining players)
Documentation
¶
Index ¶
- Constants
- Variables
- type Client
- func (c *Client) Close() error
- func (c *Client) Find(ip string) (reason string, err error)
- func (c *Client) Flush() error
- func (c *Client) Insert(ipRange, reason string) error
- func (c *Client) Remove(ipRange string) error
- func (c *Client) Reset() error
- func (c *Client) UpdateReasonOf(ip string, fn UpdateFunc) (err error)
- type Error
- type Options
- type UpdateFunc
Constants ¶
const ( // ErrConnectionFailed is returned when the connection to the redis database fails. ErrConnectionFailed = Error("failed to establish a connection to the redis database") // ErrDatabaseInit is returned when the initialization of the database boundaries fails. ErrDatabaseInit = Error("failed to initialize database ±inf boundaries") // ErrDatabaseInconsistent is returned when the initialization of the database boundaries fails. ErrDatabaseInconsistent = Error("the databe is in an inconsistent state") // ErrInvalidRange is returned when a passed string is not a valid range ErrInvalidRange = Error("invalid range passed, use either of these: <IP>, <IP>/<1-32>, <IP> - <IP>") // ErrIPv6NotSupported is returned if an IPv6 range or IP input is detected. ErrIPv6NotSupported = Error("IPv6 ranges are not supported") // ErrInvalidIP is returned when the passed argument is an invalid IP ErrInvalidIP = Error("invalid IP passed") // ErrNoResult is returned when a result slic is empty or some connection error occurs during retrieval of values. ErrNoResult = Error("could not retrieve any results from the database") // ErrIPNotFound is returned if the passed IP is not contained in any ranges ErrIPNotFound = Error("the given IP was not found in any database ranges") )
Variables ¶
var ( // IPRangesKey contains the key name of the sorted set that contains the IPs (integers) IPRangesKey = "________________IP_RANGES________________" // DeleteReason is given to a specific deltion range // on a second attept (not aromic) the range is then finally deleted. DeleteReason = "_________________DELETE_________________" )
Functions ¶
This section is empty.
Types ¶
type Client ¶
type Client struct {
// contains filtered or unexported fields
}
Client is an extended version of the redis.Client
func (*Client) Find ¶
Find searches for the requested IP in the database. If the IP is found within any previously inserted range, the associated reason is returned. If it is not found, an error is returned instead.
func (*Client) Insert ¶
Insert inserts a new IP range or IP into the database with an associated reason string
func (*Client) UpdateReasonOf ¶ added in v1.1.0
func (c *Client) UpdateReasonOf(ip string, fn UpdateFunc) (err error)
UpdateReasonOf updates the reason of the range that contains the passed ip.
type Error ¶
type Error string
Error is a wrapper for constant errors that are not supposed to be changed.
type Options ¶
type Options struct {
// The network type, either tcp or unix.
// Default is tcp.
Network string
// host:port address.
Addr string
// Dialer creates new network connection and has priority over
// Network and Addr options.
Dialer func() (net.Conn, error)
// Hook that is called when new connection is established.
OnConnect func(*redis.Conn) error
// Optional password. Must match the password specified in the
// requirepass server configuration option.
Password string
// Database to be selected after connecting to the server.
DB int
// Maximum number of retries before giving up.
// Default is to not retry failed commands.
MaxRetries int
// Minimum backoff between each retry.
// Default is 8 milliseconds; -1 disables backoff.
MinRetryBackoff time.Duration
// Maximum backoff between each retry.
// Default is 512 milliseconds; -1 disables backoff.
MaxRetryBackoff time.Duration
// Dial timeout for establishing new connections.
// Default is 5 seconds.
DialTimeout time.Duration
// Timeout for socket reads. If reached, commands will fail
// with a timeout instead of blocking. Use value -1 for no timeout and 0 for default.
// Default is 3 seconds.
ReadTimeout time.Duration
// Timeout for socket writes. If reached, commands will fail
// with a timeout instead of blocking.
// Default is ReadTimeout.
WriteTimeout time.Duration
// Maximum number of socket connections.
// Default is 10 connections per every CPU as reported by runtime.NumCPU.
PoolSize int
// Minimum number of idle connections which is useful when establishing
// new connection is slow.
MinIdleConns int
// Connection age at which client retires (closes) the connection.
// Default is to not close aged connections.
MaxConnAge time.Duration
// Amount of time client waits for connection if all connections
// are busy before returning an error.
// Default is ReadTimeout + 1 second.
PoolTimeout time.Duration
// Amount of time after which client closes idle connections.
// Should be less than server's timeout.
// Default is 5 minutes. -1 disables idle timeout check.
IdleTimeout time.Duration
// Frequency of idle checks made by idle connections reaper.
// Default is 1 minute. -1 disables idle connections reaper,
// but idle connections are still discarded by the client
// if IdleTimeout is set.
IdleCheckFrequency time.Duration
// TLS Config to use. When set TLS will be negotiated.
TLSConfig *tls.Config
}
Options configures the redis database connection
type UpdateFunc ¶ added in v1.1.0
UpdateFunc updates the previous reason to a new reason.
Source Files