threatdefender

README

Threat Defender

A blazing fast, highly customizable, modern-day defence tool using (in memory) SQL & REST/gRPC protocols.

Report Bug · Request Feature

https://threatdefender.domgolonka.com

Loved the project? Please consider donating to the bitcoin address below:

Note: This is still in alpha release, this might break over time.

About

ThreatDefender is a tool to scrape and defend against potential dangerous threats faced on the internet. It aims to be a highly customizable tools for companies and individuals to use to counter threats.

Features

• Emails (Rest/gRPC)
  • Disposable
  • Generic
  • Free
  • Spam
• IPs (Rest/gRPC)
  • VPN
  • Spam
  • Proxy
  • Tor
• Score (Rest/gRPC)
  • IP (0 to 100)
  • Email (0 to 100)
• Database
  • SQLite
  • PostgreSQL
• Editable Sources
  • You can edit all sources

Usage

Migrate

If using NON-MEMORY SQLITE or PostgreSQL, DO THIS BEFORE YOU RUN, You need to migrate the database:

make migrate

How to run

To run it on your local computer:

git clone https://github.com/domgolonka/threatdefender
cd ./threatdefender
make build && ./bin/threatdefender

The default config file is config.yml. If you want to run it with a different config file (or add your own).

git clone https://github.com/domgolonka/threatdefender make build (make sure to build it first)

./bin/threatdefender --config=/PATH/TO/CONFIG

example: ./bin/threatdefender --config=./config.prod.yml

Configs

All configurations are in the config files in the directory "config". You can add your own environment variables here.

External APIs

The application is improved if you sign up for external APIs. Leaked is paid, but all other services are free to sign up!

• haveibeenpwned.com - Check if email/password is leaked.
• maxmind.com - IP Geolocation
• ip2location.com - IP Geolocation

Change the file changeme.env to .env and save any External API Keys.

PWNEDKEY=
IP2LOCATION=
MAXMIND=

For full configuration examples, check out https://threatdefender.domgolonka.com

Change the databases

At this moment, Threat Defender only supports SQLite and PostgreSQL. You can change the databasename field with either postgresql or sqlite3

By Default, the SQLite driver is set to "in memory". To use a file, you need to specify that the host to a .sqlite3 extension, example: YOURNAME.sqlite3. This will create a new SQLite file in the root directory.

For Postgresql, I would advise using a quick read/write database like timescale.

PostgreSQL is not yet tested

Score

The overall Fraud Score of the email and IP's reputation and recent behavior across the threat network. Fraud Scores >= 75 are suspicious, but not necessarily fraudulent.

This tool saves those threats on multiple different databases and uses REST API & gRPC to output data.

Source

All sources are available in the ./resource directory. You can edit and the resources. They files get checked once a day by the different modules.

Regular Expressions

Regex expressions are saved in the ./resource/expressions.json file in JSON format.

Each regex looks like this:

{
"name": "ipv4",
"expression": "^((?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)).*",
"type": "ipv4"
},

Name: The UNIQUE name of the regular expression.

Expression: The regex

Type: The type of expression. For IPs, it is usually is a ipv4 or ipv6. For IP proxy, its http, https, sock4, sock5.

IPs

The ip files are stored in the ./resource directory and start with ip_ such as ip_tor for tor.

Spam

Proxy

VPN

Tor

Email

Disposable

Free

Spam

APIs

REST API & gRPC is enabled. For more API examples: https://threatdefender.domgolonka.com

gRPC

The default gRPC port is 8082 (you can change in the config)

REST API

The REST API to the example app is described below.

Rate Limiting

You can enable the rate limiter for REST API in the config.yml file.

ratelimit:
  enabled: true
  max: 20 
  expiration: 30 

Max number of recent connections during Duration seconds before sending a 429 response

Expiration is the time on how long to keep records of requests in memory per minute

Request

GET /health

curl -i -H 'Accept: application/json' http://localhost:8080/health

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:56:45 GMT
Content-Type: application/json
Content-Length: 13

{"http":true}

Request

GET /list/ip/proxy

curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/proxy

Response

HTTP/1.1 200 OK
Content-Type: application/json
Date: Fri, 12 Feb 2021 03:21:38 GMT
Transfer-Encoding: chunked

{"result":[{"ID":1,"URL":"103.228.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.693099-05:00","UpdatedAt":"2020-12-04T19:12:05.693099-05:00","DeletedAt":null},{"ID":2,"URL":"196.3.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.69557-05:00","UpdatedAt":"2020-12-04T19:12:05.69557-05:00","DeletedAt":null},{"ID":3,"URL":"165.227.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.696224-05:00","UpdatedAt":"2020-12-04T19:12:05.696224-05:00","DeletedAt":null},{"ID":4,"URL":"117.197.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.696876-05:00","UpdatedAt":"2020-12-04T19:12:05.696876-05:00","DeletedAt":null},{"ID":5,"URL":"180.183.xxx.xxx","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.697515-05:00","UpdatedAt":"2020-12-04T19:12:05.697515-05:00","DeletedAt":null},{"ID":6,"URL":"159.192.xxx.xxx:8080","Type":"ipv4","CreatedAt":"2020-12-04T19:12:05.698074-05:00","UpdatedAt":"2020-12-04T19:12:05.698074-05:00","DeletedAt":null},{"ID":7,"URL":"185.28.xxx.xxx","Type":"ipv4","

Request

GET /list/ip/spam

curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/spam

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:57:33 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 34952

168.0.xxx.0/22
202.49.xxx.0/24

Request

GET /list/ip/vpn

curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/vpn

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

yul-c14.xxx.com
lim-c04.xxx.com
bhx-c05.xxx.com

Request

GET /list/ip/tor

curl -i -H 'Accept: application/json' http://localhost:8080/list/ip/tor

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:58:18 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 23253

176.10.xxx.xxx
54.37.xxx.xxx
109.70.xxx.xxx

Request

GET /list/email/disposal

curl -i -H 'Accept: application/json' http://localhost:8080/list/email/disposal

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:58:18 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxx

xxx.cc
xxx.com
xxx.ca

Request

GET /list/email/generic

curl -i -H 'Accept: application/json' http://localhost:8080/list/email/generic

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx

xxx@
xxx@
xxx@

Request

GET /list/email/spam

curl -i -H 'Accept: application/json' http://localhost:8080/list/email/spam

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx

xxx.cc
xxx.com
xxx.ca

Request

GET /list/email/free

curl -i -H 'Accept: application/json' http://localhost:8080/list/email/free

Response

HTTP/1.1 200 OK
Date: Thu, 18 Feb 2021 04:59:38 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: xxxx

xxx.cc
xxx.com
xxx.ca

Request

GET /score/email/youremail@gmail.com

curl -i -H 'Accept: application/json' http://localhost:8080/score/email/youremail@gmail.com

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

10

Request

GET /score/ip/127.0.0.1

curl -i -H 'Accept: application/json' http://localhost:8080/score/ip/127.0.0.1

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/plain; charset=utf-8
Transfer-Encoding: chunked

0

Request

GET /validate/email/youremail@gmail.com

curl -i -H 'Accept: application/json' http://localhost:8080/validate/email/youremail@gmail.com

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked

{
"valid": true
}

Request

GET /email/youremail@gmail.com

curl -i -H 'Accept: application/json' http://localhost:8080/email/youremail@gmail.com

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked

{
"valid": true,
"disposable": false,
"recent_spam": false,
"free": false,
"leaked": false,
"generic": false,
"score": 0,
"domain": {
    "created_at": "1995-08-13T04:00:00Z",
    "expiration_date": "2021-08-12T04:00:00Z"
    }
}

Request

GET /ip/127.0.0.1

curl -i -H 'Accept: application/json' http://localhost:8080/ip/127.0.0.1

Response

HTTP/1.1 200 OK
Date: Fri, 12 Feb 2021 03:29:54 GMT
Content-Type: text/json; charset=utf-8
Transfer-Encoding: chunked

{
"success": false,
"proxy": false,
"ISP": "",
"organization": "",
"ASN": 0,
"host": "",
"country_code": "",
"city": "",
"region": "",
"is_crawler": false,
"connection_type": "",
"latitude": 0,
"longitude": 0,
"timezone": "",
"vpn": false,
"tor": false,
"recent_abuse": false,
"abuse_velocity": "",
"bot_status": false,
"mobile": false,
"score": 0,
"operating_system": "",
"browser": "",
"device_model": "",
"device_brand": ""
}

Metrics

Prometheus

Prometheus is enabled. Following metrices are available by default:

http_requests_total
http_request_duration_seconds
http_requests_in_progress_total

Work in progress

Lots of features are being worked on.

Roadmap

I would like a discussion going on the potential expansion of the tool.

I would like this tool to detect all modern threats.