badman

README

badman

Blacklisted Address and Domain name Manager is tool to manage blacklist network entities. The tool provides download, save and restore capability about blacklist of IP address and domain name.

Examples

Getting Started

package main

import (
	"bufio"
	"log"
	"os"

	"github.com/m-mizutani/badman"
	"github.com/m-mizutani/badman/source"
)

func main() {
	man := badman.New()

	if err := man.Download(source.DefaultSet); err != nil {
		log.Fatal("Fail to download:", err)
	}

	fd, err := os.Open("ipaddrs_in_traffic_logs.txt")
	if err != nil {
		log.Fatal("Fail to open a file:", err)
	}
	defer fd.Close()

	scanner := bufio.NewScanner(fd)
	for scanner.Scan() {
		entities, err := man.Lookup(scanner.Text())
		if err != nil {
			log.Fatal("Fail to lookup:", err)
		}

		if len(entities) > 0 {
			log.Printf("Matched %s in %s list (reason: %s)\n",
				entities[0].Name, entities[0].Src, entities[0].Reason)
		}
	}
}

ipaddrs_in_traffic_logs.txt includes only IP addresses line by line. In this case, BadMan downloads backlists prepared in this package and store entities (IP address or domain name) in the blacklist into own repository. After downloading blacklist, Lookup method is enabled to search given name (IP address or domain name, both are accepted) from the repository.

Default settings are following.

• Default sources: MVPS, MalwareDomains, URLhausRecent and URLhausOnline
• Default repository: inMemoryRepository

Insert a new bad entity one by one

	man := badman.New()

	if err := man.Insert(badman.BadEntity{
		Name:    "10.0.0.1",
		SavedAt: time.Now(),
		Src:     "It's me",
		Reason:  "testing",
	}); err != nil {
		log.Fatal("Fail to insert an entity:", err)
	}

	entities, err := man.Lookup("10.0.0.1")
	if err != nil {
		log.Fatal("Fail to lookup an entity:", err)
	}

	// Output:
	// 10.0.0.1
	fmt.Println(entities[0].Name)

Save and Restore

	man := badman.New()

	wfd, err := os.Create("repo.dat")
	if err != nil {
		log.Fatal("Fail to create a file:", err)
	}

	// Save current repository to a file
	if err := man.Dump(wfd); err != nil {
		log.Fatal("Fail to dump repository")
	}
	wfd.Close()

	// Restore repository from a file
	rfd, err := os.Open("repo.dat")
	if err != nil {
		log.Fatal("Fail to open a serialized data file:", err)
	}

	if err := man.Load(rfd); err != nil {
		log.Fatal("Fail to load repository")
	}

Change blacklist sources

	man := badman.New()
	set := []badman.Source{
		source.NewURLhausRecent(),
		source.NewURLhausOnline(),
	}
	if err := man.Download(set); err != nil {
		log.Fatal("Fail to download:", err)
	}

Usually you can use source.DefaultSet to download all badman supported blacklist providers (sources). However, if you want to use specific sources, you can choose your preffered sources. For example, above sample code downloads only URLhaus blacklist.

Change repository

	dynamoTableRegion, dynamoTableName := "ap-northeast-1", "your-table-name"
	man := badman.New()
	man.ReplaceRepository(badman.NewDynamoRepository(dynamoTableRegion, dynamoTableName))

Repository can be replaced by ReplaceRepository() with other repository. When replacing, blacklist data in old repository is NOT copied to new repository. Below 2 repositories are prepared in this pacakge.

• inMemoryRepository
• dynamoRepository

Also, you can use own repository that is implemented badman.Repository interface.

Use case

Basically badman should be used as library and a user need to implement own program by leveraging badman.

Stateless (Serveless model)

In this case, there are 2 Lambda function. 1st (left side) function retrieves blacklist and saves dumped blacklist data to S3 periodically. 2nd (right side) Lambda function is invoked by S3 ObjectCreated event of traffic log file. After that, the Lambda function downloads both of dumped blacklist data and log file and check if the IP addresses of traffic logs exist in blacklist. If existing, lambda notify it to an administrator via communication tool, such as Slack.

Stateful (Server model)

A major advantage of server model is stream processing for real-time capability. Above serverless model has latency because buffering is required to assemble tiny log data to one object before uploading to S3. Generally, using the above model, the delay will be on the order of minutes. Therefore, it is recommended to use the server model when lower-latency processing is required.

This program has a fluentd interface and receives traffic logs via fluentd. After that, use badman to check the traffic log for IP addresses included in the blacklist. Blacklist expects to be updated periodically, and uses DynamoDB as the repository so that it can recover even if the host running the program (in this case, EC2) crashes.

Terms of Use for Data Sources

The tool uses several online blacklist sites. They have each own Terms of Use and please note you need to understand their policy before operating in your environment. A part of their Terms of Use regarding usage policy is below.

Winhelp2002 ( MVPS )

Disclaimer: this file is free to use for personal use only. Furthermore it is NOT permitted to copy any of the contents or host on any other site without permission ormeeting the full criteria of the below license terms.

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. https://creativecommons.org/licenses/by-nc-sa/4.0/

http://winhelp2002.mvps.org/hosts.txt

DNS-BH – Malware Domain Blocklist by RiskAnalytics ( MalwareDomains )

This malware block lists provided here are for free for noncommercial use as part of the fight against malware.

Any use of this list commercially is strictly prohibited without prior approval. (It’s OK to use this list on an internal DNS server for which you are not charging).

http://www.malwaredomains.com/?page_id=1508

URLhaus ( URLhausRecent, URLhausOnline )

All datasets offered by URLhaus can be used for both, commercial and non-commercial purpose without any limitations (CC0)

https://urlhaus.abuse.ch/api/#tos

License

• MIT License
• Author: Masayoshi Mizutani < mizutani@sfc.wide.ad.jp >

Documentation

Overview

Example

package main

import (
	"bufio"
	"log"
	"os"

	"github.com/m-mizutani/badman"
	"github.com/m-mizutani/badman/source"
)

func main() {
	man := badman.New()

	if err := man.Download(source.DefaultSet); err != nil {
		log.Fatal("Fail to download:", err)
	}

	fd, err := os.Open("ipaddrs_in_traffic_logs.txt")
	if err != nil {
		log.Fatal("Fail to open a file:", err)
	}
	defer fd.Close()

	scanner := bufio.NewScanner(fd)
	for scanner.Scan() {
		entities, err := man.Lookup(scanner.Text())
		if err != nil {
			log.Fatal("Fail to lookup:", err)
		}

		if len(entities) > 0 {
			log.Printf("Matched %s in %s list (reason: %s)\n",
				entities[0].Name, entities[0].Src, entities[0].Reason)
		}
	}
}

Index

• type BadEntity
• type BadMan
  • func New() *BadMan
  • func (x *BadMan) Download(srcSet []Source) error
  • func (x *BadMan) Dump(w io.Writer) error
  • func (x *BadMan) Insert(entity BadEntity) error
  • func (x *BadMan) Load(r io.Reader) error
  • func (x *BadMan) Lookup(name string) ([]BadEntity, error)
  • func (x *BadMan) ReplaceRepository(repo Repository)
  • func (x BadMan) ReplaceSerializer(ser Serializer)
• type EntityQueue
• type GzipJSONSerializer
  • func NewGzipJSONSerializer() *GzipJSONSerializer
  • func (x *GzipJSONSerializer) Deserialize(r io.Reader) chan *EntityQueue
  • func (x *GzipJSONSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error
• type GzipMsgpackSerializer
  • func NewGzipMsgpackSerializer() *GzipMsgpackSerializer
  • func (x *GzipMsgpackSerializer) Deserialize(r io.Reader) chan *EntityQueue
  • func (x *GzipMsgpackSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error
• type JSONSerializer
  • func NewJSONSerializer() *JSONSerializer
  • func (x *JSONSerializer) Deserialize(r io.Reader) chan *EntityQueue
  • func (x *JSONSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error
• type MsgpackSerializer
  • func NewMsgpackSerializer() *MsgpackSerializer
  • func (x *MsgpackSerializer) Deserialize(r io.Reader) chan *EntityQueue
  • func (x *MsgpackSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error
• type Repository
  • func NewDynamoRepository(region, tableName string) Repository
  • func NewInMemoryRepository() Repository
• type Serializer
• type Source

Examples

• Package
• BadMan.Dump
• BadMan.Insert

Types

type BadEntity

type BadEntity struct {
	Name    string
	SavedAt time.Time
	Src     string
	Reason  string // optional
}

BadEntity is IP address or domain name that is appeared in BlackList. Name indicates both IP address and domain name.

type BadMan

type BadMan struct {
	// contains filtered or unexported fields
}

BadMan is Main interface of badman pacakge.

func New

func New() *BadMan

New is constructor of BadMan

func (*BadMan) Download

func (x *BadMan) Download(srcSet []Source) error

Download accesses blacklist data via Sources and store entities that is included in blacklist into repository.

func (*BadMan) Dump

func (x *BadMan) Dump(w io.Writer) error

Dump output serialized data into w to save current repository.

Example

package main

import (
	"fmt"
	"io/ioutil"
	"log"
	"os"
	"time"

	"github.com/m-mizutani/badman"
)

func main() {
	//SetUp
	tmp, err := ioutil.TempFile("", "*.dat")
	if err != nil {
		log.Fatal(err)
	}
	tmp.Close()

	// Example
	man := badman.New()

	if err := man.Insert(badman.BadEntity{
		Name:    "orange.example.com",
		SavedAt: time.Now(),
		Src:     "clock",
	}); err != nil {
		log.Fatal("Fail to insert an entity:", err)
	}

	wfd, err := os.Create(tmp.Name())
	if err != nil {
		log.Fatal("Fail to create a file:", err)
	}

	// Save current repository to a file
	if err := man.Dump(wfd); err != nil {
		log.Fatal("Fail to dump repository")
	}
	wfd.Close()

	// Restore repository from a file
	rfd, err := os.Open(tmp.Name())
	if err != nil {
		log.Fatal("Fail to open a serialized data file:", err)
	}

	if err := man.Load(rfd); err != nil {
		log.Fatal("Fail to load repository")
	}

	entities, _ := man.Lookup("orange.example.com")

	fmt.Println(entities[0].Name)

	// TearDown
	rfd.Close()
	os.Remove(tmp.Name())

}

Output:
orange.example.com

func (*BadMan) Insert

func (x *BadMan) Insert(entity BadEntity) error

Insert adds an entity one by one. It's expected to use adding IoC by feed or something like that.

Example

package main

import (
	"fmt"
	"log"
	"time"

	"github.com/m-mizutani/badman"
)

func main() {
	man := badman.New()

	if err := man.Insert(badman.BadEntity{
		Name:    "10.0.0.1",
		SavedAt: time.Now(),
		Src:     "It's me",
		Reason:  "testing",
	}); err != nil {
		log.Fatal("Fail to insert an entity:", err)
	}

	entities, err := man.Lookup("10.0.0.1")
	if err != nil {
		log.Fatal("Fail to lookup an entity:", err)
	}

	fmt.Println(entities[0].Name)
}

Output:
10.0.0.1

func (*BadMan) Load

func (x *BadMan) Load(r io.Reader) error

Load input data that is serialized by Dump(). Please note to use same Serializer for Dump and Load.

func (*BadMan) Lookup

func (x *BadMan) Lookup(name string) ([]BadEntity, error)

Lookup searches BadEntity (both of IP address and domain name). If not found, the function returns ([]BadEntity{}, nil). A reason to return list of BadEntity is that multiple blacklists may have same entity.

func (*BadMan) ReplaceRepository

func (x *BadMan) ReplaceRepository(repo Repository)

ReplaceRepository changes Repository to store entities. Entities in old repository are removed.

func (BadMan) ReplaceSerializer

func (x BadMan) ReplaceSerializer(ser Serializer)

ReplaceSerializer just changes Serializer with ser.

type EntityQueue

type EntityQueue struct {
	Error    error
	Entities []*BadEntity
}

EntityQueue is message queue via channel.

type GzipJSONSerializer

type GzipJSONSerializer struct{}

GzipJSONSerializer is simple line json serializer

func NewGzipJSONSerializer

func NewGzipJSONSerializer() *GzipJSONSerializer

NewGzipJSONSerializer is constructor of GzipJSONSerializer

func (*GzipJSONSerializer) Deserialize

func (x *GzipJSONSerializer) Deserialize(r io.Reader) chan *EntityQueue

Deserialize of GzipJSONSerializer reads reader and unmarshal gzipped nd-json.

func (*GzipJSONSerializer) Serialize

func (x *GzipJSONSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error

Serialize of GzipJSONSerializer marshals BadEntity to gzipped JSON and append line feed at tail.

type GzipMsgpackSerializer

type GzipMsgpackSerializer struct{}

GzipMsgpackSerializer is MessagePack serializer

func NewGzipMsgpackSerializer

func NewGzipMsgpackSerializer() *GzipMsgpackSerializer

NewGzipMsgpackSerializer is constructor of GzipMsgpackSerializer

func (*GzipMsgpackSerializer) Deserialize

func (x *GzipMsgpackSerializer) Deserialize(r io.Reader) chan *EntityQueue

Deserialize of GzipMsgpackSerializer reads reader and unmarshal gzipped nd-json.

func (*GzipMsgpackSerializer) Serialize

func (x *GzipMsgpackSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error

Serialize of GzipMsgpackSerializer encodes BadEntity to MessagePack format.

type JSONSerializer

type JSONSerializer struct{}

JSONSerializer is simple line json serializer

func NewJSONSerializer

func NewJSONSerializer() *JSONSerializer

NewJSONSerializer is constructor of JSONSerializer

func (*JSONSerializer) Deserialize

func (x *JSONSerializer) Deserialize(r io.Reader) chan *EntityQueue

Deserialize of JSONSerializer reads reader and unmarshal nd-json.

func (*JSONSerializer) Serialize

func (x *JSONSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error

Serialize of JSONSerializer marshals BadEntity to JSON and append line feed at tail.

type MsgpackSerializer

type MsgpackSerializer struct{}

MsgpackSerializer is MessagePack serializer

func NewMsgpackSerializer

func NewMsgpackSerializer() *MsgpackSerializer

NewMsgpackSerializer is constructor of MsgpackSerializer

func (*MsgpackSerializer) Deserialize

func (x *MsgpackSerializer) Deserialize(r io.Reader) chan *EntityQueue

Deserialize of MsgpackSerializer reads reader and unmarshal gzipped nd-json.

func (*MsgpackSerializer) Serialize

func (x *MsgpackSerializer) Serialize(ch chan *EntityQueue, w io.Writer) error

Serialize of MsgpackSerializer encodes BadEntity to MessagePack format.

type Repository

type Repository interface {
	Put(entities []*BadEntity) error
	Get(name string) ([]BadEntity, error)
	Del(name string) error
	Dump() chan *EntityQueue
}

Repository is interface of data store.

func NewDynamoRepository

func NewDynamoRepository(region, tableName string) Repository

NewDynamoRepository is constructor of dynamoRepository

func NewInMemoryRepository

func NewInMemoryRepository() Repository

NewInMemoryRepository is constructor of inMemoryRepository

type Serializer

type Serializer interface {
	Serialize(ch chan *EntityQueue, w io.Writer) error
	Deserialize(r io.Reader) chan *EntityQueue
}

Serializer converts array of BadEntity to byte array and the reverse.

type Source

type Source interface {
	Download() chan *EntityQueue
}

Source is interface of BlackList.