gofofa

README

gofofa

fofa client in Go

Background

The official library doesn't has unittests, 之前官方的库功能不全，代码质量差，完全没有社区活跃度，不符合开源项目的基本要求。因此，想就fofa的客户端作为练手，解决上述问题。

Usage

Search

• search query, only query needed:

./fofa search port=80
./fofa search 'port=80 && protocol=ftp'

• search short, default subcommand is search:

./fofa domain=qq.com

• custom fields, default 'ip,port':

./fofa search --fields host,ip,port,protocol,lastupdatetime 'port=6379'
./fofa search -f host,ip,port,protocol,lastupdatetime 'port=6379'

• custom size, default 100:

./fofa search --size 10 'port=6379'
./fofa search -s 10 'port=6379'

if size is larger than your account free limit, you can set -deductMode to decide whether deduct fcoin automatically or not

• custom out format, default csv: can be csv/json/xml, line by line

./fofa search --format=json 'port=6379'
./fofa search --format json 'port=6379'

• write to file, default stdout:

./fofa search --outFile a.txt 'port=6379'
./fofa search -o a.txt 'port=6379'

• verbose mode

./fofa --verbose search port=80

Stats

• stats subcommand

./fofa stats --fields title,country title="hacked by"

Icon

• icon subcommand

search icon at fofa:

./fofa icon --open ./data/favicon.ico
./fofa icon --open https://fofa.info/favicon.ico
./fofa icon --open http://www.baidu.com

calc local file icon hash:

./fofa icon ./data/favicon.ico

calc remote icon hash:

./fofa icon https://fofa.info/favicon.ico

calc remote homepage icon hash:

./fofa icon http://www.baidu.com

Pipeline

• pipeline subcommand

do workflow pipeline to process data:

./fofa pipeline 'fofa(`title="hacked"`,`host,ip,port,title,server`) & cut(`host,ip,port`) & to_int(`port`) & sort(`port`)'

PipeLine

Utils

• random subcommand

random generate date from fofa, line by line

./fofa random
./fofa random -f host,ip,port,lastupdatetime,title,header,body --format json

every 500ms generate one line, never stop

./fofa random -s -1 -sleep 500

• count subcommand

./fofa count port=80

• account subcommand

./fofa account

• version

./fofa --version

Features

• ☑ Cross-platform
  • ☑ Windows
  • ☑ Linux
  • ☑ Mac
• ☑ Code coverage > 90%
• ☑ As SDK
  • ☑ Client: NewClient
    • ☑ HostSearch
    • ☑ HostSize
    • ☑ AccountInfo
    • ☑ IconHash
• ☑ As Client
  • ☑ Sub Commands
    • ☑ account
    • ☑ search
      • ☑ query
      • ☑ fields/f
      • ☑ size/s
      • ☑ format
        • ☑ csv
        • ☑ json
        • ☑ xml
        • ☐ table
        • ☐ excel
      • ☑ outFile/o
    • ☑ stats
    • ☑ icon
    • ☐ web
  • ☑ Terminal color
  • ☑ Global Config
    • ☑ fofaURL
    • ☑ deductMode
  • ☑ Envirement
    • ☑ FOFA_CLIENT_URL format: /?email=<email>&key=<key>&version=<v1>
    • ☑ FOFA_SERVER
    • ☑ FOFA_EMAIL
    • ☑ FOFA_KEY
• ☐ Publish
  • ☑ github
  • ☐ brew
  • ☐ apt
  • ☐ yum

Scenes

How to dump all domains that cert is valid and contains google?

./fofa stats -f domain -s 100 'cert.is_valid=true && (cert="google")'

How to dump link icon datasets?

./fofa random -s 10 -sleep 0 -f body 'body=icon && body=link'  | jq .body | grep -Po "(<[Ll][^>]*?rel[^>]*?icon[^>]*?>)"

pipeline mode:

./fofa pipeline 'fofa("body=icon && body=link", "body,host,ip,port") & grep_add("body", "(?is)<link[^>]*?rel[^>]*?icon[^>]*?>", "icon_tag") & cut("body")'

We can check fofa's bug like this:

./fofa.exe random -s 10 -sleep 0 -f body 'body=icon && icon_hash=="0"'

Documentation

Overview

Package gofofa fofa client in Go

env settings: - FOFA_CLIENT_URL full fofa connnection string, format: <url>/?email=<email>&key=<key>&version=<v2> - FOFA_SERVER fofa server - FOFA_EMAIL fofa account email - FOFA_KEY fofa account key

Index

• func ExtractIconFromHtml(data []byte) string
• func FofaURLFromEnv() string
• func IconHash(iconUrl string) (hash string, err error)
• type AccountInfo
  • func (ai AccountInfo) String() string
• type Client
  • func NewClient(options ...ClientOption) (*Client, error)
  • func (c *Client) AccountInfo() (ac AccountInfo, err error)
  • func (c *Client) Fetch(apiURI string, params map[string]string, v interface{}) (err error)
  • func (c *Client) HostSearch(query string, size int, fields []string) (res [][]string, err error)
  • func (c *Client) HostSize(query string) (count int, err error)
  • func (c *Client) Stats(query string, size int, fields []string) (res []StatsObject, err error)
  • func (c *Client) URL() string
  • func (c *Client) Update(configURL string) error
• type ClientOption
  • func WithLogger(logger *logrus.Logger) ClientOption
  • func WithURL(configURL string) ClientOption
• type DeductMode
  • func ParseDeductMode(v string) DeductMode
• type HostResults
• type StatsItem
• type StatsObject
• type StatsResults

Functions

func ExtractIconFromHtml

func ExtractIconFromHtml(data []byte) string

ExtractIconFromHtml extract link icon from html

func FofaURLFromEnv

func FofaURLFromEnv() string

FofaURLFromEnv parse fofa connection url from env, then generate url string

func IconHash

func IconHash(iconUrl string) (hash string, err error)

IconHash if url is a local icon file, then calc the hash if url is remote icon url, the download and calc the hash if url is web homepage, then try to parse favicon url and download it, then calc the hash

Types

type AccountInfo

type AccountInfo struct {
	Error    bool   `json:"error"`            // error or not
	ErrMsg   string `json:"errmsg,omitempty"` // error string message
	FCoin    int    `json:"fcoin"`            // fcoin count
	IsVIP    bool   `json:"isvip"`            // is vip
	VIPLevel int    `json:"vip_level"`        // vip level
}

AccountInfo fofa account info

func (AccountInfo) String

func (ai AccountInfo) String() string

type Client

type Client struct {
	Server     string // can set local server for debugging, format: <scheme>://<host>
	APIVersion string // api version
	Email      string // fofa email
	Key        string // fofa key

	Account    AccountInfo // fofa account info
	DeductMode DeductMode  // 扣费提醒默认
	// contains filtered or unexported fields
}

Client of fofa connection

func NewClient

func NewClient(options ...ClientOption) (*Client, error)

NewClient from fofa connection string to config and with env config merge

func (*Client) AccountInfo

func (c *Client) AccountInfo() (ac AccountInfo, err error)

AccountInfo fetch account info from fofa

func (*Client) Fetch

func (c *Client) Fetch(apiURI string, params map[string]string, v interface{}) (err error)

Fetch http request and parse as json return to v

func (*Client) HostSearch

func (c *Client) HostSearch(query string, size int, fields []string) (res [][]string, err error)

HostSearch search fofa host data query fofa query string size data size: -1 means all，0 means just data total info, >0 means actual size fields field of fofa host struct

func (*Client) HostSize

func (c *Client) HostSize(query string) (count int, err error)

HostSize fetch query matched host count

func (*Client) Stats

func (c *Client) Stats(query string, size int, fields []string) (res []StatsObject, err error)

Stats aggs fofa host data query fofa query string size data size fields' field of fofa host struct

func (*Client) URL

func (c *Client) URL() string

URL generate fofa connection url string

func (*Client) Update

func (c *Client) Update(configURL string) error

Update merge config from config url

type ClientOption

type ClientOption func(c *Client) error

func WithLogger

func WithLogger(logger *logrus.Logger) ClientOption

WithLogger set logger

func WithURL

func WithURL(configURL string) ClientOption

WithURL configURL format: <url>/?email=<email>&key=<key>&version=<v2>&tlsdisabled=false&debuglevel=0

type DeductMode

type DeductMode int

DeductMode should deduct fcoin automatically or just use free limit

const (
	// DeductModeFree only use free limit size
	DeductModeFree DeductMode = 0
	// DeductModeFCoin deduct fcoin automatically if account has fcoin
	DeductModeFCoin DeductMode = 1
)

func ParseDeductMode

func ParseDeductMode(v string) DeductMode

ParseDeductMode parse string to DeductMode

type HostResults

type HostResults struct {
	Mode    string      `json:"mode"`
	Error   bool        `json:"error"`
	Errmsg  string      `json:"errmsg"`
	Query   string      `json:"query"`
	Page    int         `json:"page"`
	Size    int         `json:"size"` // 总数
	Results interface{} `json:"results"`
}

HostResults /search/all api results

type StatsItem

type StatsItem struct {
	Name  string
	Count int
}

StatsItem one stats item

type StatsObject

type StatsObject struct {
	Name  string
	Items []StatsItem
}

StatsObject one stats object

type StatsResults

type StatsResults struct {
	Error          bool                   `json:"error"`
	Errmsg         string                 `json:"errmsg"`
	Distinct       map[string]interface{} `json:"distinct"`
	Aggs           map[string]interface{} `json:"aggs"`
	LastUpdateTime string                 `json:"lastupdatetime"`
}

StatsResults /search/stats api results