db

package
v0.0.0-...-510dfb7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 6, 2024 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	MaxStatisticsEntry = 500
)

Variables

View Source 
var (
	Client *mongo.Client

	Domains    *mongo.Collection // The main collection to store the entries
	NotFound   *mongo.Collection // Store domains that not found by Lookup
	TopList    *mongo.Collection // Store and count successful lookups
	CTLogs     *mongo.Collection // Store informations about CT Logs
	Statistics *mongo.Collection // Store statistics history
)
View Source 
var (
	UpdaterChan chan UpdateableDomain
)

Functions

func CTLogsUpdate 

func CTLogsUpdate(name string, index int64, size int64) error

CTLogsUpdate updates the stat for the CT log with name name in the *ctlogs* collection. The name is converted to lowercase.

func Connect 

func Connect(uri string) error

Connect connects to the database using the standard Connection URI.

func Disconnect 

func Disconnect() error

Disconnect gracefully disconnect from the database.

func DomainsInsert 

func DomainsInsert(d string) (bool, error)

DomainsInsert inserts the given domain d to the *domains* database. Checks if d is valid, do a Clean() and then splits into sub|domain|tld parts.

Returns true if d is new and inserted into the database. If domain is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns ault.ErrGetPartsFailed.

NOTE: Use RecordsUpdate() after Insert()!

func DomainsInsertWithRecord 

func DomainsInsertWithRecord(d string, ignoreUpdated bool) error

DomainsInsertWithRecord inserts the given domain d to the *domains* database IF d has at least one valid record. Checks if d is valid, do a Clean() and search for records. If found at least one valid record, insert into the database. This function always updates the "updated" field, regardless of the records.

This function returns if domain d is updated recently. This function ignores common DNS errors (eg.: NXDOMAIN).

If domain is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns ault.ErrGetPartsFailed.

func DomainsLookup 

func DomainsLookup(d string, days int) ([]string, error)

DomainsLookup validate, Clean() and query the DB and returns a list subdomains only (eg.: "wwww", "mail"). days specify, that the returned subdomain must had a valid record in the previous n days. If days is -1, every subdomain returned, including domains that does not have a record. If days is 0, return every subdomain that has a record regardless of the time.

If d has a subdomain, removes it before the query.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

func DomainsLookupFull 

func DomainsLookupFull(d string, days int) ([]string, error)

DomainsLookupFull validate, Clean() and query the DB and returns a list full domains (eg.: "www.example.com", "mail.example.com"). days specify, that the returned subdomain must had a valid record in the previous n days. If days is -1, every subdomain returned, including domains that does not have a record. If days is 0, return every subdomain that has a record regardless of the time.

If d has a subdomain, removes it before the query.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

func DomainsStarts 

func DomainsStarts(d string) ([]string, error)

DomainsStarts query the DB and returns a list of Second Level Domains (eg.: "reddit", "redditmedia") that starts with d.

Domain d must be a valid Second Level Domain (eg.: "example"). This function validate with IsValidSLD() and Clean().

Returns fault.ErrInvalidDomain is d is not a valid Second Level Domain.

func DomainsTLD 

func DomainsTLD(d string) ([]string, error)

DomainsTLD query the DB and returns a list of TLDs for the given domain d (eg.: "com", "org").

Domain d must be a valid Second Level Domain (eg.: "example").

NOTE: This function not validate and Clean() d!

func DomainsUpdateUpdatedTime 

func DomainsUpdateUpdatedTime(d string) error

DomainsUpdateUpdatedTime updated the "updated" timestamp to the current time to domain d.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed.

func DomainsUpdatedRecently 

func DomainsUpdatedRecently(d string) (bool, error)

DomainsUpdatedRecently check whether domain d is updated recently (in the previous 12 hours).

Return false, nil if d is not exists in the database (ignore mongo.ErrNoDocuments).

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed.

func NotFoundInsert 

func NotFoundInsert(d string) (bool, error)

InsertNotFound inserts the given domain d to the *notFound* database. Checks if d is valid, do a Clean() and removes the subdomain from d.

Returns true if d is new and inserted into the database. If domain is invalid or failed to remove the subdomain, returns fault.ErrInvalidDomain.

func RecordsInsert 

func RecordsInsert(d string, t uint16, v string) (bool, error)

RecordsInsert insert (if not exist) or updates the "date" field for record with "type" t and "value" v. This function updates the "updated" field to the current time with DomainsUpdateUpdatedTime(). If the same record found, updates the "time" field in element. If new record found, append it to the "records" field.

Returns whether record with type t and value is a new record.

If domain d is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed.

func RecordsUpdate 

func RecordsUpdate(d string, ignoreUpdated bool) error

RecordsUpdate updates the records field for domain d if d is not update recently (in the previous hour). This function updates the "updated" field to the current time and the records in the database. If the same record found, updates the "time" field in element. If new record found, append it to the "records" field.

Checks if d is a wildcard record before update.

This function ignores the common DNS errors. If ignoreUpdated is true, ignore when was the last update based on the "updated" timestamp.

If domain d is invalid, returns fault.ErrInvalidDomain. If failed to get parts of d (eg.: d is a TLD), returns fault.ErrGetPartsFailed.

func RecordsUpdater 

func RecordsUpdater(nworker int, chanSize int)

func StatisticsCleanWorker 

func StatisticsCleanWorker()

StatisticsCleanWorker removes entries beyond MaxStatisticsEntry number.

This function is designed to run as a goroutine in the background. The errors are printed to STDERR.

func StatisticsCountTotal 

func StatisticsCountTotal() (int64, error)

StatisticsCountTotal returns the total number of entries in "domain" collection.

func StatisticsCountUpdated 

func StatisticsCountUpdated() (int64, error)

StatisticsCountUpdated returns the total number of entries that updated in "domain" collection.

func StatisticsCountValid 

func StatisticsCountValid() (int64, error)

StatisticsCountValid returns the total number of entries that has at least on valid record in the "records" field in "domain" collection.

func StatisticsInsert 

func StatisticsInsert() error

StatisticsInsert get the stats and insert a new entry in the "statistics" collection.

This function is **very** slow!

func StatisticsInsertWorker 

func StatisticsInsertWorker()

StatisticsInsertWorker insert a new Statistic entry at the beginning and at a random time in an infinite loop.

This function is designed to run as a goroutine in the background. The errors are printed to STDERR.

func TopListInsert 

func TopListInsert(d string) (bool, error)

TopListInsert inserts the given domain d to the *topList* database or increase the counter if exists. Checks if d is valid, do a Clean() and removes the subdomain from d.

Returns true if d is new and inserted into the database. If domain is invalid or failed to remove the subdomain, returns fault.ErrInvalidDomain.

Types

type CTLogSchema 

type CTLogSchema struct {
	Name  string `bson:"name" json:"name"`
	Index int64  `bson:"index" json:"index"`
	Size  int64  `bson:"size" json:"size"`
}

Schema used in "ctlogs" collection

func CTLogsGet 

func CTLogsGet(name string) (*CTLogSchema, error)

CTLogsGet returns the stat for CT log with name name from the *ctlogs* collection. The name is converted to lowercase.

func CTLogsGets 

func CTLogsGets() ([]CTLogSchema, error)

CTLogsGets returns every entry from the "ctlogs" database.

type Domain 

type Domain struct {
	Domain  string   `bson:"domain" json:"domain"`
	TLD     string   `bson:"tld" json:"tld"`
	Sub     string   `bson:"sub" json:"sub"`
	Updated int64    `bson:"updated" json:"updated"`
	Records []Record `bson:"records,omitempty" json:"records,omitempty"`
}

Domains is the schema used in the "domains" collection.

func DomainsDomains 

func DomainsDomains(d string, days int) ([]Domain, error)

DomainsDomains validate, Clean() and query the DB and returns a list of Domains. days specify, that the returned Domain must have a valid record in the previous n days. If days is -1, every Domain returned, including Domains that does not have a record. If days is 0, return every Domain that has a record regardless of the time.

If d has a subdomain, removes it before the query.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

func (*Domain) FullDomain 

func (d *Domain) FullDomain() string

Returns the domain and tld only (eg.: domain.tld)

func (*Domain) String 

func (d *Domain) String() string

Returns the full hostname (eg.: sub.domain.tld).

type FastDomain 

type FastDomain struct {
	Domain string `bson:"domain" json:"domain"`
	TLD    string `bson:"tld" json:"tld"`
	Sub    string `bson:"sub" json:"sub"`
}

FastDomain is a the schema used in Lookup() to ignore the Records field.

func (*FastDomain) String 

func (d *FastDomain) String() string

Returns the full hostname (eg.: sub.domain.tld).

type NotFoundSchema 

type NotFoundSchema struct {
	Domain string `bson:"domain" json:"domain"`
}

Schema used in *notFound* collection.

type Record 

type Record struct {
	Type  uint16 `bson:"type" json:"type"`
	Value string `bson:"value" json:"value"`
	Time  int64  `bson:"time" json:"time"`
}

Record is the schema used to store a record in Domain

func DomainsRecords 

func DomainsRecords(d string, days int) ([]Record, error)

DomainsRecords query the DB and returns a list Record. days specify, that the returned record must be updated in the previous n days. If days is 0 or -1, return every record regardless of the time.

Returns records for the exact domain d.

If d is invalid return fault.ErrInvalidDomain. If failed to get parts of d because of d is just a TLD, returns fault.ErrTLDOnly. If failed to get parts of d, returns fault.ErrGetPartsFailed. If days if < -1, returns fault.ErrInvalidDays.

type StatisticSchema 

type StatisticSchema struct {
	Date    int64         `bson:"date" json:"date"`
	Total   int64         `bson:"total" json:"total"`
	Updated int64         `bsn:"updated" json:"updated"`
	Valid   int64         `bson:"valid" json:"valid"`
	CTLogs  []CTLogSchema `bson:"ctlogs" json:"ctlogs"`
}

func StatisticsGetNewest 

func StatisticsGetNewest() (StatisticSchema, error)

StatisticsGetNewest returns the newest entry from the "statistics" collection.

func StatisticsGets 

func StatisticsGets() ([]StatisticSchema, error)

StatisticsGets returns every entry in the "statistics".

type TopListSchema 

type TopListSchema struct {
	Domain string `bson:"domain" json:"domain"`
	Count  int    `bson:"count" json:"count"`
}

Schema used in *topList* collection.

type UpdateType 

type UpdateType uint8
const (
	InsertNewDomain UpdateType = iota
	UpdateExistingDomain
)

type UpdateableDomain 

type UpdateableDomain struct {
	Domain string
	Type   UpdateType
}

UpdateableDomain used to distinguish domain coming from /api/insert and domains coming from updater functions.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.