Documentation
¶
Overview ¶
Package gib implements a gibberish string evaluator.
Package gib heuristic.go implements heuristic pattern matching on strings.
Package gib metrics.go implements accuracy metrics for rating detection on several test cases.
Package gib implements gibberish strings evaluation. file ngram.go implements N-Grams data structures and utility functions.
Index ¶
- Constants
- Variables
- func Accuracy(...) float64
- func NewDefaultScorer() (func(string) (bool, error), error)
- func NewScorer(opts *Options) (func(string) (bool, error), error)
- func Precision(truePositiveCount, falsePositiveCount int) float64
- func Recall(truePositiveCount, falseNegativeCount int) float64
- func TFIDFScoreFunction(ngramFreq NGramScores, n int, lenThres float64, lenPenalty float64, ...) func(string) float64
- type NGramScores
- type NGramSet
- type Options
- type Score
Constants ¶
const ( // MinLength represents minimal length of a string to process. MinLength = 6 // DefaultNgramLength is the default ngram length of the prepared dataset. DefaultNgramLength = 4 // Dataset is the file path to the ngram dataset collected from a corpora. Dataset = "./data/ngram.json" )
Variables ¶
var (
// MinScore represents the absolute minimal score for any given string
MinScore = 8.6
)
Functions ¶
func Accuracy ¶
func Accuracy(truePositiveCount, falsePositiveCount, trueNegativeCount, falseNegativeCount int) float64
Accuracy is the fraction of predictions gib made correctly.
func NewDefaultScorer ¶
NewDefaultScorer creates the default scoring function based on supports American English words and will generate false positives due to the problem domain
func Precision ¶
Precision is a metric to answer the question: "What proportion of positive identifications was actually correct".
func Recall ¶
Recall is a metric to answer the question: "What proportion of actual positives was identified correctly?"
func TFIDFScoreFunction ¶
func TFIDFScoreFunction(ngramFreq NGramScores, n int, lenThres float64, lenPenalty float64, repPenalty float64) func(string) float64
TFIDFScoreFunction generates a function that computes a score given a string.
Types ¶
type NGramScores ¶
NGramScores is a match between an ngram and it's score in from a text corpora.
func NewNGramDict ¶
func NewNGramDict(keys []string, values []Score) NGramScores
NewNGramDict creates a new instance of ngram dict.
func (NGramScores) Frequency ¶
func (ns NGramScores) Frequency(s string) float64
Frequency returns the string computed frequency in the dataset.
func (NGramScores) IDF ¶
func (ns NGramScores) IDF(s string) float64
IDF returns the IDF score in the corpus of a given ngram.
func (NGramScores) IsNGram ¶
func (ns NGramScores) IsNGram(s string) bool
IsNGram checks if a given string is a valid ngram in our dataset.
func (NGramScores) TotalFrequency ¶
func (ns NGramScores) TotalFrequency(s string) float64
TotalFrequency returns the string count in the dataset.
type NGramSet ¶
NGramSet defines a set of ngrams and their respective strings. Each Ngram is a key and the values are represented by a slice of strings pertaining to that ngram.
type Options ¶
type Options struct {
Dataset string
}
Options provides different option to create a new scorer.