gib

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Aug 28, 2025 License: Apache-2.0 Imports: 9 Imported by: 1

README

gib : A String Gibberish Detector in Golang

This is a fork of <github.com/saferwall/saferwall> gib

gib is a package to detect gibberish strings in Golang. This utility is useful when analyzing textual artefacts in malwares. To cite how we're using this package:

  • Macro documents/scripts usually obfuscate their code by randomizing variables and function names, applying gib over extracted tokens is a simple heuritic/ ML feature to raise when such a situation happens.
  • Be able to tell when malwares drop random files names, or uses random PE section names. Well you get the idea :)

Installation

gib is built as a go module so you can quickly get started by running:

go get github.com/sjiekak/gib

Usage

A quick example for using sh gib with the same dataset as published by nostril


import (
  "fmt"
  "github.com/sjiekak/gib"
)


func main() {

  // A couple test cases.
  randomString := "asdqwfbeqbfuilac"
  nonRandomString := "CreateNewUser"

  // Create a new gibberish detector
  isGibberish := gib.NewScorer(nil)

  // Will return `True`.
  fmt.Println(isGibberish(randomString))

  // Will return `False`.
  fmt.Println(isGibberish(nonRandomString))

Limitations

gib is strongly based on nostril and re-uses its training data for filtering strings. Thus gib limitations are similar, it only supports American English words and will generate false positives due to the problem domain.

The idea behind both is to fine tune certain scoring values such as length and repition based on its training corpus. A custom IDF value is computed for each given string, the default n-gram value is set to 4 similar to the dataset provided.

To make an analogy to how TF-IDF is used in document classification, nonsense strings are those that use unusual n-grams and thus score highly, while real/meaningful strings are those that use more common n-grams and thus score lower. The Nostril package includes a precomputed table of n-gram weights derived by training the system on a large set of strings constructed from concatenated American English words, real text corpora, and other inputs. Parameter values were optimized using the evolutionary algorithm NSGA-II.

Notes about testing

The score function acts as a classifier returning true if the given string is gibberish false otherwise. Therefore the rate of false positives points to the number of non-gibberish strings marked as gibberish a false negative in this case is when a gibberish string is marked as non-gibberish.

The score function is evaluated using accuracy, precision and recall.

References

Documentation

Overview

Package gib implements a gibberish string evaluator.

Package gib heuristic.go implements heuristic pattern matching on strings.

Package gib metrics.go implements accuracy metrics for rating detection on several test cases.

Package gib implements gibberish strings evaluation. file ngram.go implements N-Grams data structures and utility functions.

Index

Constants

View Source
const (
	// MinLength represents minimal length of a string to process.
	MinLength = 6
	// DefaultNgramLength is the default ngram length of the prepared dataset.
	DefaultNgramLength = 4
	// Dataset is the file path to the ngram dataset collected from a corpora.
	Dataset = "./data/ngram.json"
)

Variables

View Source
var (

	// MinScore represents the absolute minimal score for any given string
	MinScore = 8.6
)

Functions

func Accuracy

func Accuracy(truePositiveCount, falsePositiveCount, trueNegativeCount,
	falseNegativeCount int) float64

Accuracy is the fraction of predictions gib made correctly.

func NewDefaultScorer

func NewDefaultScorer() (func(string) (bool, error), error)

NewDefaultScorer creates the default scoring function based on supports American English words and will generate false positives due to the problem domain

func NewScorer

func NewScorer(opts *Options) (func(string) (bool, error), error)

NewScorer creates a new scoring function

func Precision

func Precision(truePositiveCount, falsePositiveCount int) float64

Precision is a metric to answer the question: "What proportion of positive identifications was actually correct".

func Recall

func Recall(truePositiveCount, falseNegativeCount int) float64

Recall is a metric to answer the question: "What proportion of actual positives was identified correctly?"

func TFIDFScoreFunction

func TFIDFScoreFunction(ngramFreq NGramScores, n int, lenThres float64,
	lenPenalty float64, repPenalty float64) func(string) float64

TFIDFScoreFunction generates a function that computes a score given a string.

Types

type NGramScores

type NGramScores map[string]Score

NGramScores is a match between an ngram and it's score in from a text corpora.

func NewNGramDict

func NewNGramDict(keys []string, values []Score) NGramScores

NewNGramDict creates a new instance of ngram dict.

func (NGramScores) Frequency

func (ns NGramScores) Frequency(s string) float64

Frequency returns the string computed frequency in the dataset.

func (NGramScores) IDF

func (ns NGramScores) IDF(s string) float64

IDF returns the IDF score in the corpus of a given ngram.

func (NGramScores) IsNGram

func (ns NGramScores) IsNGram(s string) bool

IsNGram checks if a given string is a valid ngram in our dataset.

func (NGramScores) TotalFrequency

func (ns NGramScores) TotalFrequency(s string) float64

TotalFrequency returns the string count in the dataset.

type NGramSet

type NGramSet struct {
	Set map[string][]string
}

NGramSet defines a set of ngrams and their respective strings. Each Ngram is a key and the values are represented by a slice of strings pertaining to that ngram.

func NewNGramSet

func NewNGramSet() NGramSet

NewNGramSet creates a new instance of ngram set.

func (*NGramSet) Add

func (n *NGramSet) Add(ngram string, s string)

Add a new string to an ngram set.

type Options

type Options struct {
	Dataset string
}

Options provides different option to create a new scorer.

type Score

type Score [3]float64

Score defines a table of scores for a given NGram.

func (Score) Frequency

func (s Score) Frequency() float64

Frequency returns the frequency value in a score.

func (Score) IDF

func (s Score) IDF() float64

IDF returns the IDF value in a score.

func (Score) TotalFrequency

func (s Score) TotalFrequency() float64

TotalFrequency returns the total frequency value in a score.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL