README
¶
What is Ngrams?
Ngrams is a simple N-gram index capable of learning from a corpus of data and generating a random output in the same style. The index and tokenization systems are implemented as interfaces, so you can roll your own solution.
Quick Start
You can test ngrams by running the small REST webserver in cmd/rest/trigrams.go. There's also a gRPC-proto example in cmd/grpc.
$ git clone https://github.com/mochi-co/ngrams.git
$ cd ngrams
$ go test -cover ./...
$ go build cmd/rest/trigrams.go
or
$ go run cmd/rest/trigrams.go
The webserver will serve two endpoints:
POST localhost:8080/learn
Indexes a plain-text body of data. Training texts can be found in training.
$ curl --data-binary @"training/pride-prejudice.txt" localhost:8080/learn
# {"parsed_tokens":139394}
curl -d 'posting a string of text' -H "Content-Type: text/plain" -X POST localhost:8080/learn
# {"parsed_tokens":5}
GET localhost:8080/generate[?limit=n]
Generates a random output from the learned ngrams. The limit query param can be used to change the number of tokens used to create the output (default 50).
$ curl localhost:8080/generate
# {
"body": "They have solaced their wretchedness, however, and had a most conscientious
and polite young man, that might be able to keep him quiet. The arrival of the
fugitives. Not so much wickedness existed in the only one daughter will be
having a daughter married.",
"limit": 50
}
$ curl localhost:8080/generate?limit=10
# {
"body": "Of its late possessor, she added, so untidy.",
"limit": 10
}
Basic Usage
An example of usage as a library can be found in cmd/rest/trigrams.go. The trigrams example uses the tokenizers.DefaultWord tokenizer, which will parse and format ngrams based on general latin-alphabet rules.
import "github.com/mochi-co/ngrams"
// Initialize a new ngram index for 3-grams (trigrams), with default options.
index = ngrams.NewIndex(3, nil)
// Parse and index ngrams from a dataset.
tokens, err := index.Parse("to be or not to be, that is the question.")
// Generate a random sequence from the indexed ngrams.
out, err := index.Babble("to be", 50)
Custom Index Initialization
Both the data storage and tokenization mechanisms for the indexer can be replaced by satisfying the store and tokenizer interfaces, allowing the indexer to be adjusted for different purposes. The index supports bigrams, trigrams, quadgrams, etc, by way of changing the NewIndex n (3) value.
// Initialize with custom tokenizers and memory stores.
// The DefaultWordTokenizer take a bool to strip linebreaks in parsed text.
index = ngrams.NewIndex(3, ngrams.Options{
Store: stores.NewMemoryStore(),
Tokenizer: tokenizers.NewDefaultWordTokenizer(true),
})
Tokenizers
A tokenizer consists of a Tokenize method which is used to parse the input data into ngram tokens, and a Format method which pieces them back together in an expected format. The library uses the tokenizers.DefaultWord tokenizer by default, which is a simple tokenizer for parsing most latin-based languages (english, french, etc) into ngram tokens.
The Format method will perform a best effort attempt at piecing any selected ngram tokens back together in a grammatically correct manner (or whichever is appropriate for the type of tokenization being performed).
New tokenizers can be created by satisfying the tokenizers.Tokenizer interface, such as for the tokenization of CJK datasets or amino sequences.
Stores
By default, the index uses an in-memory store, stores.MemoryStore. This is a basic memory store which stores the ngrams as-is. It's great for small examples, but if you were indexing millions of tokens it would be good to think about compression or aliasing.
New stores can be created by satisfying the stores.Store interface.
Contributions
Contributions to ngrams is encouraged. Open an issue to report a bug or make a feature request. Well-considered pull requests are welcome!
License
MIT.
Training texts are sourced from Project Gutenberg.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrEmptyIndex indicates that the index has not yet learned any ngrams. ErrEmptyIndex = errors.New("index is empty") // ErrNoResult indicates that no result could be returned. This shouldn't // really happen outside of tests. ErrNoResult = errors.New("no result found") )
Functions ¶
This section is empty.
Types ¶
type Index ¶
type Index struct {
// N is the number of grams to store per key.
N int
// Store contains the ngrams which have been indexed.
Store stores.Store
// Tokenizer is the tokenizer to use to split strings into tokens.
Tokenizer tk.Tokenizer
}
Index indexes ngrams and provides meachnisms for ngram retrieval and generation.
func NewIndex ¶
NewIndex returns a pointer to an Ngrams Index. It can be initialized with a custom store and tokenizer, otherwise the default in-memory store and latin word tokenizer will be used.
func (*Index) Babble ¶
Babble generates a random sequence of up to n ngrams. The future ngrams will be selected based on their probability. The n value total will include discrete punctuation depending on the tokenizer in use.
type Options ¶
type Options struct {
// Store contains the ngrams which have been indexed.
Store stores.Store
// Tokenizer is the tokenizer to use to split strings into tokens.
Tokenizer tk.Tokenizer
}
Options contains parameters for the ngram indexer.
type Result ¶
type Result struct {
// Prefix is the last token of the key that was matched. It is added
// to a key from the variations to make the next key, eg. Prefix+" "+VKey.
Prefix string
// Next contains the future variations of the ngram and the number of times
// they were indexed (probabilty score).
Next stores.Variations
}
Result contains the result of a ngram lookup.
Source Files
Directories