tokenizer

README

Text Tokenizer for Golang

Branch	Status	Coverage
master

import (
    "fmt"

    "github.com/euskadi31/go-tokenizer"
)

func main() {
    t := tokenizer.New()

    tokens := t.Tokenize("I believe life is an intelligent thing: that things aren't random.")

    fmt.Print(tokens) // []string{"I", "believe", "life", "is", "an", "intelligent", "thing", "that", "things", "aren't", "random"}
}

License

go-tokenizer is licensed under the MIT license.

Documentation

Index

• type Tokenizer
  • func New() Tokenizer
  • func NewWithSeparator(sep string) Tokenizer

Types

type Tokenizer

type Tokenizer interface {
	KeepSeparator()
	Tokenize(content string) []string
}

Tokenizer interface

func New

func New() Tokenizer

New Tokenizer

func NewWithSeparator

func NewWithSeparator(sep string) Tokenizer

NewWithSeparator Tokenizer