tokenizer

README

French Tokenizer

This is a basic rule-based tokenizer written in Go for French

How to install

go get github.com/justinsowhat/french-tokenizer

Usage

import (
    "fmt"

    ft "github.com/justinsowhat/french-tokenizer"
)

tokenizer := ft.Tokenizer{}

	text := " «    Je m'appelle Jean-Pierre, et j'aime faire du foot justqu'au soir. » "
	mergeProperNouns := false
	actual := tokenizer.Tokenize(text, mergeProperNouns)
	fmt.Println(actual)

When mergeProperNouns is set to true, proper nous like Jean-Pierre will not be broken into three separate tokens, like Jean - Pierre; instead, it will remain Jean-Pierre. Use it with caution, as it's written with basic heuristic.

License

The MIT license is here.

Documentation

Index

• type FrenchTokenizer
  • func (t FrenchTokenizer) Tokenize(text string, mergeProperNouns bool) []string
• type Tokenizer

Types

type FrenchTokenizer

type FrenchTokenizer struct {
}

func (FrenchTokenizer) Tokenize

func (t FrenchTokenizer) Tokenize(text string, mergeProperNouns bool) []string

type Tokenizer

type Tokenizer interface {
	Tokenize(text string, mergeProperNouns bool) []string
}