go-sentencepiece-encoder

README

SentencePiece Encoder V1

Combine two repos here:

• go-sentencepiece-encoder
• sentencepiecego
• sentencepiecego -- sunhailin-Leo by sunhailin-Leo

---

Reason:

• 1、go-sentencepiece-encoder has different result with Python sentencepiece library on same text and model, but sentencepiecego not.
• 2、sentencepiecego just have only one api to encode the text to ids, but go-sentencepiece-encoder not.
• 3、sentencepiecego -- sunhailin-Leo change to my repo (fork from sentencepiecego), because it will be caused and error in go mod tidy.

---

Example:

package main

import (
	"fmt"
	
	"github.com/sunhailin-Leo/go-sentencepiece-encoder/sentencepiece"
)

func main() {
	text := "This is a sample text"
	spm, _ := sentencepiece.NewSentencepieceFromFile("<Your Model Here>", false)
	
	tokens := spm.Tokenize(text)
	fmt.Println(tokens)
	
	spm.Free()
}