razdel

README

go-razdel

English | Русский

Rule-based Russian word and sentence tokenizer — a Go port of natasha/razdel.

go-razdel splits Russian text into tokens and sentences with the same rule set as Python razdel: abbreviations (т.д., initials), decimals (0.5, 50/64), quotes, lists, and dashes.

No models, no CGO, no extra dictionaries to download. Requires Go 1.24+.

Rules are tuned for news and fiction (the same domains as upstream). On social media, scientific papers, or legal text the result may be worse — that is expected.

Install

go get github.com/muonsoft/go-razdel

Usage

Same examples as natasha/razdel, so a Python user can compare the two by eye.

Tokens

package main

import (
	"fmt"
	"strings"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "Кружка-термос на 0.5л (50/64 см³, 516;...)"
	var parts []string
	for _, tok := range razdel.Tokenize(text) {
		parts = append(parts, tok.Text)
	}
	fmt.Println(strings.Join(parts, " | "))
}

Кружка-термос | на | 0.5 | л | ( | 50/64 | см³ | , | 516 | ; | ... | )

Sentences

text := `
- "Так в чем же дело?" - "Не ра-ду-ют".
И т. д. и т. п. В общем, вся газета
`

for _, sent := range razdel.Sentenize(text) {
	fmt.Println(sent.Text)
}

- "Так в чем же дело?"
- "Не ра-ду-ют".
И т. д. и т. п.
В общем, вся газета

Runnable copies of these snippets live in example_test.go and on pkg.go.dev.

API

tokens := razdel.Tokenize(text)   // []Token
sents  := razdel.Sentenize(text)  // []Sentence

Each item has Text plus a half-open span [Start, End) into the original string (Span is embedded, so tok.Start works):

type Span struct {
	Start int // UTF-8 byte offset, inclusive
	End   int // UTF-8 byte offset, exclusive
}

type Token struct {
	Span
	Text string // always equal to text[Start:End]
}

type Sentence struct {
	Span
	Text string
}

Empty input (and whitespace-only input for Sentenize) returns a nil slice. Neither function returns an error or panics on ordinary text, including invalid UTF-8.

Start/End are UTF-8 bytes, not Unicode code points. That is the unit Go uses for len and s[i:j], so you can slice the original string directly:

text := "a ж" // 4 bytes, 3 runes: 'a' is 1 byte, 'ж' is 2
toks := razdel.Tokenize(text)
// "a" [0:1]
// "ж" [2:4]
fmt.Println(text[toks[1].Start:toks[1].End] == "ж") // true

Python razdel counts code points, so numeric offsets often differ on Cyrillic even when the token texts match. Compare implementations by Text, not by raw indexes.

Sentence spans point at the trimmed slice (leading/trailing whitespace is dropped, same as Python chunk.strip()).

Compatibility with Python razdel

Token and sentence texts follow pinned natasha/razdel (third_party/razdel, commit 668dbe191a5cfd94bebf9155e2ffa5f94ff3fe33), checked in CI against upstream unit cases, a quick corpus, and a live Python differential.

Two intentional tokenize differences (also discussed upstream as razdel#17 and razdel#2):

Input	Python razdel	go-razdel
:-) ;-) =-)	:, -, )	one token :-)
✅Сдается	one token	✅, Сдается
счетчики💰	one token	счетчики, 💰

The public API (Tokenize, Sentenize, Token, Sentence, Span, byte offsets) is frozen in meaning for 0.x: breaking changes bump minor (0.2.0) and are listed in CHANGELOG.md. Go modules do not promise compatibility until v1.0.0.

Documentation

Document	What it covers
README.ru.md	Same guide in Russian
pkg.go.dev/github.com/muonsoft/go-razdel	Generated API reference and examples
docs/contracts.md	Offsets, empty input, invalid UTF-8, known deviations
CHANGELOG.md	User-visible changes
CONTRIBUTING.md	Tests, parity checks, releases
docs/README.md	Full documentation index

Contributing

Issues and pull requests are welcome. See CONTRIBUTING.md for tests, parity rules, and the release button.

License

MIT. Segmentation rules and abbreviation lists are derived from natasha/razdel (MIT, Copyright 2017). Attribution: NOTICE; upstream tree: third_party/razdel.

Documentation

Overview

Package razdel provides Russian text tokenization and sentence segmentation.

Public API and offset semantics are defined in docs/contracts.md (UTF-8 byte offsets, Variant A).

Index

• type Sentence
  • func Sentenize(text string) []Sentence
• type Span
• type Token
  • func Tokenize(text string) []Token

Examples

• Sentenize
• Sentenize (Razdel)
• Sentenize (TrimmedSpans)
• Tokenize
• Tokenize (Product)
• Tokenize (Utf8ByteOffsets)

Types

type Sentence

type Sentence struct {
	Span
	Text string
}

Sentence is a sentence span with its source text slice.

func Sentenize

func Sentenize(text string) []Sentence

Sentenize splits text into sentences using upstream sentenize.py SentSegmenter rules (including list_item and dash_right). Each sentence text is strings.TrimSpace on the raw segment chunk, matching Python chunk.strip(); byte spans refer to the trimmed slice in the original UTF-8 string (docs/contracts.md). Invalid UTF-8 does not panic; Python parity is not guaranteed for such input.

Example

package main

import (
	"fmt"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "Привет, мир! Это тест."
	sentences := razdel.Sentenize(text)

	for _, sent := range sentences {
		fmt.Printf("%q [%d:%d]\n", sent.Text, sent.Start, sent.End)
	}

}

Output:
"Привет, мир!" [0:21]
"Это тест." [22:38]

Example (Razdel)

package main

import (
	"fmt"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := `
- "Так в чем же дело?" - "Не ра-ду-ют".
И т. д. и т. п. В общем, вся газета
`
	for _, sent := range razdel.Sentenize(text) {
		fmt.Println(sent.Text)
	}
}

Output:
- "Так в чем же дело?"
- "Не ра-ду-ют".
И т. д. и т. п.
В общем, вся газета

Example (TrimmedSpans)

package main

import (
	"fmt"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "  Привет. Пока.  "

	for _, sent := range razdel.Sentenize(text) {
		fmt.Printf("%q [%d:%d]\n", sent.Text, sent.Start, sent.End)
	}

}

Output:
"Привет." [2:15]
"Пока." [16:25]

type Span

type Span struct {
	Start int
	End   int
}

Span is a half-open byte interval into the original UTF-8 string: [Start, End). Start and End are measured in bytes; see docs/contracts.md.

type Token

type Token struct {
	Span
	Text string
}

Token is a token span with its source text slice.

func Tokenize

func Tokenize(text string) []Token

Tokenize splits text into tokens with behavior aligned to upstream third_party/razdel/razdel/segmenters/tokenize.py (atoms, splits, join rules). Invalid UTF-8 does not panic: each invalid byte is an OTHER atom, then the usual join rules apply (see docs/contracts.md).

Example

package main

import (
	"fmt"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "Привет, мир!"
	tokens := razdel.Tokenize(text)

	for _, tok := range tokens {
		fmt.Printf("%q [%d:%d]\n", tok.Text, tok.Start, tok.End)
	}

}

Output:
"Привет" [0:12]
"," [12:13]
"мир" [14:20]
"!" [20:21]

Example (Product)

package main

import (
	"fmt"
	"strings"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "Кружка-термос на 0.5л (50/64 см³, 516;...)"
	var parts []string
	for _, tok := range razdel.Tokenize(text) {
		parts = append(parts, tok.Text)
	}
	fmt.Println(strings.Join(parts, " | "))
}

Output:
Кружка-термос | на | 0.5 | л | ( | 50/64 | см³ | , | 516 | ; | ... | )

Example (Utf8ByteOffsets)

package main

import (
	"fmt"
	"unicode/utf8"

	"github.com/muonsoft/go-razdel"
)

func main() {
	text := "a ж"
	fmt.Printf("bytes=%d runes=%d\n", len(text), utf8.RuneCountInString(text))

	for _, tok := range razdel.Tokenize(text) {
		fmt.Printf("%q [%d:%d]\n", tok.Text, tok.Start, tok.End)
	}

}

Output:
bytes=4 runes=3
"a" [0:1]
"ж" [2:4]