sentenize

Documentation

Overview

Code generated from third_party/razdel/razdel/segmenters/sokr.py — keep in sync with upstream.

Package sentenize implements sentence split primitives aligned with upstream third_party/razdel/razdel/segmenters/sentenize.py (SentSplit / SentSplitter).

Index

• Constants
• Variables
• func ByteSpans(text string, chunks []string) [][2]int
• func JoinDefault(s *SentSplit) bool
• func JoinTrivial(s *SentSplit) bool
• func PostStrip(chunks []string) []string
• func RuleCloseBracket(s *SentSplit) bool
• func RuleCloseQuote(s *SentSplit) bool
• func RuleDashRight(s *SentSplit) bool
• func RuleDelimiterRight(s *SentSplit) bool
• func RuleEmptySide(s *SentSplit) bool
• func RuleInitialsLeft(s *SentSplit) bool
• func RuleInsidePairSokr(s *SentSplit) bool
• func RuleListItem(s *SentSplit) bool
• func RuleLowerRight(s *SentSplit) bool
• func RuleNoSpacePrefix(s *SentSplit) bool
• func RuleSokrLeft(s *SentSplit) bool
• func Segment(parts []SentPart, join func(*SentSplit) bool) []string
• type SentPart
  • func SentSplitterParts(text string, window int) []SentPart
• type SentSplit
  • func (s *SentSplit) BufferFirstToken() string
  • func (s *SentSplit) BufferTokens() []string
  • func (s *SentSplit) LeftIntSokr() (word string, ok bool)
  • func (s *SentSplit) LeftPairSokr() (a, b string, ok bool)
  • func (s *SentSplit) LeftSpaceSuffix() bool
  • func (s *SentSplit) LeftToken() string
  • func (s *SentSplit) RightSpacePrefix() bool
  • func (s *SentSplit) RightToken() string
  • func (s *SentSplit) RightWord() string

Constants

const (
	// Endings is upstream ENDINGS.
	Endings = ".?!\u2026"

	// OpenQuotes is upstream OPEN_QUOTES: « “ ‘
	OpenQuotes = "\u00ab\u201c\u2018"
	// CloseQuotes is upstream CLOSE_QUOTES: » ” ’
	CloseQuotes = "\u00bb\u201d\u2019"
	// GenericQuotes is upstream GENERIC_QUOTES: ASCII ", U+201E „, ASCII apostrophe.
	GenericQuotes = "\"\u201e'"
	// Quotes is upstream QUOTES (OPEN + CLOSE + GENERIC).
	Quotes = OpenQuotes + CloseQuotes + GenericQuotes

	// OpenBrackets is upstream OPEN_BRACKETS.
	OpenBrackets = "([{"
	// CloseBrackets is upstream CLOSE_BRACKETS.
	CloseBrackets = ")]}"

	// Delimiters is upstream sentenize.DELIMITERS (splitter / delimiter_right set).
	Delimiters = Endings + ";" + GenericQuotes + CloseQuotes + CloseBrackets

	// Dashes is upstream punct.DASHES (sentenize.dash_right).
	Dashes = "\u2011\u2013\u2014\u2212-"
)

const DefaultWindow = 10

DefaultWindow is the upstream SentSplitter default (10 Unicode code points).

Variables

var TrivialJoinRules = []func(*SentSplit) bool{
	RuleEmptySide,
	RuleNoSpacePrefix,
	RuleLowerRight,
	RuleDelimiterRight,
}

TrivialJoinRules is upstream RULES[:4] order: empty_side, no_space_prefix, lower_right, delimiter_right.

Functions

func ByteSpans

func ByteSpans(text string, chunks []string) [][2]int

ByteSpans maps stripped chunks to byte offsets in text (upstream find_substrings, UTF-8 bytes).

func JoinDefault

func JoinDefault(s *SentSplit) bool

JoinDefault applies defaultJoinRules in upstream order; first matching rule wins.

func JoinTrivial

func JoinTrivial(s *SentSplit) bool

JoinTrivial returns true when upstream would JOIN for the trivial rule layer only.

func PostStrip

func PostStrip(chunks []string) []string

PostStrip applies upstream SentSegmenter.post: strings.TrimSpace on each chunk.

func RuleCloseBracket

func RuleCloseBracket(s *SentSplit) bool

RuleCloseBracket mirrors upstream close_bracket.

func RuleCloseQuote

func RuleCloseQuote(s *SentSplit) bool

RuleCloseQuote mirrors upstream close_quote (delimiter in QUOTES / CLOSE_QUOTES / GENERIC_QUOTES).

func RuleDashRight

func RuleDashRight(s *SentSplit) bool

RuleDashRight mirrors upstream dash_right.

func RuleDelimiterRight

func RuleDelimiterRight(s *SentSplit) bool

RuleDelimiterRight mirrors upstream delimiter_right.

func RuleEmptySide

func RuleEmptySide(s *SentSplit) bool

RuleEmptySide mirrors upstream empty_side.

func RuleInitialsLeft

func RuleInitialsLeft(s *SentSplit) bool

RuleInitialsLeft mirrors upstream initials_left.

func RuleInsidePairSokr

func RuleInsidePairSokr(s *SentSplit) bool

RuleInsidePairSokr mirrors upstream inside_pair_sokr.

func RuleListItem

func RuleListItem(s *SentSplit) bool

RuleListItem mirrors upstream list_item.

func RuleLowerRight

func RuleLowerRight(s *SentSplit) bool

RuleLowerRight mirrors upstream lower_right (is_lower_alpha on right_token).

func RuleNoSpacePrefix

func RuleNoSpacePrefix(s *SentSplit) bool

RuleNoSpacePrefix mirrors upstream no_space_prefix.

func RuleSokrLeft

func RuleSokrLeft(s *SentSplit) bool

RuleSokrLeft mirrors upstream sokr_left.

func Segment

func Segment(parts []SentPart, join func(*SentSplit) bool) []string

Segment merges SentSplitterParts using join(s): true means upstream JOIN (merge buffer + delimiter + right). parts must follow [text, split, text, split, ... text]. Buffer is set on each split like upstream.

Types

type SentPart

type SentPart struct {
	Text  string
	Split *SentSplit
}

SentPart is one element of the interleaved stream from SentSplitter.__call__: either a text chunk (Split == nil) or a split point (Split != nil).

func SentSplitterParts

func SentSplitterParts(text string, window int) []SentPart

SentSplitterParts returns the interleaved stream from upstream SentSplitter.__call__: text chunks and *SentSplit events. Window is in Unicode code points (Python str slice). For whitespace-only input, returns nil (upstream yields nothing).

type SentSplit

type SentSplit struct {
	Left, Delimiter, Right string
	Buffer                 *string
	// contains filtered or unexported fields
}

SentSplit mirrors upstream SentSplit(left, delimiter, right, buffer=None).

func (*SentSplit) BufferFirstToken

func (s *SentSplit) BufferFirstToken() string

BufferFirstToken mirrors SentSplit.buffer_first_token; unset buffer returns "".

func (*SentSplit) BufferTokens

func (s *SentSplit) BufferTokens() []string

BufferTokens mirrors SentSplit.buffer_tokens (upstream TOKEN.findall); unset buffer returns nil.

func (*SentSplit) LeftIntSokr

func (s *SentSplit) LeftIntSokr() (word string, ok bool)

LeftIntSokr mirrors SentSplit.left_int_sokr (upstream INT_SOKR).

func (*SentSplit) LeftPairSokr

func (s *SentSplit) LeftPairSokr() (a, b string, ok bool)

LeftPairSokr mirrors SentSplit.left_pair_sokr (upstream PAIR_SOKR).

func (*SentSplit) LeftSpaceSuffix

func (s *SentSplit) LeftSpaceSuffix() bool

LeftSpaceSuffix mirrors SentSplit.left_space_suffix (upstream SPACE_SUFFIX).

func (*SentSplit) LeftToken

func (s *SentSplit) LeftToken() string

LeftToken mirrors SentSplit.left_token (upstream LAST_TOKEN).

func (*SentSplit) RightSpacePrefix

func (s *SentSplit) RightSpacePrefix() bool

RightSpacePrefix mirrors SentSplit.right_space_prefix (upstream SPACE_PREFIX).

func (*SentSplit) RightToken

func (s *SentSplit) RightToken() string

RightToken mirrors SentSplit.right_token (upstream FIRST_TOKEN).

func (*SentSplit) RightWord

func (s *SentSplit) RightWord() string

RightWord mirrors SentSplit.right_word (upstream WORD).