README
¶
go-swd
go-swd 是一个 Go 语言的敏感词检测与过滤库。它基于 Aho-Corasick 自动机，内置约四万词的中文词库，支持自定义词库、按分类过滤、白名单、文本替换，以及大小写、全半角、数字样式等字符变体的自动归一。
安装
go get github.com/kirklin/go-swd
需要 Go 1.25 或更高版本，无第三方依赖。
使用
import "github.com/kirklin/go-swd"
engine, err := swd.New()
if err != nil {
return err
}
engine.Detect(text) // 是否包含敏感词
engine.MatchAll(text) // 全部命中，含位置与分类
engine.ReplaceWithAsterisk(text) // 用 * 遮盖命中
自定义词库
engine, err := swd.New(
swd.WithWords(map[string]swd.Category{
"示例词": swd.Custom,
"多分类词": swd.Gambling | swd.Scam,
}),
)
err = engine.AddWord("新增词", swd.Political) // 返回后即对查询可见
err = engine.RemoveWord("示例词")
不加载内置词库：swd.New(swd.WithoutDefaultDict())。批量修改使用 AddWords 和 RemoveWords。
按分类
engine.DetectIn(text, swd.Pornography, swd.Political)
engine.MatchAllIn(text, swd.All)
engine.ReplaceWithAsteriskIn(text, swd.Profanity)
分类为位掩码，一个词可以同时属于多个分类。内置分类：Pornography、Political、Violence、Gambling、Drugs、Profanity、Discrimination、Scam、Custom；All 为全部。通用词库 dict/all.txt 中的词不带分类，只有不带 In 后缀的方法会返回它们。
替换
engine.Replace(text, '#')
engine.ReplaceWithStrategy(text, func(m swd.Match) string {
return "[" + m.Category.String() + "]"
})
重叠的命中会先合并为一个区间再替换。
字符归一化
词库和文本经过同一套折叠规则，以下写法在匹配时等价，无需配置：
| 类型 | 示例 |
|---|---|
| 大小写、全半角 | Fuck、ｆｕｃｋ |
| 数字样式 | 1、１、①、⁹、一、壹、𝟙 |
| 带圈字母、数学字母 | Ⓐ、𝐚、🅰 |
| 带变音符的拉丁字母 | é、ñ、ł |
零宽字符、组合标记和变体选择符在匹配时被忽略。
可选的宽松匹配：
swd.New(swd.WithMaxGap(1)) // 容忍词内的分隔符：f*u*c*k、法 轮 功
swd.New(swd.WithCollapseRepeats(true)) // 折叠重复字符：fuuuck
白名单
engine, err := swd.New(swd.WithAllowWords("特色情怀"))
engine.Detect("特色情怀") // false：落在允许短语内部的命中被抑制
运行期使用 AddAllowWords 和 RemoveAllowWords 调整。
命中结果
type Match struct {
Word string // 词库中的原始写法
StartPos int // rune 下标
EndPos int // rune 下标，开区间
ByteStart int // 字节偏移
ByteEnd int // 字节偏移，开区间
Category Category
}
text[m.ByteStart:m.ByteEnd] 是命中的原文片段。Match 返回最早结束的命中；MatchAll 按结束位置排序，包含重叠的命中；Matches 返回迭代器，适合大量结果。
并发
Engine 可以在多个 goroutine 中共享。查询不加锁；更新词库时重建自动机，期间的查询不受影响。
词库
dict/ 目录下为内置词库：纯文本，一行一个词，# 开头为注释。文件名对应分类，all.txt 为不带分类的通用词库。
dict/strict/ 是一份更严格的词库，可自行读取后通过 AddWords 加载。
文档
赞助
如果这个项目对你有帮助，欢迎通过 GitHub Sponsors、Patreon 或 Buy Me a Coffee 支持。
许可证
Apache License 2.0，见 LICENSE。
Documentation
¶
Overview ¶
Package swd is a fast sensitive-word detection and filtering library for Chinese (and mixed) text.
Matching is done by an immutable Aho-Corasick automaton compiled from the word list (see internal/automaton). Character normalization (case, full width, digit styles, enclosed and mathematical letters, Latin diacritics) is folded into the automaton's character table, so a scan makes a single pass over the original text, allocates nothing and reports positions in the original text.
An Engine is safe for concurrent use. Queries never take a lock; updates build a new automaton and swap it in atomically, so a word added with AddWord is visible to the next query.
Basic use:
engine, err := swd.New()
if err != nil {
log.Fatal(err)
}
engine.Detect("...") // any sensitive word?
engine.MatchAll("...") // every match with positions and category
engine.ReplaceWithAsterisk("...") // mask matches with *
engine.AddWords(map[string]swd.Category{"自定义词": swd.Custom})
Index ¶
- Variables
- type Category
- type Engine
- func (e *Engine) AddAllowWords(words ...string) error
- func (e *Engine) AddWord(word string, category Category) error
- func (e *Engine) AddWords(words map[string]Category) error
- func (e *Engine) Clear() error
- func (e *Engine) Detect(text string) bool
- func (e *Engine) DetectIn(text string, categories ...Category) bool
- func (e *Engine) Len() int
- func (e *Engine) Match(text string) *Match
- func (e *Engine) MatchAll(text string) []Match
- func (e *Engine) MatchAllIn(text string, categories ...Category) []Match
- func (e *Engine) MatchIn(text string, categories ...Category) *Match
- func (e *Engine) Matches(text string) iter.Seq[Match]
- func (e *Engine) MatchesIn(text string, categories ...Category) iter.Seq[Match]
- func (e *Engine) RemoveAllowWords(words ...string) error
- func (e *Engine) RemoveWord(word string) error
- func (e *Engine) RemoveWords(words []string) error
- func (e *Engine) Replace(text string, replacement rune) string
- func (e *Engine) ReplaceIn(text string, replacement rune, categories ...Category) string
- func (e *Engine) ReplaceWithAsterisk(text string) string
- func (e *Engine) ReplaceWithAsteriskIn(text string, categories ...Category) string
- func (e *Engine) ReplaceWithStrategy(text string, strategy func(word Match) string) string
- func (e *Engine) ReplaceWithStrategyIn(text string, strategy func(word Match) string, categories ...Category) string
- func (e *Engine) Stats() Stats
- func (e *Engine) Words() map[string]Category
- type Match
- type Option
- type SWD
- type SensitiveWord
- type Stats
Constants ¶
This section is empty.
Variables ¶
var ( ErrEmptyWord = errors.New("swd: empty word") ErrInvalidCategory = errors.New("swd: invalid category") ErrWordTooLong = errors.New("swd: word longer than 255 characters") )
Errors returned by word management methods.
Functions ¶
This section is empty.
Types ¶
type Category ¶
type Category uint32
Category is a bitmask of sensitive-word categories. A word may belong to several categories at once; combine categories with |.
const ( None Category = 0 Pornography Category = 1 << 1 // 涉黄 Political Category = 1 << 2 // 涉政 Violence Category = 1 << 3 // 暴力 Gambling Category = 1 << 4 // 赌博 Drugs Category = 1 << 5 // 毒品 Profanity Category = 1 << 6 // 脏话 Discrimination Category = 1 << 7 // 歧视 Scam Category = 1 << 8 // 诈骗 Custom Category = 1 << 9 // 自定义 // All is every predefined category combined. All = Pornography | Political | Violence | Gambling | Drugs | Profanity | Discrimination | Scam | Custom )
Predefined categories. The bit values are stable across versions.
func (Category) Contains ¶ added in v0.2.0
Contains reports whether c includes every category in other. Contains(None) is always false.
type Engine ¶ added in v0.2.0
type Engine struct {
// contains filtered or unexported fields
}
Engine detects and filters sensitive words. It is safe for concurrent use: queries never block, and every update builds a new automaton and swaps it in atomically.
func New ¶
New creates an Engine loaded with the built-in dictionary (unless WithoutDefaultDict is given).
func (*Engine) AddAllowWords ¶ added in v0.2.0
AddAllowWords adds phrases that suppress matches lying inside them.
func (*Engine) AddWord ¶ added in v0.2.0
AddWord adds a word. Adding an existing word merges the categories. The word is visible to queries when AddWord returns.
func (*Engine) DetectIn ¶ added in v0.2.0
DetectIn reports whether text contains a sensitive word in any of the given categories. Words without a category never match.
func (*Engine) Match ¶ added in v0.2.0
Match returns the first match (the one that ends first; the longest one among matches ending at the same position), or nil.
func (*Engine) MatchAll ¶ added in v0.2.0
MatchAll returns every match, overlapping matches included, ordered by end position (longest first among matches ending at the same position).
func (*Engine) MatchAllIn ¶ added in v0.2.0
MatchAllIn is MatchAll restricted to the given categories.
func (*Engine) Matches ¶ added in v0.2.0
Matches iterates over every match without allocating a slice:
for m := range engine.Matches(text) { ... }
func (*Engine) RemoveAllowWords ¶ added in v0.2.0
RemoveAllowWords removes allowed phrases.
func (*Engine) RemoveWord ¶ added in v0.2.0
RemoveWord removes a word. Removing an unknown word is not an error.
func (*Engine) RemoveWords ¶ added in v0.2.0
RemoveWords removes many words with a single rebuild.
func (*Engine) Replace ¶ added in v0.2.0
Replace replaces every character of every sensitive word with replacement. Overlapping matches are merged, so the whole sensitive region is masked.
func (*Engine) ReplaceWithAsterisk ¶ added in v0.2.0
ReplaceWithAsterisk masks sensitive words with '*'.
func (*Engine) ReplaceWithAsteriskIn ¶ added in v0.2.0
ReplaceWithAsteriskIn is ReplaceWithAsterisk restricted to the given categories.
func (*Engine) ReplaceWithStrategy ¶ added in v0.2.0
ReplaceWithStrategy replaces each sensitive region with strategy(match). Overlapping matches are merged into one region: the Match passed to strategy spans the whole region, carries the union of the categories and names the leftmost-longest word.
func (*Engine) ReplaceWithStrategyIn ¶ added in v0.2.0
func (e *Engine) ReplaceWithStrategyIn(text string, strategy func(word Match) string, categories ...Category) string
ReplaceWithStrategyIn is ReplaceWithStrategy restricted to the given categories.
type Match ¶ added in v0.2.0
type Match struct {
Word string // the dictionary word, in its original spelling
StartPos int // rune index of the first character
EndPos int // rune index one past the last character
ByteStart int // byte offset of the first character
ByteEnd int // byte offset one past the last character
Category Category // categories of the word
}
Match is one occurrence of a sensitive word in a text.
StartPos and EndPos are rune indices (as in []rune(text)); ByteStart and ByteEnd are byte offsets into the text. The span covers the original characters, including any separators skipped in gap mode.
type Option ¶ added in v0.2.0
type Option func(*config)
Option configures New.
func WithAllowWords ¶ added in v0.2.0
WithAllowWords adds phrases that must never be reported: a match that lies completely inside an allowed phrase is suppressed (e.g. allow "特色情怀" so that it no longer triggers "色情").
func WithCollapseRepeats ¶ added in v0.2.0
WithCollapseRepeats treats a repeated character as part of the previous one when it cannot continue any word, so "fuuuck" matches "fuck" while words with genuine doubles such as "妈妈" keep matching.
func WithMaxGap ¶ added in v0.2.0
WithMaxGap tolerates up to n consecutive separator characters (whitespace, punctuation, symbols, emoji) between the characters of a word, so that "f*u*c*k" or "法 轮 功" are still detected. The reported match spans the separators. 0 (the default) means exact matching.
Gap matching disables the 2-gram prefilter and is therefore slower on clean text; it can also produce more false positives.
func WithWords ¶ added in v0.2.0
WithWords adds words (with their categories) on top of the dictionary.
func WithoutDefaultDict ¶ added in v0.2.0
func WithoutDefaultDict() Option
WithoutDefaultDict starts with an empty word list instead of the built-in dictionary.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Command example demonstrates the go-swd API.
|
Command example demonstrates the go-swd API. |
|
internal
|
|
|
automaton
Package automaton implements a cache-friendly Aho-Corasick automaton over folded runes.
|
Package automaton implements a cache-friendly Aho-Corasick automaton over folded runes. |
|
normalize
Package normalize folds runes into a canonical form so that the matcher can treat visually or semantically equivalent characters as identical without a separate preprocessing pass over the text.
|
Package normalize folds runes into a canonical form so that the matcher can treat visually or semantically equivalent characters as identical without a separate preprocessing pass over the text. |
|
Package version exposes the library version.
|
Package version exposes the library version. |