README
¶
go-swd
go-swd 是一个 Go 语言的敏感词检测与过滤库。它基于 Aho-Corasick 自动机，内置约四万词的中文词库，支持自定义词库、按分类过滤、白名单、文本替换，以及大小写、全半角、数字样式等字符变体的自动归一。
安装
go get github.com/kirklin/go-swd
需要 Go 1.25 或更高版本，无第三方依赖。
使用
import "github.com/kirklin/go-swd"
engine, err := swd.New()
if err != nil {
return err
}
engine.Detect(text) // 是否包含敏感词
engine.MatchAll(text) // 全部命中，含位置、标签、风险与置信度
engine.ReplaceWithAsterisk(text) // 用 * 遮盖命中
整体判定
Check 返回整段文本的判定：风险等级、处置建议、命中的一级分类，以及触发判定的主要标签。
r := engine.Check(text)
switch r.Suggestion() {
case "block": // 高风险，直接拦截
case "review": // 中风险，转人工复审
default: // 放行
}
fmt.Println(r.Risk, r.Categories, r.Label.Chinese(), r.Confidence)
风险分四态：RiskHigh 建议拦截、RiskMedium 建议人工复审、RiskLow 仅在高召回场景处理、RiskNone 无风险。
自定义词库
engine, err := swd.New(
// 按二级标签加，继承该标签的分类、风险与置信度
swd.WithLabeledWords(map[string]swd.Label{
"示例词": swd.ContrabandFraud,
"多分类词": swd.PromotionToSites,
}),
// 按一级分类加，风险默认为高
swd.WithWords(map[string]swd.Category{
"自定义词": swd.UserCategory(0),
}),
)
err = engine.AddLabeledWord("新增词", swd.PoliticalSensitive) // 返回后即对查询可见
err = engine.RemoveWord("示例词")
不加载内置词库：swd.New(swd.WithoutDefaultDict())。批量修改使用 AddWords 和 RemoveWords。
按分类
engine.DetectIn(text, swd.Pornography, swd.Political)
engine.MatchAllIn(text, swd.All)
engine.ReplaceWithAsteriskIn(text, swd.Contraband)
engine.CheckIn(text, swd.Promotion) // 只看引流广告的判定
一级分类共 10 个：
| 分类 | 说明 | 二级标签 |
|---|---|---|
Pornography |
色情低俗 | pornographic_adult sexual_suggestive sexual_terms |
Political |
涉政 | political_sensitive political_figure political_entity |
Violence |
暴恐 | violent_extremist violent_incidents violent_weapons |
Contraband |
违禁 | contraband_drug contraband_gambling contraband_act contraband_entity contraband_fraud contraband_forgery contraband_privacy |
Inappropriate |
不良内容 | inappropriate_discrimination inappropriate_profanity inappropriate_oral inappropriate_ethics inappropriate_superstition inappropriate_nonsense |
Promotion |
引流广告 | pt_to_sites pt_by_recruitment pt_to_contact |
Religion |
宗教 | religion_general |
AdCompliance |
广告法 | ad_compliance |
AIGC |
AI 生成 | aigc |
Custom |
自定义 | customized |
每个二级标签自带默认风险等级与基础置信度，Label 的方法可以查询：Category()、Risk()、Chinese()。swd.Labels() 返回全部标签。
除预定义分类外还有 21 个自定义位，用 swd.UserCategory(n) 取，n 取 0 到 20。
分类为位掩码，一个词可以同时属于多个分类，All 为全部预定义分类。
替换
engine.Replace(text, '#')
engine.ReplaceWithStrategy(text, func(m swd.Match) string {
return "[" + m.Label.Chinese() + "]"
})
重叠的命中会先合并为一个区间再替换。
字符归一化
词库和文本经过同一套折叠规则，以下写法在匹配时等价，无需配置：
| 类型 | 示例 |
|---|---|
| 大小写、全半角 | Fuck、ｆｕｃｋ |
| 数字样式 | 1、１、①、⁹、一、壹、𝟙 |
| 带圈字母、数学字母 | Ⓐ、𝐚、🅰 |
| 带变音符的拉丁字母 | é、ñ、ł |
零宽字符、组合标记和变体选择符在匹配时被忽略。
可选的宽松匹配：
swd.New(swd.WithMaxGap(1)) // 容忍词内的分隔符：f*u*c*k、法 轮 功
swd.New(swd.WithCollapseRepeats(true)) // 折叠重复字符：fuuuck
白名单
engine, err := swd.New(swd.WithAllowWords("特色情怀"))
engine.Detect("特色情怀") // false：落在允许短语内部的命中被抑制
运行期使用 AddAllowWords 和 RemoveAllowWords 调整。
命中结果
type Match struct {
Word string // 词库中的原始写法
StartPos int // rune 下标
EndPos int // rune 下标，开区间
ByteStart int // 字节偏移
ByteEnd int // 字节偏移，开区间
Label Label // 二级标签
Category Category // 一级分类
Risk Risk // 处置建议：高/中/低/无
Confidence uint8 // 0-100，该词指示真实违规的可靠程度
}
text[m.ByteStart:m.ByteEnd] 是命中的原文片段。Match 返回最早结束的命中；MatchAll 按结束位置排序，包含重叠的命中；Matches 返回迭代器，适合大量结果。
并发
Engine 可以在多个 goroutine 中共享。查询不加锁；更新词库时重建自动机，期间的查询不受影响。
词库
dict/ 目录下为内置词库：纯文本，一行一个词，# 开头为注释。文件名就是二级标签，dict/contraband_drug.txt 里的词全部带 ContrabandDrug 标签，因而自动获得对应的一级分类、风险等级与基础置信度。新增一个以标签命名的文件即可扩充词库。
词库共 15,932 词，分布在 28 个标签下。取舍依据是两份带标签的公开语料：38 万条中文新闻标题（正常文本）和 COLD 中文冒犯性语言数据集（11,754 条真实评论，人工标注冒犯与否）。
在 38 万条干净新闻标题上，按风险等级的分布：
| 风险 | 建议 | 行数 | 占比 |
|---|---|---|---|
| 高风险 | block | 490 | 0.128% |
| 中风险 | review | 719 | 0.188% |
| 低风险 | pass | 1,101 | 0.288% |
| 无风险 | pass | 380,378 | 99.396% |
在 COLD 真实评论上，按处置阈值：
| 处置阈值 | 精确率 | 召回率 |
|---|---|---|
| 仅高风险 | 79.0% | 7.3% |
| 高 + 中 | 81.5% | 11.0% |
| 任意命中 | 76.0% | 12.2% |
低风险标签（sexual_terms 性健康、religion_general 宗教、inappropriate_oral 低俗口头语、ad_compliance 广告法、aigc）本来就会在正常文本里命中，它们的作用是标注而非拦截：把性健康、宗教这类话题识别出来交给调用方判断，而不是当成违规。
召回率那一栏说明了关键词方法的边界。COLD 里的冒犯多为语境型歧视，例如「只要不来中国的外国人就是好外国人」，通篇没有敏感词。这类内容需要语义模型，任何词库都做不到；本库覆盖的是有明确词面特征的内容。
文档
赞助
如果这个项目对你有帮助，欢迎通过 GitHub Sponsors、Patreon 或 Buy Me a Coffee 支持。
许可证
Apache License 2.0，见 LICENSE。
Documentation
¶
Overview ¶
Package swd is a fast sensitive-word detection and filtering library for Chinese (and mixed) text.
Matching is done by an immutable Aho-Corasick automaton compiled from the word list (see internal/automaton). Character normalization (case, full width, digit styles, enclosed and mathematical letters, Latin diacritics) is folded into the automaton's character table, so a scan makes a single pass over the original text, allocates nothing and reports positions in the original text.
An Engine is safe for concurrent use. Queries never take a lock; updates build a new automaton and swap it in atomically, so a word added with AddWord is visible to the next query.
Basic use:
engine, err := swd.New()
if err != nil {
log.Fatal(err)
}
engine.Check("...") // risk level, categories and matches
engine.Detect("...") // any sensitive word?
engine.MatchAll("...") // every match with position, label and risk
engine.ReplaceWithAsterisk("...") // mask matches with *
engine.AddLabeledWord("自定义词", swd.ContrabandFraud)
Words are classified in two levels. Category is the first level and holds ten values; Label is the second level and decides a word's category, its default Risk and its base confidence. Risk has four states, so a caller can block, queue for review, or pass.
Index ¶
- Variables
- type Category
- type Engine
- func (e *Engine) AddAllowWords(words ...string) error
- func (e *Engine) AddLabeledWord(word string, label Label) error
- func (e *Engine) AddLabeledWords(words map[string]Label) error
- func (e *Engine) AddWord(word string, category Category) error
- func (e *Engine) AddWords(words map[string]Category) error
- func (e *Engine) Check(text string) Result
- func (e *Engine) CheckIn(text string, categories ...Category) Result
- func (e *Engine) Clear() error
- func (e *Engine) Detect(text string) bool
- func (e *Engine) DetectIn(text string, categories ...Category) bool
- func (e *Engine) Len() int
- func (e *Engine) Match(text string) *Match
- func (e *Engine) MatchAll(text string) []Match
- func (e *Engine) MatchAllIn(text string, categories ...Category) []Match
- func (e *Engine) MatchIn(text string, categories ...Category) *Match
- func (e *Engine) Matches(text string) iter.Seq[Match]
- func (e *Engine) MatchesIn(text string, categories ...Category) iter.Seq[Match]
- func (e *Engine) RemoveAllowWords(words ...string) error
- func (e *Engine) RemoveWord(word string) error
- func (e *Engine) RemoveWords(words []string) error
- func (e *Engine) Replace(text string, replacement rune) string
- func (e *Engine) ReplaceIn(text string, replacement rune, categories ...Category) string
- func (e *Engine) ReplaceWithAsterisk(text string) string
- func (e *Engine) ReplaceWithAsteriskIn(text string, categories ...Category) string
- func (e *Engine) ReplaceWithStrategy(text string, strategy func(word Match) string) string
- func (e *Engine) ReplaceWithStrategyIn(text string, strategy func(word Match) string, categories ...Category) string
- func (e *Engine) Stats() Stats
- func (e *Engine) Words() map[string]Category
- type Label
- type Match
- type Option
- type Result
- type Risk
- type SWD
- type SensitiveWord
- type Stats
Constants ¶
This section is empty.
Variables ¶
var ( ErrEmptyWord = errors.New("swd: empty word") ErrInvalidCategory = errors.New("swd: invalid category") ErrWordTooLong = errors.New("swd: word longer than 255 characters") )
Errors returned by word management methods.
Functions ¶
This section is empty.
Types ¶
type Category ¶
type Category uint32
Category is the first-level classification of a sensitive word. It is a bitmask, so a word may belong to several categories at once and several categories can be combined with | when filtering.
There are ten categories. Fraud, forged documents and personal-data trading are labels under Contraband rather than categories of their own, because all three are illegal transactions.
const ( None Category = 0 Pornography Category = 1 << 1 // 色情低俗 Political Category = 1 << 2 // 涉政 Violence Category = 1 << 3 // 暴恐 Contraband Category = 1 << 4 // 违禁，含毒品、赌博、诈骗、伪造证件、买卖个人信息 Inappropriate Category = 1 << 5 // 不良内容，含歧视、辱骂、价值观、迷信、灌水 Promotion Category = 1 << 6 // 引流广告 Religion Category = 1 << 7 // 宗教 AdCompliance Category = 1 << 8 // 广告法违规 AIGC Category = 1 << 9 // AI 生成内容 Custom Category = 1 << 10 // 自定义词库 // All is every predefined category combined. All = Pornography | Political | Violence | Contraband | Inappropriate | Promotion | Religion | AdCompliance | AIGC | Custom )
The ten first-level categories.
func UserCategory ¶ added in v0.3.0
UserCategory returns the n-th caller-defined category, counting from 0. Up to 21 of them exist; they never collide with the predefined ones and are accepted everywhere a Category is.
UserCategory panics if n is out of range.
func (Category) Contains ¶ added in v0.2.0
Contains reports whether c includes every category in other. Contains(None) is always false.
type Engine ¶ added in v0.2.0
type Engine struct {
// contains filtered or unexported fields
}
Engine detects and filters sensitive words. It is safe for concurrent use: queries never block, and every update builds a new automaton and swaps it in atomically.
func New ¶
New creates an Engine loaded with the built-in dictionary (unless WithoutDefaultDict is given).
func (*Engine) AddAllowWords ¶ added in v0.2.0
AddAllowWords adds phrases that suppress matches lying inside them.
func (*Engine) AddLabeledWord ¶ added in v0.3.0
AddLabeledWord adds a word under a second-level label, which decides its category, risk and confidence.
func (*Engine) AddLabeledWords ¶ added in v0.3.0
AddLabeledWords adds many labeled words with a single rebuild.
func (*Engine) AddWord ¶ added in v0.2.0
AddWord adds a word. Adding an existing word merges the categories. The word is visible to queries when AddWord returns.
func (*Engine) Check ¶ added in v0.3.0
Check returns the verdict for a whole text: the highest risk found, every category hit, the label that drove the decision, and all matches.
r := engine.Check(text)
switch r.Suggestion() {
case "block": // 高风险，直接拦截
case "review": // 中风险，转人工复审
default: // 放行
}
func (*Engine) DetectIn ¶ added in v0.2.0
DetectIn reports whether text contains a sensitive word in any of the given categories. Words without a category never match.
func (*Engine) Match ¶ added in v0.2.0
Match returns the first match (the one that ends first; the longest one among matches ending at the same position), or nil.
func (*Engine) MatchAll ¶ added in v0.2.0
MatchAll returns every match, overlapping matches included, ordered by end position (longest first among matches ending at the same position).
func (*Engine) MatchAllIn ¶ added in v0.2.0
MatchAllIn is MatchAll restricted to the given categories.
func (*Engine) Matches ¶ added in v0.2.0
Matches iterates over every match without allocating a slice:
for m := range engine.Matches(text) { ... }
func (*Engine) RemoveAllowWords ¶ added in v0.2.0
RemoveAllowWords removes allowed phrases.
func (*Engine) RemoveWord ¶ added in v0.2.0
RemoveWord removes a word. Removing an unknown word is not an error.
func (*Engine) RemoveWords ¶ added in v0.2.0
RemoveWords removes many words with a single rebuild.
func (*Engine) Replace ¶ added in v0.2.0
Replace replaces every character of every sensitive word with replacement. Overlapping matches are merged, so the whole sensitive region is masked.
func (*Engine) ReplaceWithAsterisk ¶ added in v0.2.0
ReplaceWithAsterisk masks sensitive words with '*'.
func (*Engine) ReplaceWithAsteriskIn ¶ added in v0.2.0
ReplaceWithAsteriskIn is ReplaceWithAsterisk restricted to the given categories.
func (*Engine) ReplaceWithStrategy ¶ added in v0.2.0
ReplaceWithStrategy replaces each sensitive region with strategy(match). Overlapping matches are merged into one region: the Match passed to strategy spans the whole region, carries the union of the categories and names the leftmost-longest word.
func (*Engine) ReplaceWithStrategyIn ¶ added in v0.2.0
func (e *Engine) ReplaceWithStrategyIn(text string, strategy func(word Match) string, categories ...Category) string
ReplaceWithStrategyIn is ReplaceWithStrategy restricted to the given categories.
type Label ¶ added in v0.3.0
type Label uint8
Label is the second-level classification of a sensitive word. Every label belongs to exactly one Category and carries a default Risk, so a match can be routed without any further configuration.
Each label corresponds to one file under dict/, so the taxonomy lives in the file layout.
const ( LabelNone Label = iota // 色情低俗 PornographicAdult // 色情内容 SexualSuggestive // 低俗性暗示 SexualTerms // 性健康与两性科普，识别但通常放行 // 涉政 PoliticalSensitive // 敏感政治内容 PoliticalFigure // 涉政人物 PoliticalEntity // 涉政组织与实体 // 暴恐 ViolentExtremist // 极端组织与恐怖主义 ViolentIncidents // 暴力伤害行为 ViolentWeapons // 武器弹药与危险品 // 违禁 ContrabandDrug // 毒品 ContrabandGambling // 赌博 ContrabandAct // 违禁行为 ContrabandEntity // 违禁物品与工具 ContrabandFraud // 诈骗话术 ContrabandForgery // 伪造证件与凭证 ContrabandPrivacy // 买卖个人信息 // 不良内容 InappropriateDiscrimination // 偏见歧视 InappropriateProfanity // 攻击辱骂 InappropriateOral // 低俗口头语 InappropriateEthics // 不良价值观 InappropriateSuperstition // 封建迷信 InappropriateNonsense // 无意义灌水 // 引流广告 PromotionToSites // 站外引流 PromotionRecruitment // 网赚兼职广告 PromotionContact // 引流联系方式 // 宗教 ReligionGeneral // 宗教内容 // 广告法 AdComplianceViolation // 广告法违规用语 // AI 生成 AIGCGenerated // AI 生成内容特征 // 自定义 Customized // 命中自定义词库 )
The labels, grouped by category.
func Labels ¶ added in v0.3.0
func Labels() []Label
Labels returns every predefined label, in declaration order.
func (Label) Category ¶ added in v0.3.0
Category returns the first-level category the label belongs to.
type Match ¶ added in v0.2.0
type Match struct {
Word string // the dictionary word, in its original spelling
Label Label // second-level label, e.g. PornographicAdult
Category Category // first-level categories of the word
Risk Risk // how this match should be handled
Confidence uint8 // 0-100, how reliably the word indicates a violation
StartPos int // rune index of the first character
EndPos int // rune index one past the last character
ByteStart int // byte offset of the first character
ByteEnd int // byte offset one past the last character
}
Match is one occurrence of a sensitive word in a text.
StartPos and EndPos are rune indices (as in []rune(text)); ByteStart and ByteEnd are byte offsets into the text. The span covers the original characters, including any separators skipped in gap mode.
type Option ¶ added in v0.2.0
type Option func(*config)
Option configures New.
func WithAllowWords ¶ added in v0.2.0
WithAllowWords adds phrases that must never be reported: a match that lies completely inside an allowed phrase is suppressed (e.g. allow "特色情怀" so that it no longer triggers "色情").
func WithCollapseRepeats ¶ added in v0.2.0
WithCollapseRepeats treats a repeated character as part of the previous one when it cannot continue any word, so "fuuuck" matches "fuck" while words with genuine doubles such as "妈妈" keep matching.
func WithLabeledWords ¶ added in v0.3.0
WithLabeledWords adds words under a second-level label. The label decides the word's category, risk level and base confidence, so results look the same as those from the built-in dictionary.
func WithMaxGap ¶ added in v0.2.0
WithMaxGap tolerates up to n consecutive separator characters (whitespace, punctuation, symbols, emoji) between the characters of a word, so that "f*u*c*k" or "法 轮 功" are still detected. The reported match spans the separators. 0 (the default) means exact matching.
Gap matching disables the 2-gram prefilter and is therefore slower on clean text; it can also produce more false positives.
func WithWords ¶ added in v0.2.0
WithWords adds words (with their categories) on top of the dictionary.
func WithoutDefaultDict ¶ added in v0.2.0
func WithoutDefaultDict() Option
WithoutDefaultDict starts with an empty word list instead of the built-in dictionary.
type Result ¶ added in v0.3.0
type Result struct {
// Risk is the highest risk among the matches, and drives the handling
// decision. Use Risk.Suggestion for the pass/review/block advice.
Risk Risk
// Categories is every first-level category the text hit.
Categories Category
// Label is the label of the highest-risk, highest-confidence match, and
// Confidence is that match's confidence. They are the "primary reason"
// the text was flagged.
Label Label
Confidence uint8
// Matches lists every hit, in the order they end in the text.
Matches []Match
}
Result is the verdict for a whole text.
func (Result) Suggestion ¶ added in v0.3.0
Suggestion returns "pass", "review" or "block".
type Risk ¶ added in v0.3.0
type Risk uint8
Risk is how a match should be handled.
const ( // RiskNone means nothing was detected. RiskNone Risk = iota // RiskLow is a weak signal: act on it only when recall matters more // than precision, otherwise treat it like RiskNone. RiskLow // RiskMedium is a probable violation: queue it for human review. RiskMedium // RiskHigh is a clear violation: block it. RiskHigh )
Risk levels, in increasing severity.
func (Risk) Suggestion ¶ added in v0.3.0
Suggestion returns the handling advice for the risk level: "pass", "review" or "block".
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Command example demonstrates the go-swd API.
|
Command example demonstrates the go-swd API. |
|
internal
|
|
|
automaton
Package automaton implements a cache-friendly Aho-Corasick automaton over folded runes.
|
Package automaton implements a cache-friendly Aho-Corasick automaton over folded runes. |
|
normalize
Package normalize folds runes into a canonical form so that the matcher can treat visually or semantically equivalent characters as identical without a separate preprocessing pass over the text.
|
Package normalize folds runes into a canonical form so that the matcher can treat visually or semantically equivalent characters as identical without a separate preprocessing pass over the text. |
|
Package version exposes the library version.
|
Package version exposes the library version. |