README
¶
简介
- gathertool是golang脚本化开发集成库，目的是提高对应场景脚本程序开发的效率；
- gathertool也是一款轻量级爬虫库，特色是分离了请求事件，通俗点理解就是对请求过程状态进行事件处理。
- gathertool也是接口测试&压力测试库，在接口测试脚本开发上有明显的效率优势，
- gathertool还集成了对三方中间件的操作，DB操作等。
使用
- go get github.com/mangenotwork/gathertool
介绍
gathertool是一个高度封装工具库，包含了http/s的请求，Mysql数据库方法，数据类型处理方法，数据提取方法，websocket相关方法， TCP|UDP相关方法，NoSql相关方法，开发常用方法等; 可以用于爬虫程序，接口&压力测试程序，常见网络协议调试程序，数据提取与存储程序等； gathertool的请求特点: 会在请求阶段执行各个事件如请求失败后的重试事件,请求前后的事件，请求成功事件等等, 可以根据请求状态码自定义这些事件； gathertool还拥有很好的可扩展性， 适配传入任意自定义http请求对象， 能适配各种代理对象等等； gathertool还拥有抓取数据存储功能, 比如存储到mysql, redis, mongo, pgsql等等; 还有很多创新的地方文档会根据具体方法进行介绍； gathertool还封装了消息队列接口，支持Nsq,Kafka,rabbitmq,redis等消息队列中间件
文档：
开始使用
go get github.com/mangenotwork/gathertool
简单的get请求
import gt "github.com/mangenotwork/gathertool"
func main(){
ctx, err := gt.Get("https://www.baidu.com")
if err != nil {
log.Println(err)
}
log.Println(ctx.RespBodyString)
}
含请求事件请求
import gt "github.com/mangenotwork/gathertool"
func main(){
gt.NewGet(`http://192.168.0.1`).SetStartFunc(func(ctx *gt.Context){
log.Println("请求前： 添加代理等等操作")
ctx.Client.Transport = &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
}
).SetSucceedFunc(func(ctx *gt.Context){
log.Println("请求成功： 处理数据或存储等等")
log.Println(ctx.RespBodyString())
}
).SetFailedFunc(func(ctx *gt.Context){
log.Println("请求失败： 记录失败等等")
log.Println(ctx.Err)
}
).SetRetryFunc(func(ctx *gt.Context){
log.Println("请求重试： 更换代理或添加等待时间等等")
ctx.Client.Transport = &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
}
).SetEndFunc(func(ctx *gt.Context){
log.Println("请求结束： 记录结束，处理数据等等")
log.Println(ctx)
}
).Do()
}
事件方法复用
func main(){
gt.NewGet(`http://192.168.0.1`).SetSucceedFunc(succeed).SetFailedFunc(failed).SetRetryFunc(retry).Do()
gt.NewGet(`http://www.baidu.com`).SetSucceedFunc(baiduSucceed).SetFailedFunc(failed).SetRetryFunc(retry).Do()
}
// 请求成功： 处理数据或存储等等
func succeed(ctx *gt.Context){
log.Println(ctx.RespBodyString())
}
// 请求失败： 记录失败等等
func failed(ctx *gt.Context){
log.Println(ctx.Err)
}
// 请求重试： 更换代理或添加等待时间等等
func retry(ctx *gt.Context){
ctx.Client.Transport = &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
}
// 百度请求成功后
func baiduSucceed(ctx *gt.Context){
log.Println(ctx.RespBodyString())
}
post请求
// FormData
postData := gt.FormData{
"aa":"aa",
}
// header
header := gt.NewHeader(map[string]string{
"Accept":"*/*",
"X-MicrosoftAjax":"Delta=true",
"Accept-Encoding":"gzip, deflate",
"XMLHttpRequest":"XMLHttpRequest",
"Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
})
// cookie
cookie := gt.Cookie{
"aa":"a"
}
// 随机休眠 2~6秒
sleep := gt.SetSleep(2,6)
c := gt.NewPostForm(caseUrl, postData, header, cookie, sleep)
c.Do()
html := c.RespBodyString()
log.Print(html)
数据存储到mysql
var (
host = "192.168.0.100"
port = 3306
user = "root"
password = "root123"
dbName = "dbName"
db,_ = gt.NewMysql(host, port, user, password, dbName)
)
//.... 执行抓取
data1 := "data1"
data2 := "data2"
inputdata := map[string]interface{} {
"data1" : data1,
"data2" : data2,
}
tableName := "data"
db.Spider2022DB.InsertAt(tableName, inputdata)
HTML数据提取
func main(){
date := "2022-07-05"
caseUrl := "***"
ctx, _ := gt.Get(fmt.Sprintf(caseUrl, date))
datas, _ := gt.GetPointHTML(ctx.Html, "div", "id", "domestic")
Data(datas, date, "内期表", "备注：内期表=国内期货主力合约表")
datas, _ = gt.GetPointHTML(ctx.Html, "div", "id", "overseas")
Data(datas, date, "外期表", "备注：外期表=国外期货主力合约表")
}
func Data(datas []string, date, typeName, note string) {
for _, data := range datas {
table, _ := gt.GetPointHTML(data, "table", "id", "fdata")
if len(table) > 0 {
trList := gt.RegHtmlTr(table[0])
jys := ""
for _, tr := range trList {
td := gt.RegHtmlTd(tr)
log.Println("td = ", td, len(td))
if len(td) == 1 {
jys = gt.RegHtmlTdTxt(td[0])[0]
continue
}
name := gt.RegHtmlTdTxt(td[0])[0]
if strings.Index(name, "商品名称") != -1 {
continue
}
zlhy := gt.RegHtmlTdTxt(td[1])[0]
jsj := gt.RegHtmlTdTxt(td[2])[0]
zd := gt.RegDelHtml(gt.RegHtmlTdTxt(td[3])[0])
cjj := gt.RegHtmlTdTxt(td[4])[0]
ccl := gt.RegHtmlTdTxt(td[5])[0]
dw := gt.RegHtmlTdTxt(td[6])[0]
log.Println("日期 = ", date)
log.Println("机构 = ", jys)
log.Println("商品名称 = ", name)
log.Println("主力合约 = ", zlhy)
log.Println("结算价 = ", jsj)
log.Println("涨跌 = ", zd)
log.Println("成交量 = ", cjj)
log.Println("持仓量 = ", ccl)
log.Println("单位 = ", dw)
}
}
}
}
Json数据提取
func main(){
txt := `{
"reason":"查询成功!",
"result":{
"city":"苏州",
"realtime":{
"temperature":"17",
"humidity":"69",
"info":"阴",
"wid":"02",
"direct":"东风",
"power":"2级",
"aqi":"30"
},
"future":[
{
"date":"2021-10-25",
"temperature":"12\/21℃",
"weather":"多云",
"wid":{
"day":"01",
"night":"01"
},
"direct":"东风"
},
{
"date":"2021-10-26",
"temperature":"13\/21℃",
"weather":"多云",
"wid":{
"day":"01",
"night":"01"
},
"direct":"东风转东北风"
},
{
"date":"2021-10-27",
"temperature":"13\/22℃",
"weather":"多云",
"wid":{
"day":"01",
"night":"01"
},
"direct":"东北风"
}
]
},
"error_code":0
}`
jx1 := "/result/future/[0]/date"
jx2 := "/result/future/[0]"
jx3 := "/result/future"
log.Println(gt.JsonFind(txt, jx1))
log.Println(gt.JsonFind2Json(txt, jx2))
log.Println(gt.JsonFind2Json(txt, jx3))
log.Println(gt.JsonFind2Map(txt, jx2))
log.Println(gt.JsonFind2Arr(txt, jx3))
}
......
实例
- Get请求
- 阳光高考招生章程抓取
- ip地址信息抓取
- 压力测试
- 文件下载
- 无登录微博抓取
- 百度题库抓取
- 搭建http/s代理与抓包
- 搭建socket5代理
- 商品报价信息抓取
- 期货信息抓取
- 行业信息网行业分类
JetBrains 开源证书支持
gathertool 项目一直以来都是在 JetBrains 公司旗下的 GoLand 集成开发环境中进行开发，基于 free JetBrains Open Source license(s) 正版免费授权，在此表达我的谢意。
使用注意&不足之处
- Mysql相关方法封装使用的字符串拼接，如抓取无需关心Sql注入这类场景放心使用，生产环境使用会出现安全风险；
- 变量类型转换使用反射，性能方面会差一些，追求性能请使用其他；
TODO
- Redis连接方法改为连接池
- 关闭重试
三方引用 感谢这些开源项目
- github.com/Shopify/sarama
- github.com/dgrijalva/jwt-go
- github.com/garyburd/redigo
- github.com/nsqio/go-nsq
- github.com/streadway/amqp
- github.com/xuri/excelize/v2
- go.mongodb.org/mongo-driver
- gopkg.in/yaml.v3
里程碑
v0.2.1 ~ v0.2.9
2021.10 ~ 2022.3
库的基本完成与完善，用于正式工作环境
v0.3.4
新增代理与抓包
启动一个代理服务，并抓包对数据包进行处理
v0.3.5
新增socket5代理
v0.3.6
新增:
1. func SimpleServer(addr string, g *grpc.Server) : grpc服务端
2. func NewTask() *Task : task的实例方法
3. func (task *Task) SetUrl(urlStr string) *Task & func (task *Task) SetJsonParam(jsonStr string) *Task
修复:
1. Context对象，Task为空的问题
v0.3.7
新增:
1. 新增html解析，指定html内容提取
2. 新增抓取实例
3. 优化部分方法
v0.3.9
新增:
1. 新增配置，支持yaml
2. 优化部分方法
v0.4.1
新增:
1. 文件相关处理
2. 文件压缩解压
3. 新增抓取实列 _examples/cnlinfo
v0.4.2
新增:
1. redis, nsq, rabbitmq, kafka 消息队列方法
2. 新增开发文档
3. 新增redis相关方法
v0.4.3
1. 新增网站链接采集应用场景的方法
2. 修复v0.4.2的json导入bug
3. 修复redis封装方法的bug
4. 请求日志添加请求地址信息
5. 优化抓取实例
v0.4.4 ~ v0.4.6
1. 移除 grpc相关方法
2. 新增DNS查询
3. 新增证书信息获取
4. 新增Url扫描
5. 新增邮件发送
6. 优化ICMP Ping的方法
v0.4.7
1. Redis连接方法改为连接池
2. 增加关闭重试方法
3. 增加Whois查询
4. 测试与优化
v0.4.8
TODO...
Documentation
¶
Index ¶
- Constants
- Variables
- func AbPathByCaller() string
- func AccountRational(str string) bool
- func Any2Arr(data interface{}) []interface{}
- func Any2Float64(data interface{}) float64
- func Any2Int(data interface{}) int
- func Any2Int64(data interface{}) int64
- func Any2Json(data interface{}) (string, error)
- func Any2Map(data interface{}) map[string]interface{}
- func Any2String(data interface{}) string
- func Any2Strings(data interface{}) []string
- func ApplicationTerminalOutClose()
- func BIG5To(dstCharset string, src string) (dst string, err error)
- func Base64Decode(str string) (string, error)
- func Base64Encode(str string) string
- func Base64UrlDecode(str string) (string, error)
- func Base64UrlEncode(str string) string
- func BeginDayUnix() int64
- func Bit2Byte(b []uint8) []byte
- func Bool2Byte(b bool) []byte
- func Byte2Bit(b []byte) []uint8
- func Byte2Bool(b []byte) bool
- func Byte2Float32(b []byte) float32
- func Byte2Float64(b []byte) float64
- func Byte2Int(b []byte) int
- func Byte2Int64(b []byte) int64
- func Byte2Str(b []byte) string
- func ByteToBinaryString(data byte) (str string)
- func ByteToGBK(strBuf []byte) []byte
- func ByteToUTF8(strBuf []byte) []byte
- func CPUMax()
- func Chdir(dir string) error
- func CleaningStr(str string) string
- func CloseLog()
- func ClosePingTerminalPrint()
- func CompressDirZip(src, outFile string) error
- func CompressFiles(files []string, dest string) error
- func ConvertByte2String(byte []byte, charset Charset) string
- func ConvertGBK2Str(gbkStr string) string
- func ConvertStr2GBK(str string) string
- func CopySlice(s []interface{}) []interface{}
- func CopySliceInt(s []int) []int
- func CopySliceInt64(s []int64) []int64
- func CopySliceStr(s []string) []string
- func DayAgo(i int) int64
- func DayDiff(beginDay string, endDay string) int
- func DeCompressTAR(tarFile, dest string) error
- func DeCompressZIP(zipFile, dest string) error
- func Debug(args ...interface{})
- func DebugF(format string, args ...interface{})
- func DebugFTimes(format string, times int, args ...interface{})
- func DebugTimes(times int, args ...interface{})
- func DecodeByte(b []byte) (interface{}, error)
- func DecompressionZipFile(src, dest string) error
- func DeepCopy(dst, src interface{}) error
- func EncodeByte(v interface{}) []byte
- func EndDayUnix() int64
- func Error(args ...interface{})
- func ErrorTimes(times int, args ...interface{})
- func Errorf(format string, args ...interface{})
- func ErrorfTimes(format string, times int, args ...interface{})
- func Exists(path string) bool
- func FileMd5(path string) (string, error)
- func FileMd5sum(fileName string) string
- func FileSizeFormat(fileSize int64) (size string)
- func Float322Byte(f float32) []byte
- func Float322Uint32(f float32) uint32
- func Float642Byte(f float64) []byte
- func Float642Uint64(f float64) uint64
- func GB18030To(dstCharset string, src string) (dst string, err error)
- func GB2312To(dstCharset string, src string) (dst string, err error)
- func GDKTo(dstCharset string, src string) (dst string, err error)
- func Get16MD5Encode(data string) string
- func GetAgent(agentType UserAgentType) string
- func GetAllFile(pathname string) ([]string, error)
- func GetChineseMonthDay(date string) (rMonth, rDay int64)
- func GetMD5Encode(data string) string
- func GetNowPath() string
- func GetPointClassHTML(htmlStr, label, val string) ([]string, error)
- func GetPointHTML(htmlStr, label, attr, val string) ([]string, error)
- func GetPointIDHTML(htmlStr, label, val string) ([]string, error)
- func GetWD() string
- func GzipCompress(src []byte) []byte
- func GzipDecompress(src []byte) []byte
- func HTTPTimes(times int, args ...interface{})
- func HZGB2312To(dstCharset string, src string) (dst string, err error)
- func Hex2Int(s string) int
- func Hex2Int64(s string) int64
- func HmacMD5(str, key string) string
- func HmacSHA1(str, key string) string
- func HmacSHA256(str, key string) string
- func HmacSHA512(str, key string) string
- func HourAgo(i int) int64
- func HumanFriendlyTraffic(bytes uint64) string
- func ID() int64
- func ID64() (int64, error)
- func IDMd5() string
- func IDStr() string
- func IF(condition bool, a, b interface{}) interface{}
- func IP2Binary(ip string) string
- func IP2Int64(ip string) int64
- func Info(args ...interface{})
- func InfoFTimes(times int, format string, args ...interface{})
- func InfoTimes(times int, args ...interface{})
- func Infof(format string, args ...interface{})
- func Int2Byte(i int) []byte
- func Int2Hex(i int64) string
- func Int642Byte(i int64) []byte
- func Int642Hex(i int64) string
- func Int642Str(i int64) string
- func IsAllCapital(str string) bool
- func IsAllLower(str string) bool
- func IsBase64(str string) bool
- func IsChinese(str string) bool
- func IsChineseAll(str string) bool
- func IsChineseMoney(str string) bool
- func IsChineseN(str string, number int) bool
- func IsChineseNumber(str string) bool
- func IsContainStr(items []string, item string) bool
- func IsDNSName(str string) bool
- func IsDir(path string) bool
- func IsDomain(str string) bool
- func IsElementStr(listData []string, element string) bool
- func IsEngAll(str string) bool
- func IsEngLen(str string, l int) bool
- func IsEngNumber(str string) bool
- func IsFile(path string) bool
- func IsFloat(str string) bool
- func IsFloat2Len(str string, l int) bool
- func IsFullWidth(str string) bool
- func IsHalfWidth(str string) bool
- func IsHaveCapital(str string) bool
- func IsHaveKey(data map[string]interface{}, key string) bool
- func IsHaveLower(str string) bool
- func IsIP(str string) bool
- func IsIPv4(str string) bool
- func IsInSlice(s []interface{}, v interface{}) bool
- func IsJson(str string) bool
- func IsLandline(str string) bool
- func IsLatitude(str string) bool
- func IsLeap(year int) bool
- func IsLeastCapital(str string, n int) bool
- func IsLeastLower(str string, n int) bool
- func IsLeastNumber(str string, n int) bool
- func IsLeastSpecial(str string, n int) bool
- func IsLongitude(str string) bool
- func IsNumber(str string) bool
- func IsNumber2Heard(str string, n int) bool
- func IsNumber2Len(str string, l int) bool
- func IsPhone(str string) bool
- func IsRGB(str string) bool
- func IsToday(timestamp int64) string
- func IsTodayList(timestamp int64) string
- func IsURL(str string) bool
- func IsUUID3(str string) bool
- func IsUUID4(str string) bool
- func IsUUID5(str string) bool
- func IsUnixPath(str string) bool
- func IsUrl(url string) bool
- func IsUtf8(buf []byte) bool
- func IsWindowsPath(str string) bool
- func IsXMLFile(str string) bool
- func Json2Map(str string) (map[string]interface{}, error)
- func JsonFind(jsonStr, find string) (interface{}, error)
- func JsonFind2Arr(jsonStr, find string) ([]interface{}, error)
- func JsonFind2Int(jsonStr, find string) (int, error)
- func JsonFind2Int64(jsonStr, find string) (int64, error)
- func JsonFind2Json(jsonStr, find string) (string, error)
- func JsonFind2Map(jsonStr, find string) (map[string]interface{}, error)
- func JsonFind2Str(jsonStr, find string) (string, error)
- func JwtDecrypt(tokenString, secret string) (data map[string]interface{}, err error)
- func JwtEncrypt(data map[string]interface{}, secret, method string) (string, error)
- func JwtEncrypt256(data map[string]interface{}, secret string) (string, error)
- func JwtEncrypt384(data map[string]interface{}, secret string) (string, error)
- func JwtEncrypt512(data map[string]interface{}, secret string) (string, error)
- func LatestDate(date int) []string
- func LinuxSendCommand(command string) (opStr string)
- func MD5(str string) string
- func Map2Json(m interface{}) (string, error)
- func Map2Slice(data interface{}) []interface{}
- func MapCopy(data map[string]interface{}) (copy map[string]interface{})
- func MapMergeCopy(src ...map[string]interface{}) (copy map[string]interface{})
- func MapStr2Any(m map[string]string) map[string]interface{}
- func MinuteAgo(i int) int64
- func MongoConn()
- func MongoConn1()
- func NewConf(appConfigPath string) error
- func NewCookiePool() *cookiePool
- func NewGDMap() *gDMap
- func NewMysqlDB(host string, port int, user, password, database string) (err error)
- func NowToEnd() (int64, error)
- func OSLine() string
- func OutJsonFile(data interface{}, fileName string) error
- func P2E()
- func PBKDF2(str, salt []byte, iterations, keySize int) []byte
- func Panic(args ...interface{})
- func PanicToError(fn func()) (err error)
- func PathExists(path string)
- func Ping(ip string) (time.Duration, error)
- func Pwd() string
- func ReadCsvFile(filename string) [][]string
- func RedisDELKeys(rds *Rds, keys string, jobNumber int)
- func RegBase64(str string, property ...string) []string
- func RegDNSName(str string, property ...string) []string
- func RegDelHtml(str string) string
- func RegDelHtmlA(str string) string
- func RegDelHtmlButton(str string, property ...string) string
- func RegDelHtmlCanvas(str string, property ...string) string
- func RegDelHtmlCode(str string, property ...string) string
- func RegDelHtmlH(str, typeH string, property ...string) string
- func RegDelHtmlHref(str string, property ...string) string
- func RegDelHtmlImg(str string, property ...string) string
- func RegDelHtmlInput(str string, property ...string) string
- func RegDelHtmlLi(str string, property ...string) string
- func RegDelHtmlMeta(str string, property ...string) string
- func RegDelHtmlP(str string, property ...string) string
- func RegDelHtmlSelect(str string, property ...string) string
- func RegDelHtmlSpan(str string, property ...string) string
- func RegDelHtmlSrc(str string, property ...string) string
- func RegDelHtmlTable(str string, property ...string) string
- func RegDelHtmlTbody(str string, property ...string) string
- func RegDelHtmlTd(str string, property ...string) string
- func RegDelHtmlTitle(str string) string
- func RegDelHtmlTr(str string) string
- func RegDelHtmlUl(str string, property ...string) string
- func RegDelHtmlVideo(str string, property ...string) string
- func RegDelNumber(str string) string
- func RegEmail(str string, property ...string) []string
- func RegEmail2(str string, property ...string) []string
- func RegFindAll(regStr, rest string) [][]string
- func RegFindAllTxt(regStr, rest string) (dataList []string)
- func RegFloat(str string, property ...string) []string
- func RegFullURL(str string, property ...string) []string
- func RegFullWidth(str string, property ...string) []string
- func RegGUID(str string, property ...string) []string
- func RegHalfWidth(str string, property ...string) []string
- func RegHtmlA(str string, property ...string) []string
- func RegHtmlATxt(str string, property ...string) []string
- func RegHtmlButton(str string, property ...string) []string
- func RegHtmlButtonTxt(str string, property ...string) []string
- func RegHtmlCanvas(str string, property ...string) []string
- func RegHtmlCode(str string, property ...string) []string
- func RegHtmlCodeTxt(str string, property ...string) []string
- func RegHtmlDescription(str string, property ...string) []string
- func RegHtmlDescriptionTxt(str string, property ...string) []string
- func RegHtmlDiv(str string, property ...string) []string
- func RegHtmlDivTxt(str string, property ...string) []string
- func RegHtmlH(str, typeH string, property ...string) []string
- func RegHtmlHTxt(str, typeH string, property ...string) []string
- func RegHtmlHref(str string, property ...string) []string
- func RegHtmlHrefTxt(str string, property ...string) []string
- func RegHtmlImg(str string, property ...string) []string
- func RegHtmlInput(str string, property ...string) []string
- func RegHtmlInputTxt(str string, property ...string) []string
- func RegHtmlKeyword(str string, property ...string) []string
- func RegHtmlKeywordTxt(str string, property ...string) []string
- func RegHtmlLi(str string, property ...string) []string
- func RegHtmlLiTxt(str string, property ...string) []string
- func RegHtmlMeta(str string, property ...string) []string
- func RegHtmlOption(str string, property ...string) []string
- func RegHtmlOptionTxt(str string, property ...string) []string
- func RegHtmlP(str string, property ...string) []string
- func RegHtmlPTxt(str string, property ...string) []string
- func RegHtmlSelect(str string, property ...string) []string
- func RegHtmlSelectTxt(str string, property ...string) []string
- func RegHtmlSpan(str string, property ...string) []string
- func RegHtmlSpanTxt(str string, property ...string) []string
- func RegHtmlSrc(str string, property ...string) []string
- func RegHtmlSrcTxt(str string, property ...string) []string
- func RegHtmlTable(str string, property ...string) []string
- func RegHtmlTableOlny(str string, property ...string) []string
- func RegHtmlTableTxt(str string, property ...string) []string
- func RegHtmlTbody(str string, property ...string) []string
- func RegHtmlTd(str string, property ...string) []string
- func RegHtmlTdTxt(str string, property ...string) []string
- func RegHtmlTitle(str string, property ...string) []string
- func RegHtmlTitleTxt(str string, property ...string) []string
- func RegHtmlTr(str string, property ...string) []string
- func RegHtmlTrTxt(str string, property ...string) []string
- func RegHtmlUl(str string, property ...string) []string
- func RegHtmlUlTxt(str string, property ...string) []string
- func RegHtmlVideo(str string, property ...string) []string
- func RegIP(str string, property ...string) []string
- func RegIPv4(str string, property ...string) []string
- func RegIPv6(str string, property ...string) []string
- func RegInt(str string, property ...string) []string
- func RegLatitude(str string, property ...string) []string
- func RegLink(str string, property ...string) []string
- func RegLongitude(str string, property ...string) []string
- func RegMACAddress(str string, property ...string) []string
- func RegMD5Hex(str string, property ...string) []string
- func RegRGBColor(str string, property ...string) []string
- func RegSHA1Hex(str string, property ...string) []string
- func RegSHA256Hex(str string, property ...string) []string
- func RegTime(str string, property ...string) []string
- func RegURLIP(str string, property ...string) []string
- func RegURLPath(str string, property ...string) []string
- func RegURLPort(str string, property ...string) []string
- func RegURLSchema(str string, property ...string) []string
- func RegURLSubdomain(str string, property ...string) []string
- func RegURLUsername(str string, property ...string) []string
- func RegUUID(str string, property ...string) []string
- func RegUUID3(str string, property ...string) []string
- func RegUUID4(str string, property ...string) []string
- func RegUUID5(str string, property ...string) []string
- func RegUnixPath(str string, property ...string) []string
- func RegValue(str string, property ...string) []string
- func RegWinPath(str string, property ...string) []string
- func ReplaceAllToOne(str string, from []string, to string) string
- func SSHClient(user string, pass string, addr string) (*ssh.Client, error)
- func SearchBytesIndex(bSrc []byte, b byte) int
- func SearchDomain(ip string)
- func SearchPort(ipStr string, vs ...interface{})
- func SetAgent(agentType UserAgentType, agent string)
- func SetLogFile(name string)
- func SetStatusCodeFailEvent(code int)
- func SetStatusCodeRetryEvent(code int)
- func SetStatusCodeSuccessEvent(code int)
- func Slice2Map(slice interface{}) map[string]interface{}
- func SliceCopy(data []interface{}) []interface{}
- func SliceTool() *sliceTool
- func SocketProxy(addr string)
- func StartJob(jobNumber int, queue TodoQueue, f func(task *Task))
- func StartJobGet(jobNumber int, queue TodoQueue, vs ...interface{})
- func StartJobPost(jobNumber int, queue TodoQueue, vs ...interface{})
- func Str2Byte(s string) []byte
- func Str2Float32(str string) float32
- func Str2Float64(str string) float64
- func Str2Int(str string) int
- func Str2Int32(str string) int32
- func Str2Int64(str string) int64
- func StrDeleteSpace(str string) string
- func StrDuplicates(a []string) []string
- func StrToSize(sizeStr string) int64
- func StringValue(i interface{}) string
- func StringValueMysql(i interface{}) string
- func Struct2Map(obj interface{}) map[string]interface{}
- func Struct2MapV2(obj interface{}, hasValue bool) (map[string]interface{}, error)
- func Struct2MapV3(obj interface{}) map[string]interface{}
- func TickerRun(t time.Duration, runFirst bool, f func())
- func Timestamp() string
- func Timestamp2Date(timestamp int64) string
- func Timestamp2Week(timestamp int64) string
- func Timestamp2WeekXinQi(timestamp int64) string
- func ToBIG5(srcCharset string, src string) (dst string, err error)
- func ToGB18030(srcCharset string, src string) (dst string, err error)
- func ToGB2312(srcCharset string, src string) (dst string, err error)
- func ToGDK(srcCharset string, src string) (dst string, err error)
- func ToHZGB2312(srcCharset string, src string) (dst string, err error)
- func ToUTF16(srcCharset string, src string) (dst string, err error)
- func ToUTF8(srcCharset string, src string) (dst string, err error)
- func UInt32ToIP(ip uint32) net.IP
- func UTF16To(dstCharset string, src string) (dst string, err error)
- func UTF8To(dstCharset string, src string) (dst string, err error)
- func Uint82Str(bs []uint8) string
- func UnescapeUnicode(raw []byte) ([]byte, error)
- func UnicodeDec(raw string) string
- func UnicodeDecByte(raw []byte) []byte
- func UrlStr(url string) string
- func UsefulUrl(str string) bool
- func Warn(args ...interface{})
- func WarnF(format string, args ...interface{})
- func WarnFTimes(format string, times int, args ...interface{})
- func WarnTimes(times int, args ...interface{})
- func WindowsSendCommand(command []string) (opStr string)
- func WindowsSendPipe(command1, command2 []string) (opStr string)
- type AES
- type Bar
- type Charset
- type Context
- func Delete(url string, vs ...interface{}) (*Context, error)
- func Get(url string, vs ...interface{}) (*Context, error)
- func NewDelete(url string, vs ...interface{}) *Context
- func NewGet(url string, vs ...interface{}) *Context
- func NewOptions(url string, vs ...interface{}) *Context
- func NewPost(url string, data []byte, contentType string, vs ...interface{}) *Context
- func NewPostForm(u string, data map[string]string, vs ...interface{}) *Context
- func NewPut(url string, data []byte, contentType string, vs ...interface{}) *Context
- func NewRequest(url, method string, data []byte, contentType string, vs ...interface{}) *Context
- func Options(url string, vs ...interface{}) (*Context, error)
- func Post(url string, data []byte, contentType string, vs ...interface{}) (*Context, error)
- func PostFile(url, paramName, filePath string, vs ...interface{}) *Context
- func PostForm(url string, data map[string]string, vs ...interface{}) (*Context, error)
- func PostJson(url string, jsonStr string, vs ...interface{}) (*Context, error)
- func Put(url string, data []byte, contentType string, vs ...interface{}) (*Context, error)
- func Req(request *http.Request, vs ...interface{}) *Context
- func Request(url, method string, data []byte, contentType string, vs ...interface{}) (*Context, error)
- func Upload(url, savePath string, vs ...interface{}) (*Context, error)
- func (c *Context) AddCookie(k, v string) *Context
- func (c *Context) AddHeader(k, v string) *Context
- func (c *Context) AddParam(key string, val interface{})
- func (c *Context) CheckMd5() string
- func (c *Context) CheckReqMd5() string
- func (c *Context) CloseLog()
- func (c *Context) CloseRetry()
- func (c *Context) CookieNext() error
- func (c *Context) DelParam(key string)
- func (c *Context) Do() func()
- func (c *Context) GetParam(key string) interface{}
- func (c *Context) GetRespHeader() string
- func (c *Context) GetRetryTimes() int
- func (c *Context) OpenErr2Retry()
- func (c *Context) RespBodyArr() []interface{}
- func (c *Context) RespBodyHtml() string
- func (c *Context) RespBodyMap() map[string]interface{}
- func (c *Context) RespBodyString() string
- func (c *Context) RespContentLength() int64
- func (c *Context) SetFailedFunc(failedFunc func(c *Context)) *Context
- func (c *Context) SetProxy(proxyUrl string) *Context
- func (c *Context) SetProxyFunc(f func() *http.Transport) *Context
- func (c *Context) SetProxyPool(pool *ProxyPool) *Context
- func (c *Context) SetRetryFunc(retryFunc func(c *Context)) *Context
- func (c *Context) SetRetryTimes(times int) *Context
- func (c *Context) SetSleep(i int) *Context
- func (c *Context) SetSleepRand(min, max int) *Context
- func (c *Context) SetSucceedFunc(successFunc func(c *Context)) *Context
- func (c *Context) Upload(filePath string) func()
- type Cookie
- type Csv
- type DES
- type DNSInfo
- type EndFunc
- type Err2Retry
- type FailedFunc
- type FormData
- type GDMapApi
- type Header
- type HostPageSpeedCheck
- func (scan *HostPageSpeedCheck) AverageSpeed() float64
- func (scan *HostPageSpeedCheck) MaxSpeed() int64
- func (scan *HostPageSpeedCheck) MinSpeed() int64
- func (scan *HostPageSpeedCheck) Report() map[string]string
- func (scan *HostPageSpeedCheck) Result() map[string]time.Duration
- func (scan *HostPageSpeedCheck) Run() ([]string, int)
- type HostScanBadLink
- type HostScanExtLinks
- type HostScanUrl
- type HostToolEr
- type HttpPackage
- type IdWorker
- type Intercept
- type IsLog
- type IsRetry
- type Level
- type MQKafkaService
- type MQNsqService
- type MQRabbitService
- type MQer
- type Mail
- type Mongo
- type Mysql
- func (m *Mysql) CloseLog()
- func (m *Mysql) Conn() (err error)
- func (m *Mysql) Delete(sql string) error
- func (m *Mysql) DeleteTable(tableName string) error
- func (m *Mysql) Describe(table string) (*tableDescribe, error)
- func (m *Mysql) Exec(sql string) error
- func (m *Mysql) GetFieldList(table string) (fieldList []string)
- func (m *Mysql) HasTable(tableName string) bool
- func (m *Mysql) Insert(table string, fieldData map[string]interface{}) error
- func (m *Mysql) InsertAt(table string, fieldData map[string]interface{}) error
- func (m *Mysql) InsertAtGd(table string, fieldData *gDMap) error
- func (m *Mysql) InsertAtJson(table, jsonStr string) error
- func (m *Mysql) IsHaveTable(table string) bool
- func (m *Mysql) NewTable(table string, fields map[string]string) error
- func (m *Mysql) NewTableGd(table string, fields *gDMap) error
- func (m *Mysql) Query(sql string) ([]map[string]string, error)
- func (m *Mysql) Select(sql string) ([]map[string]string, error)
- func (m *Mysql) SetMaxIdleConn(number int)
- func (m *Mysql) SetMaxOpenConn(number int)
- func (m *Mysql) ToVarChar(data interface{}) string
- func (m *Mysql) ToXls(sql, outPath string)
- func (m *Mysql) Update(sql string) error
- type ProxyIP
- type ProxyPool
- type ProxyUrl
- type Queue
- type RabbitMQ
- func NewRabbitMQ(queueName, exchange, key, amqpUrl string) (*RabbitMQ, error)
- func NewRabbitMQPubSub(exchangeName, amqpUrl string) (*RabbitMQ, error)
- func NewRabbitMQRouting(exchange string, routingKey string) (*RabbitMQ, error)
- func NewRabbitMQSimple(queueName string) (*RabbitMQ, error)
- func NewRabbitMQTopic(exchange string, routingKey string) (*RabbitMQ, error)
- func (r *RabbitMQ) Destroy()
- func (r *RabbitMQ) PublishPub(message []byte) (err error)
- func (r *RabbitMQ) PublishRouting(message []byte) (err error)
- func (r *RabbitMQ) PublishSimple(message []byte) (err error)
- func (r *RabbitMQ) PublishTopic(message []byte) (err error)
- func (r *RabbitMQ) RegistryConsumeSimple() (msg <-chan amqp.Delivery)
- func (r *RabbitMQ) RegistryReceiveRouting() (msg <-chan amqp.Delivery)
- func (r *RabbitMQ) RegistryReceiveSub() (msg <-chan amqp.Delivery)
- func (r *RabbitMQ) RegistryReceiveTopic() (msg <-chan amqp.Delivery)
- type Rds
- func (r *Rds) Append(key string, value interface{}) error
- func (r *Rds) BitCount(key string) (int64, error)
- func (r *Rds) DUMP(key string) bool
- func (r *Rds) Decr(key string) (int64, error)
- func (r *Rds) DecrBy(key, decrement string) (int64, error)
- func (r *Rds) DelKey(key string) bool
- func (r *Rds) Expire(key string, ttl int64) bool
- func (r *Rds) ExpireAt(key string, date int64) bool
- func (r *Rds) Get(key string) (string, error)
- func (r *Rds) GetAllKeys(match string) (ksyList map[string]int)
- func (r *Rds) GetBit(key string, offset int64) (int64, error)
- func (r *Rds) GetConn() redis.Conn
- func (r *Rds) GetRange(key string, start, end int64) (string, error)
- func (r *Rds) GetSet(key string, value interface{}) (string, error)
- func (r *Rds) HDel(key string, fields []string) error
- func (r *Rds) HExIsTs(key, fields string) bool
- func (r *Rds) HGet(key, fields string) (string, error)
- func (r *Rds) HGetAll(key string) (map[string]string, error)
- func (r *Rds) HGetAllInt(key string) (map[string]int, error)
- func (r *Rds) HGetAllInt64(key string) (map[string]int64, error)
- func (r *Rds) HIncrBy(key, field string, increment int64) (int64, error)
- func (r *Rds) HIncrByFloat(key, field string, increment float64) (float64, error)
- func (r *Rds) HKeys(key string) ([]string, error)
- func (r *Rds) HLen(key string) (int64, error)
- func (r *Rds) HMGet(key string, fields []string) ([]string, error)
- func (r *Rds) HMSet(key string, values map[interface{}]interface{}) error
- func (r *Rds) HSet(key, field string, value interface{}) (int64, error)
- func (r *Rds) HSetNx(key, field string, value interface{}) error
- func (r *Rds) HVaLs(key string) ([]string, error)
- func (r *Rds) Incr(key string) (int64, error)
- func (r *Rds) IncrBy(key, increment string) (int64, error)
- func (r *Rds) IncrByFloat(key, increment float64) (float64, error)
- func (r *Rds) LIndex(key string, index int64) (string, error)
- func (r *Rds) LInsert(direction bool, key, pivot, value string) error
- func (r *Rds) LLen(key string) (int64, error)
- func (r *Rds) LPusHx(key string, value interface{}) error
- func (r *Rds) LPush(key string, values []interface{}) error
- func (r *Rds) LRange(key string) ([]interface{}, error)
- func (r *Rds) LRangeST(key string, start, stop int64) ([]interface{}, error)
- func (r *Rds) LRem(key string, count int64, value interface{}) error
- func (r *Rds) LSet(key string, index int64, value interface{}) error
- func (r *Rds) LTrim(key string, start, stop int64) error
- func (r *Rds) ListLPOP(key string) (string, error)
- func (r *Rds) MGet(key []interface{}) ([]string, error)
- func (r *Rds) MSet(values []interface{}) error
- func (r *Rds) MSetNx(values []interface{}) error
- func (r *Rds) MqConsumer(mqName string) (reply interface{}, err error)
- func (r *Rds) MqLen(mqName string) int64
- func (r *Rds) MqProducer(mqName string, data interface{}) error
- func (r *Rds) PSetEx(key string, ttl int64, value interface{}) error
- func (r *Rds) RPop(key string) (string, error)
- func (r *Rds) RPopLPush(key, destination string) (string, error)
- func (r *Rds) RPush(key string, values []interface{}) error
- func (r *Rds) RPushX(key string, value interface{}) error
- func (r *Rds) RedisConn() (err error)
- func (r *Rds) RedisPool() error
- func (r *Rds) Rename(name, newName string) bool
- func (r *Rds) SAdd(key string, values []interface{}) error
- func (r *Rds) SCard(key string) error
- func (r *Rds) SDiff(keys []string) ([]interface{}, error)
- func (r *Rds) SDiffStore(key string, keys []string) ([]interface{}, error)
- func (r *Rds) SInter(keys []string) ([]interface{}, error)
- func (r *Rds) SInterStore(key string, keys []string) ([]interface{}, error)
- func (r *Rds) SIsMember(key string, value interface{}) (resBool bool, err error)
- func (r *Rds) SMemeRs(key string) ([]interface{}, error)
- func (r *Rds) SMove(key, destination string, member interface{}) (resBool bool, err error)
- func (r *Rds) SPop(key string) (string, error)
- func (r *Rds) SRandMember(key string, count int64) ([]interface{}, error)
- func (r *Rds) SRem(key string, member []interface{}) error
- func (r *Rds) SUnion(keys []string) ([]interface{}, error)
- func (r *Rds) SUnionStore(key string, keys []string) ([]interface{}, error)
- func (r *Rds) SearchKeys(match string) (ksyList map[string]int)
- func (r *Rds) SelectDB(dbNumber int) error
- func (r *Rds) Set(key string, value interface{}) error
- func (r *Rds) SetBit(key string, offset, value int64) error
- func (r *Rds) SetEx(key string, ttl int64, value interface{}) error
- func (r *Rds) SetNx(key string, value interface{}) error
- func (r *Rds) SetRange(key string, offset int64, value interface{}) error
- func (r *Rds) ToBool(reply interface{}, err error) (bool, error)
- func (r *Rds) ToByteSlices(reply interface{}, err error) ([][]byte, error)
- func (r *Rds) ToBytes(reply interface{}, err error) ([]byte, error)
- func (r *Rds) ToFloat64(reply interface{}, err error) (float64, error)
- func (r *Rds) ToFloat64s(reply interface{}, err error) ([]float64, error)
- func (r *Rds) ToInt(reply interface{}, err error) (int, error)
- func (r *Rds) ToInt64(reply interface{}, err error) (int64, error)
- func (r *Rds) ToInt64Map(reply interface{}, err error) (map[string]int64, error)
- func (r *Rds) ToInt64s(reply interface{}, err error) ([]int64, error)
- func (r *Rds) ToIntMap(reply interface{}, err error) (map[string]int, error)
- func (r *Rds) ToInts(reply interface{}, err error) ([]int, error)
- func (r *Rds) ToString(reply interface{}, err error) (string, error)
- func (r *Rds) ToStringMap(reply interface{}, err error) (map[string]string, error)
- func (r *Rds) ToStrings(reply interface{}, err error) ([]string, error)
- func (r *Rds) Ttl(key string) int64
- func (r *Rds) Type(key string) string
- func (r *Rds) ZAdd(key string, weight interface{}, field interface{}) error
- func (r *Rds) ZCard(key string) (int64, error)
- func (r *Rds) ZCount(key string, min, max int64) (int64, error)
- func (r *Rds) ZIncrBy(key, member string, increment int64) (string, error)
- func (r *Rds) ZRange(key string) ([]interface{}, error)
- func (r *Rds) ZRangeByScore(key string, min, max, offset, count int64) ([]interface{}, error)
- func (r *Rds) ZRangeByScoreAll(key string) ([]interface{}, error)
- func (r *Rds) ZRangeST(key string, start, stop int64) ([]interface{}, error)
- func (r *Rds) ZRank(key string, member interface{}) (int64, error)
- func (r *Rds) ZRem(key string, member []interface{}) error
- func (r *Rds) ZRemRangeByRank(key string, start, stop int64) error
- func (r *Rds) ZRemRangeByScore(key string, min, max int64) error
- func (r *Rds) ZRevRange(key string, start, stop int64) ([]interface{}, error)
- func (r *Rds) ZRevRangeByScore(key string, min, max, offset, count int64) ([]interface{}, error)
- func (r *Rds) ZRevRangeByScoreAll(key string) ([]interface{}, error)
- func (r *Rds) ZRevRank(key string, member interface{}) (int64, error)
- func (r *Rds) ZScore(key string, member interface{}) (string, error)
- type ReqTimeOut
- type ReqTimeOutMs
- type ReqUrl
- type RetryFunc
- type RetryTimes
- type SSHConnInfo
- type SSLCertificateInfo
- type Set
- type Sleep
- type Stack
- type StartFunc
- type StressUrl
- type SucceedFunc
- type TableInfo
- type Task
- type TcpClient
- func (c *TcpClient) Addr() string
- func (c *TcpClient) Close()
- func (c *TcpClient) ReConn()
- func (c *TcpClient) Read(b []byte) (int, error)
- func (c *TcpClient) Run(serverHost string, r func(c *TcpClient, data []byte), w func(c *TcpClient))
- func (c *TcpClient) Send(b []byte) (int, error)
- func (c *TcpClient) Stop()
- type TodoQueue
- type UdpClient
- type UploadQueue
- type UserAgentType
- type WSClient
- type WhoisInfo
Constants ¶
View Source
const ( CBC = "CBC" ECB = "ECB" CFB = "CFB" CTR = "CTR" )
View Source
const ( POST = "POST" GET = "GET" HEAD = "HEAD" PUT = "PUT" DELETE = "DELETE" PATCH = "PATCH" OPTIONS = "OPTIONS" ANY = "" )
View Source
const ( UTF8 = Charset("UTF-8") GB18030 = Charset("GB18030") GBK = Charset("GBK") GB2312 = Charset("GB2312") )
View Source
const ( KiB = 1024 MiB = KiB * 1024 GiB = MiB * 1024 TiB = GiB * 1024 )
字节换算
View Source
const TimeMilliTemplate = "2006-01-02 15:04:05.000"
View Source
const TimeTemplate = "2006-01-02 15:04:05"
Variables ¶
View Source
var ( ShowTablesSql = "SHOW TABLES" TABLE_NAME_NULL = fmt.Errorf("table name is null") TABLE_IS_NULL = fmt.Errorf("table is null") )
View Source
var ( ChineseNumber = []string{"一", "二", "三", "四", "五", "六", "七", "八", "九", "零"} ChineseMoney = []string{"壹", "贰", "叁", "肆", "伍", "陆", "柒", "捌", "玖"} ChineseMoneyAll = []string{"壹", "贰", "叁", "肆", "伍", "陆", "柒", "捌", "玖", "拾", "佰", "仟", "万", "亿", "元", "角", "分", "零", "整", "正", "貳", "陸", "億", "萬", "圓"} )
View Source
var ( UrlBad = errors.New("url is bad.") // 错误的url UrlNil = errors.New("url is null.") // 空的url )
View Source
var AgentType = map[UserAgentType][]int{ PCAgent: listPCAgent, WindowsAgent: listWindowsAgent, LinuxAgent: listLinuxAgent, MacAgent: listMacAgent, AndroidAgent: listAndroidAgent, IosAgent: listIosAgent, PhoneAgent: listPhoneAgent, WindowsPhoneAgent: listWindowsPhoneAgent, UCAgent: listUCAgent, }
View Source
var ApplicationTerminalOut = true
View Source
var Config *conf
View Source
var ContentType = map[string]string{}/* 315 elements not displayed */
ContentType 数据类型
View Source
var CookiePool *cookiePool
CookiePool Cookie池
View Source
var DNSServer = map[string]string{
"114.114.114.114": "公共DNS|114DNS",
"114.114.115.115": "公共DNS|114DNS",
"101.226.4.6": "公共DNS|DNS 派 电信/移动/铁通",
"218.30.118.6": "公共DNS|DNS 派 电信/移动/铁通",
"123.125.81.6": "公共DNS|DNS DNS 派 联通",
"140.207.198.6": "公共DNS|DNS DNS 派 联通",
"8.8.8.8": "公共DNS|GoogleDNS",
"223.5.5.5": "公共DNS|阿里云DNS",
"223.6.6.6": "公共DNS|阿里云DNS",
"180.76.76.76": "公共DNS|百度云DNS",
"4.2.2.1": "公共DNS|微软云DNS",
"122.112.208.1": "公共DNS|华为云DNS",
"139.9.23.90": "公共DNS|华为云DNS",
"114.115.192.11": "公共DNS|华为云DNS",
"116.205.5.1": "公共DNS|华为云DNS",
"116.205.5.30": "公共DNS|华为云DNS",
"122.112.208.175": "公共DNS|华为云DNS",
"139.159.208.206": "公共DNS|华为云DNS",
"180.184.1.1": "公共DNS|字节跳动",
"180.184.2.2": "公共DNS|字节跳动",
"168.95.192.1": "公共DNS|中華電信DNS",
"61.132.163.68": "电信|安徽",
"202.102.213.68": "电信|安徽",
"202.98.192.67": "电信|贵州",
"202.98.198.167": "电信|贵州",
"202.101.224.69": "电信|江西",
"61.139.2.69": "电信|四川",
"218.6.200.139": "电信|四川",
"202.98.96.68": "电信|四川成都",
"202.102.192.68": "电信|安徽合肥",
"221.7.1.21": "联通|新疆维吾尔自治区乌鲁木齐",
"202.106.46.151": "联通|北京",
}
DNSServer DNS服务器地址大全 ip:服务商
View Source
var Leaps = []int{0, 31, 29, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}
Leaps 闰年的天数
View Source
var LevelMap = map[Level]string{
1: "[Info] ",
2: "[Debug] ",
3: "[Warn] ",
4: "[Error] ",
5: "[HTTP/s]",
}
View Source
var LogClose bool = true
LogClose 是否关闭日志
View Source
var MysqlDB = &Mysql{}
View Source
var Pyears = []int{0, 31, 28, 31, 30, 31, 30, 31, 31, 30, 31, 30, 31}
PyearS 平年天数
View Source
var RdsNotConnError = fmt.Errorf("未连接redis")
View Source
var RootWhoisServers = "whois.iana.org:43"
View Source
var StatusCodeMap map[int]string = map[int]string{
200: "success",
201: "success",
202: "success",
203: "success",
204: "fail",
300: "success",
301: "success",
302: "success",
400: "fail",
401: "retry",
402: "retry",
403: "retry",
404: "fail",
405: "retry",
406: "retry",
407: "retry",
408: "retry",
412: "success",
500: "fail",
501: "fail",
502: "retry",
503: "retry",
504: "retry",
}
StatusCodeMap 状态码处理映射 success 该状态码对应执行成功函数 fail 该状态码对应执行失败函数 retry 该状态码对应需要重试前执行的函数
View Source
var UserAgentMap map[int]string = map[int]string{
1: "Mozilla/5.0 (Windows NT 6.2; WOW64; rv:21.0) Gecko/20100101 Firefox/21.0",
2: "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.94 Safari/537.36",
3: "Mozilla/5.0 (compatible; WOW64; MSIE 10.0; Windows NT 6.2)",
4: "Opera/9.80 (Windows NT 6.1; WOW64; U; en) Presto/2.10.229 Version/11.62",
5: "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27",
6: "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:21.0) Gecko/20130331 Firefox/21.0",
7: "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.11 (KHTML, like Gecko) Ubuntu/11.10 Chromium/27.0.1453.93 Chrome/27.0.1453.93 Safari/537.36",
8: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-US) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27",
9: "Opera/9.80 (Macintosh; Intel Mac OS X 10.6.8; U; en) Presto/2.9.168 Version/11.52",
10: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36",
11: "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:21.0) Gecko/20100101 Firefox/21.0",
12: "Mozilla/5.0 (Linux; Android 4.1.1; Nexus 7 Build/JRO03D) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19",
13: "Mozilla/5.0 (Linux; U; Android 4.0.4; en-gb; GT-I9300 Build/IMM76D) AppleWebKit/534.30 (KHTML, like Gecko) Version/4.0 Mobile Safari/534.30",
14: "Mozilla/5.0 (Linux; U; Android 2.2; en-gb; GT-P1000 Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1",
15: "Mozilla/5.0 (Android; Mobile; rv:14.0) Gecko/14.0 Firefox/14.0",
16: "Mozilla/5.0 (Android; Tablet; rv:14.0) Gecko/14.0 Firefox/14.0",
17: "Mozilla/5.0 (Linux; Android 4.0.4; Galaxy Nexus Build/IMM76B) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.133 Mobile Safari/535.19",
18: "Mozilla/5.0 (Linux; Android 4.1.2; Nexus 7 Build/JZ054K) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Safari/535.19",
19: "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3",
20: "Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3",
21: "Mozilla/5.0 (iPhone; CPU iPhone OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3",
22: "Mozilla/5.0 (iPad; CPU OS 5_0 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A334 Safari/7534.48.3",
23: "Mozilla/5.0 (iPhone; CPU iPhone OS 6_1_4 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) CriOS/27.0.1453.10 Mobile/10B350 Safari/8536.25",
24: "Mozilla/5.0 (compatible; MSIE 10.0; Windows Phone 8.0; Trident/6.0; IEMobile/10.0; ARM; Touch; NOKIA; Lumia 920)",
25: "Mozilla/5.0 (compatible; MSIE 9.0; Windows Phone OS 7.5; Trident/5.0; IEMobile/9.0; SAMSUNG; SGH-i917)",
26: "User-Agent, UCWEB7.0.2.37/28/999",
27: "User-Agent, NOKIA5700/ UCWEB7.0.2.37/28/999",
28: "User-Agent, Openwave/ UCWEB7.0.2.37/28/999",
29: "User-Agent, Mozilla/4.0 (compatible; MSIE 6.0; ) Opera/UCWEB7.0.2.37/28/999",
30: "Mozilla/5.0 (Windows; U; Windows NT 6.1; ) AppleWebKit/534.12 (KHTML, like Gecko) Maxthon/3.0 Safari/534.12",
31: "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)",
32: "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E; SE 2.X MetaSr 1.0)",
33: "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.472.33 Safari/534.3 SE 2.X MetaSr 1.0",
34: "Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; InfoPath.3; .NET4.0C; .NET4.0E)",
35: "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/13.0.782.41 Safari/535.1 QQBrowser/6.9.11079.201",
}
Functions ¶
func AbPathByCaller ¶ added in v0.3.5
func AbPathByCaller() string
AbPathByCaller 获取当前执行文件绝对路径（go run）
func Any2Arr ¶ added in v0.1.17
func Any2Arr(data interface{}) []interface{}
Any2Arr interface{} -> []interface{}
func Any2Float64 ¶ added in v0.0.10
func Any2Float64(data interface{}) float64
Any2Float64 interface{} -> float64
func Any2Int64 ¶ added in v0.4.5
func Any2Int64(data interface{}) int64
Any2Int64 interface{} -> int64
func Any2Map ¶ added in v0.0.10
func Any2Map(data interface{}) map[string]interface{}
Any2Map interface{} -> map[string]interface{}
func Any2String ¶ added in v0.0.10
func Any2String(data interface{}) string
Any2String interface{} -> string
func Any2Strings ¶ added in v0.0.10
func Any2Strings(data interface{}) []string
Any2Strings interface{} -> []string
func ApplicationTerminalOutClose ¶ added in v0.4.7
func ApplicationTerminalOutClose()
func Base64Decode ¶ added in v0.1.4
Base64Decode base64 解码
func Base64UrlDecode ¶ added in v0.1.4
Base64UrlDecode base64 url 解码
func Base64UrlEncode ¶ added in v0.1.4
Base64UrlEncode base64 url 编码
func Byte2Float32 ¶ added in v0.2.8
Byte2Float32 []byte -> float32
func Byte2Float64 ¶ added in v0.2.8
Byte2Float64 []byte -> float64
func ByteToBinaryString ¶ added in v0.1.4
ByteToBinaryString 字节 -> 二进制字符串
func ByteToUTF8 ¶ added in v0.3.5
ByteToUTF8 byte -> utf8 byte
func CleaningStr ¶ added in v0.0.11
CleaningStr 清理字符串前后空白 和回车 换行符号
func ClosePingTerminalPrint ¶ added in v0.4.5
func ClosePingTerminalPrint()
func CompressDirZip ¶ added in v0.4.1
CompressDirZip 压缩目录
func CompressFiles ¶ added in v0.4.1
CompressFiles 压缩很多文件 files 文件数组，可以是不同dir下的文件或者文件夹 dest 压缩文件存放地址
func ConvertByte2String ¶ added in v0.1.4
ConvertByte2String 编码转换
func ConvertGBK2Str ¶ added in v0.3.5
ConvertGBK2Str 将GBK编码的字符串转换为utf-8编码
func ConvertStr2GBK ¶ added in v0.3.5
ConvertStr2GBK 将utf-8编码的字符串转换为GBK编码
func CopySliceInt ¶ added in v0.2.1
func CopySliceInt64 ¶ added in v0.2.1
func CopySliceStr ¶ added in v0.2.1
func DeCompressTAR ¶ added in v0.4.1
DeCompressTAR tar 解压文件
func DeCompressZIP ¶ added in v0.4.1
DeCompressZIP zip解压文件
func DebugFTimes ¶ added in v0.4.7
DebugFTimes 日志-调试, 指定日志代码位置的定位调用层级
func DebugTimes ¶ added in v0.3.7
func DebugTimes(times int, args ...interface{})
DebugTimes 日志-调试, 指定日志代码位置的定位调用层级
func DecodeByte ¶ added in v0.2.8
DecodeByte decode byte
func DecompressionZipFile ¶ added in v0.4.1
DecompressionZipFile zip压缩文件
func ErrorTimes ¶ added in v0.3.7
func ErrorTimes(times int, args ...interface{})
ErrorTimes 日志-错误, 指定日志代码位置的定位调用层级
func ErrorfTimes ¶ added in v0.3.7
ErrorfTimes 日志-错误, 指定日志代码位置的定位调用层级
func FileSizeFormat ¶ added in v0.1.1
FileSizeFormat 字节的单位转换 保留两位小数
func Float322Byte ¶ added in v0.2.8
Float322Byte float32 -> []byte
func Float322Uint32 ¶ added in v0.2.8
Float322Uint32 float32 -> uint32
func Float642Byte ¶ added in v0.2.8
Float642Byte float64 -> []byte
func Float642Uint64 ¶ added in v0.2.8
Float642Uint64 float64 -> uint64
func Get16MD5Encode ¶ added in v0.4.1
Get16MD5Encode 返回一个16位md5加密后的字符串
func GetAgent ¶ added in v0.0.4
func GetAgent(agentType UserAgentType) string
GetAgent 随机获取那种类型的 user-agent
func GetAllFile ¶ added in v0.4.1
GetAllFile 获取目录下的所有文件
func GetChineseMonthDay ¶ added in v0.4.2
GetChineseMonthDay 获取农历
func GetPointClassHTML ¶ added in v0.3.7
GetPointClassHTML 获取指定标签class属性的html
func GetPointHTML ¶ added in v0.3.7
GetPointHTML 获取指定位置的HTML， 用标签， 标签属性， 属性值来定位
func GetPointIDHTML ¶ added in v0.3.7
GetPointIDHTML 获取指定标签id属性的html
func GzipCompress ¶ added in v0.3.5
func GzipDecompress ¶ added in v0.3.5
func HTTPTimes ¶ added in v0.4.3
func HTTPTimes(times int, args ...interface{})
HTTPTimes 日志-信息, 指定日志代码位置的定位调用层级
func HZGB2312To ¶ added in v0.2.8
HZGB2312To hzgb2312 to
func HumanFriendlyTraffic ¶ added in v0.2.4
func IF ¶ added in v0.1.12
func IF(condition bool, a, b interface{}) interface{}
IF 三元表达式 use: IF(a>b, a, b).(int)
func InfoFTimes ¶ added in v0.4.7
InfoFTimes 日志-信息, 指定日志代码位置的定位调用层级
func InfoTimes ¶ added in v0.2.8
func InfoTimes(times int, args ...interface{})
InfoTimes 日志-信息, 指定日志代码位置的定位调用层级
func IsChineseMoney ¶ added in v0.2.4
IsChineseMoney 验证是否是中文钱大写
func IsChineseN ¶ added in v0.2.4
IsChineseN 验证是否含有number个汉字
func IsChineseNumber ¶ added in v0.2.4
IsChineseNumber 验证是否全是汉字数字
func IsContainStr ¶ added in v0.1.4
IsContainStr 字符串是否等于items中的某个元素
func IsElementStr ¶ added in v0.1.9
IsElementStr 判断字符串是否与数组里的某个字符串相同
func IsFloat2Len ¶ added in v0.2.4
IsFloat2Len 验证是否含有带不超过len个小数的小数
func IsIP ¶ added in v0.2.4
IsIP IP地址：((?:(?:25[0-5]|2[0-4]\\d|[01]?\\d?\\d)\\.){3}(?:25[0-5]|2[0-4]\\d|[01]?\\d?\\d))
func IsLandline ¶ added in v0.2.4
IsLandline 验证电话号码("XXX-XXXXXXX"、"XXXX-XXXXXXXX"、"XXX-XXXXXXX"、"XXX-XXXXXXXX"、"XXXXXXX"和"XXXXXXXX)：
func IsLeastCapital ¶ added in v0.2.4
IsLeastCapital 验证不低于n个大写字母
func IsLeastLower ¶ added in v0.2.4
IsLeastLower 验证不低于n个小写字母
func IsLeastNumber ¶ added in v0.2.4
IsLeastNumber 验证不低于n个数字
func IsLeastSpecial ¶ added in v0.2.4
IsLeastSpecial 验证不低于n特殊字符
func IsNumber2Heard ¶ added in v0.2.4
IsNumber2Heard 验证是否含有n开头的number
func IsNumber2Len ¶ added in v0.2.4
IsNumber2Len 验证是否含有连续长度不超过长度l的number
func IsTodayList ¶ added in v0.4.2
IsTodayList 列表页的时间显示 "01-02 15:04"
func JsonFind ¶ added in v0.3.8
JsonFind 按路径寻找指定json值 用法参考 ./_examples/json/main.go @find : 寻找路径，与目录的url类似， 下面是一个例子： json: {a:[{b:1},{b:2}]} find=/a/[0] => {b:1} find=a/[0]/b => 1
func JsonFind2Arr ¶ added in v0.3.8
JsonFind2Arr 寻找json,输出 []interface{}
func JsonFind2Int ¶ added in v0.4.5
JsonFind2Int 寻找json,输出int
func JsonFind2Int64 ¶ added in v0.4.5
JsonFind2Int64 寻找json,输出int64
func JsonFind2Json ¶ added in v0.3.8
JsonFind2Json 寻找json,输出 json格式字符串
func JsonFind2Map ¶ added in v0.3.8
JsonFind2Map 寻找json,输出 map[string]interface{}
func JsonFind2Str ¶ added in v0.4.5
JsonFind2Str 寻找json,输出字符串
func JwtDecrypt ¶ added in v0.2.7
JwtDecrypt jwt decrypt
func JwtEncrypt ¶ added in v0.2.7
JwtEncrypt jwt Encrypt
func JwtEncrypt256 ¶ added in v0.2.7
func JwtEncrypt384 ¶ added in v0.2.7
func JwtEncrypt512 ¶ added in v0.2.7
func LinuxSendCommand ¶ added in v0.4.1
LinuxSendCommand Linux Send Command Linux执行命令
func Map2Slice ¶ added in v0.2.8
func Map2Slice(data interface{}) []interface{}
Map2Slice Eg: {"K1": "v1", "K2": "v2"} => ["K1", "v1", "K2", "v2"]
func MapMergeCopy ¶ added in v0.2.8
func MapStr2Any ¶ added in v0.2.4
MapStr2Any map[string]string -> map[string]interface{}
func NewMysqlDB ¶ added in v0.0.11
NewMysqlDB 给mysql对象进行连接
func OutJsonFile ¶ added in v0.4.1
OutJsonFile 将data输出到json文件
func PanicToError ¶ added in v0.1.3
func PanicToError(fn func()) (err error)
PanicToError panic -> error
func ReadCsvFile ¶ added in v0.1.4
ReadCsvFile csv file -> [][]string 行列
func RedisDELKeys ¶ added in v0.1.2
RedisDELKeys Del key 使用常见： 并发删除大量key
func RegDNSName ¶ added in v0.3.4
RegDNSName 提取dns
func RegDelHtmlA ¶ added in v0.2.7
func RegDelHtmlButton ¶ added in v0.2.7
func RegDelHtmlCanvas ¶ added in v0.2.7
func RegDelHtmlCode ¶ added in v0.2.7
func RegDelHtmlH ¶ added in v0.2.7
func RegDelHtmlHref ¶ added in v0.2.7
func RegDelHtmlImg ¶ added in v0.2.7
func RegDelHtmlInput ¶ added in v0.2.7
func RegDelHtmlLi ¶ added in v0.2.7
func RegDelHtmlMeta ¶ added in v0.2.7
func RegDelHtmlP ¶ added in v0.2.7
func RegDelHtmlSelect ¶ added in v0.2.7
func RegDelHtmlSpan ¶ added in v0.2.7
func RegDelHtmlSrc ¶ added in v0.2.7
func RegDelHtmlTable ¶ added in v0.2.7
func RegDelHtmlTbody ¶ added in v0.2.7
func RegDelHtmlTd ¶ added in v0.2.7
func RegDelHtmlTitle ¶ added in v0.2.7
func RegDelHtmlTr ¶ added in v0.2.7
func RegDelHtmlUl ¶ added in v0.2.7
func RegDelHtmlVideo ¶ added in v0.2.7
func RegFindAll ¶ added in v0.1.1
func RegFindAllTxt ¶ added in v0.3.7
func RegFullURL ¶ added in v0.3.4
RegFullURL 提取url
func RegFullWidth ¶ added in v0.3.4
RegFullWidth 提取全角字符
func RegHalfWidth ¶ added in v0.3.4
RegHalfWidth 提取半角字符
func RegHtmlATxt ¶ added in v0.1.4
func RegHtmlButton ¶ added in v0.2.1
func RegHtmlButtonTxt ¶ added in v0.2.1
func RegHtmlCanvas ¶ added in v0.2.1
func RegHtmlCode ¶ added in v0.2.1
func RegHtmlCodeTxt ¶ added in v0.2.1
func RegHtmlDescription ¶ added in v0.4.5
func RegHtmlDescriptionTxt ¶ added in v0.4.5
func RegHtmlDiv ¶ added in v0.3.7
func RegHtmlDivTxt ¶ added in v0.3.7
func RegHtmlHTxt ¶ added in v0.1.4
func RegHtmlHref ¶ added in v0.1.4
func RegHtmlHrefTxt ¶ added in v0.1.4
func RegHtmlImg ¶ added in v0.2.1
func RegHtmlInput ¶ added in v0.1.1
func RegHtmlInputTxt ¶ added in v0.1.4
func RegHtmlKeyword ¶ added in v0.4.5
func RegHtmlKeywordTxt ¶ added in v0.4.5
func RegHtmlLiTxt ¶ added in v0.2.1
func RegHtmlMeta ¶ added in v0.2.1
func RegHtmlOption ¶ added in v0.3.9
func RegHtmlOptionTxt ¶ added in v0.3.9
func RegHtmlPTxt ¶ added in v0.1.4
func RegHtmlSelect ¶ added in v0.2.1
func RegHtmlSelectTxt ¶ added in v0.2.1
func RegHtmlSpan ¶ added in v0.1.1
func RegHtmlSpanTxt ¶ added in v0.1.4
func RegHtmlSrc ¶ added in v0.1.4
func RegHtmlSrcTxt ¶ added in v0.1.4
func RegHtmlTable ¶ added in v0.2.1
func RegHtmlTableOlny ¶ added in v0.3.5
func RegHtmlTableTxt ¶ added in v0.2.1
func RegHtmlTbody ¶ added in v0.1.9
func RegHtmlTdTxt ¶ added in v0.1.4
func RegHtmlTitle ¶ added in v0.1.1
func RegHtmlTitleTxt ¶ added in v0.1.4
func RegHtmlTrTxt ¶ added in v0.1.4
func RegHtmlUlTxt ¶ added in v0.2.1
func RegHtmlVideo ¶ added in v0.2.1
func RegLatitude ¶ added in v0.3.4
RegLatitude 提取纬度
func RegLongitude ¶ added in v0.3.4
RegLongitude 提取经度
func RegMACAddress ¶ added in v0.2.4
RegMACAddress 提取MACAddress
func RegRGBColor ¶ added in v0.3.4
RegRGBColor 提取RGB值
func RegSHA1Hex ¶ added in v0.2.4
RegSHA1Hex 提取sha1
func RegSHA256Hex ¶ added in v0.2.4
RegSHA256Hex 提取sha256
func RegURLPath ¶ added in v0.3.4
RegURLPath 提取url path
func RegURLPort ¶ added in v0.3.4
RegURLPort 提取url port
func RegURLSchema ¶ added in v0.3.4
RegURLSchema 提取url schema
func RegURLSubdomain ¶ added in v0.3.4
RegURLSubdomain 提取 url sub domain
func RegURLUsername ¶ added in v0.3.4
RegURLUsername 提取url username
func RegUnixPath ¶ added in v0.3.4
RegUnixPath 提取 unix路径
func RegWinPath ¶ added in v0.3.4
RegWinPath 提取 windows路径
func ReplaceAllToOne ¶ added in v0.2.1
ReplaceAllToOne 批量统一替换字符串
func SSHClient ¶ added in v0.1.1
SSHClient 连接ssh addr : 主机地址, 如: 127.0.0.1:22 user : 用户 pass : 密码 返回 ssh连接
func SearchBytesIndex ¶ added in v0.1.18
SearchBytesIndex []byte 字节切片 循环查找
func SearchPort ¶ added in v0.1.4
func SearchPort(ipStr string, vs ...interface{})
SearchPort 扫描ip的端口
func SetAgent ¶ added in v0.1.9
func SetAgent(agentType UserAgentType, agent string)
func SetStatusCodeFailEvent ¶ added in v0.1.5
func SetStatusCodeFailEvent(code int)
SetStatusCodeFailEvent 将指定状态码设置为执行失败事件
func SetStatusCodeRetryEvent ¶ added in v0.1.5
func SetStatusCodeRetryEvent(code int)
SetStatusCodeRetryEvent 将指定状态码设置为执行重试事件
func SetStatusCodeSuccessEvent ¶ added in v0.1.5
func SetStatusCodeSuccessEvent(code int)
SetStatusCodeSuccessEvent 将指定状态码设置为执行成功事件
func Slice2Map ¶ added in v0.2.8
func Slice2Map(slice interface{}) map[string]interface{}
Slice2Map ["K1", "v1", "K2", "v2"] => {"K1": "v1", "K2": "v2"} ["K1", "v1", "K2"] => nil
func SliceTool ¶ added in v0.2.1
func SliceTool() *sliceTool
SliceTool use : SliceTool().CopyInt64(a)
func StartJobGet ¶ added in v0.0.7
StartJobGet 并发执行Get,直到队列任务为空 jobNumber 并发数， queue 全局队列，
func StartJobPost ¶ added in v0.0.7
StartJobPost 开始运行并发Post
func Str2Float32 ¶ added in v0.2.9
Str2Float32 string -> float32
func Str2Float64 ¶ added in v0.1.1
Str2Float64 string -> float64
func StrDeleteSpace ¶ added in v0.0.11
StrDeleteSpace 删除字符串前后的空格
func StrDuplicates ¶ added in v0.2.4
StrDuplicates 数组，切片去重和去空串
func StringValueMysql ¶ added in v0.1.12
func StringValueMysql(i interface{}) string
StringValueMysql 用于mysql字符拼接使用
func Struct2Map ¶ added in v0.1.3
func Struct2Map(obj interface{}) map[string]interface{}
Struct2Map struct -> map[string]interface{}
func Struct2MapV2 ¶ added in v0.3.7
Struct2MapV2 Struct -> map hasValue=true表示字段值不管是否存在都转换成map hasValue=false表示字段为空或者不为0则转换成map
func Struct2MapV3 ¶ added in v0.3.7
func Struct2MapV3(obj interface{}) map[string]interface{}
Struct2MapV3 struct -> map
func Timestamp2Date ¶ added in v0.3.7
func Timestamp2Week ¶ added in v0.4.2
func Timestamp2WeekXinQi ¶ added in v0.4.2
func ToHZGB2312 ¶ added in v0.2.8
ToHZGB2312 to hzgb2312
func UnescapeUnicode ¶ added in v0.1.4
UnescapeUnicode Unicode 转码
func UnicodeDec ¶ added in v0.2.4
func UnicodeDecByte ¶ added in v0.2.4
func WarnFTimes ¶ added in v0.4.7
WarnFTimes 日志-警告, 指定日志代码位置的定位调用层级
func WarnTimes ¶ added in v0.3.7
func WarnTimes(times int, args ...interface{})
WarnTimes 日志-警告, 指定日志代码位置的定位调用层级
func WindowsSendCommand ¶ added in v0.4.1
WindowsSendCommand Windows Send Command
func WindowsSendPipe ¶ added in v0.4.7
WindowsSendPipe TODO 执行windows 管道命令
Types ¶
type AES ¶ added in v0.1.18
type AES interface {
Encrypt(str, key []byte) ([]byte, error)
Decrypt(str, key []byte) ([]byte, error)
}
AES AES interface
type Bar ¶ added in v0.2.9
type Bar struct {
// contains filtered or unexported fields
}
Bar 终端显示的进度条
func (*Bar) NewOptionWithGraph ¶ added in v0.2.9
type Context ¶ added in v0.0.7
type Context struct {
// Token
Token string
// http client
Client *http.Client
// http Request
Req *http.Request
// http Response
Resp *http.Response
// Error
Err error
// Ctx context.Context
Ctx context.Context
// 最大允许重试次数
MaxTimes RetryTimes
// 请求成功了需要处理的事件
SucceedFunc SucceedFunc
// 请求失败了需要处理的事件
FailedFunc FailedFunc
// 重试请求处理的事件，可以是更换代理，设置等待时间
RetryFunc RetryFunc
// 请求开始前处理的事件
StartFunc StartFunc
// 请求完成后处理的事件
EndFunc EndFunc
// 本次请求的任务
// 用于有步骤的请求和并发执行请求
Task *Task
// 请求返回的结果
RespBody []byte
// job编号
// 在执行多并发执行抓取任务，每个并发都有一个编号
// 这个编号是递增分配的
JobNumber int
// 请求的响应时间 单位ms
Ms time.Duration
// 是否显示日志, 默认是显示的
IsLog IsLog
// 输出字符串
Text string
// 输出Json
Json string
// 输出xml
Xml string
// 输出HTML
Html string
// 请求上下文参数
Param map[string]interface{}
StateCode int
// contains filtered or unexported fields
}
Context 请求上下文
func NewOptions ¶ added in v0.1.9
NewOptions 定义一个options请求对象，请求需执行 XX.Do()
func NewPostForm ¶ added in v0.2.7
NewPostForm 定义一个post请求对象，请求参数为from data，请求需执行 XX.Do()
func NewRequest ¶ added in v0.1.9
NewRequest 请求
func Request ¶ added in v0.0.7
func Request(url, method string, data []byte, contentType string, vs ...interface{}) (*Context, error)
Request 请求
func (*Context) CheckReqMd5 ¶ added in v0.1.9
CheckReqMd5 本次请求的md5值， url+reqBodyBytes+method
func (*Context) CookieNext ¶ added in v0.1.1
CookieNext 使用上下文cookies
func (*Context) GetRespHeader ¶ added in v0.2.9
GetRespHeader 获取Response的Header
func (*Context) GetRetryTimes ¶ added in v0.3.4
GetRetryTimes 获取当前重试最大次数
func (*Context) OpenErr2Retry ¶ added in v0.1.6
func (c *Context) OpenErr2Retry()
OpenErr2Retry 开启请求失败都执行retry
func (*Context) RespBodyArr ¶ added in v0.1.12
func (c *Context) RespBodyArr() []interface{}
RespBodyArr 请求到的Body转换为[]interface{}类型
func (*Context) RespBodyHtml ¶ added in v0.2.8
RespBodyHtml 请求到的Body转换为string类型的html格式
func (*Context) RespBodyMap ¶ added in v0.1.12
RespBodyMap 请求到的Body转换为map[string]interface{}类型
func (*Context) RespBodyString ¶ added in v0.1.9
RespBodyString 请求到的Body转换为string类型
func (*Context) RespContentLength ¶ added in v0.2.9
RespContentLength 获取Response ContentLength的值
func (*Context) SetFailedFunc ¶ added in v0.0.7
SetFailedFunc 设置错误后执行的方法
func (*Context) SetProxyFunc ¶ added in v0.1.14
SetProxyFunc 给请求设置代理函数 f func() *http.Transport
func (*Context) SetProxyPool ¶ added in v0.3.1
SetProxyPool 给请求设置代理池
func (*Context) SetRetryFunc ¶ added in v0.0.7
SetRetryFunc 重试请求前执行的方法
func (*Context) SetRetryTimes ¶ added in v0.0.7
SetRetryTimes 设置重试最大次数，如果超过这个次数还没有请求成功，请求结束，返回一个错误；
func (*Context) SetSleepRand ¶ added in v0.3.4
SetSleepRand 设置延迟随机时间，单位秒
func (*Context) SetSucceedFunc ¶ added in v0.0.7
SetSucceedFunc 设置成功后执行的方法
type DES ¶ added in v0.1.18
type DES interface {
Encrypt(str, key []byte) ([]byte, error)
Decrypt(str, key []byte) ([]byte, error)
}
DES DES interface
type DNSInfo ¶ added in v0.4.7
type DNSInfo struct {
IPs []string `json:"ips"`
LookupCNAME string `json:"cname"`
DnsServerIP string `json:"dnsServerIP"`
DnsServerName string `json:"dnsServerName"`
IsCDN bool `json:"isCDN"`
Ms float64 `json:"ms"`
}
func NsLookUpAll ¶ added in v0.4.7
NsLookUpAll 在多个DNS服务器查询
func NsLookUpFromDNSServer ¶ added in v0.4.7
NsLookUpFromDNSServer 指定DNS服务器查询
type GDMapApi ¶ added in v0.2.4
type GDMapApi interface {
Add(key string, value interface{}) *gDMap
Get(key string) interface{}
Del(key string) *gDMap
Len() int
KeyList() []string
AddMap(data map[string]interface{}) *gDMap
Range(f func(k string, v interface{})) *gDMap
RangeAt(f func(id int, k string, v interface{})) *gDMap
CheckValue(value interface{}) bool // 检查是否存在某个值
Reverse() //反序
}
GDMapApi 固定顺序 Map 接口
type HostPageSpeedCheck ¶ added in v0.4.3
type HostPageSpeedCheck struct {
Host string
Depth int // 页面深度
PageSpeed map[string]time.Duration
UrlSet map[string]struct{} // 检查页面重复
}
HostPageSpeedCheck ================================================================================================= Host站点下 HTML Get 测速 应用函数
func NewHostPageSpeedCheck ¶ added in v0.4.3
func NewHostPageSpeedCheck(host string, depth int) *HostPageSpeedCheck
func (*HostPageSpeedCheck) AverageSpeed ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) AverageSpeed() float64
AverageSpeed 平均时间
func (*HostPageSpeedCheck) MaxSpeed ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) MaxSpeed() int64
MaxSpeed 最高用时
func (*HostPageSpeedCheck) MinSpeed ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) MinSpeed() int64
MinSpeed 最低用时
func (*HostPageSpeedCheck) Report ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) Report() map[string]string
Report 报告
func (*HostPageSpeedCheck) Result ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) Result() map[string]time.Duration
func (*HostPageSpeedCheck) Run ¶ added in v0.4.3
func (scan *HostPageSpeedCheck) Run() ([]string, int)
Run int 单位 ms
type HostScanBadLink ¶ added in v0.4.3
type HostScanBadLink struct {
Host string
Depth int // 页面深度
PageState map[string]int
UrlSet map[string]struct{} // 检查页面重复
}
HostScanBadLink ==================================================================================================== Host站点下 HTML Get Url 死链接扫描 应用函数
func NewHostScanBadLink ¶ added in v0.4.3
func NewHostScanBadLink(host string, depth int) *HostScanBadLink
func (*HostScanBadLink) Report ¶ added in v0.4.3
func (scan *HostScanBadLink) Report() map[string]int
func (*HostScanBadLink) Result ¶ added in v0.4.3
func (scan *HostScanBadLink) Result() map[string]int
func (*HostScanBadLink) Run ¶ added in v0.4.3
func (scan *HostScanBadLink) Run() ([]string, int)
type HostScanExtLinks ¶ added in v0.4.3
type HostScanExtLinks struct {
Host string
}
HostScanExtLinks =================================================================================================== Host站点下的外链采集 应用函数
func NewHostScanExtLinks ¶ added in v0.4.3
func NewHostScanExtLinks(host string) *HostScanExtLinks
func (*HostScanExtLinks) Run ¶ added in v0.4.3
func (scan *HostScanExtLinks) Run() ([]string, int)
type HostScanUrl ¶ added in v0.4.3
type HostScanUrl struct {
Host string
Depth int // 页面深度
UrlSet map[string]struct{}
Count int
MaxCount int64
}
HostScanUrl ======================================================================================================== Host站点下 A标签 Url扫描， 从更目录开始扫描指定深度 get Url 应用函数
func NewHostScanUrl ¶ added in v0.4.3
func NewHostScanUrl(host string, depth int) *HostScanUrl
func (*HostScanUrl) Run ¶ added in v0.4.3
func (scan *HostScanUrl) Run() ([]string, int)
type HostToolEr ¶ added in v0.4.3
HostToolEr TODO HostToolEr
type HttpPackage ¶ added in v0.3.4
HttpPackage 代理服务抓取到的HTTP的包
func (*HttpPackage) Img2Base64 ¶ added in v0.3.4
func (pack *HttpPackage) Img2Base64() string
Img2Base64 如果数据类型是image 就转换成base64的图片输出
func (*HttpPackage) SaveImage ¶ added in v0.3.4
func (pack *HttpPackage) SaveImage(path string) error
SaveImage 如果数据类型是image 就保存图片
func (*HttpPackage) ToFile ¶ added in v0.3.5
func (pack *HttpPackage) ToFile(path string) error
ToFile 抓取到的数据类型保存到文件
type IdWorker ¶ added in v0.4.1
type IdWorker struct {
// contains filtered or unexported fields
}
func (*IdWorker) InitIdWorker ¶ added in v0.4.1
type Intercept ¶ added in v0.3.4
type Intercept struct {
Ip string
HttpPackageFunc func(pack *HttpPackage)
}
Intercept http/s 代理与抓包
func (*Intercept) RunHttpIntercept ¶ added in v0.3.4
func (ipt *Intercept) RunHttpIntercept()
RunHttpIntercept 启动 http/s 代理与抓包服务
type MQKafkaService ¶ added in v0.4.2
type MQKafkaService struct {
Server []string
}
MQKafkaService Kafka消息队列
func (*MQKafkaService) Consumer ¶ added in v0.4.2
func (m *MQKafkaService) Consumer(topic string) []byte
func (*MQKafkaService) Producer ¶ added in v0.4.2
func (m *MQKafkaService) Producer(topic string, data []byte)
type MQNsqService ¶ added in v0.4.2
MQNsqService NSQ消息队列
func (*MQNsqService) Consumer ¶ added in v0.4.2
func (m *MQNsqService) Consumer(topic string) []byte
Consumer 消费者
func (*MQNsqService) Producer ¶ added in v0.4.2
func (m *MQNsqService) Producer(topic string, data []byte)
Producer 生产者
type MQRabbitService ¶ added in v0.4.2
type MQRabbitService struct {
AmqpUrl string
}
MQRabbitService Rabbit消息队列
func (*MQRabbitService) Consumer ¶ added in v0.4.2
func (m *MQRabbitService) Consumer(topic string) []byte
func (*MQRabbitService) Producer ¶ added in v0.4.2
func (m *MQRabbitService) Producer(topic string, data []byte)
type MQer ¶ added in v0.4.2
MQer 消息队列接口
type Mail ¶ added in v0.4.5
type Mongo ¶ added in v0.1.4
type Mongo struct {
User string
Password string
Host string
Port string
Conn *mongo.Client
Database *mongo.Database
Collection *mongo.Collection
MaxPoolSize int
TimeOut time.Duration
}
func (*Mongo) GetCollection ¶ added in v0.1.4
GetCollection 连接mongodb 的db的集合 dbname:DB名; name:集合名
type Mysql ¶ added in v0.0.9
Mysql 客户端对象
func GetMysqlDBConn ¶ added in v0.1.3
GetMysqlDBConn 获取mysql 连接
func (*Mysql) DeleteTable ¶ added in v0.2.9
DeleteTable 删除表
func (*Mysql) GetFieldList ¶ added in v0.3.4
GetFieldList 获取表字段
func (*Mysql) InsertAtGd ¶ added in v0.2.7
InsertAtGd 固定顺序map写入
func (*Mysql) InsertAtJson ¶ added in v0.3.4
InsertAtJson json字符串存入数据库
func (*Mysql) IsHaveTable ¶ added in v0.1.17
IsHaveTable 表是否存在
func (*Mysql) NewTableGd ¶ added in v0.2.7
NewTableGd 创建新的固定map顺序为字段的表
func (*Mysql) SetMaxIdleConn ¶ added in v0.1.11
SetMaxIdleConn 最大idle 数
func (*Mysql) SetMaxOpenConn ¶ added in v0.1.11
SetMaxOpenConn 最大连接数
type ProxyPool ¶ added in v0.3.1
type ProxyPool struct {
// contains filtered or unexported fields
}
ProxyPool 代理池
type Queue ¶ added in v0.0.4
type Queue struct {
// contains filtered or unexported fields
}
Queue 队列
type RabbitMQ ¶ added in v0.4.2
type RabbitMQ struct {
QueueName string
Exchange string //交换机
Key string //key Simple模式 几乎用不到
MqUrl string //连接信息
// contains filtered or unexported fields
}
func NewRabbitMQ ¶ added in v0.4.2
NewRabbitMQ 创建RabbitMQ结构体实例
func NewRabbitMQPubSub ¶ added in v0.4.2
NewRabbitMQPubSub 订阅模式创建 rabbitMq实例 (目前用的fanout模式)
func NewRabbitMQRouting ¶ added in v0.4.2
NewRabbitMQRouting 路由模式 创建RabbitMQ实例
func NewRabbitMQSimple ¶ added in v0.4.2
NewRabbitMQSimple 简单模式step：1。创建简单模式下RabbitMQ实例
func NewRabbitMQTopic ¶ added in v0.4.2
NewRabbitMQTopic 话题模式 创建RabbitMQ实例
func (*RabbitMQ) Destroy ¶ added in v0.4.2
func (r *RabbitMQ) Destroy()
Destroy 断开channel和connection
func (*RabbitMQ) PublishPub ¶ added in v0.4.2
PublishPub 订阅模式生成
func (*RabbitMQ) PublishRouting ¶ added in v0.4.2
PublishRouting 路由模式发送信息
func (*RabbitMQ) PublishSimple ¶ added in v0.4.2
PublishSimple 简单模式Step:2、简单模式下生产代码
func (*RabbitMQ) PublishTopic ¶ added in v0.4.2
PublishTopic 话题模式发送信息
func (*RabbitMQ) RegistryConsumeSimple ¶ added in v0.4.2
RegistryConsumeSimple 简单模式注册消费者
func (*RabbitMQ) RegistryReceiveRouting ¶ added in v0.4.2
RegistryReceiveRouting 路由模式接收信息
func (*RabbitMQ) RegistryReceiveSub ¶ added in v0.4.2
RegistryReceiveSub 订阅模式消费端代码
func (*RabbitMQ) RegistryReceiveTopic ¶ added in v0.4.2
RegistryReceiveTopic 话题模式接收信息 要注意key 其中* 用于匹配一个单词，#用于匹配多个单词（可以是零个） 匹配 xx.* 表示匹配xx.hello,但是xx.hello.one需要用xx.#才能匹配到
type Rds ¶ added in v0.1.2
type Rds struct {
SSHUser string
SSHPassword string
SSHAddr string
RedisHost string
RedisPost string
RedisPassword string
// redis DB
RedisDB int
// 单个连接
Conn redis.Conn
// 最大闲置数，用于redis连接池
RedisMaxIdle int
// 最大连接数
RedisMaxActive int
// 单条连接Timeout
RedisIdleTimeoutSec int
// 连接池
Pool *redis.Pool
}
Rds Redis客户端
func NewRedisPool ¶ added in v0.1.2
func NewRedisPool(host, port, password string, db, maxIdle, maxActive, idleTimeoutSec int, vs ...interface{}) *Rds
NewRedisPool 新建Redis连接池对象
func (*Rds) Append ¶ added in v0.4.2
Append APPEND key value 如果 key 已经存在并且是一个字符串， APPEND 命令将 value 追加到 key 原来的值的末尾。 如果 key 不存在， APPEND 就简单地将给定 key 设为 value ，就像执行 SET key value 一样。
func (*Rds) Decr ¶ added in v0.4.2
Decr key 将 key 中储存的数字值减一。 如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 DECR 操作。 如果值包含错误的类型，或字符串类型的值不能表示为数字，那么返回一个错误。
func (*Rds) GetAllKeys ¶ added in v0.4.2
GetAllKeys 获取所有的key
func (*Rds) GetBit ¶ added in v0.4.3
GetBit GETBIT key offset 对 key 所储存的字符串值，获取指定偏移量上的位(bit)。 当 offset 比字符串值的长度大，或者 key 不存在时，返回 0 。
func (*Rds) GetRange ¶ added in v0.4.2
GetRange GETRANGE key start end 返回 key 中字符串值的子字符串，字符串的截取范围由 start 和 end 两个偏移量决定(包括 start 和 end 在内)。
func (*Rds) GetSet ¶ added in v0.4.2
GetSet GETSET key value 将给定 key 的值设为 value ，并返回 key 的旧值(old value)。 当 key 存在但不是字符串类型时，返回一个错误。
func (*Rds) HGetAllInt ¶ added in v0.4.2
HGetAllInt HGETALL 获取Hash value
func (*Rds) HGetAllInt64 ¶ added in v0.4.2
HGetAllInt64 HGETALL 获取Hash value
func (*Rds) HIncrBy ¶ added in v0.4.2
HIncrBy HINCRBY key field increment 为哈希表 key 中的域 field 的值加上增量 increment 。 增量也可以为负数，相当于对给定域进行减法操作。 如果 key 不存在，一个新的哈希表被创建并执行 HINCRBY 命令。 如果域 field 不存在，那么在执行命令前，域的值被初始化为 0
func (*Rds) HIncrByFloat ¶ added in v0.4.2
HIncrByFloat HINCRBYFLOAT key field increment 为哈希表 key 中的域 field 加上浮点数增量 increment 。 如果哈希表中没有域 field ，那么 HINCRBYFLOAT 会先将域 field 的值设为 0 ，然后再执行加法操作。 如果键 key 不存在，那么 HINCRBYFLOAT 会先创建一个哈希表，再创建域 field ，最后再执行加法操作。
func (*Rds) HMGet ¶ added in v0.4.2
HMGet HMGET key field [field ...] 返回哈希表 key 中，一个或多个给定域的值。 如果给定的域不存在于哈希表，那么返回一个 nil 值。
func (*Rds) HMSet ¶ added in v0.4.2
HMSet HMSET 新建Hash 多个field HMSET key field value [field value ...] 同时将多个 field-value (域-值)对设置到哈希表 key 中。 此命令会覆盖哈希表中已存在的域。
func (*Rds) HSet ¶ added in v0.4.2
HSet HSET 新建Hash 单个field 如果 key 不存在，一个新的哈希表被创建并进行 HSET 操作。 如果域 field 已经存在于哈希表中，旧值将被覆盖。
func (*Rds) HSetNx ¶ added in v0.4.2
HSetNx HSETNX key field value 给hash追加field value 将哈希表 key 中的域 field 的值设置为 value ，当且仅当域 field 不存在。
func (*Rds) Incr ¶ added in v0.4.2
Incr INCR key 将 key 中储存的数字值增一。 如果 key 不存在，那么 key 的值会先被初始化为 0 ，然后再执行 INCR 操作。 如果值包含错误的类型，或字符串类型的值不能表示为数字，那么返回一个错误。
func (*Rds) IncrByFloat ¶ added in v0.4.2
IncrByFloat INCRBYFLOAT key increment 为 key 中所储存的值加上浮点数增量 increment 。
func (*Rds) LInsert ¶ added in v0.4.2
LInsert LINSERT key BEFORE|AFTER pivot value 将值 value 插入到列表 key 当中，位于值 pivot 之前或之后。 当 pivot 不存在于列表 key 时，不执行任何操作。 当 key 不存在时， key 被视为空列表，不执行任何操作。 如果 key 不是列表类型，返回一个错误。 direction : 方向 bool true:BEFORE(前) false: AFTER(后)
func (*Rds) LPusHx ¶ added in v0.4.2
LPusHx LPUSHX key value 将值 value 插入到列表 key 的表头，当且仅当 key 存在并且是一个列表。 和 LPUSH 命令相反，当 key 不存在时， LPUSHX 命令什么也不做。
func (*Rds) LRangeST ¶ added in v0.4.2
LRangeST LRANGE key start stop 返回列表 key 中指定区间内的元素，区间以偏移量 start 和 stop 指定。
func (*Rds) LRem ¶ added in v0.4.2
LRem LREM key count value 根据参数 count 的值，移除列表中与参数 value 相等的元素。 count 的值可以是以下几种： count > 0 : 从表头开始向表尾搜索，移除与 value 相等的元素，数量为 count 。 count < 0 : 从表尾开始向表头搜索，移除与 value 相等的元素，数量为 count 的绝对值。 count = 0 : 移除表中所有与 value 相等的值。
func (*Rds) LSet ¶ added in v0.4.2
LSet LSET key index value 将列表 key 下标为 index 的元素的值设置为 value 。 当 index 参数超出范围，或对一个空列表( key 不存在)进行 LSET 时，返回一个错误。
func (*Rds) LTrim ¶ added in v0.4.2
LTrim LTRIM key start stop 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除。 举个例子，执行命令 LTRIM list 0 2 ，表示只保留列表 list 的前三个元素，其余元素全部删除。
func (*Rds) MGet ¶ added in v0.4.2
MGet MGET key [key ...] 返回所有(一个或多个)给定 key 的值。 如果给定的 key 里面，有某个 key 不存在，那么这个 key 返回特殊值 nil 。因此，该命令永不失败。
func (*Rds) MSet ¶ added in v0.4.2
MSet MSET key value [key value ...] 同时设置一个或多个 key-value 对。 如果某个给定 key 已经存在，那么 MSET 会用新值覆盖原来的旧值，如果这不是你所希望的效果， 请考虑使用 MSETNX 命令：它只会在所有给定 key 都不存在的情况下进行设置操作。 MSET 是一个原子性(atomic)操作，所有给定 key 都会在同一时间内被设置，某些给定 key 被更新而另一些给定 key 没有改变的情况，不可能发生。
func (*Rds) MSetNx ¶ added in v0.4.2
MSetNx MSETNX key value [key value ...] 同时设置一个或多个 key-value 对，当且仅当所有给定 key 都不存在。 即使只有一个给定 key 已存在， MSETNX 也会拒绝执行所有给定 key 的设置操作。 MSETNX 是原子性的，因此它可以用作设置多个不同 key 表示不同字段(field)的唯一性逻辑对象(unique logic object)， 所有字段要么全被设置，要么全不被设置。
func (*Rds) MqConsumer ¶ added in v0.4.2
MqConsumer Redis消息队列消费方
func (*Rds) MqProducer ¶ added in v0.4.2
MqProducer Redis消息队列生产方
func (*Rds) PSetEx ¶ added in v0.4.2
PSetEx PSETEX key milliseconds value 这个命令和 SETEX 命令相似，但它以毫秒为单位设置 key 的生存时间，而不是像 SETEX 命令那样，以秒为单位。
func (*Rds) RPopLPush ¶ added in v0.4.2
RPopLPush RPOPLPUSH source destination 命令 RPOPLPUSH 在一个原子时间内，执行以下两个动作： 将列表 source 中的最后一个元素(尾元素)弹出，并返回给客户端。 将 source 弹出的元素插入到列表 destination ，作为 destination 列表的的头元素。 举个例子，你有两个列表 source 和 destination ， source 列表有元素 a, b, c ， destination 列表有元素 x, y, z ，执行 RPOPLPUSH source destination 之后， source 列表包含元素 a, b ， destination 列表包含元素 c, x, y, z ，并且元素 c 会被返回给客户端。 如果 source 不存在，值 nil 被返回，并且不执行其他动作。 如果 source 和 destination 相同，则列表中的表尾元素被移动到表头，并返回该元素，可以把这种特殊情况视作列表的旋转(rotation)操作。
func (*Rds) RPush ¶ added in v0.4.2
RPush RPUSH key value [value ...] 将一个或多个值 value 插入到列表 key 的表尾(最右边)。 如果有多个 value 值，那么各个 value 值按从左到右的顺序依次插入到表尾：比如对一个空列表 mylist 执行 RPUSH mylist a b c ，得出的结果列表为 a b c ，等同于执行命令 RPUSH mylist a 、 RPUSH mylist b 、 RPUSH mylist c 。 新创建List 将一个或多个值 value 插入到列表 key 的表尾(最右边)。
func (*Rds) RPushX ¶ added in v0.4.2
RPushX RPUSHX key value 将值 value 插入到列表 key 的表尾，当且仅当 key 存在并且是一个列表。
func (*Rds) SAdd ¶ added in v0.4.2
SAdd SADD 新创建Set 将一个或多个 member 元素加入到集合 key 当中，已经存在于集合的 member 元素将被忽略。
func (*Rds) SDiff ¶ added in v0.4.2
SDiff SDIFF key [key ...] 返回一个集合的全部成员，该集合是所有给定集合之间的差集。 不存在的 key 被视为空集。
func (*Rds) SDiffStore ¶ added in v0.4.2
SDiffStore SDIFFSTORE destination key [key ...] 这个命令的作用和 SDIFF 类似，但它将结果保存到 destination 集合，而不是简单地返回结果集。 如果 destination 集合已经存在，则将其覆盖。 destination 可以是 key 本身。
func (*Rds) SInter ¶ added in v0.4.2
SInter SINTER key [key ...] 返回一个集合的全部成员，该集合是所有给定集合的交集。 不存在的 key 被视为空集。
func (*Rds) SInterStore ¶ added in v0.4.2
SInterStore SINTERSTORE destination key [key ...] 这个命令类似于 SINTER 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。 如果 destination 集合已经存在，则将其覆盖。 destination 可以是 key 本身。
func (*Rds) SIsMember ¶ added in v0.4.2
SIsMember SISMEMBER key member 判断 member 元素是否集合 key 的成员。 返回值: 如果 member 元素是集合的成员，返回 1 。 如果 member 元素不是集合的成员，或 key 不存在，返回 0 。
func (*Rds) SMemeRs ¶ added in v0.4.2
SMemeRs SMEMBERS key 返回集合 key 中的所有成员。 获取Set value 返回集合 key 中的所有成员。
func (*Rds) SMove ¶ added in v0.4.2
SMove SMOVE source destination member 将 member 元素从 source 集合移动到 destination 集合。 SMOVE 是原子性操作。 如果 source 集合不存在或不包含指定的 member 元素，则 SMOVE 命令不执行任何操作，仅返回 0 。否则， member 元素从 source 集合中被移除，并添加到 destination 集合中去。 当 destination 集合已经包含 member 元素时， SMOVE 命令只是简单地将 source 集合中的 member 元素删除。 当 source 或 destination 不是集合类型时，返回一个错误。 返回值: 成功移除，返回 1 。失败0
func (*Rds) SRandMember ¶ added in v0.4.2
SRandMember SRANDMEMBER key [count] 如果命令执行时，只提供了 key 参数，那么返回集合中的一个随机元素。 如果 count 为正数，且小于集合基数，那么命令返回一个包含 count 个元素的数组，数组中的元素各不相同。 如果 count 大于等于集合基数，那么返回整个集合。 如果 count 为负数，那么命令返回一个数组，数组中的元素可能会重复出现多次，而数组的长度为 count 的绝对值。
func (*Rds) SRem ¶ added in v0.4.2
SRem SREM key member [member ...] 移除集合 key 中的一个或多个 member 元素，不存在的 member 元素会被忽略。
func (*Rds) SUnionStore ¶ added in v0.4.2
SUnionStore SUNIONSTORE destination key [key ...] 这个命令类似于 SUNION 命令，但它将结果保存到 destination 集合，而不是简单地返回结果集。
func (*Rds) SearchKeys ¶ added in v0.4.2
SearchKeys 搜索key
func (*Rds) SetBit ¶ added in v0.4.3
SetBit SETBIT key offset value 对 key 所储存的字符串值，设置或清除指定偏移量上的位(bit)。 value : 位的设置或清除取决于 value 参数，可以是 0 也可以是 1 。 注意 offset 不能太大，越大key越大
func (*Rds) SetNx ¶ added in v0.4.2
SetNx key value 将 key 的值设为 value ，当且仅当 key 不存在。 若给定的 key 已经存在，则 SETNX 不做任何动作。
func (*Rds) SetRange ¶ added in v0.4.2
SetRange SETRANGE key offset value 用 value 参数覆写(overwrite)给定 key 所储存的字符串值，从偏移量 offset 开始。 不存在的 key 当作空白字符串处理。
func (*Rds) ToByteSlices ¶ added in v0.4.2
func (*Rds) ToFloat64s ¶ added in v0.4.2
func (*Rds) ToInt64Map ¶ added in v0.4.2
func (*Rds) ToStringMap ¶ added in v0.4.2
func (*Rds) ZCount ¶ added in v0.4.2
ZCount ZCOUNT key min max 返回有序集 key 中， score 值在 min 和 max 之间(默认包括 score 值等于 min 或 max )的成员的数量。
func (*Rds) ZIncrBy ¶ added in v0.4.2
ZIncrBy ZINCRBY key increment member 为有序集 key 的成员 member 的 score 值加上增量 increment 。 可以通过传递一个负数值 increment ，让 score 减去相应的值，比如 ZINCRBY key -5 member ，就是让 member 的 score 值减去 5 。 当 key 不存在，或 member 不是 key 的成员时， ZINCRBY key increment member 等同于 ZADD key increment member 。
func (*Rds) ZRangeByScore ¶ added in v0.4.2
ZRangeByScore ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count] 返回有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。有序集成员按 score 值递增(从小到大)次序排列。 具有相同 score 值的成员按字典序(lexicographical order)来排列(该属性是有序集提供的，不需要额外的计算)。 可选的 LIMIT 参数指定返回结果的数量及区间(就像SQL中的 SELECT LIMIT offset, count )，注意当 offset 很大时， 定位 offset 的操作可能需要遍历整个有序集，此过程最坏复杂度为 O(N) 时间。 可选的 WITHSCORES 参数决定结果集是单单返回有序集的成员，还是将有序集成员及其 score 值一起返回。 区间及无限 min 和 max 可以是 -inf 和 +inf ，这样一来，你就可以在不知道有序集的最低和最高 score 值的情况下，使用 ZRANGEBYSCORE 这类命令。 默认情况下，区间的取值使用闭区间 (小于等于或大于等于)，你也可以通过给参数前增加 ( 符号来使用可选的开区间 (小于或大于)。
func (*Rds) ZRangeByScoreAll ¶ added in v0.4.2
ZRangeByScoreAll 获取所有
func (*Rds) ZRangeST ¶ added in v0.4.2
ZRangeST ZRANGE key start stop [WITHSCORES] 返回有序集 key 中，指定区间内的成员。 其中成员的位置按 score 值递增(从小到大)来排序。
func (*Rds) ZRank ¶ added in v0.4.2
ZRank ZRANK key member 返回有序集 key 中成员 member 的排名。其中有序集成员按 score 值递增(从小到大)顺序排列。 排名以 0 为底，也就是说， score 值最小的成员排名为 0 。
func (*Rds) ZRem ¶ added in v0.4.2
ZRem ZREM key member [member ...] 移除有序集 key 中的一个或多个成员，不存在的成员将被忽略。
func (*Rds) ZRemRangeByRank ¶ added in v0.4.2
ZRemRangeByRank ZREMRANGEBYRANK key start stop 移除有序集 key 中，指定排名(rank)区间内的所有成员。 区间分别以下标参数 start 和 stop 指出，包含 start 和 stop 在内。
func (*Rds) ZRemRangeByScore ¶ added in v0.4.2
ZRemRangeByScore ZREMRANGEBYSCORE key min max 移除有序集 key 中，所有 score 值介于 min 和 max 之间(包括等于 min 或 max )的成员。
func (*Rds) ZRevRange ¶ added in v0.4.2
ZRevRange ZREVRANGE key start stop [WITHSCORES] 返回有序集 key 中，指定区间内的成员。 其中成员的位置按 score 值递减(从大到小)来排列。 具有相同 score 值的成员按字典序的逆序(reverse lexicographical order)排列。
func (*Rds) ZRevRangeByScore ¶ added in v0.4.2
ZRevRangeByScore key max min [WITHSCORES] [LIMIT offset count] 返回有序集 key 中， score 值介于 max 和 min 之间(默认包括等于 max 或 min )的所有的成员。有序集成员按 score 值递减(从大到小)的次序排列。 具有相同 score 值的成员按字典序的逆序(reverse lexicographical order )排列。
func (*Rds) ZRevRangeByScoreAll ¶ added in v0.4.2
ZRevRangeByScoreAll 获取所有
type ReqTimeOut ¶ added in v0.0.7
type ReqTimeOut int
type ReqTimeOutMs ¶ added in v0.0.7
type ReqTimeOutMs int
type RetryFunc ¶ added in v0.0.7
type RetryFunc func(c *Context)
RetryFunc 重试请求前执行的方法类型，是否重试来自上次请求的状态码来确定，见StatusCodeMap;
type SSHConnInfo ¶ added in v0.1.2
SSHConnInfo ssh连接通道
func NewSSHInfo ¶ added in v0.1.2
func NewSSHInfo(addr, user, password string) *SSHConnInfo
NewSSHInfo 新建ssh连接通道
type SSLCertificateInfo ¶ added in v0.4.7
type SSLCertificateInfo struct {
Url string `json:"Url"` // url
EffectiveTime string `json:"EffectiveTime"` // 有效时间
NotBefore int64 `json:"NotBefore"` // 起始
NotAfter int64 `json:"NotAfter"` // 结束
DNSName string `json:"DNSName"` // DNSName
OCSPServer string `json:"OCSPServer"` // OCSPServer
CRLDistributionPoints string `json:"CRLDistributionPoints"` // CRL分发点
Issuer string `json:"Issuer"` // 颁发者
IssuingCertificateURL string `json:"IssuingCertificateURL"` // 颁发证书URL
PublicKeyAlgorithm string `json:"PublicKeyAlgorithm"` // 公钥算法
Subject string `json:"Subject"` // 颁发对象
Version string `json:"Version"` // 版本
SignatureAlgorithm string `json:"SignatureAlgorithm"` // 证书算法
}
func GetCertificateInfo ¶ added in v0.4.7
func GetCertificateInfo(caseUrl string) (SSLCertificateInfo, bool)
GetCertificateInfo 获取SSL证书信息
func (SSLCertificateInfo) Echo ¶ added in v0.4.7
func (s SSLCertificateInfo) Echo()
func (SSLCertificateInfo) Expire ¶ added in v0.4.7
func (s SSLCertificateInfo) Expire() int64
type StressUrl ¶ added in v0.0.7
type StressUrl struct {
Url string
Method string
Sum int64
Total int
TQueue TodoQueue
// 接口传入的json
JsonData string
// 接口传入类型
ContentType string
// contains filtered or unexported fields
}
StressUrl 压力测试一个url
func NewTestUrl ¶ added in v0.0.7
NewTestUrl 实例化一个新的url压测
type TableInfo ¶ added in v0.0.9
type TableInfo struct {
Field string
Type string
Null string
Key string
Default interface{}
Extra string
}
TableInfo 表信息
type Task ¶ added in v0.0.5
type Task struct {
Url string
JsonParam string
HeaderMap *http.Header
Data map[string]interface{} // 上下文传递的数据
Urls []*ReqUrl // 多步骤使用
Type string // "", "upload", "do"
SavePath string
SaveDir string
FileName string
// contains filtered or unexported fields
}
Task 任务对象
func CrawlerTask ¶ added in v0.1.3
func (*Task) GetDataStr ¶ added in v0.1.12
func (*Task) SetJsonParam ¶ added in v0.3.6
type TcpClient ¶ added in v0.2.9
type TcpClient struct {
Connection *net.TCPConn
HawkServer *net.TCPAddr
StopChan chan struct{}
CmdChan chan string
Token string
RConn chan struct{}
}
TcpClient Tcp客户端
func NewTcpClient ¶ added in v0.2.9
func NewTcpClient() *TcpClient
type TodoQueue ¶ added in v0.0.4
type TodoQueue interface {
Add(task *Task) error // 向队列中添加元素
Poll() *Task // 移除队列中最前面的元素
Clear() bool // 清空队列
Size() int // 获取队列的元素个数
IsEmpty() bool // 判断队列是否为空
Print() // 打印
}
TodoQueue 任务队列
type UdpClient ¶ added in v0.2.9
UdpClient Udp客户端
func NewUdpClient ¶ added in v0.2.9
func NewUdpClient() *UdpClient
type UploadQueue ¶ added in v0.1.1
type UploadQueue struct {
// contains filtered or unexported fields
}
UploadQueue 下载队列
func (*UploadQueue) Clear ¶ added in v0.1.1
func (q *UploadQueue) Clear() bool
func (*UploadQueue) IsEmpty ¶ added in v0.1.1
func (q *UploadQueue) IsEmpty() bool
func (*UploadQueue) Print ¶ added in v0.1.1
func (q *UploadQueue) Print()
func (*UploadQueue) Size ¶ added in v0.1.1
func (q *UploadQueue) Size() int
type UserAgentType ¶ added in v0.0.4
type UserAgentType int
const ( PCAgent UserAgentType = iota + 1 WindowsAgent LinuxAgent MacAgent AndroidAgent IosAgent PhoneAgent WindowsPhoneAgent UCAgent )
Source Files
¶
- application.go
- conf.go
- context.go
- crypto.go
- csv.go
- gathertool.go
- gd_map.go
- html.go
- intercept_package_http.go
- job.go
- log.go
- mail.go
- mongo.go
- mq.go
- mysql.go
- redis.go
- reg.go
- req.go
- slice.go
- socket_proxy.go
- ssh.go
- status_code.go
- stress_testing.go
- string_helper.go
- tcp.go
- time_helper.go
- todo_queue.go
- udp.go
- websocket.go
Directories
¶
| Path | Synopsis |
|---|---|
|
_examples
|
|
|
baojia
url : http://www.100ppi.com/mprice/mlist-1--1.html 抓取商品报价
|
url : http://www.100ppi.com/mprice/mlist-1--1.html 抓取商品报价 |
|
qgyyzs
环球医药网 爬虫 http://data.qgyyzs.net
|
环球医药网 爬虫 http://data.qgyyzs.net |
|
qihuo
url: http://futures.100ppi.com/qhb/day-2022-07-05.html 抓取期货数据
|
url: http://futures.100ppi.com/qhb/day-2022-07-05.html 抓取期货数据 |
|
search
1.
|
1. |
Click to show internal directories.
Click to hide internal directories.