esme

package module
v0.0.7 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 3, 2022 License: MIT Imports: 24 Imported by: 0

README

esme

一个go实现的多任务、多线程的网络请求SDK

1. 特性
  • 基本的网络请求实现
  • 自定义请求参数
  • http 代理的使用
  • http 请求 debug 模式
  • http 请求的回调，包括成功回调、失败回调和重试回调
  • 支持自己设置 http.Transport
  • 内存中的任务队列实现
  • redis 任务队列实现
2. 安装
go get -u github.com/zituocn/esme
3. 简单的HTTP请求

一个 http get 请求的演示代码

demo code

package main

import (
	"fmt"

	"github.com/zituocn/esme"
)

func main() {

	// 请求一个天气预报的接口
	ctx := esme.HttpGet("https://tenapi.cn/wether/?city=%E6%88%90%E9%83%BD")

	// 成功的回调
	ctx.SetSucceedFunc(func(c *esme.Context) {
		fmt.Println("请求成功了:")
		fmt.Println("返回值 :", c.ToString())
	})
	// 失败的回调
	ctx.SetFailedFunc(func(c *esme.Context) {
		fmt.Println("请求出错了...")
		fmt.Println("返回状态值 :", c.Response.StatusCode)
		fmt.Println("返回值 :", c.ToString())
	})

	// 设置http代理
	ctx.SetProxy("http://10.10.10.10:8888")

	// 执行请求
	ctx.Do()
}

返回值

2022/04/26 19:42:57.625 [I] context.go:120: [success] callback -> main.main.func1
请求成功了:
返回值 : ....
4. 使用任务队列

使用内存任务队列的演示

demo code

package main

import (
	"fmt"
	"net/http"

	"github.com/zituocn/esme"
)

var (
	// queue 一个内存中的任务队列
	queue = esme.NewMemQueue()
)

// AddTask 添加任务
func AddTask() {

	city := []string{"北京", "上海", "成都", "深圳", "西安"}

	header := &http.Header{}
	header.Set("User-Agent", "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Safari/537.36")
	for _, item := range city {
		queue.Add(&esme.Task{
			Url:    fmt.Sprintf("%s%s", "https://tenapi.cn/wether/?city=", item),
			Method: "GET",
			Header: header,
		})
	}
}

func main() {

	// 生成任务队列
	AddTask()

	// 设置 任务参数
	job := esme.NewJob("wether", 1, queue, esme.JobOptions{
		SucceedFunc: func(ctx *esme.Context) {
			fmt.Println("成功的回调")
			fmt.Println("返回信息 :", ctx.ToString())
		},
		FailedFunc: func(ctx *esme.Context) {
			fmt.Println("失败的回调")
			fmt.Println("返回状态 :", ctx.Response.StatusCode)
		},
	})

	// 执行
	job.Do()

}
5. 更多文档
  1. http请求的参数设置&&响应处理
  2. 使用 redis 任务队列
  3. 在任务队列中，使用 代理IP池 (多个代理IP使用)
  4. 和 goquery库的配合使用
  5. 和 gjson 库的配合使用
  6. 把数据存储到 mysql 中
6. 感谢

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func GetByte 

func GetByte(obj interface{}) (data []byte)

GetByte interface{} to []byte

func GetFuncName 

func GetFuncName(i interface{}) string

GetFuncName return the function name by reflection

func GetRandSleepTime 

func GetRandSleepTime(min, max int) int

GetRandSleepTime Generate random from min to max 

including min and max

func GetStatusCodeString 

func GetStatusCodeString(code int) string

func RandomUserAgent added in v0.0.2 

func RandomUserAgent(userAgentType UserAgentType) string

RandomUserAgent 取一个随机的user-agent

func Str2Int64 

func Str2Int64(s string) int64

Str2Int64 str convert to int64

Types

type CallbackFunc 

type CallbackFunc func(*Context)

CallbackFunc call back func

type Context 

type Context struct {

	// http request
	Request *http.Request

	// http response
	Response *http.Response

	// error
	Err error

	// reqeusted task
	Task *Task

	// RespBody []byte returned by the request
	RespBody []byte

	// Param context parameter
	Data map[string]interface{}
	// contains filtered or unexported fields
}

Context request and response context

func DoRequest 

func DoRequest(url, method string, vs ...interface{}) *Context

DoRequest start an http request 

returns esme.Context

func HttpGet 

func HttpGet(url string, vs ...interface{}) *Context

HttpGet start an http GET request

func HttpPost 

func HttpPost(url string, vs ...interface{}) *Context

HttpPost start an http POST request

func HttpPut 

func HttpPut(url string, data []byte, vs ...interface{}) *Context

HttpPut start an http PUT request

func NewContext 

func NewContext(req *http.Request, vs ...interface{}) *Context

NewContext returns new Context

func NewRequest 

func NewRequest(url, method string, vs ...interface{}) (*Context, error)

NewRequest returns esme.context and error

func (*Context) Do 

func (c *Context) Do()

Do execute current request

func (*Context) GetExecTime added in v0.0.2 

func (c *Context) GetExecTime() time.Duration

GetExecTime get request execution time

func (*Context) SetCompleteFunc 

func (c *Context) SetCompleteFunc(fn CallbackFunc) *Context

SetCompleteFunc Set the callback for request completion

func (*Context) SetFailedFunc 

func (c *Context) SetFailedFunc(fn CallbackFunc) *Context

SetFailedFunc Callback after setting failure

func (*Context) SetIsDebug 

func (c *Context) SetIsDebug(isDebug bool) *Context

SetIsDebug set debug

func (*Context) SetProxy 

func (c *Context) SetProxy(httpProxy string) *Context

SetProxy set http proxy

func (*Context) SetProxyLib 

func (c *Context) SetProxyLib(lib *ProxyLib) *Context

SetProxyLib set proxy lib

func (*Context) SetRetryFunc 

func (c *Context) SetRetryFunc(fn CallbackFunc) *Context

SetRetryFunc Set callback for retry

func (*Context) SetSleepTime 

func (c *Context) SetSleepTime(sleepTime int) *Context

SetSleepTime Set sleep time for http requests

func (*Context) SetStartFunc 

func (c *Context) SetStartFunc(fn CallbackFunc) *Context

SetStartFunc Set the callback for the start of the request

func (*Context) SetSucceedFunc 

func (c *Context) SetSucceedFunc(fn CallbackFunc) *Context

SetSucceedFunc Callback after successful setting

func (*Context) SetTimeOut 

func (c *Context) SetTimeOut(timeout int) *Context

SetTimeOut Set the timeout for http requests 

milli second 毫秒

func (*Context) SetTransport added in v0.0.7 

func (c *Context) SetTransport(f func() *http.Transport) *Context

SetTransport Set the client's Transport

func (*Context) ToByte 

func (c *Context) ToByte() []byte

ToByte response body to []byte

func (*Context) ToHTML 

func (c *Context) ToHTML() string

ToHTML returns string 

response body to html code

func (*Context) ToJSON 

func (c *Context) ToJSON(v interface{}) error

ToJSON returns error 

response body to struct or map or slice

func (*Context) ToSection added in v0.0.7 

func (c *Context) ToSection(path string) string

ToSection get json string by path 

use:  gjson.Get func
like: ctx.ToSection("body.data")

func (*Context) ToString 

func (c *Context) ToString() string

ToString response body to string 

type Cookie struct {
	Name     string
	Value    string
	HttpOnly bool
}

Cookie esme.Cookie

type FormData 

type FormData map[string]string

FormData esme.FormData 

type Header map[string]string

Header esme.Header

func (Header) Delete 

func (h Header) Delete(key string) Header

func (Header) Set 

func (h Header) Set(key, value string) Header

type Job 

type Job struct {
	// contains filtered or unexported fields
}

Job job struct

func NewJob 

func NewJob(name string, num int, queue TodoQueue, options JobOptions) *Job

NewJob returns a *Job

func (*Job) Do 

func (j *Job) Do()

Do start the job

type JobOptions 

type JobOptions struct {

	// StartFunc Callback at start
	StartFunc CallbackFunc

	// SucceedFunc  Callback after success
	SucceedFunc CallbackFunc

	// RetryFunc retry callback
	RetryFunc CallbackFunc

	// FailedFunc callback after failure
	FailedFunc CallbackFunc

	// CompleteFunc Callback for request completion
	CompleteFunc CallbackFunc

	// ProxyIP proxy ip
	ProxyIP string

	// ProxyLib proxy ip library
	ProxyLib *ProxyLib

	// SheepTime Sleep time for http request execution
	// millisecond
	SheepTime int

	// TimeOut http request timeout
	// millisecond
	TimeOut int

	// 是否打印调试
	IsDebug bool
}

JobOptions 任务参数

type MemQueue 

type MemQueue struct {
	// contains filtered or unexported fields
}

MemQueue in-memory queue

func (*MemQueue) Add 

func (q *MemQueue) Add(task *Task)

Add add a task

func (*MemQueue) AddTasks 

func (q *MemQueue) AddTasks(list []*Task)

AddTasks add multiple tasks at once

func (*MemQueue) Clear 

func (q *MemQueue) Clear() bool

Clear clear queue

func (*MemQueue) IsEmpty 

func (q *MemQueue) IsEmpty() bool

IsEmpty is empty 

return bool

func (*MemQueue) Pop 

func (q *MemQueue) Pop() *Task

Pop get the first task

func (*MemQueue) Print 

func (q *MemQueue) Print()

Print print

func (*MemQueue) Size 

func (q *MemQueue) Size() int

Size returns queue length

type ProxyIP 

type ProxyIP struct {
	IP    string
	Port  int
	User  string
	Pass  string
	IsTLS bool
}

ProxyIP proxy ip struct 

http proxy

func NewProxyIP 

func NewProxyIP(ip string, port int, user, pass string, isTls bool) *ProxyIP

NewProxyIP return http proxy

func (*ProxyIP) String 

func (p *ProxyIP) String() string

String return http proxy string

type ProxyLib 

type ProxyLib struct {
	// contains filtered or unexported fields
}

ProxyLib http proxy lib

func NewProxyLib 

func NewProxyLib() *ProxyLib

NewProxyLib return new ProxyLib

func (*ProxyLib) Add 

func (p *ProxyLib) Add(proxyIP *ProxyIP)

Add proxyIP to ProxyLib

func (*ProxyLib) Del 

func (p *ProxyLib) Del(n int)

Del delete a ip by n

func (*ProxyLib) Get 

func (p *ProxyLib) Get() (string, int32)

Get get a ip

type RedisQueue 

type RedisQueue struct {
	// contains filtered or unexported fields
}

RedisQueue task queue in redis

func (*RedisQueue) Add 

func (q *RedisQueue) Add(task *Task)

Add add a task to the queue

func (*RedisQueue) AddTasks 

func (q *RedisQueue) AddTasks(list []*Task)

AddTasks add multiple tasks to the queue

func (*RedisQueue) Clear 

func (q *RedisQueue) Clear() bool

Clear clear all tasks

func (*RedisQueue) IsEmpty 

func (q *RedisQueue) IsEmpty() bool

IsEmpty returns whether the queue is empty

func (*RedisQueue) Pop 

func (q *RedisQueue) Pop() *Task

Pop get a task while removing it from the queue 

from left

func (*RedisQueue) Print 

func (q *RedisQueue) Print()

func (*RedisQueue) Size 

func (q *RedisQueue) Size() int

Size returns queue length

type StatusCode 

type StatusCode map[int]string

type Task 

type Task struct {

	// Url request address
	Url string `json:"url"`

	// Method request method
	Method string `json:"method"`

	// Payload request payload
	Payload []byte `json:"payload"`

	// FormData request formData
	FormData FormData `json:"form_data"`

	// header *http.Header
	Header *http.Header `json:"header"`

	// Data Contextual data passing
	Data map[string]interface{}
}

Task http Task

type TodoQueue 

type TodoQueue interface {

	// Add add an element to the queue
	Add(task *Task)

	// AddTasks Add multiple elements to the queue at once
	AddTasks(list []*Task)

	// Pop pop and return an element
	Pop() *Task

	// Clear empty the queue
	Clear() bool

	// Size queue length
	Size() int

	// IsEmpty is empty
	IsEmpty() bool

	// Print print
	Print()
}

TodoQueue interface

func NewMemQueue 

func NewMemQueue() TodoQueue

NewMemQueue return a memory queue obj

func NewRedisQueue 

func NewRedisQueue(key string, rc *goredis.RedisConfig) TodoQueue

NewRedisQueue use redis configuration

type UserAgentType added in v0.0.2 

type UserAgentType int

UserAgentType user-agent类型 

const (
	// PCUserAgent pc版本的
	PCUserAgent UserAgentType = iota + 1

	// MobileUserAgent 移版本的
	MobileUserAgent
)

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.