README

Goreq

gopkg goproxycn Go Test codecov

Goreq是对标准库net/http的包装。目的在于简化HTTP请求和接受数据的初步处理。Goreq主要是为了HTML网页和API请求设计的。

使用文档

net/http为人类服务。

go get -u github.com/zhshch2002/goreq

Feature

  • 线程安全
  • 自动解码
  • 便捷代理设置
  • 链式配置请求
  • 支持 Multipart post
  • HTML、JSON、XML解析
  • 中间件
    • 缓存
    • 失败重试
    • 随机UA
    • 填充Referer
    • 设置速率、延时、并发限制

Goreq 是线程安全的,意味着您无论在多线程还是单线程下开发,都无需改动代码。

Goreq 会自动处理网页编码,对于下载下来的网页,Goreq 会根据 HTTP 报头、内容推断编码并加以解码。而且您任可以访问原始的未解码内容。

在Goreq中主要有三个概念。

  • Request*http.Request的封装。描述一个HTTP请求的地址、头部、代理、是否缓存等信息。
  • Client*http.Client组成。用于将Request转化为Response
  • Response*http.Response的封装。并经由Client自动处理编码。提供快速解析HTML、JSON、XML的接口。

构造请求

type Request struct {
   *http.Request
   RespEncode string
   Writer io.Writer
   Debug bool
   callback func(resp *Response) *Response
   client   *Client
   Err error
}
req := goreq.Get("https://httpbin.org/get?a=1").  // <- Notice here we got is req (as Request)
		AddParam("b", "2").
		AddHeaders(map[string]string{
			"req": "golang",
		}).
		AddCookie(&http.Cookie{
			Name:  "c",
			Value: "3",
		}).
		SetUA("goreq")
  • SetDebug(d bool)
  • AddParam(k, v string)
  • AddParams(v map[string]string)
  • AddHeader(key, value string)
  • AddHeaders(v map[string]string)
  • AddCookie(c *http.Cookie)
  • AddCookies(cs ...*http.Cookie)
  • SetUA(ua string)
  • SetBasicAuth(username, password string)
  • SetProxy(urladdr string)
  • SetTimeout(t time.Duration)
  • NoCache()
  • SetCacheExpiration(e time.Duration)
  • DisableRedirect()
  • SetCheckRedirect(fn func(req *http.Request, via []*http.Request) error)
  • 设置请求Body数据
    • SetBody(b io.Reader) basic setting
    • SetRawBody(b []byte)
    • SetFormBody(v map[string]string)
    • SetJsonBody(v interface{})
    • SetMultipartBody(data ...interface{})
  • Callback(fn func(resp *Response)
  • SetClient(c *Client) 这是一个很重要函数。Goreq有很多功能通过Client的中间件实现,为此需要使用自定义的Client执行请求。使用此函数可以改变调用Do()的目标Client

发送请求

Request需要使用Client“执行”来得到Response。

resp := goreq.Get("https://httpbin.org/get?a=1").
		AddParam("b", "2").
		AddHeaders(map[string]string{
			"req": "golang",
		}).
		AddCookie(&http.Cookie{
			Name:  "c",
			Value: "3",
		}).
		SetUA("goreq").Do()

这里使用了Goreq的全局默认Client执行。

c := goreq.NewClient(goreq.WithRandomUA())
resp := goreq.Get("https://httpbin.org/get").SetClient(c).Do()

这里使用了自定义的Client,并使用了随机UA中间件。

获取数据

type Response struct {
	*http.Response
	Body           []byte
	NotDecodedBody []byte
	Text           string
	Req            *Request
	CacheHash      string
	Err            error
}
  • Resp() (*Response, error) 获取响应本身以及网络请求错误。
  • Txt() (string, error) 自动处理完编码并解析为文本后的内容以及网络请求错误。
  • RespAndTxt() (*Response, string, error)
  • HTML() (*goquery.Document, error)
  • RespAndHTML() (*Response, *goquery.Document, error)
  • IsHTML() bool
  • XML() (*xmlpath.Node, error)
  • RespAndXML() (*Response, *xmlpath.Node, error)
  • BindXML(i interface{}) error
  • JSON() (gjson.Result, error)
  • RespAndJSON() (*Response, gjson.Result, error)
  • BindJSON(i interface{}) error
  • IsJSON() bool
  • Error() error 网络请求错误。(正常情况下为nil

Documentation

Index

Constants

This section is empty.

Variables

View Source
var Debug = false
View Source
var DefaultClient = NewClient()
View Source
var ReqRejectedErr = errors.New("request is rejected")

Functions

func GetRequestHash

func GetRequestHash(r *Request) string

GetRequestHash return a hash of url,header,cookie and body data from a request

func ModifyLink(url string) string

Types

type Client

type Client struct {
	Client *http.Client
	// contains filtered or unexported fields
}

func NewClient

func NewClient(m ...Middleware) *Client

func (*Client) Do

func (s *Client) Do(req *Request) *Response

func (*Client) Use

func (s *Client) Use(mid ...Middleware) *Client

type DelayLimiterOpinion

type DelayLimiterOpinion struct {
	LimiterMatcher
	Delay       time.Duration
	RandomDelay time.Duration
	// contains filtered or unexported fields
}

type FilterLimiterOpinion

type FilterLimiterOpinion struct {
	LimiterMatcher
	Allow bool
}

type FormField

type FormField struct {
	Name, Value string
}

type FormFile

type FormFile struct {
	FieldName, FileName, ContentType string
	File                             io.Reader
}

type Handler

type Handler func(*Request) *Response

type LimitRuleAllow

type LimitRuleAllow uint8
const (
	NotSet LimitRuleAllow = iota
	Allow
	Disallow
)

type LimiterMatcher

type LimiterMatcher struct {
	Regexp, Glob string
	// contains filtered or unexported fields
}

func (*LimiterMatcher) Compile

func (s *LimiterMatcher) Compile()

func (*LimiterMatcher) Match

func (s *LimiterMatcher) Match(u *url.URL) bool

type Middleware

type Middleware func(*Client, Handler) Handler

func WithCache

func WithCache(ca *cache.Cache) Middleware

func WithCookie

func WithCookie(urlAddr string, cookies ...*http.Cookie) Middleware

func WithDebug

func WithDebug() Middleware

func WithDelayLimiter

func WithDelayLimiter(eachSite bool, opts ...*DelayLimiterOpinion) Middleware

func WithFilterLimiter

func WithFilterLimiter(noneMatchAllow bool, opts ...*FilterLimiterOpinion) Middleware

func WithParallelismLimiter

func WithParallelismLimiter(eachSite bool, opts ...*ParallelismLimiterOpinion) Middleware

func WithProxy

func WithProxy(p ...string) Middleware

func WithRandomUA

func WithRandomUA() Middleware

func WithRateLimiter

func WithRateLimiter(eachSite bool, opts ...*RateLimiterOpinion) Middleware

func WithRefererFiller

func WithRefererFiller() Middleware

func WithRetry

func WithRetry(maxTimes int, isRespOk func(*Response) bool) Middleware

type ParallelismLimiterOpinion

type ParallelismLimiterOpinion struct {
	LimiterMatcher
	Parallelism int64
	// contains filtered or unexported fields
}

type RateLimiterOpinion

type RateLimiterOpinion struct {
	LimiterMatcher
	Rate int64
	// contains filtered or unexported fields
}

type Request

type Request struct {
	*http.Request

	RespEncode string

	Writer io.Writer

	Debug bool

	Err error
	// contains filtered or unexported fields
}

Request is a object of HTTP request

func Connect

func Connect(urladdr string) *Request

func Delete

func Delete(urladdr string) *Request

func Get

func Get(urladdr string) *Request
func Head(urladdr string) *Request

func NewRequest

func NewRequest(method, urladdr string) *Request

func Options

func Options(urladdr string) *Request

func Patch

func Patch(urladdr string) *Request

func Post

func Post(urladdr string) *Request

func Put

func Put(urladdr string) *Request

func Trace

func Trace(urladdr string) *Request

func (*Request) AddCookie

func (s *Request) AddCookie(c *http.Cookie) *Request

AddCookie adds a cookie to the request.

func (*Request) AddCookies

func (s *Request) AddCookies(cs ...*http.Cookie) *Request

AddCookies adds some cookie to the request at once.

func (*Request) AddHeader

func (s *Request) AddHeader(key, value string) *Request

AddHeader sets the header entries associated with key to the single element value.

func (*Request) AddHeaders

func (s *Request) AddHeaders(v map[string]string) *Request

func (*Request) AddParam

func (s *Request) AddParam(k, v string) *Request

AddParam adds a query param of request url.

func (*Request) AddParams

func (s *Request) AddParams(v map[string]string) *Request

func (*Request) DisableRedirect

func (s *Request) DisableRedirect() *Request

func (*Request) Do

func (s *Request) Do() *Response

func (*Request) NoCache

func (s *Request) NoCache() *Request

func (*Request) SetBasicAuth

func (s *Request) SetBasicAuth(username, password string) *Request

func (*Request) SetBody

func (s *Request) SetBody(b io.Reader) *Request

func (*Request) SetCacheExpiration

func (s *Request) SetCacheExpiration(e time.Duration) *Request

func (*Request) SetCallback

func (s *Request) SetCallback(fn func(resp *Response) *Response) *Request

func (*Request) SetCheckRedirect

func (s *Request) SetCheckRedirect(fn func(req *http.Request, via []*http.Request) error) *Request

func (*Request) SetClient

func (s *Request) SetClient(c *Client) *Request

func (*Request) SetDebug

func (s *Request) SetDebug(d bool) *Request

func (*Request) SetFormBody

func (s *Request) SetFormBody(v map[string]string) *Request

func (*Request) SetJsonBody

func (s *Request) SetJsonBody(v interface{}) *Request

func (*Request) SetMultipartBody

func (s *Request) SetMultipartBody(data ...interface{}) *Request

func (*Request) SetProxy

func (s *Request) SetProxy(urladdr string) *Request

func (*Request) SetRawBody

func (s *Request) SetRawBody(b []byte) *Request

func (*Request) SetTimeout

func (s *Request) SetTimeout(t time.Duration) *Request

func (*Request) SetUA

func (s *Request) SetUA(ua string) *Request

SetUA sets user-agent url of request header.

func (*Request) String

func (s *Request) String() string

type RequestError

type RequestError struct {
	// contains filtered or unexported fields
}

type Response

type Response struct {
	*http.Response
	Body           []byte
	NotDecodedBody []byte
	Text           string
	Req            *Request
	CacheHash      string
	Err            error
}

Response is a object of HTTP response

func Do

func Do(req *Request) *Response

func (*Response) BindJSON

func (s *Response) BindJSON(i interface{}) error

func (*Response) BindXML

func (s *Response) BindXML(i interface{}) error

func (*Response) DecodeAndParse

func (s *Response) DecodeAndParse() error

DecodeAndParas decodes the body to text and try to parse it to html or json.

func (*Response) Error

func (s *Response) Error() error

func (*Response) HTML

func (s *Response) HTML() (*goquery.Document, error)

func (*Response) IsHTML

func (s *Response) IsHTML() bool

func (*Response) IsJSON

func (s *Response) IsJSON() bool

func (*Response) JSON

func (s *Response) JSON() (gjson.Result, error)

func (*Response) Resp

func (s *Response) Resp() (*Response, error)

func (*Response) RespAndHTML

func (s *Response) RespAndHTML() (*Response, *goquery.Document, error)

func (*Response) RespAndJSON

func (s *Response) RespAndJSON() (*Response, gjson.Result, error)

func (*Response) RespAndTxt

func (s *Response) RespAndTxt() (*Response, string, error)

func (*Response) RespAndXML

func (s *Response) RespAndXML() (*Response, *xmlpath.Node, error)

func (*Response) Txt

func (s *Response) Txt() (string, error)

func (*Response) XML

func (s *Response) XML() (*xmlpath.Node, error)