fhttp

package
v1.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jun 9, 2023 License: Apache-2.0 Imports: 39 Imported by: 0

Documentation

Index

Constants

View Source
const (
	// SocketError is return when a transport error occurred: unexpected EOF, connection error, etc...
	SocketError = -1
	// RetryOnce is used internally as an error code to allow 1 retry for bad socket reuse.
	RetryOnce = -2
)

Extra error codes outside of the HTTP Status code ranges. ie negative.

View Source
const (

	// HTTPReqTimeOutDefaultValue is the default timeout value. 3s.
	HTTPReqTimeOutDefaultValue = 3 * time.Second
)

Variables

View Source
var (
	// BufferSizeKb size of the buffer (max data) for optimized client in kilobytes defaults to 128k.
	BufferSizeKb = 128
	// CheckConnectionClosedHeader indicates whether to check for server side connection closed headers.
	CheckConnectionClosedHeader = false
)
View Source
var (
	// EnvoyRequestID is the header set by envoy and we need to propagate for distributed tracing.
	EnvoyRequestID = textproto.CanonicalMIMEHeaderKey("x-request-id")
	// TraceHeader is the single aggregated open tracing header to propagate when present.
	TraceHeader = textproto.CanonicalMIMEHeaderKey("b3")
	// TraceHeadersPrefix is the prefix for the multi header version of open zipkin.
	TraceHeadersPrefix = textproto.CanonicalMIMEHeaderKey("x-b3-")
)
View Source
var (

	// EchoRequests is the number of request received. Only updated in Debug mode.
	EchoRequests            int64
	DefaultEchoServerParams = dflag.New("",
		"Default parameters/querystring to use if there isn't one provided explicitly. E.g \"status=404&delay=3s\"")
	Fetch2CopiesAllHeader = dflag.NewBool(true,
		"Determines if only tracing or all headers (and cookies) are copied from request on the fetch2 ui/server endpoint")
	ServerIdleTimeout = dflag.New(30*time.Second, "Default IdleTimeout for servers")
)
View Source
var MaxDelay = dflag.New(1500*time.Millisecond,
	"Maximum sleep time for delay= echo server parameter. dynamic flag.")

MaxDelay is the maximum delay allowed for the echoserver responses. It is a dynamic flag with default value of 1.5s so we can test the default 1s timeout in envoy.

Functions

func ASCIIToUpper

func ASCIIToUpper(str string) []byte

ASCIIToUpper returns a byte array equal to the input string but in lowercase. Only works for ASCII, not meant for unicode.

func AddHTTPS

func AddHTTPS(url string) string

AddHTTPS replaces "http://" in url with "https://" or prepends "https://" if url does not contain prefix "http://".

func CacheOn

func CacheOn(w http.ResponseWriter)

CacheOn sets the header for indefinite caching.

func CopyHeaders

func CopyHeaders(req, r *http.Request, all bool)

CopyHeaders copies all or trace headers from `r` into `req`.

func CreateProxyClient

func CreateProxyClient() *http.Client

CreateProxyClient http client for connection reuse.

func DebugHandler

func DebugHandler(w http.ResponseWriter, r *http.Request)

DebugHandler returns debug/useful info to http client. Note this can be dangerous and shouldn't be exposed to the internet. A safer version is available as part of fortio's proxy https://github.com/fortio/proxy/blob/main/rp/reverse_proxy.go

func DebugSummary

func DebugSummary(buf []byte, max int) string

DebugSummary returns a string with the size and escaped first max/2 and last max/2 bytes of a buffer (or the whole escaped buffer if small enough).

func DynamicHTTPServer

func DynamicHTTPServer(closing bool) (*http.ServeMux, *net.TCPAddr)

DynamicHTTPServer listens on an available port, sets up an http or a closing server simulating an https server (when closing is true) server on it and returns the listening port and mux to which one can attach handlers to. Note: in a future version of istio, the closing will be actually be secure on/off and create an https server instead of a closing server. As this is a dynamic tcp socket server, the address is TCP.

func EchoDebugPath

func EchoDebugPath(debugPath string) string

EchoDebugPath returns the additional echo handler path behind debugPath (ie /debug -> /debug/echo/).

func EchoHandler

func EchoHandler(w http.ResponseWriter, r *http.Request)

EchoHandler is an http server handler echoing back the input.

func Fetch

func Fetch(httpOptions *HTTPOptions) (int, []byte)

Fetch creates a client an performs a fetch according to the http options passed in. To be used only for single fetches or when performance doesn't matter as the client is closed at the end. Deprecated: use StreamFetch instead.

func FetchURL

func FetchURL(url string) (int, []byte)

FetchURL fetches the data at the given url using the standard client and default options. Returns the http status code (http.StatusOK == 200 for success) and the data. To be used only for single fetches or when performance doesn't matter as the client is closed at the end. Deprecated: use StreamURL instead.

func FetcherHandler

func FetcherHandler(w http.ResponseWriter, r *http.Request)

FetcherHandler is the handler for the fetcher/proxy.

func FetcherHandler2

func FetcherHandler2(w http.ResponseWriter, r *http.Request)

FetcherHandler2 is the handler for the fetcher/proxy that supports h2 input and makes a new request with all headers copied (allows to test sticky routing) Note this should only be made available to trusted clients.

func Flush

func Flush(w http.ResponseWriter)

func FoldFind

func FoldFind(haystack []byte, needle []byte) (bool, int)

FoldFind searches the bytes assuming ascii, ignoring the lowercase bit for testing. Not intended to work with unicode, meant for http headers and to be fast (see benchmark in test file).

func Gzip

func Gzip(next http.Handler) http.Handler

Gzip wraps a handler for automatic gzip.

func HTTPSServer

func HTTPSServer(name string, port string, to *TLSOptions) (*http.ServeMux, net.Addr)

func HTTPServer

func HTTPServer(name string, port string) (*http.ServeMux, net.Addr)

HTTPServer creates an http server named name on address/port port. Port can include binding address and/or be port 0.

func HTTPServerWithHandler

func HTTPServerWithHandler(name string, port string, hdlr http.Handler) net.Addr

HTTPServerWithHandler creates and h2c compatible server named name on address/port port. Port can include binding address and/or be port 0. Takes in a handler.

func LogAndCall

func LogAndCall(msg string, hf http.HandlerFunc) http.HandlerFunc

LogAndCall wraps an HTTP handler to log the request first.

func LogAndCallNoArg

func LogAndCallNoArg(msg string, f func()) http.HandlerFunc

LogAndCallNoArg is LogAndCall for functions not needing the response/request args.

func MakeSimpleRequest

func MakeSimpleRequest(url string, r *http.Request, copyAllHeaders bool) *http.Request

MakeSimpleRequest makes a new request for url but copies trace headers from input request r. or all the headers if copyAllHeaders is true.

func MultiServer

func MultiServer(port string, cfg *MultiServerConfig) (*http.ServeMux, net.Addr)

MultiServer starts fan out http server on the given port. Returns the mux and addr where the listening socket is bound. The port can be retrieved from it when requesting the 0 port as input for dynamic http server.

func NewHTMLEscapeWriter

func NewHTMLEscapeWriter(w io.Writer) io.Writer

NewHTMLEscapeWriter creates a io.Writer that can safely output to an http.ResponseWrite with HTMLEscape-ing.

func OnBehalfOf

func OnBehalfOf(o *HTTPOptions, r *http.Request)

OnBehalfOf adds a header with the remote addr to an http options object.

func OnBehalfOfRequest

func OnBehalfOfRequest(to *http.Request, from *http.Request)

OnBehalfOfRequest same as OnBehalfOf but places the header directly on the dst request object but also adds a X-Proxy-Agent header if the user-agent isn't already the same as this running server's version.

func ParseChunkSize

func ParseChunkSize(inp []byte) (int64, int64)

ParseChunkSize extracts the chunk size and consumes the line. Returns the offset of the data and the size of the chunk, 0, -1 when not found.

func ParseDecimal

func ParseDecimal(inp []byte) int64

ParseDecimal extracts the first positive integer number from the input. spaces are ignored. any character that isn't a digit cause the parsing to stop.

func QueryArg

func QueryArg(r *http.Request, key string) string

QueryArg(r,...) is like r.FormValue(...) but exclusively getting the values from the query string (as we use the body for data). Result of parsing the query string is cached in r.Form so we don't keep parsing it and so r.Form can be used for multivalued entries like r.Form["header"].

func RedirectToHTTPS

func RedirectToHTTPS(port string) net.Addr

RedirectToHTTPS Sets up a redirector to https on the given port. (Do not create a loop, make sure this is addressed from an ingress).

func RedirectToHTTPSHandler

func RedirectToHTTPSHandler(w http.ResponseWriter, r *http.Request)

RedirectToHTTPSHandler handler sends a redirect to same URL with https.

func RoundDuration

func RoundDuration(d time.Duration) time.Duration

RoundDuration rounds to 10th of second.

func Serve

func Serve(port, debugPath string) (*http.ServeMux, net.Addr)

Serve starts a debug / echo http server on the given port. Returns the mux and addr where the listening socket is bound. The .Port can be retrieved from it when requesting the 0 port as input for dynamic http server.

func ServeTCP

func ServeTCP(port, debugPath string) (*http.ServeMux, *net.TCPAddr)

ServeTCP is Serve() but restricted to TCP (return address is assumed to be TCP - will panic for unix domain).

func ServeTLS

func ServeTLS(port, debugPath string, to *TLSOptions) (*http.ServeMux, net.Addr)

ServeTLS starts a debug / echo server on the given port, using TLS if certPath and keyPath aren't not empty.

func SetupPPROF

func SetupPPROF(mux *http.ServeMux)

SetupPPROF add pprof to the mux (mirror the init() of http pprof).

func StreamFetch

func StreamFetch(httpOptions *HTTPOptions) int

Fetch creates a client an performs a fetch according to the http options passed in. To be used only for single fetches or when performance doesn't matter as the client is closed at the end.

func StreamURL

func StreamURL(url string, w io.Writer) int

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client object for making repeated requests of the same URL using the same http client (net/http). TODO: refactor common parts with FastClient.

func NewStdClient

func NewStdClient(o *HTTPOptions) (*Client, error)

NewStdClient creates a client object that wraps the net/http standard client.

func (*Client) ChangeURL

func (c *Client) ChangeURL(urlStr string) (err error)

ChangeURL only for standard client, allows fetching a different URL.

func (*Client) Close

func (c *Client) Close()

Close cleans up any resources used by NewStdClient.

func (*Client) Fetch

func (c *Client) Fetch(ctx context.Context) (int, []byte, int)

Fetch is the backward compatible version of StreamFetch to avoid rewriting a bunch of tests but should not be used for load tests (where we don't care about the data and only the stats) Deprecated: use StreamFetch instead.

func (*Client) GetIPAddress

func (c *Client) GetIPAddress() (*stats.Occurrence, *stats.Histogram)

GetIPAddress get the ip address that DNS resolves to when using stdClient and connection stats.

func (*Client) HasBuffer

func (c *Client) HasBuffer() bool

func (*Client) StreamFetch

func (c *Client) StreamFetch(ctx context.Context) (int, int64, uint)

StreamFetch fetches the byte and code for pre created std client. header length (3rd returned value) is always 0 for that client and only available with the fastclient.

type CreateClientTrace

type CreateClientTrace func(ctx context.Context) *httptrace.ClientTrace

type CreateTransport

type CreateTransport func(base http.RoundTripper) http.RoundTripper

type FastClient

type FastClient struct {
	// contains filtered or unexported fields
}

FastClient is a fast, lockfree single purpose http 1.0/1.1 client.

func (*FastClient) Close

func (c *FastClient) Close()

Close cleans up any resources used by FastClient.

func (*FastClient) Fetch

func (c *FastClient) Fetch(ctx context.Context) (int, []byte, int)

Fetch fetches the url content. Returns http code, data, offset of body.

func (*FastClient) GetIPAddress

func (c *FastClient) GetIPAddress() (*stats.Occurrence, *stats.Histogram)

GetIPAddress get ip address that DNS resolved to when using fast client and connection stats.

func (*FastClient) HasBuffer

func (c *FastClient) HasBuffer() bool

func (*FastClient) StreamFetch

func (c *FastClient) StreamFetch(ctx context.Context) (int, int64, uint)

Fetch fetches the url content. Returns http code, data written to the writer, length of headers.

type Fetcher

type Fetcher interface {
	// Fetch returns http code, data, offset of body (for client which returns
	// headers)
	// Deprecated: use StreamFetch with a DataWriter (nil if you don't need the data) instead.
	Fetch(context.Context) (int, []byte, int)
	// StreamFetch returns http code and body bytes read
	// (body is streamed to Dest writer or null),and header size for the fast client.
	StreamFetch(context.Context) (int, int64, uint)
	// HasBuffer is true for the fast client and false for golang standard library based client.
	// it's used to know if calling Fetch() is actually better (fast client with headers to stderr)
	HasBuffer() bool
	// Close() cleans up connections and state - must be paired with NewClient calls.
	Close()
	// GetIPAddress() returns the occurrence of ip address used by this client connection,
	// and the connection time histogram (which includes the count).
	GetIPAddress() (*stats.Occurrence, *stats.Histogram)
}

Fetcher is the Url content fetcher that the different client implements.

func NewClient

func NewClient(o *HTTPOptions) (Fetcher, error)

NewClient creates either a standard or fast client (depending on the DisableFastClient flag).

func NewFastClient

func NewFastClient(o *HTTPOptions) (Fetcher, error)

NewFastClient makes a basic, efficient http 1.0/1.1 client. This function itself doesn't need to be super efficient as it is created at the beginning and then reused many times.

type FlushWriter

type FlushWriter struct {
	// contains filtered or unexported fields
}

func (FlushWriter) Write

func (fw FlushWriter) Write(p []byte) (n int, err error)

type GzipResponseWriter

type GzipResponseWriter struct {
	io.Writer
	http.ResponseWriter
	// contains filtered or unexported fields
}

GzipResponseWriter wraps the response and gzips the content.

func NewGzipHTTPResponseWriter

func NewGzipHTTPResponseWriter(w http.ResponseWriter) *GzipResponseWriter

NewGzipHTTPResponseWriter returns a wrapper for gzip'ing the response.

func (*GzipResponseWriter) Close

func (w *GzipResponseWriter) Close() error

Close must be called in defer inside the handler using this.

func (*GzipResponseWriter) Write

func (w *GzipResponseWriter) Write(b []byte) (int, error)

Write sends the Write() to the gzip Writer.

func (*GzipResponseWriter) WriteHeader

func (w *GzipResponseWriter) WriteHeader(status int)

WriteHeader intercepts the actual to remove any Content-Length that may have been added before compression.

type HTMLEscapeWriter

type HTMLEscapeWriter struct {
	NextWriter io.Writer
	Flusher    http.Flusher
}

HTMLEscapeWriter is an io.Writer escaping the output for safe html inclusion.

func (*HTMLEscapeWriter) Write

func (w *HTMLEscapeWriter) Write(p []byte) (int, error)

type HTTPOptions

type HTTPOptions struct {
	TLSOptions
	URL               string
	NumConnections    int  // num connections (for std client)
	Compression       bool // defaults to no compression, only used by std client
	DisableFastClient bool // defaults to fast client
	HTTP10            bool // defaults to http1.1
	H2                bool // defaults to http1.1 (h2 only for stdclient)
	DisableKeepAlive  bool // so default is keep alive
	AllowHalfClose    bool // if not keepalive, whether to half close after request
	FollowRedirects   bool // For the Std Client only: follow redirects.

	Resolve string // resolve Common Name to this ip when use CN as target url

	HTTPReqTimeOut   time.Duration // timeout value for http request
	UserCredentials  string        // user credentials for authorization
	ContentType      string        // indicates request body type, implies POST instead of GET
	Payload          []byte        // body for http request, implies POST if not empty.
	LogErrors        bool          // whether to log non 2xx code as they occur or not
	ID               int           `json:"-"` // thread/connect id to use for logging (thread id when used as a runner)
	SequentialWarmup bool          // whether to do http(s) runs warmup sequentially or in parallel (new default is //)
	ConnReuseRange   [2]int        // range of max number of connection to reuse for each thread.
	// When false, re-resolve the DNS name when the connection breaks.
	NoResolveEachConn bool
	// Optional Offset Duration; to offset the histogram of the Connection duration
	Offset time.Duration
	// Optional resolution divider for the Connection duration histogram. In seconds. Defaults to 0.001 or 1 millisecond.
	Resolution float64
	// Optional ClientTrace factory to use if set. Only effective when using std client.
	ClientTrace CreateClientTrace `json:"-"`
	// Optional Transport chain factory to use if set. Only effective when using std client.
	// pass otelhttp.NewTransport for instance.
	Transport CreateTransport `json:"-"`
	// These following 2 options are only making sense for single operation (curl) mode.
	PayloadReader io.Reader `json:"-"` // if set, Payload is ignored and this is used instead.
	DataWriter    io.Writer `json:"-"` // if set, the response body is written to this writer.
	// contains filtered or unexported fields
}

HTTPOptions holds the common options of both http clients and the headers.

func NewHTTPOptions

func NewHTTPOptions(url string) *HTTPOptions

NewHTTPOptions creates and initialize a HTTPOptions object. Do not change the URL directly later, make a new HTTPOptions. It replaces plain % to %25 in the url. If you already have properly escaped URLs use o.URL = to set it.

func (*HTTPOptions) AddAndValidateExtraHeader

func (h *HTTPOptions) AddAndValidateExtraHeader(hdr string) error

AddAndValidateExtraHeader collects extra headers (see commonflags.go for example).

func (*HTTPOptions) AllHeaders

func (h *HTTPOptions) AllHeaders() http.Header

AllHeaders returns the current set of headers including virtual/special Host header.

func (*HTTPOptions) GenerateHeaders

func (h *HTTPOptions) GenerateHeaders() http.Header

GenerateHeaders completes the header generation, including Content-Type/Length and user credential coming from the http options in addition to extra headers coming from flags and AddAndValidateExtraHeader(). Warning this gets called more than once, do not generate duplicate headers.

func (*HTTPOptions) Init

func (h *HTTPOptions) Init(url string) *HTTPOptions

Init initializes the headers in an HTTPOptions (User-Agent).

func (*HTTPOptions) InitHeaders

func (h *HTTPOptions) InitHeaders()

InitHeaders initialize and/or resets the default headers (ie just User-Agent).

func (*HTTPOptions) Method

func (h *HTTPOptions) Method() string

Method returns the method of the http req.

func (*HTTPOptions) PayloadUTF8

func (h *HTTPOptions) PayloadUTF8() string

PayloadUTF8 returns the payload as a string. If payload is null return empty string This is only needed due to grpc ping proto. It takes string instead of byte array.

func (*HTTPOptions) ResetHeaders

func (h *HTTPOptions) ResetHeaders()

ResetHeaders resets all the headers, including the User-Agent: one (and the Host: logical special header). This is used from the UI as the user agent is settable from the form UI.

func (*HTTPOptions) URLSchemeCheck

func (h *HTTPOptions) URLSchemeCheck()

URLSchemeCheck makes sure the client will work with the scheme requested. it also adds missing http:// to emulate curl's behavior.

func (*HTTPOptions) ValidateAndAddBasicAuthentication

func (h *HTTPOptions) ValidateAndAddBasicAuthentication(headers http.Header) error

ValidateAndAddBasicAuthentication validates user credentials and adds basic authentication to http header, if user credentials are valid.

func (*HTTPOptions) ValidateAndSetConnectionReuseRange

func (h *HTTPOptions) ValidateAndSetConnectionReuseRange(inp string) error

type HTTPRunnerOptions

type HTTPRunnerOptions struct {
	periodic.RunnerOptions
	HTTPOptions               // Need to call Init() to initialize
	Profiler           string // file to save profiles to. defaults to no profiling
	AllowInitialErrors bool   // whether initial errors don't cause an abort
	// Which status code cause an abort of the run (default 0 = don't abort; reminder -1 is returned for socket errors)
	AbortOn int
}

HTTPRunnerOptions includes the base RunnerOptions plus http specific options.

type HTTPRunnerResults

type HTTPRunnerResults struct {
	periodic.RunnerResults

	RetCodes   map[int]int64
	IPCountMap map[string]int // TODO: Move it to a shared results struct where all runner should have this field

	// exported result
	HTTPOptions
	Sizes       *stats.HistogramData
	HeaderSizes *stats.HistogramData
	Sockets     []int64
	SocketCount int64
	// Connection Time stats
	ConnectionStats *stats.HistogramData
	// http code to abort the run on (-1 for connection or other socket error)
	AbortOn int
	// contains filtered or unexported fields
}

HTTPRunnerResults is the aggregated result of an HTTPRunner. Also is the internal type used per thread/goroutine.

func RunHTTPTest

func RunHTTPTest(o *HTTPRunnerOptions) (*HTTPRunnerResults, error)

RunHTTPTest runs an http test and returns the aggregated stats.

func (*HTTPRunnerResults) Run

func (httpstate *HTTPRunnerResults) Run(ctx context.Context, t periodic.ThreadID) (bool, string)

Run tests http request fetching. Main call being run at the target QPS. To be set as the Function in RunnerOptions.

type MultiServerConfig

type MultiServerConfig struct {
	Targets []TargetConf
	Serial  bool // Serialize or parallel queries
	//	Javascript bool // return data as UI suitable
	Name string
	// contains filtered or unexported fields
}

MultiServerConfig configures the MultiServer and holds the http client it uses for proxying.

func (*MultiServerConfig) TeeHandler

func (mcfg *MultiServerConfig) TeeHandler(w http.ResponseWriter, r *http.Request)

TeeHandler common part between TeeSerialHandler and TeeParallelHandler.

func (*MultiServerConfig) TeeParallelHandler

func (mcfg *MultiServerConfig) TeeParallelHandler(w http.ResponseWriter, r *http.Request, data []byte)

TeeParallelHandler handles teeing off traffic in parallel (one goroutine each) mode.

func (*MultiServerConfig) TeeSerialHandler

func (mcfg *MultiServerConfig) TeeSerialHandler(w http.ResponseWriter, r *http.Request, data []byte)

TeeSerialHandler handles teeing off traffic in serial (one at a time) mode.

type SyncReader

type SyncReader struct {
	// contains filtered or unexported fields
}

SyncReader is a thread-safe wrapper for a reader.

func NewSyncReader

func NewSyncReader(reader io.Reader) *SyncReader

NewSyncReader returns a new thread-safe reader.

func (*SyncReader) Read

func (r *SyncReader) Read(p []byte) (n int, err error)

type TLSOptions

type TLSOptions struct {
	Insecure         bool   // Do not verify certs
	MTLS             bool   // Use mutual TLS, require client cert
	CACert           string // `Path` to a custom CA certificate file to be used
	Cert             string // `Path` to the certificate file to be used
	Key              string // `Path` to the key file used
	UnixDomainSocket string // `Path“ of unix domain socket to use instead of host:port
}

TLSOptions are common TLS related options between https and grpc.

func (*TLSOptions) TLSConfig

func (to *TLSOptions) TLSConfig() (*tls.Config, error)

TLSConfig creates a tls.Config based on input TLSOptions. For https, ServerName is set later (once host is determined after URL parsing and depending on hostOverride). Used for both client and server TLS config.

type TargetConf

type TargetConf struct {
	Destination  string // Destination URL or base
	MirrorOrigin bool   // wether to use the incoming request as URI and data params to outgoing one (proxy like)

}

TargetConf is the structure to configure one of the multiple targets for MultiServer.

type Transport

type Transport interface {
	http.RoundTripper
	CloseIdleConnections()
}

Transport common interface between http.Transport and http2.Transport.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL