esitag

package
v0.0.0-...-8dcffa4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 3, 2019 License: Apache-2.0 Imports: 27 Imported by: 15

Documentation

Overview

Package esitag provides a parser, types and configurations for <esi:include tags.

Index

Constants

View Source 
const (
	CBStateOpen = iota + 1
	CBStateHalfOpen
	CBStateClosed
)

CBState declares the different states for the circuit breaker (CB)

View Source 
const DefaultTimeOut = 20 * time.Second

DefaultTimeOut defines the default timeout to a backend resource.

View Source 
const MaxSizeESITag = 4096

MaxSizeESITag maximum size of an Tag tag. For now this value has been returned from a dice roll.

Variables

View Source 
var CBMaxFailures uint64 = 12

CBMaxFailures maximum amount of failures before the circuit breaker is half open to try the next request.

View Source 
var CBThresholdCalc = func(failures uint64) time.Duration {
	return (1 << failures) * time.Second
}

CBThresholdCalc calculates the threshold how long the CB should wait until to set the HalfOpen state. Default implementation returns an exponentially calculated duration

View Source 
var DropHeadersForward = map[string]bool{
	"Cache-Control": true,
	"Connection":    true,
	"Host":          true,
	"Pragma":        true,
	"Upgrade":       true,
}

DropHeadersForward a list of headers which should never be forwarded to the backend resource. Initial idea of excluded fields. https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

View Source 
var DropHeadersReturn = map[string]bool{
	"Cache-Control":             true,
	"Connection":                true,
	"Content-Disposition":       true,
	"Content-Encoding":          true,
	"Content-Length":            true,
	"Content-Range":             true,
	"Content-Type":              true,
	"Date":                      true,
	"Etag":                      true,
	"Expires":                   true,
	"Last-Modified":             true,
	"Location":                  true,
	"Status":                    true,
	"Strict-Transport-Security": true,
	"Trailer":                   true,
	"Transfer-Encoding":         true,
	"Upgrade":                   true,
}

DropHeadersReturn a list of headers which should never be forwarded to the client. Initial idea of excluded fields. https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

Functions

func CloseAllResourceHandler 

func CloseAllResourceHandler() error

CloseAllResourceHandler does what the function name says returns the first occurred error.

func DeregisterResourceHandler 

func DeregisterResourceHandler(scheme string)

DeregisterResourceHandler removes a previously registered scheme/alias.

func RegisterResourceHandler 

func RegisterResourceHandler(scheme string, f ResourceHandler) struct{ DeferredDeregister func() }

RegisterResourceHandler scheme can be a protocol before the :// but also an alias to register a key-value service. This function returns a closure which lets you deregister the scheme/alias once a test has finished. Use the defer word. Scheme/alias will be transformed into an all lowercase string.

func RegisterResourceHandlerFactory 

func RegisterResourceHandlerFactory(scheme string, f ResourceHandlerFactoryFunc)

RegisterResourceHandlerFactory registers a new factory function to create a new ResourceHandler. Useful when you have entries in the resources_config.xml|json file.

func SplitAttributes 

func SplitAttributes(raw string) ([]string, error)

SplitAttributes splits an Tag tag by its attributes. This function avoids regexp.

Types

type Conditioner 

type Conditioner interface {
	OK(r *http.Request) bool
}

Conditioner does not represent your favorite shampoo but it gives you the possibility to define an expression which gets executed for every request to include the Tag resource or not.

type Config 

type Config struct {
	Log               log.Logger // optional
	ForwardHeaders    []string   // optional, already treated with http.CanonicalHeaderKey
	ReturnHeaders     []string   // optional, already treated with http.CanonicalHeaderKey
	ForwardPostData   bool       // optional
	ForwardHeadersAll bool       // optional
	ReturnHeadersAll  bool       // optional
	// Coalesce will merge n-external parallel requests into one resource
	// backend request.
	Coalesce bool
	// PrintDebug injects the time taken into the returned data as hidden HTML
	// comment in function Entities.QueryResources. It also provides the raw tag
	// and in future some other data for easier debugging.
	PrintDebug bool
	// Timeout maximum time needed for a backend request before the cancellation
	// context kills it.
	Timeout time.Duration // required
	// TTL retrieved content from a backend can live this time in the middleware
	// cache.
	TTL time.Duration // optional
	// MaxBodySize allowed max body size to read from the backend resource.
	MaxBodySize uint64 // required
	// Key defines the name of the key in an NoSQL service or as additional
	// identifier in a gRPC request.
	Key string
}

Config provides the configuration of a single Tag tag. This information gets passed on as an argument towards the backend resources and enriches the Entity type.

type DataTag 

type DataTag struct {
	Data  []byte // Data from the micro service gathered in a goroutine. Can be nil.
	Start int    // Start position in the stream
	End   int    // End position in the stream. Never smaller than Start.
}

DataTag identifies an Tag tag by its start and end position in the HTML byte stream for replacing. If the HTML changes there needs to be a refresh call to re-parse the HTML.

func (DataTag) String 

func (dt DataTag) String() string

String prints human readable the data tag for debugging purposes.

type DataTags 

type DataTags struct {
	Slice []DataTag
	// contains filtered or unexported fields
}

DataTags a list of tags with their position within a page and the content

func NewDataTagsCapped 

func NewDataTagsCapped(cap int) *DataTags

NewDataTagsCapped creates a new object with a DataTag slice and its maximum capacity.

func (*DataTags) DataLen 

func (dts *DataTags) DataLen() (l int)

DataLen returns the total length of all data fields in bytes.

func (*DataTags) InjectContent 

func (dts *DataTags) InjectContent(data []byte, w io.Writer) (nWritten int, _ error)

InjectContent inspects data argument and uses the data field in a DataTag type to injected the backend data at the current position in the data argument and then writes the output to w. DataTags must be a sorted slice. Usually this function receives the data from Entities.QueryResources(). This function can be called multiple times. It tracks the stream position and inserts the ESI tag once the correct position has been reached. This function cannot yet be used in parallel.

func (*DataTags) Len 

func (dts *DataTags) Len() int

func (*DataTags) Less 

func (dts *DataTags) Less(i, j int) bool

func (*DataTags) ResetStates 

func (dts *DataTags) ResetStates()

ResetStates exported for Benchmarks. Resets the internal state machine to re-run the injector without instantiating a new object.

func (*DataTags) String 

func (dts *DataTags) String() string

String used for debug output during development

func (*DataTags) Swap 

func (dts *DataTags) Swap(i, j int)

type Entities 

type Entities []*Entity

Entities represents a list of Tag tags found in one HTML page.

func Parse 

func Parse(r io.Reader) (Entities, error)

Parse parses a stream of data to extract Tag Tags. Malformed Tag tags won't trigger any errors, instead the parser skips them.

func (Entities) ApplyLogger 

func (et Entities) ApplyLogger(l log.Logger)

ApplyLogger sets a logger to each entity.

func (Entities) HasCoalesce 

func (et Entities) HasCoalesce() bool

HasCoalesce returns true if there is at least one tag with enabled coalesce feature.

func (Entities) ParseRaw 

func (et Entities) ParseRaw() error

ParseRaw parses all Tag tags

func (Entities) QueryResources 

func (et Entities) QueryResources(cTag chan<- DataTag, r *http.Request) error

QueryResources runs in parallel to query all available backend services / resources which are available in the current page. The channel DataTag must not be buffered. All create DataTag object will be written randomly into the channel. The developer must take care for the correct order. If the request gets canceled via its context then all resource requests gets cancelled too.

func (Entities) SplitCoalesce 

func (et Entities) SplitCoalesce() (coalesce Entities, nonCoalesce Entities)

SplitCoalesce creates two new slices whose entries contain either coalesce or non coalesce ESI tags. Returns always non-nil slices.

func (Entities) String 

func (et Entities) String() string

String for debugging only!

func (Entities) UniqueID 

func (et Entities) UniqueID() uint64

UniqueID calculates a unique ID for all tags in the slice.

type Entity 

type Entity struct {
	RawTag  []byte
	DataTag DataTag
	// OnError contains the content which gets injected into an erroneous Tag
	// tag when all reuqests are failing to its backends. If onError in the Tag
	// tag contains a file name, then that content gets loaded.
	OnError []byte
	Config
	// Race TODO(CyS) From the README: Add the attribute `race="true"` to fire
	// all resource requests at once and the one which is the fastest gets
	// served and the others dropped.
	Race bool
	// Resources contains multiple unique Resource entries, aka backend systems
	// likes redis instances or other micro services. Resources occur within one
	// single Tag tag. The resource attribute (src="") can occur multiple times.
	// The first item which successfully returns data gets its content used in
	// the response. If one item fails and we have multiple resources, the next
	// resource gets queried. All resources share the same scheme/protocol which
	// must handle the ResourceHandler.
	Resources []*Resource // Any 3rd party servers
	// Conditioner TODO(CyS) depending on a condition an Tag tag gets executed or not.
	Conditioner
}

Entity represents a single fully parsed Tag tag

func (*Entity) ParseRaw 

func (et *Entity) ParseRaw() error

ParseRaw parses the RawTag field and fills the remaining fields of the struct.

func (*Entity) QueryResources 

func (et *Entity) QueryResources(externalReq *http.Request) ([]byte, error)

QueryResources iterates sequentially over the resources and executes requests as defined in the ResourceHandler. If one resource fails it will be marked as timed out and the next resource gets tried. The exponential back-off stops when MaxBackOffs have been reached and then tries again. Returns a Temporary error behaviour when all requests to all resources have failed.

func (*Entity) SetDefaultConfig 

func (et *Entity) SetDefaultConfig(tag Config)

SetDefaultConfig used in PathConfig.UpsertESITags and in Entity.ParseRaw to set the pool function. When called in PathConfig.UpsertESITags all default config values have been applied correctly.

type Replacer 

type Replacer interface {
	Replace(string) string
}

Replacer is a type which can replace placeholder substrings in a string with actual values from a http.Request.

func MakeReplacer 

func MakeReplacer(r *http.Request, emptyValue string) Replacer

MakeReplacer makes a new replacer based on r which are used for request placeholders. Request placeholders are created immediately. emptyValue should be the string that is used in place of empty string (can still be empty string).

type Resource 

type Resource struct {
	// Index specifies the number of occurrence within the include tag to
	// allowing sorting and hence having a priority list.
	Index int
	// contains filtered or unexported fields
}

Resource specifies the location to a 3rd party remote system within an Tag tag. A resource attribute (src="") can occur n-times.

func MustNewResource 

func MustNewResource(idx int, url string) *Resource

MustNewResource same as NewResource but panics on error.

func NewResource 

func NewResource(idx int, url string) (*Resource, error)

NewResource creates a new resource to one backend. Inspects the URL if it contains a template and parses that template.

func (*Resource) CBFailures 

func (r *Resource) CBFailures() uint64

CBFailures number of failures. Thread safe.

func (*Resource) CBRecordFailure 

func (r *Resource) CBRecordFailure() (failedUnixNano int64)

CBRecordFailure records a failure and increases the internal counter. Returns the last failed time. Thread safe.

func (*Resource) CBReset 

func (r *Resource) CBReset()

CBReset resets the circuit breaker. Thread safe.

func (*Resource) CBState 

func (r *Resource) CBState() (state int, lastFailure time.Time)

CBState returns the current state of the circuit breaker and the last failure time. Thread safe.

func (*Resource) DoRequest 

func (r *Resource) DoRequest(args *ResourceArgs) (http.Header, []byte, error)

DoRequest performs the request to the backend resource. It generates the URL and then fires the request. DoRequest has the same signature as ResourceHandler

func (*Resource) String 

func (r *Resource) String() string

String returns a resource identifier, most likely the underlying URL and the template name, if defined.

type ResourceArgs 

type ResourceArgs struct {
	// ExternalReq external request object from the evil internet.
	ExternalReq *http.Request

	// URL defines the target URL to call or an alias name. This field gets set
	// from type Resource.url in the function Resource.DoRequest. Before passing
	// to DoRequest of the underlying implementation URL gets treated with
	// Replacer.
	URL string
	// Tag the configuration of a single ESI tag.
	Tag Config
	// contains filtered or unexported fields
}

ResourceArgs arguments to ResourceHandler.DoRequest. This type lives in a sync.Pool and the fields ExternalReq, repl and URL gets reset when putting it back into the pool.

func NewResourceArgs 

func NewResourceArgs(r *http.Request, url string, esi Config) *ResourceArgs

NewResourceArgs creates a new argument and initializes the internal string replacer.

func (*ResourceArgs) IsPostAllowed 

func (a *ResourceArgs) IsPostAllowed() bool

IsPostAllowed returns true if the forward post data config has been set to true and if we have a POST, PUT or PATCH request.

func (ResourceArgs) MarshalEasyJSON 

func (v ResourceArgs) MarshalEasyJSON(w *jwriter.Writer)

MarshalEasyJSON supports easyjson.Marshaler interface

func (ResourceArgs) MarshalJSON 

func (v ResourceArgs) MarshalJSON() ([]byte, error)

MarshalJSON supports json.Marshaler interface

func (*ResourceArgs) MarshalLog 

func (a *ResourceArgs) MarshalLog(kv log.KeyValuer) error

MarshalLog special crafted log format, does not log the external request

func (*ResourceArgs) MaxBodySizeHumanized 

func (a *ResourceArgs) MaxBodySizeHumanized() string

MaxBodySizeHumanized converts the bytes into a human readable format

func (*ResourceArgs) PrepareForwardHeaders 

func (a *ResourceArgs) PrepareForwardHeaders() []string

PrepareForwardHeaders returns all headers which must get forwarded to the backend resource. Returns a non-nil slice when no headers should be forwarded. Slice is balanced: i == key and i+1 == value

func (*ResourceArgs) PrepareReturnHeaders 

func (a *ResourceArgs) PrepareReturnHeaders(fromBE http.Header) http.Header

PrepareReturnHeaders extracts the required headers fromBE as defined in the struct fields ReturnHeaders*. fromBE means: From Back End. These are the headers from the queried backend resource. Might return a nil map.

func (*ResourceArgs) ReplaceKeyURLForTesting 

func (a *ResourceArgs) ReplaceKeyURLForTesting() *ResourceArgs

ReplaceKeyURLForTesting only used for integration tests in the backend package.

func (*ResourceArgs) UnmarshalEasyJSON 

func (v *ResourceArgs) UnmarshalEasyJSON(l *jlexer.Lexer)

UnmarshalEasyJSON supports easyjson.Unmarshaler interface

func (*ResourceArgs) UnmarshalJSON 

func (v *ResourceArgs) UnmarshalJSON(data []byte) error

UnmarshalJSON supports json.Unmarshaler interface

func (*ResourceArgs) Validate 

func (a *ResourceArgs) Validate() (err error)

Validate checks if required arguments have been set

func (*ResourceArgs) ValidateWithKey 

func (a *ResourceArgs) ValidateWithKey() (err error)

ValidateWithKey same as Validate but validates the key instead of the URL field.

type ResourceHandler 

type ResourceHandler interface {
	// DoRequest fires the request to the resource and it may return a header
	// and content or an error. All three return values can be nil. An error can
	// have the behaviour of NotFound which calls the next resource in the
	// sequence and does not trigger the circuit breaker. Any other returned
	// error will trigger the increment of the circuit breaker. See the variable
	// CBMaxFailures for the maximum amount of allowed failures until the
	// circuit breaker opens.
	DoRequest(*ResourceArgs) (header http.Header, content []byte, err error)
	// Closes closes the resource when Caddy restarts or reloads. If supported
	// by the resource.
	Close() error
}

ResourceHandler gets implemented by any client which is able to speak to any remote backend. A handler gets registered in a global map and has a long lived state.

func LookupResourceHandler 

func LookupResourceHandler(scheme string) (rf ResourceHandler, ok bool)

LookupResourceHandler uses the scheme/alias to retrieve the backend request function.

func NewResourceHandler 

func NewResourceHandler(opt *ResourceOptions) (ResourceHandler, error)

NewResourceHandler a given URL gets checked which service it should instantiate and connect to. A handler must be previously registered via function RegisterResourceHandlerFactory.

type ResourceHandlerFactoryFunc 

type ResourceHandlerFactoryFunc func(*ResourceOptions) (ResourceHandler, error)

ResourceHandlerFactoryFunc can create a new resource handler or an error.

type ResourceOptions 

type ResourceOptions struct {
	// Alias ,optional, can have any name which gets used in an Tag tag and
	// refers to the connection to a resource.
	Alias string
	// URL, required, defines the authentication and target to a resource. If an
	// URL contains the name of an Alias then the URl data from that alias will
	// be copied into this URL field.
	URL string
	// Query, optional, contains mostly a SQL query which runs as a prepared
	// statement so you must use the question mark or any other placeholder.
	Query string
}

ResourceOptions has the same structure as caddyesi.ResourceItem. Defines a single configuration item for creating a new backend resource, especially in ResourceHandlerFactoryFunc.

func NewResourceOptions 

func NewResourceOptions(url string, aliasQuery ...string) *ResourceOptions

NewResourceOptions creates a new option. URL is mandatory but alias and query are optional. Up to three arguments in total are supported.

func (*ResourceOptions) ParseNoSQLURL 

func (ro *ResourceOptions) ParseNoSQLURL() (address, password string, params url.Values, err error)

ParseNoSQLURL parses a given URL using a custom URI scheme. For example: 

redis://localhost:6379/?db=3
memcache://localhost:1313/?lazy=1
redis://:6380/?db=0 => connects to localhost:6380
redis:// => connects to localhost:6379 with DB 0
memcache:// => connects to localhost:11211
memcache://?server=localhost:11212&server=localhost:11213
	=> connects to: localhost:11211, localhost:11212, localhost:11213
redis://empty:myPassword@clusterName.xxxxxx.0001.usw2.cache.amazonaws.com:6379/?db=0

Available parameters: db, max_active (int, Connections), max_idle (int, Connections), idle_timeout (time.Duration, Connection), cancellable (0,1 request towards redis), lazy (0, 1 disables ping during connection setup). On successful parse the key "scheme" is also set in the params return value.

Source Files

View all Source files

Directories

Path Synopsis
Package backend provides integrations for 3rd party resources.
 Package backend provides integrations for 3rd party resources.
esigrpc
Package esigrpc is a generated protocol buffer package.
 Package esigrpc is a generated protocol buffer package.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.