Package esitag provides a parser, types and configurations for <esi:include tags.



const (
	CBStateOpen = iota + 1

CBState declares the different states for the circuit breaker (CB)

const DefaultTimeOut = 20 * time.Second

DefaultTimeOut defines the default timeout to a backend resource.

const MaxSizeESITag = 4096

MaxSizeESITag maximum size of an Tag tag. For now this value has been returned from a dice roll.


var CBMaxFailures uint64 = 12

CBMaxFailures maximum amount of failures before the circuit breaker is half open to try the next request.

var CBThresholdCalc = func(failures uint64) time.Duration {
	return (1 << failures) * time.Second

CBThresholdCalc calculates the threshold how long the CB should wait until to set the HalfOpen state. Default implementation returns an exponentially calculated duration

var DropHeadersForward = map[string]bool{
	"Cache-Control": true,
	"Connection":    true,
	"Host":          true,
	"Pragma":        true,
	"Upgrade":       true,

DropHeadersForward a list of headers which should never be forwarded to the backend resource. Initial idea of excluded fields.

var DropHeadersReturn = map[string]bool{
	"Cache-Control":             true,
	"Connection":                true,
	"Content-Disposition":       true,
	"Content-Encoding":          true,
	"Content-Length":            true,
	"Content-Range":             true,
	"Content-Type":              true,
	"Date":                      true,
	"Etag":                      true,
	"Expires":                   true,
	"Last-Modified":             true,
	"Location":                  true,
	"Status":                    true,
	"Strict-Transport-Security": true,
	"Trailer":                   true,
	"Transfer-Encoding":         true,
	"Upgrade":                   true,

DropHeadersReturn a list of headers which should never be forwarded to the client. Initial idea of excluded fields.


func CloseAllResourceHandler

func CloseAllResourceHandler() error

CloseAllResourceHandler does what the function name says returns the first occurred error.

func DeregisterResourceHandler

func DeregisterResourceHandler(scheme string)

DeregisterResourceHandler removes a previously registered scheme/alias.

func RegisterResourceHandler

func RegisterResourceHandler(scheme string, f ResourceHandler) struct{ DeferredDeregister func() }

RegisterResourceHandler scheme can be a protocol before the :// but also an alias to register a key-value service. This function returns a closure which lets you deregister the scheme/alias once a test has finished. Use the defer word. Scheme/alias will be transformed into an all lowercase string.

func RegisterResourceHandlerFactory

func RegisterResourceHandlerFactory(scheme string, f ResourceHandlerFactoryFunc)

RegisterResourceHandlerFactory registers a new factory function to create a new ResourceHandler. Useful when you have entries in the resources_config.xml|json file.

func SplitAttributes

func SplitAttributes(raw string) ([]string, error)

SplitAttributes splits an Tag tag by its attributes. This function avoids regexp.


type Conditioner

type Conditioner interface {
	OK(r *http.Request) bool

Conditioner does not represent your favorite shampoo but it gives you the possibility to define an expression which gets executed for every request to include the Tag resource or not.

type Config

type Config struct {
	Log               log.Logger // optional
	ForwardHeaders    []string   // optional, already treated with http.CanonicalHeaderKey
	ReturnHeaders     []string   // optional, already treated with http.CanonicalHeaderKey
	ForwardPostData   bool       // optional
	ForwardHeadersAll bool       // optional
	ReturnHeadersAll  bool       // optional
	// Coalesce will merge n-external parallel requests into one resource
	// backend request.
	Coalesce bool
	// PrintDebug injects the time taken into the returned data as hidden HTML
	// comment in function Entities.QueryResources. It also provides the raw tag
	// and in future some other data for easier debugging.
	PrintDebug bool
	// Timeout maximum time needed for a backend request before the cancellation
	// context kills it.
	Timeout time.Duration // required
	// TTL retrieved content from a backend can live this time in the middleware
	// cache.
	TTL time.Duration // optional
	// MaxBodySize allowed max body size to read from the backend resource.
	MaxBodySize uint64 // required
	// Key defines the name of the key in an NoSQL service or as additional
	// identifier in a gRPC request.
	Key string

Config provides the configuration of a single Tag tag. This information gets passed on as an argument towards the backend resources and enriches the Entity type.

type DataTag

type DataTag struct {
	Data  []byte // Data from the micro service gathered in a goroutine. Can be nil.
	Start int    // Start position in the stream
	End   int    // End position in the stream. Never smaller than Start.

DataTag identifies an Tag tag by its start and end position in the HTML byte stream for replacing. If the HTML changes there needs to be a refresh call to re-parse the HTML.

func (DataTag) String

func (dt DataTag) String() string

String prints human readable the data tag for debugging purposes.

type DataTags

type DataTags struct {
	Slice []DataTag
	// contains filtered or unexported fields

DataTags a list of tags with their position within a page and the content

func NewDataTagsCapped

func NewDataTagsCapped(cap int) *DataTags

NewDataTagsCapped creates a new object with a DataTag slice and its maximum capacity.

func (*DataTags) DataLen

func (dts *DataTags) DataLen() (l int)

DataLen returns the total length of all data fields in bytes.

func (*DataTags) InjectContent

func (dts *DataTags) InjectContent(data []byte, w io.Writer) (nWritten int, _ error)

InjectContent inspects data argument and uses the data field in a DataTag type to injected the backend data at the current position in the data argument and then writes the output to w. DataTags must be a sorted slice. Usually this function receives the data from Entities.QueryResources(). This function can be called multiple times. It tracks the stream position and inserts the ESI tag once the correct position has been reached. This function cannot yet be used in parallel.

func (*DataTags) Len

func (dts *DataTags) Len() int

func (*DataTags) Less

func (dts *DataTags) Less(i, j int) bool

func (*DataTags) ResetStates

func (dts *DataTags) ResetStates()

ResetStates exported for Benchmarks. Resets the internal state machine to re-run the injector without instantiating a new object.

func (*DataTags) String

func (dts *DataTags) String() string

String used for debug output during development

func (*DataTags) Swap

func (dts *DataTags) Swap(i, j int)

type Entities

type Entities []*Entity

Entities represents a list of Tag tags found in one HTML page.

func Parse

func Parse(r io.Reader) (Entities, error)

Parse parses a stream of data to extract Tag Tags. Malformed Tag tags won't trigger any errors, instead the parser skips them.

func (Entities) ApplyLogger

func (et Entities) ApplyLogger(l log.Logger)

ApplyLogger sets a logger to each entity.

func (Entities) HasCoalesce

func (et Entities) HasCoalesce() bool

HasCoalesce returns true if there is at least one tag with enabled coalesce feature.

func (Entities) ParseRaw

func (et Entities) ParseRaw() error

ParseRaw parses all Tag tags

func (Entities) QueryResources

func (et Entities) QueryResources(cTag chan<- DataTag, r *http.Request) error

QueryResources runs in parallel to query all available backend services / resources which are available in the current page. The channel DataTag must not be buffered. All create DataTag object will be written randomly into the channel. The developer must take care for the correct order. If the request gets canceled via its context then all resource requests gets cancelled too.

func (Entities) SplitCoalesce

func (et Entities) SplitCoalesce() (coalesce Entities, nonCoalesce Entities)

SplitCoalesce creates two new slices whose entries contain either coalesce or non coalesce ESI tags. Returns always non-nil slices.

func (Entities) String

func (et Entities) String() string

String for debugging only!

func (Entities) UniqueID

func (et Entities) UniqueID() uint64

UniqueID calculates a unique ID for all tags in the slice.

type Entity

type Entity struct {
	RawTag  []byte
	DataTag DataTag
	// OnError contains the content which gets injected into an erroneous Tag
	// tag when all reuqests are failing to its backends. If onError in the Tag
	// tag contains a file name, then that content gets loaded.
	OnError []byte
	// Race TODO(CyS) From the README: Add the attribute `race="true"` to fire
	// all resource requests at once and the one which is the fastest gets
	// served and the others dropped.
	Race bool
	// Resources contains multiple unique Resource entries, aka backend systems
	// likes redis instances or other micro services. Resources occur within one
	// single Tag tag. The resource attribute (src="") can occur multiple times.
	// The first item which successfully returns data gets its content used in
	// the response. If one item fails and we have multiple resources, the next
	// resource gets queried. All resources share the same scheme/protocol which
	// must handle the ResourceHandler.
	Resources []*Resource // Any 3rd party servers
	// Conditioner TODO(CyS) depending on a condition an Tag tag gets executed or not.

Entity represents a single fully parsed Tag tag

func (*Entity) ParseRaw

func (et *Entity) ParseRaw() error

ParseRaw parses the RawTag field and fills the remaining fields of the struct.

func (*Entity) QueryResources

func (et *Entity) QueryResources(externalReq *http.Request) ([]byte, error)

QueryResources iterates sequentially over the resources and executes requests as defined in the ResourceHandler. If one resource fails it will be marked as timed out and the next resource gets tried. The exponential back-off stops when MaxBackOffs have been reached and then tries again. Returns a Temporary error behaviour when all requests to all resources have failed.

func (*Entity) SetDefaultConfig

func (et *Entity) SetDefaultConfig(tag Config)

SetDefaultConfig used in PathConfig.UpsertESITags and in Entity.ParseRaw to set the pool function. When called in PathConfig.UpsertESITags all default config values have been applied correctly.

type Replacer

type Replacer interface {
	Replace(string) string

Replacer is a type which can replace placeholder substrings in a string with actual values from a http.Request.

func MakeReplacer

func MakeReplacer(r *http.Request, emptyValue string) Replacer

MakeReplacer makes a new replacer based on r which are used for request placeholders. Request placeholders are created immediately. emptyValue should be the string that is used in place of empty string (can still be empty string).

type Resource

type Resource struct {
	// Index specifies the number of occurrence within the include tag to
	// allowing sorting and hence having a priority list.
	Index int
	// contains filtered or unexported fields

Resource specifies the location to a 3rd party remote system within an Tag tag. A resource attribute (src="") can occur n-times.

func MustNewResource

func MustNewResource(idx int, url string) *Resource

MustNewResource same as NewResource but panics on error.

func NewResource

func NewResource(idx int, url string) (*Resource, error)

NewResource creates a new resource to one backend. Inspects the URL if it contains a template and parses that template.

func (*Resource) CBFailures

func (r *Resource) CBFailures() uint64

CBFailures number of failures. Thread safe.

func (*Resource) CBRecordFailure

func (r *Resource) CBRecordFailure() (failedUnixNano int64)

CBRecordFailure records a failure and increases the internal counter. Returns the last failed time. Thread safe.

func (*Resource) CBReset

func (r *Resource) CBReset()

CBReset resets the circuit breaker. Thread safe.

func (*Resource) CBState

func (r *Resource) CBState() (state int, lastFailure time.Time)

CBState returns the current state of the circuit breaker and the last failure time. Thread safe.

func (*Resource) DoRequest

func (r *Resource) DoRequest(args *ResourceArgs) (http.Header, []byte, error)

DoRequest performs the request to the backend resource. It generates the URL and then fires the request. DoRequest has the same signature as ResourceHandler

func (*Resource) String

func (r *Resource) String() string

String returns a resource identifier, most likely the underlying URL and the template name, if defined.

type ResourceArgs

type ResourceArgs struct {
	// ExternalReq external request object from the evil internet.
	ExternalReq *http.Request

	// URL defines the target URL to call or an alias name. This field gets set
	// from type Resource.url in the function Resource.DoRequest. Before passing
	// to DoRequest of the underlying implementation URL gets treated with
	// Replacer.
	URL string
	// Tag the configuration of a single ESI tag.
	Tag Config
	// contains filtered or unexported fields

ResourceArgs arguments to ResourceHandler.DoRequest. This type lives in a sync.Pool and the fields ExternalReq, repl and URL gets reset when putting it back into the pool.

func NewResourceArgs

func NewResourceArgs(r *http.Request, url string, esi Config) *ResourceArgs

NewResourceArgs creates a new argument and initializes the internal string replacer.

func (*ResourceArgs) IsPostAllowed

func (a *ResourceArgs) IsPostAllowed() bool

IsPostAllowed returns true if the forward post data config has been set to true and if we have a POST, PUT or PATCH request.

func (ResourceArgs) MarshalEasyJSON

func (v ResourceArgs) MarshalEasyJSON(w *jwriter.Writer)

MarshalEasyJSON supports easyjson.Marshaler interface

func (ResourceArgs) MarshalJSON

func (v ResourceArgs) MarshalJSON() ([]byte, error)

MarshalJSON supports json.Marshaler interface

func (*ResourceArgs) MarshalLog

func (a *ResourceArgs) MarshalLog(kv log.KeyValuer) error

MarshalLog special crafted log format, does not log the external request

func (*ResourceArgs) MaxBodySizeHumanized

func (a *ResourceArgs) MaxBodySizeHumanized() string

MaxBodySizeHumanized converts the bytes into a human readable format

func (*ResourceArgs) PrepareForwardHeaders

func (a *ResourceArgs) PrepareForwardHeaders() []string

PrepareForwardHeaders returns all headers which must get forwarded to the backend resource. Returns a non-nil slice when no headers should be forwarded. Slice is balanced: i == key and i+1 == value

func (*ResourceArgs) PrepareReturnHeaders

func (a *ResourceArgs) PrepareReturnHeaders(fromBE http.Header) http.Header

PrepareReturnHeaders extracts the required headers fromBE as defined in the struct fields ReturnHeaders*. fromBE means: From Back End. These are the headers from the queried backend resource. Might return a nil map.

func (*ResourceArgs) ReplaceKeyURLForTesting

func (a *ResourceArgs) ReplaceKeyURLForTesting() *ResourceArgs

ReplaceKeyURLForTesting only used for integration tests in the backend package.

func (*ResourceArgs) UnmarshalEasyJSON

func (v *ResourceArgs) UnmarshalEasyJSON(l *jlexer.Lexer)

UnmarshalEasyJSON supports easyjson.Unmarshaler interface

func (*ResourceArgs) UnmarshalJSON

func (v *ResourceArgs) UnmarshalJSON(data []byte) error

UnmarshalJSON supports json.Unmarshaler interface

func (*ResourceArgs) Validate

func (a *ResourceArgs) Validate() (err error)

Validate checks if required arguments have been set

func (*ResourceArgs) ValidateWithKey

func (a *ResourceArgs) ValidateWithKey() (err error)

ValidateWithKey same as Validate but validates the key instead of the URL field.

type ResourceHandler

type ResourceHandler interface {
	// DoRequest fires the request to the resource and it may return a header
	// and content or an error. All three return values can be nil. An error can
	// have the behaviour of NotFound which calls the next resource in the
	// sequence and does not trigger the circuit breaker. Any other returned
	// error will trigger the increment of the circuit breaker. See the variable
	// CBMaxFailures for the maximum amount of allowed failures until the
	// circuit breaker opens.
	DoRequest(*ResourceArgs) (header http.Header, content []byte, err error)
	// Closes closes the resource when Caddy restarts or reloads. If supported
	// by the resource.
	Close() error

ResourceHandler gets implemented by any client which is able to speak to any remote backend. A handler gets registered in a global map and has a long lived state.

func LookupResourceHandler

func LookupResourceHandler(scheme string) (rf ResourceHandler, ok bool)

LookupResourceHandler uses the scheme/alias to retrieve the backend request function.

func NewResourceHandler

func NewResourceHandler(opt *ResourceOptions) (ResourceHandler, error)

NewResourceHandler a given URL gets checked which service it should instantiate and connect to. A handler must be previously registered via function RegisterResourceHandlerFactory.

type ResourceHandlerFactoryFunc

type ResourceHandlerFactoryFunc func(*ResourceOptions) (ResourceHandler, error)

ResourceHandlerFactoryFunc can create a new resource handler or an error.

type ResourceOptions

type ResourceOptions struct {
	// Alias ,optional, can have any name which gets used in an Tag tag and
	// refers to the connection to a resource.
	Alias string
	// URL, required, defines the authentication and target to a resource. If an
	// URL contains the name of an Alias then the URl data from that alias will
	// be copied into this URL field.
	URL string
	// Query, optional, contains mostly a SQL query which runs as a prepared
	// statement so you must use the question mark or any other placeholder.
	Query string

ResourceOptions has the same structure as caddyesi.ResourceItem. Defines a single configuration item for creating a new backend resource, especially in ResourceHandlerFactoryFunc.

func NewResourceOptions

func NewResourceOptions(url string, aliasQuery ...string) *ResourceOptions

NewResourceOptions creates a new option. URL is mandatory but alias and query are optional. Up to three arguments in total are supported.

func (*ResourceOptions) ParseNoSQLURL

func (ro *ResourceOptions) ParseNoSQLURL() (address, password string, params url.Values, err error)

ParseNoSQLURL parses a given URL using a custom URI scheme. For example:

redis://:6380/?db=0 => connects to localhost:6380
redis:// => connects to localhost:6379 with DB 0
memcache:// => connects to localhost:11211
	=> connects to: localhost:11211, localhost:11212, localhost:11213

Available parameters: db, max_active (int, Connections), max_idle (int, Connections), idle_timeout (time.Duration, Connection), cancellable (0,1 request towards redis), lazy (0, 1 disables ping during connection setup). On successful parse the key "scheme" is also set in the params return value.


Path Synopsis
backend Package backend provides integrations for 3rd party resources.
backend/esigrpc Package esigrpc is a generated protocol buffer package.