Documentation

Overview

    Package esitag provides a parser, types and configurations for <esi:include tags.

    Index

    Constants

    View Source
    const (
    	CBStateOpen = iota + 1
    	CBStateHalfOpen
    	CBStateClosed
    )

      CBState declares the different states for the circuit breaker (CB)

      View Source
      const DefaultTimeOut = 20 * time.Second

        DefaultTimeOut defines the default timeout to a backend resource.

        View Source
        const MaxSizeESITag = 4096

          MaxSizeESITag maximum size of an Tag tag. For now this value has been returned from a dice roll.

          Variables

          View Source
          var CBMaxFailures uint64 = 12

            CBMaxFailures maximum amount of failures before the circuit breaker is half open to try the next request.

            View Source
            var CBThresholdCalc = func(failures uint64) time.Duration {
            	return (1 << failures) * time.Second
            }

              CBThresholdCalc calculates the threshold how long the CB should wait until to set the HalfOpen state. Default implementation returns an exponentially calculated duration

              View Source
              var DropHeadersForward = map[string]bool{
              	"Cache-Control": true,
              	"Connection":    true,
              	"Host":          true,
              	"Pragma":        true,
              	"Upgrade":       true,
              }

                DropHeadersForward a list of headers which should never be forwarded to the backend resource. Initial idea of excluded fields. https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

                View Source
                var DropHeadersReturn = map[string]bool{
                	"Cache-Control":             true,
                	"Connection":                true,
                	"Content-Disposition":       true,
                	"Content-Encoding":          true,
                	"Content-Length":            true,
                	"Content-Range":             true,
                	"Content-Type":              true,
                	"Date":                      true,
                	"Etag":                      true,
                	"Expires":                   true,
                	"Last-Modified":             true,
                	"Location":                  true,
                	"Status":                    true,
                	"Strict-Transport-Security": true,
                	"Trailer":                   true,
                	"Transfer-Encoding":         true,
                	"Upgrade":                   true,
                }

                  DropHeadersReturn a list of headers which should never be forwarded to the client. Initial idea of excluded fields. https://en.wikipedia.org/wiki/List_of_HTTP_header_fields

                  Functions

                  func CloseAllResourceHandler

                  func CloseAllResourceHandler() error

                    CloseAllResourceHandler does what the function name says returns the first occurred error.

                    func DeregisterResourceHandler

                    func DeregisterResourceHandler(scheme string)

                      DeregisterResourceHandler removes a previously registered scheme/alias.

                      func RegisterResourceHandler

                      func RegisterResourceHandler(scheme string, f ResourceHandler) struct{ DeferredDeregister func() }

                        RegisterResourceHandler scheme can be a protocol before the :// but also an alias to register a key-value service. This function returns a closure which lets you deregister the scheme/alias once a test has finished. Use the defer word. Scheme/alias will be transformed into an all lowercase string.

                        func RegisterResourceHandlerFactory

                        func RegisterResourceHandlerFactory(scheme string, f ResourceHandlerFactoryFunc)

                          RegisterResourceHandlerFactory registers a new factory function to create a new ResourceHandler. Useful when you have entries in the resources_config.xml|json file.

                          func SplitAttributes

                          func SplitAttributes(raw string) ([]string, error)

                            SplitAttributes splits an Tag tag by its attributes. This function avoids regexp.

                            Types

                            type Conditioner

                            type Conditioner interface {
                            	OK(r *http.Request) bool
                            }

                              Conditioner does not represent your favorite shampoo but it gives you the possibility to define an expression which gets executed for every request to include the Tag resource or not.

                              type Config

                              type Config struct {
                              	Log               log.Logger // optional
                              	ForwardHeaders    []string   // optional, already treated with http.CanonicalHeaderKey
                              	ReturnHeaders     []string   // optional, already treated with http.CanonicalHeaderKey
                              	ForwardPostData   bool       // optional
                              	ForwardHeadersAll bool       // optional
                              	ReturnHeadersAll  bool       // optional
                              	// Coalesce will merge n-external parallel requests into one resource
                              	// backend request.
                              	Coalesce bool
                              	// PrintDebug injects the time taken into the returned data as hidden HTML
                              	// comment in function Entities.QueryResources. It also provides the raw tag
                              	// and in future some other data for easier debugging.
                              	PrintDebug bool
                              	// Timeout maximum time needed for a backend request before the cancellation
                              	// context kills it.
                              	Timeout time.Duration // required
                              	// TTL retrieved content from a backend can live this time in the middleware
                              	// cache.
                              	TTL time.Duration // optional
                              	// MaxBodySize allowed max body size to read from the backend resource.
                              	MaxBodySize uint64 // required
                              	// Key defines the name of the key in an NoSQL service or as additional
                              	// identifier in a gRPC request.
                              	Key string
                              }

                                Config provides the configuration of a single Tag tag. This information gets passed on as an argument towards the backend resources and enriches the Entity type.

                                type DataTag

                                type DataTag struct {
                                	Data  []byte // Data from the micro service gathered in a goroutine. Can be nil.
                                	Start int    // Start position in the stream
                                	End   int    // End position in the stream. Never smaller than Start.
                                }

                                  DataTag identifies an Tag tag by its start and end position in the HTML byte stream for replacing. If the HTML changes there needs to be a refresh call to re-parse the HTML.

                                  func (DataTag) String

                                  func (dt DataTag) String() string

                                    String prints human readable the data tag for debugging purposes.

                                    type DataTags

                                    type DataTags struct {
                                    	Slice []DataTag
                                    	// contains filtered or unexported fields
                                    }

                                      DataTags a list of tags with their position within a page and the content

                                      func NewDataTagsCapped

                                      func NewDataTagsCapped(cap int) *DataTags

                                        NewDataTagsCapped creates a new object with a DataTag slice and its maximum capacity.

                                        func (*DataTags) DataLen

                                        func (dts *DataTags) DataLen() (l int)

                                          DataLen returns the total length of all data fields in bytes.

                                          func (*DataTags) InjectContent

                                          func (dts *DataTags) InjectContent(data []byte, w io.Writer) (nWritten int, _ error)

                                            InjectContent inspects data argument and uses the data field in a DataTag type to injected the backend data at the current position in the data argument and then writes the output to w. DataTags must be a sorted slice. Usually this function receives the data from Entities.QueryResources(). This function can be called multiple times. It tracks the stream position and inserts the ESI tag once the correct position has been reached. This function cannot yet be used in parallel.

                                            func (*DataTags) Len

                                            func (dts *DataTags) Len() int

                                            func (*DataTags) Less

                                            func (dts *DataTags) Less(i, j int) bool

                                            func (*DataTags) ResetStates

                                            func (dts *DataTags) ResetStates()

                                              ResetStates exported for Benchmarks. Resets the internal state machine to re-run the injector without instantiating a new object.

                                              func (*DataTags) String

                                              func (dts *DataTags) String() string

                                                String used for debug output during development

                                                func (*DataTags) Swap

                                                func (dts *DataTags) Swap(i, j int)

                                                type Entities

                                                type Entities []*Entity

                                                  Entities represents a list of Tag tags found in one HTML page.

                                                  func Parse

                                                  func Parse(r io.Reader) (Entities, error)

                                                    Parse parses a stream of data to extract Tag Tags. Malformed Tag tags won't trigger any errors, instead the parser skips them.

                                                    func (Entities) ApplyLogger

                                                    func (et Entities) ApplyLogger(l log.Logger)

                                                      ApplyLogger sets a logger to each entity.

                                                      func (Entities) HasCoalesce

                                                      func (et Entities) HasCoalesce() bool

                                                        HasCoalesce returns true if there is at least one tag with enabled coalesce feature.

                                                        func (Entities) ParseRaw

                                                        func (et Entities) ParseRaw() error

                                                          ParseRaw parses all Tag tags

                                                          func (Entities) QueryResources

                                                          func (et Entities) QueryResources(cTag chan<- DataTag, r *http.Request) error

                                                            QueryResources runs in parallel to query all available backend services / resources which are available in the current page. The channel DataTag must not be buffered. All create DataTag object will be written randomly into the channel. The developer must take care for the correct order. If the request gets canceled via its context then all resource requests gets cancelled too.

                                                            func (Entities) SplitCoalesce

                                                            func (et Entities) SplitCoalesce() (coalesce Entities, nonCoalesce Entities)

                                                              SplitCoalesce creates two new slices whose entries contain either coalesce or non coalesce ESI tags. Returns always non-nil slices.

                                                              func (Entities) String

                                                              func (et Entities) String() string

                                                                String for debugging only!

                                                                func (Entities) UniqueID

                                                                func (et Entities) UniqueID() uint64

                                                                  UniqueID calculates a unique ID for all tags in the slice.

                                                                  type Entity

                                                                  type Entity struct {
                                                                  	RawTag  []byte
                                                                  	DataTag DataTag
                                                                  	// OnError contains the content which gets injected into an erroneous Tag
                                                                  	// tag when all reuqests are failing to its backends. If onError in the Tag
                                                                  	// tag contains a file name, then that content gets loaded.
                                                                  	OnError []byte
                                                                  	Config
                                                                  	// Race TODO(CyS) From the README: Add the attribute `race="true"` to fire
                                                                  	// all resource requests at once and the one which is the fastest gets
                                                                  	// served and the others dropped.
                                                                  	Race bool
                                                                  	// Resources contains multiple unique Resource entries, aka backend systems
                                                                  	// likes redis instances or other micro services. Resources occur within one
                                                                  	// single Tag tag. The resource attribute (src="") can occur multiple times.
                                                                  	// The first item which successfully returns data gets its content used in
                                                                  	// the response. If one item fails and we have multiple resources, the next
                                                                  	// resource gets queried. All resources share the same scheme/protocol which
                                                                  	// must handle the ResourceHandler.
                                                                  	Resources []*Resource // Any 3rd party servers
                                                                  	// Conditioner TODO(CyS) depending on a condition an Tag tag gets executed or not.
                                                                  	Conditioner
                                                                  }

                                                                    Entity represents a single fully parsed Tag tag

                                                                    func (*Entity) ParseRaw

                                                                    func (et *Entity) ParseRaw() error

                                                                      ParseRaw parses the RawTag field and fills the remaining fields of the struct.

                                                                      func (*Entity) QueryResources

                                                                      func (et *Entity) QueryResources(externalReq *http.Request) ([]byte, error)

                                                                        QueryResources iterates sequentially over the resources and executes requests as defined in the ResourceHandler. If one resource fails it will be marked as timed out and the next resource gets tried. The exponential back-off stops when MaxBackOffs have been reached and then tries again. Returns a Temporary error behaviour when all requests to all resources have failed.

                                                                        func (*Entity) SetDefaultConfig

                                                                        func (et *Entity) SetDefaultConfig(tag Config)

                                                                          SetDefaultConfig used in PathConfig.UpsertESITags and in Entity.ParseRaw to set the pool function. When called in PathConfig.UpsertESITags all default config values have been applied correctly.

                                                                          type Replacer

                                                                          type Replacer interface {
                                                                          	Replace(string) string
                                                                          }

                                                                            Replacer is a type which can replace placeholder substrings in a string with actual values from a http.Request.

                                                                            func MakeReplacer

                                                                            func MakeReplacer(r *http.Request, emptyValue string) Replacer

                                                                              MakeReplacer makes a new replacer based on r which are used for request placeholders. Request placeholders are created immediately. emptyValue should be the string that is used in place of empty string (can still be empty string).

                                                                              type Resource

                                                                              type Resource struct {
                                                                              	// Index specifies the number of occurrence within the include tag to
                                                                              	// allowing sorting and hence having a priority list.
                                                                              	Index int
                                                                              	// contains filtered or unexported fields
                                                                              }

                                                                                Resource specifies the location to a 3rd party remote system within an Tag tag. A resource attribute (src="") can occur n-times.

                                                                                func MustNewResource

                                                                                func MustNewResource(idx int, url string) *Resource

                                                                                  MustNewResource same as NewResource but panics on error.

                                                                                  func NewResource

                                                                                  func NewResource(idx int, url string) (*Resource, error)

                                                                                    NewResource creates a new resource to one backend. Inspects the URL if it contains a template and parses that template.

                                                                                    func (*Resource) CBFailures

                                                                                    func (r *Resource) CBFailures() uint64

                                                                                      CBFailures number of failures. Thread safe.

                                                                                      func (*Resource) CBRecordFailure

                                                                                      func (r *Resource) CBRecordFailure() (failedUnixNano int64)

                                                                                        CBRecordFailure records a failure and increases the internal counter. Returns the last failed time. Thread safe.

                                                                                        func (*Resource) CBReset

                                                                                        func (r *Resource) CBReset()

                                                                                          CBReset resets the circuit breaker. Thread safe.

                                                                                          func (*Resource) CBState

                                                                                          func (r *Resource) CBState() (state int, lastFailure time.Time)

                                                                                            CBState returns the current state of the circuit breaker and the last failure time. Thread safe.

                                                                                            func (*Resource) DoRequest

                                                                                            func (r *Resource) DoRequest(args *ResourceArgs) (http.Header, []byte, error)

                                                                                              DoRequest performs the request to the backend resource. It generates the URL and then fires the request. DoRequest has the same signature as ResourceHandler

                                                                                              func (*Resource) String

                                                                                              func (r *Resource) String() string

                                                                                                String returns a resource identifier, most likely the underlying URL and the template name, if defined.

                                                                                                type ResourceArgs

                                                                                                type ResourceArgs struct {
                                                                                                	// ExternalReq external request object from the evil internet.
                                                                                                	ExternalReq *http.Request
                                                                                                
                                                                                                	// URL defines the target URL to call or an alias name. This field gets set
                                                                                                	// from type Resource.url in the function Resource.DoRequest. Before passing
                                                                                                	// to DoRequest of the underlying implementation URL gets treated with
                                                                                                	// Replacer.
                                                                                                	URL string
                                                                                                	// Tag the configuration of a single ESI tag.
                                                                                                	Tag Config
                                                                                                	// contains filtered or unexported fields
                                                                                                }

                                                                                                  ResourceArgs arguments to ResourceHandler.DoRequest. This type lives in a sync.Pool and the fields ExternalReq, repl and URL gets reset when putting it back into the pool.

                                                                                                  func NewResourceArgs

                                                                                                  func NewResourceArgs(r *http.Request, url string, esi Config) *ResourceArgs

                                                                                                    NewResourceArgs creates a new argument and initializes the internal string replacer.

                                                                                                    func (*ResourceArgs) IsPostAllowed

                                                                                                    func (a *ResourceArgs) IsPostAllowed() bool

                                                                                                      IsPostAllowed returns true if the forward post data config has been set to true and if we have a POST, PUT or PATCH request.

                                                                                                      func (ResourceArgs) MarshalEasyJSON

                                                                                                      func (v ResourceArgs) MarshalEasyJSON(w *jwriter.Writer)

                                                                                                        MarshalEasyJSON supports easyjson.Marshaler interface

                                                                                                        func (ResourceArgs) MarshalJSON

                                                                                                        func (v ResourceArgs) MarshalJSON() ([]byte, error)

                                                                                                          MarshalJSON supports json.Marshaler interface

                                                                                                          func (*ResourceArgs) MarshalLog

                                                                                                          func (a *ResourceArgs) MarshalLog(kv log.KeyValuer) error

                                                                                                            MarshalLog special crafted log format, does not log the external request

                                                                                                            func (*ResourceArgs) MaxBodySizeHumanized

                                                                                                            func (a *ResourceArgs) MaxBodySizeHumanized() string

                                                                                                              MaxBodySizeHumanized converts the bytes into a human readable format

                                                                                                              func (*ResourceArgs) PrepareForwardHeaders

                                                                                                              func (a *ResourceArgs) PrepareForwardHeaders() []string

                                                                                                                PrepareForwardHeaders returns all headers which must get forwarded to the backend resource. Returns a non-nil slice when no headers should be forwarded. Slice is balanced: i == key and i+1 == value

                                                                                                                func (*ResourceArgs) PrepareReturnHeaders

                                                                                                                func (a *ResourceArgs) PrepareReturnHeaders(fromBE http.Header) http.Header

                                                                                                                  PrepareReturnHeaders extracts the required headers fromBE as defined in the struct fields ReturnHeaders*. fromBE means: From Back End. These are the headers from the queried backend resource. Might return a nil map.

                                                                                                                  func (*ResourceArgs) ReplaceKeyURLForTesting

                                                                                                                  func (a *ResourceArgs) ReplaceKeyURLForTesting() *ResourceArgs

                                                                                                                    ReplaceKeyURLForTesting only used for integration tests in the backend package.

                                                                                                                    func (*ResourceArgs) UnmarshalEasyJSON

                                                                                                                    func (v *ResourceArgs) UnmarshalEasyJSON(l *jlexer.Lexer)

                                                                                                                      UnmarshalEasyJSON supports easyjson.Unmarshaler interface

                                                                                                                      func (*ResourceArgs) UnmarshalJSON

                                                                                                                      func (v *ResourceArgs) UnmarshalJSON(data []byte) error

                                                                                                                        UnmarshalJSON supports json.Unmarshaler interface

                                                                                                                        func (*ResourceArgs) Validate

                                                                                                                        func (a *ResourceArgs) Validate() (err error)

                                                                                                                          Validate checks if required arguments have been set

                                                                                                                          func (*ResourceArgs) ValidateWithKey

                                                                                                                          func (a *ResourceArgs) ValidateWithKey() (err error)

                                                                                                                            ValidateWithKey same as Validate but validates the key instead of the URL field.

                                                                                                                            type ResourceHandler

                                                                                                                            type ResourceHandler interface {
                                                                                                                            	// DoRequest fires the request to the resource and it may return a header
                                                                                                                            	// and content or an error. All three return values can be nil. An error can
                                                                                                                            	// have the behaviour of NotFound which calls the next resource in the
                                                                                                                            	// sequence and does not trigger the circuit breaker. Any other returned
                                                                                                                            	// error will trigger the increment of the circuit breaker. See the variable
                                                                                                                            	// CBMaxFailures for the maximum amount of allowed failures until the
                                                                                                                            	// circuit breaker opens.
                                                                                                                            	DoRequest(*ResourceArgs) (header http.Header, content []byte, err error)
                                                                                                                            	// Closes closes the resource when Caddy restarts or reloads. If supported
                                                                                                                            	// by the resource.
                                                                                                                            	Close() error
                                                                                                                            }

                                                                                                                              ResourceHandler gets implemented by any client which is able to speak to any remote backend. A handler gets registered in a global map and has a long lived state.

                                                                                                                              func LookupResourceHandler

                                                                                                                              func LookupResourceHandler(scheme string) (rf ResourceHandler, ok bool)

                                                                                                                                LookupResourceHandler uses the scheme/alias to retrieve the backend request function.

                                                                                                                                func NewResourceHandler

                                                                                                                                func NewResourceHandler(opt *ResourceOptions) (ResourceHandler, error)

                                                                                                                                  NewResourceHandler a given URL gets checked which service it should instantiate and connect to. A handler must be previously registered via function RegisterResourceHandlerFactory.

                                                                                                                                  type ResourceHandlerFactoryFunc

                                                                                                                                  type ResourceHandlerFactoryFunc func(*ResourceOptions) (ResourceHandler, error)

                                                                                                                                    ResourceHandlerFactoryFunc can create a new resource handler or an error.

                                                                                                                                    type ResourceOptions

                                                                                                                                    type ResourceOptions struct {
                                                                                                                                    	// Alias ,optional, can have any name which gets used in an Tag tag and
                                                                                                                                    	// refers to the connection to a resource.
                                                                                                                                    	Alias string
                                                                                                                                    	// URL, required, defines the authentication and target to a resource. If an
                                                                                                                                    	// URL contains the name of an Alias then the URl data from that alias will
                                                                                                                                    	// be copied into this URL field.
                                                                                                                                    	URL string
                                                                                                                                    	// Query, optional, contains mostly a SQL query which runs as a prepared
                                                                                                                                    	// statement so you must use the question mark or any other placeholder.
                                                                                                                                    	Query string
                                                                                                                                    }

                                                                                                                                      ResourceOptions has the same structure as caddyesi.ResourceItem. Defines a single configuration item for creating a new backend resource, especially in ResourceHandlerFactoryFunc.

                                                                                                                                      func NewResourceOptions

                                                                                                                                      func NewResourceOptions(url string, aliasQuery ...string) *ResourceOptions

                                                                                                                                        NewResourceOptions creates a new option. URL is mandatory but alias and query are optional. Up to three arguments in total are supported.

                                                                                                                                        func (*ResourceOptions) ParseNoSQLURL

                                                                                                                                        func (ro *ResourceOptions) ParseNoSQLURL() (address, password string, params url.Values, err error)

                                                                                                                                          ParseNoSQLURL parses a given URL using a custom URI scheme. For example:

                                                                                                                                          redis://localhost:6379/?db=3
                                                                                                                                          memcache://localhost:1313/?lazy=1
                                                                                                                                          redis://:6380/?db=0 => connects to localhost:6380
                                                                                                                                          redis:// => connects to localhost:6379 with DB 0
                                                                                                                                          memcache:// => connects to localhost:11211
                                                                                                                                          memcache://?server=localhost:11212&server=localhost:11213
                                                                                                                                          	=> connects to: localhost:11211, localhost:11212, localhost:11213
                                                                                                                                          redis://empty:myPassword@clusterName.xxxxxx.0001.usw2.cache.amazonaws.com:6379/?db=0
                                                                                                                                          

                                                                                                                                          Available parameters: db, max_active (int, Connections), max_idle (int, Connections), idle_timeout (time.Duration, Connection), cancellable (0,1 request towards redis), lazy (0, 1 disables ping during connection setup). On successful parse the key "scheme" is also set in the params return value.

                                                                                                                                          Directories

                                                                                                                                          Path Synopsis
                                                                                                                                          Package backend provides integrations for 3rd party resources.
                                                                                                                                          Package backend provides integrations for 3rd party resources.
                                                                                                                                          esigrpc
                                                                                                                                          Package esigrpc is a generated protocol buffer package.
                                                                                                                                          Package esigrpc is a generated protocol buffer package.