config

Documentation

Index

• type Config
  • func NewConfig() *Config
  • func Parse(file io.Reader) (*Config, error)
  • func (c *Config) Bind() string
  • func (c *Config) CrawleraURL() string
  • func (c *Config) MaybeDoNotVerifyCrawleraCert(value bool)
  • func (c *Config) MaybeSetAPIKey(value string)
  • func (c *Config) MaybeSetAdblockLists(value []string)
  • func (c *Config) MaybeSetBindIP(value net.IP)
  • func (c *Config) MaybeSetBindPort(value int)
  • func (c *Config) MaybeSetConcurrentConnections(value int)
  • func (c *Config) MaybeSetCrawleraHost(value string)
  • func (c *Config) MaybeSetCrawleraPort(value int)
  • func (c *Config) MaybeSetDebug(value bool)
  • func (c *Config) MaybeSetDirectAccessExceptHostPathRegexps(value []string)
  • func (c *Config) MaybeSetDirectAccessHostPathRegexps(value []string)
  • func (c *Config) MaybeSetNoAutoSessions(value bool)
  • func (c *Config) MaybeSetProxyAPIIP(value net.IP)
  • func (c *Config) MaybeSetProxyAPIPort(value int)
  • func (c *Config) MaybeSetTLSCaCertificate(value string)
  • func (c *Config) MaybeSetTLSPrivateKey(value string)
  • func (c *Config) SetXHeader(key, value string)

Types

type Config

type Config struct {
	Debug                             bool     `toml:"debug"`
	DoNotVerifyCrawleraCert           bool     `toml:"dont_verify_crawlera_cert"`
	NoAutoSessions                    bool     `toml:"no_auto_sessions"`
	ConcurrentConnections             int      `toml:"concurrent_connections"`
	BindPort                          int      `toml:"bind_port"`
	CrawleraPort                      int      `toml:"crawlera_port"`
	ProxyAPIPort                      int      `toml:"proxy_api_port"`
	BindIP                            string   `toml:"bind_ip"`
	ProxyAPIIP                        string   `toml:"proxy_api_ip"`
	APIKey                            string   `toml:"api_key"`
	CrawleraHost                      string   `toml:"crawlera_host"`
	TLSCaCertificate                  string   `toml:"tls_ca_certificate"`
	TLSPrivateKey                     string   `toml:"tls_private_key"`
	AdblockLists                      []string `toml:"adblock_lists"`
	DirectAccessHostPathRegexps       []string `toml:"direct_access_hostpath_regexps"`
	DirectAccessExceptHostPathRegexps []string `toml:"direct_access_except_hostpath_regexps"`
	XHeaders                          map[string]string
}

Config stores global configuration data of the application.

func NewConfig

func NewConfig() *Config

NewConfig returns new instance of configuration data structure with fields set to sensible defaults.

func Parse

func Parse(file io.Reader) (*Config, error)

Parse processes incoming file handler (usually, an instance of *os.File) and returns an instance of Config with fields set.

Basically, new Config instance gets its fields in this order:

1. Defaults
2. Values from the config file.

func (*Config) Bind

func (c *Config) Bind() string

Bind returns a string for the http.ListenAndServe based on config information.

func (*Config) CrawleraURL

func (c *Config) CrawleraURL() string

CrawleraURL builds and returns URL to crawlera. Basically, this is required for http.ProxyURL to have embedded credentials etc.

func (*Config) MaybeDoNotVerifyCrawleraCert

func (c *Config) MaybeDoNotVerifyCrawleraCert(value bool)

MaybeDoNotVerifyCrawleraCert defines is it necessary to verify Crawlera TLS certificate. If given value is not defined (false) then changes nothing.

func (*Config) MaybeSetAPIKey

func (c *Config) MaybeSetAPIKey(value string)

MaybeSetAPIKey sets an API key of Crawlera. If given value is not defined ("") then changes nothing.

func (*Config) MaybeSetAdblockLists

func (c *Config) MaybeSetAdblockLists(value []string)

MaybeSetAdblockLists sets a list to URLs

func (*Config) MaybeSetBindIP

func (c *Config) MaybeSetBindIP(value net.IP)

MaybeSetBindIP sets an IP crawlera-headless-proxy should listen on. If given value is not defined (0) then changes nothing.

If you want to have a global access (which is not recommended) please set it to 0.0.0.0.

func (*Config) MaybeSetBindPort

func (c *Config) MaybeSetBindPort(value int)

MaybeSetBindPort sets a port crawlera-headless-proxy should listen on. If given value is not defined (0) then changes nothing.

func (*Config) MaybeSetConcurrentConnections

func (c *Config) MaybeSetConcurrentConnections(value int)

MaybeSetConcurrentConnections sets a number of concurrent connections if necessary.

func (*Config) MaybeSetCrawleraHost

func (c *Config) MaybeSetCrawleraHost(value string)

MaybeSetCrawleraHost sets a host of Crawlera (usually it is 'proxy.crawlera.com'). If given value is not defined ("") then changes nothing.

func (*Config) MaybeSetCrawleraPort

func (c *Config) MaybeSetCrawleraPort(value int)

MaybeSetCrawleraPort a port Crawlera is listening to (usually it is 8010). If given value is not defined (0) then changes nothing.

func (*Config) MaybeSetDebug

func (c *Config) MaybeSetDebug(value bool)

MaybeSetDebug enabled debug mode of crawlera-headless-proxy (verbosity mostly). If given value is not defined (false) then changes nothing.

func (*Config) MaybeSetDirectAccessExceptHostPathRegexps

func (c *Config) MaybeSetDirectAccessExceptHostPathRegexps(value []string)

MaybeSetDirectAccessExceptHostPathRegexps sets a list of regular expressions for proxied access. Takes priority over DirectAccessHostPathRegexps.

func (*Config) MaybeSetDirectAccessHostPathRegexps

func (c *Config) MaybeSetDirectAccessHostPathRegexps(value []string)

MaybeSetDirectAccessHostPathRegexps sets a list of regular expressions for direct access.

func (*Config) MaybeSetNoAutoSessions

func (c *Config) MaybeSetNoAutoSessions(value bool)

MaybeSetNoAutoSessions defines is it is required to enable automatic session management or not.

func (*Config) MaybeSetProxyAPIIP

func (c *Config) MaybeSetProxyAPIIP(value net.IP)

MaybeSetProxyAPIIP sets an ip for own API of crawlera-headless-proxy. If given value is not defined ("") then changes nothing.

func (*Config) MaybeSetProxyAPIPort

func (c *Config) MaybeSetProxyAPIPort(value int)

MaybeSetProxyAPIPort sets a port for own API of crawlera-headless-proxy. If given value is not defined (0) then changes nothing.

func (*Config) MaybeSetTLSCaCertificate

func (c *Config) MaybeSetTLSCaCertificate(value string)

MaybeSetTLSCaCertificate sets a content of the given file as TLS CA certificate.

func (*Config) MaybeSetTLSPrivateKey

func (c *Config) MaybeSetTLSPrivateKey(value string)

MaybeSetTLSPrivateKey sets a content of the given file as TLS private key.

func (*Config) SetXHeader

func (c *Config) SetXHeader(key, value string)

SetXHeader sets a header value of Crawlera X-Header. It is actually allowed to pass values in both ways: with full name (x-crawlera-profile) for example, and in the short form: just 'profile'. This effectively the same.