ekalog_writer_http

package
v2.8.15 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 6, 2021 License: MIT Imports: 11 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	DATADOG_ADDR_US = "https://http-intake.logs.datadoghq.com/v1/input"
	DATADOG_ADDR_EU = "https://http-intake.logs.datadoghq.eu/v1/input"
)

noinspection GoSnakeCaseUsage

View Source 
const (
	ROLLBAR_ADDR = "https://api.rollbar.com/api/1/items/"
)

noinspection GoSnakeCaseUsage

Variables

View Source 
var (
	ErrWriterIsNil      = fmt.Errorf("CI_WriterHttp: writer is nil (not initialized)")
	ErrWriterDisabled   = fmt.Errorf("CI_WriterHttp: writer is disabled (stopped)")
	ErrWriterBufferFull = fmt.Errorf("CI_WriterHttp: writer's buffer is full")
)

Functions

This section is empty.

Types

type CI_WriterHttp 

type CI_WriterHttp struct {
	// contains filtered or unexported fields
}

CI_WriterHttp is a type that implements an io.Writer - legacy Golang interface, doing write encoded log's entry as []byte to the some HTTP service using desired API.

Features: -----------

  1. Async transport. When you calling Write() it just pushes encoded entry (as []byte) to the worker and does not blocks the routine. Spawn as many workers (at the initialization) as you want. See SetWorkersNum() method.
  1. Thread-safe. You may call CI_WriterHttp to as many goroutines as you want. The CommonIntegrator under the hood just calls Write() for all writers they holds on.

  1. Accumulated bulk deferred requests. Has an internal buffer of encoded log entries (as []byte) (see SetBufferCap() method), each worker pulling data from and putting to its own internal buffer (see SetWorkerBufferCap() method).

    When worker buffer is full or when it's time to flush the accumulated data (the stopping of app or flush time has come), sends accumulated data to the HTTP service.

  1. Control how encoded log entries will be combined before sending. Each worker aggregates a single encoded log entries with others to the log entries pack (contains many encoded log entries) before send it.

    You may set HOW they will be combined:

    - Need to put something to buffer before first encoded log entry added? (see AddBefore() method).

    - Need to put something to buffer after last encoded log entry added? (see AddAfter() method).

    - Need to put something to buffer between encoded log entries? I mean between all of them, excluding before first and after last. (see AddBetween() method).

  1. Flushing data each N time intervals. Each worker flushes accumulated data on demand (the internal buffer is full) or at the timeout you may set (see SetWorkerAutoFlushDelay() method).

  1. Slow down when service unavailable or network error. If request sending at the some worker is failed (it's only network or log service issue if you configured writer well), the usage of HTTP API service will slow down, logging what happens and why and saving (not dropping) your logs to the internal buffer.

    They will send when connection will be restored. You may set the deferred log entries pack buffer capacity (pack, because each worker builds a pack of log entries, remember? see p.3) (see SetDeferredBufferCap() method).

  1. Graceful shutdown. Of course, if you're familiar of ekadeath package. If you're not yet, it's time to: https://github.com/qioalice/ekago/ekadeath .

    When you calling ekadeath.Die(), ekadeath.Exit() or writing a log with the level that marked as fatal, you won't lost aggregated logs!

    The package will sends the rest of logs for the last time for you. And you do not need to do something for that.

    Need more? No problem, RegisterGracefulShutdown() allows you to specify context, using which you may finally disable CI_WriterHttp and a sync.WaitGroup, using which you may be sure, that you get your control only when all accumulated logs are flushed. You may specify only one of them.

  1. Auto-initialization: You do need to call methods like Start() or something like that. You may check whether you configuration is valid calling Ping() method, but it's not necessary.

    Just call all configuration methods with the chaining style and pass CI_WriterHttp object to the CommonIntegrator's WriteTo() method (or to your own logging integrator) and there is! The CI_WriterHttp will be initialized at the first Write() call.

  1. Configurable to use any service. It's a very customizable type using which you may stream your logs safely to the log aggregation services like: - DataDog: https://www.datadoghq.com/ : UseProviderDataDog(), - Rollbar: https://rollbar.com/ : UseProviderRollbar(), - GrayLog: https://www.graylog.org/ : UseProviderGrayLog(), - Sentry: https://sentry.io/ : UseProviderSentry(), etc.

    For those services, CI_WriterHttp has a methods (3rd column) that makes it easy to configure it for, but you may configure it manually.

    If you want to manually set HTTP service use UseProviderManual() method, providing a fasthttp's Request initializer for your service.

  1. Fast. Uses fasthttp ( https://github.com/valyala/fasthttp ) under the hood, as http client. Pools, reusing, caching, optimizations. All you need.

--------

WARNING! DO NOT CALL Write() or Ping() METHODS UNTIL YOU FINISH ALL PREPARATIONS! DO NOT PASS WRITER TO THE CommonIntegrator's WriteTo() METHOD UNTIL YOU FINISH ALL PREPARATIONS! IF YOU DO, THE CHANGES WILL NOT BE SAVED!

WARNING! PANIC CAUTION! YOU MUST SET THE LOG SERVICE YOU WANT TO WRITE LOG ENTRIES TO. CHOOSE A PREDEFINED OR USE YOUR OWN. IF YOU DO NOT DO THAT, THE INITIALIZATION WILL PANIC!

func (*CI_WriterHttp) AddAfter 

func (dw *CI_WriterHttp) AddAfter(data []byte) *CI_WriterHttp

AddAfter sets the data that will be added to the encoded entries pack's buffer after the last encoded entry is added.

Take a look: If your log service provider accepts many records as JSON array of objects, 'data' is "]" (JSON ending of array char).

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) AddAfterS added in v2.8.15 

func (dw *CI_WriterHttp) AddAfterS(data string) *CI_WriterHttp

AddAfterS is the same as AddAfter() but accepts string instead of []byte, doing no-copy conversion.

func (*CI_WriterHttp) AddBefore 

func (dw *CI_WriterHttp) AddBefore(data []byte) *CI_WriterHttp

AddBefore sets the data that will be added to the encoded entries pack's buffer before the first encoded entry is added.

Take a look: If your log service provider accepts many records as JSON array of objects, 'data' is "[" (JSON beginning of array char).

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) AddBeforeAfterBetween added in v2.8.15 

func (dw *CI_WriterHttp) AddBeforeAfterBetween(args ...[]byte) *CI_WriterHttp

AddBeforeAfterBetween replaces all of AddBefore(), AddAfter(), AddBetween() calls.

It accepts 1 or 3 arguments. If 3 arguments is provided, they passed to corresponding calls. If 1 argument is passed, and it's length is multiple of 3, it will be split to the equal length 3 pieces, and they are passed to corresponding calls.

All other variants of arguments are ignored and does no-op.

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) AddBeforeAfterBetweenS added in v2.8.15 

func (dw *CI_WriterHttp) AddBeforeAfterBetweenS(args ...string) *CI_WriterHttp

AddBeforeAfterBetweenS is the same as AddBeforeAfterBetweenS() but accepts strings instead of [][]byte, doing no-copy conversion.

func (*CI_WriterHttp) AddBeforeS added in v2.8.15 

func (dw *CI_WriterHttp) AddBeforeS(data string) *CI_WriterHttp

AddBeforeS is the same as AddBefore() but accepts string instead of []byte, doing no-copy conversion.

func (*CI_WriterHttp) AddBetween 

func (dw *CI_WriterHttp) AddBetween(data []byte) *CI_WriterHttp

AddBetween sets the data that will be added to the encoded entries pack's buffer between encoded log entries (but neither before first nor after last).

Take a look: If your log service provider accepts many records as JSON array of objects, 'data' is "," (JSON separator of objects inside an array).

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) AddBetweenS added in v2.8.15 

func (dw *CI_WriterHttp) AddBetweenS(data string) *CI_WriterHttp

AddBetweenS is the same as AddBetween() but accepts string instead of []byte, doing no-copy conversion.

func (*CI_WriterHttp) Ping 

func (dw *CI_WriterHttp) Ping(cb ...func(req *fasthttp.Request)) *ekaerr.Error

Ping checks whether provider settings are correct and connection can be established.

Keep in mind, that if you do not do ping by yourself, the CI_WriterHttp will try to ping when Write() will be called first time (initialization). But if you call method Ping(), there will be no internal Ping() call at the initialization. You may prefer explicit Ping() call, because here you may specify some callbacks 'cb' that are applied to the HTTP request before it will be send.

WARNING! If you call Ping() by yourself and it returns an error, you may change something (provider settings, etc), and then try to ping again. But if Ping() is called at the initialization, the CI_WriterHttp can not be recover.

func (*CI_WriterHttp) RegisterGracefulShutdown added in v2.8.15 

func (dw *CI_WriterHttp) RegisterGracefulShutdown(ctx context.Context, wg *sync.WaitGroup) *CI_WriterHttp

RegisterGracefulShutdown allows you to pass context.Context and sync.WaitGroup, that will be used to provide you graceful shutdown, meaning:

  1. Context. Specify, when running CI_WriterHttp must be disabled.
  1. sync.WaitGroup. If specified, one second before all workers are started, your waitgroup's counter will be increased, and it will be decreased, when all of them are stopped, guaranteeing to you, when disabling is requested, flushing all accumulated logs is important.

Read p.1, p.8 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

You may pass only context or only sync.WaitGroup. It's OK.

func (*CI_WriterHttp) SetBufferCap 

func (dw *CI_WriterHttp) SetBufferCap(cap uint32) *CI_WriterHttp

SetBufferCap sets a limit of internal pool of encoded []byte entries, to which Write() method places them, and where they are extracted from later for being processed and sent.

If this cap is reached, Write() will be IGNORED all next entries, until old ones are processed.

So, you need to specify this buffer as big as it must be never reached, to keep your logs safe even at the load, but not as big to spend gigabytes of RAM for them. Keep in mind, that buffer stores only pointers to []byte (it's up to 24 bytes per item), but the data of []byte also takes a RAM.

Read p.1, p.3 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

Allowed range: [256..1'048'576] (2**8..2**20). Do not overwrite default value if you're not understand how much RAM your log entries consume per item, and what RAM consumption will be at the upper bound. Default: 4096.

func (*CI_WriterHttp) SetDeferredBufferCap 

func (dw *CI_WriterHttp) SetDeferredBufferCap(cap uint32) *CI_WriterHttp

SetDeferredBufferCap looks like SetBufferCap(), but sets a capacity of those encoded []byte entries, that is tried to be sent, while CI_WriterHttp is temporary disabled.

Usually, you set this bigger than value used in SetBufferCap(), to store your entries when your provider is not available you don't know why, because exactly in this buffer all those entries that are tried to be processed, while your provider is not available, will be stored.

If is set to 0, the entries will be discarded when CI_WriterHttp is temporary disabled.

Read p.1, p.3, p.6 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

Allowed range: [0..8'388'608]. Do not overwrite default value if you're not understand how much RAM your log entries consume per item, and what RAM consumption will be at the upper bound. Default: 16384.

func (*CI_WriterHttp) SetWorkerAutoFlushDelay 

func (dw *CI_WriterHttp) SetWorkerAutoFlushDelay(delay time.Duration) *CI_WriterHttp

SetWorkerAutoFlushDelay sets how often accumulated []byte entries will be sent to your provider, even if their buffer is not full.

Read p.3, p.5 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

Allowed range: [100ms..24h]. Default: 10s.

func (*CI_WriterHttp) SetWorkerBufferCap 

func (dw *CI_WriterHttp) SetWorkerBufferCap(cap uint16) *CI_WriterHttp

SetWorkerBufferCap sets a limit of each worker's internal pool of encoded []byte entries, when that accumulated set will be sent to your provider.

Less entries may be sent (if timeout that you may set by SetWorkerAutoFlushDelay() is reached), but this value tells a maximum.

Read p.1, p.3 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

Allowed range: [1..16384]. Default: 32.

Hint. It's useful when your service may accept a bulk requests. Do less requests but much loaded. But if your provider doesn't accept bulk requests, you may need to set this to 1.

func (*CI_WriterHttp) SetWorkersNum 

func (dw *CI_WriterHttp) SetWorkersNum(num uint16) *CI_WriterHttp

SetWorkersNum sets how much goroutines will be spawned to handle all passed encoded []byte entries and send them to your provider.

Read p.1 of CI_WriterHttp doc for more info.

Does nothing, if CI_WriterHttp already running, stopped or disabled (Write() has been called at least once).

Allowed range: [1..32]. Set high values only if there is a really high throughput is required. Default: 2. Recommended: [1..4].

func (*CI_WriterHttp) UseProviderDataDog 

func (dw *CI_WriterHttp) UseProviderDataDog(addr, token string) *CI_WriterHttp

UseProviderDataDog setups CI_WriterHttp for DataDog log service provider ( https://www.datadoghq.com/ ).

You MUST specify 'addr' as desired DataDog's HTTP addr (you may use predefined constants DATADOG_ADDR_US, DATADOG_ADDR_EU) or use your own and DataDog service's token as 'token'.

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) UseProviderManual 

func (dw *CI_WriterHttp) UseProviderManual(

	cb func(req *fasthttp.Request),
	bodyPreparer ...func(reader io.Reader) io.Reader,

) *CI_WriterHttp

UseProviderManual is a log service provider manual configurator. You MUST specify a callback that will set-up an HTTP request for desired provider.

A 2nd argument allows you to modify generated HTTP request's body before it will be sent. Your callback (if provided) must return a new body, that will be used instead of old one.

WARNING! You MUST NOT save and reuse old body that you receiver in your 2nd callback!

Nil safe. There is no-op if CI_WriterHttp already initialized.

func (*CI_WriterHttp) UseProviderRollbar added in v2.8.15 

func (dw *CI_WriterHttp) UseProviderRollbar(token string) *CI_WriterHttp

func (*CI_WriterHttp) Write 

func (dw *CI_WriterHttp) Write(p []byte) (n int, err error)

Write sends 'p' to the internal entries being processed buffer and returns len(p) and nil if 'p' has been successfully queued.

Initializes CI_WriterHttp object if it's not. If initialization once failed, the CI_WriterHttp can not be used anymore.

Returned errors:

  • nil: OK, 'p' has been queued.
  • ErrWriterIsNil: CI_WriterHttp receiver is nil.
  • ErrWriterDisabled: CI_WriterHttp is stopped and will never start again.
  • ErrWriterBufferFull: Internal CI_WriterHttp's buffer of processed entries is full. Next time set bigger buffer's length using SetBufferCap().

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.