pinqloq

README

Pinqloq (Go)

Pinqloq is a structured logging and log shipping SDK for centralized application logs. It captures HTTP request/response logs through standard net/http middleware and sends manual application events to the Pinqloq log management platform using in-memory buffering, batching, and HTTPS delivery. This is the Go counterpart of the .NET, Node.js, and Ruby pinqloq SDKs — same platform, same wire protocol, idiomatic API on each side.

Because the middleware is plain func(http.Handler) http.Handler, it works with any router built on net/http.Handler — chi, gorilla/mux, http.ServeMux, or a framework's own http.Handler adapter (e.g. Gin's gin.WrapH).

Features

• Automatic HTTP request/response logging via standard net/http middleware
• Correlation id read from the caller's header, falling back to a generated UUID
• Name-based redaction of sensitive fields, headers, and whole endpoints
• Manual structured application events
• Buffered and batched HTTPS delivery, backed by a single background goroutine and a Go channel
• Graceful shutdown flush, context-aware

Requirements

• Go 1.22 or later
• A Pinqloq project and secret key

Installation

go get github.com/pinqponq/pinqloq-go-sdk

Quick Start

Store your secret key in an environment variable or a secret manager. Do not hardcode production credentials.

package main

import (
	"context"
	"net/http"
	"os"

	pinqloq "github.com/pinqponq/pinqloq-go-sdk"
)

func main() {
	client, err := pinqloq.New(pinqloq.Options{
		SecretKey:             os.Getenv("PINQLOQ_SECRET_KEY"),
		APILogsCollectionName: "myapp_api_logs",
		DeviceIdentifier:      "myapp-instance-1",
	})
	if err != nil {
		panic(err)
	}
	defer client.Shutdown(context.Background())

	mux := http.NewServeMux()
	mux.HandleFunc("/orders", ordersHandler)

	middleware := client.Middleware(pinqloq.RequestLoggingOptions{
		ExcludePaths: []string{"/health"},
	})

	http.ListenAndServe(":8080", middleware(mux))
}

The middleware captures the HTTP method, path, and status code as searchable metadata. The request body, response body, request headers, and response headers go to the log detail as InputJson, OutputJson, RequestHeaders, and ResponseHeaders. Bodies are truncated at 32 KB.

Manual Logging

Call Enqueue directly on the client to send structured application events:

client.Enqueue(pinqloq.LogEntry{
	Event:            "order.created",
	DeviceIdentifier: order.CustomerID,
	LogLevel:         pinqloq.LogLevelInformation,
	LogSourceType:    pinqloq.LogSourceTypeBackend,
	Metadata:         map[string]string{"orderId": order.ID},
}, nil, nil)

client.Logger() still returns the same Logger interface — useful when you want to pass just the logging capability into a function or struct without handing it the whole client (middleware, shutdown, and all).

Event and DeviceIdentifier are required on every entry. Leave DeviceIdentifier unset on an entry to inherit the global Options.DeviceIdentifier. Enqueue returns an error if an entry has no DeviceIdentifier and no global fallback is set — a missing required field fails loudly rather than being silently dropped.

Add Request Metadata

By default the middleware reads the required DeviceIdentifier from the Device-Identifier request header automatically. Override how it is resolved with ResolveDeviceIdentifier; the override wins, and if it returns an empty string the middleware falls back to the Device-Identifier header, then to the global Options.DeviceIdentifier. If none of these resolve a value, the middleware rejects the request with HTTP 400 before it runs.

middleware := client.Middleware(pinqloq.RequestLoggingOptions{
	ExcludePaths: []string{"/health"},
	ResolveDeviceIdentifier: func(r *http.Request) string {
		return r.Header.Get("X-User-Id")
	},
	Metadata: map[string]pinqloq.EnricherFunc{
		"userId": func(r *http.Request, statusCode int, headers http.Header) string {
			return r.Header.Get("X-User-Id")
		},
	},
})

Use Metadata for searchable values such as user and tenant IDs. Use Detail for additional drill-down information. The event key (the panel title) defaults to "{method} {path}" and can be overridden via a Metadata["event"] enricher.

Correlation ID

Every log carries a CorrelationID that ties together the records of a single request or flow. The request-logging middleware fills it with no configuration: the caller's Correlation-Id request header when present, otherwise a generated UUID.

client.Enqueue(pinqloq.LogEntry{
	Event:            "order.created",
	DeviceIdentifier: order.CustomerID,
	CorrelationID:    currentCorrelationID,
}, nil, nil)

Redacting Sensitive Values

Request and response bodies and headers may contain credentials, tokens, or personal information. Unlike the .NET SDK's attribute-based redaction (which relies on C# reflection over typed DTOs — not available in Go's interface{}-based JSON decoding), this SDK redacts by name, exactly like the Node.js and Ruby SDKs:

• RedactFields — case-insensitive field/header names masked with *****REDACTED***** wherever they appear in a captured body or header, at any nesting depth.
• RedactPaths — path prefixes (matched the same way as ExcludePaths) where every value in InputJson, OutputJson, RequestHeaders, and ResponseHeaders is masked, keeping the JSON structure and header names intact.

middleware := client.Middleware(pinqloq.RequestLoggingOptions{
	RedactFields: []string{"ssnLastFour"},
	RedactPaths:  []string{"/payment"},
})

A built-in, unconditional floor of common credential names (password, token, Authorization, card numbers, ...) is always masked, even with no configuration — see redaction.go for the full list. pinqloq.NewRedactionPlan and pinqloq.ApplyBodyRedaction/pinqloq.SerializeHeaders are exported for a project running its own request-logging middleware that wants the same redaction behavior.

Security and Reliability

Logs are buffered in memory and sent in batches by a single background goroutine. Buffered logs may be lost if the process is terminated without a graceful shutdown — call client.Shutdown(ctx) on exit, passing a context with a deadline generous enough for the final flush.

Delivery failures are reported through onFailed callbacks and, even without callbacks, as throttled log lines via the standard log package — never silently discarded, but also never blocking. If your secret key is authorized for more than one collection, set APILogsCollectionName (or a per-entry CollectionName); otherwise the batch is rejected.

Reading a request body larger than 32 MB is capped (io.LimitReader) before restoring it for the next handler, to bound memory use on an unexpectedly large upload; only the first 32 KB of that buffer is ever sent to Pinqloq.

Documentation

• .NET SDK, Node.js SDK, Ruby SDK — the other implementations of this platform's wire protocol and feature set.
• Full documentation

License

MIT

Documentation

Index

• func ApplyBodyRedaction(body string, plan *RedactionPlan) string
• func ContainsAlwaysRedactedName(body string) bool
• func SerializeHeaders(headers http.Header, plan *RedactionPlan) string
• type Client
  • func New(opts Options) (*Client, error)
  • func (c *Client) Enqueue(entry LogEntry, onSent OnSent, onFailed OnFailed) (bool, error)
  • func (c *Client) EnqueueMany(entries []LogEntry, onSent OnSent, onFailed OnFailed) (int, error)
  • func (c *Client) Logger() Logger
  • func (c *Client) Middleware(opts RequestLoggingOptions) func(http.Handler) http.Handler
  • func (c *Client) Shutdown(ctx context.Context) error
• type EnricherFunc
• type LogEntry
• type LogError
  • func (e *LogError) Error() string
• type LogFailureReason
• type LogLevel
• type LogSourceType
• type Logger
• type OnFailed
• type OnSent
• type Options
• type RedactionPlan
  • func NewRedactionPlan(redactAll bool, declaredNames []string) *RedactionPlan
  • func (p *RedactionPlan) ContainsDeclaredName(body string) bool
  • func (p *RedactionPlan) HasDeclaredRedactions() bool
  • func (p *RedactionPlan) ShouldRedact(name string) bool
• type RequestLoggingOptions

Functions

func ApplyBodyRedaction

func ApplyBodyRedaction(body string, plan *RedactionPlan) string

ApplyBodyRedaction processes a request/response body according to the redaction plan. If plan.RedactAll, every JSON value is redacted; otherwise only fields the plan names (plus the built-in floor) are redacted. A body that mentions nothing sensitive is returned unprocessed.

func ContainsAlwaysRedactedName

func ContainsAlwaysRedactedName(body string) bool

ContainsAlwaysRedactedName reports whether a raw body mentions any always-redacted name, as a whole name.

func SerializeHeaders

func SerializeHeaders(headers http.Header, plan *RedactionPlan) string

SerializeHeaders converts an http.Header into single-line JSON, redacting values whose name matches the plan. Multi-value headers are joined with ", ".

Types

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client bundles a buffered, batched log shipper with an HTTP request-logging middleware factory.

func New

func New(opts Options) (*Client, error)

New creates a Pinqloq client: a buffered, batched log shipper plus an HTTP middleware factory.

func (*Client) Enqueue

func (c *Client) Enqueue(entry LogEntry, onSent OnSent, onFailed OnFailed) (bool, error)

Enqueue is a shortcut for Logger().Enqueue.

func (*Client) EnqueueMany

func (c *Client) EnqueueMany(entries []LogEntry, onSent OnSent, onFailed OnFailed) (int, error)

EnqueueMany is a shortcut for Logger().EnqueueMany.

func (*Client) Logger

func (c *Client) Logger() Logger

Logger sends structured application events; buffered and delivered in the background.

func (*Client) Middleware

func (c *Client) Middleware(opts RequestLoggingOptions) func(http.Handler) http.Handler

Middleware returns standard net/http middleware that captures every HTTP request, produces one API log, and enqueues it via the client's Logger. Compatible with any router built on net/http.Handler (chi, gorilla/mux, ServeMux, ...).

func (*Client) Shutdown

func (c *Client) Shutdown(ctx context.Context) error

Shutdown stops the background dispatcher and sends whatever is left in the queue, waiting up to ctx's deadline for delivery to finish.

type EnricherFunc

type EnricherFunc func(r *http.Request, statusCode int, responseHeaders http.Header) string

EnricherFunc computes a metadata/detail value from the request and the completed response.

type LogEntry

type LogEntry struct {
	LogLevel         LogLevel
	Event            string
	Date             time.Time
	AppVersionName   string
	DeviceIdentifier string
	LogSourceType    LogSourceType
	CollectionName   string
	CorrelationID    string
	Path             string
	Metadata         map[string]string
	Detail           map[string]string
}

LogEntry is a single log record. The producer creates it and places it on the queue.

type LogError

type LogError struct {
	Reason     LogFailureReason
	StatusCode int
	Message    string
	Cause      error
}

LogError carries the reason a log could not be sent; passed to an OnFailed callback.

func (*LogError) Error

func (e *LogError) Error() string

type LogFailureReason

type LogFailureReason string

LogFailureReason explains why a log could not be sent.

const (
	LogFailureReasonUnauthorized LogFailureReason = "Unauthorized"
	LogFailureReasonForbidden    LogFailureReason = "Forbidden"
	LogFailureReasonQueueFull    LogFailureReason = "QueueFull"
	LogFailureReasonHTTPError    LogFailureReason = "HttpError"
	LogFailureReasonTimeout      LogFailureReason = "Timeout"
	LogFailureReasonNetwork      LogFailureReason = "Network"
)

type LogLevel

type LogLevel int

LogLevel matches the server-side ClientLogLevel one to one.

const (
	LogLevelDebug       LogLevel = 1
	LogLevelInformation LogLevel = 2
	LogLevelWarning     LogLevel = 3
	LogLevelError       LogLevel = 4
	LogLevelFatal       LogLevel = 5
)

type LogSourceType

type LogSourceType int

LogSourceType matches the server-side ClientLogSourceType one to one.

const (
	LogSourceTypeDevice  LogSourceType = 1
	LogSourceTypeBackend LogSourceType = 2
)

type Logger

type Logger interface {
	Enqueue(entry LogEntry, onSent OnSent, onFailed OnFailed) (bool, error)
	EnqueueMany(entries []LogEntry, onSent OnSent, onFailed OnFailed) (int, error)
}

Logger is the main interface the Pinqloq SDK uses to send logs.

type OnFailed

type OnFailed func(entry LogEntry, err *LogError)

OnFailed is called when a log could not be sent.

type OnSent

type OnSent func(entry LogEntry)

OnSent is called when a log has been successfully delivered.

type Options

type Options struct {
	SecretKey             string
	APILogsCollectionName string
	BulkPath              string
	BatchSize             int
	FlushInterval         time.Duration
	QueueCapacity         int
	HTTPTimeout           time.Duration
	AppVersionName        string
	DeviceIdentifier      string
}

Options configures a Client.

type RedactionPlan

type RedactionPlan struct {
	RedactAll bool
	// contains filtered or unexported fields
}

RedactionPlan is the redaction decision for a request. RedactAll masks every field/header value wholesale (the RedactPaths equivalent of the .NET SDK's [PinqloqRedactEndpoint]); otherwise ShouldRedact is true for names configured via RedactFields plus the always-redacted floor.

func NewRedactionPlan

func NewRedactionPlan(redactAll bool, declaredNames []string) *RedactionPlan

NewRedactionPlan builds a plan that redacts the given declared field/header names (case insensitive) in addition to the built-in credential-name floor.

func (*RedactionPlan) ContainsDeclaredName

func (p *RedactionPlan) ContainsDeclaredName(body string) bool

ContainsDeclaredName reports whether a raw body mentions one of this plan's declared names, as a whole name.

func (*RedactionPlan) HasDeclaredRedactions

func (p *RedactionPlan) HasDeclaredRedactions() bool

HasDeclaredRedactions reports whether this plan redacts anything beyond the built-in floor.

func (*RedactionPlan) ShouldRedact

func (p *RedactionPlan) ShouldRedact(name string) bool

ShouldRedact reports whether the given JSON field or header name needs to be redacted.

type RequestLoggingOptions

type RequestLoggingOptions struct {
	ExcludePaths            []string
	ResolveDeviceIdentifier func(r *http.Request) string
	ResolveAppVersionName   func(r *http.Request) string
	Metadata                map[string]EnricherFunc
	Detail                  map[string]EnricherFunc
	RedactFields            []string
	RedactPaths             []string
}

RequestLoggingOptions configures Client.Middleware.