convoy

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Dec 25, 2021 License: MPL-2.0 Imports: 16 Imported by: 22

README

Convoy

golangci-lint Build and run all tests

convoy image

Convoy is a fast & secure webhooks service. It receives event data from a HTTP API and sends these event data to the configured endpoints. To get started download the openapi spec into Postman or Insomnia.

It includes the following features

  • Sign payload: Configure hash function to use in signing payload.
  • Retry events: Retry events to endpoints.
  • Delivery Attempt Logs: View request headers and body as well as response headers and body.
  • Rich UI: To easily debug and retry failed events.

Install

There are various ways of installing Convoy.

Precompiled binaries

Precompiled binaries for released versions are available in the releases section on Github.

Docker images

Docker images are available on Github Container Registry.

You can launch a Convoy Container to try it out with

$ docker run \
	-p 5005:5005 \
	--name convoy-server \
	-v `pwd`/convoy.json:/convoy.json \
	ghcr.io/frain-dev/convoy:v0.2.5

You can download a sample configuration of convoy.json.

Building from source

To build Convoy from source code, you need:

git clone https://github.com/frain-dev/convoy.git && cd convoy
go build -o convoy ./convoy

Concepts

  1. Apps: An app is an abstraction representing a user who wants to receive webhooks. Currently, an app contains one endpoint to receive webhooks.
  2. Events: An event represents a webhook event to be sent to an app.
  3. Delivery Attempts: A delivery attempt represents an attempt to send an event to it's respective app's endpoint. It contains the event body, status code and response body received on attempt. The amount of attempts on a failed delivery depends on your configured retry strategy.

Configuration

Convoy is configured using a json file with a sample configuration below:

{
	"database": {
		"dsn": "mongo-url-with-username-and-password"
	},
	"queue": {
		"type": "redis",
		"redis": {
			"dsn": "redis-url-with-username-and-password"
		}
	},
	"server": {
		"http": {
			"port": 5005
		}
	},
	"auth": {
		"type": "none"
	},
	"strategy": {
		"type": "default",
		"default": {
			"intervalSeconds": 125,
			"retryLimit": 15
		}
	},
	"signature": {
		"header": "X-Company-Event-Webhook-Signature"
	}
}
Notes to Configuration
  • You can set basic auth mechanism with the following:
{
	"auth": {
		"type": "basic",
		"basic": {
			"username": "username",
			"password": "password"
		}
	}
}

License

Mozilla Public License v2.0

Documentation

Index

Constants

View Source 
const (
	// With this Convoy will not process more than 3000
	// concurrent requests per minute. We use github.com/go-chi/httprate
	// which uses a sliding window algorithm, so we should be fine :)
	// TODO(subomi): We need to configure rate limiting to be per
	// client as well as implement distributed limiting backed by
	// a distributed key value store.
	RATE_LIMIT          = 5000
	RATE_LIMIT_DURATION = 1 * time.Minute
)

Variables

View Source 
var (
	ErrApplicationNotFound = errors.New("application not found")

	ErrEndpointNotFound = errors.New("endpoint not found")
)
View Source 
var (
	ErrEventDeliveryNotFound        = errors.New("event not found")
	ErrEventDeliveryAttemptNotFound = errors.New("delivery attempt not found")
)
View Source 
var DefaultLevels = []log.Level{
	log.ErrorLevel,
	log.PanicLevel,
	log.FatalLevel,
	log.WarnLevel,
}
View Source 
var (
	ErrEventNotFound = errors.New("event not found")
)
View Source 
var ErrGroupNotFound = errors.New("group not found")
View Source 
var PeriodValues = map[string]Period{
	"daily":   Daily,
	"weekly":  Weekly,
	"monthly": Monthly,
	"yearly":  Yearly,
}

Functions

func IsValidPeriod added in v0.1.0 

func IsValidPeriod(period string) bool

Types

type AppMetadata added in v0.1.0 

type AppMetadata struct {
	UID          string `json:"uid" bson:"uid"`
	Title        string `json:"title" bson:"title"`
	GroupID      string `json:"group_id" bson:"group_id"`
	SupportEmail string `json:"support_email" bson:"support_email"`
}

type Application added in v0.1.0 

type Application struct {
	ID           primitive.ObjectID `json:"-" bson:"_id"`
	UID          string             `json:"uid" bson:"uid"`
	GroupID      string             `json:"group_id" bson:"group_id"`
	Title        string             `json:"name" bson:"title"`
	SupportEmail string             `json:"support_email" bson:"support_email"`

	Endpoints []Endpoint         `json:"endpoints" bson:"endpoints"`
	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`

	Events int64 `json:"events" bson:"-"`

	DocumentStatus DocumentStatus `json:"-" bson:"document_status"`
}

type ApplicationRepository added in v0.1.0 

type ApplicationRepository interface {
	CreateApplication(context.Context, *Application) error
	LoadApplicationsPaged(context.Context, string, models.Pageable) ([]Application, pager.PaginationData, error)
	FindApplicationByID(context.Context, string) (*Application, error)
	UpdateApplication(context.Context, *Application) error
	DeleteApplication(context.Context, *Application) error
	LoadApplicationsPagedByGroupId(context.Context, string, models.Pageable) ([]Application, pager.PaginationData, error)
	SearchApplicationsByGroupId(context.Context, string, models.SearchParams) ([]Application, error)
	FindApplicationEndpointByID(context.Context, string, string) (*Endpoint, error)
	UpdateApplicationEndpointsStatus(context.Context, string, []string, EndpointStatus) error
}

type Datastore added in v0.1.0 

type Datastore interface {
	GroupRepository
	ApplicationRepository
	// EndpointRepository
	io.Closer
	Migrate() error
}

Datastore provides an abstraction for all database related operations

type DeliveryAttempt added in v0.3.0 

type DeliveryAttempt struct {
	ID         primitive.ObjectID `json:"-" bson:"_id"`
	UID        string             `json:"uid" bson:"uid"`
	MsgID      string             `json:"msg_id" bson:"msg_id"`
	URL        string             `json:"url" bson:"url"`
	Method     string             `json:"method" bson:"method"`
	EndpointID string             `json:"endpoint_id" bson:"endpoint_id"`
	APIVersion string             `json:"api_version" bson:"api_version"`

	IPAddress        string     `json:"ip_address,omitempty" bson:"ip_address,omitempty"`
	RequestHeader    HttpHeader `json:"request_http_header,omitempty" bson:"request_http_header,omitempty"`
	ResponseHeader   HttpHeader `json:"response_http_header,omitempty" bson:"response_http_header,omitempty"`
	HttpResponseCode string     `json:"http_status,omitempty" bson:"http_status,omitempty"`
	ResponseData     string     `json:"response_data,omitempty" bson:"response_data,omitempty"`
	Error            string     `json:"error,omitempty" bson:"error,omitempty"`
	Status           bool       `json:"status,omitempty" bson:"status,omitempty"`

	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`
}

type DocumentStatus added in v0.1.0 

type DocumentStatus string
const (
	ActiveDocumentStatus   DocumentStatus = "Active"
	InactiveDocumentStatus DocumentStatus = "Inactive"
	DeletedDocumentStatus  DocumentStatus = "Deleted"
)

type Endpoint added in v0.1.0 

type Endpoint struct {
	UID         string         `json:"uid" bson:"uid"`
	TargetURL   string         `json:"target_url" bson:"target_url"`
	Description string         `json:"description" bson:"description"`
	Status      EndpointStatus `json:"status" bson:"status"`
	Secret      string         `json:"secret" bson:"secret"`

	Events []string `json:"events" bson:"events"`

	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`

	DocumentStatus DocumentStatus `json:"-" bson:"document_status"`
}

type EndpointMetadata added in v0.1.0 

type EndpointMetadata struct {
	UID       string         `json:"uid" bson:"uid"`
	TargetURL string         `json:"target_url" bson:"target_url"`
	Status    EndpointStatus `json:"status" bson:"status"`
	Secret    string         `json:"secret" bson:"secret"`

	Sent bool `json:"sent" bson:"sent"`
}

type EndpointStatus added in v0.2.0 

type EndpointStatus string
const (
	ActiveEndpointStatus   EndpointStatus = "active"
	InactiveEndpointStatus EndpointStatus = "inactive"
	PendingEndpointStatus  EndpointStatus = "pending"
)

type Event added in v0.3.0 

type Event struct {
	ID               primitive.ObjectID `json:"-" bson:"_id"`
	UID              string             `json:"uid" bson:"uid"`
	EventType        EventType          `json:"event_type" bson:"event_type"`
	MatchedEndpoints int                `json:"matched_endpoints" bson:"matched_enpoints"`

	// ProviderID is a custom ID that can be used to reconcile this Event
	// with your internal systems.
	// This is optional
	// If not provided, we will generate one for you
	ProviderID string `json:"provider_id" bson:"provider_id"`

	// Data is an arbitrary JSON value that gets sent as the body of the
	// webhook to the endpoints
	Data json.RawMessage `json:"data" bson:"data"`

	AppMetadata *AppMetadata `json:"app_metadata,omitempty" bson:"app_metadata"`

	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`

	DocumentStatus DocumentStatus `json:"-" bson:"document_status"`
}

Event defines a payload to be sent to an application

type EventDelivery added in v0.3.0 

type EventDelivery struct {
	ID            primitive.ObjectID `json:"-" bson:"_id"`
	UID           string             `json:"uid" bson:"uid"`
	EventMetadata *EventMetadata     `json:"event_metadata" bson:"event_metadata"`

	// Endpoint contains the destination of the event.
	EndpointMetadata *EndpointMetadata `json:"endpoint" bson:"endpoint"`

	AppMetadata      *AppMetadata        `json:"app_metadata,omitempty" bson:"app_metadata"`
	Metadata         *Metadata           `json:"metadata" bson:"metadata"`
	Description      string              `json:"description,omitempty" bson:"description"`
	Status           EventDeliveryStatus `json:"status" bson:"status"`
	DeliveryAttempts []DeliveryAttempt   `json:"-" bson:"attempts"`

	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`

	DocumentStatus DocumentStatus `json:"-" bson:"document_status"`
}

Event defines a payload to be sent to an application

type EventDeliveryRepository added in v0.3.0 

type EventDeliveryRepository interface {
	CreateEventDelivery(context.Context, *EventDelivery) error
	FindEventDeliveryByID(context.Context, string) (*EventDelivery, error)
	FindEventDeliveriesByIDs(context.Context, []string) ([]EventDelivery, error)
	FindEventDeliveriesByEventID(context.Context, string) ([]EventDelivery, error)
	UpdateStatusOfEventDelivery(context.Context, EventDelivery, EventDeliveryStatus) error
	UpdateEventDeliveryWithAttempt(context.Context, EventDelivery, DeliveryAttempt) error
	LoadEventDeliveriesPaged(context.Context, string, string, string, []EventDeliveryStatus, models.SearchParams, models.Pageable) ([]EventDelivery, pager.PaginationData, error)
}

type EventDeliveryStatus added in v0.3.0 

type EventDeliveryStatus string
const (
	// ScheduledEventStatus : when  a Event has been scheduled for delivery
	ScheduledEventStatus  EventDeliveryStatus = "Scheduled"
	ProcessingEventStatus EventDeliveryStatus = "Processing"
	DiscardedEventStatus  EventDeliveryStatus = "Discarded"
	FailureEventStatus    EventDeliveryStatus = "Failure"
	SuccessEventStatus    EventDeliveryStatus = "Success"
	RetryEventStatus      EventDeliveryStatus = "Retry"
)

type EventMetadata added in v0.3.0 

type EventMetadata struct {
	UID       string    `json:"uid" bson:"uid"`
	EventType EventType `json:"name" bson:"name"`
}

type EventRepository added in v0.3.0 

type EventRepository interface {
	CreateEvent(context.Context, *Event) error
	LoadEventIntervals(context.Context, string, models.SearchParams, Period, int) ([]models.EventInterval, error)
	LoadEventsPagedByAppId(context.Context, string, models.SearchParams, models.Pageable) ([]Event, pager.PaginationData, error)
	FindEventByID(ctx context.Context, id string) (*Event, error)
	LoadEventsScheduledForPosting(context.Context) ([]Event, error)
	LoadEventsForPostingRetry(context.Context) ([]Event, error)
	LoadAbandonedEventsForPostingRetry(context.Context) ([]Event, error)
	LoadEventsPaged(context.Context, string, string, models.SearchParams, models.Pageable) ([]Event, pager.PaginationData, error)
}

type EventType added in v0.1.0 

type EventType string

EventType is used to identify an specific event. This could be "user.new" This will be used for data indexing Makes it easy to filter by a list of events

type Group added in v0.2.0 

type Group struct {
	ID      primitive.ObjectID  `json:"-" bson:"_id"`
	UID     string              `json:"uid" bson:"uid"`
	Name    string              `json:"name" bson:"name"`
	LogoURL string              `json:"logo_url" bson:"logo_url"`
	Config  *config.GroupConfig `json:"config" bson:"config"`

	CreatedAt primitive.DateTime `json:"created_at,omitempty" bson:"created_at,omitempty" swaggertype:"string"`
	UpdatedAt primitive.DateTime `json:"updated_at,omitempty" bson:"updated_at,omitempty" swaggertype:"string"`
	DeletedAt primitive.DateTime `json:"deleted_at,omitempty" bson:"deleted_at,omitempty" swaggertype:"string"`

	DocumentStatus DocumentStatus `json:"-" bson:"document_status"`
}

func (*Group) IsDeleted added in v0.2.0 

func (o *Group) IsDeleted() bool

func (*Group) IsOwner added in v0.2.0 

func (o *Group) IsOwner(a *Application) bool

type GroupFilter added in v0.2.0 

type GroupFilter struct {
	Names []string `json:"name" bson:"name"`
}

type GroupRepository added in v0.2.0 

type GroupRepository interface {
	LoadGroups(context.Context, *GroupFilter) ([]*Group, error)
	CreateGroup(context.Context, *Group) error
	UpdateGroup(context.Context, *Group) error
	FetchGroupByID(context.Context, string) (*Group, error)
}

type HttpHeader added in v0.2.0 

type HttpHeader map[string]string

type HttpMethod 

type HttpMethod string
const (
	HttpPost HttpMethod = "POST"
)

type Metadata added in v0.3.0 

type Metadata struct {
	// Data to be sent to endpoint.
	Data     json.RawMessage         `json:"data" bson:"data"`
	Strategy config.StrategyProvider `json:"strategy" bson:"strategy"`
	// NextSendTime denotes the next time a Event will be published in
	// case it failed the first time
	NextSendTime primitive.DateTime `json:"next_send_time" bson:"next_send_time"`

	// NumTrials: number of times we have tried to deliver this Event to
	// an application
	NumTrials uint64 `json:"num_trials" bson:"num_trials"`

	IntervalSeconds uint64 `json:"interval_seconds" bson:"interval_seconds"`

	RetryLimit uint64 `json:"retry_limit" bson:"retry_limit"`
}

func (Metadata) Value added in v0.3.0 

func (em Metadata) Value() (driver.Value, error)

type Period added in v0.1.0 

type Period int
const (
	Daily Period = iota
	Weekly
	Monthly
	Yearly
)

type Plugin 

type Plugin interface {
	Apply(http.ResponseWriter, *http.Request) error
	Name() string
	IsEnabled() bool
}

type SentryHook 

type SentryHook struct {
	LogLevels []log.Level
}

func NewSentryHook 

func NewSentryHook(levels []log.Level) *SentryHook

func (*SentryHook) Fire 

func (s *SentryHook) Fire(entry *log.Entry) error

func (*SentryHook) Levels 

func (s *SentryHook) Levels() []log.Level

type TaskName 

type TaskName string
const (
	EventProcessor      TaskName = "EventProcessor"
	DeadLetterProcessor TaskName = "DeadLetterProcessor"
)

func (TaskName) SetPrefix 

func (t TaskName) SetPrefix(prefix string) TaskName

Source Files

View all Source files

Directories

Path Synopsis
realm/file
realm/noop
realm_chain
algo
Package docs GENERATED BY THE COMMAND ABOVE; DO NOT EDIT This file was generated by swaggo/swag at 2021-12-20 17:08:50.079095 +0100 WAT m=+39.807422918
 Package docs GENERATED BY THE COMMAND ABOVE; DO NOT EDIT This file was generated by swaggo/swag at 2021-12-20 17:08:50.079095 +0100 WAT m=+39.807422918
Package mocks is a generated GoMock package.
 Package mocks is a generated GoMock package.
redis
models
task

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.