README 

.. image:: docs/_static/logo_with_text.svg
   :alt: Gazette Logo


.. image:: https://github.com/gazette/core/workflows/Gazette%20Continuous%20Integration/badge.svg
   :target: https://github.com/gazette/core/actions?query=workflow%3A%22Gazette+Continuous+Integration%22
   :alt: Gazette Continuous Integration
.. image:: https://godoc.org/go.gazette.dev/core?status.svg
   :target: https://godoc.org/go.gazette.dev/core
   :alt: GoDoc
.. image:: https://img.shields.io/badge/slack-@gazette/dev-yellow.svg?logo=slack
   :target: https://join.slack.com/t/gazette-dev/shared_invite/enQtNjQxMzgyNTEzNzk1LTU0ZjZlZmY5ODdkOTEzZDQzZWU5OTk3ZTgyNjY1ZDE1M2U1ZTViMWQxMThiMjU1N2MwOTlhMmVjYjEzMjEwMGQ
   :alt: Slack
.. image:: https://goreportcard.com/badge/github.com/gazette/core
   :target: https://goreportcard.com/report/github.com/gazette/core
   :alt: Go Report Card


Overview
=========

Gazette makes it easy to build platforms that flexibly mix *SQL*, *batch*,
and *millisecond-latency streaming* processing paradigms. It enables teams,
applications, and analysts to work from a common catalog of data in the way
that's most convenient **to them**. Gazette's core abstraction is a "journal"
-- a streaming append log that's represented using regular files in a BLOB
store (i.e., S3).

The magic of this representation is that journals are simultaneously a
low-latency data stream *and* a collection of immutable, organized files
in cloud storage (aka, a data lake) -- a collection which can be directly
integrated into familiar processing tools and SQL engines.

Atop the journal *broker service*, Gazette offers a powerful *consumers
framework* for building streaming applications in Go. Gazette has served
production use cases for nearly five years, with deployments scaled to
millions of streamed records per second.

Where to Start
===============

- Official documentation_, with tutorials and examples.
- API Godocs_.

.. _documentation: https://gazette.dev
.. _Godocs: https://godoc.org/go.gazette.dev/core
Expand ▾ Collapse ▴

Directories

Path Synopsis
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator.
 Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator.
push_relabel
Package push_relabel implements a greedy variant of the push/relabel algorithm.
 Package push_relabel implements a greedy variant of the push/relabel algorithm.
sparse_push_relabel
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply).
 Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply).
client
Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers.
 Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers.
codecs
fragment
Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content.
 Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content.
http_gateway
journalspace
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again.
 Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again.
protocol
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers.
 Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers.
protocol/ext
Package ext defines extensions to broker/protocol that depend on keyspace and allocator (which in turn depend on etcd).
 Package ext defines extensions to broker/protocol that depend on keyspace and allocator (which in turn depend on etcd).
teststub
Package brokertest provides utilities for testing components requiring a live Gazette broker.
 Package brokertest provides utilities for testing components requiring a live Gazette broker.
cmd
gazctl
gazctl/editor
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt.
 Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt.
gazette
Package consumer is a framework for distributed, stateful consumption of Gazette journals.
 Package consumer is a framework for distributed, stateful consumption of Gazette journals.
protocol
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers.
 Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers.
recoverylog
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal.
 Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal.
shardspace
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again.
 Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again.
store-rocksdb
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance.
 Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance.
store-sqlite
Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance.
 Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance.
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications.
 Package consumertest provides utilities for in-process unit testing of Gazette consumer applications.
Package etcdtest provides test support for obtaining a client to an Etcd server.
 Package etcdtest provides test support for obtaining a client to an Etcd server.
Package examples parents packages which implement example Gazette consumer applications.
 Package examples parents packages which implement example Gazette consumer applications.
bike-share
Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data.
 Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data.
bike-share/bike-share
stream-sum
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks.
 Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks.
stream-sum/chunker
stream-sum/summer
Package summer runs the stream_sum.Summer consumer.
 Package summer runs the stream_sum.Summer consumer.
word-count
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts.
 Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts.
word-count/counter
Package counter runs the word_count.Counter consumer.
 Package counter runs the word_count.Counter consumer.
word-count/wordcountctl
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation.
 Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation.
Package labels defines well-known label names and values of Gazette.
 Package labels defines well-known label names and values of Gazette.
runconsumer
Package runconsumer extends consumer.Application with support for configuration and application initialization.
 Package runconsumer extends consumer.Application with support for configuration and application initialization.
Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams.
 Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams.