README
¶
.. image:: docs/_static/logo_with_text.svg :alt: Gazette Logo .. image:: https://github.com/gazette/core/workflows/Gazette%20Continuous%20Integration/badge.svg :target: https://github.com/gazette/core/actions?query=workflow%3A%22Gazette+Continuous+Integration%22 :alt: Gazette Continuous Integration .. image:: https://godoc.org/go.gazette.dev/core?status.svg :target: https://godoc.org/go.gazette.dev/core :alt: GoDoc .. image:: https://img.shields.io/badge/slack-@gazette/dev-yellow.svg?logo=slack :target: https://join.slack.com/t/gazette-dev/shared_invite/enQtNjQxMzgyNTEzNzk1LTU0ZjZlZmY5ODdkOTEzZDQzZWU5OTk3ZTgyNjY1ZDE1M2U1ZTViMWQxMThiMjU1N2MwOTlhMmVjYjEzMjEwMGQ :alt: Slack .. image:: https://goreportcard.com/badge/github.com/gazette/core :target: https://goreportcard.com/report/github.com/gazette/core :alt: Go Report Card Overview ========= Gazette makes it easy to build platforms that flexibly mix *SQL*, *batch*, and *millisecond-latency streaming* processing paradigms. It enables teams, applications, and analysts to work from a common catalog of data in the way that's most convenient **to them**. Gazette's core abstraction is a "journal" -- a streaming append log that's represented using regular files in a BLOB store (i.e., S3). The magic of this representation is that journals are simultaneously a low-latency data stream *and* a collection of immutable, organized files in cloud storage (aka, a data lake) -- a collection which can be directly integrated into familiar processing tools and SQL engines. Atop the journal *broker service*, Gazette offers a powerful *consumers framework* for building streaming applications in Go. Gazette has served production use cases for nearly five years, with deployments scaled to millions of streamed records per second. Where to Start =============== - Official documentation_, with tutorials and examples. - API Godocs_. .. _documentation: https://gazette.dev .. _Godocs: https://godoc.org/go.gazette.dev/core
Directories
¶
| Path | Synopsis |
|---|---|
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator.
|
Package allocator implements a distributed algorithm for assigning a number of "Items" across a number of "Members", where each Member runs an instance of the Allocator. |
|
push_relabel
Package push_relabel implements a greedy variant of the push/relabel algorithm.
|
Package push_relabel implements a greedy variant of the push/relabel algorithm. |
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply).
|
Package broker implements the broker runtime and protocol.JournalServer APIs (Read, Append, Replicate, List, Apply). |
|
client
Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers.
|
Package client implements a Go client for interacting with the gRPC Journal service of Gazette brokers. |
|
fragment
Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content.
|
Package fragment is a broker-only package concerned with the mapping of journal offsets to protocol.Fragments, and from there to corresponding local or remote journal content. |
|
journalspace
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again.
|
Package journalspace provides mechanisms for mapping a collection of JournalSpecs into a minimally-described hierarchical structure, and for mapping back again. |
|
protocol
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers.
|
Package protocol defines the core broker datamodel, validation behaviors, and gRPC APIs which are shared across clients and broker servers. |
|
protocol/ext
Package ext defines extensions to broker/protocol that depend on keyspace and allocator (which in turn depend on etcd).
|
Package ext defines extensions to broker/protocol that depend on keyspace and allocator (which in turn depend on etcd). |
|
Package brokertest provides utilities for testing components requiring a live Gazette broker.
|
Package brokertest provides utilities for testing components requiring a live Gazette broker. |
|
cmd
|
|
|
gazctl/editor
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt.
|
Package editor provides utilities for editing and re-editing text files in an editor until the changes are accepted or the user aborts the attempt. |
|
Package consumer is a framework for distributed, stateful consumption of Gazette journals.
|
Package consumer is a framework for distributed, stateful consumption of Gazette journals. |
|
protocol
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers.
|
Package protocol defines the consumer datamodel, validation behaviors, and gRPC APIs which are shared across clients and consumer application servers. |
|
recoverylog
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal.
|
Package recoverylog specifies a finite state machine for recording and replaying observed filesystem operations into a Gazette journal. |
|
shardspace
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again.
|
Package shardspace provides mechanisms for mapping a collection of ShardSpecs into a minimally-described, semi hierarchical structure, and for mapping back again. |
|
store-rocksdb
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance.
|
Package store_rocksdb implements the consumer.Store interface via an embedded RocksDB instance. |
|
store-sqlite
Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance.
|
Package store_sqlite implements the consumer.Store interface via an embedded SQLite instance. |
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications.
|
Package consumertest provides utilities for in-process unit testing of Gazette consumer applications. |
|
Package etcdtest provides test support for obtaining a client to an Etcd server.
|
Package etcdtest provides test support for obtaining a client to an Etcd server. |
|
Package examples parents packages which implement example Gazette consumer applications.
|
Package examples parents packages which implement example Gazette consumer applications. |
|
bike-share
Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data.
|
Package bike_share implements a Gazette consumer application which processes and serves streaming Citi Bike system data. |
|
stream-sum
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks.
|
Package stream_sum is an example application consisting of three stages: 1) A `chunker` job randomly generates a number of unique "streams", with stream content emitted across a number of interleaved data chunks. |
|
stream-sum/summer
Package summer runs the stream_sum.Summer consumer.
|
Package summer runs the stream_sum.Summer consumer. |
|
word-count
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts.
|
Package word_count is an example application which provides a gRPC API for publishing texts and querying running counts of NGrams extracted from previously published texts. |
|
word-count/counter
Package counter runs the word_count.Counter consumer.
|
Package counter runs the word_count.Counter consumer. |
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation.
|
Package keyspace implements an efficient mechanism to mirror a decoded Etcd key/value space into a local KeySpace, which may be kept updated via a long- lived Watch operation. |
|
Package labels defines well-known label names and values of Gazette.
|
Package labels defines well-known label names and values of Gazette. |
|
runconsumer
Package runconsumer extends consumer.Application with support for configuration and application initialization.
|
Package runconsumer extends consumer.Application with support for configuration and application initialization. |
|
Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams.
|
Package message is a client-only library which implements exactly-once message semantics atop raw, at-least-once Journal byte-streams. |