taxman

README

Zedge Event Taxonomy Framework

An Event Taxonomy describes a schema for events in an application. This repository contains tools and services to implement event taxonomies and event pipelines.

Events

Events are structured log messages used to describe user actions in an application, for product analytics and business reporting.

This is a simple example event:

{
  "event": "browse",
  "zid": "c3f7e8fba1ebb506ce646b4bf07e8bd566fa456b",
  "client_version": "5.56.0",
  "serverTimestamp": 1547581321926,
  "properties": {
    "event_type": "BROWSE",
    "content_type": "CONTENT_EDITORIAL",
    "offset": 1,
    "section": "stories",
    "from_content_type": "NOTIFICATION_SOUND"
  }
}

Events consist of a bunch of properties, most of which are optional. Each key in the JSON object above is an event property, even those outside of the "properties" object.

Each property has a scope that (among other things) determines which codebase owns it, a type, and a lifecycle that says whether it's in production or is being phased out - read more about lifecycles here

The possible event scopes are:

• "Event" properties: these contain specific information to the action the event is describing, and typically comprise the bulk of properties. In the example above, "section" is an event property.
• "User" properties: these contain information about the user or device. In the example above, "client_version" is a user property.
• "Envelope" properties: these are used for routing or filtering events, and are always required, and are always in the outermost JSON object, to allow looking them up as efficiently as possible (using fast JSON scanners such as the Jsoniter family). "zid" is an envelope property in the example above.
• "Internal" properties: these are handled by the client library or server-side pipeline. Examples are geoip-based country and server-side timestamp (added by the server-side API).

Application developers only have to worry about User and Event properties, but it's good to know about the others, because they share the same namespace, and naming conflicts may occur.

Event Taxonomies

The example above is a payload format, and doesn't tell you which fields have which types, scopes or lifecycles. This is where the Event Taxonomy comes in.

The taxonomy descibes, per appid, which properties exist, and which enums are supported. (It also defines internal data bindings for the event pipelines such as Kafka topics and ClickHouse tables, but these details are not exposed to clients.)

Read more about event taxonomies in docs/event-taxonomies.md.

Tools

evtail

The evtail tool lets you "tail" the Kafka event stream in real time. Full docs are here.

Services

Event Taxonomy Controller

This service implements event taxonomies as a Kubernetes Custom Resource, which allows us to treat them as deployable objects in Kubernetes, and leverage the Kubernetes controller pattern to manage resources on behalf of the taxonomy.

This service is deployed to the test and analytics (prod) environments under the release name event-taxonomy-controller.

See full docs on the EventTaxonomy custom resource.

Event Taxonomy API

...also known as taxman.

Exposes the following endpoints:

GET /taxonomy/{appid} - fetch the current event taxonomy for appid

PUT /taxonomy/{appid} - post an updated event taxonomy for appid (subject to all kinds of validation)

POST /taxonomy?validate=true - validate a taxonomy (does not validate wrt update rules)

GET /topics/{appid} - fetch the Kafka topics used for appid

This service is deployed to the test and analytics (prod) environments under the release name event-taxonomy-api.

Evtail Server

This service is deployed to the test and analytics (prod) environments under the release name evtail-server.