sharding

command
Version: v0.1.0-alpha Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 19, 2021 License: MIT Imports: 5 Imported by: 0

Documentation

Overview

sharding example shows how to use an output that supports sharding When a shardable output is used, the parallel outputs identified by the "procs" configuration value in the toml, receive a subset of the processed records. The sharding function returns a shard idx (based on the sharded field value) which value is assigned to an output process calculating the modulo with the available output processes. This means that using a procs=1 configuration is the same as disabling the sharding, while procs=N where N is the number of possible values of the sharded field means that each output receives records with always the same value for that field

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL