sharding

Documentation

Overview

sharding example shows how to use an output that supports sharding When a shardable output is used, the parallel outputs identified by the "procs" configuration value in the toml, receive a subset of the processed records. The sharding function returns a shard idx (based on the sharded field value) which value is assigned to an output process calculating the modulo with the available output processes. This means that using a procs=1 configuration is the same as disabling the sharding, while procs=N where N is the number of possible values of the sharded field means that each output receives records with always the same value for that field