Topicmappr was created as a replacement for Kafka's provided
kafka-reassign-partition.sh tool, providing additional enhancements:
Given the same input, topicmappr should always provide the same output map.
Minimal Partition Movement
Avoids reassigning partitions where movement isn't necessary, greatly reducing reassignment times and resource load for simple recoveries.
Balancing Partition Placement With Constraints
For each partition placement, topicmappr chooses the least utilized candidate broker (configurable as by storage or by partition counts) that satisfies the following constraints:
- the broker isn't already in the replica set
- the broker isn't in any of the existing replica set localities (using the Kafka
Provided enough brokers, topicmapper determines the appropriate leadership, follower and failure domain balance.
An output of what's changed along with advisory notices (e.g. insufficient broker counts supplied to satisfy all constraints at the desired partition/replica count) that helps users make clear decisions.
Additional statistical output is included where available. For instance, broker-to-broker relationships are represented as node degree counts (where edges are defined as brokers that belong in a common replica set for any given partition). These values can be used as a probabilistic indicator of replication bandwidth; replacing a broker with more edges will likely replicate from more source brokers than one with fewer edges.
go get -u github.com/DataDog/kafka-kit/...
Binary will be found at
Tested with Go 1.10+ (required), Kafka 0.10.x, ZooKeeper 3.4.x.
Most operations are performed through the
rebuild command. Partial rebalances are performed through a dedicated
rebalance command (beta).
Usage: topicmappr [command] Available Commands: help Help about any command rebalance Rebalance partition allotments among a set of topics and brokers rebuild Rebuild a partition map for one or more topics Flags: -h, --help help for topicmappr --zk-addr string ZooKeeper connect string [TOPICMAPPR_ZK_ADDR] (default "localhost:2181") --zk-prefix string ZooKeeper prefix (if Kafka is configured with a chroot path prefix) [TOPICMAPPR_ZK_PREFIX] Use "topicmappr [command] --help" for more information about a command.
rebuild requires at least two inputs: a reference of target topics and a list of broker IDs to which those topics should be mapped. Target topics are provided as a comma delimited list of topic names and/or regex patterns via the --topics parameter, which discovers matching topics in ZooKeeper (additionally, the --zk-addr and --zk-prefix global flags should be set). Alternatively, a JSON map can be provided via the --map-string flag. Target broker IDs are provided via the --broker flag. Usage: topicmappr rebuild [flags] Flags: --brokers string Broker list to scope all partition placements to --force-rebuild Forces a complete map rebuild -h, --help help for rebuild --ignore-warns Produce a map even if warnings are encountered --map-string string Rebuild a partition map provided as a string literal --optimize string Optimization priority for the storage placement strategy: [distribution, storage] (default "distribution") --out-file string If defined, write a combined map of all topics to a file --out-path string Path to write output map files to --partition-size-factor float Factor by which to multiply partition sizes when using storage placement (default 1) --placement string Partition placement strategy: [count, storage] (default "count") --replication int Normalize the topic replication factor across all replica sets (0 results in a no-op) --sub-affinity Replacement broker substitution affinity --topics string Rebuild topics (comma delim. list) by lookup in ZooKeeper --use-meta Use broker metadata in placement constraints (default true) --zk-metrics-prefix string ZooKeeper namespace prefix for Kafka metrics (when using storage placement) (default "topicmappr") Global Flags: --zk-addr string ZooKeeper connect string (for broker metadata or rebuild-topic lookups) (default "localhost:2181") --zk-prefix string ZooKeeper namespace prefix (for Kafka brokers)
Rebalance partition allotments among a set of topics and brokers Usage: topicmappr rebalance [flags] Flags: --brokers string Broker list to scope all partition placements to -h, --help help for rebalance --locality-scoped Disallow a relocation to traverse rack.id values among brokers (default true) --optimize-leaders Perform a naive leadership optimization --out-file string If defined, write a combined map of all topics to a file --out-path string Path to write output map files to --partition-limit int Limit the number of top partitions by size eligible for relocation per broker (default 30) --storage-threshold float Percent below the harmonic mean storage free to target for partition offload (default 0.2) --storage-threshold-gb float Storage free in gigabytes to target for partition offload (those below the specified value); 0 [default] defers target selection to --storage-threshold --tolerance float Percent distance from the mean storage free to limit storage scheduling (0 targets a brokers) (default 0.1) --topics string Rebuild topics (comma delim. list) by lookup in ZooKeeper --verbose Verbose output --zk-metrics-prefix string ZooKeeper namespace prefix for Kafka metrics (when using storage placement) (default "topicmappr") Global Flags: --zk-addr string ZooKeeper connect string (for broker metadata or rebuild-topic lookups) (default "localhost:2181") --zk-prefix string ZooKeeper namespace prefix (for Kafka brokers)
Managing and Repairing Topics
See the wiki Usage Guide section for examples of common topic management tasks.
There is no documentation for this package.