README
¶
Depaginator Paginated API Iterator
This repository contains the Depaginator. The Depaginator is a tool for traversing all items presented by a paginated API: all pages are retrieved by independent goroutines, then each item in each page is iterated over, calling a Handle method. The index of the item in the list is also passed to Handle for the benefit of ordering-sensitive applications.
How to Use
For full details, refer to the package documentation. The basic concept is for the consuming application to create one object that implements a GetPage, which conforms to the PageGetter interface, and a second object that implements Handle, conforming to the Handler interface. The GetPage method is passed a PageRequest, which bundles a PageIndex integer with an application-defined Request. The GetPage method must then retrieve the desired page of the results, add any relevant metadata--including requests for subsequent pages--via calls to the Depaginator object, and return a an array of items. The Depaginator will then call Handle for each element in the returned list. Optionally, the Handler may implement additional Start, Update, or Done methods which will be called at appropriate parts of the workflow.
To actually perform the depagination operation, the application passes instances of these objects and any appropriate options to the Depaginate function; this returns a Depaginator object which the application may then Wait on. Any errors encountered during the operation will be returned by Wait.
The PageGetter and the Handler interfaces are distinct to aid in code reuse; this architecture allows for general handlers like the provided ListHandler, as well as allowing the PageGetter to be reused with different handlers, depending on the needs of the application.
For convenience, the ListHandler type is provided; this is a Handler implementation which assembles the list of retrieved items into the correct order.
Why to Use
Many server APIs that return lists of objects will "paginate" the response to avoid overwhelming the connection or the client--or the server or database. However, many clients consuming that API need to perform some operation on all the returned objects, such as displaying them to the user or applying additional filters to select specific items. Especially for large lists, this process can be quite slow; this may be fine for user-interactive clients, such as command line clients, but if some other operation is being performed, such as bulk modifications, this can be unacceptably slow. The Depaginator is intended to simplify the implementation of code that iterates over all the items in a list by allowing their retrieval as fast as the server API will permit.
Notes on Panics
The Depaginate function makes extensive use of internal goroutines. This means that panics could be lost without specific handling. To avoid this being an issue, the library is designed to capture and report panics occurring within internal goroutines as errors of type *PanicError, which preserves the panic object and a stack trace. This only occurs for calls that occur within internal goroutines, which covers PageGetter.GetPage, Handler.Handle, and Updater.Update callbacks. The Starter.Start and Doner.Done callbacks are executed in the caller goroutine, and so no effort is made to catch panics caused by these callbacks.
Documentation
¶
Overview ¶
Package depaginator contains an implementation of a function, Depaginate, that allows iterating over all items in a paginated API. The consuming application needs to pass Depaginate a context.Context, a PageGetter page retriever, and an item Handler, then call Depaginator.Wait on the result; the Depaginator will then take care of the rest, calling the [PageGetter.GetPage] method to retrieve pages of results and the [Handler.Handle] method to handle the found items. Options exist to call a [Starter.Start] method before beginning, [Updater.Update] method when the total number of pages or items is discovered; and [Doner.Done] when the iteration is complete.
Index ¶
- Constants
- type Capacity
- type Depaginator
- type Doner
- type DonerFunc
- type Handler
- type HandlerFunc
- type ListHandler
- func (lh *ListHandler[T]) Done(_ context.Context, totalItems, _, _ int)
- func (lh *ListHandler[T]) Handle(_ context.Context, idx int, item T)
- func (lh *ListHandler[T]) Start(_ context.Context, totalItems, totalPages, perPage int)
- func (lh *ListHandler[T]) Update(_ context.Context, totalItems, totalPages, perPage int)
- type Option
- type PageError
- type PageGetter
- type PageGetterFunc
- type PageRequest
- type PanicError
- type PerPage
- type Starter
- type StarterFunc
- type State
- type TotalItems
- type TotalPages
- type Updater
- type UpdaterFunc
- type WithDonerOption
- type WithRequestOption
- type WithStarterOption
- type WithUpdaterOption
Constants ¶
const DefaultCapacity = 500
DefaultCapacity is the default capacity for the updates channel.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Capacity ¶ added in v0.3.0
type Capacity int
Capacity may be passed to Depaginate to control the size of the updates queue on the Depaginator. This defaults to DefaultCapacity, which is set to a generous size. Applications should only need to use this option if the default is insufficient for efficient operation.
type Depaginator ¶
type Depaginator[T any] struct { // contains filtered or unexported fields }
Depaginator is returned by the Depaginate function to allow the caller to wait for the iteration to complete. This object is also passed to [PageGetter.GetPage], and may be used to call Depaginator.Update and Depaginator.Request to update the number of items/pages or to request fetching additional pages, respectively.
func Depaginate ¶
func Depaginate[T any](ctx context.Context, pager PageGetter[T], handler Handler[T], opts ...Option) *Depaginator[T]
Depaginate is a tool for iterating over all items in a paginated response. It uses goroutines to perform its work, and is capable of issuing requests for every available page simultaneously, so callers should ensure the [PageGetter.GetPage] routine passed to Depaginate incorporates some sort of limiter to ensure they don't overwhelm any rate limits that may be set on the target API. The [Handler.Handle] method will be called for each item. Note that Depaginate returns a Depaginator, and the calling application is expected to call Depaginator.Wait.
func (*Depaginator[T]) PerPage ¶ added in v0.3.0
func (dp *Depaginator[T]) PerPage() int
PerPage retrieves the configured "per page" value for Depaginator. This allows a consumer to set the number of items per page when calling Depaginate (using the PerPage option). Applications should be careful to not mix this functionality with dynamic collection of the "per page" value, as the value is not protected by any mutex; if using this method, avoid passing PerPage to Depaginator.Update and arrange for a reasonable default if PerPage is not passed to Depaginate (in which case, this method will return 0).
func (*Depaginator[T]) Request ¶ added in v0.3.0
func (dp *Depaginator[T]) Request(idx int, req any)
Request requests the Depaginator retrieve a page. Note that the page index is 0-based; the first page always has index 0. The request is optional, and can contain any page-specific data, such as a page link. Duplicate page requests are ignored, as is any request with an index greater than the total number of pages (if known).
func (*Depaginator[T]) Update ¶ added in v0.3.0
func (dp *Depaginator[T]) Update(updates ...any)
Update allows updating the total number of items, total number of pages, or the items per page. The arguments passed to Update should be TotalItems, TotalPages, or PerPage; any other argument types will be ignored.
func (*Depaginator[T]) Wait ¶
func (dp *Depaginator[T]) Wait() error
Wait waits for the iteration to complete. It returns the errors encountered during the iteration, wrapped by errors.Join. Each error in the list is a PageError, which bundles together the error and the corresponding page request.
type Doner ¶ added in v0.3.0
type Doner interface {
// Done is called with the most up-to-date values of total items,
// total pages, and items per page. It is called once all pages
// have been retrieved and all items handled.
Done(ctx context.Context, totalItems, totalPages, perPage int)
}
Doner is an interface that can be additionally implemented by [Handle] implementations. The Done method will be called once all pages have been retrieved and all items have been handled.
type DonerFunc ¶ added in v0.3.0
DonerFunc is a wrapper for a function matching the [Doner.Done] signature. The wrapper implements the Doner interface, allowing a function to be passed instead of an interface implementation.
type Handler ¶ added in v0.3.0
type Handler[T any] interface { // Handle is called for each item in a page of items retrieved by // the [PageGetter]. It is called with the item index and the // item. // // If the Handle method panics, the panic will be captured and // reported to the caller of [Depaginate] as an error of type // [PanicError]. Handle(ctx context.Context, idx int, item T) }
Handler is an interface for handling items iterated over in a given page. Note that the handler is called from a common goroutine, so if extensive processing will be performed, a new goroutine should be started from the Handle method.
type HandlerFunc ¶ added in v0.3.0
HandlerFunc is a wrapper for a function matching the [Handler.Handle] signature. The wrapper implements the Handler interface, allowing a function to be passed instead of an interface implementation.
func (HandlerFunc[T]) Handle ¶ added in v0.3.0
func (f HandlerFunc[T]) Handle(ctx context.Context, idx int, item T)
Handle is called for each item in a page of items retrieved by the PageGetter. It is called with the item index and the item.
If the Handle method panics, the panic will be captured and reported to the caller of Depaginate as an error of type PanicError.
type ListHandler ¶ added in v0.3.0
type ListHandler[T any] struct { Items []T // Final list of items // contains filtered or unexported fields }
ListHandler is an implementation of Handler that constructs a slice containing all the retrieved items in order. It can be passed to Depaginate multiple times, with additional items added at the end of the list. Once ListHandler.Done is called (which is called by Depaginator.Wait), the Items field of the object will contain the properly ordered list of items retrieved via the PageGetter. No constructor is necessary, as a pointer to the zero value of ListHandler is valid.
func (*ListHandler[T]) Done ¶ added in v0.3.0
func (lh *ListHandler[T]) Done(_ context.Context, totalItems, _, _ int)
Done is called with the most up-to-date values of total items, total pages, and items per page. It is called once all pages have been retrieved and all items handled.
func (*ListHandler[T]) Handle ¶ added in v0.3.0
func (lh *ListHandler[T]) Handle(_ context.Context, idx int, item T)
Handle is called for each item in a page of items retrieved by the PageGetter. It is called with the item index and the item.
If the Handle method panics, the panic will be captured and reported to the caller of Depaginate as an error of type PanicError.
func (*ListHandler[T]) Start ¶ added in v0.3.0
func (lh *ListHandler[T]) Start(_ context.Context, totalItems, totalPages, perPage int)
Start is called with the initial values of total items, total pages, and items per page. It should perform any initialization that may be required.
func (*ListHandler[T]) Update ¶ added in v0.3.0
func (lh *ListHandler[T]) Update(_ context.Context, totalItems, totalPages, perPage int)
Update is called with the new values of total items, total pages, and items per page. It should not undertake extensive processing.
If the Update method panics, the panic will be captured and reported to the caller of Depaginate as an error of type PanicError.
type Option ¶ added in v0.3.0
type Option interface {
// contains filtered or unexported methods
}
Option describes an option that may be passed to Depaginate.
type PageError ¶
type PageError struct {
PageRequest PageRequest // The request that failed
Err error // The error that occurred
}
PageError contains an error returned by the [PageGetter.GetPage] callback, along with the failing page request.
type PageGetter ¶ added in v0.3.0
type PageGetter[T any] interface { // GetPage is a page retriever function. It is passed the // [Depaginator] object and a [PageRequest] object describing the // page to request, and returns a list of items of the appropriate // type, or an error. Methods of [Depaginator] should be called // to update data on the maximum number of items, maximum number // of pages, items per page, or additional pages to request. Note // that page requests for page indexes that are greater than the // maximum known number of pages will be ignored. // // If the GetPage method panics, the panic will be captured and // reported to the caller of [Depaginate] as an error of type // [PanicError]. GetPage(ctx context.Context, depag State, req PageRequest) ([]T, error) }
PageGetter is an interface for a GetPage method that retrieves a page specified by the given PageRequest. It returns a slice of some type or an error. It may also call methods on the Depaginator object to submit metadata information, such as additional requests to make.
type PageGetterFunc ¶ added in v0.3.0
PageGetterFunc is a wrapper for a function matching the [PageGetter.GetPage] signature. The wrapper implements the PageGetter interface, allowing a function to be passed instead of an interface implementation.
func (PageGetterFunc[T]) GetPage ¶ added in v0.3.0
func (f PageGetterFunc[T]) GetPage(ctx context.Context, depag State, req PageRequest) ([]T, error)
GetPage is a page retriever function. It is passed the Depaginator object and a PageRequest object describing the page to request, and returns a list of items of the appropriate type, or an error. Methods of Depaginator should be called to update data on the maximum number of items, maximum number of pages, items per page, or additional pages to request. Note that page requests for page indexes that are greater than the maximum known number of pages will be ignored.
If the GetPage method panics, the panic will be captured and reported to the caller of Depaginate as an error of type PanicError.
type PageRequest ¶
type PageRequest struct {
PageIndex int // The index of the page
Request any // The actual data needed to request the page
}
PageRequest describes a request for a specific page. Most of the page request lives in the Request field, which can be anything needed by the application; the only other field is the PageIndex field, which identifies the index of the page. Note that PageIndex is 0-based.
type PanicError ¶ added in v0.4.0
type PanicError struct {
Panic any // The panic value
Trace string // The captured stack trace
// contains filtered or unexported fields
}
PanicError is an error that represents a panic that occurred during go routine execution.
func NewPanicError ¶ added in v0.4.0
func NewPanicError(panicVal any) *PanicError
NewPanicError constructs a new PanicError instance with the specified panic value. Constructs and saves a stack trace.
func (*PanicError) Error ¶ added in v0.4.0
func (pe *PanicError) Error() string
Error returns the error message.
type PerPage ¶ added in v0.3.0
type PerPage int
PerPage is used to indicate an update to the number of items per page to be expected. It may also be passed to Depaginate to hint to the number of items per page to be expected.
type Starter ¶ added in v0.3.0
type Starter interface {
// Start is called with the initial values of total items, total
// pages, and items per page. It should perform any
// initialization that may be required.
Start(ctx context.Context, totalItems, totalPages, perPage int)
}
Starter is an interface that can be additionally implemented by Handler implementations. The Start method will be called before Depaginate begins its work, allowing the Handler to implement any initialization it requires.
type StarterFunc ¶ added in v0.3.0
StarterFunc is a wrapper for a function matching the [Starter.Start] signature. The wrapper implements the Starter interface, allowing a function to be passed instead of an interface implementation.
type State ¶ added in v0.3.1
type State interface {
// Update allows updating the total number of items, total number
// of pages, or the items per page. The arguments passed to
// Update should be [TotalItems], [TotalPages], or [PerPage]; any
// other argument types will be ignored.
Update(updates ...any)
// Request requests the [Depaginator] retrieve a page. Note that
// the page index is 0-based; the first page always has index 0.
// The request is optional, and can contain any page-specific
// data, such as a page link. Duplicate page requests are
// ignored, as is any request with an index greater than the total
// number of pages (if known).
Request(idx int, req any)
// PerPage retrieves the configured "per page" value for
// [Depaginator]. This allows a consumer to set the number of
// items per page when calling [Depaginate] (using the [PerPage]
// option). Applications should be careful to not mix this
// functionality with dynamic collection of the "per page" value,
// as the value is not protected by any mutex; if using this
// method, avoid passing [PerPage] to [Depaginator.Update] and
// arrange for a reasonable default if [PerPage] is not passed to
// [Depaginate] (in which case, this method will return 0).
PerPage() int
}
State describes the state of depagination. It provides the feedback mechanism for requesting updates to the depaginator state, such as updating the total number of items or making additional page requests.
type TotalItems ¶ added in v0.3.0
type TotalItems int
TotalItems is used to indicate an update to the total number of items to be expected. It may also be passed to Depaginate to hint to the total number of items to be expected.
type TotalPages ¶ added in v0.3.0
type TotalPages int
TotalPages is used to indicate an update to the total number of pages to be expected. It may also be passed to Depaginate to hint to the total number of pages to be expected.
type Updater ¶ added in v0.3.0
type Updater interface {
// Update is called with the new values of total items, total
// pages, and items per page. It should not undertake extensive
// processing.
//
// If the Update method panics, the panic will be captured and
// reported to the caller of [Depaginate] as an error of type
// [PanicError].
Update(ctx context.Context, totalItems, totalPages, perPage int)
}
Updater is an interface that can be additionally implemented by Handler implementations. The Update method will be called with the updated values of the total items, total pages, and items per page, every time the [PageGetter.GetPage] method makes appropriate calls to update these values.
type UpdaterFunc ¶ added in v0.3.0
UpdaterFunc is a wrapper for a function matching the [Updater.Update] signature. The wrapper implements the Updater interface, allowing a function to be passed instead of an interface implementation.
func (UpdaterFunc) Update ¶ added in v0.3.0
func (f UpdaterFunc) Update(ctx context.Context, totalItems, totalPages, perPage int)
Update is called with the new values of total items, total pages, and items per page. It should not undertake extensive processing.
If the Update method panics, the panic will be captured and reported to the caller of Depaginate as an error of type PanicError.
type WithDonerOption ¶ added in v0.3.0
type WithDonerOption struct {
// contains filtered or unexported fields
}
WithDonerOption is an Option implementation that explicitly sets the Doner to use.
func WithDoner ¶ added in v0.3.0
func WithDoner(doner Doner) WithDonerOption
WithDoner returns an Option that can be passed to Depaginate which sets an Doner to be called once all pages are retrieved. The default is the Handler, if it implements Doner.
type WithRequestOption ¶ added in v0.3.0
type WithRequestOption struct {
// contains filtered or unexported fields
}
WithRequestOption is an Option implementation that sets the initial request.
func WithRequest ¶ added in v0.3.0
func WithRequest(req any) WithRequestOption
WithRequest returns an Option which sets the request object for the initial page load. By default, the request will be set to nil.
type WithStarterOption ¶ added in v0.3.0
type WithStarterOption struct {
// contains filtered or unexported fields
}
WithStarterOption is an Option implementation that explicitly sets the Starter to use.
func WithStarter ¶ added in v0.3.0
func WithStarter(starter Starter) WithStarterOption
WithStarter returns an Option that can be passed to Depaginate which sets an Starter to be called when Depaginate begins its work. The [Starter.Start] method is called with the initial values for total pages, total items, and per-page. The default is the Handler, if it implements Starter.
type WithUpdaterOption ¶ added in v0.3.0
type WithUpdaterOption struct {
// contains filtered or unexported fields
}
WithUpdaterOption is an Option implementation that explicitly sets the Updater to use.
func WithUpdater ¶ added in v0.3.0
func WithUpdater(updater Updater) WithUpdaterOption
WithUpdater returns an Option that can be passed to Depaginate which sets an Updater to be called when the total pages, total items, or per-page values are altered. The default is the Handler, if it implements Updater.
Source Files