processor

Documentation

Index

• type AddressGetter
• type FakeModelSyncer
  • func NewFakeModelSyncer() *FakeModelSyncer
  • func (s *FakeModelSyncer) ListInProgressModels() []string
  • func (s *FakeModelSyncer) ListSyncedModelIDs(ctx context.Context) []string
  • func (s *FakeModelSyncer) PullModel(ctx context.Context, modelID string) error
• type FixedAddressGetter
  • func NewFixedAddressGetter(addr string) *FixedAddressGetter
  • func (g *FixedAddressGetter) GetLLMAddress(modelID string) (string, error)
• type ModelSyncer
• type NoopMetricsCollector
  • func (NoopMetricsCollector) Add(modelID string, v float64)
• type P
  • func NewP(engineID string, client v1.InferenceWorkerServiceClient, ...) *P
  • func (p *P) IsReady() (bool, string)
  • func (p *P) NeedLeaderElection() bool
  • func (p *P) SetupWithManager(mgr ctrl.Manager, leaderElection bool) error
  • func (p *P) Start(ctx context.Context) error

Types

type AddressGetter

type AddressGetter interface {
	GetLLMAddress(modelID string) (string, error)
}

AddressGetter gets an address of a model.

type FakeModelSyncer

type FakeModelSyncer struct {
	// contains filtered or unexported fields
}

FakeModelSyncer is a fake implementation of model syncer.

func NewFakeModelSyncer

func NewFakeModelSyncer() *FakeModelSyncer

NewFakeModelSyncer returns a FakeModelSyncer.

func (*FakeModelSyncer) ListInProgressModels

func (s *FakeModelSyncer) ListInProgressModels() []string

ListInProgressModels lists all models that are in progress.

func (*FakeModelSyncer) ListSyncedModelIDs

func (s *FakeModelSyncer) ListSyncedModelIDs(ctx context.Context) []string

ListSyncedModelIDs lists all models that have been synced.

func (*FakeModelSyncer) PullModel

func (s *FakeModelSyncer) PullModel(ctx context.Context, modelID string) error

PullModel downloads and registers a model from model manager.

type FixedAddressGetter

type FixedAddressGetter struct {
	// contains filtered or unexported fields
}

FixedAddressGetter is a fixed address getter.

func NewFixedAddressGetter

func NewFixedAddressGetter(addr string) *FixedAddressGetter

NewFixedAddressGetter returns a new FixedAddressGetter.

func (*FixedAddressGetter) GetLLMAddress

func (g *FixedAddressGetter) GetLLMAddress(modelID string) (string, error)

GetLLMAddress returns a fixed address.

type ModelSyncer

type ModelSyncer interface {
	ListSyncedModelIDs() []string
	PullModel(ctx context.Context, modelID string) error
	ListInProgressModels() []string
}

ModelSyncer syncs models.

type NoopMetricsCollector

type NoopMetricsCollector struct{}

NoopMetricsCollector is a no-op metrics collector.

func (NoopMetricsCollector) Add

func (NoopMetricsCollector) Add(modelID string, v float64)

Add does nothing.

type P

type P struct {
	// contains filtered or unexported fields
}

P processes tasks.

func NewP

func NewP(
	engineID string,
	client v1.InferenceWorkerServiceClient,
	addrGetter AddressGetter,
	modelSyncer ModelSyncer,
	logger logr.Logger,
	collector metrics.Collector,
	gracefulShutdownTimeout time.Duration,
) *P

NewP returns a new processor.

func (*P) IsReady

func (p *P) IsReady() (bool, string)

IsReady returns true if the processor is ready. If not, it returns a message describing why it is not ready.

func (*P) NeedLeaderElection

func (p *P) NeedLeaderElection() bool

NeedLeaderElection implements LeaderElectionRunnable

func (*P) SetupWithManager

func (p *P) SetupWithManager(mgr ctrl.Manager, leaderElection bool) error

SetupWithManager sets up the processor with the manager.

func (*P) Start

func (p *P) Start(ctx context.Context) error

Start runs the processor.

TODO(kenji): Gracefully handle an error from the server.