processor

Documentation

Index

• type AddressGetter
• type ModelSyncer
• type P
  • func NewP(engineID string, client v1.InferenceWorkerServiceClient, ...) *P
  • func (p *P) IsReady() (bool, string)
  • func (p *P) NeedLeaderElection() bool
  • func (p *P) SetupWithManager(mgr ctrl.Manager, leaderElection bool) error
  • func (p *P) Start(ctx context.Context) error

Types

type AddressGetter

type AddressGetter interface {
	GetLLMAddress(modelID string) (string, error)
}

AddressGetter gets an address of a model.

type ModelSyncer

type ModelSyncer interface {
	ListSyncedModels() []runtime.ModelRuntimeInfo
	PullModel(ctx context.Context, modelID string) error
	ListInProgressModels() []runtime.ModelRuntimeInfo
	DeleteModel(ctx context.Context, modelID string) error
}

ModelSyncer syncs models.

type P

type P struct {
	// contains filtered or unexported fields
}

P processes tasks.

func NewP

func NewP(
	engineID string,
	client v1.InferenceWorkerServiceClient,
	addrGetter AddressGetter,
	modelSyncer ModelSyncer,
	logger logr.Logger,
	collector metrics.Collector,
	gracefulShutdownTimeout time.Duration,
) *P

NewP returns a new processor.

func (*P) IsReady

func (p *P) IsReady() (bool, string)

IsReady returns true if the processor is ready. If not, it returns a message describing why it is not ready.

func (*P) NeedLeaderElection

func (p *P) NeedLeaderElection() bool

NeedLeaderElection implements LeaderElectionRunnable

func (*P) SetupWithManager

func (p *P) SetupWithManager(mgr ctrl.Manager, leaderElection bool) error

SetupWithManager sets up the processor with the manager.

func (*P) Start

func (p *P) Start(ctx context.Context) error

Start runs the processor.

TODO(kenji): Gracefully handle an error from the server.