processor

Documentation

Index

• type FakeModelSyncer
  • func NewFakeModelSyncer() *FakeModelSyncer
  • func (s *FakeModelSyncer) ListSyncedModelIDs(ctx context.Context) []string
  • func (s *FakeModelSyncer) PullModel(ctx context.Context, modelID string) error
• type ModelSyncer
• type P
  • func NewP(engineID string, client v1.InferenceWorkerServiceClient, llmAddr string, ...) *P
  • func (p *P) Run(ctx context.Context) error

Types

type FakeModelSyncer

type FakeModelSyncer struct {
	// contains filtered or unexported fields
}

FakeModelSyncer is a fake implementation of model syncer.

func NewFakeModelSyncer

func NewFakeModelSyncer() *FakeModelSyncer

NewFakeModelSyncer returns a FakeModelSyncer.

func (*FakeModelSyncer) ListSyncedModelIDs

func (s *FakeModelSyncer) ListSyncedModelIDs(ctx context.Context) []string

ListSyncedModelIDs lists all models that have been synced.

func (*FakeModelSyncer) PullModel

func (s *FakeModelSyncer) PullModel(ctx context.Context, modelID string) error

PullModel downloads and registers a model from model manager.

type ModelSyncer

type ModelSyncer interface {
	ListSyncedModelIDs(ctx context.Context) []string
	PullModel(ctx context.Context, modelID string) error
}

ModelSyncer syncs models.

type P

type P struct {
	// contains filtered or unexported fields
}

P processes tasks.

func NewP

func NewP(
	engineID string,
	client v1.InferenceWorkerServiceClient,
	llmAddr string,
	llmKind llmkind.K,
	modelSyncer ModelSyncer,
) *P

NewP returns a new processor.

func (*P) Run

func (p *P) Run(ctx context.Context) error

Run runs the processor.

TODO(kenji): Gracefully handle an error from the server.