infprocessor

Documentation

Index

• type P
  • func NewP(queue *TaskQueue, engineGetter engineGetter) *P
  • func (p *P) AddOrUpdateEngineStatus(srv v1.InferenceWorkerService_ProcessTasksServer, ...)
  • func (p *P) Run(ctx context.Context) error
• type Task
  • func (t *Task) WaitForCompletion() (*http.Response, error)
• type TaskQueue
  • func NewTaskQueue() *TaskQueue
  • func (q *TaskQueue) Dequeue(ctx context.Context) (*Task, error)
  • func (q *TaskQueue) Enqueue(t *Task)

Types

type P

type P struct {
	// contains filtered or unexported fields
}

P processes inference tasks.

func NewP

func NewP(queue *TaskQueue, engineGetter engineGetter) *P

NewP creates a new processor.

func (*P) AddOrUpdateEngineStatus

func (p *P) AddOrUpdateEngineStatus(
	srv v1.InferenceWorkerService_ProcessTasksServer,
	engineStatus *v1.EngineStatus,
)

AddOrUpdateEngineStatus adds or updates the engine status.

func (*P) Run

func (p *P) Run(ctx context.Context) error

Run runs the processor.

type Task

type Task struct {
	ID     string
	Req    *v1.CreateChatCompletionRequest
	Header http.Header
	RespCh chan *http.Response
	ErrCh  chan error
}

Task is an inference task. TODO(kenji): Consider preserving the request context as well.

func (*Task) WaitForCompletion

func (t *Task) WaitForCompletion() (*http.Response, error)

WaitForCompletion waits for the completion of the task.

type TaskQueue

type TaskQueue struct {
	// contains filtered or unexported fields
}

TaskQueue is a queue for inference tasks.

func NewTaskQueue

func NewTaskQueue() *TaskQueue

NewTaskQueue creates a new task queue.

func (*TaskQueue) Dequeue

func (q *TaskQueue) Dequeue(ctx context.Context) (*Task, error)

Dequeue removes a task from the queue.

func (*TaskQueue) Enqueue

func (q *TaskQueue) Enqueue(t *Task)

Enqueue inserts a task into the queue.