vllm

Documentation

Index

• func ChatTemplate(modelName string) (string, error)
• func ModelFilePath(modelDir, modelName string) string
• type Manager
  • func New(c *config.Config, modelDir string) *Manager
  • func (m *Manager) CreateNewModel(modelName string, spec *manager.ModelSpec) error
  • func (m *Manager) IsReady() (bool, string)
  • func (m *Manager) Run() error
  • func (m *Manager) UpdateModelTemplateToLatest(modelName string) error
  • func (m *Manager) WaitForReady() error

Functions

func ChatTemplate

func ChatTemplate(modelName string) (string, error)

ChatTemplate returns the chat template for the given model.

func ModelFilePath

func ModelFilePath(modelDir, modelName string) string

ModelFilePath returns the file path of the model.

Types

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages the Ollama service.

TODO(kenji): Refactor this class once we completely switch to the one-odel-per-pod implementation where inference-manager-engine doesn't directly run vLLM or Ollama.

func New

func New(c *config.Config, modelDir string) *Manager

New returns a new Manager.

func (*Manager) CreateNewModel

func (m *Manager) CreateNewModel(modelName string, spec *manager.ModelSpec) error

CreateNewModel creates a new model with the given name and spec.

func (*Manager) IsReady

func (m *Manager) IsReady() (bool, string)

IsReady returns true if the processor is ready. If not, it returns a message describing why it is not ready.

func (*Manager) Run

func (m *Manager) Run() error

Run starts the vLLM service.

func (*Manager) UpdateModelTemplateToLatest

func (m *Manager) UpdateModelTemplateToLatest(modelName string) error

UpdateModelTemplateToLatest updates the model template to the latest.

func (*Manager) WaitForReady

func (m *Manager) WaitForReady() error

WaitForReady waits for the vllm service to be ready.