serving

Documentation

Overview

Package serving is the entry point for model inference (serving). Models can have more than one inference engine, this will pick the correct (fastest) one according to the model.

Index

• Constants
• func DetectCompatibility(model model.Model) example.CompatibilityType
• func NewEngine(model model.Model) (engine.Engine, error)
• func NewEngineWithCompatibility(model model.Model, compatibility example.CompatibilityType) (engine.Engine, error)

Constants

const (
	TFEstimatorFramework = "TF Estimator" // Generally used in TF 1.0
	TFKerasFramework     = "TF Keras"     // Generally used in TF 2.0
)

Different frameworks on which the model may have been trained.

Functions

func DetectCompatibility

func DetectCompatibility(model model.Model) example.CompatibilityType

DetectCompatibility detects the most likely compatibility of the model. If unsure, returns the Yggdrasil (i.e. native) compatibility.

func NewEngine

func NewEngine(model model.Model) (engine.Engine, error)

NewEngine creates the best available engine for the model. It fails if no engine is available for the model.

func NewEngineWithCompatibility

func NewEngineWithCompatibility(model model.Model, compatibility example.CompatibilityType) (engine.Engine, error)

NewEngineWithCompatibility creates the best available engine for the model. It fails if no engine is available for the model.

The "compatibility" argument facilitates cross API modeling. For example, to use a model trained with TensorFlow Decision Forests, the compatibility can be set to compatibilityTensorFlowDecisionForests. Not setting a compatibility corresponding to how the model was trained does not prevent the model to be used. However, in this case, the user need to make sure to understand the difference between the APIs (if any) and adapt the code accordingly. See the definition of the compatibilities (e.g. "compatibilityTensorFlowDecisionForests") for a description of the effects.