fvt

package
v0.11.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 10, 2024 License: Apache-2.0 Imports: 54 Imported by: 0

README

Functional Verification Test (FVT) suite

Functional Verification Test (FVT) suite for ModelMesh Serving using Ginkgo.

How the tests are structured

  • The entry points for FVT suite are located in predictor/predictor_suite_test.go, scaleToZero/scaleToZero_suite_test.go and hpa/hpa_suite_test.go.
  • Framework, utility, and helper functions for all suites are in the fvt package in this directory.
  • Manifests used to create predictors, inference services, and runtimes are in the testdata folder.

How to run the FVT suite

Prerequisites
  • CLIs:
  • A Kubernetes or OpenShift cluster:
    • Kubernetes version 1.23+
    • Cluster-administrative privileges
    • 12 vCPUs (3-nodes a 4 vCPU, or, 2-nodes a 8 vCPU)
    • 16 GB memory

For more details on cluster sizing, please see here

Install ModelMesh Serving

The FVTs rely on a set of models existing in a configured localMinIO storage. The easiest way to get these models is to use a quick-start installation with an instance of MinIO running the kserve/modelmesh-minio-dev-examples image.

If starting with a fresh namespace, install ModelMesh Serving configured for the FVTs with:

kubectl create namespace modelmesh-serving
./scripts/install.sh --namespace modelmesh-serving --fvt --dev-mode-logging

To re-configure an existing "quickstart" deployment for FVTs, run:

kubectl apply -f config/dependencies/fvt.yaml
Development Environment

The FVTs run using the ginkgo CLI tool and need kubectl configured to communicate to a Kubernetes cluster. It is recommended to use the containerized development environment to run the FVTs.

First, verify that you have access to your Kubernetes cluster:

kubectl config current-context

If you are using an OpenShift cluster, you can run:

oc login --token=${YOUR_TOKEN} --server=https://${YOUR_ADDRESS}

Then build and start the development environment with:

make develop

This will drop you in a shell in the development container where the ./kube/config is mounted to root so that you should be able to communicate to your Kubernetes cluster from inside the container. If not, you can manually export a functioning kubectl configuration and copy it into the container at /root/.kube/config or, for OpenShift, run the oc login ... command inside the development container.

# in shell that is has `kubectl` configured with the desired cluster as the
# current context, the following command will print a portable kubeconfig file
kubectl config view --minify --flatten
Run the FVTs

With a suitable development environment and ModelMesh Serving installation as described above, the FVTs can be executed with a make target:

make fvt

Set the NAMESPACE environment variable, if you installed to a namespace other than modelmesh-serving

NAMESPACE="<your-namespace>" make fvt

Enabling or disabling specific tests

Thanks to the Ginkgo framework, we have the ability to run or not run specific tests. See this doc for details. This is useful when you'd like to skip failing tests or want to debug specific test(s).

You can exclude a test by adding X or P in front of Describe or It:

XDescribe("some behavior", func() { ... })
XIt("some assertion", func() {...})

And to run only specific tests, add F in front Describe or It:

FDescribe("some behavior", func() { ... })
FIt("some assertion", func() { ... })

Documentation

Index

Constants

View Source 
const (
	ServingRuntimeKind         = "ServingRuntime"
	ClusterServingRuntimeKind  = "ClusterServingRuntime"
	PredictorKind              = "Predictor"
	IsvcKind                   = "InferenceService"
	ConfigMapKind              = "ConfigMap"
	SecretKind                 = "Secret"
	DefaultTestNamespace       = "modelmesh-serving"
	DefaultTestServiceName     = "modelmesh-serving"
	DefaultControllerNamespace = "modelmesh-serving"
	UserConfigMapName          = "model-serving-config"
	SamplesPath                = "predictors/"
	IsvcSamplesPath            = "isvcs/"
	RuntimeSamplesPath         = "runtimes/"
	TLSSecretName              = "fvt-tls-secret"
	StorageConfigSecretName    = "storage-config"
)
View Source 
const EPSILON float64 = 0.000001

Used for checking if floats are sufficiently close enough.

View Source 
const PredictorTimeout = time.Second * 120 // absolute time to wait for predictor to become ready
View Source 
const TimeForStatusToStabilize = time.Second * 10 // time to wait between watcher events before assuming a stable state

Variables

View Source 
var BasicTLSConfig = map[string]interface{}{
	"tls": map[string]interface{}{
		"secretName": TLSSecretName,
		"clientAuth": "optional",

		"headlessService": false,
	},
}
View Source 
var DefaultConfig = map[string]interface{}{
	"podsPerRuntime": 1,
	"restProxy": map[string]interface{}{
		"enabled": true,
	},
	"scaleToZero": map[string]interface{}{
		"enabled": false,
	},
	"internalModelMeshEnvVars": []map[string]interface{}{
		{
			"name":  "BOOTSTRAP_CLEARANCE_PERIOD_MS",
			"value": "0",
		},
	},
}
View Source 
var DefaultTimeout = int64(120) // absolute timeout for watcher event channels
View Source 
var Log logr.Logger
View Source 
var MutualTLSConfig = map[string]interface{}{
	"tls": map[string]interface{}{
		"secretName": TLSSecretName,
		"clientAuth": "require",

		"headlessService": false,
	},
}
View Source 
var NameSpaceScopeMode = false
View Source 
var StorageConfigDataMinio = map[string]interface{}{
	"localMinIO": map[string]string{
		"type":              "s3",
		"access_key_id":     "AKIAIOSFODNN7EXAMPLE",
		"secret_access_key": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
		"endpoint_url":      "http://minio:9000",
		"default_bucket":    "modelmesh-example-models",
		"region":            "us-south",
	},
}
View Source 
var StorageConfigDataPVC = map[string]interface{}{
	"pvc1": map[string]string{
		"type": "pvc",
		"name": "models-pvc-1",
	},
	"pvc2": map[string]string{
		"type": "pvc",
		"name": "models-pvc-2",
	},
}

Functions

func CreateIsvcAndWaitAndExpectFailed added in v0.11.0 

func CreateIsvcAndWaitAndExpectFailed(isvcManifest *unstructured.Unstructured) *unstructured.Unstructured

func CreateIsvcAndWaitAndExpectReady added in v0.9.0 

func CreateIsvcAndWaitAndExpectReady(isvcManifest *unstructured.Unstructured, timeout time.Duration) *unstructured.Unstructured

func CreatePredictorAndWaitAndExpectFailed 

func CreatePredictorAndWaitAndExpectFailed(predictorManifest *unstructured.Unstructured) *unstructured.Unstructured

func CreatePredictorAndWaitAndExpectInvalidSpec 

func CreatePredictorAndWaitAndExpectInvalidSpec(predictorManifest *unstructured.Unstructured) *unstructured.Unstructured

func CreatePredictorAndWaitAndExpectLoaded 

func CreatePredictorAndWaitAndExpectLoaded(predictorManifest *unstructured.Unstructured) *unstructured.Unstructured

func CreateSecret added in v0.9.0 

func CreateSecret(secret *corev1.Secret, namespace string, fvt *FVTClient)

func DecodeResourceFromFile 

func DecodeResourceFromFile(resourcePath string) *unstructured.Unstructured

func ExpectIsvcFailureInfo added in v0.11.0 

func ExpectIsvcFailureInfo(obj *unstructured.Unstructured, reason string, hasLocation bool, hasTime bool, message string)

func ExpectIsvcState added in v0.11.0 

func ExpectIsvcState(obj *unstructured.Unstructured, activeModelState, targetModelState, transitionStatus string)

func ExpectPredictorFailureInfo 

func ExpectPredictorFailureInfo(obj *unstructured.Unstructured, reason string, hasLocation bool, hasTime bool, message string)

func ExpectPredictorState 

func ExpectPredictorState(obj *unstructured.Unstructured, available bool, activeModelState, targetModelState, transitionStatus string)

func ExpectSuccessfulInference_kerasMnist added in v0.9.0 

func ExpectSuccessfulInference_kerasMnist(predictorName string)

Keras MNIST COS path: fvt/tensorflow/keras-mnist

func ExpectSuccessfulInference_lightgbmMushroom 

func ExpectSuccessfulInference_lightgbmMushroom(predictorName string)

LightGBM Mushroom COS path: fvt/lightgbm/mushroom

func ExpectSuccessfulInference_onnxMnist 

func ExpectSuccessfulInference_onnxMnist(predictorName string)

ONNX MNIST COS path: fvt/onnx/onnx-mnist

func ExpectSuccessfulInference_openvinoMnistTFSPredict added in v0.9.0 

func ExpectSuccessfulInference_openvinoMnistTFSPredict(predictorName string)

func ExpectSuccessfulInference_pytorchCifar 

func ExpectSuccessfulInference_pytorchCifar(predictorName string)

PyTorch CIFAR COS path: fvt/pytorch/pytorch-cifar

func ExpectSuccessfulInference_sklearnMnistSvm 

func ExpectSuccessfulInference_sklearnMnistSvm(predictorName string)

SKLearn MNIST SVM COS path: fvt/sklearn/mnist-svm

func ExpectSuccessfulInference_tensorflowMnist 

func ExpectSuccessfulInference_tensorflowMnist(predictorName string)

Tensorflow MNIST COS path: fvt/tensorflow/mnist.savedmodel

func ExpectSuccessfulInference_torchserveMARPredict added in v0.11.0 

func ExpectSuccessfulInference_torchserveMARPredict(predictorName string)

func ExpectSuccessfulInference_xgboostMushroom 

func ExpectSuccessfulInference_xgboostMushroom(predictorName string)

XGBoost Mushroom COS path: fvt/xgboost/mushroom

func ExpectSuccessfulRESTInference_sklearnMnistSvm added in v0.9.0 

func ExpectSuccessfulRESTInference_sklearnMnistSvm(predictorName string)

func ExpectSuccessfulRESTInference_xgboostMushroom added in v0.9.0 

func ExpectSuccessfulRESTInference_xgboostMushroom(predictorName string, tls bool)

func GetBool 

func GetBool(obj *unstructured.Unstructured, fieldPath ...string) bool

func GetInt64 

func GetInt64(obj *unstructured.Unstructured, fieldPath ...string) int64

func GetMap 

func GetMap(obj *unstructured.Unstructured, fieldPath ...string) map[string]interface{}

func GetSlice added in v0.9.0 

func GetSlice(obj *unstructured.Unstructured, fieldPath ...string) ([]interface{}, bool)

func GetString 

func GetString(obj *unstructured.Unstructured, fieldPath ...string) string

func InitializeFVTClient added in v0.11.0 

func InitializeFVTClient()

func LoadCifarImage 

func LoadCifarImage(index int) []float32

func LoadMnistImage 

func LoadMnistImage(index int) []float32

func MakeUniquePredictorName 

func MakeUniquePredictorName(base string) string

to enable tests to run in parallel even when loading from the same Predictor sample

func NewIsvcForFVT added in v0.9.0 

func NewIsvcForFVT(filename string) *unstructured.Unstructured

func NewPredictorForFVT 

func NewPredictorForFVT(filename string) *unstructured.Unstructured

func SetMap added in v0.11.0 

func SetMap(obj *unstructured.Unstructured, value map[string]interface{}, fieldPath ...string)

func SetString 

func SetString(obj *unstructured.Unstructured, value string, fieldPath ...string)

func TestDataPath added in v0.9.0 

func TestDataPath(resourcePathWithinTestData string) string

Utility function to return the testdata directory

func UpdatePredictorAndWaitAndExpectFailed 

func UpdatePredictorAndWaitAndExpectFailed(predictorManifest *unstructured.Unstructured) *unstructured.Unstructured

func UpdatePredictorAndWaitAndExpectLoaded 

func UpdatePredictorAndWaitAndExpectLoaded(predictorManifest *unstructured.Unstructured) *unstructured.Unstructured

func WaitForIsvcState added in v0.11.0 

func WaitForIsvcState(watcher watch.Interface, anyOfDesiredStates []api.ModelState, name string, isvcTimeout time.Duration) *unstructured.Unstructured

func WaitForLastStateInExpectedList 

func WaitForLastStateInExpectedList(statusAttribute string, expectedStates []string, watcher watch.Interface) *unstructured.Unstructured

Waiting for predictor state to reach the last one in the expected list Predictor state is allowed to directly reach the last state in the expected list i.e; Loaded. Also, Predictor state can be one of the earlier states (i.e; Pending or Loading), but state change should happen in the following order: [Pending => Loaded] (or) [Pending => Loading => Loaded] (or) [Loading => Loaded] (or) [Pending => Loading => FailedToLoad]

func WaitForRuntimeDeploymentsToBeStable added in v0.11.0 

func WaitForRuntimeDeploymentsToBeStable(timeToStabilize time.Duration, watcher watch.Interface)

func WaitForStableActiveDeployState 

func WaitForStableActiveDeployState(timeToStabilize time.Duration)

Types

type CertGenerator added in v0.9.0 

type CertGenerator struct {
	Namespaces    []string
	ServiceName   string
	CAPEM         *bytes.Buffer
	PublicKeyPEM  *bytes.Buffer
	PrivateKeyPEM *bytes.Buffer
}

type FVTClient 

type FVTClient struct {
	dynamic.Interface
	// contains filtered or unexported fields
}
var FVTClientInstance *FVTClient

func GetFVTClient 

func GetFVTClient(log logr.Logger, namespace, serviceName, controllerNamespace string) (*FVTClient, error)

func (*FVTClient) ApplyClusterServingRuntimeExpectSuccess added in v0.11.0 

func (fvt *FVTClient) ApplyClusterServingRuntimeExpectSuccess(clusterServingRuntime *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) ApplyPredictorExpectSuccess 

func (fvt *FVTClient) ApplyPredictorExpectSuccess(predictor *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) ApplyServingRuntimeExpectSuccess added in v0.11.0 

func (fvt *FVTClient) ApplyServingRuntimeExpectSuccess(servingRuntime *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) ApplyUserConfigMap 

func (fvt *FVTClient) ApplyUserConfigMap(config map[string]interface{})

func (*FVTClient) ConnectToModelServing 

func (fvt *FVTClient) ConnectToModelServing(connectionType ModelServingConnectionType) error

func (*FVTClient) CreateIsvcExpectSuccess added in v0.9.0 

func (fvt *FVTClient) CreateIsvcExpectSuccess(resource *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) CreatePredictorExpectSuccess 

func (fvt *FVTClient) CreatePredictorExpectSuccess(resource *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) CreateStorageConfigSecret added in v0.11.0 

func (fvt *FVTClient) CreateStorageConfigSecret(storageConfigData map[string]interface{})

func (*FVTClient) CreateTLSSecrets 

func (fvt *FVTClient) CreateTLSSecrets()

func (*FVTClient) DeleteAllIsvcs added in v0.9.0 

func (fvt *FVTClient) DeleteAllIsvcs()

func (*FVTClient) DeleteAllPredictors 

func (fvt *FVTClient) DeleteAllPredictors()

func (*FVTClient) DeleteConfigMap 

func (fvt *FVTClient) DeleteConfigMap(resourceName string) error

func (*FVTClient) DeleteIsvc added in v0.9.0 

func (fvt *FVTClient) DeleteIsvc(resourceName string)

func (*FVTClient) DeletePredictor 

func (fvt *FVTClient) DeletePredictor(resourceName string)

func (*FVTClient) DeleteSecret 

func (fvt *FVTClient) DeleteSecret(resourceName string, namespace string) error

func (*FVTClient) DeleteStorageConfigSecret added in v0.11.0 

func (fvt *FVTClient) DeleteStorageConfigSecret()

func (FVTClient) DeleteTLSSecrets 

func (fvt FVTClient) DeleteTLSSecrets()

func (*FVTClient) DisconnectFromModelServing 

func (fvt *FVTClient) DisconnectFromModelServing()

func (*FVTClient) GetClusterServingRuntime added in v0.11.0 

func (fvt *FVTClient) GetClusterServingRuntime(name string) *unstructured.Unstructured

func (*FVTClient) GetPredictor 

func (fvt *FVTClient) GetPredictor(name string) *unstructured.Unstructured

func (*FVTClient) GetRandomReadyRuntimePod added in v0.11.0 

func (fvt *FVTClient) GetRandomReadyRuntimePod() string

func (*FVTClient) GetServingRuntime 

func (fvt *FVTClient) GetServingRuntime(name string) *unstructured.Unstructured

func (*FVTClient) ListClusterServingRuntimes added in v0.10.0 

func (fvt *FVTClient) ListClusterServingRuntimes(options metav1.ListOptions) (*unstructured.UnstructuredList, error)

func (*FVTClient) ListDeploys 

func (fvt *FVTClient) ListDeploys() appsv1.DeploymentList

func (*FVTClient) ListHPAs added in v0.11.0 

func (fvt *FVTClient) ListHPAs() hpav2.HorizontalPodAutoscalerList

func (*FVTClient) ListPredictors 

func (fvt *FVTClient) ListPredictors(options metav1.ListOptions) *unstructured.UnstructuredList

func (*FVTClient) ListReadyRuntimePods added in v0.11.0 

func (fvt *FVTClient) ListReadyRuntimePods() corev1.PodList

func (*FVTClient) ListServingRuntimes 

func (fvt *FVTClient) ListServingRuntimes(options metav1.ListOptions) (*unstructured.UnstructuredList, error)

func (*FVTClient) PrintDescribeIsvc added in v0.11.0 

func (fvt *FVTClient) PrintDescribeIsvc(name string)

func (*FVTClient) PrintDescribeNodes added in v0.9.0 

func (fvt *FVTClient) PrintDescribeNodes()

func (*FVTClient) PrintEvents added in v0.9.0 

func (fvt *FVTClient) PrintEvents()

func (*FVTClient) PrintHPAs added in v0.11.0 

func (fvt *FVTClient) PrintHPAs()

func (*FVTClient) PrintIsvcs added in v0.9.0 

func (fvt *FVTClient) PrintIsvcs()

func (*FVTClient) PrintPods added in v0.9.0 

func (fvt *FVTClient) PrintPods()

func (*FVTClient) PrintPredictors 

func (fvt *FVTClient) PrintPredictors()

func (*FVTClient) RestartDeploys 

func (fvt *FVTClient) RestartDeploys()

func (*FVTClient) RunKfsInference 

func (fvt *FVTClient) RunKfsInference(req *inference.ModelInferRequest) (*inference.ModelInferResponse, error)

func (*FVTClient) RunKfsModelMetadata added in v0.9.0 

func (fvt *FVTClient) RunKfsModelMetadata(req *inference.ModelMetadataRequest) (*inference.ModelMetadataResponse, error)

func (*FVTClient) RunKfsRestInference added in v0.9.0 

func (fvt *FVTClient) RunKfsRestInference(modelName string, body []byte, tls bool) (string, error)

func (*FVTClient) RunKubectl 

func (fvt *FVTClient) RunKubectl(args ...string) error

func (*FVTClient) RunTfsInference added in v0.9.0 

func (fvt *FVTClient) RunTfsInference(req *tfsapi.PredictRequest) (*tfsapi.PredictResponse, error)

func (*FVTClient) RunTorchserveInference added in v0.11.0 

func (fvt *FVTClient) RunTorchserveInference(req *torchserveapi.PredictionsRequest) (*torchserveapi.PredictionResponse, error)

func (*FVTClient) SetDefaultUserConfigMap added in v0.9.0 

func (fvt *FVTClient) SetDefaultUserConfigMap()

func (*FVTClient) SetServingRuntimeAnnotation added in v0.11.0 

func (fvt *FVTClient) SetServingRuntimeAnnotation(expectedRuntimeName string, annotations map[string]interface{})

func (*FVTClient) StartWatchingDeploys 

func (fvt *FVTClient) StartWatchingDeploys() watch.Interface

func (*FVTClient) StartWatchingIsvcs added in v0.9.0 

func (fvt *FVTClient) StartWatchingIsvcs(options metav1.ListOptions, timeoutSeconds int64) watch.Interface

func (*FVTClient) StartWatchingPredictors 

func (fvt *FVTClient) StartWatchingPredictors(options metav1.ListOptions, timeoutSeconds int64) watch.Interface

func (*FVTClient) TailPodLogs 

func (fvt *FVTClient) TailPodLogs(sinceTime string)

func (*FVTClient) UpdateConfigMapTLS 

func (fvt *FVTClient) UpdateConfigMapTLS(tlsConfig map[string]interface{})

func (*FVTClient) UpdatePredictorExpectSuccess 

func (fvt *FVTClient) UpdatePredictorExpectSuccess(resource *unstructured.Unstructured) *unstructured.Unstructured

func (*FVTClient) WatchPredictorsAsync 

func (fvt *FVTClient) WatchPredictorsAsync(c chan *unstructured.Unstructured, options metav1.ListOptions, timeoutSeconds int64)

type ModelMeshPortForward added in v0.9.0 

type ModelMeshPortForward struct {
	// contains filtered or unexported fields
}

func NewModelMeshPortForward added in v0.9.0 

func NewModelMeshPortForward(namespace string, podName string, localPort int, targetPort int, log logr.Logger) *ModelMeshPortForward

NewModelMeshPortForward switched to port-forwarding to a pod instead of the service, since, when port-forwarding to a Service, it just picks any pod to port-forward to without any guardrails against selecting a Terminating pod. It doesn't use readiness checks for pod selection, it seems to actually select the oldest pod which ends up being most likely to be terminated soon

func (*ModelMeshPortForward) EnsureStarted added in v0.9.0 

func (pf *ModelMeshPortForward) EnsureStarted() error

func (*ModelMeshPortForward) EnsureStopped added in v0.9.0 

func (pf *ModelMeshPortForward) EnsureStopped()

type ModelServingConnectionType 

type ModelServingConnectionType int
const (
	Insecure ModelServingConnectionType = iota
	TLS
	MutualTLS
)

Source Files

View all Source files

Directories

Path Synopsis
tensorflow/core/framework
tensorflow_serving/apis
torchserve/apis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.