tutils

package
v0.0.0-...-fe65812 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 24, 2019 License: MIT Imports: 34 Imported by: 0

Documentation

Overview

  • Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. *

Package tutils provides common low-level utilities for all aistore unit and integration tests 

FIXME -- FIXME: split and transform it into the: a) client API and b) test utilities
FIXME -- FIXME: the client API must then move into the api package

* Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
*

Package tutils provides common low-level utilities for all aistore unit and integration tests

  • Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. *

Package tutils provides common low-level utilities for all aistore unit and integration tests

  • Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. *

Package tutils provides common low-level utilities for all aistore unit and integration tests

  • Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. *

Package tutils provides common low-level utilities for all aistore unit and integration tests

  • Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved. *

Index

Constants

View Source 
const (
	// ReaderTypeFile defines the name for file reader
	ReaderTypeFile = "file"
	// ReaderTypeSG defines the name for sg reader
	ReaderTypeSG = "sg"
	// ReaderTypeRand defines the name for rand reader
	ReaderTypeRand = "rand"
	// ReaderTypeInMem defines the name for inmem reader
	ReaderTypeInMem = "inmem"
)

Variables

View Source 
var (
	BaseHTTPClient = &http.Client{}
	HTTPClient     = &http.Client{
		Timeout:   600 * time.Second,
		Transport: transport,
	}
	Mem2 *memsys.Mem2
)

Functions

func AbortDSort 

func AbortDSort(proxyURL, managerUUID string) error

func BaseAPIParams 

func BaseAPIParams(url string) *api.BaseParams

func CheckFatal 

func CheckFatal(err error, t *testing.T)

func ClusterCount 

func ClusterCount() int

ClusterCount returns the current number of clusters running

func ConnectContainer 

func ConnectContainer(containerID string, networks []string) error

ConnectContainer connects specific containerID to all provided networks.

func ContainerCount 

func ContainerCount(clusterNumber ...int) (proxyCnt int, targetCnt int)

ContainerCount is used by clusterHealthCheck to test if any container crashed after a test completes

func ContainerExec 

func ContainerExec(containerName string, args ...string) error

ContainerExec executes a docker exec command for containerName

func CreateFreshLocalBucket 

func CreateFreshLocalBucket(t *testing.T, proxyURL, bucketFQN string)

func CreateTarWithCustomFiles 

func CreateTarWithCustomFiles(tarName string, fileCnt, fileSize int, customFileType string, customFileExt string) error

func CreateTarWithRandomFiles 

func CreateTarWithRandomFiles(tarName string, gzipped bool, fileCnt int, fileSize int) error

CreateTarWithRandomFiles creates tar with specified number of files. Tar is also gziped if necessary.

func CreateZipWithRandomFiles 

func CreateZipWithRandomFiles(zipName string, fileCnt, fileSize int) error

func DefaultBaseAPIParams 

func DefaultBaseAPIParams(t *testing.T) *api.BaseParams

func Del 

func Del(proxyURL, bucket string, object string, wg *sync.WaitGroup, errCh chan error, silent bool) error

func DeleteList 

func DeleteList(proxyURL, bucket string, fileslist []string, wait bool, deadline time.Duration) error

func DeleteRange 

func DeleteRange(proxyURL, bucket, prefix, regex, rng string, wait bool, deadline time.Duration) error

func DestroyLocalBucket 

func DestroyLocalBucket(t *testing.T, proxyURL, bucket string)

func DisconnectContainer 

func DisconnectContainer(containerID string) ([]string, error)

DisconnectContainer diconnects specific containerID from all networks. Returns all networks from which has been disconnected.

func DockerCreateMpathDir 

func DockerCreateMpathDir(c int, mpathFQN string) (err error)

DockerCreateMpathDir creates a directory that will be used as a mountpath for each target in cluster c

func DockerRemoveMpathDir 

func DockerRemoveMpathDir(c int, mpathFQN string) (err error)

DockerRemoveMpathDir removes a directory named mpathFQN for each target in cluster c

func DockerRunning 

func DockerRunning() bool

DockerRunning returns true if docker-based AIStore cluster is detected

func DoesLocalBucketExist 

func DoesLocalBucketExist(serverURL string, bucket string) (bool, error)

DoesLocalBucketExist queries a proxy or target to get a list of all local buckets, returns true if the bucket exists.

func EvictList 

func EvictList(proxyURL, bucket string, fileslist []string, wait bool, deadline time.Duration) error

func EvictRange 

func EvictRange(proxyURL, bucket, prefix, regex, rng string, wait bool, deadline time.Duration) error

func FastRandomFilename 

func FastRandomFilename(src *rand.Rand, fnlen int) string

func GetFileInfosFromTarBuffer 

func GetFileInfosFromTarBuffer(buffer bytes.Buffer, gzipped bool) ([]os.FileInfo, error)

GetFileInfosFromTarBuffer returns all file infos contained in buffer which assumably is tar or gzipped tar.

func GetFileInfosFromZipBuffer 

func GetFileInfosFromZipBuffer(buffer bytes.Buffer) ([]os.FileInfo, error)

GetFileInfosFromZipBuffer returns all file infos contained in buffer which assumably is zip.

func GetPrimaryProxy 

func GetPrimaryProxy(proxyURL string) (string, error)

GetPrimaryProxy returns the primary proxy's url of a cluster

func GetWhatRawQuery 

func GetWhatRawQuery(getWhat string, getProps string) string

func GetXactionResponse 

func GetXactionResponse(proxyURL string, kind string) ([]byte, error)

TODO: rename, and move to the api package

func IsCached 

func IsCached(proxyURL, bucket, objname string) (bool, error)

func ListObjects 

func ListObjects(proxyURL, bucket, prefix string, objectCountLimit int) ([]string, error)

ListObjects returns a slice of object names of all objects that match the prefix in a bucket

func Logf 

func Logf(msg string, args ...interface{})

func Logln 

func Logln(msg string)

func MetricsDSort 

func MetricsDSort(proxyURL, managerUUID string) (map[string]*dsort.Metrics, error)

func ParseEnvVariables 

func ParseEnvVariables(fpath string, delimiter ...string) map[string]string

ParseEnvVariables takes in a .env file and parses its contents

func PrefetchList 

func PrefetchList(proxyURL, bucket string, fileslist []string, wait bool, deadline time.Duration) error

func PrefetchRange 

func PrefetchRange(proxyURL, bucket, prefix, regex, rng string, wait bool, deadline time.Duration) error

func Progress 

func Progress(id int, period int)

func PutAsync 

func PutAsync(wg *sync.WaitGroup, proxyURL, bucket, object string, reader Reader, errCh chan error)

PutAsync sends a PUT request to the given URL

func PutObjsFromList 

func PutObjsFromList(proxyURL, bucket, readerPath, readerType, objPath string, objSize uint64, objList []string,
	errCh chan error, objsPutCh chan string, sgl *memsys.SGL)

func PutRandObjs 

func PutRandObjs(proxyURL, bucket, readerPath, readerType, objPath string, objSize uint64, numPuts int,
	errCh chan error, objsPutCh chan string, sgl *memsys.SGL)

func RandomObjDir 

func RandomObjDir(src *mrand.Rand, dirLen, maxDepth int) (dir string)

func RegisterTarget 

func RegisterTarget(proxyURL string, targetNode *cluster.Snode, smap cluster.Smap) error

func ReplicateMultipleObjects 

func ReplicateMultipleObjects(proxyURL string, bucketToObjects map[string][]string) map[string]error

ReplicateMultipleObjects replicates all the objects in the map bucketToObjects. bucketsToObjects is a key value pairing where the keys are bucket names and the corresponding value is a slice of objects. ReplicateMultipleObjects returns a map of errors where the key is bucket+"/"+object and the corresponding value is the error that caused replication to fail.

func RestartContainer 

func RestartContainer(cid string) error

RestartContainer restores previously killed target or proxy with given container id

func StartDSort 

func StartDSort(proxyURL string, rs dsort.RequestSpec) (string, error)

func StopContainer 

func StopContainer(cid string) error

StopContainer simulates killing a target or proxy (by given container id)

func TargetsInCluster 

func TargetsInCluster(i int) (ans []string)

TargetsInCluster returns the names of the targets in cluster i

func Tcping 

func Tcping(url string) (err error)

func UnregisterTarget 

func UnregisterTarget(proxyURL, sid string) error

func WaitForDSortToFinish 

func WaitForDSortToFinish(proxyURL, managerUUID string) (bool, error)

func WaitMapVersionSync 

func WaitMapVersionSync(timeout time.Time, smap cluster.Smap, prevVersion int64, idsToIgnore []string) error

Types

type FileContent 

type FileContent struct {
	Name    string
	Ext     string
	Content []byte
}

func GetFilesFromTarBuffer 

func GetFilesFromTarBuffer(buffer bytes.Buffer, extension string) ([]FileContent, error)

GetFilesFromTarBuffer returns all file infos contained in buffer which assumably is tar or gzipped tar.

type HTTPLatencies 

type HTTPLatencies struct {
	ProxyConn           time.Duration // from request is created to proxy connection is established
	Proxy               time.Duration // from proxy connection is established to redirected
	TargetConn          time.Duration // from request is redirected to target connection is established
	Target              time.Duration // from target connection is established to request is completed
	PostHTTP            time.Duration // from http ends to after read data from http response and verify hash (if specified)
	ProxyWroteHeader    time.Duration // from ProxyConn to header is written
	ProxyWroteRequest   time.Duration // from ProxyWroteHeader to response body is written
	ProxyFirstResponse  time.Duration // from ProxyWroteRequest to first byte of response
	TargetWroteHeader   time.Duration // from TargetConn to header is written
	TargetWroteRequest  time.Duration // from TargetWroteHeader to response body is written
	TargetFirstResponse time.Duration // from TargetWroteRequest to first byte of response
}

HTTPLatencies stores latency of a http request

func GetConfig 

func GetConfig(server string) (HTTPLatencies, error)

GetConfig sends a {what:config} request to the url and discard the message For testing purpose only

func GetWithMetrics 

func GetWithMetrics(url, bucket string, keyname string, validate bool, offset, length int64) (int64, HTTPLatencies, error)

Get sends a GET request to url and discards returned data

func PutWithMetrics 

func PutWithMetrics(url, bucket, object, hash string, reader cmn.ReadOpenCloser) (HTTPLatencies, error)

Put sends a PUT request to url

type ParamReader 

type ParamReader struct {
	Type       string      // file | sg | inmem | rand
	SGL        *memsys.SGL // When Type == sg
	Path, Name string      // When Type == file; path and name of file to be created (if not already existing)
	Size       int64
}

ParamReader is used to pass in parameters when creating a new reader

type Reader 

type Reader interface {
	io.ReadCloser
	io.Seeker
	Open() (io.ReadCloser, error)
	XXHash() string
	Description() string
}

Reader is the interface a client works with to read in data and send to a HTTP server

func NewBytesReader 

func NewBytesReader(buf []byte) Reader

NewBytesReader returns a new bytesReader

func NewFileReader 

func NewFileReader(filepath, name string, size int64, withHash bool) (Reader, error)

NewFileReader creates/opens the file, populates it with random data, closes it and returns a new fileReader

func NewFileReaderFromFile 

func NewFileReaderFromFile(fn string, withHash bool) (Reader, error)

NewFileReaderFromFile opens an existing file, reads it to compute checksum, closes it, and returns a new reader. See also (and note the difference from): NewFileReader

func NewInMemReader 

func NewInMemReader(size int64, withHash bool) (Reader, error)

NewInMemReader returns a new inMemReader

func NewRandReader 

func NewRandReader(size int64, withHash bool) (Reader, error)

NewRandReader returns a new randReader

func NewReader 

func NewReader(p ParamReader) (Reader, error)

NewReader returns a data reader; type of reader returned is based on the parameters provided

func NewSGReader 

func NewSGReader(sgl *memsys.SGL, size int64, withHash bool) (Reader, error)

NewSGReader returns a new sgReader

type ReqError 

type ReqError struct {
	// contains filtered or unexported fields
}

func NewReqError 

func NewReqError(msg string, code int) ReqError

func (ReqError) Error 

func (err ReqError) Error() string

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.