README
¶
Generic Network Function (GNF)
Introduction
There are two types of GNFs: GNF (with controller) and GNF-Cli (without controller). While the normal GNFs wait for controller commands to stat/stop benchmarking at almost the same time, a GNF-Cli instance can be used for debugging or testing purposes.
Architecture & Design
Please see our paper for HotNets 2019
Implementation
Interfaces
Database Client
DBClient in database.go. When adding a new type of storage node (i.e. other than Redis), need to implement this interface with two methods.
type DBClient interface {
DBWrite(key, val string) error
DBRead(key string) (string, error)
}
Also, add an entry in function getRemoteDBClients() in database.go, and an entry in isValidRemoteDB() in utilities.go
to make a string that represents this type of DB can be picked up from a workload file.
Operation Generator
OpGenerator in generator.go. When implementing a new kind of request distribution, need to implement this interface.
type OpGenerator interface {
GenThread(allToExe chan<- exeCmd, exeToGen <-chan bool, genToCli chan<- genCmd, wl *Workload, phase exePhase)
}
Also, add an entry in function getOpGenerator() in generator.go, and en entry in isValidDistribution() in utilities.go
Threads / Goroutines
Besides GenThread() above in Operation Generator section, there are a few more goroutines:
exeRecvThread() in control.go: receives controller signals (`UserData` struct in outer package) and interpret into executor commands;
type UserData struct {
Action string // load, run, stop, or interrupt
NewWorkLoad bool // NewWorkLoad is true if WorkLoadFile is a string of wl parameters and values;
WorkLoadFile string // NewWorkLoad is false if WorkLoadFile is a gnf instance local file path
GnfIPS []string
}
exeSendThread() in control.go: sends benchmark statistics from executor to the controller;
package gnf
type BmStats struct {
IP string
Timestamp string // e.g. get from time.Now().String()
Runtime float64
Throughput float64
SRead int
SReadAvgLat float64
SRead95pLat float64
SWrite int
SWriteAvgLat float64
SWrite95pLat float64
FRead int
FReadAvgLat float64
FRead95pLat float64
FWrite int
FWriteAvgLat float64
FWrite95pLat float64
}
func (bm *BmStats) String() string {
// performs some string formatting
// returns a YCSB-like string representation of bm
}
func EncodeBmStat(data *BmStats) []byte {
// returns an empty byte array upon error
// returns the encoded byte array if succeed
}
func DecodeBmStat(dataBytes []byte) BmStats {
// returns an empty BmStats upon error
// returns the decoded byte array if succeed
}
exeSignThread() in control.go: receives OS signals and interpret into executor commands;
cliThread() in gnf.go: performs DB operations according to GenThread() commands;
staThread() in gnf.go: converges DB Client Threads' per-request statistics to one benchmark statistics.
Concurrent Logic
GnfMain() in gnf.go: interpret command line arguments and spawns GNF or GNF-Cli
mainRoutine() in gnf.go: GNF executor's main routine, incl. responses to executor commands (e.g. start benchmarking)
benchmarkRoutine() in gnf.go: GNF executor's routine while benchmarking.
Compared to the main routine, responses to commands and exception handling are different.
For Developers:
Add a Workload Parameter
- choose a proper section (e.g. remoteDB), a proper parameter name and the default value in
workload_template- leave the right hand side of the equal sign empty is the default value is the zero value of that type
- In
workload.go, add the parameter (need to capitalize the first letter) in the corresponding place- method
UpdateWorkloadByLineautomatically check whether int > 0 or float between 0 - 1 (inclusive). - If there are special rules, add a method that starts with
isValidinutilities.go(e.g.isValidRemoteDB,isValidDistribution, andisValidKeyRange). Then insert this method in the rightcasestatement inUpdateWorkloadByLine
- method
- Use it wherever you have a *Workload
- Add some unit tests if you introduce new methods
Future Work
Implements more operation generator according to different distributions;
Add more DB backend support;
Add more unit tests.
Documentation
¶
Index ¶
Constants ¶
const ( MaxIntInFloat = float64(math.MaxInt64) // assume the platform supports MaxUintInFloat = float64(math.MaxUint64) )
const ( LoadPhase exePhase = "load" RunPhase = "run" DoRead genSig = "read" DoWrite = "write" NormalExit exeSig = "exit" // followed by a return ErrorExit = "exception" // followed by a return GnfStop = "stop" // never followed by a return BmkStop = "bmStop" // never followed by a return CtrlLoad = "load" CtrlRun = "run" Ready = "ready" // for two comm threads only )
const Letters = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
Variables ¶
This section is empty.
Functions ¶
func EncodeBmStat ¶
func GnfMain ¶
func GnfMain() error
gnf main method parses command line argument and spawns one of the two gnfs: gnf: gnf with controller, need to specify controller ip, nf port and stat port gnf-cli: gnf without controller, need to specify load/run phase and workload file
returns some_error_with_msg if there's an exception or nil if everything works as intended
Types ¶
type BmStats ¶
type BmStats struct {
IP string
Timestamp string // e.g. get from time.Now().String()
Runtime float64
Throughput float64
SRead int
SReadAvgLat float64
SRead95pLat float64
SWrite int
SWriteAvgLat float64
SWrite95pLat float64
FRead int
FReadAvgLat float64
FRead95pLat float64
FWrite int
FWriteAvgLat float64
FWrite95pLat float64
}
TODO: change the name to StatsMsg from statistics thread to executor then to pub thread / print out
func DecodeBmStat ¶
type OpGenerator ¶
type OpGenerator interface {
/*
exit condition: receive from exeToGen OR offered operation keys required by the workload
upon exit: it closes genToCli channel
upon exception: no known possible exception, no exception signal
*/
GenThread(allToExe chan<- exeCmd, exeToGen <-chan bool, genToCli chan<- genCmd, wl *Workload, phase exePhase)
}
type RedisClient ¶
type UniformOpGenerator ¶
type UniformOpGenerator struct {
}
func (*UniformOpGenerator) GenThread ¶
func (gen *UniformOpGenerator) GenThread(allToExe chan<- exeCmd, exeToGen <-chan bool, genToCli chan<- genCmd, wl *Workload, phase exePhase)
type Workload ¶
type Workload struct {
RemoteDB string
RemoteDBHost string
RemoteDBPort int
RemoteDBPassword string
RemoteDBLoadThreadCount int
RemoteDBRunThreadCount int
RemoteDBInsertKeyRange keyRanges
RemoteDBInsertValueSizeInByte int
RemoteDBOperationCount int
RemoteDBOperationRange keyRanges
RemoteDBOperationDistribution string
RemoteDBGlobalMaximumKeyIndex int // not in use for now
RemoteDBReadRatio float64
RemoteDBWriteRatio float64
}
func InitWorkload ¶
func InitWorkload() *Workload
func (*Workload) UpdateWorkloadByFile ¶
return -1 if there's an exception (no workload file found) or a updateCount, could >= 0, if the workload object is updated
func (*Workload) UpdateWorkloadByLine ¶
Update a workload parameter by feeding a line that looks like param=value if found no update: return 0, nil; if found one update: return 1, nil; if found an ill update (found a field but unsuitable value): return -1, some_error_message
type ZipfianOpGenerator ¶
type ZipfianOpGenerator struct {
}
func (*ZipfianOpGenerator) GenThread ¶
func (gen *ZipfianOpGenerator) GenThread(allToExe chan<- exeCmd, exeToGen <-chan bool, genToCli chan<- genCmd, wl *Workload, phase exePhase)
Source Files