utils

package
v0.0.0-...-bc93099 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 29, 2024 License: MIT Imports: 36 Imported by: 0

README

(C) 2020 Geotab Inc

All files and artifacts in this repository are licensed under the provisions of the license provided by the LICENSE file in this repository.

Two experimental (i. e. not part of the VISSv2 standard) compression solutions are implemented:

  • Protobuf: Based on protobuf. For more information, see below, in README in the protobuf, and client/client-1.0 directories. The code that transforms payload messages from json to protobuf, and back, are found in pbutils.go.

  • Proprietary: Based on a proprietary algorithm that is explained below.
    The code that transforms payload messages from json to protobuf, and back, are found in computils.go.
    The proprietary solution currently only supports requests where the path points to a leaf node in the VSS tree.

The compression solutions are currently supported over the websocket transport only, where it is signalled in the subprotocol parameter at session initiation. For HTTP support, it could be signalled in a header. For MQTT, a new key-value parameter could be added to the application protocol.

Proprietary reference compression design

The design is context aware, i. e. it knows that the payload is JSON, and the set of key-value pairs that can be expected. As the payload sizes are typically less than 100 characters, this is probably necessary to achieve high compression ratios. Tests with standard compression algorithms like GZIP, LZW, and DEFLATE showed comparably low compression ratios.

The overall compression consists of the following compression features:

  1. Paths are replaced by a two byte index from the list of paths found in vsspathlist.json.
  2. Fixed keywords, such as "action", "path", "get", etc., are replaced by a one byte value.
  3. Timestamps are replaced by a four byte value.
  4. The value part for the keys "value", "requestId", "subscriptionId" are replaced by a binary value, if possible.
  5. JSON reserved characters are removed in the case their excistence in a certain position is a MUST according to the JSON format rules.
  6. Payload data that does not fall under any of the features above is left uncompressed.

Below follows further details about the compression features.

  1. The list vsspathist.json is generated by the VISSv2 server at every startup. The input to the server comes from the "vss_vissv2.binary" file, which is in its turn generated by the VSS Tools, that take the VSS tree as represented by the vspec files from the VSS repo as input. This means that if the VISSv2 server starts up with an updated VSS tree, then the clients should be aware of this, and get a copy of the updated vsspathlist.json file from the server. This could be handled by a request to the server asking for this file, but the VISSv2 spec does not currenly support that.
    The pathlist in the file generated by the VISSv2 server is ordered, so searches in the list can be optimised.
    The list index is represented by two bytes, which means that a maximum of 65536 paths can be handled. The list only contains paths for leaf nodes, so this is the maximum number of leaf nodes that the VSS tree can have for this compression feature to work.

  2. The value replacing the keywords are the index from the kewordlist plus 128. The reason to add 128 is that an uncompressed UTF8 JSON string only contains byte values less than 128, see an ASCII table. The keywordlist original is found in the "common.go" source file. It is not expected to change. A client compression impl needs a copy of it.

  3. The uncompressed timestamps follow the ISO8601 standard and has the format YYYY-MM-DDTHH:MM:SS.ssssssZ
    This is compressed as follows:

  • The least significant year digit is represented by four bits.
  • The month is represented by four bits.
  • The day is represented by five bits.
  • The hour is represented by five bits.
  • The minute is represented by six bits.
  • The second is represented by six bits.

Subsecond usage is not supported currently.

The sum of bits above is 30 bits, which fits into 4 bytes as shown below.
Byte 1 | Byte 2 | Byte 3 | Byte 4
00yyyymm | mmdddddh | hhhhmmmm | mmssssss

The timestamp encoding has the limitation that when the year goes from 2029 to 2030, any messages that are received by a client after the new year has begun, that contain a timestamp from the previous year, will be incorrectly decoded as they will be assigned the year 2039 instead of 2029. A solution to this could be to use Unix time instead, but then the 2038 problem will be coming. Another would be for clients to check for this 10-year problem around the time when it becomes likely. Vehicles “should not” respond with too stale data, so a few months after 2030 begins should be max what is needed.

  1. The value types that are supported are integer, float32, and boolean. The first byte is and index from the keywordlist, which also contains this (+128). The type indicated also sets the number of bytes used for the representation, as shown below. The type "nuintx" means it is a negative uint value, as all integer values are represented as uints.
    Type | #bytes
    nuint8 | 1
    uint8 | 1
    nuint16 | 2
    uint16v | 2
    nuint24 | 3
    uint24 | 3
    nuint32 | 4
    uint32 | 4
    bool | 1
    float32 | 4

The absolute value of integers are compared to the max value of the different types, and the smallest possible size is selected. Values that are not of any of these types are kept uncompressed.

  1. The following JSON reserved character usages can be removed as at decompression the JSON rules will infer their reinstatement.
  • The leading and trailing curly brackets.
  • The colon between key and value.
  • The comma between JSON data entities.
  1. The fact that data that does not relate to any of the mentioned compression features is left uncompressed leads to that the usage of this compression does not apply any restriction to the syntax scope of VISSv2. It also enables the client to send uncompressed requests, where the response will be compressed.

The current compression solution does not support service discovery requests.

Protobuf reference compression design

This design provides two levels of compression:
PB_LEVEL1:
All payload data parameters are represented in string format.
PB_LEVEL2:
Paths are represented by an index into the path array found in the vsspathlist.json file. The index has int32 format.
Timestamps are represented as Unix time using int32 format.
This is currently implemented in get and subscribe responses/notifications.
All other payload data parameters are represented in string format.

Documentation

Overview

* * (C) 2023 Ford Motor Company * (C) 2021 Geotab * * All files and artifacts in the repository at https://github.com/covesa/vissr * are licensed under the provisions of the license provided by the LICENSE file in this repository. * *

* * (C) 2021 Geotab * * All files and artifacts in the repository at https://github.com/covesa/vissr * are licensed under the provisions of the license provided by the LICENSE file in this repository. * *

Index

Constants

View Source
const (
	NONE        Compression = 0
	PROPRIETARY             = 1
	PB_LEVEL1               = 2 // path has string format, e. g. "Vehicle.Acceleration.Longitudinal"
	PB_LEVEL2               = 3 // path is represented by integer index, retrieved from vsspathlist.json
)
View Source
const CODELISTINDEXPATH = 4 // must be set to the list index of the "path" element
View Source
const CODELISTINDEXREQID = 1 // must be set to the list index of the "requestId" element
View Source
const CODELISTINDEXSUBID = 5 // must be set to the list index of the "subscriptionId" element
View Source
const CODELISTINDEXTS = 3 // must be set to the list index of the "ts" element
View Source
const CODELISTINDEXVALUE = 2 // must be set to the list index of the "value" element
View Source
const CODELISTKEYS = 10 // must be set to the number of keys in the list
View Source
const CODELISTKEYVALUES = 15 // must be set to the number of keys plus values in the list (excl value types)
View Source
const IpEnvVarName = "GEN2MODULEIP"
View Source
const IpModel = 0 // IpModel = [0,1,2] = [localhost,extIP,envVarIP]

Variables

View Source
var (
	//    Trace   *log.Logger
	Info    *logrus.Logger
	Warning *logrus.Logger
	Error   *logrus.Logger
)
View Source
var ErrorInfoList [8]ErrorInformation = [8]ErrorInformation{
	{"400", "bad_request", "The request is malformed."},
	{"400", "invalid_data", "Data present in the request is invalid."},
	{"401", "expired_token", "Access token has expired."},
	{"401", "invalid_token", "Access token is invalid."},
	{"401", "missing_token", "Access token is missing."},
	{"403", "forbidden_request", "The server refuses to carry out the request."},
	{"404", "unavailable_data", "The requested data was not found."},
	{"503", "service_unavailable", "The server is temporarily unable to handle the request."}}
View Source
var HostIP string
View Source
var Logfile *os.File

const LOG_FILE = "servercore-log.txt"

View Source
var TrSecConfigPath string = "../transport_sec/" // relative path to the directory containing the transportSec.json file
View Source
var Upgrader = websocket.Upgrader{
	ReadBufferSize:  1024,
	WriteBufferSize: 1024,
}
View Source
var WsClientIndexList = []bool{
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
	true,
}

len of WsClientIndexList must match the number of select cases in wsMgr

Functions

func AddKeyValue

func AddKeyValue(message string, key string, value string) string

func AddRoutingForwardRequest

func AddRoutingForwardRequest(reqMessage string, mgrId int, clientId int, transportMgrChan chan string)

func AnalyzeValueType

func AnalyzeValueType(value string) int

func CertOptToInt

func CertOptToInt(serverCertOpt string) int

func CloseLogFile

func CloseLogFile()

func CompressMessage

func CompressMessage(message []byte) []byte

func CompressPath

func CompressPath(path string) *int32

func CompressTS

func CompressTS(ts string) int32

func DecompressMessage

func DecompressMessage(message []byte) []byte

func DecompressPath

func DecompressPath(index int32) string

func DecompressTs

func DecompressTs(tsCompressed int32) string

func ExportKeyPair

func ExportKeyPair(privKey crypto.PrivateKey, privFileName string, pubFileName string) error

Export KeyPair to files named as given (ECDSA and RSA supported, pointers to privKey must be given)

func ExtractFromToken

func ExtractFromToken(token string, claim string) string

func ExtractRootName

func ExtractRootName(path string) string

func ExtractSubscriptionId

func ExtractSubscriptionId(jsonSubResponse string) string

func FileExists

func FileExists(filename string) bool

func FinalizeMessage

func FinalizeMessage(responseMap map[string]interface{}) string

func GenEcdsaKey

func GenEcdsaKey(curve elliptic.Curve, privKey **ecdsa.PrivateKey) error

Generates ECDSA private Key using given curve

func GenRsaKey

func GenRsaKey(size int, privKey **rsa.PrivateKey) error

********* KEY GENERATION *********** Generates RSA private key of given size

func GenerateHmac

func GenerateHmac(input string, key string) string

func GetMaxValidation

func GetMaxValidation(newValidation int, currentMaxValidation int) int

func GetModelIP

func GetModelIP(ipModel int) string

func GetRequestJsonToPb

func GetRequestJsonToPb(vssGetReq string, compression Compression) *pb.GetRequestMessage

func GetRequestPbToJson

func GetRequestPbToJson(pbGetReq *pb.GetRequestMessage, compression Compression) string

func GetResponseJsonToPb

func GetResponseJsonToPb(vssGetResp string, compression Compression) *pb.GetResponseMessage

func GetResponsePbToJson

func GetResponsePbToJson(pbGetResp *pb.GetResponseMessage, compression Compression) string

func GetRfcTime

func GetRfcTime() string

func GetServerIP

func GetServerIP() string

func GetTLSConfig

func GetTLSConfig(host string, caCertFile string, certOpt tls.ClientAuthType, serverCert *tls.Certificate) *tls.Config

Obtains a tls.Config struct, giving support to https.listenandservetls

func GetTimeInMilliSecs

func GetTimeInMilliSecs() string

func GetUdsConn

func GetUdsConn(path string, connectionName string) net.Conn

func GetUdsPath

func GetUdsPath(path string, connectionName string) string

func ImportEcdsaKey

func ImportEcdsaKey(filename string, privKey **ecdsa.PrivateKey) error

Gets ecdsa private key from pem file

func ImportRsaKey

func ImportRsaKey(filename string, privKey **rsa.PrivateKey) error

********* PEM KEY IMPORT / EXPORT *********** Gets rsa private key from pem file

func ImportRsaPubKey

func ImportRsaPubKey(filename string, pubKey **rsa.PublicKey) error

Gets rsa public key from pem file

func InitCompression

func InitCompression(vsspathlistFname string) bool

must be called before calling the methods CompressMessage, DecompressMessage, CompressTS, DecompressTs, CompressPath, DecompressPath

func InitLog

func InitLog(filename string, logdir string, logFile bool, logLevel string)

func JsonRecursiveMarshall

func JsonRecursiveMarshall(key string, value string, jplain *string)

Gets Json string (or nothing) and adds received key and value, if it doesnt receive a value or key, it does nothing

func JsonToProtobuf

func JsonToProtobuf(jsonMessage string, compression Compression) []byte

func MapRequest

func MapRequest(request string, rMap *map[string]interface{}) int

func NextQuoteMark

func NextQuoteMark(message []byte, offset int) int

func PathToUrl

func PathToUrl(path string) string

func PemDecodeECDSA

func PemDecodeECDSA(pemKey string, privKey **ecdsa.PrivateKey) error

Gets ECDSA key in pem format and decodes it into ecdsa.PrivateKey

func PemDecodeRSA

func PemDecodeRSA(pemKey string, privKey **rsa.PrivateKey) error

********* KEY ENCODING / DECODING *********** Gets rsa key in pem format and decodes it into rsa.privatekey

func PemDecodeRSAPub

func PemDecodeRSAPub(pemKey string, pubKey **rsa.PublicKey) error

Gets rsa pub key in pem format and decodes it into rsa.publickey

func PemEncodeECDSA

func PemEncodeECDSA(privKey *ecdsa.PrivateKey) (strPrivKey string, strPubKey string, err error)

Returns ECDSA Keys as string in PEM format

func PemEncodeRSA

func PemEncodeRSA(privKey *rsa.PrivateKey) (strPrivKey string, strPubKey string, err error)

Returns RSA Keys as string in PEM format

func ProtobufToJson

func ProtobufToJson(serialisedMessage []byte, compression Compression) string

func ReadTransportSecConfig

func ReadTransportSecConfig()

Initializes TransportSec Variables

func RemoveInternalData

func RemoveInternalData(response string) (string, int)

func ReturnWsClientIndex

func ReturnWsClientIndex(index int)

func SetErrorResponse

func SetErrorResponse(reqMap map[string]interface{}, errRespMap map[string]interface{}, errorListIndex int, altErrorMessage string)

func SetErrorResponse(reqMap map[string]interface{}, errRespMap map[string]interface{}, number string, reason string, message string) {

func SetRequestJsonToPb

func SetRequestJsonToPb(vssSetReq string, compression Compression) *pb.SetRequestMessage

func SetRequestPbToJson

func SetRequestPbToJson(pbSetReq *pb.SetRequestMessage, compression Compression) string

func SetResponseJsonToPb

func SetResponseJsonToPb(vssSetResp string, compression Compression) *pb.SetResponseMessage

func SetResponsePbToJson

func SetResponsePbToJson(pbSetResp *pb.SetResponseMessage, compression Compression) string

func SubscribeRequestJsonToPb

func SubscribeRequestJsonToPb(vssSubscribeReq string, compression Compression) *pb.SubscribeRequestMessage

func SubscribeRequestPbToJson

func SubscribeRequestPbToJson(pbSubscribeReq *pb.SubscribeRequestMessage, compression Compression) string

func SubscribeStreamJsonToPb

func SubscribeStreamJsonToPb(vssSubscribeStream string, compression Compression) *pb.SubscribeStreamMessage

func SubscribeStreamPbToJson

func SubscribeStreamPbToJson(pbSubscribeResp *pb.SubscribeStreamMessage, compression Compression) string

func TrimLogFile

func TrimLogFile(logFile *os.File)

* * The log file is trimmed to 20% of its size when exceeding 10MB. *

func UnpackFilter

func UnpackFilter(filter interface{}, fList *[]FilterObject)

func UnsubscribeRequestJsonToPb

func UnsubscribeRequestJsonToPb(vssUnsubscribeReq string, compression Compression) *pb.UnsubscribeRequestMessage

func UnsubscribeRequestPbToJson

func UnsubscribeRequestPbToJson(pbUnsubscribeReq *pb.UnsubscribeRequestMessage, compression Compression) string

func UnsubscribeResponseJsonToPb

func UnsubscribeResponseJsonToPb(vssUnsubscribeResp string, compression Compression) *pb.UnsubscribeResponseMessage

func UnsubscribeResponsePbToJson

func UnsubscribeResponsePbToJson(pbUnsubscribeResp *pb.UnsubscribeResponseMessage, compression Compression) string

func UrlToPath

func UrlToPath(url string) string

func VerifyTokenSignature

func VerifyTokenSignature(token string, key string) error

Types

type ClientHandler

type ClientHandler interface {
	// contains filtered or unexported methods
}

*********** Client response handlers *******************************************************************************

type ClientServer

type ClientServer interface {
	InitClientServer(muxServer *http.ServeMux)
}

type CodeList

type CodeList struct {
	Code []string `json:"codes"`
}

type Compression

type Compression int

type ErrorInformation

type ErrorInformation struct {
	Number  string
	Reason  string
	Message string
}

type ExtendedJwt

type ExtendedJwt struct {
	Token         JsonWebToken
	HeaderClaims  map[string]string
	PayloadClaims map[string]string
}

********* EXTENDED JSON WEB TOKEN *********** ********* Extends the JsonWebToken type, including a map with the claims in header ********* and a map with the claims in payload

func (*ExtendedJwt) DecodeFromFull

func (ext *ExtendedJwt) DecodeFromFull(input string) error

type FilterObject

type FilterObject struct {
	Type      string
	Parameter string
}

type HttpChannel

type HttpChannel struct {
}

type HttpServer

type HttpServer struct {
}

func (HttpServer) InitClientServer

func (server HttpServer) InitClientServer(muxServer *http.ServeMux, httpClientChan []chan string)

Launches the HTTP Manager

type JsonWebKey

type JsonWebKey struct {
	Thumb  string `json:"-"`
	Type   string `json:"kty"`
	Use    string `json:"use,omitempty"`
	PubMod string `json:"n,omitempty"`   // RSA
	PubExp string `json:"e,omitempty"`   // RSA
	Curve  string `json:"crv,omitempty"` //ECDSA
	Xcoord string `json:"x,omitempty"`   //ECDSA
	Ycoord string `json:"y,omitempty"`   //ECDSA
}

********* JSON WEB KEY ENCODING *********** ********* Contained in PoP, follows RFC7517 standard. Support for RSA and ECDSA keys

func (*JsonWebKey) GenThumbprint

func (jkey *JsonWebKey) GenThumbprint() string

Generates thumbprint of the JWK

func (*JsonWebKey) Initialize

func (jkey *JsonWebKey) Initialize(pubKey crypto.PublicKey, use string) error

Initializes json web key from public key

func (*JsonWebKey) Marshal

func (jkey *JsonWebKey) Marshal() string

From JsonWebKey struct, returns marshalled text

func (*JsonWebKey) Unmarshall

func (jkey *JsonWebKey) Unmarshall(rcv string) error

Gets the received JWK and unmarshalls it, returns error if fails to unmarshall

type JsonWebToken

type JsonWebToken struct {
	Header           string
	Payload          string
	EncodedHeader    string
	EncodedPayload   string
	EncodedSignature string
}

********* JSON WEB TOKEN *********** ********* Basic JWT including Header, Payload and encoded parts. ********* Methods for decoding and signature check avaliable

func (*JsonWebToken) AddClaim

func (token *JsonWebToken) AddClaim(key string, value string)

Adds a claim to the payload

func (*JsonWebToken) AddHeader

func (token *JsonWebToken) AddHeader(key string, value string)

Adds a claim to the header

func (*JsonWebToken) AssymSign

func (token *JsonWebToken) AssymSign(privKey crypto.PrivateKey) error

Signs the token using an assymetric key

func (JsonWebToken) CheckAssymSignature

func (token JsonWebToken) CheckAssymSignature(key crypto.PublicKey) (err error)

Checks the assymetric signature of the token

func (JsonWebToken) CheckSignature

func (token JsonWebToken) CheckSignature(key interface{}) error

Checks if the token is signed correctly. In case of symm sign, key as string must be passed. In case of assym, a crypto.PublicKey must be passed

func (*JsonWebToken) DecodeFromFull

func (token *JsonWebToken) DecodeFromFull(input string) error

From a signed jwt received, gets header and payload

func (*JsonWebToken) Encode

func (token *JsonWebToken) Encode()

Encodes the Token

func (JsonWebToken) GetFullToken

func (token JsonWebToken) GetFullToken() string

Returns the full token

func (JsonWebToken) GetHeader

func (token JsonWebToken) GetHeader() string

Returns the header of the token

func (JsonWebToken) GetPayload

func (token JsonWebToken) GetPayload() string

Returns the payload of the token

func (*JsonWebToken) SetHeader

func (token *JsonWebToken) SetHeader(algorithm string)

Sets the algorithm used

func (*JsonWebToken) SymmSign

func (token *JsonWebToken) SymmSign(key string)

Signs the token using a symmetric key

type PathList

type PathList struct {
	Path []string `json:"LeafPaths"`
}

type PopToken

type PopToken struct {
	HeaderClaims  map[string]string // TYP, ALG, JWK
	PayloadClaims map[string]string // IAT, JTI
	Jwk           JsonWebKey
	Jwt           JsonWebToken
}

********* POP TOKEN *********** ********* POP Token is used by the client to attest its possession of a private key ********* More info in the README of the repo

func (*PopToken) CheckAud

func (popToken *PopToken) CheckAud(aud string) (bool, string)

func (PopToken) CheckExp

func (popToken PopToken) CheckExp() (bool, string)

Check exp time

func (PopToken) CheckIat

func (popToken PopToken) CheckIat(gap int, lifetime int) (bool, string)

Check iats. Gap is the possible error between clocks. lifetime is the maximum time after is creation that the token can be used

func (*PopToken) CheckSignature

func (popToken *PopToken) CheckSignature() error

Checks signature, checks that alg used to sign is the same as in key (to avoid exploits)

func (PopToken) CheckThumb

func (popToken PopToken) CheckThumb(thumprint string) (bool, string)

Validates keys: same alg, same thumprint...

func (*PopToken) GenerateToken

func (popToken *PopToken) GenerateToken(privKey crypto.PrivateKey) (token string, err error)

Generates popToken using a PrivateKey, can be used even if popToken is not initialized (claims are auto-fulfilled)

func (PopToken) GetPubEcdsa

func (popToken PopToken) GetPubEcdsa() (*ecdsa.PublicKey, error)

Obtains ECDSA public ket in the PoP token. Returns nil + error if fails

func (PopToken) GetPubRsa

func (popToken PopToken) GetPubRsa() (*rsa.PublicKey, error)

Obtains Rsa public key included in the PoP token. Returns nil + error if fails

func (*PopToken) Initialize

func (popToken *PopToken) Initialize(headerMap, payloadMap map[string]string, pubKey crypto.PublicKey) error

Initializes popToken from claims and public key. Make sure the private key used to sign is the same used to initialize

func (*PopToken) Unmarshal

func (popToken *PopToken) Unmarshal(token string) error

Gets the received PoP token as string, and unmarshalls it. JWK, JWT and claims fields are all filled

func (*PopToken) Validate

func (popToken *PopToken) Validate(thumbprint, aud string, gap, lifetime int) (valid bool, info string)

Returns a bool that tells if the pop token is valid.

type SecConfig

type SecConfig struct {
	TransportSec  string `json:"transportSec"`  // "yes" or "no"
	HttpSecPort   string `json:"httpSecPort"`   // HTTPS port number
	WsSecPort     string `json:"wsSecPort"`     // WSS port number
	MqttSecPort   string `json:"mqttSecPort"`   // MQTTS port number
	GrpcSecPort   string `json:"grpcSecPort"`   // MQTTS port number
	AgtsSecPort   string `json:"agtsSecPort"`   // AGTS port number
	AtsSecPort    string `json:"atsSecPort"`    // ATS port number
	CaSecPath     string `json:"caSecPath"`     // relative path from the directory containing the transportSec.json file
	ServerSecPath string `json:"serverSecPath"` // relative path from the directory containing the transportSec.json file
	ServerCertOpt string `json:"serverCertOpt"` // one of  "NoClientCert"/"ClientCertNoVerification"/"ClientCertVerification"
	ClientSecPath string `json:"clientSecPath"` // relative path from the directory containing the transportSec.json file
}
var SecureConfiguration SecConfig // name change to caps allowing to export outside utils

type UdsReg

type UdsReg struct {
	RootName     string `json:"root"`
	ServerFeeder string `json:"serverFeeder"`
	Redis        string `json:"redis"`
	Memcache     string `json:"memcache"`
	History      string `json:"history"`
}

func ReadUdsRegistrations

func ReadUdsRegistrations(sockFile string) []UdsReg

type WsChannel

type WsChannel struct {
	// contains filtered or unexported fields
}

type WsServer

type WsServer struct {
	ClientBackendChannel []chan string
}

func (WsServer) InitClientServer

func (server WsServer) InitClientServer(muxServer *http.ServeMux, wsClientChan []chan string, mgrIndex int, clientIndex *int)

Launches the WebSocket Manager

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL