Documentation
¶
Overview ¶
Package xdstest tests an xDS control plane against a real Envoy, run in Docker with testcontainers-go.
Record the xDS conversation with a Recorder, check it with Sotw or Delta, wait for it to go quiet with Settled, and check Envoy itself with Envoy.Healthy and Envoy.WarningsSince.
Every check returns an error rather than taking a *testing.T, so use them with WaitFor or with your own assertions.
Index ¶
- Constants
- func IsWildcard(typeURL string) bool
- func NameOf(resource proto.Message) string
- func Settled(c *Conversation, quietPeriod time.Duration) func() error
- func SettledAfter(c *Conversation, fromMessage int, settled func() error) func() error
- func SettledIdle(c *Conversation, idle func() bool) func() error
- func WaitFor(timeout time.Duration, check func() error) error
- type Checkpoint
- type Checks
- type Conversation
- type Envoy
- func (e *Envoy) Accept(l testcontainers.Log)
- func (e *Envoy) Clusters(ctx context.Context) (map[string]uint64, error)
- func (e *Envoy) ConfigDump(ctx context.Context, query string) (*adminv3.ConfigDump, error)
- func (e *Envoy) Container() testcontainers.Container
- func (e *Envoy) HTTPStatus(ctx context.Context, listenerPort int, virtualHost string) (int, error)
- func (e *Envoy) Healthy(ctx context.Context, baseline map[string]uint64) error
- func (e *Envoy) LogSince(fromLine int) []string
- func (e *Envoy) LogSize() int
- func (e *Envoy) RoutedVia(ctx context.Context, listenerPort int, virtualHost, cluster string) error
- func (e *Envoy) Stats(ctx context.Context) (map[string]uint64, error)
- func (e *Envoy) Terminate(ctx context.Context) error
- func (e *Envoy) URL(ctx context.Context, port int) (string, error)
- func (e *Envoy) WarningsSince(fromLine int) error
- type EnvoyConfig
- type Recorder
Constants ¶
const ( LDS = "type.googleapis.com/envoy.config.listener.v3.Listener" RDS = "type.googleapis.com/envoy.config.route.v3.RouteConfiguration" SRDS = "type.googleapis.com/envoy.config.route.v3.ScopedRouteConfiguration" VHDS = "type.googleapis.com/envoy.config.route.v3.VirtualHost" CDS = "type.googleapis.com/envoy.config.cluster.v3.Cluster" EDS = "type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment" SDS = "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret" RTDS = "type.googleapis.com/envoy.service.runtime.v3.Runtime" )
xDS type URLs.
const ADSPortPlaceholder = "${ADS_PORT}"
ADSPortPlaceholder is replaced in Envoy's bootstrap with EnvoyConfig.ADSPort, e.g. "port_value: ${ADS_PORT}".
const AdminPort = 9901
AdminPort is where Envoy's config must put its admin interface, on 0.0.0.0.
const DefaultQuietPeriod = 500 * time.Millisecond
DefaultQuietPeriod is how long a conversation must be silent to count as settled, if the control plane can't say.
const HostInternal = testcontainers.HostInternal
HostInternal is the name Envoy's config uses to reach your control plane on the host, e.g. in the ADS cluster.
Variables ¶
This section is empty.
Functions ¶
func IsWildcard ¶
IsWildcard reports whether Envoy subscribes to a type as a wildcard (all resources) by default: LDS, CDS, SRDS and VHDS.
func NameOf ¶
NameOf is the name Envoy knows a resource by: its name field, or cluster_name for a ClusterLoadAssignment. It's "" if the resource has neither.
func Settled ¶
func Settled(c *Conversation, quietPeriod time.Duration) func() error
Settled returns a condition, for WaitFor, that every response was answered and nothing has been sent either way for quietPeriod.
func SettledAfter ¶
func SettledAfter(c *Conversation, fromMessage int, settled func() error) func() error
SettledAfter returns a condition that something was sent after fromMessage, then settled. Use it to wait for a change: when you start waiting, the control plane may not have begun sending it yet, and look idle.
func SettledIdle ¶
func SettledIdle(c *Conversation, idle func() bool) func() error
SettledIdle returns a condition that the control plane says it is idle and every response was answered. Use it if your control plane can tell whether it has anything in flight or waiting to be sent on the stream: it knows whether more is coming, and there's no quiet period to wait out.
Types ¶
type Checkpoint ¶
type Checkpoint struct {
// Message is the conversation's size when taken, e.g. for SettledAfter.
Message int
// LogLine is Envoy's log length when taken.
LogLine int
// Stats are Envoy's stats when taken.
Stats map[string]uint64
// contains filtered or unexported fields
}
Checkpoint is where a conversation and Envoy were at one moment: the conversation's size, Envoy's log length and Envoy's stats. Take one once Envoy has synced, make changes, then check that everything since was sane:
cp, err := xdstest.TakeCheckpoint(ctx, envoy, c)
controlPlane.AddService("foo")
err = xdstest.WaitFor(30*time.Second, xdstest.SettledAfter(c, cp.Message, xdstest.Settled(c, xdstest.DefaultQuietPeriod)))
err = cp.Sane(ctx)
func TakeCheckpoint ¶
func TakeCheckpoint(ctx context.Context, envoy *Envoy, c *Conversation) (*Checkpoint, error)
TakeCheckpoint marks where things are now.
func (*Checkpoint) EnvoyHealthy ¶
func (cp *Checkpoint) EnvoyHealthy(ctx context.Context) error
EnvoyHealthy checks that Envoy is connected, has nothing warming, and since the checkpoint rejected nothing, timed out no initial fetches and logged no warnings or errors.
func (*Checkpoint) Sane ¶
func (cp *Checkpoint) Sane(ctx context.Context) error
Sane checks everything that should hold once things have settled after changes since the checkpoint. Envoy ACKed every response and was sent nothing it didn't ask for. Since the checkpoint: no empty responses, make before break, and EnvoyHealthy.
func (*Checkpoint) SaneAfterReconnect ¶
func (cp *Checkpoint) SaneAfterReconnect(ctx context.Context, before *Conversation) error
SaneAfterReconnect is Sane for a conversation Envoy started by reconnecting. It allows Envoy before 1.19 resending resource names for wildcard types. Given before, the previous conversation with this Envoy, it also checks make before break from the start of this one, starting from what Envoy last ACKed there: Envoy keeps its config across a disconnect. With before nil, make before break isn't checked.
type Checks ¶
type Checks interface {
// AllWereAcked checks that Envoy ACKed every response, and NACKed none.
AllWereAcked() error
// NothingSentUnasked checks that every response is for a type, and resource names, that Envoy subscribed to.
// Before 1.19, Envoy resent resource names for wildcard types when it reconnected; pass true to allow that.
NothingSentUnasked(allowPre119WildcardReconnect bool) error
// UnsubscribedFrom checks that Envoy dropped its subscription to a resource, e.g. EDS for a removed cluster.
UnsubscribedFrom(typeURL, name string) error
// SentNoEmptyResponses checks that no response from fromMessage on was empty. Earlier ones may be: an empty first
// response for a wildcard type tells Envoy there is nothing yet.
SentNoEmptyResponses(fromMessage int) error
// WasMakeBeforeBreak checks that, from fromMessage on, no route was sent before Envoy ACKed the clusters and
// endpoints it uses, and no cluster was removed while an ACKed route still used it. Start after the initial sync:
// answers to Envoy's first subscriptions aren't ordered, Envoy's own startup sequencing covers those.
WasMakeBeforeBreak(fromMessage int) error
// WasMakeBeforeBreakAfter is WasMakeBeforeBreak for a conversation Envoy started by reconnecting. Envoy keeps its
// config across a disconnect, so this starts from what it last ACKed in before, the previous conversation, and
// checks this one from fromMessage on, usually 0.
WasMakeBeforeBreakAfter(before *Conversation, fromMessage int) error
// AskedForEveryRouteConfigInUse checks that Envoy subscribed to the route config of every listener it ACKed. Until
// it has, it hasn't synced, though it may look done: a rate limited Envoy can ACK a listener well before it asks
// for the listener's routes.
AskedForEveryRouteConfigInUse() error
// AnsweredEveryType checks that every type Envoy asked for got at least one response. Without one Envoy waits out
// its initial fetch timeout, or forever if that is disabled.
AnsweredEveryType() error
// HadConversationsWith checks that Envoy asked for each of these types.
HadConversationsWith(typeURLs ...string) error
}
Checks are checks on one recorded conversation. Each looks at the conversation as it is when called, and returns nil or what is wrong. Get them with Sotw or Delta, whichever protocol Envoy uses.
func ChecksFor ¶
func ChecksFor(c *Conversation) Checks
ChecksFor returns Sotw or Delta checks, whichever protocol the conversation uses.
func Delta ¶
func Delta(c *Conversation) Checks
Delta returns checks for an incremental (delta) conversation.
func Sotw ¶
func Sotw(c *Conversation) Checks
Sotw returns checks for a state-of-the-world conversation.
type Conversation ¶
type Conversation struct {
// contains filtered or unexported fields
}
Conversation is one recorded stream: requests from Envoy and responses to it, in order.
func NewConversation ¶
func NewConversation() *Conversation
NewConversation returns an empty conversation, e.g. to test your own checks against.
func (*Conversation) Finished ¶
func (c *Conversation) Finished() bool
Finished reports whether the stream has ended.
func (*Conversation) Messages ¶
func (c *Conversation) Messages() []proto.Message
Messages returns a copy of the messages so far.
func (*Conversation) QuietFor ¶
func (c *Conversation) QuietFor(d time.Duration) bool
QuietFor reports whether nothing has been sent either way for at least d.
func (*Conversation) Size ¶
func (c *Conversation) Size() int
Size is the number of messages so far. Take it as a mark before a change, so later checks only look at what came after.
type Envoy ¶
type Envoy struct {
// contains filtered or unexported fields
}
Envoy is a running Envoy container. It keeps Envoy's log.
func StartEnvoy ¶
func StartEnvoy(ctx context.Context, cfg EnvoyConfig) (*Envoy, error)
StartEnvoy starts Envoy and waits for its admin interface. Old Envoy builds occasionally crash at startup, so it tries up to three times.
func (*Envoy) Accept ¶
func (e *Envoy) Accept(l testcontainers.Log)
Accept implements testcontainers.LogConsumer.
func (*Envoy) Clusters ¶
Clusters returns what Envoy has per host, from its /clusters admin page: the numeric values by everything before the last "::", e.g. "foo::127.0.0.2:9901::rq_total".
func (*Envoy) ConfigDump ¶
ConfigDump returns Envoy's config dump. query is added to the admin URL, e.g. "?resource=dynamic_active_clusters", or "" for everything. Decoding needs every type in the dump to be linked in: blank import the go-control-plane packages for any extensions your config uses.
func (*Envoy) Container ¶
func (e *Envoy) Container() testcontainers.Container
Container is the underlying container, for anything this doesn't cover.
func (*Envoy) HTTPStatus ¶
HTTPStatus sends GET / with the given Host header through one of Envoy's listeners and returns the status code.
func (*Envoy) Healthy ¶
Healthy checks that Envoy is connected to its control plane, has nothing warming, and rejected no updates and timed out no initial fetches (Envoy 1.24 on; older ones have no stat for it) since baseline, an earlier Envoy.Stats or nil for ever. Stats can trail an ACK a little, so it retries for up to 10 seconds.
func (*Envoy) LogSize ¶
LogSize is the number of log lines so far. Take it as a mark once Envoy has synced, to skip startup noise.
func (*Envoy) RoutedVia ¶
RoutedVia checks that GET / with this Host header through one of Envoy's listeners answers 200 and went to this cluster, going by its upstream_rq_total. A 200 alone doesn't say which cluster answered. It retries for up to 5 seconds, while Envoy's workers catch up with an ACKed change.
func (*Envoy) WarningsSince ¶
WarningsSince checks that Envoy logged no warnings or errors from fromLine on, e.g. a Envoy.LogSize taken once Envoy has synced.
type EnvoyConfig ¶
type EnvoyConfig struct {
// Image is the Envoy image, e.g. "envoyproxy/envoy:v1.40-latest".
Image string
// Bootstrap is Envoy's YAML config. It must put the admin interface on 0.0.0.0:9901 and point ADS at
// HostInternal on ADSPortPlaceholder, which is replaced with ADSPort.
Bootstrap []byte
// ADSPort is the port your control plane listens on, on the host. Start it before Envoy, or Envoy backs off and
// can time out its initial fetches. Listen on ":0" and pass the port it got, so tests can run in parallel.
ADSPort int
// ListenerPorts are listener ports to reach from the test, e.g. with HTTPStatus. Bind those listeners to 0.0.0.0.
ListenerPorts []int
// Args are extra Envoy arguments, e.g. "-l", "debug".
Args []string
}
EnvoyConfig says how to run Envoy.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder is a gRPC stream interceptor that records every stream to and from your server:
rec := &xdstest.Recorder{}
server := grpc.NewServer(grpc.StreamInterceptor(rec.Intercept))
Use one Recorder for the whole test, so a reconnect shows up as a second conversation.
func (*Recorder) Conversations ¶
func (r *Recorder) Conversations() []*Conversation
Conversations returns the streams recorded so far, oldest first.
func (*Recorder) Intercept ¶
func (r *Recorder) Intercept(srv any, ss grpc.ServerStream, _ *grpc.StreamServerInfo, handler grpc.StreamHandler) error
Intercept is a grpc.StreamServerInterceptor.