Version: v0.0.0-...-4638b96 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jan 7, 2020 License: Apache-2.0 Imports: 44 Imported by: 0


Debug interface

The debug handlers are configured on the monitoring port (default 15014) as well as on the http port (8080).


# What is sent to envoy
# Listeners and routes
curl $PILOT/debug/adsz

# Endpoints
curl $PILOT/debug/edsz

# Clusters
curl $PILOT/debug/cdsz

Each handler takes an extra parameter, "debug=0|1" which flips the verbosity of the messages for that component (similar with envoy).

An extra parameter "push=1" triggers a config push to all connected endpoints.

Handlers should list, in json format:

  • one entry for each connected envoy
  • the timestamp of the connection

In addition, Pilot debug interface can show pilot's internal view of the config:

# General metrics
curl $PILOT/metrics

# All services/external services from all registries
curl $PILOT/debug/registryz

# All endpoints
curl $PILOT/debug/endpointz[?brief=1]

# All configs.
curl $PILOT/debug/configz

Example for EDS:


   // Cluster
   "echosrv.istio-system.svc.cluster.local|grpc-ping": {
    "EdsClients": {
      // One for each connected envoy.
      "sidecar~": {
        // Should match the info in the node (this is the real remote address)
        "PeerAddr": "",
        "Clusters": [
          // Should match the cluster, this is what is monitored
        // Time the sidecar connected to pilot
        "Connect": "2018-03-22T15:01:07.527304202Z"
      "sidecar~": {
        "PeerAddr": "",
        "Clusters": [
        "Connect": "2018-03-22T15:01:00.465066249Z"
    // The info pushed to each connected sidecar watching the cluster.
    "LoadAssignment": {
      "cluster_name": "echosrv.istio-system.svc.cluster.local|grpc-ping",
      "endpoints": [
          "locality": {},
          "lb_endpoints": [
              // Should match the endpoint and port from 'kubectl get ep'
              "endpoint": {
                "address": {
                  "Address": {
                    "SocketAddress": {
                      "address": "",
                      "PortSpecifier": {
                        "PortValue": 8079
                      "ipv4_compat": true

Log messages

Verbose messages for v2 is controlled by env variables PILOT_DEBUG_{EDS,CDS,LDS}. Setting it to "0" disables debug, setting it to "1" enables - debug is currently enabled by default, since it is not very verbose.

Messages are prefixed with EDS/LDS/CDS.

What we log and how to use it:

  • sidecar connecting to pilot: "EDS/CSD/LDS: REQ ...". This includes the node, IP and the discovery request proto. Should show up when the sidecar starts up.
  • sidecar disconnecting from pilot: xDS: close. This happens when a pod is stopped.
  • push events - whenever we push a config to the sidecar.
  • "XDS: Registry event..." - indicates a registry event, should be followed by PUSH messages for each endpoint.
  • "EDS: no instances": pay close attention to this event, it indicates that Envoy asked for a cluster but pilot doesn't have any valid instance. At some point after, when the instance eventually shows up you should see an EDS PUSH message.

In addition, the registry has slightly more verbose messages about the events, so it is possible to map an event in the registry to config pushes.

Example requests and responses


    build_version:"0/1.6.0-dev//RELEASE" >




View Source
const (

	// ClusterType is used for cluster discovery. Typically first request received
	ClusterType = typePrefix + "Cluster"
	// EndpointType is used for EDS and ADS endpoint discovery. Typically second request.
	EndpointType = typePrefix + "ClusterLoadAssignment"
	// ListenerType is sent after clusters and endpoints.
	ListenerType = typePrefix + "Listener"
	// RouteType is sent after listeners.
	RouteType = typePrefix + "RouteConfiguration"


View Source
var (

	// SendTimeout is the max time to wait for a ADS send to complete. This helps detect
	// clients in a bad state (not reading). In future it may include checking for ACK
	SendTimeout = 5 * time.Second

	// PushTimeout is the time to wait for a push on a client. Pilot iterates over
	// clients and pushes them serially for now, to avoid large CPU/memory spikes.
	// We measure and reports cases where pushing a client takes longer.
	PushTimeout = 5 * time.Second
View Source
var (

	// DebounceAfter is the delay added to events to wait
	// after a registry/config event for debouncing.
	// This will delay the push by at least this interval, plus
	// the time getting subsequent events. If no change is
	// detected the push will happen, otherwise we'll keep
	// delaying until things settle.
	DebounceAfter time.Duration

	// DebounceMax is the maximum time to wait for events
	// while debouncing. Defaults to 10 seconds. If events keep
	// showing up with no break for this time, we'll trigger a push.
	DebounceMax time.Duration


func AdsPushAll

func AdsPushAll(s *DiscoveryServer)

AdsPushAll will send updates to all nodes, for a full config or incremental EDS.

func EndpointsByNetworkFilter

func EndpointsByNetworkFilter(endpoints []endpoint.LocalityLbEndpoints, conn *XdsConnection, env *model.Environment) []endpoint.LocalityLbEndpoints

EndpointsByNetworkFilter is a network filter function to support Split Horizon EDS - filter the endpoints based on the network of the connected sidecar. The filter will filter out all endpoints which are not present within the sidecar network and add a gateway endpoint to remote networks that have endpoints (if gateway exists). Information for the mesh networks is provided as a MeshNetwork config map.

func LoadBalancingWeightNormalize

func LoadBalancingWeightNormalize(endpoints []endpoint.LocalityLbEndpoints) []endpoint.LocalityLbEndpoints

LoadBalancingWeightNormalize set LoadBalancingWeight with a valid value.

func Syncz

func Syncz(w http.ResponseWriter, req *http.Request)

Syncz dumps the synchronization status of all Envoys connected to this Pilot instance


type AuthenticationDebug

type AuthenticationDebug struct {
	Host                     string `json:"host"`
	Port                     int    `json:"port"`
	AuthenticationPolicyName string `json:"authentication_policy_name"`
	DestinationRuleName      string `json:"destination_rule_name"`
	ServerProtocol           string `json:"server_protocol"`
	ClientProtocol           string `json:"client_protocol"`
	TLSConflictStatus        string `json:"TLS_conflict_status"`

AuthenticationDebug holds debug information for service authentication policy.

type DiscoveryServer

type DiscoveryServer struct {
	// Env is the model environment.
	Env *model.Environment

	// MemRegistry is used for debug and load testing, allow adding services. Visible for testing.
	MemRegistry *MemServiceDiscovery

	// ConfigGenerator is responsible for generating data plane configuration using Istio networking
	// APIs and service registry info
	ConfigGenerator core.ConfigGenerator

	// ConfigController provides readiness info (if initial sync is complete)
	ConfigController model.ConfigStoreCache

	// KubeController provides readiness info (if initial sync is complete)
	KubeController *kube.Controller

	// DebugConfigs controls saving snapshots of configs for /debug/adsz.
	// Defaults to false, can be enabled with PILOT_DEBUG_ADSZ_CONFIG=1
	DebugConfigs bool

	// EndpointShards for a service. This is a global (per-server) list, built from
	// incremental updates.
	EndpointShardsByService map[string]*EndpointShards

	// WorkloadsById keeps track of information about a workload, based on direct notifications
	// from registry. This acts as a cache and allows detecting changes.
	WorkloadsByID map[string]*Workload
	// contains filtered or unexported fields

DiscoveryServer is Pilot's gRPC implementation for Envoy's v2 xds APIs

func NewDiscoveryServer

func NewDiscoveryServer(
	env *model.Environment,
	generator core.ConfigGenerator,
	ctl model.Controller,
	kuebController *kube.Controller,
	configCache model.ConfigStoreCache) *DiscoveryServer

NewDiscoveryServer creates DiscoveryServer that sources data from Pilot's internal mesh data structures

func (*DiscoveryServer) AdsPushAll

func (s *DiscoveryServer) AdsPushAll(version string, push *model.PushContext,
	full bool, edsUpdates map[string]struct{})

AdsPushAll implements old style invalidation, generated when any rule or endpoint changes. Primary code path is from v1 discoveryService.clearCache(), which is added as a handler to the model ConfigStorageCache and Controller.

func (*DiscoveryServer) ClearCache

func (s *DiscoveryServer) ClearCache()

ClearCache is wrapper for clearCache method, used when new controller gets instantiated dynamically

func (*DiscoveryServer) ConfigDump

func (s *DiscoveryServer) ConfigDump(w http.ResponseWriter, req *http.Request)

ConfigDump returns information in the form of the Envoy admin API config dump for the specified proxy The dump will only contain dynamic listeners/clusters/routes and can be used to compare what an Envoy instance should look like according to Pilot vs what it currently does look like.

func (*DiscoveryServer) ConfigUpdate

func (s *DiscoveryServer) ConfigUpdate(full bool)

ConfigUpdate implements ConfigUpdater interface, used to request pushes. It replaces the 'clear cache' from v1.

func (*DiscoveryServer) DeltaAggregatedResources

DeltaAggregatedResources is not implemented.

func (*DiscoveryServer) EDSUpdate

func (s *DiscoveryServer) EDSUpdate(shard, serviceName string,
	istioEndpoints []*model.IstioEndpoint) error

EDSUpdate computes destination address membership across all clusters and networks. This is the main method implementing EDS. It replaces InstancesByPort in model - instead of iterating over all endpoints it uses the hostname-keyed map. And it avoids the conversion from Endpoint to ServiceEntry to envoy on each step: instead the conversion happens once, when an endpoint is first discovered.

func (*DiscoveryServer) InitDebug

func (s *DiscoveryServer) InitDebug(mux *http.ServeMux, sctl *aggregate.Controller)

InitDebug initializes the debug handlers and adds a debug in-memory registry.

func (*DiscoveryServer) Push

func (s *DiscoveryServer) Push(full bool, edsUpdates map[string]struct{})

Push is called to push changes on config updates using ADS. This is set in DiscoveryService.Push, to avoid direct dependencies.

func (*DiscoveryServer) PushStatusHandler

func (s *DiscoveryServer) PushStatusHandler(w http.ResponseWriter, req *http.Request)

PushStatusHandler dumps the last PushContext

func (*DiscoveryServer) Register

func (s *DiscoveryServer) Register(rpcs *grpc.Server)

Register adds the ADS and EDS handles to the grpc server

func (*DiscoveryServer) StreamAggregatedResources

StreamAggregatedResources implements the ADS interface.

func (*DiscoveryServer) SvcUpdate

func (s *DiscoveryServer) SvcUpdate(cluster, hostname string, ports map[string]uint32, _ map[uint32]string)

SvcUpdate is a callback from service discovery when service info changes.

func (*DiscoveryServer) WorkloadUpdate

func (s *DiscoveryServer) WorkloadUpdate(id string, labels map[string]string, _ map[string]string)

WorkloadUpdate is called when workload labels/annotations are updated.

type DiscoveryStream

type DiscoveryStream interface {
	Send(*xdsapi.DiscoveryResponse) error
	Recv() (*xdsapi.DiscoveryRequest, error)

DiscoveryStream is a common interface for EDS and ADS. It also has a shorter name.

type EdsCluster

type EdsCluster struct {

	// LoadAssignment has the pre-computed EDS response for this cluster. Any sidecar asking for the
	// cluster will get this response.
	LoadAssignment *xdsapi.ClusterLoadAssignment

	// FirstUse is the time the cluster was first used, for debugging
	FirstUse time.Time

	// EdsClients keeps track of all nodes monitoring the cluster.
	EdsClients map[string]*XdsConnection `json:"-"`

	// NonEmptyTime is the time the cluster first had a non-empty set of endpoints
	NonEmptyTime time.Time
	// contains filtered or unexported fields

EdsCluster tracks eds-related info for monitored cluster. Used in 1.0, where cluster info is not source-dependent.

type EndpointShards

type EndpointShards struct {

	// Shards is used to track the shards. EDS updates are grouped by shard.
	// Current implementation uses the registry name as key - in multicluster this is the
	// name of the k8s cluster, derived from the config (secret).
	Shards map[string][]*model.IstioEndpoint

	// ServiceAccounts has the concatenation of all service accounts seen so far in endpoints.
	// This is updated on push, based on shards. If the previous list is different than
	// current list, a full push will be forced, to trigger a secure naming update.
	// Due to the larger time, it is still possible that connection errors will occur while
	// CDS is updated.
	ServiceAccounts map[string]bool
	// contains filtered or unexported fields

EndpointShards holds the set of endpoint shards of a service. Registries update individual shards incrementally. The shards are aggregated and split into clusters when a push for the specific cluster is needed.

type EndpointsFilterFunc

type EndpointsFilterFunc func(endpoints []endpoint.LocalityLbEndpoints, conn *XdsConnection, env *model.Environment) []endpoint.LocalityLbEndpoints

EndpointsFilterFunc is a function that filters data from the ClusterLoadAssignment and returns updated one

type MemServiceController

type MemServiceController struct {
	// contains filtered or unexported fields

MemServiceController is a mock service controller

func (*MemServiceController) AppendInstanceHandler

func (c *MemServiceController) AppendInstanceHandler(f func(*model.ServiceInstance, model.Event)) error

AppendInstanceHandler appends a service instance handler to the controller

func (*MemServiceController) AppendServiceHandler

func (c *MemServiceController) AppendServiceHandler(f func(*model.Service, model.Event)) error

AppendServiceHandler appends a service handler to the controller

func (*MemServiceController) Run

func (c *MemServiceController) Run(<-chan struct{})

Run will run the controller

type MemServiceDiscovery

type MemServiceDiscovery struct {
	WantGetProxyServiceInstances  []*model.ServiceInstance
	ServicesError                 error
	GetServiceError               error
	InstancesError                error
	GetProxyServiceInstancesError error

	ClusterID string

	// XDSUpdater will push EDS changes to the ADS model.
	EDSUpdater model.XDSUpdater
	// contains filtered or unexported fields

MemServiceDiscovery is a mock discovery interface

func NewMemServiceDiscovery

func NewMemServiceDiscovery(services map[model.Hostname]*model.Service, versions int) *MemServiceDiscovery

NewMemServiceDiscovery builds an in-memory MemServiceDiscovery

func (*MemServiceDiscovery) AddEndpoint

func (sd *MemServiceDiscovery) AddEndpoint(service model.Hostname, servicePortName string, servicePort int, address string, port int) *model.ServiceInstance

AddEndpoint adds an endpoint to a service.

func (*MemServiceDiscovery) AddHTTPService

func (sd *MemServiceDiscovery) AddHTTPService(name, vip string, port int)

AddHTTPService is a helper to add a service of type http, named 'http-main', with the specified vip and port.

func (*MemServiceDiscovery) AddInstance

func (sd *MemServiceDiscovery) AddInstance(service model.Hostname, instance *model.ServiceInstance)

AddInstance adds an in-memory instance.

func (*MemServiceDiscovery) AddService

func (sd *MemServiceDiscovery) AddService(name model.Hostname, svc *model.Service)

AddService adds an in-memory service.

func (*MemServiceDiscovery) AddWorkload

func (sd *MemServiceDiscovery) AddWorkload(ip string, labels model.Labels)

func (*MemServiceDiscovery) ClearErrors

func (sd *MemServiceDiscovery) ClearErrors()

ClearErrors clear errors used for mocking failures during model.MemServiceDiscovery interface methods

func (*MemServiceDiscovery) GetIstioServiceAccounts

func (sd *MemServiceDiscovery) GetIstioServiceAccounts(hostname model.Hostname, ports []int) []string

GetIstioServiceAccounts gets the Istio service accounts for a service hostname.

func (*MemServiceDiscovery) GetProxyServiceInstances

func (sd *MemServiceDiscovery) GetProxyServiceInstances(node *model.Proxy) ([]*model.ServiceInstance, error)

GetProxyServiceInstances returns service instances associated with a node, resulting in 'in' services.

func (*MemServiceDiscovery) GetProxyWorkloadLabels

func (sd *MemServiceDiscovery) GetProxyWorkloadLabels(proxy *model.Proxy) (model.LabelsCollection, error)

func (*MemServiceDiscovery) GetService

func (sd *MemServiceDiscovery) GetService(hostname model.Hostname) (*model.Service, error)

GetService implements discovery interface Each call to GetService() should return a new *model.Service

func (*MemServiceDiscovery) Instances

func (sd *MemServiceDiscovery) Instances(hostname model.Hostname, ports []string,
	labels model.LabelsCollection) ([]*model.ServiceInstance, error)

Instances filters the service instances by labels. This assumes single port, as is used by EDS/ADS.

func (*MemServiceDiscovery) InstancesByPort

func (sd *MemServiceDiscovery) InstancesByPort(hostname model.Hostname, port int,
	labels model.LabelsCollection) ([]*model.ServiceInstance, error)

InstancesByPort filters the service instances by labels. This assumes single port, as is used by EDS/ADS.

func (*MemServiceDiscovery) ManagementPorts

func (sd *MemServiceDiscovery) ManagementPorts(addr string) model.PortList

ManagementPorts implements discovery interface

func (*MemServiceDiscovery) Services

func (sd *MemServiceDiscovery) Services() ([]*model.Service, error)

Services implements discovery interface Each call to Services() should return a list of new *model.Service

func (*MemServiceDiscovery) SetEndpoints

func (sd *MemServiceDiscovery) SetEndpoints(service string, endpoints []*model.IstioEndpoint)

SetEndpoints update the list of endpoints for a service, similar with K8S controller.

func (*MemServiceDiscovery) WorkloadHealthCheckInfo

func (sd *MemServiceDiscovery) WorkloadHealthCheckInfo(addr string) model.ProbeList

WorkloadHealthCheckInfo implements discovery interface

type SyncStatus

type SyncStatus struct {
	ProxyID         string `json:"proxy,omitempty"`
	ProxyVersion    string `json:"proxy_version,omitempty"`
	ClusterSent     string `json:"cluster_sent,omitempty"`
	ClusterAcked    string `json:"cluster_acked,omitempty"`
	ListenerSent    string `json:"listener_sent,omitempty"`
	ListenerAcked   string `json:"listener_acked,omitempty"`
	RouteSent       string `json:"route_sent,omitempty"`
	RouteAcked      string `json:"route_acked,omitempty"`
	EndpointSent    string `json:"endpoint_sent,omitempty"`
	EndpointAcked   string `json:"endpoint_acked,omitempty"`
	EndpointPercent int    `json:"endpoint_percent,omitempty"`

SyncStatus is the synchronization status between Pilot and a given Envoy

type Workload

type Workload struct {
	// Labels
	Labels map[string]string

Workload has the minimal info we need to detect if we need to push workloads, and to cache data to avoid expensive model allocations.

type XdsConnection

type XdsConnection struct {

	// PeerAddr is the address of the client envoy, from network layer
	PeerAddr string

	// Time of connection, for debugging
	Connect time.Time

	// ConID is the connection identifier, used as a key in the connection table.
	// Currently based on the node name and a counter.
	ConID string

	LDSListeners []*xdsapi.Listener                    `json:"-"`
	RouteConfigs map[string]*xdsapi.RouteConfiguration `json:"-"`
	CDSClusters  []*xdsapi.Cluster

	// Last nonce sent and ack'd (timestamps) used for debugging
	ClusterNonceSent, ClusterNonceAcked   string
	ListenerNonceSent, ListenerNonceAcked string
	RouteNonceSent, RouteNonceAcked       string
	RouteVersionInfoSent                  string
	EndpointNonceSent, EndpointNonceAcked string
	EndpointPercent                       int

	// current list of clusters monitored by the client
	Clusters []string

	// Routes is the list of watched Routes.
	Routes []string

	// LDSWatch is set if the remote server is watching Listeners
	LDSWatch bool
	// CDSWatch is set if the remote server is watching Clusters
	CDSWatch bool

	// Time of last push
	LastPush time.Time

	// Time of last push failure.
	LastPushFailure time.Time
	// contains filtered or unexported fields

XdsConnection is a listener connection type.

type XdsEvent

type XdsEvent struct {
	// contains filtered or unexported fields

XdsEvent represents a config or registry event that results in a push.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to