mdatagen

README

Metadata Generator

Status
Stability	alpha: metrics
Issues
Code Owners	@dmitryax

Every component's documentation should include a brief description of the component and guidance on how to use it. There is also some information about the component (or metadata) that should be included to help end-users understand the current state of the component and whether it is right for their use case. Examples of this metadata about a component are:

• its stability level
• the distributions containing it
• the types of pipelines it supports
• metrics emitted in the case of a scraping receiver, a scraper, or a connector

The metadata generator defines a schema for specifying this information to ensure it is complete and well-formed. The metadata generator is then able to ingest the metadata, validate it against the schema and produce documentation in a standardized format. An example of how this generated documentation looks can be found in documentation.md.

Using the Metadata Generator

In order for a component to benefit from the metadata generator (mdatagen) these requirements need to be met:

1. A yaml file containing the metadata that needs to be included in the component
2. The component should declare a go:generate mdatagen directive which tells mdatagen what to generate

As an example, here is a minimal metadata.yaml for the OTLP receiver:

type: otlp
status:
  class: receiver
  stability:
    beta: [logs]
    stable: [metrics, traces]

Detailed information about the schema of metadata.yaml can be found in metadata-schema.yaml.

The go:generate mdatagen directive is usually defined in a doc.go file in the same package as the component, for example:

//go:generate mdatagen metadata.yaml

package main

Below are some more examples that can be used for reference:

• The ElasticSearch receiver has an extensive metadata.yaml
• The host metrics receiver has internal subcomponents, each with their own metadata.yaml and doc.go. See cpuscraper for example.

You can run cd cmd/mdatagen && $(GOCMD) install . to install the mdatagen tool in GOBIN and then run mdatagen metadata.yaml to generate documentation for a specific component or you can run make generate to generate documentation for all components.

Central configuration file

mdatagen supports a repository-level configuration file named .mdatagen.yaml.

This is used for skipping validation and configuring project-level hooks.

Component Config Documentation

The metadata generator supports automatic generation of configuration schemas for components. This generates JSON Schema files that enable IDE autocompletion, validation, and documentation for component configuration. In the future it will also generate Go config structs and human-readable documentation for configuration options

To define a configuration schema, add a config section to your metadata.yaml:

type: myreceiver
status:
  class: receiver
  stability:
    beta: [metrics, traces]

config:
  type: object
  properties:
    endpoint:
      type: string
      description: The endpoint to listen on
      default: "localhost:4317"
    timeout:
      type: duration
      description: Request timeout duration
      default: "30s"
    tls:
      $ref: go.opentelemetry.io/collector/config/configtls.server_config
  required: [endpoint]

The config section uses Go-centric types directly in the type field, and also supports:

• Validation constraints: minLength, maxLength, pattern, minimum, maximum, enum, etc.
• References: Internal ($ref: definition_name), external ($ref: package.path.type), or relative ($ref: ./internal/config.type)
• Reusable definitions: Define common schemas in $defs and reference them with $ref
• Schema composition: Use allOf for complex configurations

Supported types

Primitive types

Use Go type names directly in the type field:

Type	Go type	JSON Schema type
string	string	string
bool	bool	boolean
int	int	integer
int8	int8	integer
int16	int16	integer
int32	int32	integer
int64	int64	integer
uint	uint	integer
uint8	uint8	integer
uint16	uint16	integer
uint32	uint32	integer
uint64	uint64	integer
byte	byte	integer
rune	rune	integer
float32	float32	number
float64	float64	number
any	any	(no type constraint)

Container types

Type	Go type	JSON Schema type	Notes
object	struct	object	Requires properties:
slice	[]T	array	Requires values: for the element type
map	map[string]T	object	Requires values: for the value type

Example using container types:

config:
  type: object
  properties:
    endpoints:
      type: slice
      values:
        type: string
      description: List of endpoints to connect to.
    headers:
      type: map
      values:
        type: string
      description: Extra HTTP headers to attach to each request.
    tls:
      $ref: go.opentelemetry.io/collector/config/configtls.server_config

Alias types

Shorthand types that expand to a primitive or container type with additional annotations:

Alias	Go type	JSON Schema representation	Notes
float	float32	number	Shorthand for float32
double	float64	number	Shorthand for float64
duration	time.Duration	string with Go duration pattern	e.g. "30s", "1h30m"
time	time.Time	string with format: date-time	RFC 3339 format
opaque_string	configopaque.String	string	Masked in logs
component_id	component.ID	string	Collector component ID
opaque_map	configopaque.MapList	object of name/value pairs	Values masked in logs

Example using alias types:

config:
  type: object
  properties:
    max_results:
      type: int64
      default: 100
    timeout:
      type: duration
      default: 30s
    api_token:
      type: opaque_string

Metrics Builder Configuration

For receivers, scrapers, and other components that emit metrics, mdatagen can generate metrics builder configuration from metadata.yaml.

type: myreceiver
status:
  class: receiver
  stability:
    beta: [metrics]

resource_attributes:
  transport:
    description: Transport used by the request.
    type: string
    enabled: true

attributes:
  status_code:
    description: Response status code.
    type: int
    requirement_level: opt_in

metrics:
  http.server.request.count:
    enabled: true
    description: Number of received requests.
    unit: "{request}"
    sum:
      values: int
      monotonic: true
      aggregation_temporality: cumulative
    attributes: [status_code]

This lets users:

• enable or disable individual metrics
• enable or disable resource attributes
• use metrics_include and metrics_exclude on resource attributes to only emit metrics with matching resource attribute values

Metric Reaggregation Configuration

mdatagen lets users reduce metric cardinality by dropping selected metric attributes and aggregating the resulting datapoints.

attributes:
  transport:
    description: Transport used by the request.
    type: string
    requirement_level: recommended
  status_code:
    description: Response status code.
    type: int
    requirement_level: opt_in

This adds two per-metric settings for metrics that declare attributes:

• attributes: the subset of metric attributes to keep in the emitted metric stream
• aggregation_strategy: how collapsed datapoints are merged, using sum, avg, min, or max

Defaults:

• sum metrics use sum; gauge metrics use avg
• required attributes are always kept
• recommended and conditionally_required attributes are kept by default, but users can remove them
• opt_in attributes are omitted by default, so that dimension is aggregated unless the user adds it

Example user configuration:

receivers:
  myreceiver:
    metrics:
      http.server.request.count:
        enabled: true
        aggregation_strategy: sum
        attributes: [transport]

In this example, datapoints that only differ by status_code are aggregated together, while transport remains part of the output identity.

Feature Gates Documentation

The metadata generator supports automatic documentation generation for feature gates used by components. Feature gates are documented by adding a feature_gates section to your metadata.yaml:

type: mycomponent
status:
  class: receiver
  stability:
    beta: [metrics, traces]

feature_gates:
  - id: receiver.mycomponent.newFeature
    description: 'Enables new feature functionality that improves performance'
    stage: alpha
    from_version: 'v0.100.0'
    reference_url: 'https://github.com/open-telemetry/opentelemetry-collector/issues/12345'

  - id: receiver.mycomponent.stableFeature
    description: 'A feature that has reached stability'
    stage: stable
    from_version: 'v0.90.0'
    to_version: 'v0.95.0'
    reference_url: 'https://github.com/open-telemetry/opentelemetry-collector/issues/11111'

This will generate a "Feature Gates" section in the component's documentation.md file with a table containing:

• Feature Gate: The gate identifier
• Stage: The lifecycle stage (alpha, beta, stable, deprecated)
• Description: Brief description of what the gate controls
• From Version: Version when the gate was introduced
• To Version: Version when stable/deprecated gates will be removed (if applicable)
• Reference: Link to additional contextual information

The feature gate definitions should correspond to actual gates registered in your component code using the Feature Gates API.

By default, mdatagen applies strict validation to feature gate entries:

• Every gate id must be prefixed with <status.class>.<type>. (e.g. receiver.mycomponent.newFeature), so gates are namespaced to the component that owns them.
• Every reference_url must be a GitHub issue URL of the form https://github.com/<owner>/<repo>/issues/<number>. Pull requests, blog posts, and other URLs are rejected.

Individual gates that predate these rules can be grandfathered in by setting skip_strict_validation: true on the gate entry itself. New gates should not set this flag.

Generate multiple metadata packages

By default, mdatagen will generate a package called metadata in the internal directory. If you want to generate a package with a different name, you can use the generated_package_name configuration field to provide an alternate name.

type: otlp
generated_package_name: customname
status:
  class: receiver
  stability:
    beta: [logs]
    stable: [metrics, traces]

The most common scenario for this would be making major changes to a receiver's metadata without breaking what exists. In this scenario, mdatagen could produce separate packages for different metadata specs in the same receiver:

//go:generate mdatagen metadata.yaml
//go:generate mdatagen custom.yaml

package main

With two different packages generated, the behaviour for which metadata is used can be easily controlled via featuregate or a similar mechanism.

Contributing to the Metadata Generator

The code for generating the documentation can be found in loader.go and the templates for rendering the documentation can be found in templates. When making updates to the metadata generator or introducing support for new functionality:

1. Ensure the metadata-schema.yaml and metadata.yaml files reflect the changes.
2. Run make mdatagen-test.
3. Make sure all tests are passing including generated tests.
4. Run make generate.