adaptivetelemetryprocessor

README

Adaptive Telemetry Processor

Status
Stability	alpha: metrics
Distributions	nrdot
Issues
Code Owners	@newrelic/dbi, @newrelic/ohai

Overview

The Adaptive Telemetry Processor (ATP) is an intelligent metric filtering and adaptive sampling component designed for the New Relic OpenTelemetry Collector. It dynamically monitors system and process metrics, filtering out low-value telemetry based on configurable thresholds while maintaining high observability during critical events. ATP reduces telemetry costs by up to 70% while preserving visibility into performance anomalies.

Key Features

• Metric Threshold Filtering: Filters metrics based on configurable thresholds for CPU, memory, disk, and network
• Process-Based Sampling: Monitors specific processes by full path (e.g., /usr/sbin/nginx, /usr/bin/java)
• Dynamic Threshold Adjustment: Automatically adapts thresholds based on historical baselines and workload patterns
• Multi-Metric Composite Scoring: Combines multiple metrics with configurable weights for holistic health assessment
• Anomaly Detection: Detects sudden metric changes and ensures anomalous data is always captured
• Stateful Processing: Maintains persistent state in JSON file across collector restarts
• Low Overhead: Minimal performance impact with efficient metric evaluation

Use Cases

• Cost Optimization: Reduce telemetry volume by 50-70% by filtering "normal" metrics while preserving anomalies
• Noise Reduction: Eliminate low-value metrics from idle or baseline systems
• Selective Monitoring: Monitor only specific processes (applications, containers, services)
• Dynamic Environments: Adapt to changing workload patterns in containerized or cloud environments
• Performance Troubleshooting: Automatically increase sampling during performance degradation
• Compliance & Auditing: Maintain complete audit trail during critical events while reducing steady-state data

Configuration

Basic Configuration

processors:
  adaptivetelemetry:
    # Storage is enabled by default and uses platform-specific paths:
    #   Linux: /var/lib/nrdot-collector/adaptiveprocess.db
    #   Windows: %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db
    # To disable storage entirely, set: enable_storage: false

    # Data retention period in minutes
    retention_minutes: 30

    # Process list to monitor
    # SECURITY: Use full paths in production to prevent spoofing
    include_process_list:
      - "/usr/sbin/nginx"        # Full path (recommended)
      - "/usr/bin/java"          # Full path (recommended)
      - "/usr/bin/postgres"      # Full path (recommended)
    
    # Metric thresholds - metrics below these values are filtered out
    metric_thresholds:
      # CPU metrics (utilization is 0-1 scale, time in seconds)
      system.cpu.utilization: 0.05           # 5% CPU utilization
      system.cpu.time: 5.0                   # 5 seconds CPU time
      
      # Memory metrics (utilization 0-1 scale, usage in bytes)
      system.memory.utilization: 0.05        # 5% memory utilization
      system.memory.usage: 104857600         # 100 MB
      
      # Process-specific metrics
      process.cpu.utilization: 0.05          # 5% process CPU
      process.memory.usage: 104857600        # 100 MB process memory
      
      # Disk metrics
      system.disk.io: 1048576                # 1 MB disk I/O
      system.filesystem.utilization: 0.05    # 5% disk utilization
      
      # Network metrics
      system.network.io: 1048576             # 1 MB network I/O

Advanced Configuration

processors:
  adaptivetelemetry:
    # Storage is enabled by default and uses platform-specific paths
    # Linux: /var/lib/nrdot-collector/adaptiveprocess.db
    # Windows: %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db
    # To disable storage, set: enable_storage: false
    retention_minutes: 30
    
    # Process filtering - monitor specific processes
    # Use full paths for security (prevents process name spoofing)
    include_process_list:
      - "/usr/bin/stress-ng"
      - "/usr/sbin/nginx"
      - "/usr/bin/java"
      - "/usr/bin/postgres"
    
    # Static metric thresholds - metrics below these values are filtered
    metric_thresholds:
      # CPU metrics from hostmetrics receiver
      system.cpu.utilization: 0.05           # 5% CPU utilization (0-1 scale)
      system.cpu.time: 5.0                   # CPU time in seconds
      
      # Memory metrics from hostmetrics receiver
      system.memory.utilization: 0.05        # 5% memory utilization (0-1 scale)
      system.memory.usage: 100               # in bytes (minimal for testing)
      
      # Process metrics from hostmetrics receiver
      process.cpu.utilization: 0.05          # 5% process CPU (0-1 scale)
      process.memory.usage: 100              # process memory in bytes
      process.memory.virtual: 100            # virtual memory in bytes
      
      # Disk metrics from hostmetrics receiver
      system.disk.io.read_bytes: 100         # 100B disk read
      system.disk.io.write_bytes: 100        # 100B disk write
      system.filesystem.utilization: 0.05    # 5% disk utilization (0-1 scale)
      
      # Network metrics from hostmetrics receiver
      system.network.io.receive_bytes: 100   # 100B network receive
      system.network.io.transmit_bytes: 100  # 100B network transmit
    
    # Dynamic threshold configuration
    enable_dynamic_thresholds: true
    dynamic_smoothing_factor: 0.1            # Exponential moving average smoothing
    
    # Minimum thresholds (dynamic thresholds cannot go below these)
    min_thresholds:
      system.cpu.utilization: 0.04           # Minimum 4% CPU
      system.memory.utilization: 0.04        # Minimum 4% memory
      process.cpu.utilization: 0.04          # Minimum 4% process CPU
      system.filesystem.utilization: 0.04    # Minimum 4% disk
    
    # Maximum thresholds (dynamic thresholds cannot exceed these)
    max_thresholds:
      system.cpu.utilization: 0.30           # Maximum 30% CPU
      system.memory.utilization: 0.30        # Maximum 30% memory
      process.cpu.utilization: 0.30          # Maximum 30% process CPU
      system.filesystem.utilization: 0.30    # Maximum 30% disk
    
    # Multi-metric composite scoring
    enable_multi_metric: true
    composite_threshold: 0.8                 # Composite score threshold (0-1)
    
    # Metric weights for composite score calculation
    weights:
      system.cpu.utilization: 0.2            # 20% weight
      system.memory.utilization: 0.2         # 20% weight
      process.cpu.utilization: 0.3           # 30% weight
      system.filesystem.utilization: 0.3     # 30% weight
    
    # Anomaly detection configuration
    enable_anomaly_detection: true
    anomaly_history_size: 15                 # Number of historical data points
    anomaly_change_threshold: 50.0           # 50% change triggers anomaly
    anomaly_min_data_points: 3               # Minimum data points before detection

Complete Working Example with Full Pipeline

This example shows ATP integrated with hostmetrics receiver and other processors in a complete OpenTelemetry Collector configuration:

receivers:
  otlp:
    protocols:
      grpc:
      http:

  hostmetrics:
    collection_interval: 60s
    scrapers:
      processes:
      process:
        metrics:
          process.cpu.time:
            enabled: true
          process.cpu.utilization:
            enabled: true
          process.memory.usage:
            enabled: true
          process.memory.virtual:
            enabled: true
          process.disk.io:
            enabled: true
          process.threads:
            enabled: true
      cpu:
        metrics:
          system.cpu.time:
            enabled: true
          system.cpu.utilization:
            enabled: true
      memory:
        metrics:
          system.memory.usage:
            enabled: true
          system.memory.utilization:
            enabled: true
      disk:
        metrics:
          system.disk.io:
            enabled: true
      filesystem:
        metrics:
          system.filesystem.usage:
            enabled: true
          system.filesystem.utilization:
            enabled: true
      network:
        metrics:
          system.network.io:
            enabled: true

processors:
  adaptivetelemetry:
    # Storage uses default platform-specific path
    # Linux: /var/lib/nrdot-collector/adaptiveprocess.db
    # Windows: %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db
    retention_minutes: 30
    include_process_list:
      - "/usr/bin/stress-ng"
    
    metric_thresholds:
      system.cpu.utilization: 0.05
      system.memory.utilization: 0.05
      process.cpu.utilization: 0.05
      process.memory.usage: 100
      system.disk.io.read_bytes: 100
      system.network.io.receive_bytes: 100
    
    enable_dynamic_thresholds: true
    dynamic_smoothing_factor: 0.1
    min_thresholds:
      system.cpu.utilization: 0.04
      system.memory.utilization: 0.04
    max_thresholds:
      system.cpu.utilization: 0.30
      system.memory.utilization: 0.30
    
    enable_multi_metric: true
    composite_threshold: 0.8
    weights:
      system.cpu.utilization: 0.2
      system.memory.utilization: 0.2
      process.cpu.utilization: 0.3
      system.filesystem.utilization: 0.3
    
    enable_anomaly_detection: true
    anomaly_history_size: 15
    anomaly_change_threshold: 50.0
    anomaly_min_data_points: 3
  
  memory_limiter:
    check_interval: 1s
    limit_mib: 100
  
  batch:
  
  resourcedetection:
    detectors: ["system"]
    system:
      hostname_sources: ["os"]

exporters:
  otlphttp:
    endpoint: https://otlp.nr-data.net
    headers:
      api-key: ${env:NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors:
        - memory_limiter
        - adaptivetelemetry
        - resourcedetection
        - batch
      exporters: [otlphttp]

    metrics:
      receivers: [otlp]
      processors:
        - adaptivetelemetry
        - resourcedetection
        - batch
      exporters: [otlphttp]

Kubernetes DaemonSet Configuration

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: monitoring
data:
  collector-config.yaml: |
    receivers:
      hostmetrics:
        collection_interval: 60s
        scrapers:
          cpu:
          memory:
          disk:
          filesystem:
          network:
          process:
    
    processors:
      adaptivetelemetry:
        # Storage uses default path: /var/lib/nrdot-collector/adaptiveprocess.db
        # To disable storage, set: enable_storage: false
        retention_minutes: 30
        include_process_list:
          - "/usr/bin/java"
          - "/usr/sbin/nginx"
        metric_thresholds:
          system.cpu.utilization: 0.05
          system.memory.utilization: 0.05
          process.cpu.utilization: 0.05
        enable_dynamic_thresholds: true
        enable_multi_metric: true
        enable_anomaly_detection: true
      
      resourcedetection:
        detectors: ["env", "system"]
      
      batch:
    
    exporters:
      otlphttp:
        endpoint: https://otlp.nr-data.net
        headers:
          api-key: ${env:NEW_RELIC_LICENSE_KEY}
    
    service:
      pipelines:
        metrics:
          receivers: [hostmetrics]
          processors: [adaptivetelemetry, resourcedetection, batch]
          exporters: [otlphttp]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otel-collector
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: otel-collector
        image: ghcr.io/newrelic/nrdot-collector:latest
        env:
        - name: NEW_RELIC_LICENSE_KEY
          valueFrom:
            secretKeyRef:
              name: newrelic-license
              key: license-key
        volumeMounts:
        - name: config
          mountPath: /etc/otelcol
        - name: atp-state
          mountPath: /var/lib/nrdot-collector
        - name: hostfs
          mountPath: /hostfs
          readOnly: true
        resources:
          limits:
            memory: 200Mi
            cpu: 200m
          requests:
            memory: 100Mi
            cpu: 100m
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
      - name: atp-state
        hostPath:
          path: /var/lib/nrdot-collector
          type: DirectoryOrCreate
      - name: hostfs
        hostPath:
          path: /
      serviceAccountName: otel-collector
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: otel-collector
  namespace: monitoring

How It Works

Architecture

┌─────────────────────────────────────────────────────────────────┐
│              Adaptive Telemetry Processor (ATP)                  │
│                                                                  │
│  ┌──────────────┐   ┌─────────────────┐   ┌──────────────────┐ │
│  │   Metric     │──▶│  Threshold      │──▶│  Decision        │ │
│  │  Ingestion   │   │  Evaluation     │   │  Engine          │ │
│  └──────────────┘   └─────────────────┘   └──────────────────┘ │
│                             │                       │            │
│                             ▼                       ▼            │
│                     ┌─────────────────┐    ┌──────────────────┐ │
│                     │ Dynamic         │    │  Anomaly         │ │
│                     │ Thresholds      │    │  Detection       │ │
│                     └─────────────────┘    └──────────────────┘ │
│                             │                       │            │
│                             └───────┬───────────────┘            │
│                                     ▼                            │
│                           ┌──────────────────┐                  │
│                           │  Composite Score │                  │
│                           │  Calculator      │                  │
│                           └──────────────────┘                  │
│                                     │                            │
│                                     ▼                            │
│                           ┌──────────────────┐                  │
│                           │  Filter/Pass     │                  │
│                           │  Decision        │                  │
│                           └──────────────────┘                  │
└─────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
                            JSON File State
             /var/lib/nrdot-collector/adaptiveprocess.db
                  (Historical data, thresholds, baselines)
                  (Secured: Path validation, symlink detection)

Processing Flow

1. Metric Ingestion: Receives metrics from hostmetrics or other receivers
2. Process Filtering: Checks if metric is from a monitored process (if include_process_list configured). If yes, metric is passed immediately.
3. Threshold Evaluation: Compares metric value against configured static thresholds
4. Dynamic Adjustment: If enabled, adjusts thresholds based on historical baseline using exponential moving average
5. Composite Scoring: If enabled, calculates weighted composite score across multiple metrics
6. Anomaly Detection: If enabled, detects sudden percentage changes compared to historical data
7. Pass/Filter Decision:
   • Pass: If process in include list OR metric exceeds threshold OR anomaly detected OR composite score high
   • Filter: If process NOT in include list AND metric below threshold AND no anomaly AND composite score low
8. State Persistence: Updates JSON file with latest metric values and thresholds
9. Downstream Emission: Sends passed metrics to next processor or exporter

Filtering Logic

ATP uses a multi-layered decision process:

For each metric data point:
  ├─ Is process in include_process_list? (if configured)
  │  ├─ YES → PASS metric
  │  └─ NO → Continue
  │
  ├─ Does metric exceed static threshold?
  │  └─ YES → PASS metric
  │
  ├─ Is dynamic thresholds enabled?
  │  ├─ Calculate dynamic threshold from baseline
  │  ├─ Does metric exceed dynamic threshold?
  │  │  └─ YES → PASS metric
  │
  ├─ Is multi-metric scoring enabled?
  │  ├─ Calculate composite score across metrics
  │  ├─ Does composite score exceed threshold?
  │  │  └─ YES → PASS metric
  │
  ├─ Is anomaly detection enabled?
  │  ├─ Calculate % change from historical average
  │  ├─ Does change exceed anomaly threshold?
  │  │  └─ YES → PASS metric (anomaly!)
  │
  └─ All checks failed → DROP metric

State Management

The processor maintains a persistent JSON file containing:

• Historical metric values for baseline calculation
• Dynamic threshold values per metric
• Exponential moving averages for anomaly detection
• Process monitoring state
• Timestamp of last update

JSON Structure:

{
  "metrics": {
    "system.cpu.utilization": {
      "history": [0.05, 0.06, 0.04],
      "dynamic_threshold": 0.08,
      "last_updated": 1678900000
    }
  },
  "processes": {
    "nginx": {
      "last_seen": 1678900000,
      "metric_count": 150
    }
  }
}

Retention Policy:

Data older than retention_minutes is automatically purged to prevent file growth:

Every collection cycle:
  Remove entries where timestamp < (current_time - retention_minutes * 60)

State File:

• Location: Automatically uses platform-specific paths:
  • Linux: /var/lib/nrdot-collector/adaptiveprocess.db
  • Windows: %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db
• Security: Paths are hardcoded for security and cannot be changed by users
• Format: JSON
• Size: Typically 1-10 MB depending on retention and metric cardinality
• Permissions: Automatically set to 0600 (owner read/write only) for security
• Directory Permissions: Automatically set to 0700 (owner access only)

Security Features

1. File Permissions

• State file: Should be 0600 (owner read/write only)
• Directory: Should be 0700 (owner access only)
• Prevents unauthorized access to historical metric data

2. Storage Path Security

• Hardcoded paths: Storage paths are platform-specific and cannot be changed by users:
  • Linux: /var/lib/nrdot-collector/adaptiveprocess.db
  • Windows: %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db
• Symlink protection: Detects and rejects symlinks in the path to prevent redirection attacks
• Reparse point protection (Windows): Detects and rejects junctions and mount points
• Path traversal prevention: Uses filepath.Clean() to prevent .. escapes
• Linux FHS compliant: Follows Filesystem Hierarchy Standard for application state data
• Directory permissions: Creates directories with 0700 (owner-only access)
• File permissions: Creates files with 0600 (owner read/write only)

Security benefits:

• No user-controlled paths eliminates entire class of path traversal attacks
• Prevents writing to world-writable directories (like /tmp) that could be exploited
• Eliminates symlink redirection to sensitive system locations (like /etc/passwd)
• Platform-specific defaults follow OS security best practices

3. Process Path Filtering

• Full path matching: Filters by complete executable path (e.g., "/usr/sbin/nginx")
• Prevents spoofing: Requires path separator (/ or ) in include list entries
• Exact match: Only processes with matching full paths are included

4. Resource Limits

• Retention limits: Automatic cleanup of old data prevents disk exhaustion
• Memory limits: Use memory_limiter processor upstream to prevent OOM

Performance Considerations

Resource Usage

Metric	Typical Value	Notes
CPU Usage	< 2%	During steady state
Memory Usage	10-50 MB	Scales with metric cardinality and history size
Disk I/O	Low	Periodic JSON file writes
State File Size	1-10 MB	Depends on retention and metric count
Network	None	Local-only operations

Optimization Tips

1. Adjust Retention Period: Shorter retention (15-30 min) reduces state file size
2. Tune History Size: Lower anomaly_history_size reduces memory usage
3. Disable Optional Features: Turn off anomaly detection or dynamic thresholds if not needed
4. Batch Downstream: Use batch processor after ATP to reduce exporter load
5. Monitor Storage: Check adaptiveprocess.db file size periodically

Scalability

• Metrics Filtered: Tested with 500+ metrics/second per host
• Processes Monitored: Supports 100+ concurrent processes
• Throughput Impact: < 5ms latency added to metric pipeline
• Storage Performance: Efficient JSON serialization/deserialization

Example Pipelines

Scenario 1: Production Web Application

Filter telemetry to only include production Nginx and application servers:

receivers:
  otlp:
    protocols:
      grpc:

processors:
  adaptivetelemetry:
    # Uses default path: /var/lib/nrdot-collector/adaptiveprocess.db (Linux)
    # or %LOCALAPPDATA%\nrdot-collector\adaptiveprocess.db (Windows)
    include_process_list:
      - "/usr/sbin/nginx"
      - "/usr/bin/java"
    
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  otlp/newrelic:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      api-key: ${NEW_RELIC_LICENSE_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [adaptivetelemetry, batch]
      exporters: [otlp/newrelic]
    metrics:
      receivers: [otlp]
      processors: [adaptivetelemetry, batch]
      exporters: [otlp/newrelic]

Scenario 2: Multi-Tenant Environment

Note: Since storage paths are now fixed and platform-specific, multi-tenant setups share the same storage file. The processor maintains state for all processes together.

processors:
  adaptivetelemetry/tenant1:
    # Uses default path (shared storage for all instances)
    include_process_list:
      - "/opt/tenant1/bin/app-tenant1"

  adaptivetelemetry/tenant2:
    # Uses default path (shared storage for all instances)
    include_process_list:
      - "/opt/tenant2/bin/app-tenant2"

service:
  pipelines:
    traces/tenant1:
      receivers: [otlp]
      processors: [adaptivetelemetry/tenant1, batch]
      exporters: [otlp/newrelic]
    traces/tenant2:
      receivers: [otlp]
      processors: [adaptivetelemetry/tenant2, batch]
      exporters: [otlp/newrelic]

Troubleshooting

Common Issues

Issue: No telemetry passing through processor

Symptoms: All telemetry is being dropped

Solutions:

1. Verify include_process_list uses full paths with separators (e.g., /usr/sbin/nginx not nginx)
2. Check executable paths match running processes (use which <command> or readlink -f /proc/<pid>/exe)
3. Ensure path separators are present - entries without / or \ will not match any process
4. Review collector logs for filtering decisions: grep "adaptivetelemetry" collector.log

Issue: State file errors

Symptoms: permission denied or storage-related errors

Solutions:

1. Check directory exists and is writable:
   • Linux: /var/lib/nrdot-collector/
   • Windows: %LOCALAPPDATA%\nrdot-collector\
2. Ensure no symlinks/junctions exist in the storage path (use ls -la on Linux, dir /AL on Windows)
3. Verify collector has write permissions:
   • Linux: chown -R collector-user:collector-group /var/lib/nrdot-collector/
   • Windows: Ensure user running collector has write access
4. Check directory permissions:
   • Linux: chmod 700 /var/lib/nrdot-collector/
   • Windows: Use Security tab in folder properties
5. Check SELinux/AppArmor policies aren't blocking access (Linux)
6. Delete state file to force recreation:
   • Linux: rm -rf /var/lib/nrdot-collector/*
   • Windows: del /Q %LOCALAPPDATA%\nrdot-collector\*
7. To disable storage entirely, set enable_storage: false in config

Issue: High CPU usage

Symptoms: Collector CPU usage > 10%

Solutions:

1. Check for process churn (many short-lived processes)

Debug Logging

Enable debug logging to troubleshoot issues:

service:
  telemetry:
    logs:
      level: debug
      
processors:
  adaptivetelemetry:
    # ... configuration ...

Look for log entries:

• [adaptivetelemetry] Discovered process: <details> - Process discovery
• [adaptivetelemetry] Filtering telemetry from PID <pid> - Filter decisions
• [adaptivetelemetry] Enriching telemetry with attributes - Enrichment actions
• [adaptivetelemetry] State file saved - Persistence operations

Validation Commands

Linux:

# Check state file exists and permissions (should be 0600)
ls -la /var/lib/nrdot-collector/

# Verify directory permissions (should be 0700)
stat -c "%a %n" /var/lib/nrdot-collector/

# View state file contents
cat /var/lib/nrdot-collector/adaptiveprocess.db | jq

# Monitor collector logs in real-time
tail -f /var/log/nrdot-collector/collector.log | grep adaptivetelemetry

# List matching processes
ps aux | grep -E "nginx|java|postgres"

# Check process details
ls -l /proc/<pid>/exe
cat /proc/<pid>/cmdline

Windows:

# Check state directory exists
dir $env:LOCALAPPDATA\nrdot-collector

# View state file contents (requires jq)
Get-Content $env:LOCALAPPDATA\nrdot-collector\adaptiveprocess.db | jq

# Monitor collector logs
Get-Content -Path "C:\ProgramData\nrdot-collector\logs\collector.log" -Wait | Select-String "adaptivetelemetry"

# List matching processes
Get-Process | Where-Object {$_.Name -match "nginx|java|postgres"}

# Check process details
Get-Process -Id <pid> | Select-Object Path, CommandLine

Security Best Practices

1. Run with Minimal Privileges: Use a dedicated service account with only required capabilities
2. Storage Security: Storage paths are automatically set to secure platform-specific locations and cannot be changed by users. Ensure the storage directory has proper ownership and permissions:
   • Linux: /var/lib/nrdot-collector/ with 0700 permissions
   • Windows: %LOCALAPPDATA%\nrdot-collector\ with restricted user access
3. Audit Access: Monitor who/what accesses the state directory using filesystem audit tools (auditd/inotify on Linux, auditing policies on Windows)
4. Keep Updated: Apply security patches promptly
5. Use Full Paths in include_process_list: Always specify full executable paths (e.g., /usr/sbin/nginx) instead of just process names to prevent process name spoofing attacks. See "Process Include List Security" below.
6. Avoid Symlinks: Do not use symlinks in the storage path as they are detected and rejected to prevent redirection attacks
7. Directory Permissions: The processor automatically creates storage directories with 0700 (owner-only) and files with 0600 (owner read/write only) permissions

Process Include List Security

The include_process_list feature uses full path matching only for security.

Full Path Matching (Required)

include_process_list:
  - "/usr/sbin/nginx"           # Matches only this exact path
  - "/usr/bin/postgres"         # Matches only this exact path
  - "/opt/redis/bin/redis-server"

Security Features:

• Prevents process spoofing: Malicious processes cannot bypass filters by using the same basename
• Exact path matching: A process at /tmp/nginx will NOT match /usr/sbin/nginx
• Path separator required: Entries without path separators (/ or ) will not match any process
• Secure by default: No basename-only matching to eliminate attack vectors

Invalid Configuration (Will Not Match):

include_process_list:
  - "nginx"                     # ❌ No path separator - will not match any process
  - "postgres"                  # ❌ No path separator - will not match any process

Attack Scenario Prevented:

# An attacker attempts to spoof a trusted process:
cp /malicious/backdoor /tmp/nginx
/tmp/nginx  # ❌ Will NOT match - only "/usr/sbin/nginx" matches

# Valid configuration requires full paths:
include_process_list:
  - "/usr/sbin/nginx"          # ✅ Only matches this exact path

Best Practices:

• Always use absolute paths with full directory structure
• Verify executable paths using which <command> or readlink -f /proc/<pid>/exe
• Use Unix paths on Linux/macOS: /usr/sbin/nginx
• Use Windows paths on Windows: C:\Program Files\nginx\nginx.exe

Contributing

See CONTRIBUTING.md for general contribution guidelines.

Support

• Issues: GitHub Issues
• Security Issues: Report to security@newrelic.com
• Documentation: New Relic Docs

Acknowledgments

• Built on the OpenTelemetry Collector
• Inspired by process filtering patterns in observability tools
• Security hardening guided by OWASP and CIS benchmarks

Documentation

Overview

Package adaptivetelemetryprocessor implements a processor for adaptive telemetry.

Index

• func NewFactory() processor.Factory
• type Config
  • func (cfg *Config) Normalize()
  • func (cfg *Config) Validate() error
• type EntityStateStorage

Functions

func NewFactory

func NewFactory() processor.Factory

NewFactory creates the processor.Factory used by the Collector to construct this processor.

Types

type Config

type Config struct {
	// Generic thresholds per metric (required for those metrics to participate in evaluation)
	MetricThresholds map[string]float64 `mapstructure:"metric_thresholds"`

	// Common settings
	RetentionMinutes int64 `mapstructure:"retention_minutes"`
	EnableStorage    *bool `mapstructure:"enable_storage"`

	// Dynamic thresholds
	EnableDynamicThresholds bool               `mapstructure:"enable_dynamic_thresholds"`
	DynamicSmoothingFactor  float64            `mapstructure:"dynamic_smoothing_factor"`
	MinThresholds           map[string]float64 `mapstructure:"min_thresholds"`
	MaxThresholds           map[string]float64 `mapstructure:"max_thresholds"`

	// Multi metric (composite) scoring
	EnableMultiMetric  bool               `mapstructure:"enable_multi_metric"`
	CompositeThreshold float64            `mapstructure:"composite_threshold"`
	Weights            map[string]float64 `mapstructure:"weights"`

	// Anomaly detection
	EnableAnomalyDetection bool    `mapstructure:"enable_anomaly_detection"`
	AnomalyHistorySize     int     `mapstructure:"anomaly_history_size"`
	AnomalyChangeThreshold float64 `mapstructure:"anomaly_change_threshold"`
	AnomalyMinDataPoints   int     `mapstructure:"anomaly_min_data_points"` // Minimum data points before detection starts

	// Include list - processes that should always be monitored (bypass all filters)
	IncludeProcessList []string `mapstructure:"include_process_list"`

	// Debug options
	DebugShowAllFilterStages bool `mapstructure:"debug_show_all_filter_stages"`
}

Config defines processor configuration. All numeric fields are validated & capped during processor construction so startup does not fail for mildly invalid values; only clearly invalid (e.g. negative thresholds) return an error.

func (*Config) Normalize

func (cfg *Config) Normalize()

Normalize applies defaults & caps. Must be called before processor usage. It does not log; caller should.

func (*Config) Validate

func (cfg *Config) Validate() error

Validate performs strict validation after normalization for values that must not be negative.

type EntityStateStorage

type EntityStateStorage interface {
	Load() (map[string]*trackedEntity, error)

	Save(map[string]*trackedEntity) error

	Close() error
}