README
¶
OpenTelemetry eBPF Profiler
This repository implements a whole-system, cross-language profiler for Linux via eBPF.
Core features and strengths
- Implements the Alpha OTel Profiles signal
- Very low CPU and memory overhead (1% CPU and 250MB memory are our upper limits in testing and the agent typically manages to stay way below that)
- Support for native C/C++ executables without the need for DWARF debug
information (by leveraging
.eh_framedata as described in US11604718B1) - Support profiling of system libraries without frame pointers and without debug symbols on the host.
- Support for mixed stacktraces between runtimes - stacktraces go from Kernel space through unmodified system libraries all the way into high-level languages.
- Support for native code (C/C++, Rust, Zig, Go, etc. without debug symbols on host)
- Support for a broad set of HLLs, like Hotspot JVM, Python, Ruby, PHP, Node.JS, V8, Perl, Erlang and .NET.
- 100% non-intrusive: there's no need to load agents or libraries into the processes that are being profiled.
- No need for any reconfiguration, instrumentation or restarts of HLL interpreters and VMs: the agent supports unwinding each of the supported languages in the default configuration.
- ARM64 support for all unwinders.
- Support for native
inline frames, which provide insights into compiler optimizations and offer a higher precision of function call chains.
Supported Linux kernel version
The minimum required Linux kernel version has increased with certain commits. Specifically:
- Commit 8047150e was the last to support kernel version 5.4. Subsequent changes may require a minimal Linux kernel version of 5.10 or greater.
- Commit 7ddc23ea was the last to support kernel version 4.19. Subsequent changes may require a minimal Linux kernel version of at least 5.4.
Updating the supported Linux kernel version
The project maintains its minimum supported kernel version in line with the lowest kernel version currently provided by actively maintained major Linux distributions, which include Debian stable, Red Hat Enterprise Linux, Ubuntu LTS, Amazon Linux and SUSE Linux. The minimum requirement may be increased when all such distributions no longer ship a specific kernel version. This approach enables the codebase to utilize newer eBPF features and avoids the need to maintain compatibility shims for obsolete kernels.
It should be noted that certain distributions incorporate eBPF features from newer kernels into their supported versions. When this occurs, the distribution's stated kernel version does not accurately reflect its true eBPF capabilities and will not prevent us from increasing the minimum supported version. On such kernels, the no-kernel-version-check configuration option can be used to bypass the checks and allow the profiler to execute.
Building
We have integrated the profiler into the OTel Collector as a receiver, and this is the supported configuration going forward.
To aid with development, testing and debugging, we also offer a standalone profiling agent binary named ebpf-profiler,
and a local build of an OTel Collector profiling receiver binary (otelcol-ebpf-profiler). These binaries are not
supported in any way, can be dropped in the future and should not be deployed in production.
Platform Requirements
The agent can be built with the provided make targets. Docker is required for containerized builds, and both amd64 and arm64 architectures are supported.
For Linux, the following steps apply:
- Build the agent for your current machine's architecture:
make agent - To cross-compile for a different architecture (e.g. arm64):
make agent TARGET_ARCH=arm64
The resulting binary will be named ebpf-profiler in the current directory.
Other OSes
Since the profiler is Linux-only, macOS and Windows users need to set up a Linux VM to build and run the agent. Ensure the appropriate architecture is specified if using cross-compilation. Use the same make targets as above after the Linux environment is configured in the VM.
Alternative Build (Without Docker)
You can build the agent without Docker by directly installing the dependencies listed in the Dockerfile. Once dependencies are set up, simply run:
make
This will build the profiler natively on your machine.
Building otelcol-ebpf-profiler locally (Without Docker)
You can build the local otelcol-ebpf-profiler binary by running:
make otelcol-ebpf-profiler
or to cross-compile for a different architecture (e.g. arm64):
make otelcol-ebpf-profiler TARGET_ARCH=arm64
See local.example.yaml for an example configuration.
Running
You can start the agent with the following command:
sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls
To start the OTel Collector profiling receiver, run:
sudo ./otelcol-ebpf-profiler --feature-gates=+service.profilesSupport --config cmd/otelcol-ebpf-profiler/local.example.yaml
The agent comes with a functional but work-in-progress / evolving implementation of the recently released Alpha OTel Profiles signal.
The agent loads the eBPF program and its maps, starts unwinding and reports captured traces to the backend.
Open Source Backends
As the OTel Profiles signal is still in development, mature production-ready backends have yet to emerge. The following open source projects can be used as backends:
- devfiler — to speed up development and experimentation, Elastic has open-sourced a desktop application that reimplements the backend (collection, data storage, symbolization and UI) portion of the eBPF profiler. Note that devfiler is not a real production backend and should not be used as such. It is solely aimed at testing, experimentation and development.
- Pyroscope — an open source continuous profiling database that natively supports ingesting OTel profiling data.
Development
To understand how this project works and learn more about profiling, check out Profiling internals
Contributing
Emeritus
- Dmitry Filimonov, Maintainer
- Joel Höner, Approver
- Tim Rühsen, Approver
For more information about the emeritus role, see the community repository.
Legal
Licensing Information
This project is licensed under the Apache License 2.0 (Apache-2.0). Apache License 2.0
The eBPF source code is licensed under the GPL 2.0 license. GPL 2.0
Licenses of dependencies
To display a summary of the dependencies' licenses:
make legal
Documentation
¶
There is no documentation for this package.
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
asm
|
|
|
internal/metadata
Package metadata contains the autogenerated telemetry and build information for the receiver/profiling component.
|
Package metadata contains the autogenerated telemetry and build information for the receiver/profiling component. |
|
Package host implements types and methods specific to interacting with eBPF maps.
|
Package host implements types and methods specific to interacting with eBPF maps. |
|
internal
|
|
|
apmint
Package apmint implements a pseudo interpreter handler that detects APM agent libraries, establishes socket connections with them and notifies them about the stack traces that we collected for their process.
|
Package apmint implements a pseudo interpreter handler that detects APM agent libraries, establishes socket connections with them and notifies them about the stack traces that we collected for their process. |
|
interpreterconfig
Package interpreterconfig aggregates per-interpreter configuration.
|
Package interpreterconfig aggregates per-interpreter configuration. |
|
Package kallsyms provides functionality for reading /proc/kallsyms and using it to symbolize kernel addresses.
|
Package kallsyms provides functionality for reading /proc/kallsyms and using it to symbolize kernel addresses. |
|
basehash
Package basehash provides basic types to implement hash identifiers.
|
Package basehash provides basic types to implement hash identifiers. |
|
hash
Package hash provides the same hash primitives as used by the eBPF.
|
Package hash provides the same hash primitives as used by the eBPF. |
|
pfbufio
Package pfbufio provides caching reader implementation similar to bufio, but with additional primitives and being more GC friendly.
|
Package pfbufio provides caching reader implementation similar to bufio, but with additional primitives and being more GC friendly. |
|
pfelf
package pfelf implements functions for processing of ELF files and extracting data from them.
|
package pfelf implements functions for processing of ELF files and extracting data from them. |
|
pfelf/internal/mmap
Package mmap is inspired by golang.org/x/exp/mmap with additional functionality.
|
Package mmap is inspired by golang.org/x/exp/mmap with additional functionality. |
|
xsync
Package xsync provides thin wrappers around locking primitives in an effort towards better documenting the relationship between locks and the data they protect.
|
Package xsync provides thin wrappers around locking primitives in an effort towards better documenting the relationship between locks and the data they protect. |
|
Package log provides a public logging interface for go.opentelemetry.io/ebpf-profiler.
|
Package log provides a public logging interface for go.opentelemetry.io/ebpf-profiler. |
|
lpm package provides helpers for calculating prefix lists from ranges
|
lpm package provides helpers for calculating prefix lists from ranges |
|
Package maccess provides functionality to check if a certain bug in copy_from_user_nofault is patched.
|
Package maccess provides functionality to check if a certain bug in copy_from_user_nofault is patched. |
|
Package metrics contains the code for reporting metrics.
|
Package metrics contains the code for reporting metrics. |
|
genids
command
|
|
|
stackdeltatypes
Package stackdeltatypes provides types used to represent stack delta information as constructed by `nativeunwind.GetIntervalStructures` This information is a post-processed form of the stack delta information that is used in all relevant packages.
|
Package stackdeltatypes provides types used to represent stack delta information as constructed by `nativeunwind.GetIntervalStructures` This information is a post-processed form of the stack delta information that is used in all relevant packages. |
|
nopanicslicereader provides little convenience utilities to read "native" endian values from a slice at given offset.
|
nopanicslicereader provides little convenience utilities to read "native" endian values from a slice at given offset. |
|
Package periodiccaller allows periodic calls of functions.
|
Package periodiccaller allows periodic calls of functions. |
|
Package processmanager manages the loading and unloading of information related to processes.
|
Package processmanager manages the loading and unloading of information related to processes. |
|
remotememory provides access to memory space of a process.
|
remotememory provides access to memory space of a process. |
|
rust-crates
|
|
|
symblib-capi/go
command
|
|
|
successfailurecounter provides a wrapper to atomically increment success or failure counters.
|
successfailurecounter provides a wrapper to atomically increment success or failure counters. |
|
support maps the definitions from headers in the C world into a nice go way
|
support maps the definitions from headers in the C world into a nice go way |
|
tools
|
|
|
coredump
command
|
|
|
coredump/cloudstore
cloudstore provides access to the cloud based storage used in the tests.
|
cloudstore provides access to the cloud based storage used in the tests. |
|
coredump/testsources/go
command
|
|
|
coredump/testsources/go/cgo
command
|
|
|
coredump/testsources/go/time
command
|
|
|
errors-codegen
command
errors-codegen generates the code containing the host agent error code enums.
|
errors-codegen generates the code containing the host agent error code enums. |
|
gooffsets
command
|
|
|
stackdeltas
command
A command-line tool to parse stack deltas from given ELF files.
|
A command-line tool to parse stack deltas from given ELF files. |
|
zstpak
command
|
|
|
Package tracer
|
Package tracer |