README
¶
Inspektor Gadget is a set of tools and framework for data collection and system inspection on Kubernetes clusters and Linux hosts using eBPF. It manages the packaging, deployment and execution of Gadgets (eBPF programs encapsulated in OCI images) and provides mechanisms to customize and extend Gadget functionality.
Note: Major new functionality was released in v0.31.0. Please read the blog post for a detailed overview.
Features
- Build and package eBPF programs into OCI images called Gadgets
- Targets Kubernetes clusters and Linux hosts
- Collect and export data to observability tools with a simple command (and soon via declarative configuration)
- Security mechanisms to restrict and lock-down which Gadgets can be run
- Automatic enrichment: map kernel data to high-level resources like Kubernetes and container runtimes
- Supports WebAssembly modules to post-process data and customize IG operators; using any WASM-supported language
- Supports many modes of operation; cli, client-server, API, embeddable via Golang library
Quick start
The following examples use the trace_open Gadget which triggers when a file is open on the system.
Kubernetes
Deployed to the Cluster
krew is the recommended way to install
kubectl gadget. You can follow the
krew's quickstart
to install it and then install kubectl gadget by executing the following
commands.
kubectl krew install gadget
kubectl gadget deploy
kubectl gadget run trace_open:latest
Check Installing on Kubernetes to learn more about different options.
Kubectl Node Debug
We can use kubectl node debug to run ig on a Kubernetes node:
kubectl debug --profile=sysadmin node/minikube-docker -ti --image=ghcr.io/inspektor-gadget/ig:latest -- ig run trace_open:latest
For more information on how to use ig without installation on Kubernetes, please refer to the ig documentation.
Linux
Install Locally
Install the ig binary locally on Linux and run a Gadget:
IG_ARCH=amd64
IG_VERSION=$(curl -s https://api.github.com/repos/inspektor-gadget/inspektor-gadget/releases/latest | jq -r .tag_name)
curl -sL https://github.com/inspektor-gadget/inspektor-gadget/releases/download/${IG_VERSION}/ig-linux-${IG_ARCH}-${IG_VERSION}.tar.gz | sudo tar -C /usr/local/bin -xzf - ig
sudo ig run trace_open:latest
Check Installing on Linux to learn more.
Run in a Container
docker run -ti --rm --privileged -v /:/host --pid=host ghcr.io/inspektor-gadget/ig:latest run trace_open:latest
For more information on how to use ig without installation on Linux, please check Using ig in a container.
MacOS or Windows
It's possible to control an ig running in Linux from different operating systems by using the gadgetctl binary.
Run the following on a Linux machine to make ig available to clients.
sudo ig daemon --host=tcp://0.0.0.0:1234
Download the gadgetctl tools for MacOS
(amd64,
arm64) or windows and run the Gadget specifying the IP address of the Linux machine:
gadgetctl run trace_open:latest --remote-address=tcp://$IP:1234
The above demonstrates the simplest command. To learn how to filter, export, etc. please consult the documentation for the run command.
Core concepts
What is a Gadget?
Gadgets are the central component in the Inspektor Gadget framework. A Gadget is an OCI image that includes one or more eBPF programs, metadata YAML file and, optionally, WASM modules for post processing.
As OCI images, they can be stored in a container registry (compliant with the OCI specifications), making them easy to distribute and share.
Gadgets are built using the ig image build command.
You can find a growing collection of Gadgets on Artifact HUB. This includes both in-tree Gadgets (hosted in this git repository in the /gadgets directory and third-party Gadgets).
See the Gadget documentation for more information.
⚠ For versions prior to v0.31.0
Prior to v0.31.0, Inspektor Gadget only shipped gadgets embedded in its executable file. As of v0.31.0 these built-in Gadgets are still available and work as before, but their use is discouraged as they will be deprecated at some point. We encourage users to use image-based Gadgets going forward, as they provide more features and decouple the eBPF programs from the Inspektor Gadget release process.
What is enrichment?
The data that eBPF collects from the kernel includes no knowledge about Kubernetes, container runtimes or any other high-level user-space concepts. In order to relate this data to these high-level concepts and make the eBPF data immediately more understandable, Inspektor Gadget automatically uses kernel primitives such as mount namespaces, pids or similar to infer which high-level concepts they relate to; Kubernetes pods, container names, DNS names, etc. The process of augmenting the eBPF data with these high-level concepts is called enrichment.
Enrichment flows the other way, too. Inspektor Gadget enables users to do high-performance in-kernel filtering by only referencing high-level concepts such as Kubernetes pods, container names, etc.; automatically translating these to the corresponding low-level kernel resources.
What is an operator?
In Inspektor Gadget, an operator is any part of the framework where an action is taken. Some operators are under-the-hood (i.e. fetching and loading Gadgets) while others are user-exposed (enrichment, filtering, export, etc.) and can be reordered and overridden.
Further learning
Use the project documentation to learn more about:
Kernel requirements
Kernel requirements are largely determined by the specific eBPF functionality a Gadget makes use of. The eBPF functionality available to Gadgets depend on the version and configuration of the kernel running running in the node/machine where the Gadget is being loaded. Gadgets developed by the Inspektor Gadget project require at least Linux 5.10 with BTF enabled.
Refer to the documentation for a specific Gadget for any notes regarding requirements.
Code examples
There are some examples in this folder showing the usage
of the Golang packages provided by Inspektor Gadget. These examples are
designed for developers that want to use the Golang packages exposed by
Inspektor Gadget directly. End-users do not need this and can use
kubectl-gadget or ig directly.
Contributing
Contributions are welcome, see CONTRIBUTING.
Community Meeting
We hold community meetings regularly. Please check our calendar for the full schedule of up-coming meetings and please add any topic you want to discuss to our meeting notes document.
Slack
Join the discussions on the #inspektor-gadget channel in the Kubernetes Slack.
Talks
- Demystifying DNS: A Guide to Understanding and Debugging Request Flows in Kubernetes Clusters, Container Days Hamburg - September 2024 (video)
- Collecting Low-Level Metrics with eBPF, KubeCon + CloudNativeCon North America 2023 (video, slides)
- A (re)introduction of Inspektor Gadget: A Containerized Framework for eBPF Systems Inspection, Cloud Native Rejekts Chicago - November 2023 (video)
- Gaining Linux insights with Inspektor Gadget, an eBPF tool and systems inspection framework, All Systems Go - September 2023 (video)
- Overcoming the Challenges of Debugging Containers, Container Days Hamburg - September 2023 (video)
- Using the EBPF Superpowers To Generate Kubernetes Security Policies, KubeCon + CloudNativeCon North America 2022 (video, slides)
- Debug Your Clusters with eBPF-Powered Tools, Cloud Native eBPF Day North America 2022 (video, slides)
- Who Needs an API Server to Debug a Kubernetes Cluster?, Cloud Native eBPF Day North America 2022 (video, slides)
- Inspektor Gadget, introduction and demos, eCHO Livestream - September 2021 (video)
- OpenShift Commons Briefing: Unleash eBPF Superpowers with Kubectl Gadget, Openshift Commons 2020 (video)
- Tutorial: Understanding What Happens Inside Kubernetes Clusters Using BPF Tools, Open Source Summit EU 2020 (video)
- Inspektor Gadget and traceloop: Tracing containers syscalls using BPF, FOSDEM 2020 (video, slides)
- Traceloop for systemd and Kubernetes + Inspektor Gadget, All Systems Go 2019 (video)
Thanks
- BPF Compiler Collection (BCC): some of the gadgets are based on BCC tools.
- kubectl-trace: the Inspektor Gadget architecture was inspired from kubectl-trace.
- cilium/ebpf: the gadget tracer manager and some other gadgets use the cilium/ebpf library.
License
The Inspektor Gadget user-space components are licensed under the Apache License, Version 2.0. The BPF code templates are licensed under the General Public License, Version 2.0, with the Linux-syscall-note.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
gadget-container
|
|
|
gadgets
|
|
|
internal
|
|
|
version
Package version stores the semver of this binary.
|
Package version stores the semver of this binary. |
|
pkg
|
|
|
apis/gadget/v1alpha1
Package v1alpha1 contains API Schema definitions for the gadget v1alpha1 API group +kubebuilder:object:generate=true +groupName=gadget.kinvolk.io
|
Package v1alpha1 contains API Schema definitions for the gadget v1alpha1 API group +kubebuilder:object:generate=true +groupName=gadget.kinvolk.io |
|
btfgen
Package btfgen provides a way to load BTF information generated with btfgen.
|
Package btfgen provides a way to load BTF information generated with btfgen. |
|
btfhelpers
Package btfhelpers provides a couple of helper functions to bridge Go's reflection system with types from BTF
|
Package btfhelpers provides a couple of helper functions to bridge Go's reflection system with types from BTF |
|
cachedmap
Package cachedmap provides a CachedMap which functions as a map with a TTL for old entries.
|
Package cachedmap provides a CachedMap which functions as a map with a TTL for old entries. |
|
client/clientset/versioned
This package has the automatically generated clientset.
|
This package has the automatically generated clientset. |
|
client/clientset/versioned/fake
This package has the automatically generated fake clientset.
|
This package has the automatically generated fake clientset. |
|
client/clientset/versioned/scheme
This package contains the scheme of the automatically generated clientset.
|
This package contains the scheme of the automatically generated clientset. |
|
client/clientset/versioned/typed/gadget/v1alpha1
This package has the automatically generated typed clients.
|
This package has the automatically generated typed clients. |
|
client/clientset/versioned/typed/gadget/v1alpha1/fake
Package fake has the automatically generated clients.
|
Package fake has the automatically generated clients. |
|
columns
Package columns is a library that helps to carry data structs in a more generic way using a combination of reflection and generics.
|
Package columns is a library that helps to carry data structs in a more generic way using a combination of reflection and generics. |
|
columns/ellipsis
Package ellipsis helps to truncate text at a specific width and adding an optional ellipsis ("…") to indicate that the text has been truncated.
|
Package ellipsis helps to truncate text at a specific width and adding an optional ellipsis ("…") to indicate that the text has been truncated. |
|
columns/filter
Package filter helps filtering an array of structs that were analyzed by the columns library.
|
Package filter helps filtering an array of structs that were analyzed by the columns library. |
|
columns/formatter/textcolumns
Package textcolumns helps to output structs (and events of structs) using metadata from a `Columns` instance in a tabular way suitable for consoles or other frontends using fixed-width characters / fonts.
|
Package textcolumns helps to output structs (and events of structs) using metadata from a `Columns` instance in a tabular way suitable for consoles or other frontends using fixed-width characters / fonts. |
|
columns/group
Package group can group the entries of an array by one or more columns.
|
Package group can group the entries of an array by one or more columns. |
|
columns/sort
Package sort can be used to sort an array by their columns in either ascending or descending order.
|
Package sort can be used to sort an array by their columns in either ascending or descending order. |
|
container-collection
Package containercollection provides the ContainerCollection struct to keep track of the set of running containers and primitives to query that set with various criteria.
|
Package containercollection provides the ContainerCollection struct to keep track of the set of running containers and primitives to query that set with various criteria. |
|
container-hook
Package containerhook detects when a container is created or terminated.
|
Package containerhook detects when a container is created or terminated. |
|
environment
Package environment is a temporary workaround to have a simple means of knowing what environment we're running in.
|
Package environment is a temporary workaround to have a simple means of knowing what environment we're running in. |
|
environment/k8s
Package k8s sets the environment to Kubernetes.
|
Package k8s sets the environment to Kubernetes. |
|
environment/local
Package local sets the environment to Local.
|
Package local sets the environment to Local. |
|
gadget-context
Package gadgetcontext handles initializing gadgets and installed operators before handing them over to a specified runtime.
|
Package gadgetcontext handles initializing gadgets and installed operators before handing them over to a specified runtime. |
|
gadget-service/api-helpers
Package apihelpers provides some helper functions for the API package; these were extracted into this package to avoid having additional dependencies on the API package itself
|
Package apihelpers provides some helper functions for the API package; these were extracted into this package to avoid having additional dependencies on the API package itself |
|
histogram
Package histogram provides a Histogram struct that represents a histogram of the number of events that occurred in each interval.
|
Package histogram provides a Histogram struct that represents a histogram of the number of events that occurred in each interval. |
|
kallsyms
Package kallsyms provides functions to resolve kernel symbols.
|
Package kallsyms provides functions to resolve kernel symbols. |
|
kfilefields
Package kfilefields provides functions to read kernel "struct file" fields against a file descriptor.
|
Package kfilefields provides functions to read kernel "struct file" fields against a file descriptor. |
|
logger
Package logger provides a logger interface that is available to gadgets and runtimes.
|
Package logger provides a logger interface that is available to gadgets and runtimes. |
|
networktracer
Package networktracer installs the dispatcher ebpf program in each network namespace of interest.
|
Package networktracer installs the dispatcher ebpf program in each network namespace of interest. |
|
operators/combiner
Package combiner is a data operator that combines data from a same data source coming from different targets into a single data source.
|
Package combiner is a data operator that combines data from a same data source coming from different targets into a single data source. |
|
operators/ebpf
Package ebpfoperator provides an operator that is capable of analyzing and running an eBFP based gadget.
|
Package ebpfoperator provides an operator that is capable of analyzing and running an eBFP based gadget. |
|
operators/ebpf/types
Package types contains the names and Golang representation of the types used by the eBPF programs.
|
Package types contains the names and Golang representation of the types used by the eBPF programs. |
|
operators/kubeipresolver
Package kubeipresolver provides an operator that enriches events by looking up IP addresses in Kubernetes resources such as pods and services.
|
Package kubeipresolver provides an operator that enriches events by looking up IP addresses in Kubernetes resources such as pods and services. |
|
operators/kubenameresolver
Package kubenameresolver provides an operator that enriches events by looking up the pod name and namespace and enriches it with its ip information.
|
Package kubenameresolver provides an operator that enriches events by looking up the pod name and namespace and enriches it with its ip information. |
|
operators/limiter
Package limiter is a data operator that limits the number of entries in each batch of data.
|
Package limiter is a data operator that limits the number of entries in each batch of data. |
|
operators/otel-metrics
Package otelmetrics implements an operator that can export data sources to OpenTelemetry metrics.
|
Package otelmetrics implements an operator that can export data sources to OpenTelemetry metrics. |
|
operators/socketenricher
Package socketenricher creates an eBPF map exposing processes owning each socket.
|
Package socketenricher creates an eBPF map exposing processes owning each socket. |
|
operators/uidgidresolver
Package uidgidresolver provides an operator that enriches events by looking up uid and gid resolving them to the corresponding username and groupname.
|
Package uidgidresolver provides an operator that enriches events by looking up uid and gid resolving them to the corresponding username and groupname. |
|
params
Package params provides a generic way to describe parameters used by gadgets, operators and runtimes including validation.
|
Package params provides a generic way to describe parameters used by gadgets, operators and runtimes including validation. |
|
parser
Package parser wraps a couple of helper libraries with the intention of hiding type information and simplifying data handling outside the gadgets.
|
Package parser wraps a couple of helper libraries with the intention of hiding type information and simplifying data handling outside the gadgets. |
|
tchandler
Package tchandler handles how SchedCLS programs are attached to containers and network interfaces.
|
Package tchandler handles how SchedCLS programs are attached to containers and network interfaces. |
|
testing/command
Package command provides a generic way for running testing commands.
|
Package command provides a generic way for running testing commands. |
|
testing/ig
Package ig provides executable wrapper for ig binary.
|
Package ig provides executable wrapper for ig binary. |
|
testing/match
Package match provides various helper functions for matching actual output to expected output.
|
Package match provides various helper functions for matching actual output to expected output. |
|
uprobetracer
Package uprobetracer handles how uprobe/uretprobe/USDT programs are attached to containers.
|
Package uprobetracer handles how uprobe/uretprobe/USDT programs are attached to containers. |
|
utils/bpf-iter-ns
Package bpfiterns reads a ebpf iterator in a different namespace.
|
Package bpfiterns reads a ebpf iterator in a different namespace. |
|
utils/host
Package host provides ways to access the host filesystem.
|
Package host provides ways to access the host filesystem. |
|
utils/netns
Package netns is a small wrapper around github.com/vishvananda/netns that provides GetFromPidWithAltProcfs() and GetFromThreadWithAltProcfs().
|
Package netns is a small wrapper around github.com/vishvananda/netns that provides GetFromPidWithAltProcfs() and GetFromThreadWithAltProcfs(). |
|
utils/secureopen
Package secureopen provides a way to securely open a file in a container and checking that the path didn't move outside of the container rootfs.
|
Package secureopen provides a way to securely open a file in a container and checking that the path didn't move outside of the container rootfs. |
|
tools
|
|
|
dnstester
Module
|
|
|
testjson2md
Module
|
|
|
wasmapi
|
|
|
go
Package api contains the reference implementation of the wasm API for Inspektor Gadget.
|
Package api contains the reference implementation of the wasm API for Inspektor Gadget. |