journaldreceiver

README

Journald Receiver

The Journald Receiver parses Journald events from systemd journal.

Status
Stability	alpha: logs
Unsupported Platforms	darwin, windows
Distributions	contrib, k8s
Issues
Code coverage
Code Owners	@belimawr, @namco1992
Emeritus	@djaglowski, @sumo-drosiek

Prerequisites

The journald receiver has specific requirements that must be met:

journalctl Binary Requirement

The journald receiver requires the journalctl binary to be available in the container or on the system. This is because the receiver uses journalctl to read logs from the systemd journal. The receiver does not implement its own journal reading logic and relies on journalctl for this functionality. If running the collector in a container, the journalctl binary should match the host system, and this receiver should be configured with the root_path/journalctl_path options so that the journalctl from the host is used.

Permissions

• The collector's user must have sufficient permissions to access the journal through journalctl
• In containerized environments, this typically requires root privileges and specific capabilities (see Kubernetes Security)

Configuration

Field	Default	Description
directory	/run/log/journal or /run/journal	A directory containing journal files to read entries from. Relative to root_path.
files		A list of journal files to read entries from. Relative to root_path.
start_at	end	At startup, where to start reading logs from the file. Options are beginning or end
units		A list of units to read entries from. See Multiple filtering options examples.
identifiers		Filter output by message identifiers (SYSTEMD_IDENTIFIER). See Multiple filtering options examples.
matches		A list of matches to read entries from. See Matches and Multiple filtering options examples.
priority	info	Filter output by message priorities or priority ranges. See Multiple filtering options examples.
grep		Filter output to entries where the MESSAGE= field matches the specified regular expression. See Multiple filtering options examples.
dmesg	'false'	Show only kernel messages. This shows logs from current boot and adds the match _TRANSPORT=kernel. See Multiple filtering options examples.
storage	none	The ID of a storage extension to be used to store cursors. Cursors allow the receiver to pick up where it left off in the case of a collector restart. If no storage extension is used, the receiver will manage cursors in memory only.
all	'false'	If true, very long logs and logs with unprintable characters will also be included.
namespace		Will query the given namespace. See man page systemd-journald.service(8) for details.
convert_message_bytes	'false'	If true and if the MESSAGE field is read as an array of bytes, the array will be converted to string.
include_log_record_original	'false'	If true, the raw data read from journalctl is added as the 'log.record.original' attribute
merge	'false'	If true, read from all available journals, including remote ones.
retry_on_failure.enabled	false	If true, the receiver will pause reading a file and attempt to resend the current batch of logs if it encounters an error from downstream components.
retry_on_failure.initial_interval	1 second	Time to wait after the first failure before retrying.
retry_on_failure.max_interval	30 seconds	Upper bound on retry backoff interval. Once this value is reached the delay between consecutive retries will remain constant at the specified value.
retry_on_failure.max_elapsed_time	5 minutes	Maximum amount of time (including retries) spent trying to send a logs batch to a downstream consumer. Once this value is reached, the data is discarded. Retrying never stops if set to 0.
root_path		Chroot to use when executing the journalctl command. Must be an absolute path or empty. When empty (default), no chroot is used when executing journalctl.
journalctl_path	journalctl	journalctl command to execute. Relative to root_path. Must be an absolute path if root_path is non-empty. See below for more details
operators	[]	An array of operators. See below for more details

Operators

Each operator performs a simple responsibility, such as parsing a timestamp or JSON. Chain together operators to process logs into a desired format.

• Every operator has a type.
• Every operator can be given a unique id. If you use the same type of operator more than once in a pipeline, you must specify an id. Otherwise, the id defaults to the value of type.
• Operators will output to the next operator in the pipeline. The last operator in the pipeline will emit from the receiver. Optionally, the output parameter can be used to specify the id of another operator to which logs will be passed directly.
• Only parsers and general purpose operators should be used.

Example Configurations

Minimal configuration

The following configuration is the minimal configuration to read journald logs:

receivers:
  journald:

will be passed to journalctl as the following arguments: journalctl ... --priority info. This will read the 10 most recent entries and any subsequent entry. --priority info is the default priority, the following examples will omit it for simplicity.

Cursor tracking

receivers:
  journald:
    storage: file_storage/journald

extensions:
  file_storage/journald:
    directory: .

service:
  extensions: [file_storage/journald]

If you stop and start the otel collector, only new entries will be read.

Reading from the beginning

receivers:
  journald:
    start_at: beginning

will be passed to journalctl as the following arguments: journalctl ... --no-tail. This will read all messages from the current boot.

Units

receivers:
  journald:
    directory: /run/log/journal
    units:
      - ssh
      - kubelet
      - docker
      - containerd
    priority: info

Matches

The following configuration:

- type: journald_input
  matches:
    - _SYSTEMD_UNIT: ssh
    - _SYSTEMD_UNIT: kubelet
      _UID: "1000"

will be passed to journalctl as the following arguments: journalctl ... _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000, which is going to retrieve all entries which match at least one of the following rules:

• _SYSTEMD_UNIT is ssh
• _SYSTEMD_UNIT is kubelet and _UID is 1000

Multiple filtering options

In case of using multiple following options, conditions between them are logically ANDed and within them are logically ORed:

( dmesg )
AND
( priority )
AND
( units[0] OR units[1] OR units[2] OR ... units[U] )
AND
( identifier[0] OR identifier[1] OR identifier[2] OR ... identifier[I] )
AND
( matches[0] OR matches[1] OR matches[2] OR ... matches[M] )
AND
( grep )

Consider the following example:

- type: journald_input
  matches:
    - _SYSTEMD_UNIT: ssh
    - _SYSTEMD_UNIT: kubelet
      _UID: "1000"
  units:
    - kubelet
    - systemd
  priority: info
  identifiers:
    - systemd

The above configuration will be passed to journalctl as the following arguments journalctl ... --priority=info --unit=kubelet --unit=systemd --identifier=systemd _SYSTEMD_UNIT=ssh + _SYSTEMD_UNIT=kubelet _UID=1000, which is going to effectively retrieve all entries which matches the following set of rules:

• _PRIORITY is 6, and

• _SYSTEMD_UNIT is kubelet or systemd, and

• SYSLOG_IDENTIFIER systemd, and

• entry matches at least one of the following rules:

  • _SYSTEMD_UNIT is ssh
  • _SYSTEMD_UNIT is kubelet and _UID is 1000

Performance Considerations

start_at parameter

The start_at configuration parameter has significant performance implications:

• end (default - recommended for production): Reads only new entries after collector start
  • Minimal startup impact
  • No historical log replay
• beginning: Reads entire journal history (potentially GBs of logs)
  • Use only for testing or initial backfill
  • Can cause 10-60 second startup delay
  • High CPU/memory usage during catchup

Unit Filtering Impact

• Unfiltered: Reads all systemd units
• Filtered: Reads only specified units

Setup and Deployment

Journal Location in Container Environments

Journal file locations differ between traditional systems and containerized environments:

Traditional Systems:

• Persistent journal: /var/log/journal/
• Requires Storage=persistent in journald.conf

Container Platforms (kind, minikube, k3s, most Kubernetes clusters):

• Volatile journal: /run/log/journal/
• Default Storage=volatile or Storage=auto with missing /var/log/journal

Discovery Command:

# Find journal location
journalctl --header | grep "File path"

Docker & Kubernetes

When running otelcol in a container, you need:

1. Root permissions (journal files are root-owned)
2. The journal directory mounted (usually /run/log/journal in containerized environments)
3. The journalctl binary available

Option 1: Bundle journalctl in your image (Recommended)

Build a collector image that includes journalctl. See the example Dockerfile in examples/container with step-by-step instructions for Kubernetes deployments.

This works well if your hosts run similar systemd versions.

Option 2: Use host's journalctl via chroot

Some collector containers need to work across a variety of arbitrary hosts that may be running mutually incompatible versions of journalctl, making it difficult to select a single version of journalctl to bundle into the container.

One way to solve this problem is to ensure that the collector running in the container invokes the exact same journalctl that was used to write the logs in the first place. To achieve this, you can mount the host's rootfs to the container and then configure the receiver to run the host's journalctl in a chroot for that mount.

You can pass -v /:/host to docker run to mount the host's rootfs:

docker run -v /:/host otel/opentelemetry-collector-contrib

Then, you can configure the receiver with root_path to use that mount as a chroot for journalctl. Due to a Go issue, running executables from $PATH does not work well in a chroot, so you must also use journalctl_path to configure a full path to journalctl inside of the chroot:

receivers:
  journald:
    root_path: /host
    journalctl_path: /usr/bin/journalctl

You also need to ensure the user running the collector can run the journalctl from the chroot, one way to do this is to run the collector as root:

docker run -v /:/host --user 0 otel/opentelemetry-collector-contrib

Linux packaging

When installing otelcol as a linux package, you will most likely need to add the otelcol-contrib or otel user to the systemd-journal group. The exact user and group might vary depending on your package and linux distribution of choice.

You can test if the user has sufficient permissions by running something like (you might need to adjust according to available shell and opentelemetry user)

sudo su -s /bin/bash -c 'journalctl --lines 5' otelcol-contrib

if the permissions are set correctly you will see some logs, otherwise a clear error message.

Troubleshooting

Kubernetes Security Requirements

Minimum Requirements:

• Run as root (runAsUser: 0) - journal files are root-owned
• Read access to /run/log/journal (requires root permission level)
• Required capabilities:
  • CAP_DAC_READ_SEARCH: Read any file regardless of permissions
  • CAP_SYS_PTRACE: Required by journalctl for some operations

Recommended security context:

securityContext:
  runAsUser: 0
  capabilities:
    add:
      - DAC_READ_SEARCH
      - SYS_PTRACE

Common Errors and Solutions

Error Message	Cause	Solution
journalctl: executable file not found	systemd not installed in container	Install systemd package in your container image
Journal file uses an unsupported feature	systemd version mismatch	Use Ubuntu 24.04, Debian 12, or newer systemd version
permission denied reading journal	Insufficient permissions	Add CAP_DAC_READ_SEARCH capability and run as root
directory "/var/log/journal" is empty	Wrong journal path for container environment	Change to /run/log/journal (see Journal Location)
No logs appearing	Various causes	See diagnostic commands below
Failed to get journal file list	Incorrect mount or permissions	Verify volume mount and security context

Diagnostic Commands

# 1. Check journal location on the host
ls -la /var/log/journal /run/log/journal
journalctl --header | grep "File path"

# 2. Verify journalctl works inside the container
kubectl exec -it <pod> -- journalctl -n 20

# 3. Check journalctl version
kubectl exec -it <pod> -- journalctl --version

# 4. Test reading from mounted directory
kubectl exec -it <pod> -- journalctl -D /run/log/journal --priority=info -n 10

# 5. Check collector logs for errors
kubectl logs <pod> | grep -i "journald\|error"

# 6. Verify volume mounts
kubectl exec -it <pod> -- ls -la /run/log/journal

# 7. Check pod security context
kubectl get pod <pod> -o jsonpath='{.spec.containers[0].securityContext}'

# 8. Verify capabilities (from within pod)
kubectl exec -it <pod> -- grep Cap /proc/self/status

Documentation

Index

• func NewFactory() receiver.Factory
• type JournaldConfig

Functions

func NewFactory

func NewFactory() receiver.Factory

NewFactory creates a factory for journald receiver

Types

type JournaldConfig

type JournaldConfig struct {
	BaseConfig  adapter.BaseConfig `mapstructure:",squash"`
	InputConfig journald.Config    `mapstructure:",squash"`
	// contains filtered or unexported fields
}

JournaldConfig defines configuration for the journald receiver