softstat

README

Report file descriptors, processes count vs limits

Example usage

Make sure you run this utility under root (it needs access to /proc).

# Build binary
go build softstat.go

# Alternatively, download already compiled version
wget https://github.com/mihgen/softstat/releases/download/0.2/softstat
chmod +x softstat

# Show 10 mostly loaded processes
./softstat

# Show 1000 mostly loaded processes
./softstat -n 1000

# Show all
./softstat -1

Example output

./softstat
  PID FD FD-max FD% Proc Proc-Max Proc%         CMD
    1 52  65536 0.1  126      500  25.2     systemd
 6615  4   1024 0.4  126      500  25.2    softstat
 5379  4   1024 0.4  126      500  25.2        bash
 5378  4   1024 0.4  126      500  25.2          su
 5234  4   1024 0.4  126      500  25.2        bash
 5233  4   1024 0.4  126      500  25.2          su
 5178  5   1024 0.5  126      500  25.2        sshd
 6597  3   1024 0.3  126      500  25.2          go
  132  0   1024 0.0  126     5843   2.2      bioset
    9  0   1024 0.0  126     5843   2.2 migration/0

How it works

This utility will go over all processes in /proc, get their cmd name from /proc/<pid>/status, and count number of open file descriptors from /proc/<pid>/fd/. prlimit64() system call is used to get limits set for the process, and SoftLimit value is used in calculations.

Use prlimit Linux utility if you need to get or change limits of running process. Alternatively, /proc/<pid>/limits can be used to get current limits.

There are a few ceilings process count can hit. Currently, only these two are used:

1. /proc/sys/kernel/threads-max is maximum task count (processes and threads) you may have in a system
2. /proc/<pid>/limits, Max processes, Soft Limit. (Hard limit is not in use by kernel and ignored). This value is checked when this given PID tries to create a new process (or thread). However, this limit is NOT how much threads or children given PID can have; this limit is total number of processes ran under real UID which PID rans under. For example, there are 10 processes running under user "buddy". One of process is apache2, and we set a limit for this process to 10 using prlimit. After that, no new forks could be created by apache2, as we are pressed by a limit for this process. At the same time, any other process running with the same UID can create new processes or threads.

In the output of softstat, only one set of processes count / limit / % is given, the one with higher usage.

Limitations

Note, that currently softstat only works on x86_64 Linux. Limits related to processes count are not yet calculated:

• cgroups
• pid_max - to check if we don't exhaust PID namespace

Documentation

Index

• func CmdName(pid string) (string, error)
• func FileNr() (used, max uint64)
• func FilePerProcMax() uint64
• func PidTotalLimit() uint64
• func Prlimit(pid int, resource int, new_rlim *syscall.Rlimit, old_rlim *syscall.Rlimit) (err error)
• func ProcTotalLimit() uint64
• func ReadAndTrim(file string) (string, error)
• type Entry
• type Limits
  • func GetLimits(pid string) (Limits, error)
• type Task
  • func (t Task) FdsRlim() (Entry, error)
  • func (t Task) NprocRlim() (Entry, error)
• type Tasks
  • func TasksInit() *Tasks

Functions

func CmdName

func CmdName(pid string) (string, error)

func FileNr

func FileNr() (used, max uint64)

func FilePerProcMax

func FilePerProcMax() uint64

func PidTotalLimit

func PidTotalLimit() uint64

func Prlimit

func Prlimit(pid int, resource int, new_rlim *syscall.Rlimit, old_rlim *syscall.Rlimit) (err error)

func ProcTotalLimit

func ProcTotalLimit() uint64

func ReadAndTrim

func ReadAndTrim(file string) (string, error)

Types

type Entry

type Entry struct {
	V   uint64
	Max uint64
}

type Limits

type Limits struct {
	// contains filtered or unexported fields
}

func GetLimits

func GetLimits(pid string) (Limits, error)

type Task

type Task struct {
	Pid    string
	UidMap map[string]uint64
	// contains filtered or unexported fields
}

func (Task) FdsRlim

func (t Task) FdsRlim() (Entry, error)

func (Task) NprocRlim

func (t Task) NprocRlim() (Entry, error)

type Tasks

type Tasks struct {
	Pids  []Task
	Total uint64
}

func TasksInit

func TasksInit() *Tasks