unused

README

CLI tool, Prometheus exporter, and Go module to list your unused disks in all cloud providers

This repository contains a Go library to list your unused persistent disks in different cloud providers, and binaries for displaying them at the CLI or exporting Prometheus metrics.

At Grafana Labs we host our workloads in multiple cloud providers. Our workloads orchestration is managed by Kubernetes, and we've found that due to some misconfiguration in the backend storage, we used to have lots of unused resources, specially persistent disks. These leaked resources cost money, and because these are resources that are not in use anymore, it translates into wasted money. This library and its companion tools should help you out to identify these resources and clean them up.

Go module github.com/grafana/unused

This module exports some interfaces and implementations to easily list all your unsed persistent disks in GCP, AWS, and Azure. You can find the API in the Go package documentation.

Binaries

This repository also provides two binaries ready to use to interactively view unused disks (cmd/unused) or expose unused disk metrics (cmd/unused-exporter) to Prometheus. Both programs can authenticate against the following providers using the listed CLI flags:

Provider	Flag	Description
GCP	-gcp.project	ID of the GCP project
AWS	-aws.profile	AWS configuration profile name
Azure	-azure.sub	Azure subscription ID

These flags can be specified more than once, allowing to have different configurations for each provider.

Notes on authentication

Both binaries are opinionated on how to authenticate against each Cloud Service Provider (CSP).

Provider	Notes
GCP	Depends on default credentials
AWS	Uses profile names from your credentials file or AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_REGION env variables
Azure	Either specify an AZURE_CLIENT_ID, AZURE_CLIENT_SECRET, and AZURE_TENANT_ID, or requires Azure CLI installed on the host and signed in

unused binary

TUI tool to query all given providers and list them as a neat table. It also supports an interactive mode which allows to select and delete disks in an easy way.

go install github.com/grafana/unused/cmd/unused@latest

unused-exporter Prometheus exporter

Web server exposing Prometheus metrics about each providers count of unused disks. It exposes the following metrics:

Metric	Description
unused_disks_count	How many unused disks are in this provider
unused_disks_total_size_bytes	Total size of unused disks in this provider in bytes
unused_disk_size_bytes	Size of each disk in bytes
unused_disks_last_used_timestamp_seconds	Last timestamp (unix seconds) when this disk was used. GCP only!
unused_provider_duration_seconds	How long in seconds took to fetch this provider information
unused_provider_info	CSP information
unused_provider_success	Static metric indicating if collecting the metrics succeeded or not

All metrics have the provider and provider_id labels to identify to which provider instance they belong. The unused_disks_count, unused_disk_size_bytes, and unused_disks_total_size_bytes metrics have an additional k8s_namespace metric mapped to the kubernetes.io/created-for/pvc/namespace annotation assigned to persistent disks created by Kubernetes.

Information about each unused disk is currently logged to stdout given that it contains more changing information that could lead to cardinality explosion.

go install github.com/grafana/unused/cmd/unused-exporter@latest

Documentation

Overview

Package unused provides the basic interfaces to obtain information and in some cases, manipulate, unused resources in different Cloud Service Providers (CSPs).

Currently only unused disks are supported.

The following providers are already implemented:

• Google Cloud Platform (GCP)
• Amazon Web Services (AWS)
• Azure

Index

• Constants
• func ByCreatedAt(p, q Disk) bool
• func ByName(p, q Disk) bool
• func ByProvider(p, q Disk) bool
• type ByFunc
• type Disk
• type DiskType
• type Disks
  • func (d Disks) Sort(by ByFunc)
• type Meta
  • func (m Meta) CreatedForNamespace() string
  • func (m Meta) CreatedForPV() string
  • func (m Meta) CreatedForPVC() string
  • func (m Meta) Equals(b Meta) bool
  • func (m Meta) Keys() []string
  • func (m Meta) Matches(key, val string) bool
  • func (m Meta) String() string
  • func (m Meta) Zone() string
• type Provider

Constants

const GiBbytes = 1_073_741_824 // 2^30

Functions

func ByCreatedAt

func ByCreatedAt(p, q Disk) bool

ByCreatedAt sorts a Disks collection by disk creation time.

func ByName

func ByName(p, q Disk) bool

ByName sorts a Disks collection by disk name.

func ByProvider

func ByProvider(p, q Disk) bool

ByProvider sorts a Disks collection by provider name.

Types

type ByFunc

type ByFunc func(p, q Disk) bool

ByFunc is the type of sorting functions for Disks.

type Disk

type Disk interface {
	// ID should return a unique ID for a disk within each cloud
	// provider.
	ID() string

	// Provider returns a reference to the provider used to instantiate
	// this disk
	Provider() Provider

	// Name returns the disk name.
	Name() string

	// SizeGB returns the disk size in GB (Azure/GCP) and GiB for AWS.
	SizeGB() int

	// SizeBytes returns the disk size in bytes.
	SizeBytes() float64

	// CreatedAt returns the time when the disk was created.
	CreatedAt() time.Time

	// LastUsedAt returns the date when the disk was last used.
	LastUsedAt() time.Time

	// Meta returns the disk metadata.
	Meta() Meta

	// DiskType returns the normalized type of disk.
	DiskType() DiskType
}

Disk represents an unused disk on a given cloud provider.

type DiskType

type DiskType string

const (
	SSD     DiskType = "ssd"
	HDD     DiskType = "hdd"
	Unknown DiskType = "unknown"
)

type Disks

type Disks []Disk

Disks is a collection of Disk.

func (Disks) Sort

func (d Disks) Sort(by ByFunc)

Sort sorts the collection by the given function.

type Meta

type Meta map[string]string

Meta is a map of key/value pairs.

func (Meta) CreatedForNamespace

func (m Meta) CreatedForNamespace() string

func (Meta) CreatedForPV

func (m Meta) CreatedForPV() string

func (Meta) CreatedForPVC

func (m Meta) CreatedForPVC() string

func (Meta) Equals

func (m Meta) Equals(b Meta) bool

func (Meta) Keys

func (m Meta) Keys() []string

Keys returns all the keys in the map sorted alphabetically.

func (Meta) Matches

func (m Meta) Matches(key, val string) bool

Matches returns true when the given key exists in the map with the given value.

func (Meta) String

func (m Meta) String() string

String returns a string representation of metadata.

func (Meta) Zone

func (m Meta) Zone() string

type Provider

type Provider interface {
	// Name returns the provider name.
	Name() string

	// ID returns the project (GCP) or profile (AWS) or subscription (Azure) for this provider.
	ID() string

	// ListUnusedDisks returns a list of unused disks for the given
	// provider.
	ListUnusedDisks(ctx context.Context) (Disks, error)

	// Meta returns the provider metadata.
	Meta() Meta

	// Delete deletes a disk from the provider.
	Delete(ctx context.Context, disk Disk) error
}

Provider represents a cloud provider.