gcmmap

README

gcmmap

Package gcmmap provides mmap(2) that can be garbage collected by Go's garbage collector. There is no explicit munmap.

It works by allocating a []byte from the Go allocator, which we can set a finalizer on, and then using mmap with MAP_FIXED to overwrite it with your requested mmap. When the finalizer runs, we undo that and put a normal anonymous read/write mapping back.

USE AT YOUR OWN RISK.

Impact on garbage collection

Contrary to normal unix.Mmap()s, memory mapped with gcmmap is considered to be part of Go's heap. Go's garbage collector by default (GOGC=100) targets to waste no more than another 100% of your heap size. This becomes especially relevant if you have file backed mappings that aren't fully in RSS.

We ran into trouble because we mmapped 20GB, but had only 8GB physical memory. The GC decided it would run the next garbage collection around a heap size of 40GB. The heap kept growing, the page cache kept shrinking and the machine came to a grinding halt. We worked around this by setting GOMEMLIMIT to 22GB, giving the Go heap basically 2GB to work with.

Documentation

Overview

Package gcmmap provides mmap(2) that can be garbage collected by Go's garbage collector. There is no explicit munmap.

It uses unsafe and evil trickery. USE AT YOUR OWN RISK.

Index

• Variables
• func Mmap(fd int, offset int64, len, prot, flags int) ([]byte, error)

Variables

var (
	// NumActive is the number of mmaps that have not yet been garbage collected.
	NumActive atomic.Int32

	// BytesActive is the number of bytes across all mmaps that have not yet been garbage collected.
	BytesActive atomic.Int64
)

Functions

func Mmap

func Mmap(fd int, offset int64, len, prot, flags int) ([]byte, error)

Mmap calls mmap(2) and uses the garbage collector to unmap when no more references exist.