gommap

README

gommap

This is a git mirror of launchpad.net/gommap. The master branch includes this patch, which adds support for darwin64 (MacOS X), as well as Windows support by @matt-farmer.

Read more

API documentation

Notes on Windows Support by @matt-farmer

Gommap is a dependency within the liftbridge streaming engine which we are using, and we need to be able to run the streaming server on windows.

The Gommap windows implementation was done by Qing Miao from the NSIP team, but we drew heavily on work from others which are referenced below to make sure they get the credit they deserve - it's also ended up being a bit of a sample of known golang memory maps that work cross-platform, so putting it here in case others find it useful.

The state of many implementations of the mem-maps is unclear in terms of support, how active the repos are etc. Of course this is a low-level piece of any system so probably once people have one working they don't do any changes until an OS or processor architecture change forces the need.

We created this fork really because we're prepared to support the implementation for now, and we're passing the changes back up to the original repo so they can accept them if they want to.

The windows version is designed to require no changes to existing code; certainly this is the case for our liftbridge requirement, the liftbridge server now builds and runs fine on windows, and it's been subjected to high volume tests that would force the flushing of the map without incident.

Limitations

1. We have only tested this on 64-bit Windows
2. We have not been able to implement the (mmap)Advise() or (mmap)IsResident() functions - later versions of windows may have apis that can help to support these (from the documentation we've been able to find), but go can only distinguish os and architecture and those 'advanced' features are not generically available to 'windows'. Please raise an issue if this is a show-stopper.

Prior Art

Here’s a list of the alternative go mem-map packages that we looked at for help and/or borrowed from directly, and which anyone else may find helpful if you need cross-platform memory-mapping in go:

https://github.com/edsrzf/mmap-go This one has a very similar interface to the original labix library, but does seem to provide support for windows and various linux distros and Mac.

https://github.com/golang/exp/tree/master/mmap Package from the golang devs for mmap. Not immediately useful for our requirement as it only offers a reader interface, but does have nice use of the syscall package in the windows golang file that might be useful if we end up having to create our own library completely from scratch.

https://github.com/justmao945/tama/tree/master/mmap This one uses a full go File paradigm rather than a byte array.

https://github.com/influxdata/platform/tree/master/pkg/mmap This one is used by influxdb, so we know it works on multiple platforms (we also use influxdb as part of the same project). Difference here is that the API is much simpler, just open - returns a byte array, and then close!

https://github.com/go-util/mmap An interesting one that is actively developed. Uses a mixture of file/byte array paradigms, so may operate on Windows using a file-based approach, with lower-level calls for unix systems; offers reader and writer interfaces if required.

https://github.com/AlexStocks/goext/tree/master/syscall Another active repo, with a mmap for unix and windows, offers the simple interface for byte array which should be compatible with the simple calls used by liftbridge.

Documentation

Overview

This package offers the MMap type that manipulates a memory mapped file or device.

IMPORTANT NOTE (1): The MMap type is backed by an unsafe memory region, which is not covered by the normal rules of Go's memory management. If a slice is taken out of it, and then the memory is explicitly unmapped through one of the available methods, both the MMap value itself and the slice obtained will now silently point to invalid memory. Attempting to access data in them will crash the application.

Index

• type AdviseFlags
• type MMap
  • func Map(fd uintptr, prot ProtFlags, flags MapFlags) (MMap, error)
  • func MapAt(addr uintptr, fd uintptr, offset, length int64, prot ProtFlags, flags MapFlags) (MMap, error)
  • func MapRegion(fd uintptr, offset, length int64, prot ProtFlags, flags MapFlags) (MMap, error)
  • func (mmap MMap) Advise(advice AdviseFlags) error
  • func (mmap MMap) IsResident() ([]bool, error)
  • func (mmap MMap) Lock() error
  • func (mmap MMap) Protect(prot ProtFlags) error
  • func (mmap MMap) Sync(flags SyncFlags) error
  • func (mmap MMap) Unlock() error
  • func (mmap MMap) UnsafeUnmap() error
• type MapFlags
• type ProtFlags
• type SyncFlags

Types

type AdviseFlags

type AdviseFlags uint

const (
	MADV_NORMAL     AdviseFlags = 0x0
	MADV_RANDOM     AdviseFlags = 0x1
	MADV_SEQUENTIAL AdviseFlags = 0x2
	MADV_WILLNEED   AdviseFlags = 0x3
	MADV_DONTNEED   AdviseFlags = 0x4
	MADV_REMOVE     AdviseFlags = 0x9
	MADV_DONTFORK   AdviseFlags = 0xa
	MADV_DOFORK     AdviseFlags = 0xb
)

type MMap

type MMap []byte

The MMap type represents a memory mapped file or device. The slice offers direct access to the memory mapped content.

IMPORTANT: Please see note in the package documentation regarding the way in which this type behaves.

func Map

func Map(fd uintptr, prot ProtFlags, flags MapFlags) (MMap, error)

Map creates a new mapping in the virtual address space of the calling process. This function will attempt to map the entire file by using the fstat system call with the provided file descriptor to discover its length.

func MapAt

func MapAt(addr uintptr, fd uintptr, offset, length int64, prot ProtFlags, flags MapFlags) (MMap, error)

MapAt creates a new mapping in the virtual address space of the calling process, using the specified region of the provided file or device. The provided addr parameter will be used as a hint of the address where the kernel should position the memory mapped region. If -1 is provided as length, this function will attempt to map until the end of the provided file descriptor by using the fstat system call to discover its length.

func MapRegion

func MapRegion(fd uintptr, offset, length int64, prot ProtFlags, flags MapFlags) (MMap, error)

MapRegion creates a new mapping in the virtual address space of the calling process, using the specified region of the provided file or device. If -1 is provided as length, this function will attempt to map until the end of the provided file descriptor by using the fstat system call to discover its length.

func (MMap) Advise

func (mmap MMap) Advise(advice AdviseFlags) error

Advise advises the kernel about how to handle the mapped memory region in terms of input/output paging within the memory region defined by the mmap slice.

func (MMap) IsResident

func (mmap MMap) IsResident() ([]bool, error)

IsResident returns a slice of booleans informing whether the respective memory page in mmap was mapped at the time the call was made.

func (MMap) Lock

func (mmap MMap) Lock() error

Lock locks the mapped region defined by the mmap slice, preventing it from being swapped out.

func (MMap) Protect

func (mmap MMap) Protect(prot ProtFlags) error

Protect changes the protection flags for the memory mapped region defined by the mmap slice.

func (MMap) Sync

func (mmap MMap) Sync(flags SyncFlags) error

Sync flushes changes made to the region determined by the mmap slice back to the device. Without calling this method, there are no guarantees that changes will be flushed back before the region is unmapped. The flags parameter specifies whether flushing should be done synchronously (before the method returns) with MS_SYNC, or asynchronously (flushing is just scheduled) with MS_ASYNC.

func (MMap) Unlock

func (mmap MMap) Unlock() error

Unlock unlocks the mapped region defined by the mmap slice, allowing it to swap out again.

func (MMap) UnsafeUnmap

func (mmap MMap) UnsafeUnmap() error

UnsafeUnmap deletes the memory mapped region defined by the mmap slice. This will also flush any remaining changes, if necessary. Using mmap or any other slices based on it after this method has been called will crash the application.

type MapFlags

type MapFlags uint

const (
	MAP_SHARED    MapFlags = 0x1
	MAP_PRIVATE   MapFlags = 0x2
	MAP_FIXED     MapFlags = 0x10
	MAP_ANONYMOUS MapFlags = 0x20
	MAP_GROWSDOWN MapFlags = 0x100
	MAP_LOCKED    MapFlags = 0x2000
	MAP_NONBLOCK  MapFlags = 0x10000
	MAP_NORESERVE MapFlags = 0x4000
	MAP_POPULATE  MapFlags = 0x8000
)

type ProtFlags

type ProtFlags uint

const (
	PROT_NONE  ProtFlags = 0x0
	PROT_READ  ProtFlags = 0x1
	PROT_WRITE ProtFlags = 0x2
	PROT_EXEC  ProtFlags = 0x4
)

type SyncFlags

type SyncFlags uint

const (
	MS_SYNC       SyncFlags = 0x4
	MS_ASYNC      SyncFlags = 0x1
	MS_INVALIDATE SyncFlags = 0x2
)