diskfs

README

go-diskfs

go-diskfs is a go library for performing manipulation of disks, disk images and filesystems natively in go.

You can do nearly everything that go-diskfs provides using shell tools like gdisk/fdisk/mkfs.vfat/mtools/sgdisk/sfdisk/dd. However, these have the following limitations:

• they need to be installed on your system
• you need to fork/exec to the command (and possibly a shell) to run them
• some are difficult to run without mounting disks, which may not be possible or may be risky in your environment, and almost certainly will require root privileges
• you do not want to launch a VM to run the excellent libguestfs and it may not be installed

go-diskfs performs all modifications natively in go, without mounting any disks.

Usage

Note: detailed go documentation is available at godoc.org.

Concepts

go-diskfs has a few basic concepts:

• Backend
• Disk
• Partition
• Filesystem

Backend

Backend is a (relatively) thin layer which abstracts low-level read/write operations. Through a backend you can seamlessly operate different disk formats.

Currently there is only one implementation - file.

Use file backend to access block devices and raw image files.

Disk

A disk represents either a file or block device that you access and manipulate. With access to the disk, you can:

• read, modify or create a partition table
• open an existing or create a new filesystem

Partition

A partition is a slice of a disk, beginning at one point and ending at a later one. You can have multiple partitions on a disk, and a partition table that describes how partitions are laid out on the disk.

Filesystem

A filesystem is a construct that gives you access to create, read and write directories and files.

You do not need a partitioned disk to work with a filesystem; filesystems can be an entire disk, just as they can be an entire block device. However, they also can be in a partition in a disk

Working With a Disk

Before you can do anything with a disk - partitions or filesystems - you need to access it.

• If you have an existing disk or image file, you Open() it
• If you are creating a new one, usually just disk image files, you Create() it

The disk will be opened read-write, with exclusive access. If it cannot do either, it will fail.

Once you have a Disk, you can work with partitions or filesystems in it.

Partitions on a Disk

The following are the partition actions you can take on a disk:

• GetPartitionTable() - if one exists. Will report the table layout and type.
• Partition() - partition the disk, overwriting any previous table if it exists

As of this writing, supported partition formats are Master Boot Record (mbr) and GUID Partition Table (gpt).

Filesystems on a Disk

Once you have a valid disk, and optionally partition, you can access filesystems on that disk image or partition.

• CreateFilesystem() - create a filesystem in an individual partition or the entire disk
• GetFilesystem() - access an existing filesystem in a partition or the entire disk

As of this writing, supported filesystems include FAT32 and ISO9660 (a.k.a. .iso).

With a filesystem in hand, you can create, access and modify directories and files.

• Mkdir() - make a directory in a filesystem
• Readdir() - read all of the entries in a directory
• OpenFile() - open a file for read, optionally write, create and append

Note that OpenFile() is intended to match os.OpenFile and returns a godiskfs.File that closely matches os.File

With a File in hand, you then can:

• Write(p []byte) to the file
• Read(b []byte) from the file
• Seek(offset int64, whence int) to set the next read or write to an offset in the file

Read-Only Filesystems

Some filesystem types are intended to be created once, after which they are read-only, for example ISO9660/.iso and squashfs.

godiskfs recognizes read-only filesystems and limits working with them to the following:

• You can GetFilesystem() a read-only filesystem and do all read activities, but cannot write to them. Any attempt to Mkdir() or OpenFile() in write/append/create modes or Write() to the file will result in an error.
• You can CreateFilesystem() a read-only filesystem and write anything to it that you want. It will do all of its work in a "scratch" area, or temporary "workspace" directory on your local filesystem. When you are ready to complete it, you call Finalize(), after which it becomes read-only. If you forget to Finalize() it, you get... nothing. The Finalize() function exists only on read-only filesystems.

Example

There are examples in the examples/ directory. See for example how to create a fully bootable EFI disk image.

Tests

There are two ways to run tests: unit and integration (somewhat loosely defined).

• Unit: these tests run entirely within the go process, primarily test unexported and some exported functions, and may use pre-defined test fixtures in a directory's testdata/ subdirectory. By default, these are run by running go test ./... or just make unit_test.
• Integration: these test the exported functions and their ability to create or manipulate correct files. They are validated by running a docker container with the right utilities to validate the output. These are run by running TEST_IMAGE=diskfs/godiskfs go test ./... or just make test. The value of TEST_IMAGE will be the image to use to run tests.

For integration tests to work, the correct docker image must be available. You can create it by running make image. Check the Makefile to see the docker build command used to create it. Running make test automatically creates the image for you.

Integration Test Image

The integration test image contains the various tools necessary to test images: mtools, fdisk, gdisk, etc. It works on precisely one file at a time. In order to avoid docker volume mounting limitations with various OSes, instead of mounting the image -v, it expects to receive the image as a stdin stream, and saves it internally to the container as /file.img.

For example, to test the existence of directory /abc on file $PWD/foo.img:

cat $PWD/foo.img | docker run -i --rm $INT_IMAGE mdir -i /file.img /abc

Plans

Future plans are to add the following:

• embed boot code in mbr e.g. altmbr.bin (no need for gpt since an ESP with /EFI/BOOT/BOOT<arch>.EFI will boot)
• Joliet extensions to iso9660
• Rock Ridge sparse file support - supports the flag, but not yet reading or writing
• squashfs sparse file support - currently treats sparse files as regular files
• qcow disk format

Documentation

Overview

Package diskfs implements methods for creating and manipulating disks and filesystems

methods for creating and manipulating disks and filesystems, whether block devices in /dev or direct disk images. This does **not** mount any disks or filesystems, neither directly locally nor via a VM. Instead, it manipulates the bytes directly.

This is not intended as a replacement for operating system filesystem and disk drivers. Instead, it is intended to make it easy to work with partitions, partition tables and filesystems directly without requiring operating system mounts.

Index

• func Create(device string, size int64, sectorSize SectorSize) (*disk.Disk, error)
• func Open(device string, opts ...OpenOpt) (*disk.Disk, error)
• func OpenBackend(b backend.Storage, opts ...OpenOpt) (*disk.Disk, error)
• type OpenModeOption
  • func (m OpenModeOption) String() string
• type OpenOpt
  • func WithOpenMode(mode OpenModeOption) OpenOpt
  • func WithSectorSize(sectorSize SectorSize) OpenOpt
• type SectorSize

Examples

• Create (Fat32)
• Create (Fat32WithDirsAndFiles)
• Create (Gpt)
• Create (Mbr)
• OpenBackend (Backend)
• OpenBackend (Create)
• OpenBackend (Open)

Functions

func Create

func Create(device string, size int64, sectorSize SectorSize) (*disk.Disk, error)

Might be deprecated in future: use <backend>.CreateFromPath + diskfs.OpenBackend Create a Disk from a path to a device Should pass a path to a block device e.g. /dev/sda or a path to a file /tmp/foo.img The provided device must not exist at the time you call Create()

Example (Fat32)

Create a disk image of size 10MB with a FAT32 filesystem spanning the entire disk.

package main

import (
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/disk"
	"github.com/diskfs/go-diskfs/filesystem"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	var size int64 = 10 * 1024 * 1024 // 10 MB

	diskImg := "/tmp/disk.img"
	defer os.Remove(diskImg)
	theDisk, _ := diskfs.Create(diskImg, size, diskfs.SectorSizeDefault)

	fs, err := theDisk.CreateFilesystem(disk.FilesystemSpec{
		Partition: 0,
		FSType:    filesystem.TypeFat32,
	})
	check(err)
	unused(fs)
}

Example (Fat32WithDirsAndFiles)

Create a disk of size 20MB with an MBR partition table, a single partition beginning at block 2048 (1MB), of size 10MB filled with a FAT32 filesystem, and create some directories and files in that filesystem.

package main

import (
	"crypto/rand"
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/disk"
	"github.com/diskfs/go-diskfs/filesystem"
	"github.com/diskfs/go-diskfs/partition/mbr"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	var size int64 = 20 * 1024 * 1024 // 20 MB

	diskImg := "/tmp/disk.img"
	defer os.Remove(diskImg)
	theDisk, _ := diskfs.Create(diskImg, size, diskfs.SectorSizeDefault)

	table := &mbr.Table{
		LogicalSectorSize:  512,
		PhysicalSectorSize: 512,
		Partitions: []*mbr.Partition{
			{
				Bootable: false,
				Type:     mbr.Linux,
				Start:    2048,
				Size:     20480,
			},
		},
	}

	check(theDisk.Partition(table))

	fs, err := theDisk.CreateFilesystem(disk.FilesystemSpec{
		Partition: 1,
		FSType:    filesystem.TypeFat32,
	})
	check(err)

	err = fs.Mkdir("/FOO/BAR")
	check(err)

	rw, err := fs.OpenFile("/FOO/BAR/AFILE.EXE", os.O_CREATE|os.O_RDWR)
	check(err)
	b := make([]byte, 1024)

	_, err = rand.Read(b)
	check(err)

	written, err := rw.Write(b)
	check(err)
	unused(written)
}

Example (Gpt)

Create a disk of size 20MB with a GPT partition table, a single partition beginning at block 2048 (1MB), of size 10MB, and fill with the contents from the 10MB file "/root/contents.dat"

package main

import (
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/partition/gpt"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	var size int64 = 20 * 1024 * 1024 // 20 MB

	diskImg := "/tmp/disk.img"
	defer os.Remove(diskImg)
	theDisk, _ := diskfs.Create(diskImg, size, diskfs.SectorSizeDefault)

	table := &gpt.Table{
		LogicalSectorSize:  512,
		PhysicalSectorSize: 512,
		ProtectiveMBR:      true,
		Partitions: []*gpt.Partition{
			{
				Start: 1 * 1024 * 1024 / 512,
				Size:  10 * 1024 * 1024,
			},
		},
	}

	check(theDisk.Partition(table))

	f, err := os.Open("/root/contents.dat")
	check(err)

	written, err := theDisk.WritePartitionContents(1, f)
	check(err)
	unused(written)
}

Example (Mbr)

Create a disk of size 20MB with an MBR partition table, a single partition beginning at block 2048 (1MB), of size 10MB filled with a FAT32 filesystem.

package main

import (
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/disk"
	"github.com/diskfs/go-diskfs/filesystem"
	"github.com/diskfs/go-diskfs/partition/mbr"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	var size int64 = 20 * 1024 * 1024 // 20 MB

	diskImg := "/tmp/disk.img"
	defer os.Remove(diskImg)
	theDisk, _ := diskfs.Create(diskImg, size, diskfs.SectorSizeDefault)

	table := &mbr.Table{
		LogicalSectorSize:  512,
		PhysicalSectorSize: 512,
		Partitions: []*mbr.Partition{
			{
				Bootable: false,
				Type:     mbr.Linux,
				Start:    2048,
				Size:     20480,
			},
		},
	}

	check(theDisk.Partition(table))

	fs, err := theDisk.CreateFilesystem(disk.FilesystemSpec{
		Partition: 1,
		FSType:    filesystem.TypeFat32,
	})
	check(err)
	unused(fs)
}

func Open

func Open(device string, opts ...OpenOpt) (*disk.Disk, error)

Might be deprecated in future: use <backend>.New + diskfs.OpenBackend Open a Disk from a path to a device in read-write exclusive mode Should pass a path to a block device e.g. /dev/sda or a path to a file /tmp/foo.img The provided device must exist at the time you call Open(). Use OpenOpt to control options, such as sector size or open mode.

func OpenBackend

func OpenBackend(b backend.Storage, opts ...OpenOpt) (*disk.Disk, error)

Open a Disk using provided fs.File to a device in read-only mode Use OpenOpt to control options, such as sector size or open mode.

Example (Backend)

Instantiate the backend manually for later use in diskfs.OpenBackend

package main

import (
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/backend/file"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	// some exemplary fs.FS
	dirfs := os.DirFS("/tmp/images")
	// the fs.File we'll use
	f, err := dirfs.Open("theImage.raw")
	check(err)

	theBackend := file.New(f, false)

	// As opposed to File example, theBackend will be used as is (i.e. not wrapped).
	// This usage scenario is for upcoming support for other backends (qcow2 and such).
	theDisk, err := diskfs.OpenBackend(theBackend)
	check(err)
	unused(theDisk)
}

Example (Create)

Create a disk of size 20MB at provided pathname using backend abstraction. This is a proposed way of usage after factoring out underlying file operations to the backend. diskfs.Create() is slowly deprecating now.

package main

import (
	"log"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/backend/file"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	theBackend, err := file.CreateFromPath("/tmp/my.img", 20*1024*1024)
	check(err)

	theDisk, err := diskfs.OpenBackend(theBackend)
	check(err)
	unused(theDisk)
}

Example (Open)

Open existing image using backend abstraction This is a proposed way of usage after factoring out underlying file operations to the backend. diskfs.Open() is slowly deprecating now.

package main

import (
	"log"
	"os"

	diskfs "github.com/diskfs/go-diskfs"
	"github.com/diskfs/go-diskfs/backend/file"
)

func check(err error) {
	if err != nil {
		log.Fatal(err)
	}
}

func unused(_ ...any) {
}

func main() {
	readOnly := true
	theBackend, err := file.OpenFromPath("/tmp/my.img", readOnly)
	check(err)
	defer os.Remove("/tmp/my.img")

	theDisk, err := diskfs.OpenBackend(theBackend)
	check(err)
	unused(theDisk)
}

Types

type OpenModeOption

type OpenModeOption int

OpenModeOption represents file open modes

const (
	// ReadOnly open file in read only mode
	ReadOnly OpenModeOption = iota
	// ReadWriteExclusive open file in read-write exclusive mode
	ReadWriteExclusive
	// ReadWrite open file in read-write mode
	ReadWrite
)

func (OpenModeOption) String

func (m OpenModeOption) String() string

OpenModeOption.String()

type OpenOpt

type OpenOpt func(o *openOpts) error

OpenOpt func that process Open options

func WithOpenMode

func WithOpenMode(mode OpenModeOption) OpenOpt

WithOpenMode sets the opening mode to the requested mode of type OpenModeOption. Default is ReadWriteExclusive, i.e. os.O_RDWR | os.O_EXCL

func WithSectorSize

func WithSectorSize(sectorSize SectorSize) OpenOpt

WithSectorSize opens the disk file or block device with the provided sector size. Defaults to the physical block size.

type SectorSize

type SectorSize int

SectorSize represents the sector size to use

const (
	// SectorSizeDefault default behavior, defaulting to defaultBlocksize
	SectorSizeDefault SectorSize = 0
	// SectorSize512 override sector size to 512
	SectorSize512 SectorSize = 512
	// SectorSize4k override sector size to 4k
	SectorSize4k SectorSize = 4096
)