magic

package module
v0.5.1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 15, 2026 License: MIT Imports: 11 Imported by: 0

README


Unit Tests Go Reference Releases Discord Discussion

magic

magic is a Go package that identifies the type of a file or of bytes, as the file command does. It is a port of libmagic, the library of file, to pure Go.

It gives the output of libmagic 5.48, with the commits of file after the release, to the commit in docs/upstream/upstreams.tsv: the same description, MIME type, charset, extensions and Apple code. The rules of file are compiled into the package. You need only Go 1.27 or later. You do not need a C compiler or libmagic, and a program that imports magic builds for every target of Go.

Install

go get github.com/xo/magic

Use

Detect identifies bytes or a string:

r, err := magic.Detect(ctx, buf)
if err != nil {
	return err
}
fmt.Println(r.Description)   // GIF image data, version 89a, 1375 x 825
fmt.Println(r.MIME)          // image/gif
fmt.Println(r.ContentType()) // image/gif; charset=binary
fmt.Println(r.Extensions)    // [gif]

A Result holds every form of the answer at once:

Field What it holds
Description the text that file prints
MIME the MIME type, such as image/gif
Charset the MIME encoding, such as binary or utf-8
Extensions the usual extensions, without the dot, or nil
Apple the creator and type code of classic Mac OS, or ""
Matches each match, when WithAll is set, and else the one match

DetectFile identifies a file. A directory, a link, a pipe or a device is named by its kind, as file names it. A file that cannot be found or opened is an error that wraps the error of the os package:

r, err := magic.DetectFile(ctx, "photo.jpg")

DetectReader reads from an io.Reader, and DetectReaderAt reads from an io.ReaderAt of a known size. A rule can read from the end of a file, so DetectReaderAt gives the same answer as a file, and DetectReader does not when the data is longer than what it reads.

Each function takes a context.Context first. A detection stops between the rules when the context is done, and its error wraps magic.ErrStopped and the error of the context. Every function is safe to call from many goroutines at once.

Options

The options change how a call detects:

Option What it does Flag of libmagic
WithAll() keeps every match, one on each line of Description, and one in each entry of Matches MAGIC_CONTINUE
WithDecompress() identifies what is inside compressed data MAGIC_COMPRESS
WithLimit(n) reads and examines n bytes. The default is 7 MiB bytes_max
WithRaw() keeps the characters that cannot be printed, which are octal escapes by default MAGIC_RAW
WithFollowSymlinks() identifies the target of a link MAGIC_SYMLINK
WithDevices() reads a device MAGIC_DEVICES
WithoutCheck(c) turns off one built-in test, such as CheckJSON MAGIC_NO_CHECK_*
Compressed data

WithDecompress decodes a format only when the program imports its decoder, as a program imports the driver of database/sql. Each decoder adds to the size of the program, so a program imports the ones it needs:

import (
	_ "github.com/xo/magic/compress/bzip2" // bzip2
	_ "github.com/xo/magic/compress/xz"    // xz and lzma
	_ "github.com/xo/magic/compress/zlib"  // gzip and zlib
	_ "github.com/xo/magic/compress/zstd"  // zstd
)

compress/xz and compress/zstd are Go modules of their own, because they need other modules. Get them with go get as well. For a format with no decoder, magic answers as libmagic does when it was built without the library and finds no program that decodes it. magic never runs a program.

Size

A program that prints the MIME type of a file grows by about 980 KB with magic, stripped, on linux/amd64. Most of it is the compiled rules. The decoders add 45 KB for zlib, 74 KB for bzip2, 442 KB for xz and 594 KB for zstd.

How it is tested

The test module builds libmagic from the source of file, at the commit that the port follows, and compares the package with it, on 2,471 files and damaged copies of them, with 14 combinations of flags, as bytes and as files. Every answer is the same, except 50 where a decoder of Go reads damaged compressed data in another way than the library of C. All 113 tests of file pass. CONTRIBUTING.md says how to run the tests.

Versions

Up to v0.4.1, magic wrapped libmagic with cgo, and carried static archives of it for eight targets. The port is the next minor version, and its API is new (D39). docs/API.md gives the reasons.

To move from v0.4.1:

v0.4.1 The port
m, err := magic.New(magic.MimeType), then m.Buffer(buf) r, err := magic.Detect(ctx, buf), then r.MIME
m.File(name) magic.DetectFile(ctx, name)
magic.None r.Description
magic.MimeEncoding r.Charset
magic.Mime r.ContentType()
magic.Extension r.Extensions
magic.Apple r.Apple
magic.Continue magic.WithAll()
magic.Compress magic.WithDecompress(), and a decoder from compress/
m.Close() nothing. The port has no handle
magic.Version() magic.FileVersion
magic.MagicDB() nothing. The port has no .mgc (D47)

A file that cannot be opened is an error now, where libmagic gave the text "cannot open".

Documents

Document What it holds
README.md this page, for a person who uses the package
CONTRIBUTING.md how to change the package and test it
AGENTS.md the rules for a coding agent. CLAUDE.md imports it
docs/PLAN.md the plan of the port and its phases
docs/API.md the design of the API
docs/decisions/ every decision, one file each, with an index
docs/UPSTREAM.md the rules for following file, the upstream of the port
docs/BACKLOG.md the work that is known and not done, and the binary walker

magic is one of the xo projects. Several of them, as magic does, port a library of C or Rust to pure Go. Each one is a separate repository:

Project What it is
okage A Go port of tiny-skia and resvg, which draws 2D graphics and SVG images
transit A Go port of tree-sitter, a parser of source code
rline A readline package for Go, which reads a line of text that a person edits. It started as a port of isocline
zuzu Draws DOT graphs, Mermaid diagrams, and Vega and Vega-Lite charts
ox Parses the flags and arguments of a command line
usql A command line client for SQL and NoSQL databases

iv, an image viewer for the terminal, uses magic to find the type of each file that it shows.

License

The code of magic that is not a translation, such as the API and the tests, is under the MIT license, in LICENSE. The port translates file, which is under its own BSD license, in LICENSE.file. Each translated Go file opens with the notice of the file of file that it translates, and the compiled rules carry the notices of file and of its rule files, in internal/table/LICENSE. A program that ships magic must reproduce these notices in its documentation.

Documentation

Overview

Package magic identifies the type of a file or of bytes, as the file command does. It is a port of libmagic, the library of file, to pure Go (D22). It gives the output of libmagic 5.48 and the commits of file after it (D48), and needs no C compiler.

r, err := magic.Detect(ctx, buf)
if err != nil {
	return err
}
fmt.Println(r.MIME)        // image/gif
fmt.Println(r.Description) // GIF image data, version 89a, 1375 x 825
fmt.Println(r.Extensions)  // [gif]

The rules of file are compiled into the package (D35). A caller cannot add rules. Every function is safe to call from many goroutines at once.

Index

Examples

Constants

View Source
const FileVersion = "5.48"

FileVersion is the version of file whose behavior the package copies. The package follows the commits of file after the release, to the commit that docs/upstream/upstreams.tsv names (D48), and file still calls that 5.48.

Variables

View Source
var ErrStopped = errors.New("magic: the detection stopped")

ErrStopped is the error of a detection that stopped because its context is done. The error also wraps the error of the context.

Functions

This section is empty.

Types

type Check

type Check int

Check is one built-in test of libmagic.

const (
	CheckCompress Check = iota
	CheckTar
	CheckRules
	CheckAppType
	CheckELF
	CheckText
	CheckCDF
	CheckCSV
	CheckTokens
	CheckEncoding
	CheckJSON
	CheckSIMH
)

The built-in tests, as the flags MAGIC_NO_CHECK_* name them.

type Match added in v0.5.0

type Match struct {
	Description string
	// MIME, Extensions and Apple are set for a match after the first only
	// when libmagic gives them for each match, as it does when every match
	// has one.
	MIME       string
	Extensions []string
	Apple      string
}

Match is one match of the rules, or of a built-in test: one line that file prints with its option --keep-going.

type Option added in v0.5.0

type Option func(*options) error

Option changes how a call detects.

func WithAll added in v0.5.0

func WithAll() Option

WithAll keeps every match, as MAGIC_CONTINUE does. Description then holds each match on a line of its own, and Matches holds one Match for each.

func WithDecompress added in v0.5.0

func WithDecompress() Option

WithDecompress identifies what is inside compressed data, as MAGIC_COMPRESS does. It decodes the formats whose decoder a program imports from github.com/xo/magic/compress (D45). magic runs no outside program (D36).

func WithDevices added in v0.5.0

func WithDevices() Option

WithDevices makes DetectFile read a device, as MAGIC_DEVICES does.

func WithFollowSymlinks() Option

WithFollowSymlinks makes DetectFile identify the target of a link, as MAGIC_SYMLINK does. By default a link is named as a link.

func WithLimit added in v0.5.0

func WithLimit(n int) Option

WithLimit sets how many bytes are read and examined. The default is 7 MiB, as in libmagic.

func WithRaw added in v0.5.0

func WithRaw() Option

WithRaw keeps characters that cannot be printed, as MAGIC_RAW does. By default they are written as octal escapes, as file writes them.

func WithoutCheck added in v0.5.0

func WithoutCheck(c Check) Option

WithoutCheck turns off one built-in test, as a MAGIC_NO_CHECK_* flag does.

type Result added in v0.5.0

type Result struct {
	// Description is the text that file prints, such as
	// "GIF image data, version 89a, 1375 x 825". With WithAll, it holds each
	// match on a line of its own, after the first with the prefix "- ".
	Description string

	// MIME is the MIME type, such as "image/gif".
	MIME string

	// Charset is the MIME encoding, such as "binary", "us-ascii" or
	// "utf-8".
	Charset string

	// Extensions are the usual extensions without the dot, such as
	// ["jpeg" "jpg" "jpe" "jfif"], or nil when no rule gives one.
	Extensions []string

	// Apple is the creator and type code of classic Mac OS, such as
	// "8BIMGIFf", or "" when no rule gives one.
	Apple string

	// Matches holds every match when WithAll is set, the first one included.
	// Without WithAll it holds the one match that the fields above describe.
	Matches []Match
}

Result is what libmagic says of the content, in every form at once.

func Detect added in v0.5.0

func Detect[S Source](ctx context.Context, src S, opts ...Option) (Result, error)

Detect identifies src. Bytes that match nothing are "data", and no bytes are "empty". It returns an error when ctx is done or an option is not valid, such as a negative limit. It does not keep src, and does not change it.

Example
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/xo/magic"
)

func main() {
	buf, err := os.ReadFile("testdata/gif.gif")
	if err != nil {
		fmt.Println(err)
		return
	}
	r, err := magic.Detect(context.Background(), buf)
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(r.Description)
	fmt.Println(r.ContentType())
	fmt.Println(r.Extensions)
}
Output:
GIF image data, version 89a, 1375 x 825
image/gif; charset=binary
[gif]

func DetectFile added in v0.5.0

func DetectFile(ctx context.Context, name string, opts ...Option) (Result, error)

DetectFile identifies the file at name. A directory, a link, a pipe or a device is named by its kind, as file names it. A file that cannot be found or opened is an error, which wraps the error of the os package.

Example
package main

import (
	"context"
	"fmt"

	"github.com/xo/magic"
)

func main() {
	r, err := magic.DetectFile(context.Background(), "testdata")
	if err != nil {
		fmt.Println(err)
		return
	}
	fmt.Println(r.Description)
	fmt.Println(r.MIME)
}
Output:
directory
inode/directory

func DetectReader added in v0.5.0

func DetectReader(ctx context.Context, r io.Reader, opts ...Option) (Result, error)

DetectReader reads from r up to the limit (WithLimit), and identifies what it read. A stream has no end that a rule can read relative to, so a rule that reads from the end of the file sees the end of what was read. Use DetectReaderAt when the size is known.

func DetectReaderAt added in v0.5.0

func DetectReaderAt(ctx context.Context, r io.ReaderAt, size int64, opts ...Option) (Result, error)

DetectReaderAt identifies the size bytes that r holds. It reads at the offsets that the rules ask for, including offsets from the end, and reads at most the limit from the start and the limit from the end. An *os.File, a *bytes.Reader and a *strings.Reader are each an io.ReaderAt.

func (Result) ContentType added in v0.5.0

func (r Result) ContentType() string

ContentType returns MIME and Charset in the form of an HTTP header, such as "image/gif; charset=binary".

func (Result) String added in v0.5.0

func (r Result) String() string

String returns Description.

type Source added in v0.5.0

type Source interface {
	~[]byte | ~string
}

Source is the content that Detect reads: bytes, or a string of bytes.

Directories

Path Synopsis
cmd
file command
Command file is a small example of the package magic: it prints the type of each file that it gets, as the file command does.
Command file is a small example of the package magic: it prints the type of each file that it gets, as the file command does.
Package compress is the registry of the decoders that magic uses to look inside compressed data.
Package compress is the registry of the decoders that magic uses to look inside compressed data.
bzip2
Package bzip2 registers the decoder of bzip2 data with github.com/xo/magic/compress, as the library bzlib does in libmagic.
Package bzip2 registers the decoder of bzip2 data with github.com/xo/magic/compress, as the library bzlib does in libmagic.
zlib
Package zlib registers the decoders of gzip and zlib data with github.com/xo/magic/compress, as the library zlib does in libmagic.
Package zlib registers the decoders of gzip and zlib data with github.com/xo/magic/compress, as the library zlib does in libmagic.
xz module
zstd module
internal
engine
Package engine is the port of the engine of libmagic 5.48: it runs the compiled rules and the detectors on a buffer or a file, and gives the text that libmagic gives (D22, D34).
Package engine is the port of the engine of libmagic 5.48: it runs the compiled rules and the detectors on a buffer or a file, and gives the text that libmagic gives (D22, D34).
rules
Package rules compiles the rules of file, the text of magic/Magdir, into the entries that the engine of the port matches (D35).
Package rules compiles the rules of file, the text of magic/Magdir, into the entries that the engine of the port matches (D35).
table/gen command
libmagic
darwin_amd64 module
darwin_arm64 module
freebsd_amd64 module
linux_amd64 module
linux_arm module
linux_arm64 module
windows_amd64 module
windows_arm64 module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL