eszip

package module
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 5, 2026 License: MIT Imports: 14 Imported by: 0

README

eszip

A Go library for serializing and deserializing ECMAScript module graphs into a compact binary format (eszip). The eszip format is designed to be compact and streaming-capable, allowing efficient storage and loading of large JavaScript and TypeScript module collections.

Installation

go get github.com/JakeChampion/eszip

Library usage

Parsing an eszip archive
import (
    "context"
    "os"

    "github.com/JakeChampion/eszip"
)

data, _ := os.ReadFile("archive.eszip2")
archive, _ := eszip.ParseBytes(context.Background(), data)

for _, spec := range archive.Specifiers() {
    module := archive.GetModule(spec)
    source, _ := module.Source(context.Background())
    fmt.Printf("%s: %d bytes\n", spec, len(source))
}
Creating an eszip archive
archive := eszip.NewV2()
archive.SetChecksum(eszip.ChecksumSha256)
archive.AddModule("file:///main.js", eszip.ModuleKindJavaScript, sourceBytes, nil)

data, _ := archive.IntoBytes(ctx)
os.WriteFile("output.eszip2", data, 0644)

CLI tool

Build the CLI:

go build -o eszip ./cmd/eszip
Commands
eszip view archive.eszip2              # View contents
eszip view -s file:///main.ts archive  # View specific module
eszip view -m archive.eszip2           # View with source maps
eszip extract -o ./output archive      # Extract to disk
cat archive.eszip2 | eszip extract -o ./output  # Extract from stdin
eszip create -o archive.eszip2 *.js    # Create from files
eszip create --checksum xxhash3 -o archive.eszip2 *.js  # With checksum option
eszip info archive.eszip2              # Show archive metadata

Development

make test       # Run tests with race detection
make lint       # Run golangci-lint
make coverage   # Generate coverage report
make ci         # Run lint + tests (CI pipeline)

File format

Eszip:
| Magic (8) | Header size (4) | Header (n) | Header hash (32) | Sources size (4) | Sources (n) | SourceMaps size (4) | SourceMaps (n) |

Header:
( | Specifier size (4) | Specifier (n) | Entry type (1) | Entry (n) | )*

Entry (redirect):
| Specifier size (4) | Specifier (n) |

Entry (module):
| Source offset (4) | Source size (4) | SourceMap offset (4) | SourceMap size (4) | Module type (1) |

Sources:
( | Source (n) | Hash (32) | )*

SourceMaps:
( | SourceMap (n) | Hash (32) | )*

If both the offset and size for a source or source map are 0, no entry and no hash is present in the data sections for that module.

License

MIT

Documentation

Overview

Package eszip provides functionality for reading and writing eszip archives. Eszip is a binary serialization format for ECMAScript module graphs, used by Deno.

Index

Constants

View Source 
const LatestVersion = VersionV2_3

LatestVersion is the latest supported version

Variables

View Source 
var (
	MagicV2   = [8]byte{'E', 'S', 'Z', 'I', 'P', '_', 'V', '2'}
	MagicV2_1 = [8]byte{'E', 'S', 'Z', 'I', 'P', '2', '.', '1'}
	MagicV2_2 = [8]byte{'E', 'S', 'Z', 'I', 'P', '2', '.', '2'}
	MagicV2_3 = [8]byte{'E', 'S', 'Z', 'I', 'P', '2', '.', '3'}
)

Magic bytes for V2 versions

Functions

func HasMagic 

func HasMagic(buffer []byte) bool

HasMagic checks if the buffer starts with a V2 magic

Types

type ChecksumType 

type ChecksumType uint8

ChecksumType represents the hash algorithm used for checksums 

const (
	ChecksumNone   ChecksumType = 0
	ChecksumSha256 ChecksumType = 1
	ChecksumXxh3   ChecksumType = 2
)

func ChecksumFromU8 

func ChecksumFromU8(b uint8) (ChecksumType, bool)

FromU8 creates a ChecksumType from a byte value

func (ChecksumType) DigestSize 

func (c ChecksumType) DigestSize() uint8

DigestSize returns the size in bytes of the hash digest

func (ChecksumType) Hash 

func (c ChecksumType) Hash(data []byte) []byte

Hash computes the checksum of the given data

func (ChecksumType) Verify 

func (c ChecksumType) Verify(data, hash []byte) bool

Verify checks if the given hash matches the data

type EszipUnion 

type EszipUnion struct {
	// contains filtered or unexported fields
}

EszipUnion wraps either V1 or V2 eszip

func Parse 

func Parse(ctx context.Context, r io.Reader) (*EszipUnion, func(context.Context) error, error)

Parse parses an eszip archive from the given reader. Returns the eszip and a function to complete parsing of source data (for streaming). The completion function must be called to fully load sources.

func ParseBytes 

func ParseBytes(ctx context.Context, data []byte) (*EszipUnion, error)

ParseBytes parses an eszip from a byte slice

func ParseSync 

func ParseSync(ctx context.Context, r io.Reader) (*EszipUnion, error)

ParseSync parses an eszip archive completely (blocking)

func (*EszipUnion) GetImportMap 

func (e *EszipUnion) GetImportMap(specifier string) *Module

GetImportMap returns the import map module for the given specifier

func (*EszipUnion) GetModule 

func (e *EszipUnion) GetModule(specifier string) *Module

GetModule returns the module for the given specifier

func (*EszipUnion) IsV1 

func (e *EszipUnion) IsV1() bool

IsV1 returns true if this is a V1 archive

func (*EszipUnion) IsV2 

func (e *EszipUnion) IsV2() bool

IsV2 returns true if this is a V2 archive

func (*EszipUnion) LookupModule added in v0.2.2 

func (e *EszipUnion) LookupModule(specifier string) ModuleEntry

LookupModule returns the raw module entry for the given specifier without following redirects. Returns nil if the specifier is not found. Unlike GetModule, this does not filter out ModuleKindJsonc entries.

func (*EszipUnion) NpmSnapshot added in v0.2.0 

func (e *EszipUnion) NpmSnapshot() *NpmResolutionSnapshot

NpmSnapshot returns the NPM snapshot without removing it

func (*EszipUnion) Specifiers 

func (e *EszipUnion) Specifiers() []string

Specifiers returns all module specifiers

func (*EszipUnion) TakeNpmSnapshot 

func (e *EszipUnion) TakeNpmSnapshot() *NpmResolutionSnapshot

TakeNpmSnapshot removes and returns the NPM snapshot

func (*EszipUnion) V1 

func (e *EszipUnion) V1() (*EszipV1, bool)

V1 returns the V1 archive, or nil and false if not V1

func (*EszipUnion) V2 

func (e *EszipUnion) V2() (*EszipV2, bool)

V2 returns the V2 archive, or nil and false if not V2

type EszipV1 

type EszipV1 struct {
	Version uint32                     `json:"version"`
	Modules map[string]json.RawMessage `json:"modules"`
	// contains filtered or unexported fields
}

EszipV1 represents a V1 eszip archive (JSON format)

func ParseV1 

func ParseV1(data []byte) (*EszipV1, error)

ParseV1 parses a V1 eszip from JSON data

func (*EszipV1) GetImportMap 

func (e *EszipV1) GetImportMap(specifier string) *Module

GetImportMap returns nil for V1 (V1 never contains import maps)

func (*EszipV1) GetModule 

func (e *EszipV1) GetModule(specifier string) *Module

GetModule returns the module for the given specifier, following redirects

func (*EszipV1) IntoBytes 

func (e *EszipV1) IntoBytes() ([]byte, error)

IntoBytes serializes the V1 eszip to JSON. The output reflects any mutations made via TakeSource.

func (*EszipV1) Iterate 

func (e *EszipV1) Iterate() []struct {
	Specifier string
	Module    *Module
}

Iterate returns all modules as an iterator

func (*EszipV1) LookupModule added in v0.2.2 

func (e *EszipV1) LookupModule(specifier string) ModuleEntry

LookupModule returns the raw module entry for the given specifier without following redirects. Returns nil if the specifier is not found. Unlike GetModule, this does not filter out ModuleKindJsonc entries.

func (*EszipV1) Specifiers 

func (e *EszipV1) Specifiers() []string

Specifiers returns all module specifiers in sorted order

type EszipV2 

type EszipV2 struct {
	// contains filtered or unexported fields
}

EszipV2 represents a V2 eszip archive

func NewV2 

func NewV2() *EszipV2

NewV2 creates a new empty V2 eszip archive

func ParseV2 

func ParseV2(ctx context.Context, r io.Reader) (*EszipV2, func(context.Context) error, error)

ParseV2 parses a V2 eszip from a reader. Returns the eszip and a completion function that loads sources in background.

func ParseV2Sync 

func ParseV2Sync(ctx context.Context, r io.Reader) (*EszipV2, error)

ParseV2Sync parses a V2 eszip completely (blocking)

func (*EszipV2) AddImportMap 

func (e *EszipV2) AddImportMap(kind ModuleKind, specifier string, source []byte)

AddImportMap adds an import map at the front of the archive

func (*EszipV2) AddModule 

func (e *EszipV2) AddModule(specifier string, kind ModuleKind, source, sourceMap []byte)

AddModule adds a module to the archive

func (*EszipV2) AddOpaqueData 

func (e *EszipV2) AddOpaqueData(specifier string, data []byte)

AddOpaqueData adds opaque data to the archive

func (*EszipV2) AddRedirect 

func (e *EszipV2) AddRedirect(specifier, target string)

AddRedirect adds a redirect entry

func (*EszipV2) GetImportMap 

func (e *EszipV2) GetImportMap(specifier string) *Module

GetImportMap returns the import map module for the given specifier

func (*EszipV2) GetModule 

func (e *EszipV2) GetModule(specifier string) *Module

GetModule returns the module for the given specifier, following redirects

func (*EszipV2) IntoBytes 

func (e *EszipV2) IntoBytes(ctx context.Context) ([]byte, error)

IntoBytes serializes the eszip archive to bytes. The context allows cancellation of source slot waits during serialization.

func (*EszipV2) Iterate 

func (e *EszipV2) Iterate() []struct {
	Specifier string
	Module    *Module
}

Iterate returns all modules

func (*EszipV2) LookupModule added in v0.2.2 

func (e *EszipV2) LookupModule(specifier string) ModuleEntry

LookupModule returns the raw module entry for the given specifier without following redirects. Returns nil if the specifier is not found. Unlike GetModule, this does not filter out ModuleKindJsonc entries.

func (*EszipV2) NpmSnapshot added in v0.2.0 

func (e *EszipV2) NpmSnapshot() *NpmResolutionSnapshot

NpmSnapshot returns the NPM snapshot without removing it

func (*EszipV2) SetChecksum 

func (e *EszipV2) SetChecksum(checksum ChecksumType)

SetChecksum sets the checksum algorithm

func (*EszipV2) Specifiers 

func (e *EszipV2) Specifiers() []string

Specifiers returns all module specifiers

func (*EszipV2) TakeNpmSnapshot 

func (e *EszipV2) TakeNpmSnapshot() *NpmResolutionSnapshot

TakeNpmSnapshot removes and returns the NPM snapshot

type EszipV2Module 

type EszipV2Module interface {
	// contains filtered or unexported methods
}

EszipV2Module represents a module entry in V2 format

type EszipVersion 

type EszipVersion int

EszipVersion represents the V2 version 

const (
	VersionV2   EszipVersion = 0
	VersionV2_1 EszipVersion = 1
	VersionV2_2 EszipVersion = 2
	VersionV2_3 EszipVersion = 3
)

func VersionFromMagic 

func VersionFromMagic(magic []byte) (EszipVersion, bool)

VersionFromMagic returns the version from magic bytes

func (EszipVersion) SupportsNpm 

func (v EszipVersion) SupportsNpm() bool

SupportsNpm returns true if the version supports npm

func (EszipVersion) SupportsOptions 

func (v EszipVersion) SupportsOptions() bool

SupportsOptions returns true if the version supports options header

func (EszipVersion) ToMagic 

func (v EszipVersion) ToMagic() [8]byte

ToMagic returns the magic bytes for the version

type HeaderFrameKind 

type HeaderFrameKind uint8

HeaderFrameKind represents the type of entry in the modules header 

const (
	HeaderFrameModule       HeaderFrameKind = 0
	HeaderFrameRedirect     HeaderFrameKind = 1
	HeaderFrameNpmSpecifier HeaderFrameKind = 2
)

type Module 

type Module struct {
	Specifier string
	Kind      ModuleKind
	// contains filtered or unexported fields
}

Module represents a module in the eszip archive

func (*Module) Source 

func (m *Module) Source(ctx context.Context) ([]byte, error)

Source returns the source code of the module. This may block if the source hasn't been loaded yet (streaming).

func (*Module) SourceMap 

func (m *Module) SourceMap(ctx context.Context) ([]byte, error)

SourceMap returns the source map of the module (V2 only).

func (*Module) TakeSource 

func (m *Module) TakeSource(ctx context.Context) ([]byte, error)

TakeSource returns and removes the source from memory.

func (*Module) TakeSourceMap 

func (m *Module) TakeSourceMap(ctx context.Context) ([]byte, error)

TakeSourceMap returns and removes the source map from memory.

type ModuleData 

type ModuleData struct {
	Kind      ModuleKind
	Source    *SourceSlot
	SourceMap *SourceSlot
}

ModuleData represents an actual module with source

type ModuleEntry 

type ModuleEntry interface {
	// contains filtered or unexported methods
}

ModuleEntry is a sealed interface for module map entries returned by LookupModule. It is implemented by *Module and *ModuleRedirect. Unlike GetModule, LookupModule does not filter out ModuleKindJsonc entries.

type ModuleKind 

type ModuleKind uint8

ModuleKind represents the type of module stored 

const (
	ModuleKindJavaScript ModuleKind = 0
	ModuleKindJson       ModuleKind = 1
	ModuleKindJsonc      ModuleKind = 2
	ModuleKindOpaqueData ModuleKind = 3
	ModuleKindWasm       ModuleKind = 4
)

func (ModuleKind) String 

func (k ModuleKind) String() string

type ModuleMap 

type ModuleMap struct {
	// contains filtered or unexported fields
}

ModuleMap is a thread-safe ordered map of modules

func NewModuleMap 

func NewModuleMap() *ModuleMap

NewModuleMap creates a new module map

func (*ModuleMap) Get 

func (m *ModuleMap) Get(specifier string) (EszipV2Module, bool)

Get retrieves a module

func (*ModuleMap) Insert 

func (m *ModuleMap) Insert(specifier string, module EszipV2Module)

Insert adds or updates a module

func (*ModuleMap) InsertFront 

func (m *ModuleMap) InsertFront(specifier string, module EszipV2Module)

InsertFront adds a module at the front (for import maps)

func (*ModuleMap) Keys 

func (m *ModuleMap) Keys() []string

Keys returns all specifiers in order

func (*ModuleMap) Len 

func (m *ModuleMap) Len() int

Len returns the number of modules

type ModuleRedirect 

type ModuleRedirect struct {
	Target string
}

ModuleRedirect represents a redirect to another specifier

type NpmPackage 

type NpmPackage struct {
	ID           *NpmPackageID
	Dependencies map[string]*NpmPackageID // req -> id
}

NpmPackage represents a resolved NPM package

type NpmPackageID 

type NpmPackageID struct {
	Name    string
	Version string
}

NpmPackageID represents an NPM package identifier (name@version)

func ParseNpmPackageID 

func ParseNpmPackageID(s string) (*NpmPackageID, error)

ParseNpmPackageID parses a serialized NPM package ID (name@version)

func (*NpmPackageID) String 

func (id *NpmPackageID) String() string

String returns the serialized form of the package ID

type NpmPackageIndex 

type NpmPackageIndex struct {
	Index uint32
}

NpmPackageIndex represents an npm package index

type NpmResolutionSnapshot 

type NpmResolutionSnapshot struct {
	Packages     []*NpmPackage
	RootPackages map[string]*NpmPackageID // req -> id
}

NpmResolutionSnapshot represents the NPM package resolution

type NpmSpecifierEntry 

type NpmSpecifierEntry struct {
	PackageID uint32
}

NpmSpecifierEntry represents an npm specifier entry

type Options 

type Options struct {
	Checksum     ChecksumType
	ChecksumSize uint8
}

Options represents V2 options

func DefaultOptionsForVersion 

func DefaultOptionsForVersion(version EszipVersion) Options

DefaultOptionsForVersion returns the default options for a version

func (Options) GetChecksumSize 

func (o Options) GetChecksumSize() uint8

GetChecksumSize returns the effective checksum size

type ParseError 

type ParseError struct {
	Type    ParseErrorType
	Message string
	Offset  int
}

ParseError represents an error that occurred during parsing

func (*ParseError) Error 

func (e *ParseError) Error() string

type ParseErrorType 

type ParseErrorType int

ParseErrorType represents the type of parse error 

const (
	ErrInvalidV1Json ParseErrorType = iota
	ErrInvalidV1Version
	ErrInvalidV2
	ErrInvalidV2HeaderHash
	ErrInvalidV2EntryKind
	ErrInvalidV2ModuleKind
	ErrInvalidV2Header
	ErrInvalidV2SourceOffset
	ErrInvalidV2SourceHash
	ErrInvalidV2NpmSnapshotHash
	ErrInvalidV2NpmPackageOffset
	ErrInvalidV2NpmPackage
	ErrInvalidV2NpmPackageReq
	ErrInvalidV22OptionsHeader
	ErrInvalidV22OptionsHeaderHash
	ErrIO
)

type Section 

type Section struct {
	// contains filtered or unexported fields
}

Section represents a parsed section with content and hash

func (*Section) Content 

func (s *Section) Content() []byte

Content returns the section content

func (*Section) ContentLen 

func (s *Section) ContentLen() int

ContentLen returns the content length

func (*Section) IntoContent 

func (s *Section) IntoContent() []byte

IntoContent returns and takes ownership of the content

func (*Section) IsChecksumValid 

func (s *Section) IsChecksumValid() bool

IsChecksumValid verifies the section checksum

func (*Section) TotalLen 

func (s *Section) TotalLen() int

TotalLen returns the total length including hash

type SourceSlot 

type SourceSlot struct {
	// contains filtered or unexported fields
}

SourceSlot represents a pending or loaded source

func NewEmptySourceSlot 

func NewEmptySourceSlot() *SourceSlot

NewEmptySourceSlot creates a new ready source slot with empty data

func NewPendingSourceSlot 

func NewPendingSourceSlot(offset, length uint32) *SourceSlot

NewPendingSourceSlot creates a new pending source slot

func NewReadySourceSlot 

func NewReadySourceSlot(data []byte) *SourceSlot

NewReadySourceSlot creates a new ready source slot with data

func (*SourceSlot) Get 

func (s *SourceSlot) Get(ctx context.Context) ([]byte, error)

Get returns the source data, blocking until ready or context cancelled

func (*SourceSlot) Length 

func (s *SourceSlot) Length() uint32

Length returns the length in the sources section

func (*SourceSlot) Offset 

func (s *SourceSlot) Offset() uint32

Offset returns the offset in the sources section

func (*SourceSlot) SetReady 

func (s *SourceSlot) SetReady(data []byte)

SetReady marks the slot as ready with the given data. It is a no-op if the slot is already ready or taken.

func (*SourceSlot) State 

func (s *SourceSlot) State() SourceSlotState

State returns the current state

func (*SourceSlot) Take 

func (s *SourceSlot) Take(ctx context.Context) ([]byte, error)

Take returns and removes the source data

type SourceSlotState 

type SourceSlotState int

SourceSlotState represents the state of a source slot 

const (
	SourceSlotPending SourceSlotState = iota
	SourceSlotReady
	SourceSlotTaken
)

Source Files

View all Source files

Directories

Path Synopsis
cmd
eszip command
eszip is a CLI tool for working with eszip archives.
 eszip is a CLI tool for working with eszip archives.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.