archive

package
Version: v0.0.1-0...-c966c28 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 22, 2020 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source
const (
	NamespaceUnopened = iota
	NamespaceOpened
	NamespaceClosed
)
View Source
const MagicNumber uint32 = 0x8199e26d

MagicNumber is four bytes that are found at the beginning of the archive that indicate that the byte stream is an archive, as opposed to anything else, including a stream of BSON documents

Variables

This section is empty.

Functions

This section is empty.

Types

type CollectionMetadata

type CollectionMetadata struct {
	Database   string `bson:"db"`
	Collection string `bson:"collection"`
	Metadata   string `bson:"metadata"`
	Size       int    `bson:"size"`
}

CollectionMetadata is a data structure that, as BSON, is found in the prelude of the archive. There is one CollectionMetadata per collection that will be in the archive.

type Demultiplexer

type Demultiplexer struct {
	In io.Reader

	// NamespaceChan is used to send a namespace to a consumer of namespaces.
	NamespaceChan chan string

	// NamespaceErrorChan is used to receive an error or nil from a namespace
	// consumer immediately after each namespace is sent and validated.
	NamespaceErrorChan chan error

	NamespaceStatus map[string]int
	// contains filtered or unexported fields
}

Demultiplexer implements Parser.

func CreateDemux

func CreateDemux(namespaceMetadatas []*CollectionMetadata, in io.Reader) *Demultiplexer

func (*Demultiplexer) BodyBSON

func (demux *Demultiplexer) BodyBSON(buf []byte) error

BodyBSON is part of the ParserConsumer interface and receives BSON bodies from the parser. Its main role is to dispatch the body to the Read() function of the current DemuxOut.

func (*Demultiplexer) End

func (demux *Demultiplexer) End() error

End is part of the ParserConsumer interface and receives the end of archive notification.

func (*Demultiplexer) HeaderBSON

func (demux *Demultiplexer) HeaderBSON(buf []byte) error

HeaderBSON is part of the ParserConsumer interface and receives headers from parser. Its main role is to implement opens and EOFs of the embedded stream.

func (*Demultiplexer) NewPrioritizer

func (demux *Demultiplexer) NewPrioritizer(mgr *intents.Manager) *Prioritizer

NewPrioritizer creates a new Prioritizer and hooks up its Namespace channels to the ones in demux

func (*Demultiplexer) Open

func (demux *Demultiplexer) Open(ns string, out DemuxOut)

Open installs the DemuxOut as the handler for data for the namespace ns

func (*Demultiplexer) Run

func (demux *Demultiplexer) Run() error

Run creates and runs a parser with the Demultiplexer as a consumer

type DemuxOut

type DemuxOut interface {
	Write([]byte) (int, error)
	End()
	Sum64() (uint64, bool)
}

DemuxOut is a Demultiplexer output consumer The Write() and End() occur in the same thread as the Demultiplexer runs in.

type DirLike

type DirLike interface {
	Name() string
	Path() string
	Size() int64
	IsDir() bool
	Stat() (DirLike, error)
	ReadDir() ([]DirLike, error)
	Parent() DirLike
}

DirLike represents the group of methods done on directories and files in dump directories, or in archives, when mongorestore is figuring out what intents to create.

type Header struct {
	ConcurrentCollections int32  `bson:"concurrent_collections"`
	FormatVersion         string `bson:"version"`
	ServerVersion         string `bson:"server_version"`
	ToolVersion           string `bson:"tool_version"`
}

Header is a data structure that, as BSON, is found immediately after the magic number in the archive, before any CollectionMetadatas. It is the home of any archive level information

type MetadataFile

type MetadataFile struct {
	*bytes.Buffer
	Intent *intents.Intent
	// contains filtered or unexported fields
}

MetadataFile implements intents.file

func (*MetadataFile) Close

func (md *MetadataFile) Close() error

func (*MetadataFile) Open

func (md *MetadataFile) Open() error

func (*MetadataFile) Pos

func (md *MetadataFile) Pos() int64

func (*MetadataFile) Read

func (md *MetadataFile) Read(p []byte) (int, error)

type MetadataPreludeFile

type MetadataPreludeFile struct {
	Intent  *intents.Intent
	Origin  string
	Prelude *Prelude
	*bytes.Buffer
	// contains filtered or unexported fields
}

MetadataPreludeFile is part of the intents.file. It allows the metadata contained in the prelude to be opened and read

func (*MetadataPreludeFile) Close

func (mpf *MetadataPreludeFile) Close() error

Close is part of the intents.file interface.

func (*MetadataPreludeFile) Open

func (mpf *MetadataPreludeFile) Open() error

Open is part of the intents.file interface, it finds the metadata in the prelude and creates a bytes.Buffer from it.

func (*MetadataPreludeFile) Pos

func (mpf *MetadataPreludeFile) Pos() int64

func (*MetadataPreludeFile) Read

func (mpf *MetadataPreludeFile) Read(p []byte) (int, error)

type Multiplexer

type Multiplexer struct {
	Out       io.WriteCloser
	Control   chan *MuxIn
	Completed chan error
	// contains filtered or unexported fields
}

Multiplexer is what one uses to create interleaved intents in an archive

func NewMultiplexer

func NewMultiplexer(out io.WriteCloser, shutdownInputs notifier) *Multiplexer

NewMultiplexer creates a Multiplexer and populates its Control/Completed chans it takes a WriteCloser, which is where in inputs will get multiplexed on to, and it takes a notifier, which should allow the multiplexer to ask for the shutdown of the inputs.

func (*Multiplexer) Run

func (mux *Multiplexer) Run()

Run multiplexes until it receives an EOF on its Control chan.

type MutedCollection

type MutedCollection struct {
	Intent *intents.Intent
	Demux  *Demultiplexer
}

MutedCollection implements both DemuxOut as well as intents.file. It serves as a way to let the demutiplexer ignore certain embedded streams

func (*MutedCollection) Close

func (*MutedCollection) Close() error

Close is part of the intents.file interface, and does nothing

func (*MutedCollection) End

func (*MutedCollection) End()

End is part of the DemuxOut interface and does nothing.

func (*MutedCollection) Open

func (*MutedCollection) Open() error

Open is part of the intents.file interface, and does nothing

func (*MutedCollection) Read

func (*MutedCollection) Read([]byte) (int, error)

Read is part of the intents.file interface, and does nothing

func (*MutedCollection) Sum64

func (*MutedCollection) Sum64() (uint64, bool)

Sum64 is part of the DemuxOut interface

func (*MutedCollection) Write

func (*MutedCollection) Write(b []byte) (int, error)

Write is part of the intents.file interface, and does nothing

type MuxIn

type MuxIn struct {
	Intent *intents.Intent
	Mux    *Multiplexer
	// contains filtered or unexported fields
}

MuxIn is an implementation of the intents.file interface. They live in the intents, and are potentially owned by different threads than the thread owning the Multiplexer. They are out the intents write data to the multiplexer

func (*MuxIn) Close

func (muxIn *MuxIn) Close() error

Close closes the chans in the MuxIn. Ultimately the multiplexer will detect that they are closed and cause a formatEOF to occur.

func (*MuxIn) Open

func (muxIn *MuxIn) Open() error

Open is implemented in Mux.open, but in short, it creates chans and a select case and adds the SelectCase and the MuxIn in to the Multiplexer.

func (*MuxIn) Pos

func (muxIn *MuxIn) Pos() int64

func (*MuxIn) Read

func (muxIn *MuxIn) Read([]byte) (int, error)

Read does nothing for MuxIns

func (*MuxIn) Write

func (muxIn *MuxIn) Write(buf []byte) (int, error)

Write hands a buffer to the Multiplexer and receives a written length from the multiplexer after the length is received, the buffer is free to be reused.

type NamespaceHeader

type NamespaceHeader struct {
	Database   string `bson:"db"`
	Collection string `bson:"collection"`
	EOF        bool   `bson:"EOF"`
	CRC        int64  `bson:"CRC"`
}

NamespaceHeader is a data structure that, as BSON, is found in archives where it indicates that either the subsequent stream of BSON belongs to this new namespace, or that the indicated namespace will have no more documents (EOF)

type Parser

type Parser struct {
	In io.Reader
	// contains filtered or unexported fields
}

Parser encapsulates the small amount of state that the parser needs to keep

func (*Parser) ReadAllBlocks

func (parse *Parser) ReadAllBlocks(consumer ParserConsumer) (err error)

ReadAllBlocks calls ReadBlock() until it returns an error. If the error is EOF, then nil is returned, otherwise it returns the error

func (*Parser) ReadBlock

func (parse *Parser) ReadBlock(consumer ParserConsumer) (err error)

ReadBlock reads one archive block ( header + body* + terminator ) calling consumer.HeaderBSON() on the header, consumer.BodyBSON() on each piece of body, and consumer.EOF() when EOF is encountered before any data was read. It returns nil if a whole block was read, io.EOF if nothing was read, and a parserError if there was any io error in the middle of the block, if either of the consumer methods return error, or if there was any sort of parsing failure.

type ParserConsumer

type ParserConsumer interface {
	HeaderBSON([]byte) error
	BodyBSON([]byte) error
	End() error
}

ParserConsumer is the interface that one needs to implement to consume data from the Parser

type Prelude

type Prelude struct {
	Header                 *Header
	DBS                    []string
	NamespaceMetadatas     []*CollectionMetadata
	NamespaceMetadatasByDB map[string][]*CollectionMetadata
}

Prelude represents the knowledge gleaned from reading the prelude out of the archive.

func NewPrelude

func NewPrelude(manager *intents.Manager, concurrentColls int, serverVersion, toolVersion string) (*Prelude, error)

NewPrelude generates a Prelude using the contents of an intent.Manager.

func (*Prelude) AddMetadata

func (prelude *Prelude) AddMetadata(cm *CollectionMetadata)

AddMetadata adds a metadata data structure to a prelude and does the required bookkeeping.

func (*Prelude) NewPreludeExplorer

func (prelude *Prelude) NewPreludeExplorer() (*PreludeExplorer, error)

NewPreludeExplorer creates a PreludeExplorer from a Prelude.

func (*Prelude) Read

func (prelude *Prelude) Read(in io.Reader) error

Read consumes and checks the magic number at the beginning of the archive, then it runs the parser with a Prelude as its consumer.

func (*Prelude) Write

func (prelude *Prelude) Write(out io.Writer) error

Write writes the archive header.

type PreludeExplorer

type PreludeExplorer struct {
	// contains filtered or unexported fields
}

PreludeExplorer implements DirLike. PreludeExplorer represent the databases, collections, and their metadata json files, of an archive, in such a way that they can be explored like a filesystem.

func (*PreludeExplorer) IsDir

func (pe *PreludeExplorer) IsDir() bool

IsDir is part of the DirLike interface. All pes that are not collections are Dirs.

func (*PreludeExplorer) Name

func (pe *PreludeExplorer) Name() string

Name is part of the DirLike interface. It synthesizes a filename for the given "location" the prelude.

func (*PreludeExplorer) Parent

func (pe *PreludeExplorer) Parent() DirLike

Parent is part of the DirLike interface. It returns a pe without a collection, if there is one, otherwise, without a database.

func (*PreludeExplorer) Path

func (pe *PreludeExplorer) Path() string

Path is part of the DirLike interface. It creates the full path for the "location" in the prelude.

func (*PreludeExplorer) ReadDir

func (pe *PreludeExplorer) ReadDir() ([]DirLike, error)

ReadDir is part of the DirLIke interface. ReadDir generates a list of PreludeExplorers whose "locations" are encapsulated by the current pes "location".

"dump/oplog.bson"     => &PreludeExplorer{ database: "", collection: "oplog.bson" }
"dump/test/"          => &PreludeExplorer{ database: "test", collection: "foo.bson" }
"dump/test/foo.bson"  => &PreludeExplorer{ database: "test", collection: "" }
"dump/test/foo.json"  => &PreludeExplorer{ database: "test", collection: "foo", isMetadata: true }

func (*PreludeExplorer) Size

func (pe *PreludeExplorer) Size() int64

Size is part of the DirLike interface. It returns the size from the metadata of the prelude, if the "location" is a collection.

func (*PreludeExplorer) Stat

func (pe *PreludeExplorer) Stat() (DirLike, error)

Stat is part of the DirLike interface. os.Stat returns a FileInfo, and since DirLike is similar to FileInfo, we just return the pe, here.

type Prioritizer

type Prioritizer struct {
	NamespaceChan      <-chan string
	NamespaceErrorChan chan<- error
	// contains filtered or unexported fields
}

Prioritizer is a completely reactive prioritizer Intents are handed out as they arrive in the archive

func (*Prioritizer) Finish

func (prioritizer *Prioritizer) Finish(*intents.Intent)

Finish is part of the IntentPrioritizer interface, and does nothing

func (*Prioritizer) Get

func (prioritizer *Prioritizer) Get() *intents.Intent

Get waits for a new namespace from the NamespaceChan, and returns a Intent found for it

type Reader

type Reader struct {
	In      io.ReadCloser
	Demux   *Demultiplexer
	Prelude *Prelude
}

Reader is the top level object to contain information about archives in mongorestore

type RegularCollectionReceiver

type RegularCollectionReceiver struct {
	Intent *intents.Intent
	Origin string
	Demux  *Demultiplexer
	// contains filtered or unexported fields
}

RegularCollectionReceiver implements the intents.file interface.

func (*RegularCollectionReceiver) Close

func (receiver *RegularCollectionReceiver) Close() error

Close is part of the DemuxOut as well as the intents.file interface. It only closes the readLenChan, as that is what will cause the RegularCollectionReceiver.Read() to receive EOF Close will get called twice, once in the demultiplexer, and again when the restore goroutine is done with its intent.file

func (*RegularCollectionReceiver) End

func (receiver *RegularCollectionReceiver) End()

End signals to any waiting readers that there is nothing more to read, then it waits on the readBufChan, which is closed by the reader-side Close() method.

func (*RegularCollectionReceiver) Open

func (receiver *RegularCollectionReceiver) Open() error

Open is part of the intents.file interface. It creates the chan's in the RegularCollectionReceiver and adds the RegularCollectionReceiver to the set of RegularCollectionReceivers in the demultiplexer

func (*RegularCollectionReceiver) Pos

func (receiver *RegularCollectionReceiver) Pos() int64

func (*RegularCollectionReceiver) Read

func (receiver *RegularCollectionReceiver) Read(r []byte) (int, error)

Read() runs in the restoring goroutine

func (*RegularCollectionReceiver) ReleaseIOBuffer

func (receiver *RegularCollectionReceiver) ReleaseIOBuffer()

func (*RegularCollectionReceiver) Sum64

func (receiver *RegularCollectionReceiver) Sum64() (uint64, bool)

func (*RegularCollectionReceiver) TakeIOBuffer

func (receiver *RegularCollectionReceiver) TakeIOBuffer(ioBuf []byte)

func (*RegularCollectionReceiver) Write

func (receiver *RegularCollectionReceiver) Write(buf []byte) (int, error)

Write is part of the DemuxOut interface.

type SpecialCollectionCache

type SpecialCollectionCache struct {
	Intent *intents.Intent
	Demux  *Demultiplexer
	// contains filtered or unexported fields
}

SpecialCollectionCache implements both DemuxOut as well as intents.file

func NewSpecialCollectionCache

func NewSpecialCollectionCache(intent *intents.Intent, demux *Demultiplexer) *SpecialCollectionCache

func (*SpecialCollectionCache) Close

func (cache *SpecialCollectionCache) Close() error

Close is part of the intents.file interface, and does nothing

func (*SpecialCollectionCache) End

func (cache *SpecialCollectionCache) End()

End indicates we've read all there is to read, so we update the intent size based on the final length.

func (*SpecialCollectionCache) Open

func (cache *SpecialCollectionCache) Open() error

Open is part of the both interfaces, and it does nothing

func (*SpecialCollectionCache) Pos

func (cache *SpecialCollectionCache) Pos() int64

func (*SpecialCollectionCache) Read

func (cache *SpecialCollectionCache) Read(p []byte) (int, error)

func (*SpecialCollectionCache) Sum64

func (cache *SpecialCollectionCache) Sum64() (uint64, bool)

func (*SpecialCollectionCache) Write

func (cache *SpecialCollectionCache) Write(b []byte) (int, error)

type Writer

type Writer struct {
	Out     io.WriteCloser
	Prelude *Prelude
	Mux     *Multiplexer
}

Writer is the top level object to contain information about archives in mongodump

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL