archive

package
v0.0.0-...-ae3bf8c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Nov 16, 2017 License: Apache-2.0 Imports: 15 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	NamespaceUnopened = iota
	NamespaceOpened
	NamespaceClosed
)
View Source 
const MagicNumber uint32 = 0x8199e26d

MagicNumber is four bytes that are found at the beginning of the archive that indicate that the byte stream is an archive, as opposed to anything else, including a stream of BSON documents

Variables

This section is empty.

Functions

This section is empty.

Types

type CollectionMetadata 

type CollectionMetadata struct {
	Database   string `bson:"db"`
	Collection string `bson:"collection"`
	Metadata   string `bson:"metadata"`
	Size       int    `bson:"size"`
}

CollectionMetadata is a data structure that, as BSON, is found in the prelude of the archive. There is one CollectionMetadata per collection that will be in the archive.

type Demultiplexer 

type Demultiplexer struct {
	In io.Reader

	NamespaceChan      chan string
	NamespaceErrorChan chan error
	NamespaceStatus    map[string]int
	// contains filtered or unexported fields
}

Demultiplexer implements Parser.

func CreateDemux 

func CreateDemux(namespaceMetadatas []*CollectionMetadata, in io.Reader) *Demultiplexer

func (*Demultiplexer) BodyBSON 

func (demux *Demultiplexer) BodyBSON(buf []byte) error

BodyBSON is part of the ParserConsumer interface and receives BSON bodies from the parser. Its main role is to dispatch the body to the Read() function of the current DemuxOut.

func (*Demultiplexer) End 

func (demux *Demultiplexer) End() error

End is part of the ParserConsumer interface and receives the end of archive notification.

func (*Demultiplexer) HeaderBSON 

func (demux *Demultiplexer) HeaderBSON(buf []byte) error

HeaderBSON is part of the ParserConsumer interface and receives headers from parser. Its main role is to implement opens and EOFs of the embedded stream.

func (*Demultiplexer) NewPrioritizer 

func (demux *Demultiplexer) NewPrioritizer(mgr *intents.Manager) *Prioritizer

NewPrioritizer careates a new Prioritizer and hooks up its Namespace channels to the ones in demux

func (*Demultiplexer) Open 

func (demux *Demultiplexer) Open(ns string, out DemuxOut)

Open installs the DemuxOut as the handler for data for the namespace ns

func (*Demultiplexer) Run 

func (demux *Demultiplexer) Run() error

Run creates and runs a parser with the Demultiplexer as a consumer

type DemuxOut 

type DemuxOut interface {
	Write([]byte) (int, error)
	Close() error
	Sum64() (uint64, bool)
}

DemuxOut is a Demultiplexer output consumer The Write() and Close() occur in the same thread as the Demultiplexer runs in.

type DirLike 

type DirLike interface {
	Name() string
	Path() string
	Size() int64
	IsDir() bool
	Stat() (DirLike, error)
	ReadDir() ([]DirLike, error)
	Parent() DirLike
}

DirLike represents the group of methods done on directories and files in dump directories, or in archives, when mongorestore is figuring out what intents to create. 

type Header struct {
	ConcurrentCollections int32  `bson:"concurrent_collections"`
	FormatVersion         string `bson:"version"`
	ServerVersion         string `bson:"server_version"`
	ToolVersion           string `bson:"tool_version"`
}

Header is a data structure that, as BSON, is found immediately after the magic number in the archive, before any CollectionMetadatas. It is the home of any archive level information

type MetadataFile 

type MetadataFile struct {
	*bytes.Buffer
	Intent *intents.Intent
	// contains filtered or unexported fields
}

MetadataFile implements intents.file

func (*MetadataFile) Close 

func (md *MetadataFile) Close() error

func (*MetadataFile) Open 

func (md *MetadataFile) Open() error

func (*MetadataFile) Pos 

func (md *MetadataFile) Pos() int64

func (*MetadataFile) Read 

func (md *MetadataFile) Read(p []byte) (int, error)

type MetadataPreludeFile 

type MetadataPreludeFile struct {
	Intent  *intents.Intent
	Origin  string
	Prelude *Prelude
	*bytes.Buffer
	// contains filtered or unexported fields
}

MetadataPreludeFile is part of the intents.file. It allows the metadata contained in the prelude to be opened and read

func (*MetadataPreludeFile) Close 

func (mpf *MetadataPreludeFile) Close() error

Close is part of the intents.file interface.

func (*MetadataPreludeFile) Open 

func (mpf *MetadataPreludeFile) Open() error

Open is part of the intents.file interface, it finds the metadata in the prelude and creates a bytes.Buffer from it.

func (*MetadataPreludeFile) Pos 

func (mpf *MetadataPreludeFile) Pos() int64

func (*MetadataPreludeFile) Read 

func (mpf *MetadataPreludeFile) Read(p []byte) (int, error)

type Multiplexer 

type Multiplexer struct {
	Out       io.WriteCloser
	Control   chan *MuxIn
	Completed chan error
	// contains filtered or unexported fields
}

Multiplexer is what one uses to create interleaved intents in an archive

func NewMultiplexer 

func NewMultiplexer(out io.WriteCloser, shutdownInputs notifier) *Multiplexer

NewMultiplexer creates a Multiplexer and populates its Control/Completed chans it takes a WriteCloser, which is where in imputs will get multiplexed on to, and it takes a notifier, which should allow the multiplexer to ask for the shutdown of the inputs.

func (*Multiplexer) Run 

func (mux *Multiplexer) Run()

Run multiplexes until it receives an EOF on its Control chan.

type MutedCollection 

type MutedCollection struct {
	Intent *intents.Intent
	Demux  *Demultiplexer
}

MutedCollection implements both DemuxOut as well as intents.file. It serves as a way to let the demutiplexer ignore certain embedded streams

func (*MutedCollection) Close 

func (*MutedCollection) Close() error

Close is part of the intents.file interface, and does nothing

func (*MutedCollection) Open 

func (*MutedCollection) Open() error

Open is part of the intents.file interface, and does nothing

func (*MutedCollection) Read 

func (*MutedCollection) Read([]byte) (int, error)

Read is part of the intents.file interface, and does nothing

func (*MutedCollection) Sum64 

func (*MutedCollection) Sum64() (uint64, bool)

Sum64 is part of the DemuxOut interface

func (*MutedCollection) Write 

func (*MutedCollection) Write(b []byte) (int, error)

Write is part of the intents.file interface, and does nothing

type MuxIn 

type MuxIn struct {
	Intent *intents.Intent
	Mux    *Multiplexer
	// contains filtered or unexported fields
}

MuxIn is an implementation of the intents.file interface. They live in the intents, and are potentially owned by different threads than the thread owning the Multiplexer. They are out the intents write data to the multiplexer

func (*MuxIn) Close 

func (muxIn *MuxIn) Close() error

Close closes the chans in the MuxIn. Ultimately the multiplexer will detect that they are closed and cause a formatEOF to occur.

func (*MuxIn) Open 

func (muxIn *MuxIn) Open() error

Open is implemented in Mux.open, but in short, it creates chans and a select case and adds the SelectCase and the MuxIn in to the Multiplexer.

func (*MuxIn) Pos 

func (muxIn *MuxIn) Pos() int64

func (*MuxIn) Read 

func (muxIn *MuxIn) Read([]byte) (int, error)

Read does nothing for MuxIns

func (*MuxIn) Write 

func (muxIn *MuxIn) Write(buf []byte) (int, error)

Write hands a buffer to the Multiplexer and receives a written length from the multiplexer after the length is received, the buffer is free to be reused.

type NamespaceHeader 

type NamespaceHeader struct {
	Database   string `bson:"db"`
	Collection string `bson:"collection"`
	EOF        bool   `bson:"EOF"`
	CRC        int64  `bson:"CRC"`
}

NamespaceHeader is a data structure that, as BSON, is found in archives where it indicates that either the subsequent stream of BSON belongs to this new namespace, or that the indicated namespace will have no more documents (EOF)

type Parser 

type Parser struct {
	In io.Reader
	// contains filtered or unexported fields
}

Parser encapsulates the small amount of state that the parser needs to keep

func (*Parser) ReadAllBlocks 

func (parse *Parser) ReadAllBlocks(consumer ParserConsumer) (err error)

ReadAllBlocks calls ReadBlock() until it returns an error. If the error is EOF, then nil is returned, otherwise it returns the error

func (*Parser) ReadBlock 

func (parse *Parser) ReadBlock(consumer ParserConsumer) (err error)

ReadBlock reads one archive block ( header + body* + terminator ) calling consumer.HeaderBSON() on the header, consumer.BodyBSON() on each piece of body, and consumer.EOF() when EOF is encountered before any data was read. It returns nil if a whole block was read, io.EOF if nothing was read, and a parserError if there was any io error in the middle of the block, if either of the consumer methods return error, or if there was any sort of parsing failure.

type ParserConsumer 

type ParserConsumer interface {
	HeaderBSON([]byte) error
	BodyBSON([]byte) error
	End() error
}

ParserConsumer is the interface that one needs to implement to consume data from the Parser

type Prelude 

type Prelude struct {
	Header                 *Header
	DBS                    []string
	NamespaceMetadatas     []*CollectionMetadata
	NamespaceMetadatasByDB map[string][]*CollectionMetadata
}

Prelude represents the knowledge gleaned from reading the prelude out of the archive.

func NewPrelude 

func NewPrelude(manager *intents.Manager, concurrentColls int, serverVersion string) (*Prelude, error)

NewPrelude generates a Prelude using the contents of an intent.Manager.

func (*Prelude) AddMetadata 

func (prelude *Prelude) AddMetadata(cm *CollectionMetadata)

AddMetadata adds a metadata data structure to a prelude and does the required bookkeeping.

func (*Prelude) NewPreludeExplorer 

func (prelude *Prelude) NewPreludeExplorer() (*PreludeExplorer, error)

NewPreludeExplorer creates a PreludeExplorer from a Prelude.

func (*Prelude) Read 

func (prelude *Prelude) Read(in io.Reader) error

Read consumes and checks the magic number at the beginning of the archive, then it runs the parser with a Prelude as its consumer.

func (*Prelude) Write 

func (prelude *Prelude) Write(out io.Writer) error

Write writes the archive header.

type PreludeExplorer 

type PreludeExplorer struct {
	// contains filtered or unexported fields
}

PreludeExplorer implements DirLike. PreludeExplorer represent the databases, collections, and their metadata json files, of an archive, in such a way that they can be explored like a filesystem.

func (*PreludeExplorer) IsDir 

func (pe *PreludeExplorer) IsDir() bool

IsDir is part of the DirLike interface. All pes that are not collections are Dirs.

func (*PreludeExplorer) Name 

func (pe *PreludeExplorer) Name() string

Name is part of the DirLike interface. It synthesizes a filename for the given "location" the prelude.

func (*PreludeExplorer) Parent 

func (pe *PreludeExplorer) Parent() DirLike

Parent is part of the DirLike interface. It returns a pe without a collection, if there is one, otherwise, without a database.

func (*PreludeExplorer) Path 

func (pe *PreludeExplorer) Path() string

Path is part of the DirLike interface. It creates the full path for the "location" in the prelude.

func (*PreludeExplorer) ReadDir 

func (pe *PreludeExplorer) ReadDir() ([]DirLike, error)

ReadDir is part of the DirLIke interface. ReadDir generates a list of PreludeExplorers whose "locations" are encapsulated by the current pes "location". 

"dump/oplog.bson"     => &PreludeExplorer{ database: "", collection: "oplog.bson" }
"dump/test/"          => &PreludeExplorer{ database: "test", collection: "foo.bson" }
"dump/test/foo.bson"  => &PreludeExplorer{ database: "test", collection: "" }
"dump/test/foo.json"  => &PreludeExplorer{ database: "test", collection: "foo", isMetadata: true }

func (*PreludeExplorer) Size 

func (pe *PreludeExplorer) Size() int64

Size is part of the DirLike interface. It returns the size from the metadata of the prelude, if the "location" is a collection.

func (*PreludeExplorer) Stat 

func (pe *PreludeExplorer) Stat() (DirLike, error)

Stat is part of the DirLike interface. os.Stat returns a FileInfo, and since DirLike is similar to FileInfo, we just return the pe, here.

type Prioritizer 

type Prioritizer struct {
	NamespaceChan      <-chan string
	NamespaceErrorChan chan<- error
	// contains filtered or unexported fields
}

Prioritizer is a completely reactive prioritizer Intents are handed out as they arrive in the archive

func (*Prioritizer) Finish 

func (prioritizer *Prioritizer) Finish(*intents.Intent)

Finish is part of the IntentPrioritizer interface, and does nothing

func (*Prioritizer) Get 

func (prioritizer *Prioritizer) Get() *intents.Intent

Get waits for a new namespace from the NamespaceChan, and returns a Intent found for it

type Reader 

type Reader struct {
	In      io.ReadCloser
	Demux   *Demultiplexer
	Prelude *Prelude
}

Reader is the top level object to contain information about archives in mongorestore

type RegularCollectionReceiver 

type RegularCollectionReceiver struct {
	Intent *intents.Intent
	Origin string
	Demux  *Demultiplexer
	// contains filtered or unexported fields
}

RegularCollectionReceiver implements the intents.file interface.

func (*RegularCollectionReceiver) Close 

func (receiver *RegularCollectionReceiver) Close() error

Close is part of the DemuxOut as well as the intents.file interface. It only closes the readLenChan, as that is what will cause the RegularCollectionReceiver.Read() to receive EOF Close will get called twice, once in the demultiplexer, and again when the restore goroutine is done with its intent.file

func (*RegularCollectionReceiver) Open 

func (receiver *RegularCollectionReceiver) Open() error

Open is part of the intents.file interface. It creates the chan's in the RegularCollectionReceiver and adds the RegularCollectionReceiver to the set of RegularCollectonReceivers in the demultiplexer

func (*RegularCollectionReceiver) Pos 

func (receiver *RegularCollectionReceiver) Pos() int64

func (*RegularCollectionReceiver) Read 

func (receiver *RegularCollectionReceiver) Read(r []byte) (int, error)

Read() runs in the restoring goroutine

func (*RegularCollectionReceiver) ReleaseIOBuffer 

func (receiver *RegularCollectionReceiver) ReleaseIOBuffer()

func (*RegularCollectionReceiver) Sum64 

func (receiver *RegularCollectionReceiver) Sum64() (uint64, bool)

func (*RegularCollectionReceiver) TakeIOBuffer 

func (receiver *RegularCollectionReceiver) TakeIOBuffer(ioBuf []byte)

func (*RegularCollectionReceiver) Write 

func (receiver *RegularCollectionReceiver) Write(buf []byte) (int, error)

Write is part of the DemuxOut interface.

type SpecialCollectionCache 

type SpecialCollectionCache struct {
	Intent *intents.Intent
	Demux  *Demultiplexer
	// contains filtered or unexported fields
}

SpecialCollectionCache implemnts both DemuxOut as well as intents.file

func NewSpecialCollectionCache 

func NewSpecialCollectionCache(intent *intents.Intent, demux *Demultiplexer) *SpecialCollectionCache

func (*SpecialCollectionCache) Close 

func (cache *SpecialCollectionCache) Close() error

Close is part of the both interfaces, and it does nothing

func (*SpecialCollectionCache) Open 

func (cache *SpecialCollectionCache) Open() error

Open is part of the both interfaces, and it does nothing

func (*SpecialCollectionCache) Pos 

func (cache *SpecialCollectionCache) Pos() int64

func (*SpecialCollectionCache) Read 

func (cache *SpecialCollectionCache) Read(p []byte) (int, error)

func (*SpecialCollectionCache) Sum64 

func (cache *SpecialCollectionCache) Sum64() (uint64, bool)

func (*SpecialCollectionCache) Write 

func (cache *SpecialCollectionCache) Write(b []byte) (int, error)

type Writer 

type Writer struct {
	Out     io.WriteCloser
	Prelude *Prelude
	Mux     *Multiplexer
}

Writer is the top level object to contain information about archives in mongodump

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.