filemgr

Documentation

Index

• type FileManager
  • func LoadFileManager(db *gorm.DB, alloc *datanode_alloc.DataNodeAllocator, ...) *FileManager
  • func (f *FileManager) CreateSheet(filename string) (uint64, error)
  • func (f *FileManager) DoCheckpoint()
  • func (f *FileManager) GetAllSheets() []*fs_rpc.Sheet
  • func (f *FileManager) HandleMasterEntry(entry *journal_entry.MasterEntry) error
  • func (f *FileManager) Monitor()
  • func (f *FileManager) OpenSheet(filename string) (uint64, error)
  • func (f *FileManager) Persistent() error
  • func (f *FileManager) ReadFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)
  • func (f *FileManager) ReadSheet(fd uint64) ([]*sheetfile.Chunk, error)
  • func (f *FileManager) RecycleSheet(filename string) error
  • func (f *FileManager) ResumeSheet(filename string) error
  • func (f *FileManager) WriteFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)

Types

type FileManager

type FileManager struct {

	// All directory entries in the directory. All of them should be loaded into memory once.
	Entries map[string]*mgr_entry.MapEntry
	// Maps filename to a already opened SheetFile. This map is fulfilled on-demand. If a SheetFile
	// is not being opened currently, it's not presented in the map.
	Opened map[string]*sheetfile.SheetFile
	// Maps a fd to a opened filename. Multiple fds are allowed to be pointed to the same file. So
	// their entries in this map will contain same filename.
	Fds map[uint64]string
	// contains filtered or unexported fields
}

FileManager Represents a top-level directory of SheetFiles stored in the filesystem. There is only one level of directory, mapping filename to SheetFile directly, or raise errors for invalid filename.

FileManager exposes both filename-oriented and fd-oriented API at the same time. The relationship between them is Unix-alike. In other words, applications need to Open a file with its filename, then a fd will be returned. Subsequently operations must be invoked by a fd. The management of fds is also Unix-alike: multiple fds can be pointed to the same underlying SheetFile. The only difference is that there is only a global fd namespace, not one for per client.

FileManager is responsible for implementing almost all APIs provided to outer applications. It's a goroutine-safe data structure. To maximize concurrency, it exploits a two-level locking strategy. It acquires itself RWMutex, mu, to lookup directory Entries or fd table safely. Then it release mu immediately, and rely on SheetFile.mu to guarantee goroutine-safe access to specific SheetFiles.

As a directory, FileManager should be persisted during checkpointing. Besides, it manages all SheetFiles at the same time, so it should not only persist itself, but also persist those files' metadata as a helper. It also plays as an in-memory cache for filesystem metadata of those files, loading them into memory on-demand. So a gorm connection is required.

func LoadFileManager

func LoadFileManager(db *gorm.DB, alloc *datanode_alloc.DataNodeAllocator, writer *common_journal.Writer) *FileManager

LoadFileManager Load all MapEntry from database and construct FileManager.Entries. Opened file table and fd table may be recovered from journal.

This method should only be used to load checkpoints in sqlite.

@para

db: a gorm connection. It can be a transaction

@return

*FileManager

func (*FileManager) CreateSheet

func (f *FileManager) CreateSheet(filename string) (uint64, error)

CreateSheet Creates a new SheetFile and open it immediately.

@para

filename

@return

uint64: fd of the newly created file.
error:
	*errors.FileExistsError if there has been a file with the same filename.
	Although the existing file has been recycled, creating a file with the
	same filename is not allowed.
	errors raised during sheetfile.CreateSheetFile

func (*FileManager) DoCheckpoint

func (f *FileManager) DoCheckpoint()

func (*FileManager) GetAllSheets

func (f *FileManager) GetAllSheets() []*fs_rpc.Sheet

GetAllSheets List all sheets stored in the directory.

@return

[]*fs_rpc.Sheet: a slice of protobuf fs_rpc.Sheet model, contains
filename and recycled field.

func (*FileManager) HandleMasterEntry

func (f *FileManager) HandleMasterEntry(entry *journal_entry.MasterEntry) error

func (*FileManager) Monitor

func (f *FileManager) Monitor()

Monitor Continuously monitoring all files marked as recycled, and if some file has exceeded configured retain time period, delete it forever.

func (*FileManager) OpenSheet

func (f *FileManager) OpenSheet(filename string) (uint64, error)

OpenSheet @para

filename

@return

uint64: fd to access this file if filename is valid
error:
	*errors.FileNotFoundError if the filename is invalid or file has been
	recycled.

func (*FileManager) Persistent

func (f *FileManager) Persistent() error

Persistent Flush the MapEntry and SheetFile data stored in a FileManager to sqlite in a transaction.

@return

error: error during the persistent transaction.

func (*FileManager) ReadFileCell

func (f *FileManager) ReadFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)

ReadFileCell Read Cell located at (row, col) in a file pointed by fd.

@para

fd, row, col

@return

*Cell, *Chunk: snapshots of corresponding Cell and Chunk
error:
	*errors.FdNotFoundError if the fd is invalid
	*errors.CellNotFoundError if row, col passed in is invalid.

func (*FileManager) ReadSheet

func (f *FileManager) ReadSheet(fd uint64) ([]*sheetfile.Chunk, error)

ReadSheet Read all contents of a SheetFile. This method returns all Chunk of a sheet, applications should then contact with DataNodes to fetch actual content of those Chunks.

@para

fd

@returns

[]*sheetfile.Chunk: List of all Chunks in a file.
error:
	*errors.FdNotFoundError if the fd is invalid

func (*FileManager) RecycleSheet

func (f *FileManager) RecycleSheet(filename string) error

RecycleSheet Mark a file as recycled and record RecycledAt. Do nothing if the filename is invalid.

The semantic of this method is Unix-alike too. It won't prevent applications who have opened this file from editing this file, but later Open or Create operations will fail.

func (*FileManager) ResumeSheet

func (f *FileManager) ResumeSheet(filename string) error

ResumeSheet Mark a file as not recycled, so it can be Opened afterwards.

func (*FileManager) WriteFileCell

func (f *FileManager) WriteFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)

WriteFileCell Write Cell located at (row, col) in a file pointed by fd. Create Cell if not existed.

@para

fd, row, col

@return

*Cell, *Chunk: snapshots of corresponding Cell and Chunk
error:
	*errors.FdNotFoundError if the fd is invalid
	*errors.NoDataNodeError if there is no DataNode registered.