Version: v1.0.0 Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jul 16, 2021 License: MIT Imports: 14 Imported by: 0




This section is empty.


This section is empty.


This section is empty.


type FileManager

type FileManager struct {

	// All directory entries in the directory. All of them should be loaded into memory once.
	Entries map[string]*mgr_entry.MapEntry
	// Maps filename to a already opened SheetFile. This map is fulfilled on-demand. If a SheetFile
	// is not being opened currently, it's not presented in the map.
	Opened map[string]*sheetfile.SheetFile
	// Maps a fd to a opened filename. Multiple fds are allowed to be pointed to the same file. So
	// their entries in this map will contain same filename.
	Fds map[uint64]string
	// contains filtered or unexported fields

FileManager Represents a top-level directory of SheetFiles stored in the filesystem. There is only one level of directory, mapping filename to SheetFile directly, or raise errors for invalid filename.

FileManager exposes both filename-oriented and fd-oriented API at the same time. The relationship between them is Unix-alike. In other words, applications need to Open a file with its filename, then a fd will be returned. Subsequently operations must be invoked by a fd. The management of fds is also Unix-alike: multiple fds can be pointed to the same underlying SheetFile. The only difference is that there is only a global fd namespace, not one for per client.

FileManager is responsible for implementing almost all APIs provided to outer applications. It's a goroutine-safe data structure. To maximize concurrency, it exploits a two-level locking strategy. It acquires itself RWMutex, mu, to lookup directory Entries or fd table safely. Then it release mu immediately, and rely on to guarantee goroutine-safe access to specific SheetFiles.

As a directory, FileManager should be persisted during checkpointing. Besides, it manages all SheetFiles at the same time, so it should not only persist itself, but also persist those files' metadata as a helper. It also plays as an in-memory cache for filesystem metadata of those files, loading them into memory on-demand. So a gorm connection is required.

func LoadFileManager

func LoadFileManager(db *gorm.DB, alloc *datanode_alloc.DataNodeAllocator, writer *common_journal.Writer) *FileManager

LoadFileManager Load all MapEntry from database and construct FileManager.Entries. Opened file table and fd table may be recovered from journal.

This method should only be used to load checkpoints in sqlite.


db: a gorm connection. It can be a transaction



func (*FileManager) CreateSheet

func (f *FileManager) CreateSheet(filename string) (uint64, error)

CreateSheet Creates a new SheetFile and open it immediately.




uint64: fd of the newly created file.
	*errors.FileExistsError if there has been a file with the same filename.
	Although the existing file has been recycled, creating a file with the
	same filename is not allowed.
	errors raised during sheetfile.CreateSheetFile

func (*FileManager) DoCheckpoint added in v1.0.0

func (f *FileManager) DoCheckpoint()

func (*FileManager) GetAllSheets

func (f *FileManager) GetAllSheets() []*fs_rpc.Sheet

GetAllSheets List all sheets stored in the directory.


[]*fs_rpc.Sheet: a slice of protobuf fs_rpc.Sheet model, contains
filename and recycled field.

func (*FileManager) HandleMasterEntry added in v1.0.0

func (f *FileManager) HandleMasterEntry(entry *journal_entry.MasterEntry) error

func (*FileManager) Monitor

func (f *FileManager) Monitor()

Monitor Continuously monitoring all files marked as recycled, and if some file has exceeded configured retain time period, delete it forever.

func (*FileManager) OpenSheet

func (f *FileManager) OpenSheet(filename string) (uint64, error)

OpenSheet @para



uint64: fd to access this file if filename is valid
	*errors.FileNotFoundError if the filename is invalid or file has been

func (*FileManager) Persistent

func (f *FileManager) Persistent() error

Persistent Flush the MapEntry and SheetFile data stored in a FileManager to sqlite in a transaction.


error: error during the persistent transaction.

func (*FileManager) ReadFileCell

func (f *FileManager) ReadFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)

ReadFileCell Read Cell located at (row, col) in a file pointed by fd.


fd, row, col


*Cell, *Chunk: snapshots of corresponding Cell and Chunk
	*errors.FdNotFoundError if the fd is invalid
	*errors.CellNotFoundError if row, col passed in is invalid.

func (*FileManager) ReadSheet

func (f *FileManager) ReadSheet(fd uint64) ([]*sheetfile.Chunk, error)

ReadSheet Read all contents of a SheetFile. This method returns all Chunk of a sheet, applications should then contact with DataNodes to fetch actual content of those Chunks.




[]*sheetfile.Chunk: List of all Chunks in a file.
	*errors.FdNotFoundError if the fd is invalid

func (*FileManager) RecycleSheet

func (f *FileManager) RecycleSheet(filename string) error

RecycleSheet Mark a file as recycled and record RecycledAt. Do nothing if the filename is invalid.

The semantic of this method is Unix-alike too. It won't prevent applications who have opened this file from editing this file, but later Open or Create operations will fail.

func (*FileManager) ResumeSheet

func (f *FileManager) ResumeSheet(filename string) error

ResumeSheet Mark a file as not recycled, so it can be Opened afterwards.

func (*FileManager) WriteFileCell

func (f *FileManager) WriteFileCell(fd uint64, row, col uint32) (*sheetfile.Cell, *sheetfile.Chunk, error)

WriteFileCell Write Cell located at (row, col) in a file pointed by fd. Create Cell if not existed.


fd, row, col


*Cell, *Chunk: snapshots of corresponding Cell and Chunk
	*errors.FdNotFoundError if the fd is invalid
	*errors.NoDataNodeError if there is no DataNode registered.

Source Files


Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL