esl

README

enchanted-sleeve

enchanted-sleeve is a key-value (KV) storage engine based on Log-Structured Merge-tree (LSM). It is designed for efficient storage and retrieval of key-value pairs with high write throughput. It is a simple key-value store that supports basic operations like get, put, delete, and list.

Enchanted sleeves is from the chinese myth item which called "蟒袖" that can store things in it and what the most important is that it store things more than its size, even a mountain.

Installation

Before installing enchanted-sleeve, make sure you have Go installed on your system. You can download Go from here.

Once you have Go installed, you can clone and install enchanted-sleeve with the following commands:

git clone https://github.com/yeqown/enchanted-sleeve.git
cd enchanted-sleeve
go build

This will build the enchanted-sleeve executable in the current directory.

Getting started

Usage

The repository's key methods include Open, Put, Get, Delete, and Close. Below is a brief look at how to use them to create and manipulate a key-value store.

// Create a new database instance with default options
db, err := esl.Open("path/to/directory")
if err != nil {
    log.Fatalf("failed to open database: %v", err)
}

defer db.Close()

// Set key-value pairs into the database
err = db.Put([]byte("key"), []byte("value"))
if err != nil {
    log.Fatalf("failed to put data: %v", err)
}

// Retrieve the value associated with the key
value, err := db.Get([]byte("key"))
if err != nil {
    log.Fatalf("failed to get data: %v", err)
}
fmt.Printf("key: %s", value)

To handle concurrent read and write operations, refer to the example in the examples/race directory. It demonstrates the use of goroutines to perform operations concurrently. Always use appropriate synchronization mechanisms like mutexes or channels to ensure thread safety in concurrent environments.

Testing

To run the tests included with enchanted-sleeve, make sure you have installed Go and configured your environment. Run the following command from the root of the project directory:

go test ./...

This will execute all tests across all packages in the repository.

Contributing

We welcome contributions to enchanted-sleeve! To contribute:

• Submit issues to report bugs or request features.
• Propose changes via pull requests (PRs), following the Pull Request process.
• Follow the code of conduct and the coding style guidelines of the project.

Always write meaningful commit messages and provide context for your changes.

Contact

For questions and support, please open an issue in the project's GitHub repository.

License

enchanted-sleeve is licensed under the MIT License - see the LICENSE file for details.

Documentation

Index

• Variables
• type DB
  • func Open(path string, options ...Option) (*DB, error)
  • func (db *DB) Close() error
  • func (db *DB) Delete(key []byte) error
  • func (db *DB) Get(key []byte) (value []byte, err error)
  • func (db *DB) ListKeys() []Key
  • func (db *DB) Merge() error
  • func (db *DB) Put(key, value []byte) error
  • func (db *DB) Sync()
• type FileSystem
• type Key
• type Option
  • func WithCompactInterval(compactInterval time.Duration) Option
  • func WithCompactThreshold(compactThreshold uint32) Option
  • func WithFileSystem(fs FileSystem) Option
  • func WithMaxFileBytes(maxFileBytes uint32) Option
  • func WithMaxKeyBytes(maxKeyBytes uint16) Option
  • func WithMaxValueBytes(maxValueBytes uint16) Option

Variables

var (
	ErrKeyOrValueTooLong  = errors.New("key or value is oversize")
	ErrKeyNotFound        = errors.New("key not found")
	ErrInvalidEntryHeader = errors.New("invalid entry header")
	ErrEntryCorrupted     = errors.New("entry corrupted")

	ErrInvalidKeydirData     = errors.New("invalid keydir data")
	ErrInvalidKeydirFileData = errors.New("invalid keydir file data")
)

Types

type DB

type DB struct {
	// contains filtered or unexported fields
}

DB is a simple key-value store backed by a log file, which is an append-only and immutable sequence of key-value pairs. The key-value pairs are stored in the log file in the order they are written. The log file is structured as follows:

| crc | tstamp | key_sz | value_sz | key | value | | crc | tstamp | key_sz | value_sz | key | value |

Since it's append-only, so modification and deletion would also append a new entry to overwrite old value.

func Open

func Open(path string, options ...Option) (*DB, error)

Open create or restore from the path.

func (*DB) Close

func (db *DB) Close() error

func (*DB) Delete

func (db *DB) Delete(key []byte) error

Delete removes the key from the DB. Note that the key is not actually removed from the DB, but marked as deleted, and the key will be removed from the DB when the DB is compacted.

func (*DB) Get

func (db *DB) Get(key []byte) (value []byte, err error)

func (*DB) ListKeys

func (db *DB) ListKeys() []Key

func (*DB) Merge

func (db *DB) Merge() error

Merge compacts the DB which is used by developer to reduce disk usage manually.

func (*DB) Put

func (db *DB) Put(key, value []byte) error

func (*DB) Sync

func (db *DB) Sync()

Sync force any writes to sync to disk

type FileSystem

type FileSystem = afero.Fs

FileSystem is the interface that wraps the basic methods for a file system, it is used by the file system abstraction layer to access, so that the default os file system can be replaced by other implementations.

It's useful for testing, since it can be replaced by a mock file system.

type Key

type Key []byte

type Option

type Option interface {
	// contains filtered or unexported methods
}

func WithCompactInterval

func WithCompactInterval(compactInterval time.Duration) Option

WithCompactInterval set the interval to check whether the compaction process should be triggered. NOTE: The interval is recommended to be greater than 1 minute. but it depends on the case of the application.

func WithCompactThreshold

func WithCompactThreshold(compactThreshold uint32) Option

WithCompactThreshold set the maximum number of files to keep.

func WithFileSystem

func WithFileSystem(fs FileSystem) Option

WithFileSystem set the file system to access.

func WithMaxFileBytes

func WithMaxFileBytes(maxFileBytes uint32) Option

WithMaxFileBytes set the maximum number of bytes for a single file.

func WithMaxKeyBytes

func WithMaxKeyBytes(maxKeyBytes uint16) Option

WithMaxKeyBytes set the maximum number of bytes for a single key.

func WithMaxValueBytes

func WithMaxValueBytes(maxValueBytes uint16) Option

WithMaxValueBytes set the maximum number of bytes for a single value.