README
¶
Write-Ahead Log (WAL) Library
Overview
The Write-Ahead Log (WAL) library is a simple, production-ready Go package that implements a Write-Ahead Log for database applications. It allows you to efficiently store and replay database changes for crash recovery and replication purposes. The WAL is broken down into smaller segments, enabling log rotation and space management.
Features
- Efficient and safe Write-Ahead Log for databases
- Log rotation and space management with smaller segments
- Support for crash recovery and replication scenarios
- Monotonically incrementing offset for log replay
Installation
To use the Write-Ahead Log library, you need Go version 1.17 or above. Install it using the following command:
go get github.com/aarthikrao/wal
Usage
Here's a simple example demonstrating how to use the Write-Ahead Log library:
package main
import (
"fmt"
wal "github.com/aarthikrao/wal"
"go.uber.org/zap"
)
func main() {
log, err := zap.NewProduction()
if err != nil {
panic(err)
}
wal, err := NewWriteAheadLog(&WALOptions{
LogDir: "data/",
MaxLogSize: 40 * 1024 * 1024, // 400 MB (log rotation size)
MaxSegments: 2,
Log: log,
MaxWaitBeforeSync: 1 * time.Second,
SyncMaxBytes: 1000,
})
if err != nil {
fmt.Println("Error creating Write-Ahead Log:", err)
return
}
defer wal.Close()
for i := 0; i < 10000000; i++ {
_, err := wal.Write([]byte("Simulate some database changes and write them to the WAL"))
if err != nil {
panic(err)
}
}
// Sync all the logs to the file at the end so that we dont loose any data
err = wal.Sync()
if err != nil {
fmt.Println("Error in final sync", err)
}
startTime := time.Now()
var numLines int
wal.Replay(0, func(b []byte) error {
numLines++
return nil
})
fmt.Println("Total lines replayed:", numLines)
fmt.Println("time taken:", time.Since(startTime))
}
TODO
- Start from where we left off during restart
- Test cases
- More comments for common people
License
This project is licensed under the MIT License - see the LICENSE file for details.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type WALOptions ¶
type WALOptions struct {
// LogDir is where the wal logs will be stored
LogDir string
// Maximum size in bytes for each file
MaxLogSize int64
// The entire wal is broken down into smaller segments.
// This will be helpful during log rotation and management
// maximum number of log segments
MaxSegments int
MaxWaitBeforeSync time.Duration
SyncMaxBytes int64
Log *zap.Logger
}
type WriteAheadLog ¶
type WriteAheadLog struct {
// contains filtered or unexported fields
}
A Write Ahead Log (WAL) is a data structure used to record changes to a database or any persistent storage system in a sequential and durable manner. This allows for crash recovery and data integrity.
func NewWriteAheadLog ¶
func NewWriteAheadLog(opts *WALOptions) (*WriteAheadLog, error)
NewWriteAheadLog creates a new instance of the WriteAheadLog with the provided options.
func (*WriteAheadLog) Close ¶
func (wal *WriteAheadLog) Close() error
Close closes the underneath storage file, it flushes data remaining in the memory buffer and file systems in-memory copy of recently written data to file to ensure persistent commit of the log
func (*WriteAheadLog) GetOffset ¶
func (wal *WriteAheadLog) GetOffset() int64
GetOffset returns the current log offset.
func (*WriteAheadLog) Replay ¶
func (wal *WriteAheadLog) Replay(offset int64, f func([]byte) error) error
Replay replays log entries starting from the specified offset, invoking the provided callback.
func (*WriteAheadLog) Sync ¶ added in v0.0.4
func (wal *WriteAheadLog) Sync() error
Sync writes all the data to the disk ensuring data durability. Since it is a expensive call, calling this often will slow down the throughput.
Source Files