history

package module
v0.0.0-...-8c5321b Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 14, 2025 License: MIT Imports: 23 Imported by: 0

README ΒΆ

nntp-history - High-Performance NNTP History Management

nntp-history is a Go module designed for managing and storing NNTP (Network News Transfer Protocol) history records efficiently with support for both MySQL RocksDB and SQLite3.

It provides a scalable way to store and retrieve historical message records with duplicate detection and high-throughput capabilities.

This module is suitable for building applications related to Usenet news servers or any system that requires managing message history efficiently.

πŸš€ Features

  • Dual Backend Support: MySQL RocksDB and SQLite3 with RocksDB-style optimizations
  • High Performance: Optimized for millions of records with minimal latency
  • Duplicate Detection: Efficient hash-based duplicate checking with L1 cache
  • Connection Pooling: Efficient database connection management
  • Concurrent Processing: Multi-threaded design for high throughput
  • Memory Mapped I/O: SQLite3 backend uses memory mapping for performance

πŸ“¦ Installation

go get github.com/go-while/nntp-history

πŸ”§ Database Backends

  • File-based: Simple deployment and backup
  • RocksDB Optimizations: WAL mode, memory mapping, large caches
  • 4096 Table Sharding: Optimal distribution using 3-character hex prefixes
  • Always Available: No build tags required
MySQL RocksDB (Enterprise use cases)
  • Distributed: Suitable for cluster environments
  • High Concurrency: Supports more concurrent connections
  • Complex Setup: Requires MySQL with RocksDB storage engine

πŸ—οΈ Quick Start

Initialize with SQLite3 (Default)
import "github.com/go-while/nntp-history"

his := &history.HISTORY{DIR: "/path/to/history"}
err := his.InitializeDatabase(false) // false = SQLite3
if err != nil {
    log.Fatal(err)
}
Initialize with MySQL RocksDB
his := &history.HISTORY{DIR: "/path/to/history"}
err := his.InitializeDatabase(true) // true = MySQL
if err != nil {
    log.Fatal(err)
}

πŸ“Š Performance Comparison & Sharding

The nntp-history module supports multiple SQLite3 sharding strategies to optimize for different workloads. Detailed benchmarks with 1 million hash insertions show:

Sharding Mode Insertion Rate (1M) Go Heap RAM (Post GC) SQLite Cache (Est. Max Potential) Initialization Time Key Characteristics
Mode 0 (1 DB, 4096 tables) ~333K/sec ~0-1 MB ~400 MB (Shared) 8.55s Simplicity, best for read-heavy (large shared cache)
Mode 2 (16 DBs, 256 tables) ~332K/sec ~0-1 MB ~112 MB (Adaptive) 2.42s Recommended Default: Fast init, balanced R/W, good concurrency
Mode 3 (64 DBs, 64 tables) ~322K/sec ~0-1 MB ~192 MB (Adaptive) 3.05s Write-focused, higher concurrency needs

Key Takeaways from 1M Hash Tests:

  • High Write Performance: All tested modes exceed 320,000 inserts/second.
  • Low Go Application RAM: Go's heap usage is minimal (~1MB after GC) due to SQLite's memory management.
  • Adaptive SQLite Cache: SQLite's page cache is managed efficiently. Multi-DB modes use adaptive sizing (e.g., Mode 2 estimates up to 112MB total cache if fully utilized, Mode 3 up to 192MB). This is OS-managed memory, not Go heap.
  • Fast Initialization for Multi-DB: Mode 2 (16 DBs) initializes significantly faster than Mode 0 (single DB with many tables).

Recommendation:

  • Mode 2 (16 DBs, 256 tables each) is recommended for most use cases, offering a strong balance of fast startup, high throughput, and efficient memory utilization.
  • Use Mode 0 if simplicity is paramount or for predominantly read-heavy workloads that benefit from a single, large shared cache.
  • Consider Mode 3 or other multi-DB options for very specific high-concurrency write scenarios.

Refer to examples/README.md and sqlite-sharded-integration-log.md for more detailed analysis.

  • Test Suite: nntp-history_test - Comprehensive testing and benchmarking
  • Examples: See examples/ directory for usage examples

πŸ§ͺ Testing & Benchmarking

For comprehensive testing and benchmarking, use the dedicated test repository: nntp-history_test

Quick Benchmark Examples
# Clone the test repository
git clone https://github.com/go-while/nntp-history_test
cd nntp-history_test

# Build and run basic test
./build.sh
./nntp-history-test -todo=100000

# Performance test with SQLite3
./nntp-history-test -backend=sqlite3 -todo=1000000

# High-volume duplicate detection test
./nntp-history-test -todo=1000000 -DBG_BS_LOG=true
Performance Notes
  • Performance varies based on hardware, database configuration, and workload patterns
  • Both SQLite3 and MySQL RocksDB backends are optimized for high-throughput duplicate detection
  • Run your own benchmarks with the test repository for accurate measurements on your system
  • See the nntp-history_test repository for comprehensive testing tools

οΏ½ Storage Size Estimates

Based on the actual code structure, here are precise storage calculations for planning:

πŸ“„ history.dat File Size

Each record in history.dat has a fixed format:

{sha256hash}\ttimestamp~expires~date\tstoragetoken\n

Per record breakdown:

  • { + } = 2 bytes (hash wrapper)
  • SHA256 hash = 64 bytes
  • \t = 1 byte (tab)
  • Arrival timestamp = 10 bytes (zero-padded)
  • ~ = 1 byte
  • Expires = 10 bytes (zero-padded or ----------)
  • ~ = 1 byte
  • Date = 10 bytes (zero-padded)
  • \t = 1 byte (tab)
  • StorageToken = 1 byte (typically "F")
  • \n = 1 byte (newline)

Total per record: 102 bytes

πŸ—„οΈ Database Size (SQLite3 with keylen=7)

Table Structure:

  • 4096 tables (s000 to sfff) using 3-character hex prefixes
  • Each table stores: 7-char key + comma-separated offsets
  • Distribution: ~256 records per table (1M Γ· 4096)

Storage Components:

  • Keys storage: ~7 bytes per record
  • Offsets storage: ~8 bytes per record (low collision rate)
  • SQLite overhead: ~12 bytes per record
  • Table/index overhead: ~5 MB total
πŸ“Š Size Summary
Records history.dat SQLite3 DB Total Storage
1M 102 MB 32 MB 134 MB
10M 1.02 GB 320 MB 1.34 GB
100M 10.2 GB 3.2 GB 13.4 GB
πŸ” Key Performance Factors
  • keylen=7: Uses first 10 chars (3 for table + 7 for key)
  • Low Collision Rate: 10-char prefixes provide excellent distribution
  • Efficient Storage: SQLite3 WITHOUT ROWID tables minimize overhead
  • Linear Scalability: Storage grows predictably with record count

οΏ½πŸ”¨ Building

# Simple build (SQLite3 included by default)
go build

# Using Makefile
make build

# Get dependencies
make deps

πŸ“– Documentation

  • SQLite3 Backend: See SQLITE3_README.md for detailed SQLite3 documentation
  • API Reference: See inline documentation in source files

Code Hints

BootHistory Function

The BootHistory function in this Go code is responsible for initializing and booting a history management system.

It provides essential configuration options and prepares the system for historical data storage and retrieval.

Usage

To use the BootHistory function, follow these steps:

  1. Call the BootHistory function with the desired configuration options.

  2. The history management system will be initialized and ready for use.

history.History.WriterChan

  • history.History.WriterChan is a Go channel used for sending and processing historical data entries.

  • It is primarily responsible for writing data to a historical data storage system, using a HashDB (SQLite3/MySQL) to avoid duplicate entries.

  • To send data for writing, you create a HistoryObject and send it through the channel.

  • If the ResponseChan channel is provided, it receives one of the following (int) values:

  /*
  0: Indicates "not a duplicate."
  1: Indicates "duplicate."
  2: Indicates "retry later."
  */

history.History.IndexChan

  • The history.History.IndexChan is a Go channel with a dual purpose.

  • Its primary function is to facilitate the detection of duplicate message-ID hashes within the history file.

  • When the offset is set to -1, the channel performs a check for duplicate hashes but does not add the hash to the database.

  • When the offset is set to a value greater than zero, the channel functions as a mechanism for adding these message-ID hashes to the database.

  • Beware: Adding message-ID hashes is then normally done via history.History.WriterChan if you want to write the history file too!

  • If desired, one could only use the IndexChan and avoid writing the history file. Use the full KeyLen of hash and provide a uniq up-counter for their Offsets.

  • If the IndexRetChan channel is provided, it receives one of the following (int) values:

  /*
  0: Indicates "not a duplicate."
  1: Indicates "duplicate."
  2: Indicates "retry later."
  */

Message-ID Hash Distribution with SQLite3

KeyAlgo (HashShort)

  • The standard key algorithm used is HashShort (-keyalgo=11).

Database Organization

To improve processing speed and optimize data storage, we organize data using SQLite3 table sharding based on hash prefixes:

  • We create 4096 separate tables (s000 to sfff), each corresponding to a 3-character hexadecimal prefix.

  • This provides optimal distribution and parallel access for high-performance operations.

Key Structure

The hash distribution uses a sophisticated sharding approach:

  • The first 3 characters of the hash determine the table (s000-sfff)
  • The next 7 characters (or KeyLen setting) are used as the key within that table
  • The recommended KeyLen is 7. The minimum KeyLen is 1. The maximum KeyLen is the length of the hash -3.
  • Reasonable values for KeyLen range from 4 to 7. Use higher values if you expect more than 100M messages.
  • Choose wisely. You can not change KeyLen after database creation.

*** These are outdated benchmarks from the previous BoltDB implementation ***
*** SQLite3 backend provides better performance and simpler management ***

# With SQLite3 sharding (4096 tables), collision rates are significantly lower:
# - keylen=7: Optimal balance of performance and storage efficiency
# - keylen=4-6: Higher collision rates, more multi-offset storage
# - keylen=8+: Near-zero collisions, maximum storage usage

# Run your own benchmarks with nntp-history_test for current performance metrics

Example

Let's illustrate this approach with an example:

Suppose you have a Message-ID hash of "1a2b3c4d5e6f0...":

  • We use the first 3 characters "1a2" to select the SQLite3 table "s1a2"
  • The next 7 characters "b3c4d5e" (based on KeyLen=7) are used as the key within that table
  • The full hash is stored in history.dat file and database holds the offsets as comma-separated for that key

By following this approach, you can efficiently organize and retrieve data based on Message-ID hashes while benefiting from the performance and storage optimizations provided by SQLite3 with RocksDB-style optimizations.

Feel free to customize the KeyLen setting to meet your specific performance and storage requirements.

Smaller KeyLen values save space but may result in more collisions and multi-offset storage, while larger KeyLen values reduce collisions but consume more space.

File Descriptors

SQLite3 backend uses significantly fewer file descriptors:

  • Writer History.dat: 1
  • SQLite3 Database: 1 (main database file)
  • SQLite3 WAL: 1 (write-ahead log)
  • Total: ~3-5 file descriptors (much more efficient than the previous 33 FDs)
# Monitor open file descriptors
ls -lha /proc/$(pidof nntp-history-test)/fd

# Count all open FDs
ls -l /proc/$(pidof nntp-history-test)/fd | wc -l

# Count open history.dat
ls -lha /proc/$(pidof nntp-history-test)/fd|grep history.dat$|wc -l

# Count open SQLite3 database files
ls -lha /proc/$(pidof nntp-history-test)/fd|grep "\.sqlite3"|wc -l

SQLite3 Database Statistics

When you retrieve and examine the statistics of a SQLite3 database in Go, you can gather valuable information about the database's performance, resource usage, and structure. The statistics provide insights into how the database is operating and can be useful for optimizing your application. Here are some of the key metrics you can see from SQLite3 database statistics:

Database Size Metrics:

  • Total database file size in bytes
  • Number of tables and their individual sizes
  • WAL (Write-Ahead Log) file size and checkpoint frequency
  • Page count and page utilization

Performance Metrics:

  • Cache hit/miss ratios for database pages
  • Query execution times and optimization statistics
  • Transaction throughput and commit rates
  • Lock contention and wait times

Storage Efficiency:

  • Table and index storage utilization
  • Fragmentation levels and vacuum effectiveness
  • Compression ratios (if enabled)
  • Free space and page reuse statistics

Connection and Concurrency:

  • Active connection count
  • Concurrent reader/writer statistics
  • Transaction isolation levels and conflicts
  • Checkpoint frequency and WAL size management

SQLite3-Specific Features:

  • Memory-mapped I/O effectiveness
  • Journal mode efficiency (WAL vs. DELETE)
  • PRAGMA settings impact on performance
  • Query planner statistics and index usage

These statistics can help you monitor and optimize your SQLite3 database. For example, you can use them to identify performance bottlenecks, understand resource usage patterns, and assess the efficiency of your data organization. The SQLite3 backend in nntp-history is optimized with RocksDB-style settings including WAL mode, large caches, and memory mapping for maximum performance.

Contributing

Contributions to this code are welcome.

If you have suggestions for improvements or find issues, please feel free to open an issue or submit a pull request.

License

This code is provided under the MIT License. See the LICENSE file for details.

Documentation ΒΆ

Index ΒΆ

Constants ΒΆ

View Source 
const (
	// never change this
	FlagExpires      bool = true
	FlagNeverExpires bool = false
)
View Source 
const (
	CR                = "\r"
	LF                = "\n"
	CRLF              = CR + LF
	DefaultSocketPath = "./history.socket"
	// default launches a tcp port with a telnet interface @ localhost:49119
	DefaultServerTCPAddr = "[::]:49119"
)
View Source 
const (
	SHARD_SINGLE_DB  = 0 // 1 DB with 4096 tables (current default)
	SHARD_FULL_SPLIT = 1 // 4096 separate DBs
	SHARD_16_256     = 2 // 16 DBs with 256 tables each
	SHARD_64_64      = 3 // 64 DBs with 64 tables each
	SHARD_128_32     = 4 // 128 DBs with 32 tables each
	SHARD_512_8      = 5 // 512 DBs with 8 tables each
)

Sharding configuration constants

View Source 
const (
	HashShort = 0x0B // 11
	//KeyIndex   = 0
	KeyLen      = 7    // Fixed key length for MySQL 3-level hex structure (7 chars after 3-char table prefix)
	NumCacheDBs = 4096 // Changed from 16 to 4096 for 3-level hex (16^3 = 4096)
	ALWAYS      = true
	// DefExpiresStr use 10 digits as spare so we can update it later without breaking offsets
	DefExpiresStr string = "----------" // never expires
	CaseLock             = 0xFF         // internal cache state. reply with CaseRetry while CaseLock
	CasePass             = 0xF1         // is a reply to L1Lock and IndexQuery
	CaseDupes            = 0x1C         // is a reply and cache state
	CaseRetry            = 0x2C         // is a reply to if CaseLock or CaseWrite or if history.dat returns EOF
	CaseAdded            = 0x3C         // is a reply to WriterChan:responseChan
	CaseWrite            = 0x4C         // internal cache state. is not a reply. reply with CaseRetry while CaseWrite is happening
	CaseError            = 0xE1         // some things drop this error
	ZEROPADLEN           = 0xFFF        // zeropads the header

)
View Source 
const (
	FlagSearch = -1
)
View Source 
const (
	MinRetryWaiter = 100
)

Variables ΒΆ

View Source 
var (
	DBG_CGS               bool       // DEBUG_CACHE_GROW_SHRINK
	DefaultCacheExpires   int64 = 5  // gets x2 BatchFlushEvery x2
	DefaultCacheExtend    int64 = 5  // extends cached items after writes
	DefaultCachePurge     int64 = 1  // checks ttl every N seconds. affects CacheExpires/Extend max to + Purge
	DefaultEvictsCapacity       = 16 // his.cEvCap (size of Extend chan) is normally fine as is.
	ClearEveryN                 = 16
)
View Source 
var (
	BootHisCli           bool
	DefaultHistoryServer = "[::1]:49119" // localhost:49119
	// set only once before boot
	TCPchanQ           = 128
	DefaultDialTimeout = 5   // seconds
	DefaultRetryWaiter = 500 // milliseconds
	DefaultDialRetries = -1  // try N times and fail or <= 0 enables infinite retry
)
View Source 
var (
	DEBUGL1         bool  = true
	L1              bool  = true // better not disable L1 cache...
	L1CacheExpires  int64 = DefaultCacheExpires
	L1ExtendExpires int64 = DefaultCacheExtend
	L1Purge         int64 = DefaultCachePurge
	L1InitSize      int   = 128

	// L1LockDelay: delays L1 locking by N milliseconds
	// L1 locking is most likely done per client-connection
	// settings this greater 0 limits the amount of articles a client can lock&send
	//    1ms is a max of 1000 messages/sec per conn
	//  100ms is a max of   10 messages/sec per conn
	//  250ms is a max of    4 messages/sec per conn
	// 1000ms is a max of    1 message /sec per conn
	// text peers mostly dont need more than 4 msg per sec
	L1LockDelay int = 0
)
View Source 
var (
	DEBUGL2         bool  = false
	L2              bool  = true
	L2CacheExpires  int64 = DefaultCacheExpires
	L2ExtendExpires int64 = DefaultCacheExtend
	L2Purge         int64 = DefaultCachePurge
	L2InitSize      int   = 64 * 1024
)

L2Cache: offset => hash less requests to hisDat

View Source 
var (
	DEBUGL3         bool  = false
	L3              bool  = true // do not disable!
	L3CacheExpires  int64 = DefaultCacheExpires
	L3ExtendExpires int64 = DefaultCacheExtend
	L3Purge         int64 = DefaultCachePurge
	L3InitSize      int   = 64 * 1024
)

* * L3Cache: key => offsets * less requests to hashDB * * disabling L3 is not a good idea!! * queues hold offsets which DB does not know about! * cache keeps track of duplicate writes * duplicate keys will get an empty_offsets * and the latest will overwrite the past write maybe still in queue *

View Source 
var (
	IndexParallel     int = NumCacheDBs
	NumQueueWriteChan int = NumCacheDBs
	HisDatWriteBuffer int = 4 * 1024
)
View Source 
var (
	ForcedReplay         bool
	NoReplayHisDat       bool
	UseHashDB            bool  = true                             // controls whether to use hash database for duplicate detection
	BatchFlushEvery      int64 = 5120                             // milliseconds
	HISTORY_INDEX_LOCK         = make(chan struct{}, 1)           // main lock
	HISTORY_INDEX_LOCK16       = make(chan struct{}, NumCacheDBs) // sub locks
	BootVerbose                = true
	//TESTHASH0                  = "0f05e27ca579892a63a256dacd657f5615fab04bf81e85f53ee52103e3a4fae8"
	//TESTHASH1                  = "f0d784ae13ce7cf1f3ab076027a6265861eb003ad80069cdfb1549dd1b8032e8"
	//TESTHASH2                  = "f0d784ae1747092974d02bd3359f044a91ed4fd0a39dc9a1feffe646e6c7ce09"
	TESTHASH = ""
	//TESTCACKEY                 = "f0d784ae1"
	//TESTKEY                    = "784ae1"
	//TESTBUK                    = "0d"
	//TESTDB                     = "f"
	//TESTOFFSET                 = 123456
	ROOTDBS []string
	//ROOTBUCKETS          []string
	//SUBBUCKETS           []string
	BUFLINES    = 10
	BUFIOBUFFER = 102 * BUFLINES // a history line with sha256 is 102 bytes long including LF or 38 bytes of payload + hashLen
	History     HISTORY
	DEBUG       bool = true
	DEBUG0      bool = false
	DEBUG1      bool = false
	DEBUG2      bool = false
	DEBUG9      bool = false
	LOCKHISTORY      = make(chan struct{}, 1)
	HEXCHARS         = []string{"0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "a", "b", "c", "d", "e", "f"}
)
View Source 
var (
	CPUProfile bool // set before boot

)
View Source 
var (
	// set HEX true: converts offset into hex strings to store in hashdb
	// dont change later once db is initialized!
	HEX bool = true
)

Functions ΒΆ

func CRC ΒΆ 

func CRC(input string) string

func ConvertHistoryObjectToString ΒΆ 

func ConvertHistoryObjectToString(obj *HistoryObject) string

func GetShardConfig ΒΆ 

func GetShardConfig(mode int) (numDBs, tablesPerDB int, description string)

GetShardConfig returns the configuration for a given shard mode

func IsPow2 ΒΆ 

func IsPow2(n int) bool

func LOCKfunc ΒΆ 

func LOCKfunc(achan chan struct{}, src string) bool

func LeftPad ΒΆ 

func LeftPad(input *string, length int)

func NullPad ΒΆ 

func NullPad(input *string, length int)

func PrintMemoryStats ΒΆ 

func PrintMemoryStats()

func PrintMemoryStatsEvery ΒΆ 

func PrintMemoryStatsEvery(interval time.Duration)

func RemoveNullPad ΒΆ 

func RemoveNullPad(input string) string

func UNLOCKfunc ΒΆ 

func UNLOCKfunc(achan chan struct{}, src string)

func UnixTimeMicroSec ΒΆ 

func UnixTimeMicroSec() int64

func UnixTimeMilliSec ΒΆ 

func UnixTimeMilliSec() int64

func UnixTimeNanoSec ΒΆ 

func UnixTimeNanoSec() int64

func UnixTimeSec ΒΆ 

func UnixTimeSec() int64

Types ΒΆ

type AccessControlList ΒΆ 

type AccessControlList struct {
	// contains filtered or unexported fields
}
var (
	ACL        AccessControlList
	DefaultACL map[string]bool // can be set before booting
)

func (*AccessControlList) IsAllowed ΒΆ 

func (a *AccessControlList) IsAllowed(ip string) bool

func (*AccessControlList) SetACL ΒΆ 

func (a *AccessControlList) SetACL(ip string, val bool)

func (*AccessControlList) SetupACL ΒΆ 

func (a *AccessControlList) SetupACL()

type CCC ΒΆ 

type CCC struct {
	Counter map[string]uint64
}

CCC is a counter structure for cache statistics

type DBconn ΒΆ 

type DBconn struct {
	// contains filtered or unexported fields
}

type DBopts ΒΆ 

type DBopts struct {
	// contains filtered or unexported fields
}

type HISTORY ΒΆ 

type HISTORY struct {
	/*   it is mostly NOT safe to change any values or read maps from outside! *
	 *   set, change, update values only inside (his *HISTORY) functions and
	 *   don't forget mutex where needed or run into race conditions.
	 */
	DIR string // path to folder: history/

	Offset int64 // the actual offset for history.dat

	WriterChan chan *HistoryObject // history.dat writer channel
	IndexChan  chan *HistoryIndex  // main index query channel

	CutCharRO int

	Counter map[string]uint64
	WBR     bool // WatchDBRunning

	CPUfile *os.File // ptr to file for cpu profiling
	MEMfile *os.File // ptr to file for mem profiling
	// TCPchan: used to send hobj via handleRConn to a remote historyServer
	TCPchan chan *HistoryObject
	// MySQL RocksDB connection pool
	MySQLPool *SQL
	// SQLite3 RocksDB-optimized connection pool (interface{} to avoid import issues)
	SQLite3Pool interface{}
	// SQLite3 sharding configuration
	ShardMode   int // 0=1DB/4096tables, 1=4096DBs, 2=16DB/256tables, 3=64DB/64tables, 4=128DB/32tables, 5=512DB/8tables
	ShardDBs    int // number of database files
	ShardTables int // number of tables per database
	// L1 cache for lightweight duplicate detection when hash DB is disabled
	L1 L1CACHE
	// contains filtered or unexported fields
}

func (*HISTORY) AddHistory ΒΆ 

func (his *HISTORY) AddHistory(hobj *HistoryObject, useL1Cache bool) int

func (*HISTORY) BatchTicker ΒΆ 

func (his *HISTORY) BatchTicker(char string, ticker chan struct{})

func (*HISTORY) BootHistory ΒΆ 

func (his *HISTORY) BootHistory(history_dir string, keylen int)

BootHistory initializes the history component, configuring its settings and preparing it for operation. It sets up the necessary directories for history and hash databases, and opens the history data file. The function also manages the communication channels for reading and writing historical data. If the `useHashDB` parameter is set to true, it initializes the history database (HashDB) and starts worker routines. Parameters:

  • history_dir: The directory where history data will be stored.
  • keylen: The length of the hash values used for indexing.

func (*HISTORY) BootHistoryClient ΒΆ 

func (his *HISTORY) BootHistoryClient(historyServer string)

func (*HISTORY) CLOSE_HISTORY ΒΆ 

func (his *HISTORY) CLOSE_HISTORY()

func (*HISTORY) DoCacheEvict ΒΆ 

func (his *HISTORY) DoCacheEvict(char string, hash string, offset int64, key string)

DoCacheEvict - stub function (cache layers removed)

func (*HISTORY) FseekHistoryHeader ΒΆ 

func (his *HISTORY) FseekHistoryHeader(output *[]byte) (int, error)

func (*HISTORY) FseekHistoryLine ΒΆ 

func (his *HISTORY) FseekHistoryLine(offset int64) (string, error)

func (*HISTORY) FseekHistoryMessageHash ΒΆ 

func (his *HISTORY) FseekHistoryMessageHash(file *os.File, offset int64, char string, rethash *string) error

FseekHistoryMessageHash seeks to a specified offset in the history file and extracts a message-ID hash. It reads characters from the file until a tab character ('\t') is encountered, extracting the hash enclosed in curly braces. If a valid hash is found, it returns the hash as a string without curly braces. If the end of the file (EOF) is reached, it returns a special EOF marker.

func (*HISTORY) GetCounter ΒΆ 

func (his *HISTORY) GetCounter(k string) uint64

func (*HISTORY) GetSQLite3Pool ΒΆ 

func (his *HISTORY) GetSQLite3Pool() *SQLite3DB

GetSQLite3Pool returns the SQLite3 pool if available

func (*HISTORY) GetShardedDB ΒΆ 

func (his *HISTORY) GetShardedDB() (*SQLite3ShardedDB, bool)

GetShardedDB returns the underlying SQLite3ShardedDB if available

func (*HISTORY) GetShardingInfo ΒΆ 

func (his *HISTORY) GetShardingInfo() (mode, numDBs, tablesPerDB int, description string)

GetShardingInfo returns current sharding configuration

func (*HISTORY) IndexQuery ΒΆ 

func (his *HISTORY) IndexQuery(hash string, indexRetChan chan int, offset int64) (int, error)

func (*HISTORY) InitSQLite3 ΒΆ 

func (his *HISTORY) InitSQLite3() error

Initialize SQLite3 for history system

func (*HISTORY) InitSQLite3WithSharding ΒΆ 

func (his *HISTORY) InitSQLite3WithSharding(shardMode int) error

Initialize SQLite3 with specific sharding mode

func (*HISTORY) Initialize128DB32Tables ΒΆ 

func (his *HISTORY) Initialize128DB32Tables() error

Initialize128DB32Tables initializes with 128 databases, 32 tables each

func (*HISTORY) Initialize16DB256Tables ΒΆ 

func (his *HISTORY) Initialize16DB256Tables() error

Initialize16DB256Tables initializes with 16 databases, 256 tables each

func (*HISTORY) Initialize512DB8Tables ΒΆ 

func (his *HISTORY) Initialize512DB8Tables() error

Initialize512DB8Tables initializes with 512 databases, 8 tables each

func (*HISTORY) Initialize64DB64Tables ΒΆ 

func (his *HISTORY) Initialize64DB64Tables() error

Initialize64DB64Tables initializes with 64 databases, 64 tables each

func (*HISTORY) InitializeDatabase ΒΆ 

func (his *HISTORY) InitializeDatabase(useMySQL bool) error

InitializeDatabase initializes either MySQL or SQLite3 database backend

func (*HISTORY) InitializeDatabaseWithSharding ΒΆ 

func (his *HISTORY) InitializeDatabaseWithSharding(useMySQL bool, shardMode int) error

InitializeDatabaseWithSharding initializes database backend with specific sharding mode

func (*HISTORY) InitializeFullSplit ΒΆ 

func (his *HISTORY) InitializeFullSplit() error

InitializeFullSplit initializes with 4096 separate databases

func (*HISTORY) InitializeSingleDB ΒΆ 

func (his *HISTORY) InitializeSingleDB() error

InitializeSingleDB initializes with 1 database and 4096 tables

func (*HISTORY) NewRConn ΒΆ 

func (his *HISTORY) NewRConn(historyServer string) *RemoteConn

func (*HISTORY) PrintCacheStats ΒΆ 

func (his *HISTORY) PrintCacheStats()

func (*HISTORY) SET_DEBUG ΒΆ 

func (his *HISTORY) SET_DEBUG(debug int)

func (*HISTORY) Sync_upcounter ΒΆ 

func (his *HISTORY) Sync_upcounter(k string)

func (*HISTORY) Sync_upcounterN ΒΆ 

func (his *HISTORY) Sync_upcounterN(k string, v uint64)

func (*HISTORY) UsingSQLite3 ΒΆ 

func (his *HISTORY) UsingSQLite3() bool

UsingSQLite3 returns true if SQLite3 backend is being used

func (*HISTORY) WatchDB ΒΆ 

func (his *HISTORY) WatchDB()

type HistoryIndex ΒΆ 

type HistoryIndex struct {
	Hash         string
	Char         string   // first N chars of hash
	Offset       int64    // used to search: -1 or add: > 0 a hash
	IndexRetChan chan int // receives a 0,1,2 :: pass|duplicate|retrylater
}

used to query the index

type HistoryObject ΒΆ 

type HistoryObject struct {
	MessageIDHash string
	StorageToken  string // "F" = flatstorage | "M" = mongodb | "X" = deleted
	Char          string
	Arrival       int64
	Expires       int64
	Date          int64
	ResponseChan  chan int // receives a 0,1,2 :: pass|duplicate|retrylater
}

func ConvertStringToHistoryObject ΒΆ 

func ConvertStringToHistoryObject(parts []string) (*HistoryObject, error)

type HistorySettings ΒΆ 

type HistorySettings struct {
	// constant values once DBs are initalized
	Ka int // keyalgo
	Kl int // keylen

}

builds the history.dat header

type L1CACHE ΒΆ 

type L1CACHE struct {
	Caches  map[string]*L1CACHEMAP
	Extend  map[string]*L1ECH
	Muxers  map[string]*L1MUXER
	Counter map[string]*CCC
	// contains filtered or unexported fields
}

func (*L1CACHE) BootL1Cache ΒΆ 

func (l1 *L1CACHE) BootL1Cache(his *HISTORY)

The BootL1Cache method initializes the cache system. It creates cache maps, initializes them with initial sizes, and starts goroutines to periodically purge expired entries.

func (*L1CACHE) L1Stats ΒΆ 

func (l1 *L1CACHE) L1Stats(statskey string) (retval uint64, retmap map[string]uint64)

func (*L1CACHE) LockL1Cache ΒΆ 

func (l1 *L1CACHE) LockL1Cache(hash string, value int, his *HISTORY) (retcase int)

The LockL1Cache method is used to LOCK a `MessageIDHash` for processing. If the value is not in the cache or has expired, it locks the cache, updates the cache with a new value, and returns the value. Possible return values: 

CaseLock == already in processing
CaseWrite == already in processing
CaseDupes == is a duplicate
CasePass == not a duplicate == locked article for processing

func (*L1CACHE) Set ΒΆ 

func (l1 *L1CACHE) Set(hash string, char string, value int, flagexpires bool, his *HISTORY)

The Set method is used to set a value in the cache. If the cache size is close to its maximum, it grows the cache.

type L1CACHEMAP ΒΆ 

type L1CACHEMAP struct {
	// contains filtered or unexported fields
}

type L1ECH ΒΆ 

type L1ECH struct {
	// contains filtered or unexported fields
}

L1ExtendChan

type L1ITEM ΒΆ 

type L1ITEM struct {
	// contains filtered or unexported fields
}

type L1MUXER ΒΆ 

type L1MUXER struct {
	// contains filtered or unexported fields
}

type L1PQ ΒΆ 

type L1PQ []*L1PQItem

type L1PQItem ΒΆ 

type L1PQItem struct {
	Key     string
	Expires int64
}

type L1pqQ ΒΆ 

type L1pqQ struct {
	// contains filtered or unexported fields
}

func (*L1pqQ) Pop ΒΆ 

func (pq *L1pqQ) Pop() (*L1PQItem, int)

func (*L1pqQ) Push ΒΆ 

func (pq *L1pqQ) Push(item *L1PQItem)

type L2CACHE ΒΆ 

type L2CACHE struct {
	Caches map[string]*L2CACHEMAP
	Extend map[string]*L2ECH
	Muxers map[string]*L2MUXER

	Counter map[string]*CCC
	// contains filtered or unexported fields
}

func (*L2CACHE) BootL2Cache ΒΆ 

func (l2 *L2CACHE) BootL2Cache(his *HISTORY)

The BootL2Cache method initializes the L2 cache. It creates cache maps, initializes them with initial sizes, and starts goroutines to periodically clean up expired entries.

func (*L2CACHE) GetHashFromOffset ΒΆ 

func (l2 *L2CACHE) GetHashFromOffset(offset int64, rethash *string)

The GetHashFromOffset method retrieves a hash from the L2 cache using an offset as the key.

func (*L2CACHE) L2Stats ΒΆ 

func (l2 *L2CACHE) L2Stats(statskey string) (retval uint64, retmap map[string]uint64)

func (*L2CACHE) OffsetToChar ΒΆ 

func (l2 *L2CACHE) OffsetToChar(offset int64) (retval string)

func (*L2CACHE) SetOffsetHash ΒΆ 

func (l2 *L2CACHE) SetOffsetHash(offset int64, hash string, flagexpires bool)

The SetOffsetHash method sets a cache item in the L2 cache using an offset as the key and a hash as the value. It also dynamically grows the cache when necessary.

type L2CACHEMAP ΒΆ 

type L2CACHEMAP struct {
	// contains filtered or unexported fields
}

type L2ECH ΒΆ 

type L2ECH struct {
	// contains filtered or unexported fields
}

L2ExtendChan

type L2ITEM ΒΆ 

type L2ITEM struct {
	// contains filtered or unexported fields
}

type L2MUXER ΒΆ 

type L2MUXER struct {
	// contains filtered or unexported fields
}

type L2PQ ΒΆ 

type L2PQ []*L2PQItem

type L2PQItem ΒΆ 

type L2PQItem struct {
	Key     int64
	Expires int64
}

type L2pqQ ΒΆ 

type L2pqQ struct {
	// contains filtered or unexported fields
}

func (*L2pqQ) Pop ΒΆ 

func (pq *L2pqQ) Pop() (*L2PQItem, int)

func (*L2pqQ) Push ΒΆ 

func (pq *L2pqQ) Push(item *L2PQItem)

type L3CACHE ΒΆ 

type L3CACHE struct {
	Caches map[string]*L3CACHEMAP
	Extend map[string]*L3ECH
	Muxers map[string]*L3MUXER

	Counter map[string]*CCC
	// contains filtered or unexported fields
}

func (*L3CACHE) BootL3Cache ΒΆ 

func (l3 *L3CACHE) BootL3Cache(his *HISTORY)

The BootL3Cache method initializes the L3 cache. It creates cache maps, initializes them with initial sizes, and starts goroutines to periodically clean up expired entries.

func (*L3CACHE) GetOffsets ΒΆ 

func (l3 *L3CACHE) GetOffsets(key string, char string, offsets *[]int64, his *HISTORY) int

The GetOffsets method retrieves a slice of offsets from the L3 cache using a key and a char.

func (*L3CACHE) L3Stats ΒΆ 

func (l3 *L3CACHE) L3Stats(statskey string) (retval uint64, retmap map[string]uint64)

func (*L3CACHE) SetOffsets ΒΆ 

func (l3 *L3CACHE) SetOffsets(key string, char string, offsets []int64, flagexpires bool, src string, his *HISTORY)

The SetOffsets method sets a cache item in the L3 cache using a key, char and a slice of offsets as the value. It also dynamically grows the cache when necessary.

type L3CACHEMAP ΒΆ 

type L3CACHEMAP struct {
	// contains filtered or unexported fields
}

type L3ECH ΒΆ 

type L3ECH struct {
	// contains filtered or unexported fields
}

L3ExtendChan

type L3ITEM ΒΆ 

type L3ITEM struct {
	// contains filtered or unexported fields
}

type L3MUXER ΒΆ 

type L3MUXER struct {
	// contains filtered or unexported fields
}

type L3PQ ΒΆ 

type L3PQ []*L3PQItem

type L3PQItem ΒΆ 

type L3PQItem struct {
	Key     string
	Expires int64
}

type L3pqQ ΒΆ 

type L3pqQ struct {
	// contains filtered or unexported fields
}

func (*L3pqQ) Pop ΒΆ 

func (pq *L3pqQ) Pop() (*L3PQItem, int)

func (*L3pqQ) Push ΒΆ 

func (pq *L3pqQ) Push(item *L3PQItem)

type OffsetData ΒΆ 

type OffsetData struct {
	Shorthash string // first N chars of hash
	Offset    int64
}

type Offsets ΒΆ 

type Offsets struct {
	// contains filtered or unexported fields
}

type RemoteConn ΒΆ 

type RemoteConn struct {
	// contains filtered or unexported fields
}

holds connection to historyServer

type SQL ΒΆ 

type SQL struct {
	DBs chan *DBconn
	// contains filtered or unexported fields

} // end func SQLhandler

func NewSQLpool ΒΆ 

func NewSQLpool(opts *DBopts, createTables bool) (*SQL, error)

createTables := true 

shortHashDBpool, err := history.NewSQLpool(&history.DBopts{
    username: "xxx",
    password: "xxx",
    hostname: "xxx",
    dbname: "xxx",
    maxopen: "128",
    initopen: "16",
    tcpmode: "tcp4",
    timeout: 55,
}, createTables)

func (*SQL) CloseDB ΒΆ 

func (s *SQL) CloseDB(db *sql.DB)

func (*SQL) ClosePool ΒΆ 

func (s *SQL) ClosePool()

func (*SQL) GetDB ΒΆ 

func (s *SQL) GetDB(wait bool) (db *sql.DB, err error)

func (*SQL) GetDSN ΒΆ 

func (s *SQL) GetDSN() string

func (*SQL) GetIsOpen ΒΆ 

func (s *SQL) GetIsOpen() int

func (*SQL) GetOffsets ΒΆ 

func (s *SQL) GetOffsets(key string, db *sql.DB) ([]int64, error)

func (*SQL) InsertOffset ΒΆ 

func (s *SQL) InsertOffset(key string, offset int64, db *sql.DB) error

func (*SQL) NewConn ΒΆ 

func (s *SQL) NewConn() (*sql.DB, error)

func (*SQL) ReturnDB ΒΆ 

func (s *SQL) ReturnDB(db *sql.DB)

func (*SQL) SetMaxOpen ΒΆ 

func (s *SQL) SetMaxOpen(maxopen int)

func (*SQL) ShortHashDB_CreateTables ΒΆ 

func (s *SQL) ShortHashDB_CreateTables() error

type SQLite3Conn ΒΆ 

type SQLite3Conn struct {
	// contains filtered or unexported fields
}

type SQLite3DB ΒΆ 

type SQLite3DB struct {
	DBs chan *SQLite3Conn
	// contains filtered or unexported fields
}

func NewSQLite3Pool ΒΆ 

func NewSQLite3Pool(opts *SQLite3Opts, createTables bool) (*SQLite3DB, error)

func (*SQLite3DB) CloseDB ΒΆ 

func (s *SQLite3DB) CloseDB(db *sql.DB)

func (*SQLite3DB) ClosePool ΒΆ 

func (s *SQLite3DB) ClosePool()

func (*SQLite3DB) CreateTables ΒΆ 

func (s *SQLite3DB) CreateTables() error

func (*SQLite3DB) GetDB ΒΆ 

func (s *SQLite3DB) GetDB(wait bool) (db *sql.DB, err error)

func (*SQLite3DB) GetOffsets ΒΆ 

func (s *SQLite3DB) GetOffsets(key string, db *sql.DB) ([]int64, error)

func (*SQLite3DB) InsertOffset ΒΆ 

func (s *SQLite3DB) InsertOffset(key string, offset int64, db *sql.DB) error

func (*SQLite3DB) NewConn ΒΆ 

func (s *SQLite3DB) NewConn() (*sql.DB, error)

func (*SQLite3DB) OptimizeDB ΒΆ 

func (s *SQLite3DB) OptimizeDB()

Optimize SQLite3 database periodically

func (*SQLite3DB) ReturnDB ΒΆ 

func (s *SQLite3DB) ReturnDB(db *sql.DB)

func (*SQLite3DB) StartOptimizer ΒΆ 

func (s *SQLite3DB) StartOptimizer()

Start periodic optimization

type SQLite3Opts ΒΆ 

type SQLite3Opts struct {
	// contains filtered or unexported fields
}

type SQLite3ShardedDB ΒΆ 

type SQLite3ShardedDB struct {
	DBPools []*SQLite3DB // Array of database pools
	// contains filtered or unexported fields
}

SQLite3ShardedDB manages multiple SQLite databases for sharding

func NewSQLite3ShardedDB ΒΆ 

func NewSQLite3ShardedDB(config *ShardConfig, createTables bool) (*SQLite3ShardedDB, error)

NewSQLite3ShardedDB creates a new sharded SQLite3 database system

func (*SQLite3ShardedDB) Close ΒΆ 

func (s *SQLite3ShardedDB) Close() error

Close closes all database connections

func (*SQLite3ShardedDB) CreateAllTables ΒΆ 

func (s *SQLite3ShardedDB) CreateAllTables() error

CreateAllTables creates all required tables across all databases

func (*SQLite3ShardedDB) GetDBAndTable ΒΆ 

func (s *SQLite3ShardedDB) GetDBAndTable(hash string) (*sql.DB, string, int, error)

GetDBAndTable returns the appropriate database connection and table name for a hash

func (*SQLite3ShardedDB) GetMemoryInfo ΒΆ 

func (s *SQLite3ShardedDB) GetMemoryInfo() map[string]interface{}

GetMemoryInfo returns memory usage information for the current sharding configuration

func (*SQLite3ShardedDB) GetStats ΒΆ 

func (s *SQLite3ShardedDB) GetStats() map[string]interface{}

GetStats returns statistics for all databases

func (*SQLite3ShardedDB) ReturnDB ΒΆ 

func (s *SQLite3ShardedDB) ReturnDB(db *sql.DB, dbIndex int)

ReturnDB returns a database connection to its pool

type SQLiteData ΒΆ 

type SQLiteData struct {
	// contains filtered or unexported fields
}

type ShardConfig ΒΆ 

type ShardConfig struct {
	Mode         int    // Sharding mode (0-5)
	BaseDir      string // Base directory for database files
	MaxOpenPerDB int    // Max connections per database
	Timeout      int64  // Connection timeout
}

ShardConfig defines the sharding configuration

Source Files ΒΆ

View all Source files

Directories ΒΆ

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.