dedup

README

Dedup Package

The dedup package provides object deduplication and fingerprinting functionality using MD5 hashing.

Features

• Object Fingerprinting: Generate unique MD5 fingerprints for objects
• Deduplication: Track and detect duplicate objects
• Database Persistence: Store fingerprints in database with GORM
• Flexible Hashing: Support custom hash providers or JSON serialization

Main Components

ObjectFingerprint

Database model for storing object fingerprints:

• ObjectKey: Unique identifier for object category
• ObjectName: Specific object name
• MD5: MD5 hash fingerprint
• Composite unique index on (ObjectKey, ObjectName)

ObjectFingerprintService

Service for managing object fingerprints:

• Set(key, name, obj): Store or update object fingerprint
• Get(key, name): Retrieve stored fingerprint
• IsDuplicated(key, name, obj): Check if object is duplicate

HashBytesProvider

Interface for custom hashing:

• Implement HashBytes() to provide custom byte representation
• Falls back to JSON marshaling if not implemented

BuildObjectMD5

Utility function to generate MD5 hash:

• Accepts any object type
• Uses custom hasher or JSON serialization
• Returns hex-encoded MD5 string

Usage

// Store object fingerprint
fingerprint, err := ServiceObjectFingerprint.Set("category", "item1", myObject)

// Check for duplicates
isDup, err := ServiceObjectFingerprint.IsDuplicated("category", "item1", newObj)
if isDup {
    // Object is duplicate
}

Dependencies

• GORM for database operations
• Core package for dependency injection
• Crypto/md5 for hashing

Documentation

Index

• func BuildObjectMD5(obj any) (string, error)
• type HashBytesProvider
• type ObjectFingerprint
• type ObjectFingerprintService
  • func NewObjectFingerprintService(db *gorm.DB) *ObjectFingerprintService
  • func (s *ObjectFingerprintService) CheckAndSave(objectKey string, objectName string, obj any) (bool, error)
  • func (s *ObjectFingerprintService) Get(objectKey string, objectName string) (*ObjectFingerprint, error)
  • func (s *ObjectFingerprintService) IsDuplicated(objectKey string, objectName string, obj any) (bool, error)
  • func (s *ObjectFingerprintService) Set(objectKey string, objectName string, obj any) (*ObjectFingerprint, error)

Functions

func BuildObjectMD5

func BuildObjectMD5(obj any) (string, error)

Types

type HashBytesProvider

type HashBytesProvider interface {
	HashBytes() []byte
}

type ObjectFingerprint

type ObjectFingerprint struct {
	gorm.Model
	ObjectKey  string `gorm:"size:255;not null;uniqueIndex:uk_object_dedup_key_name,priority:1"`
	ObjectName string `gorm:"size:255;not null;uniqueIndex:uk_object_dedup_key_name,priority:2"`
	MD5        string `gorm:"size:32;not null;index"`
}

type ObjectFingerprintService

type ObjectFingerprintService struct {
	// contains filtered or unexported fields
}

var ServiceObjectFingerprint *ObjectFingerprintService

func NewObjectFingerprintService

func NewObjectFingerprintService(db *gorm.DB) *ObjectFingerprintService

func (*ObjectFingerprintService) CheckAndSave

func (s *ObjectFingerprintService) CheckAndSave(objectKey string, objectName string, obj any) (bool, error)

func (*ObjectFingerprintService) Get

func (s *ObjectFingerprintService) Get(objectKey string, objectName string) (*ObjectFingerprint, error)

func (*ObjectFingerprintService) IsDuplicated

func (s *ObjectFingerprintService) IsDuplicated(objectKey string, objectName string, obj any) (bool, error)

func (*ObjectFingerprintService) Set

func (s *ObjectFingerprintService) Set(objectKey string, objectName string, obj any) (*ObjectFingerprint, error)