utils

Documentation

Overview

Package utils provides data management utilities for the RSS feed backend.

Key Functions:

• OptimizeQueries: Provides query optimization recommendations
• ValidateIndexes: Checks if required indexes exist
• GetDataManagementConfig: Returns data management configuration

Usage:

config := GetDataManagementConfig()
optimized := OptimizeQueries(query, config)

Package utils provides helper functions for the RSS feed backend.

Package utils provides utility functions for RSS feed parsing.

Key Functions:

• FetchRSSFeed: Parses an RSS feed from a URL and returns a slice of feed items.

Dependencies:

• Uses the `gofeed` library for RSS parsing.

Usage:

items, err := FetchRSSFeed("https://example.com/rss")
if err != nil {
    log.Fatalf("Failed to fetch RSS feed: %v", err)
}

Index

• func EstimateIndexUsage() map[string]float64
• func GenerateRequestID() string
• func GetCleanupCutoffDate(retentionDays int) time.Time
• func GetRecommendedIndexes() []string
• func RandomString(length int) string
• func ValidateDataManagementConfig(config DataManagementConfig) error
• type CleanupConfig
• type DataManagementConfig
  • func GetDataManagementConfig() DataManagementConfig
• type DuplicateDetectionConfig
• type FeedItem
  • func FetchRSSFeed(url string) ([]*FeedItem, error)
  • func (f *FeedItem) GenerateContentHash() string
  • func (f *FeedItem) IsDuplicate(other *FeedItem) bool
  • func (f *FeedItem) Sanitize()
  • func (f *FeedItem) Validate() error
• type IndexConfig
• type ValidationConfig

Functions

func EstimateIndexUsage

func EstimateIndexUsage() map[string]float64

EstimateIndexUsage provides usage estimates for different index types

func GenerateRequestID

func GenerateRequestID() string

GenerateRequestID generates a unique request ID

func GetCleanupCutoffDate

func GetCleanupCutoffDate(retentionDays int) time.Time

GetCleanupCutoffDate returns the cutoff date for cleanup based on retention days

func GetRecommendedIndexes

func GetRecommendedIndexes() []string

GetRecommendedIndexes returns a list of recommended indexes based on query patterns

func RandomString

func RandomString(length int) string

RandomString generates a random string of specified length

func ValidateDataManagementConfig

func ValidateDataManagementConfig(config DataManagementConfig) error

ValidateDataManagementConfig validates the data management configuration

Types

type CleanupConfig

type CleanupConfig struct {
	DefaultRetentionDays int  `json:"default_retention_days"`
	EnableAutoCleanup    bool `json:"enable_auto_cleanup"`
	CleanupBatchSize     int  `json:"cleanup_batch_size"`
	ScheduleCleanup      bool `json:"schedule_cleanup"`
	CleanupHour          int  `json:"cleanup_hour"` // Hour of day to run cleanup (0-23)
}

CleanupConfig contains cleanup settings

type DataManagementConfig

type DataManagementConfig struct {
	// Validation settings
	Validation ValidationConfig `json:"validation"`

	// Duplicate detection settings
	DuplicateDetection DuplicateDetectionConfig `json:"duplicate_detection"`

	// Cleanup settings
	Cleanup CleanupConfig `json:"cleanup"`

	// Index optimization settings
	Indexes IndexConfig `json:"indexes"`
}

DataManagementConfig contains configuration for data management operations

func GetDataManagementConfig

func GetDataManagementConfig() DataManagementConfig

GetDataManagementConfig returns the default data management configuration

type DuplicateDetectionConfig

type DuplicateDetectionConfig struct {
	UseLinkComparison   bool   `json:"use_link_comparison"`
	UseContentHash      bool   `json:"use_content_hash"`
	UseTitleAuthorMatch bool   `json:"use_title_author_match"`
	HashAlgorithm       string `json:"hash_algorithm"`
	CaseSensitive       bool   `json:"case_sensitive"`
}

DuplicateDetectionConfig contains duplicate detection settings

type FeedItem

type FeedItem struct {
	Title       string `datastore:"title,noindex"` // noindex to exclude from indexes
	Link        string `datastore:"link"`
	Description string `datastore:"description,noindex"`
	Author      string `datastore:"author,noindex"`
	PubDate     string `datastore:"pub_date,noindex"`
}

FeedItem represents an RSS feed item

func FetchRSSFeed

func FetchRSSFeed(url string) ([]*FeedItem, error)

FetchRSSFeed fetches and parses an RSS feed from the given URL.

Parameters:

• url: The URL of the RSS feed.

Returns:

• A slice of FeedItem objects containing parsed RSS feed data.
• An error if parsing fails.

Example:

items, err := FetchRSSFeed("https://example.com/rss")
if err != nil {
    log.Fatalf("Failed to fetch RSS feed: %v", err)
}

FeedItem Structure:

• Title: The title of the RSS feed item.
• Link: The URL link to the original article.
• Description: A short description of the RSS feed item.
• PubDate: The publication date of the RSS feed item.

func (*FeedItem) GenerateContentHash

func (f *FeedItem) GenerateContentHash() string

GenerateContentHash generates a hash for content-based duplicate detection

func (*FeedItem) IsDuplicate

func (f *FeedItem) IsDuplicate(other *FeedItem) bool

IsDuplicate checks if this item is likely a duplicate of another

func (*FeedItem) Sanitize

func (f *FeedItem) Sanitize()

Sanitize sanitizes the FeedItem fields

func (*FeedItem) Validate

func (f *FeedItem) Validate() error

Validate validates the FeedItem fields

type IndexConfig

type IndexConfig struct {
	RequiredIndexes     []string `json:"required_indexes"`
	OptimizedQueries    []string `json:"optimized_queries"`
	EnableIndexHints    bool     `json:"enable_index_hints"`
	QueryTimeoutSeconds int      `json:"query_timeout_seconds"`
	MaxQueryResults     int      `json:"max_query_results"`
}

IndexConfig contains index optimization settings

type ValidationConfig

type ValidationConfig struct {
	MaxTitleLength       int  `json:"max_title_length"`
	MaxDescriptionLength int  `json:"max_description_length"`
	MaxAuthorLength      int  `json:"max_author_length"`
	RequireTitle         bool `json:"require_title"`
	RequireLink          bool `json:"require_link"`
	ValidateURL          bool `json:"validate_url"`
	ValidateDate         bool `json:"validate_date"`
}

ValidationConfig contains validation settings