largescale

Documentation

Overview

Package largescale provides efficient large-scale repository operations for GitHub.

This package contains specialized implementations for handling bulk operations on thousands of repositories with optimized memory usage, adaptive rate limiting, and concurrent processing capabilities.

Key features:

• Large-scale repository listing with pagination support
• Bulk repository cloning with memory-efficient batching
• Adaptive rate limiting based on GitHub API response headers
• Progress tracking and statistics for long-running operations
• Memory pressure monitoring and garbage collection management
• Retry mechanisms with exponential backoff

Example usage:

config := largescale.DefaultLargeScaleConfig()
manager := largescale.NewLargeScaleManager(config, progressCallback)

repos, err := manager.ListAllRepositories(ctx, "organization")
if err != nil {
	log.Fatal(err)
}

err = manager.BulkCloneRepositories(ctx, repos, "/target/path")
if err != nil {
	log.Fatal(err)
}

Index

• type AdaptiveRateLimiter
  • func NewAdaptiveRateLimiter() *AdaptiveRateLimiter
  • func (rl *AdaptiveRateLimiter) EstimateTimeToCompletion(requestsNeeded int) time.Duration
  • func (rl *AdaptiveRateLimiter) GetStatus() (remaining int, resetTime time.Time, estimatedDelay time.Duration)
  • func (rl *AdaptiveRateLimiter) Reset()
  • func (rl *AdaptiveRateLimiter) SetConfiguration(maxPerSecond int, bufferRatio float64, enableAdaptive bool)
  • func (rl *AdaptiveRateLimiter) UpdateRemaining(remaining int)
  • func (rl *AdaptiveRateLimiter) UpdateResetTime(resetTime time.Time)
  • func (rl *AdaptiveRateLimiter) Wait(ctx context.Context) error
• type LargeScaleConfig
  • func DefaultLargeScaleConfig() *LargeScaleConfig
• type LargeScaleManager
  • func NewLargeScaleManager(config *LargeScaleConfig, progressCallback ProgressCallback) *LargeScaleManager
  • func (m *LargeScaleManager) BulkCloneRepositories(ctx context.Context, repos []LargeScaleRepository, targetPath string) error
  • func (m *LargeScaleManager) GetStats() OperationStats
  • func (m *LargeScaleManager) ListAllRepositories(ctx context.Context, org string) ([]LargeScaleRepository, error)
• type LargeScaleRepository
• type OperationStats
• type ProgressCallback

Types

type AdaptiveRateLimiter

type AdaptiveRateLimiter struct {
	// contains filtered or unexported fields
}

AdaptiveRateLimiter provides intelligent rate limiting for GitHub API operations.

func NewAdaptiveRateLimiter

func NewAdaptiveRateLimiter() *AdaptiveRateLimiter

NewAdaptiveRateLimiter creates a new adaptive rate limiter.

func (*AdaptiveRateLimiter) EstimateTimeToCompletion

func (rl *AdaptiveRateLimiter) EstimateTimeToCompletion(requestsNeeded int) time.Duration

EstimateTimeToCompletion estimates how long it will take to make N requests.

func (*AdaptiveRateLimiter) GetStatus

func (rl *AdaptiveRateLimiter) GetStatus() (remaining int, resetTime time.Time, estimatedDelay time.Duration)

GetStatus returns current rate limiter status.

func (*AdaptiveRateLimiter) Reset

func (rl *AdaptiveRateLimiter) Reset()

Reset resets the rate limiter state (useful for testing or manual reset).

func (*AdaptiveRateLimiter) SetConfiguration

func (rl *AdaptiveRateLimiter) SetConfiguration(maxPerSecond int, bufferRatio float64, enableAdaptive bool)

SetConfiguration allows customizing rate limiter behavior.

func (*AdaptiveRateLimiter) UpdateRemaining

func (rl *AdaptiveRateLimiter) UpdateRemaining(remaining int)

UpdateRemaining updates the remaining request count from API response.

func (*AdaptiveRateLimiter) UpdateResetTime

func (rl *AdaptiveRateLimiter) UpdateResetTime(resetTime time.Time)

UpdateResetTime updates the rate limit reset time from API response.

func (*AdaptiveRateLimiter) Wait

func (rl *AdaptiveRateLimiter) Wait(ctx context.Context) error

Wait blocks until it's safe to make a request.

type LargeScaleConfig

type LargeScaleConfig struct {
	MaxConcurrency    int
	BatchSize         int
	UseShallowClone   bool
	EnableCompression bool
	ProgressInterval  time.Duration
	MaxRetries        int
	MemoryThreshold   int64 // bytes
}

LargeScaleConfig holds configuration for large-scale repository operations.

func DefaultLargeScaleConfig

func DefaultLargeScaleConfig() *LargeScaleConfig

DefaultLargeScaleConfig returns optimized configuration for large-scale operations.

type LargeScaleManager

type LargeScaleManager struct {
	// contains filtered or unexported fields
}

LargeScaleManager handles large-scale repository operations efficiently.

func NewLargeScaleManager

func NewLargeScaleManager(config *LargeScaleConfig, progressCallback ProgressCallback) *LargeScaleManager

NewLargeScaleManager creates a new manager for large-scale repository operations.

func (*LargeScaleManager) BulkCloneRepositories

func (m *LargeScaleManager) BulkCloneRepositories(ctx context.Context, repos []LargeScaleRepository, targetPath string) error

BulkCloneRepositories clones multiple repositories with optimized concurrency.

func (*LargeScaleManager) GetStats

func (m *LargeScaleManager) GetStats() OperationStats

GetStats returns current operation statistics.

func (*LargeScaleManager) ListAllRepositories

func (m *LargeScaleManager) ListAllRepositories(ctx context.Context, org string) ([]LargeScaleRepository, error)

ListAllRepositories fetches all repositories from an organization with proper pagination.

type LargeScaleRepository

type LargeScaleRepository struct {
	Name          string `json:"name"`
	FullName      string `json:"fullName"`
	HTMLURL       string `json:"htmlUrl"`
	CloneURL      string `json:"cloneUrl"`
	DefaultBranch string `json:"defaultBranch"`
	Size          int    `json:"size"` // Size in KB
	Fork          bool   `json:"fork"`
	Archived      bool   `json:"archived"`
}

LargeScaleRepository represents a minimal repository structure for large-scale operations.

type OperationStats

type OperationStats struct {
	TotalRepos     int
	ProcessedRepos int
	FailedRepos    int
	SkippedRepos   int
	TotalSize      int64 // Total size in KB
	StartTime      time.Time
	LastUpdateTime time.Time
	APICallsUsed   int
	// contains filtered or unexported fields
}

OperationStats tracks statistics for large-scale operations.

type ProgressCallback

type ProgressCallback func(processed, total int, current string)

ProgressCallback is called periodically during large operations.