Documentation
¶
Overview ¶
Package godav - Memory-efficient buffer management
This file provides buffer pooling functionality to reduce memory allocations during upload operations. The BufferPool reuses byte buffers to minimize garbage collection pressure and improve performance for large file uploads.
Features:
- Reusable byte buffer pooling
- Configurable pool size and buffer size
- Automatic buffer size validation
- Non-blocking buffer acquisition and return
Package godav - Upload checkpoint and resumption functionality ¶
This file provides upload resumption capabilities through checkpoint persistence. Checkpoints contain all necessary information to resume an interrupted upload, including upload progress, chunk information, and configuration settings.
Features:
- JSON-based checkpoint serialization
- File-based checkpoint persistence
- Resume upload from saved checkpoints
- Configuration restoration from checkpoints
Package godav - Chunked upload implementation ¶
This file contains the core chunked upload logic for Nextcloud WebDAV, implementing the protocol for large file uploads with pause/resume support, retry logic, and progress tracking.
The chunked upload protocol follows these steps:
- MKCOL /uploads/<user>/<upload-id>
- PUT /uploads/<user>/<upload-id>/<offset> for each chunk
- MOVE /uploads/<user>/<upload-id>/.file -> /files/<user>/<dst>
Package godav provides a high-level client for Nextcloud WebDAV operations, including chunked uploads that bypass proxy body-size limits.
The library is organized into several modules for better maintainability:
- client.go: Core client functionality and upload methods
- types.go: Type definitions, constants, and configuration structures
- chunked_upload.go: Chunked upload implementation with retry logic
- upload_controller.go: Pause/resume/cancel functionality for uploads
- upload_manager.go: Multi-session upload coordination and management
- checkpoint.go: Upload resumption and checkpoint persistence
- buffer_pool.go: Memory-efficient buffer management
- utils.go: Helper functions and utilities
Basic usage:
client := godav.NewClient("https://nextcloud.example.com/remote.php/dav/", "username", "password")
config := godav.DefaultConfig()
config.Verbose = true
// Upload a single file
err := client.UploadFile("/path/to/local/file.txt", "remote/path/file.txt", config)
if err != nil {
log.Fatal(err)
}
Advanced features include pause/resume support, progress tracking, event handling, and automatic checkpoint saving for large file uploads.
Package godav - Type definitions and configuration structures ¶
This file contains all the core type definitions used throughout the godav library, including progress tracking, event handling, error types, and configuration options.
Package godav - Upload control and state management ¶
This file provides pause/resume/cancel functionality for individual uploads and global control across multiple upload sessions. It includes thread-safe state management and coordination between upload sessions.
Package godav - Multi-session upload management ¶
This file provides coordination and management for multiple concurrent upload sessions. It includes session lifecycle management, global pause/resume functionality, and thread-safe coordination between different upload clients.
Features:
- Multi-client upload coordination
- Session lifecycle management (queued, running, paused, completed, failed, cancelled)
- Global pause/resume across all uploads
- Thread-safe session state management
- Session cleanup and resource management
Package godav - Utility functions and helpers ¶
This file contains helper functions used throughout the godav library, including path manipulation, configuration validation, event emission, and other utility functions that support the core upload functionality.
Functions include:
- Path manipulation and joining
- Configuration validation and sanitization
- Event emission for upload lifecycle
- Error checking utilities
- Upload ID generation
Index ¶
- func SaveCheckpoint(checkpoint Checkpoint, filePath string) error
- type BufferPool
- type Checkpoint
- type Client
- func (c *Client) ResumeUpload(checkpoint Checkpoint) error
- func (c *Client) SetConfig(cfg *Config)
- func (c *Client) SetVerbose(verbose bool)
- func (c *Client) UploadDir(localDir, dstDir string) error
- func (c *Client) UploadFile(localPath, dstPath string) error
- func (c *Client) UploadFileResumable(localPath, dstPath string) (*UploadController, error)
- func (c *Client) UploadFileWithConfig(localPath, dstPath string, cfg *Config) error
- func (c *Client) UploadFileWithContext(ctx context.Context, localPath, dstPath string) error
- func (c *Client) UploadFileWithContextWithConfig(ctx context.Context, localPath, dstPath string, cfg *Config) error
- type Config
- type EventInfo
- type GlobalController
- type ProgressInfo
- type UploadController
- type UploadError
- type UploadEvent
- type UploadManager
- func (um *UploadManager) AddUploadSession(localPath, remotePath string, client *Client) (*UploadSession, error)
- func (um *UploadManager) GetUploadSession(sessionID string) (*UploadSession, error)
- func (um *UploadManager) GetUploadSessions() map[string]*UploadSession
- func (um *UploadManager) IsGloballyPaused() bool
- func (um *UploadManager) PauseAllUploads()
- func (um *UploadManager) PauseUpload(sessionID string) error
- func (um *UploadManager) RemoveUploadSession(sessionID string) error
- func (um *UploadManager) ResumeAllUploads()
- func (um *UploadManager) ResumeUpload(sessionID string) error
- func (um *UploadManager) StartUpload(sessionID string) error
- type UploadSession
- type UploadState
- type UploadStatus
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SaveCheckpoint ¶ added in v1.0.0
func SaveCheckpoint(checkpoint Checkpoint, filePath string) error
SaveCheckpoint saves a checkpoint to a file in JSON format. The checkpoint can later be loaded and used to resume an interrupted upload.
Parameters:
- checkpoint: The checkpoint data to save
- filePath: Path where the checkpoint file will be created
Returns an error if the checkpoint cannot be serialized or written to file.
Example:
checkpoint := Checkpoint{...}
err := godav.SaveCheckpoint(checkpoint, "/tmp/upload.checkpoint")
if err != nil {
log.Printf("Failed to save checkpoint: %v", err)
}
Types ¶
type BufferPool ¶ added in v1.0.0
type BufferPool struct {
// contains filtered or unexported fields
}
BufferPool manages reusable byte buffers to reduce allocations
func NewBufferPool ¶ added in v1.0.0
func NewBufferPool(chunkSize int64, poolSize int) *BufferPool
NewBufferPool creates a new buffer pool with the specified chunk size and pool size
func (*BufferPool) Get ¶ added in v1.0.0
func (bp *BufferPool) Get() []byte
Get retrieves a buffer from the pool or creates a new one
func (*BufferPool) Put ¶ added in v1.0.0
func (bp *BufferPool) Put(buf []byte)
Put returns a buffer to the pool for reuse
type Checkpoint ¶ added in v1.0.0
type Checkpoint struct {
LocalPath string `json:"local_path"` // Original local file path
RemotePath string `json:"remote_path"` // Target remote path
UploadID string `json:"upload_id"` // Unique upload session ID
FileSize int64 `json:"file_size"` // Total file size in bytes
ChunkSize int64 `json:"chunk_size"` // Size of each chunk
BytesUploaded int64 `json:"bytes_uploaded"` // Bytes successfully uploaded
ChunksUploaded int `json:"chunks_uploaded"` // Number of chunks uploaded
TotalChunks int `json:"total_chunks"` // Total number of chunks
Timestamp time.Time `json:"timestamp"` // When checkpoint was created
// Essential config values (function pointers cannot be serialized)
ConfigChunkSize int64 `json:"config_chunk_size"` // Original chunk size setting
ConfigSkipExisting bool `json:"config_skip_existing"` // Skip existing files setting
ConfigMaxRetries int `json:"config_max_retries"` // Max retry attempts setting
}
Checkpoint represents a resumable upload checkpoint containing all necessary information to resume an interrupted upload. It includes upload progress, file information, and essential configuration settings.
Checkpoints are typically saved periodically during upload and can be persisted to files, databases, or other storage systems for later resumption.
func LoadCheckpoint ¶ added in v1.0.0
func LoadCheckpoint(filePath string) (*Checkpoint, error)
LoadCheckpoint loads a checkpoint from a JSON file. The loaded checkpoint can be used to resume an interrupted upload.
Parameters:
- filePath: Path to the checkpoint file to load
Returns the loaded checkpoint and any error during loading or parsing.
Example:
checkpoint, err := godav.LoadCheckpoint("/tmp/upload.checkpoint")
if err != nil {
log.Printf("No checkpoint found: %v", err)
return
}
err = client.ResumeUpload(*checkpoint, config)
type Client ¶
Client wraps a gowebdav.Client with Nextcloud-specific operations. It provides high-level methods for uploading files and directories with support for chunked uploads, progress tracking, and pause/resume functionality.
func NewClient ¶
NewClient creates a new Nextcloud WebDAV client.
Parameters:
- baseURL: The base URL of the Nextcloud WebDAV endpoint (e.g., "https://nextcloud.example.com/remote.php/dav/")
- username: The username for authentication
- password: The password or app password for authentication
Returns a configured Client ready for upload operations.
Example:
client := godav.NewClient("https://nextcloud.example.com/remote.php/dav/", "username", "password")
func (*Client) ResumeUpload ¶ added in v1.0.0
func (c *Client) ResumeUpload(checkpoint Checkpoint) error
ResumeUpload resumes an upload from a checkpoint. It creates an isolated config copy for the resume operation so that c.config is never mutated and no stale ResumeFromCheckpoint lingers.
func (*Client) SetConfig ¶ added in v1.0.2
SetConfig replaces the client's default configuration used by methods that do not take an explicit config parameter (e.g., UploadFile, UploadDir, UploadManager flows). The provided config is validated and sanitized. Not safe to change concurrently with ongoing uploads on the same client.
func (*Client) SetVerbose ¶
SetVerbose enables or disables verbose logging for upload operations. When enabled, the client will log detailed information about upload progress, chunk operations, and directory creation.
func (*Client) UploadDir ¶
UploadDir uploads a directory recursively using chunked uploads. Errors from individual file uploads are collected and returned as a combined error; the walk continues even when a file fails.
func (*Client) UploadFile ¶
UploadFile uploads a single file using Nextcloud's chunked upload protocol. The dstPath is relative to the user's files directory.
This method uses chunked uploads to bypass proxy body-size limits and provides better reliability for large files. It supports pause/resume functionality, progress tracking, and automatic retry logic for failed chunks.
Parameters:
- localPath: Local file path to upload
- dstPath: Remote destination path (relative to user's files directory)
- config: Upload configuration (use DefaultConfig() for sensible defaults)
Returns an error if the upload fails. Use UploadError type assertion for detailed error information.
Example:
config := godav.DefaultConfig()
config.Verbose = true
err := client.UploadFile("/path/to/file.txt", "remote/file.txt", config)
if err != nil {
var uploadErr *godav.UploadError
if errors.As(err, &uploadErr) {
fmt.Printf("Upload failed: %s (retries: %d)\n", uploadErr.Op, uploadErr.Retries)
}
}
func (*Client) UploadFileResumable ¶ added in v1.0.0
func (c *Client) UploadFileResumable(localPath, dstPath string) (*UploadController, error)
UploadFileResumable uploads a file with built-in pause/resume support
func (*Client) UploadFileWithConfig ¶ added in v1.0.2
UploadFileWithConfig uploads a single file using the provided config (does not mutate the client's default config). Safe to call concurrently with other uploads since c.config is not touched.
func (*Client) UploadFileWithContext ¶ added in v1.0.0
UploadFileWithContext uploads a single file with context support for cancellation and timeouts. This method provides the same functionality as UploadFile but allows for cancellation and timeout control through the provided context.
Parameters:
- ctx: Context for cancellation and timeout control
- localPath: Local file path to upload
- dstPath: Remote destination path (relative to user's files directory)
- config: Upload configuration
The context is checked at the beginning of the upload operation. For more granular cancellation control during upload, use the pause/resume functionality via UploadController.
Example:
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Minute)
defer cancel()
err := client.UploadFileWithContext(ctx, localPath, remotePath, config)
if err == context.DeadlineExceeded {
fmt.Println("Upload timed out")
}
func (*Client) UploadFileWithContextWithConfig ¶ added in v1.0.2
func (c *Client) UploadFileWithContextWithConfig(ctx context.Context, localPath, dstPath string, cfg *Config) error
UploadFileWithContextWithConfig uploads with context and the provided config (does not mutate the client's default config). Safe to call concurrently with other uploads since c.config is not touched.
type Config ¶
type Config struct {
// ChunkSize specifies the size of each chunk in bytes (default 10MB).
// Larger chunks reduce the number of requests but use more memory.
// Minimum: 1KB, Maximum: 1GB
ChunkSize int64
// SkipExisting when true, skips files that already exist with the same size.
// This provides efficient synchronization by avoiding unnecessary uploads.
SkipExisting bool
// Verbose enables detailed logging of upload operations, including
// chunk progress, directory creation, and retry attempts.
Verbose bool
// ProgressFunc is called during upload to report detailed progress information.
// The callback receives ProgressInfo with current progress, percentages,
// and chunk information. Called after each chunk upload.
ProgressFunc func(info ProgressInfo)
// EventFunc is called for various upload lifecycle events such as
// upload started, chunk uploaded, upload completed, etc.
// Use this for implementing custom upload monitoring and logging.
EventFunc func(info EventInfo)
// MaxRetries specifies the maximum number of retry attempts for failed chunks.
// Each chunk will be retried up to this many times before giving up.
// Range: 0-10 (default 3)
MaxRetries int
// BufferPool provides memory-efficient buffer reuse for upload operations.
// When specified, buffers will be reused to reduce garbage collection.
// Use NewBufferPool() to create a pool with desired size and count.
BufferPool *BufferPool
// Controller enables pause/resume/cancel functionality for uploads.
// When specified, the upload can be controlled programmatically.
// Use NewUploadController() or NewSimpleUploadController() to create.
Controller *UploadController
// CheckpointFunc is called periodically to save upload progress.
// The callback receives a Checkpoint struct that can be persisted
// and used later to resume interrupted uploads.
CheckpointFunc func(cp Checkpoint)
// ResumeFromCheckpoint when specified, resumes an upload from the
// given checkpoint instead of starting a new upload.
// Load checkpoints using LoadCheckpoint().
ResumeFromCheckpoint *Checkpoint
}
Config holds options for upload operations and provides extensive customization for upload behavior, performance optimization, and event handling.
Use DefaultConfig() to get sensible defaults, then customize as needed:
config := godav.DefaultConfig()
config.Verbose = true
config.MaxRetries = 5
config.ProgressFunc = func(info ProgressInfo) {
fmt.Printf("Progress: %.1f%%\n", info.Percentage)
}
func DefaultConfig ¶
func DefaultConfig() *Config
DefaultConfig returns sensible defaults for upload operations.
type EventInfo ¶ added in v1.0.0
type EventInfo struct {
Event UploadEvent // Type of event
Filename string // Name of the file
Path string // Remote path
Message string // Optional message
Error error // Error if applicable
SessionID string // Upload session ID for multi-client support
}
EventInfo contains information about upload events
type GlobalController ¶ added in v1.0.2
type GlobalController struct {
// contains filtered or unexported fields
}
GlobalController provides global pause/resume control across all uploads
type ProgressInfo ¶
type ProgressInfo struct {
Filename string // Name of the file being uploaded
Current int64 // Bytes uploaded so far
Total int64 // Total file size in bytes
Percentage float64 // Upload progress as percentage (0.0 to 100.0)
ChunkIndex int // Current chunk number (0-based)
TotalChunks int // Total number of chunks
SessionID string // Upload session ID for multi-client support
}
ProgressInfo contains detailed progress information for uploads.
type UploadController ¶ added in v1.0.0
type UploadController struct {
// contains filtered or unexported fields
}
UploadController provides pause/resume functionality for individual uploads
func NewSimpleUploadController ¶ added in v1.0.2
func NewSimpleUploadController() *UploadController
NewSimpleUploadController creates a basic upload controller (for backward compatibility)
func NewUploadController ¶ added in v1.0.0
func NewUploadController(sessionID string, manager *UploadManager) *UploadController
NewUploadController creates a new upload controller for a specific session
func (*UploadController) Cancel ¶ added in v1.0.0
func (uc *UploadController) Cancel()
Cancel cancels the upload
func (*UploadController) Pause ¶ added in v1.0.0
func (uc *UploadController) Pause()
Pause pauses the upload
func (*UploadController) Resume ¶ added in v1.0.0
func (uc *UploadController) Resume()
Resume resumes the upload
func (*UploadController) State ¶ added in v1.0.0
func (uc *UploadController) State() UploadState
State returns the current upload state
type UploadError ¶ added in v1.0.0
type UploadError struct {
Op string // Operation that failed
Path string // File path involved
Err error // Underlying error
Retries int // Number of retries attempted
}
UploadError represents errors that occur during upload
func (*UploadError) Error ¶ added in v1.0.0
func (e *UploadError) Error() string
func (*UploadError) Unwrap ¶ added in v1.0.0
func (e *UploadError) Unwrap() error
type UploadEvent ¶ added in v1.0.0
type UploadEvent string
UploadEvent represents different stages of the upload process
const ( EventUploadStarted UploadEvent = "upload_started" // Upload process initiated EventChunkUploaded UploadEvent = "chunk_uploaded" // Individual chunk uploaded EventChunksComplete UploadEvent = "chunks_complete" // All chunks uploaded, before move EventMoveStarted UploadEvent = "move_started" // Starting final move operation EventMoveComplete UploadEvent = "move_complete" // Move operation completed EventUploadComplete UploadEvent = "upload_complete" // Entire upload process finished EventUploadFailed UploadEvent = "upload_failed" // Upload failed EventUploadSkipped UploadEvent = "upload_skipped" // File skipped (already exists) EventUploadPaused UploadEvent = "upload_paused" // Upload paused EventUploadResumed UploadEvent = "upload_resumed" // Upload resumed from checkpoint )
type UploadManager ¶ added in v1.0.2
type UploadManager struct {
// contains filtered or unexported fields
}
UploadManager manages multiple concurrent uploads across different clients
func NewUploadManager ¶ added in v1.0.2
func NewUploadManager() *UploadManager
NewUploadManager creates a new upload manager for coordinating multiple uploads. The manager provides session lifecycle management, global pause/resume functionality, and thread-safe coordination between different upload clients.
Returns a configured UploadManager ready to manage upload sessions.
Example:
manager := godav.NewUploadManager()
session, err := manager.AddUploadSession(localPath, remotePath, client, config)
if err != nil {
log.Fatal(err)
}
err = manager.StartUpload(session.ID)
func (*UploadManager) AddUploadSession ¶ added in v1.0.2
func (um *UploadManager) AddUploadSession(localPath, remotePath string, client *Client) (*UploadSession, error)
AddUploadSession adds a new upload session to the manager. This creates a new session in "queued" status that can be started later. Each session gets a unique ID and its own upload controller.
Parameters:
- localPath: Local file path to upload
- remotePath: Remote destination path
- client: Configured godav client for uploads
- config: Upload configuration (nil uses DefaultConfig)
Returns the created session and any error during session creation.
Example:
session, err := manager.AddUploadSession("/local/file.txt", "remote/file.txt", client, config)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Created session: %s\n", session.ID)
func (*UploadManager) GetUploadSession ¶ added in v1.0.2
func (um *UploadManager) GetUploadSession(sessionID string) (*UploadSession, error)
GetUploadSession returns a specific upload session
func (*UploadManager) GetUploadSessions ¶ added in v1.0.2
func (um *UploadManager) GetUploadSessions() map[string]*UploadSession
GetUploadSessions returns all upload sessions
func (*UploadManager) IsGloballyPaused ¶ added in v1.0.2
func (um *UploadManager) IsGloballyPaused() bool
IsGloballyPaused returns whether all uploads are globally paused
func (*UploadManager) PauseAllUploads ¶ added in v1.0.2
func (um *UploadManager) PauseAllUploads()
PauseAllUploads pauses all running uploads
func (*UploadManager) PauseUpload ¶ added in v1.0.2
func (um *UploadManager) PauseUpload(sessionID string) error
PauseUpload pauses a specific upload session
func (*UploadManager) RemoveUploadSession ¶ added in v1.0.2
func (um *UploadManager) RemoveUploadSession(sessionID string) error
RemoveUploadSession removes a completed or failed upload session
func (*UploadManager) ResumeAllUploads ¶ added in v1.0.2
func (um *UploadManager) ResumeAllUploads()
ResumeAllUploads resumes all paused uploads
func (*UploadManager) ResumeUpload ¶ added in v1.0.2
func (um *UploadManager) ResumeUpload(sessionID string) error
ResumeUpload resumes a specific upload session
func (*UploadManager) StartUpload ¶ added in v1.0.2
func (um *UploadManager) StartUpload(sessionID string) error
StartUpload starts an upload session
type UploadSession ¶ added in v1.0.2
type UploadSession struct {
ID string
LocalPath string
RemotePath string
Client *Client
Controller *UploadController
Config *Config
Status UploadStatus
CreatedAt time.Time
UpdatedAt time.Time
}
UploadSession represents a single upload session
type UploadState ¶ added in v1.0.0
type UploadState int
UploadState represents the current state of an upload
const ( StateRunning UploadState = iota StatePaused StateCancelled )
type UploadStatus ¶ added in v1.0.2
type UploadStatus string
UploadStatus represents the status of an upload session
const ( StatusQueued UploadStatus = "queued" StatusRunning UploadStatus = "running" StatusPaused UploadStatus = "paused" StatusCompleted UploadStatus = "completed" StatusFailed UploadStatus = "failed" StatusCancelled UploadStatus = "cancelled" )