dataprep

package
v0.5.13 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Mar 7, 2024 License: Apache-2.0, MIT Imports: 21 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func ValidateCreateRequest 

func ValidateCreateRequest(ctx context.Context, db *gorm.DB, request CreateRequest) (*model.Preparation, error)

ValidateCreateRequest processes and validates the creation request parameters. The function checks the validity of the input parameters such as maxSize, pieceSize, and the existence of source and output storages. The function also ensures that provided parameters meet certain criteria, like the pieceSize being a power of two, and maxSize allowing for padding. The encryption and storages compatibility is also validated.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • request: The CreateRequest structure containing the parameters for the creation request.

Returns:

  • A pointer to the validated Preparation model which can be used for subsequent operations.
  • An error, if any occurred during the validation. This includes errors such as invalid parameter values, storage not found, or incompatibility between encryption and storage options.

Note: If certain parameters are not provided in the request, they are computed based on certain defaults or constraints, like the pieceSize defaulting to a power of two value.

Types

type AddPieceRequest 

type AddPieceRequest struct {
	PieceCID  string `binding:"required" json:"pieceCid"`      // CID of the piece
	PieceSize string `binding:"required" json:"pieceSize"`     // Size of the piece
	FilePath  string `json:"filePath"    swaggerignore:"true"` // Path to the CAR file, used to determine the size of the file and root CID
	RootCID   string `json:"rootCid"`                          // Root CID of the CAR file, used to populate the label field of storage deal
	FileSize  int64  `json:"fileSize"`                         // File size of the CAR file, this is required for boost online deal
}

type CreateRequest 

type CreateRequest struct {
	Name              string   `binding:"required"    json:"name"`              // Name of the preparation
	SourceStorages    []string `json:"sourceStorages"`                          // Name of Source storage systems to be used for the source
	OutputStorages    []string `json:"outputStorages"`                          // Name of Output storage systems to be used for the output
	MaxSizeStr        string   `default:"31.5GiB"     json:"maxSize"`           // Maximum size of the CAR files to be created
	PieceSizeStr      string   `default:""            json:"pieceSize"`         // Target piece size of the CAR files used for piece commitment calculation
	DeleteAfterExport bool     `default:"false"       json:"deleteAfterExport"` // Whether to delete the source files after export
	NoInline          bool     `default:"false"       json:"noInline"`          // Whether to disable inline storage for the preparation. Can save database space but requires at least one output storage.
	NoDag             bool     `default:"false"       json:"noDag"`             // Whether to disable maintaining folder dag structure for the sources. If disabled, DagGen will not be possible and folders will not have an associated CID.
}

type DefaultHandler 

type DefaultHandler struct{}

func (DefaultHandler) AddOutputStorageHandler 

func (DefaultHandler) AddOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

AddOutputStorageHandler associates a given output storage to a Preparation based on the provided ID. The function verifies the existence of the output storage and the specified Preparation. If both are valid, it creates an association between the output storage and the Preparation.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name of the Preparation to which the output storage should be attached.
  • output: The ID or name of the output storage to be attached.

Returns:

  • A pointer to the updated Preparation model with the new output storage associated.
  • An error, if any occurred during the verification or attachment process.

Note: This function performs several checks to ensure the output storage and the Preparation exist. It also checks for potential duplicate associations and handles potential errors accordingly.

func (DefaultHandler) AddPieceHandler 

func (DefaultHandler) AddPieceHandler(
	ctx context.Context,
	db *gorm.DB,
	id string,
	request AddPieceRequest,
) (*model.Car, error)

AddPieceHandler adds a new piece (represented by the Car model) to a given preparation.

This function fetches a preparation based on the provided ID. It then parses and validates the provided piece CID and size from the request. If a root CID is provided in the request, it is parsed; if a file path is provided instead, the root CID is extracted from the file.

Once the necessary information is extracted and validated, a new piece (Car model) is created in the database associated with the given preparation.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name for the desired Preparation record.
  • request: A struct of AddPieceRequest which contains the information for the piece to be added.

Returns:

  • A pointer to the newly created Car model.
  • An error, if any occurred during the operation.

func (DefaultHandler) AddSourceStorageHandler 

func (DefaultHandler) AddSourceStorageHandler(ctx context.Context, db *gorm.DB, id string, source string) (*model.Preparation, error)

AddSourceStorageHandler associates a given source storage to a Preparation based on the provided ID. It first checks if the source storage exists. If it does, it then creates an association between the source storage and the specified Preparation.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name of the Preparation to which the source storage should be attached.
  • source: The ID or name of the source storage to be attached.

Returns:

  • A pointer to the updated Preparation model with the new source storage associated.
  • An error, if any occurred during the verification or attachment process.

Note: This function ensures that the given source storage exists and that the given Preparation exists before creating an association. It also ensures there are no duplicate associations and handles potential errors accordingly.

func (DefaultHandler) CreatePreparationHandler 

func (DefaultHandler) CreatePreparationHandler(
	ctx context.Context,
	db *gorm.DB,
	request CreateRequest,
) (*model.Preparation, error)

CreatePreparationHandler handles the creation of a new Preparation entity based on the provided CreateRequest parameters. Initially, it validates the request parameters and, if valid, creates a new Preparation record in the database.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • request: The CreateRequest structure containing the parameters for the creation request.

Returns:

  • A pointer to the newly created Preparation model.
  • An error, if any occurred during the validation or creation process.

Note: This function relies on the ValidateCreateRequest function to ensure that the provided parameters meet the required criteria before creating a Preparation record.

func (DefaultHandler) ExploreHandler 

func (DefaultHandler) ExploreHandler(
	ctx context.Context,
	db *gorm.DB,
	id string,
	name string,
	path string,
) (*ExploreResult, error)

ExploreHandler fetches directory entries (files and sub-directories) associated with a specific preparation in a given storage system and directory path. The function retrieves information from a local database rather than directly exploring the remote storage, making use of the stored relationships between files, directories, and storage systems.

This function starts by fetching the desired Storage record based on the provided name. It then fetches the associated SourceAttachment record which connects a preparation to a storage. Using the RootDirectoryID method of the source, it retrieves the root directory's ID and navigates to the desired directory by iterating through the path segments. Once at the desired directory, it fetches the contained directories and files, constructing a result list from the gathered data.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name of the preparation associated with the storage.
  • name: The ID or name of the desired Storage record.
  • path: The directory path in the storage system to explore.

Returns:

  • ExploreResult struct representing the entries in the explored directory.
  • An error, if any occurred during the operation.

func (DefaultHandler) ListHandler 

func (DefaultHandler) ListHandler(ctx context.Context, db *gorm.DB) ([]model.Preparation, error)

ListHandler fetches and returns a list of all Preparation records from the database. It also preloads the associated source and output storages for each Preparation.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.

Returns:

  • A slice containing all Preparation records from the database.
  • An error, if any occurred during the database query operation.

Note: The function uses the Preload() method of gorm to automatically load the related source and output storage records for each returned Preparation, simplifying subsequent operations on these records.

func (DefaultHandler) ListPiecesHandler 

func (DefaultHandler) ListPiecesHandler(
	ctx context.Context,
	db *gorm.DB,
	id string,
) ([]PieceList, error)

ListPiecesHandler retrieves the list of pieces associated with a particular preparation and its source attachments.

This function retrieves the SourceAttachment associated with a given preparation ID. For each source attachment, the associated pieces (represented by the Car model) are fetched and grouped. If there are pieces that are not associated with any source attachment but are linked to the preparation, they are also fetched.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name for the desired Preparation record.

Returns:

  • A slice of PieceList, each representing a source attachment and its associated pieces.
  • An error, if any occurred during the operation.

func (DefaultHandler) ListSchedulesHandler 

func (DefaultHandler) ListSchedulesHandler(
	ctx context.Context,
	db *gorm.DB,
	id string) ([]model.Schedule, error)

ListSchedulesHandler retrieves and returns the list of schedules associated with the specified preparation.

This function searches for a preparation in the database based on the given ID (which could be either a primary key ID or name). Once the preparation is successfully fetched, it retrieves all schedules linked with this preparation.

Parameters:

  • ctx: The context for managing timeouts and cancellation.
  • db: The gorm.DB instance for database operations.
  • id: The ID or name of the preparation to find associated schedules for.

Returns:

  • A slice of model.Schedule if the operation is successful.
  • An error if any issues occur during the operation, including database retrieval errors.

func (DefaultHandler) RemoveOutputStorageHandler 

func (DefaultHandler) RemoveOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

RemoveOutputStorageHandler disassociates a specified output storage from a Preparation using the provided ID. It ensures that the output storage and Preparation both exist before attempting the removal. Special checks are in place to ensure:

  1. The output storage is currently attached to the Preparation.
  2. Removing the only output storage while using encryption is disallowed.

Parameters:

  • ctx: The context for database transactions and other operations.
  • db: A pointer to the gorm.DB instance representing the database connection.
  • id: The ID or name of the Preparation from which the output storage should be detached.
  • output: The ID or name of the output storage to be detached.

Returns:

  • A pointer to the updated Preparation model with the output storage removed.
  • An error, if any occurred during the verification or detachment process.

Note: This function performs several validation steps to ensure integrity while removing the association. It also preloads associated storages to return an updated version of the Preparation.

func (DefaultHandler) RemovePreparationHandler added in v0.5.2 

func (DefaultHandler) RemovePreparationHandler(ctx context.Context, db *gorm.DB, name string, request RemoveRequest) error

func (DefaultHandler) RenamePreparationHandler added in v0.5.0 

func (DefaultHandler) RenamePreparationHandler(
	ctx context.Context,
	db *gorm.DB,
	name string,
	request RenameRequest,
) (*model.Preparation, error)

RenamePreparationHandler updates the name of a preparation entry in the database. This handler finds a preparation entry by its ID or name and then updates its name with the new name provided in the request payload. The new name cannot be entirely numeric or empty.

Parameters:

  • ctx: The context for managing timeouts and cancellation.
  • db: The gorm.DB instance for making database queries.
  • name: The current name or ID of the preparation entry to be renamed.
  • request: A RenameRequest object containing the new name for the preparation entry.

Returns:

  • A pointer to the updated model.Preparation entry, reflecting the new name.
  • An error if any issues occur during the operation, especially if the provided new name is invalid, or if there are database-related errors.

type DirEntry 

type DirEntry struct {
	Path         string    `json:"path"`
	IsDir        bool      `json:"isDir"`
	CID          string    `json:"cid"`
	FileVersions []Version `json:"fileVersions" table:"verbose;expand"`
}

type ExploreResult 

type ExploreResult struct {
	Path       string     `json:"path"`
	CID        string     `json:"cid"`
	SubEntries []DirEntry `json:"subEntries" table:"expand"`
}

type Handler 

type Handler interface {
	CreatePreparationHandler(
		ctx context.Context,
		db *gorm.DB,
		request CreateRequest,
	) (*model.Preparation, error)

	ExploreHandler(
		ctx context.Context,
		db *gorm.DB,
		id string,
		name string,
		path string,
	) (*ExploreResult, error)

	ListHandler(ctx context.Context, db *gorm.DB) ([]model.Preparation, error)

	AddOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

	RemoveOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

	ListPiecesHandler(
		ctx context.Context,
		db *gorm.DB,
		id string,
	) ([]PieceList, error)

	AddPieceHandler(
		ctx context.Context,
		db *gorm.DB,
		id string,
		request AddPieceRequest,
	) (*model.Car, error)

	AddSourceStorageHandler(ctx context.Context, db *gorm.DB, id string, source string) (*model.Preparation, error)
	ListSchedulesHandler(
		ctx context.Context,
		db *gorm.DB,
		id string) ([]model.Schedule, error)
	RenamePreparationHandler(
		ctx context.Context,
		db *gorm.DB,
		name string,
		request RenameRequest,
	) (*model.Preparation, error)

	RemovePreparationHandler(
		ctx context.Context,
		db *gorm.DB,
		name string,
		request RemoveRequest) error
}
var Default Handler = &DefaultHandler{}

type MockDataPrep added in v0.5.0 

type MockDataPrep struct {
	mock.Mock
}

func (*MockDataPrep) AddOutputStorageHandler added in v0.5.0 

func (m *MockDataPrep) AddOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

func (*MockDataPrep) AddPieceHandler added in v0.5.0 

func (m *MockDataPrep) AddPieceHandler(ctx context.Context, db *gorm.DB, id string, request AddPieceRequest) (*model.Car, error)

func (*MockDataPrep) AddSourceStorageHandler added in v0.5.0 

func (m *MockDataPrep) AddSourceStorageHandler(ctx context.Context, db *gorm.DB, id string, source string) (*model.Preparation, error)

func (*MockDataPrep) CreatePreparationHandler added in v0.5.0 

func (m *MockDataPrep) CreatePreparationHandler(ctx context.Context, db *gorm.DB, request CreateRequest) (*model.Preparation, error)

func (*MockDataPrep) ExploreHandler added in v0.5.0 

func (m *MockDataPrep) ExploreHandler(ctx context.Context, db *gorm.DB, id string, name string, path string) (*ExploreResult, error)

func (*MockDataPrep) ListHandler added in v0.5.0 

func (m *MockDataPrep) ListHandler(ctx context.Context, db *gorm.DB) ([]model.Preparation, error)

func (*MockDataPrep) ListPiecesHandler added in v0.5.0 

func (m *MockDataPrep) ListPiecesHandler(ctx context.Context, db *gorm.DB, id string) ([]PieceList, error)

func (*MockDataPrep) ListSchedulesHandler added in v0.5.0 

func (m *MockDataPrep) ListSchedulesHandler(ctx context.Context, db *gorm.DB, id string) ([]model.Schedule, error)

func (*MockDataPrep) RemoveOutputStorageHandler added in v0.5.0 

func (m *MockDataPrep) RemoveOutputStorageHandler(ctx context.Context, db *gorm.DB, id string, output string) (*model.Preparation, error)

func (*MockDataPrep) RemovePreparationHandler added in v0.5.2 

func (m *MockDataPrep) RemovePreparationHandler(ctx context.Context, db *gorm.DB, name string, request RemoveRequest) error

func (*MockDataPrep) RenamePreparationHandler added in v0.5.0 

func (m *MockDataPrep) RenamePreparationHandler(ctx context.Context, db *gorm.DB, name string, request RenameRequest) (*model.Preparation, error)

type PieceList 

type PieceList struct {
	AttachmentID    *model.SourceAttachmentID `json:"attachmentId"`
	SourceStorageID *model.StorageID          `json:"storageId"`
	SourceStorage   *model.Storage            `json:"source"       table:"expand"`
	Pieces          []model.Car               `json:"pieces"       table:"expand"`
}

type RemoveRequest added in v0.5.2 

type RemoveRequest struct {
	RemoveCars bool `json:"removeCars"`
}

type RenameRequest added in v0.5.0 

type RenameRequest struct {
	Name string `binding:"required" json:"name"`
}

type Version 

type Version struct {
	ID           model.FileID `json:"id"`
	CID          string       `json:"cid"`
	Hash         string       `json:"hash"`
	Size         int64        `json:"size"`
	LastModified time.Time    `json:"lastModified" table:"format:2006-01-02 15:04:05"`
}

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.