storage

package module
v0.0.0-...-6326fb3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: AGPL-3.0 Imports: 16 Imported by: 0

README

StuHelper Storage

StuHelper 的通用存储核心,管理存储实例、准入、容量预占、完整副本、校验、修复和清理。课程资料、文件元数据、用户贡献与奖励由独立模块管理。

目标为三份完整副本,两份独立合规副本校验通过后可供业务发布。受限文件应用层加密、阅读器与真实长期授权验收分别按开发记录推进。

开发

使用 Next 声明的工具链。make setup 安装锁定工具,make check 检查生成结果、场景、Go、文档与工作流,并强制要求 PostgreSQL。用 STUHELPER_STORAGE_TEST_DSN 指定专用测试服务器,或运行 bash scripts/check-with-postgres.sh 建立临时集群。

边界

每个入口都要求当前授权。运行身份只访问 storage schema,迁移使用独立身份。供应商凭据只通过引用解析,数据库、任务、日志与事件不保存密码或令牌。adapter 的返回成功仍需内容校验。

当前存储实例规划上限为 100 个,每批维护最多处理 100 个对象。executor 提供固定 worker、维护扫描与停止恢复。传输大小、并发、带宽、队列和临时磁盘预算均有上限;Linux 私有暂存目录通过进程锁回收上次崩溃残留,服务关闭等待在途操作退出。

容量刷新与副本巡检保持定期执行,周期由装配配置决定。QuotaValidity 默认五分钟,QuotaRefreshAhead 默认一分钟,VerificationAge 的一半为开始巡检的时间。调整后的有效期用于新样本,过期样本仍拒绝新写,具体范围见模块规格。

并发和带宽限制作用于单个 Service 实例,多个执行进程须分别配置目录并汇总机器预算;数据库任务与容量预占共享同一账本。执行器库可以嵌入 API 或独立入口,真实身份、凭据恢复及生产部署仍由 backend/ops 装配和验收。

行为、实际测试与生产装配缺项见模块规格。BHpan 驱动已用真实账号验证隔离写入、内容读取和永久清理,backend 另完成三账号副本、重启恢复及会话失效后的凭据重登。本轮真实观察已结束,生产稳定性另行验收;候选依赖固定到 BHpan 源码提交,不代表已经发行。

运维接口使用独立权限,展示容量、健康副本缺口和失败任务,不触发供应商访问。缺失文件的重新复制与容量回收已有数据库测试;供应商会话失效后的凭据重登由 backend 托管层负责。

运维命令支持暂停、恢复暂停、排空、撤销与平台额度调整。任务人工重试检查当前权限、修订和队列预算,保留原写入意图并记录操作者与任务标识。管理页面使用专用 DTO,不读取完整实例或凭据配置。

在线托管接口支持首次接入、原请求恢复和凭据更新。凭据存储与供应商登录由 backend 注入,首次接入不自动开始贡献。托管期限与贡献条件分别管理,接口及浏览器 SDK 仍需实际身份与生产配置接入。

共享内容创建保留独立逻辑引用,实际字节校验后按服务身份、保护范围、长度、摘要和副本策略共用一组物理副本。已有来源免上传需要独立复用权限。引用释放、过期和保护迁移分别保存进度,最后一个引用释放后才清理物理内容。

Documentation

Overview

Package storage owns provider-independent placement, replicas and recovery. Callers supply current authorization; credentials are resolved outside this package.

Index

Constants

This section is empty.

Variables

View Source
var (
	ErrInput       = errors.New("storage_invalid_input")
	ErrDenied      = errors.New("storage_permission_denied")
	ErrUnavailable = errors.New("storage_unavailable")
	ErrNotFound    = errors.New("storage_not_found")
	ErrConflict    = errors.New("storage_conflict")
	ErrCapacity    = errors.New("storage_capacity_unavailable")
	ErrIntegrity   = errors.New("storage_integrity_failed")
	ErrStale       = errors.New("storage_stale_operation")
	ErrUnknown     = errors.New("storage_remote_result_unknown")
)

Functions

This section is empty.

Types

type Admission

type Admission struct {
	Class     string    `json:"class"`
	Scope     string    `json:"scope"`
	Plaintext bool      `json:"plaintext"`
	ExpiresAt time.Time `json:"expires_at"`
}

type Authorizer

type Authorizer interface {
	Authorize(context.Context, Principal, string, string, Protection) error
}

Authorizer must check the current actor, action, object and protection scope. HTTP headers and previously issued publication events are not authorization facts.

type Config

type Config struct {
	Clock                 func() time.Time
	MaxObjectSize         int64
	TemporaryDirectory    string
	ParallelTransfers     int
	Lease                 time.Duration
	VerificationAge       time.Duration
	QuotaValidity         time.Duration
	MaxAttempts           int
	TemporaryBytes        int64
	MaxOutstandingJobs    int
	AccountTransfers      int
	BytesPerSecond        int64
	AccountBytesPerSecond int64
}

type ContentGroup

type ContentGroup struct {
	CleanupCheckedAt time.Time  `json:"cleanup_checked_at"`
	ID               string     `json:"id"`
	Fingerprint      string     `json:"fingerprint"`
	ObjectID         string     `json:"object_id"`
	Owner            Principal  `json:"owner"`
	Size             int64      `json:"size"`
	SHA256           string     `json:"sha256"`
	Protection       Protection `json:"protection"`
	Policy           Policy     `json:"policy"`
	State            string     `json:"state"`
}

ContentGroup owns one immutable physical object. Handles retain independent authorization, references and publication state. Group identity includes the service principal and every placement requirement; a digest is not access proof.

type ContentHandle

type ContentHandle struct {
	CreatedAt    time.Time   `json:"created_at"`
	ID           string      `json:"id"`
	GroupID      string      `json:"group_id"`
	Owner        Principal   `json:"owner"`
	CreationHash string      `json:"creation_hash"`
	Revision     int64       `json:"revision"`
	State        string      `json:"state"`
	Verified     bool        `json:"verified"`
	Published    bool        `json:"published"`
	References   []Reference `json:"references"`
	SourceGroup  string      `json:"source_group,omitempty"`
	SourceObject string      `json:"source_object,omitempty"`
	SourceID     string      `json:"source_id,omitempty"`
	PendingGroup string      `json:"pending_group,omitempty"`
}

type ContributionCandidate

type ContributionCandidate struct {
	ID                     string    `json:"id"`
	Adapter                string    `json:"adapter"`
	AvailableBytes         int64     `json:"available_bytes"`
	AuthorizationExpiresAt time.Time `json:"authorization_expires_at"`
}

type ContributionCandidates

type ContributionCandidates struct {
	Items      []ContributionCandidate `json:"items"`
	NextCursor string                  `json:"next_cursor"`
}

type ContributionControl

type ContributionControl struct {
	ID               string    `json:"id"`
	OwnerDomain      string    `json:"owner_domain"`
	Revision         int64     `json:"revision"`
	ExpectedRevision int64     `json:"expected_revision"`
	State            string    `json:"state"`
	LimitBytes       int64     `json:"limit_bytes"`
	PersonalReserve  int64     `json:"personal_reserve"`
	Until            time.Time `json:"until"`
}

ContributionControl is a permanent, monotonically revised business binding. It contains no provider credential and cannot extend provider authorization.

type ContributionSnapshot

type ContributionSnapshot struct {
	InstanceID             string               `json:"instance_id"`
	Adapter                string               `json:"adapter"`
	Revision               int64                `json:"revision"`
	Control                *ContributionControl `json:"control,omitempty"`
	State                  string               `json:"state"`
	Authorized             bool                 `json:"authorized"`
	AuthorizationExpiresAt time.Time            `json:"authorization_expires_at"`
	QuotaExpiresAt         time.Time            `json:"quota_expires_at"`
	UsedBytes              int64                `json:"used_bytes"`
	ReservedBytes          int64                `json:"reserved_bytes"`
	AvailableBytes         int64                `json:"available_bytes"`
	RemainingReplicas      int64                `json:"remaining_replicas"`
	ObservedAt             time.Time            `json:"observed_at"`
}

type CreateRequest

type CreateRequest struct {
	Deduplicate  bool       `json:"deduplicate,omitempty"`
	ReuseFrom    string     `json:"reuse_from,omitempty"`
	ContentGroup string     `json:"-"`
	ClientKey    string     `json:"client_key,omitempty"`
	Size         int64      `json:"size"`
	SHA256       string     `json:"sha256"`
	Protection   Protection `json:"protection"`
	Policy       Policy     `json:"policy"`
}

type CredentialRenewal

type CredentialRenewal struct {
	ID                 string
	ExpectedRevision   int64
	CredentialRef      string
	CredentialRevision int64
	Until              time.Time
}

type Driver

Driver's intent is a unique, platform-owned name in an isolated root. Find reconciles a write whose result was lost. Checkpoint must run before data write. Errors must be sanitized; absence must be explicit ErrNotFound.

type Event

type Event struct {
	SpecVersion     string    `json:"specversion"`
	Source          string    `json:"source"`
	Subject         string    `json:"subject"`
	DataContentType string    `json:"datacontenttype"`
	Data            EventData `json:"data"`
	ID              string    `json:"id"`
	Type            string    `json:"type"`
	ObjectID        string    `json:"-"`
	Revision        int64     `json:"-"`
	Time            time.Time `json:"time"`
	Actor           Principal `json:"-"`
	JobID           string    `json:"-"`
}

type EventData

type EventData struct {
	JobID          string `json:"job_id,omitempty"`
	Revision       int64  `json:"revision"`
	PolicyRevision int64  `json:"policy_revision"`
	ObjectID       string `json:"object_id"`
}

type Idempotency

type Idempotency struct {
	Key, Digest, ObjectID string
	Result                json.RawMessage
	ExpiresAt             time.Time
}

type Instance

type Instance struct {
	ContributionControl *ContributionControl `json:"contribution_control,omitempty"`
	ID                  string               `json:"id"`
	Adapter             string               `json:"adapter"`
	AccountDomain       string               `json:"account_domain"`
	OwnerDomain         string               `json:"owner_domain"`
	Ownership           string               `json:"ownership"`
	CredentialRef       string               `json:"credential_ref"`
	CredentialRevision  int64                `json:"credential_revision"`
	Root                string               `json:"root"`
	Revision            int64                `json:"revision"`
	State               string               `json:"state"`
	Authorized          bool                 `json:"authorized"`
	Limit               int64                `json:"limit"`
	Used                int64                `json:"used"`
	Reserved            int64                `json:"reserved"`
	ProviderFree        int64                `json:"provider_free"`
	PersonalReserve     int64                `json:"personal_reserve"`
	SafetyMargin        int64                `json:"safety_margin"`
	QuotaRevision       int64                `json:"quota_revision"`
	QuotaExpiresAt      time.Time            `json:"quota_expires_at"`
	ExpiresAt           time.Time            `json:"expires_at"`
	Admissions          []Admission          `json:"admissions"`
}

func (Instance) Allows

func (i Instance) Allows(p Protection, now time.Time) bool

func (Instance) CanWrite

func (i Instance) CanWrite(p Protection, size int64, now time.Time) bool

func (Instance) Draining

func (i Instance) Draining(now time.Time) bool

type InstanceCommand

type InstanceCommand struct {
	Revision   int64  `json:"revision"`
	Action     string `json:"action"`
	LimitBytes *int64 `json:"limit_bytes,omitempty"`
}

type InstanceEvidence

type InstanceEvidence struct {
	AccountDomain string
	ProviderFree  int64
	VerifiedAt    time.Time
}

type InstanceHealth

type InstanceHealth struct {
	Revision               int64     `json:"revision"`
	LimitBytes             int64     `json:"limit_bytes"`
	UsedBytes              int64     `json:"used_bytes"`
	ReservedBytes          int64     `json:"reserved_bytes"`
	ID                     string    `json:"id"`
	State                  string    `json:"state"`
	Reason                 string    `json:"reason"`
	AvailableBytes         int64     `json:"available_bytes"`
	AuthorizationExpiresAt time.Time `json:"authorization_expires_at"`
	QuotaExpiresAt         time.Time `json:"quota_expires_at"`
}

type Job

type Job struct {
	Intent             string    `json:"intent"`
	ObjectRevision     int64     `json:"object_revision"`
	InstanceRevision   int64     `json:"instance_revision"`
	CredentialRevision int64     `json:"credential_revision"`
	ID                 string    `json:"id"`
	ObjectID           string    `json:"object_id"`
	ReplicaID          string    `json:"replica_id"`
	Kind               string    `json:"kind"`
	State              string    `json:"state"`
	Generation         int64     `json:"generation"`
	LeaseOwner         string    `json:"lease_owner"`
	LeaseUntil         time.Time `json:"lease_until"`
	AvailableAt        time.Time `json:"available_at"`
	Attempts           int       `json:"attempts"`
	ErrorCode          string    `json:"error_code"`
}

type JobFailure

type JobFailure struct {
	Generation  int64     `json:"generation"`
	ID          string    `json:"id"`
	ObjectID    string    `json:"object_id"`
	Kind        string    `json:"kind"`
	State       string    `json:"state"`
	Attempts    int       `json:"attempts"`
	ErrorCode   string    `json:"error_code"`
	NextAttempt time.Time `json:"next_attempt"`
}

type JobTotals

type JobTotals struct {
	Pending int64 `json:"pending"`
	Running int64 `json:"running"`
	Blocked int64 `json:"blocked"`
}

type Locator

type Locator struct {
	ID       string `json:"id"`
	Revision string `json:"revision"`
}

type MaintenanceResult

type MaintenanceResult struct {
	Processed  int    `json:"processed"`
	NextCursor string `json:"next_cursor"`
}

type Object

type Object struct {
	ContentGroup string      `json:"content_group,omitempty"`
	CreationHash string      `json:"creation_hash,omitempty"`
	ID           string      `json:"id"`
	Owner        Principal   `json:"owner"`
	Size         int64       `json:"size"`
	SHA256       string      `json:"sha256"`
	Protection   Protection  `json:"protection"`
	Policy       Policy      `json:"policy"`
	Revision     int64       `json:"revision"`
	State        string      `json:"state"`
	Published    bool        `json:"published"`
	CreatedAt    time.Time   `json:"created_at"`
	References   []Reference `json:"references"`
}

type ObjectHealth

type ObjectHealth struct {
	ID      string `json:"id"`
	State   string `json:"state"`
	Healthy int    `json:"healthy"`
	Desired int    `json:"desired"`
	Missing int    `json:"missing"`
	Reason  string `json:"reason"`
}

type Operations

type Operations struct {
	GeneratedAt        time.Time        `json:"generated_at"`
	Instances          []InstanceHealth `json:"instances"`
	Objects            []ObjectHealth   `json:"objects"`
	Jobs               JobTotals        `json:"jobs"`
	Failures           []JobFailure     `json:"failures"`
	NextCursor         string           `json:"next_cursor"`
	FailuresNextCursor string           `json:"failures_next_cursor"`
}

type Policy

type Policy struct {
	Target  int `json:"target"`
	Publish int `json:"publish"`
}

func DefaultPolicy

func DefaultPolicy() Policy

func (Policy) Valid

func (p Policy) Valid() bool

type Principal

type Principal struct {
	Issuer  string `json:"issuer"`
	Subject string `json:"subject"`
}

func (Principal) Valid

func (p Principal) Valid() bool

type Protection

type Protection struct {
	Class     string `json:"class"`
	Scope     string `json:"scope"`
	Public    bool   `json:"public"`
	Encrypted bool   `json:"encrypted"`
}

func (Protection) Valid

func (p Protection) Valid() bool

type Reference

type Reference struct {
	Module string `json:"module"`
	ID     string `json:"id"`
}

type Replica

type Replica struct {
	ID                 string    `json:"id"`
	ObjectID           string    `json:"object_id"`
	InstanceID         string    `json:"instance_id"`
	Intent             string    `json:"intent"`
	Remote             Locator   `json:"remote"`
	State              string    `json:"state"`
	InstanceRevision   int64     `json:"instance_revision"`
	CredentialRevision int64     `json:"credential_revision"`
	ObjectRevision     int64     `json:"object_revision"`
	VerifiedAt         time.Time `json:"verified_at"`
	Size               int64     `json:"size"`
	Reserved           bool      `json:"reserved"`
	Accounted          bool      `json:"accounted"`
	IssuedWrite        bool      `json:"issued_write"`
}

type Repository

type Repository interface {
	Transaction(context.Context, func(Transaction) error) error
	Claim(context.Context, string, time.Time, time.Duration) (Job, error)
	PendingEvents(context.Context, int) ([]Event, error)
	AcknowledgeEvent(context.Context, string) error
}

type Resolver

type Resolver interface {
	Resolve(context.Context, Instance) (Driver, error)
}

Resolver uses credential references and revisions, never credentials in jobs.

type Service

type Service struct {
	// contains filtered or unexported fields
}

func New

func New(repo Repository, auth Authorizer, resolver Resolver, verifier Verifier, cfg Config) (*Service, error)

func (*Service) ApplyContribution

func (s *Service) ApplyContribution(ctx context.Context, p Principal, id string, c ContributionControl) (ContributionSnapshot, error)

func (*Service) ChangeProtection

func (s *Service) ChangeProtection(ctx context.Context, p Principal, id, key string, revision int64, protection Protection) (Status, error)

func (*Service) Close

func (s *Service) Close(ctx context.Context) error

Close rejects new transfers, cancels active I/O and waits for cleanup. The directory stays locked until the last operation actually exits, even on timeout.

func (*Service) CommandInstance

func (s *Service) CommandInstance(ctx context.Context, p Principal, id, key string, command InstanceCommand) (InstanceHealth, error)

CommandInstance deliberately cannot change identity, credentials, admissions or authorization deadlines. Restoring revoked custody requires fresh evidence.

func (*Service) Create

func (s *Service) Create(ctx context.Context, p Principal, key string, r CreateRequest) (Status, error)

func (*Service) Delete

func (s *Service) Delete(ctx context.Context, p Principal, id, key string, revision int64) (Status, error)

func (*Service) Download

func (s *Service) Download(ctx context.Context, p Principal, id string, dst io.Writer) error

Download buffers within the configured size/concurrency budget so corrupted or replaced remote content cannot partially escape before checksum validation.

func (*Service) Ingest

func (s *Service) Ingest(ctx context.Context, p Principal, id, key string, src io.Reader) (Status, error)

Ingest hashes bounded local staging before any provider write. The staging file is removed on return; retry uses the immutable object's size and digest.

func (*Service) InspectContribution

func (s *Service) InspectContribution(ctx context.Context, p Principal, id, owner string) (ContributionSnapshot, error)

func (*Service) Instances

func (s *Service) Instances(ctx context.Context, p Principal, limit int, after string) ([]Instance, error)

func (*Service) ListContributionCandidates

func (s *Service) ListContributionCandidates(ctx context.Context, p Principal, owner string, n int, after string) (ContributionCandidates, error)

func (*Service) Operations

func (s *Service) Operations(ctx context.Context, p Principal, limit int, after, failuresAfter string) (Operations, error)

Operations is a global administration privilege, independent of content read access. It reads the ledger only and never extends supplier sessions.

func (*Service) ReauthorizeInstance

func (s *Service) ReauthorizeInstance(ctx context.Context, p Principal, key string, q CredentialRenewal) (Instance, error)

ReauthorizeInstance changes only custody facts. Current contribution, quota, state, ownership and admissions are copied inside the committing transaction.

func (*Service) Reconcile

func (s *Service) Reconcile(ctx context.Context, p Principal, limit int, after string) (MaintenanceResult, error)

Reconcile is bounded and resumable. Expired unreferenced uploads are tombstoned; no operation recursively deletes a provider directory or personal files.

func (*Service) Reference

func (s *Service) Reference(ctx context.Context, p Principal, id, key string, ref Reference, attach bool) (Status, error)

func (*Service) RefreshInstance

func (s *Service) RefreshInstance(ctx context.Context, p Principal, id string) error

RefreshInstance samples capacity only without outstanding writes. This avoids counting the same in-flight write both in provider usage and local reservations.

func (*Service) Register

func (s *Service) Register(ctx context.Context, p Principal, key string, v Instance) (Instance, error)

Register verifies account identity before accepting administrative placement claims. An alias or a second directory under the same account cannot add capacity.

func (*Service) RetryJob

func (s *Service) RetryJob(ctx context.Context, p Principal, objectID, jobID, key string, generation int64) (JobFailure, error)

RetryJob does not reconstruct a write. It requeues the original intent only when the bindings still agree, leaving unknown-write reconciliation to workers.

func (*Service) ReviewDownload

func (s *Service) ReviewDownload(ctx context.Context, p Principal, id string, dst io.Writer) error

ReviewDownload uses a separate permission to inspect an unpublished verified copy. It never changes publication, references, replica policy or protection.

func (*Service) RunOne

func (s *Service) RunOne(ctx context.Context, worker string) (bool, error)

RunOne claims a durable lease and executes one bounded operation. No in-memory queue is required to resume after a worker restart.

func (*Service) Status

func (s *Service) Status(ctx context.Context, p Principal, id string) (Status, error)

func (*Service) UpdateInstance

func (s *Service) UpdateInstance(ctx context.Context, p Principal, key string, v Instance) (Instance, error)

type Status

type Status struct {
	ID         string     `json:"id"`
	Size       int64      `json:"size"`
	SHA256     string     `json:"sha256"`
	Protection Protection `json:"protection"`
	Policy     Policy     `json:"policy"`
	Revision   int64      `json:"revision"`
	State      string     `json:"state"`
	Published  bool       `json:"published"`
	Healthy    int        `json:"healthy"`
	Desired    int        `json:"desired"`
}

func (Status) String

func (s Status) String() string

type Transaction

type Transaction interface {
	ContentGroup(string) (ContentGroup, error)
	ActiveContentGroup(string) (ContentGroup, error)
	SaveContentGroup(ContentGroup) error
	ContentHandle(string) (ContentHandle, error)
	SaveContentHandle(ContentHandle) error
	ContentGroupHeld(string) (bool, error)
	ContentCleanup(int, string) ([]ContentGroup, error)
	ExpiredContentHandles(int, time.Time) ([]ContentHandle, error)
	Lock(string) error
	Instance(string) (Instance, error)
	Instances(int, string) ([]Instance, error)
	ContributionCandidates(string, int, string) ([]Instance, error)
	SaveInstance(Instance) error
	InstanceReplicas(string) (int64, error)
	Object(string) (Object, error)
	Objects(int, string) ([]Object, error)
	SaveObject(Object) error
	Replicas(string) ([]Replica, error)
	SaveReplica(Replica) error
	Job(string) (Job, error)
	// JobSnapshot reads a binding without a row lock, for instance-before-job locking.
	JobSnapshot(string) (Job, error)
	SaveJob(Job) error
	OutstandingJobs(string, int) (int, error)
	ActiveWrites(string, time.Time) (bool, error)
	JobTotals() (JobTotals, error)
	FailedJobs(int, string) ([]Job, error)
	Idempotency(string) (Idempotency, error)
	SaveIdempotency(Idempotency) error
	Event(Event) error
}

Locks are acquired object -> instances in sorted ID order -> job. Claim only locks a job and commits before execution, so it cannot invert that order.

type Verifier

type Verifier interface {
	Verify(context.Context, Instance) (InstanceEvidence, error)
}

Verifier resolves the provider's stable account identity and validates the isolated root. Caller-supplied aliases must never establish replica independence.

Directories

Path Synopsis
Package custody coordinates explicit provider custody separately from storage contribution.
Package custody coordinates explicit provider custody separately from storage contribution.
drivers
bhpan
Package bhpan connects the independent BHpan protocol library to storage.
Package bhpan connects the independent BHpan protocol library to storage.
Package executor runs bounded workers against the storage-owned durable queue.
Package executor runs bounded workers against the storage-owned durable queue.
Package httpapi exposes the storage contract.
Package httpapi exposes the storage contract.
Package postgres implements the storage-owned transactional ledger.
Package postgres implements the storage-owned transactional ledger.
Package sdk provides primitives to interact with the openapi HTTP API.
Package sdk provides primitives to interact with the openapi HTTP API.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL