upload

Documentation

Overview

api.go: thin HTTP client for the per-user files-backend's Drive v2 upload-related endpoints. The wire surface mirrors what apps/packages/app/src/api/files/v2/drive/data.ts (getFileServerUploadLink, getFileUploadedBytes) and apps/packages/app/src/api/files/v2/drive/utils.ts (createDir / postCreateFile) call from the web app, so probe / resume / chunk POST behavior stay byte-compatible across both clients.

The HTTP client is supplied by the caller (so tests can use httptest) and the access token is passed in via X-Authorization on every request (same convention as the rest of olares-cli — see pkg/cmdutil/factory.go's authTransport for the rationale).

uploader.go: single-file chunked uploader. Drives the resumable upload protocol the LarePass web app uses (Resumable.js + Drive v2 endpoints):

1. probe the server for already-uploaded bytes via /upload/file-uploaded-bytes/<node>/ (GetUploadedBytes)
2. align to a chunk boundary by flooring (matches the web app's `Math.floor(uploadedBytes / chunkSize)` — re-uploading the "overflow" within that chunk is harmless and identical-byte)
3. ask the server for an upload link via /upload/upload-link/<node>/ (GetUploadLink, once per file)
4. POST each remaining chunk as multipart/form-data with the Resumable.js parameter shape (resumableChunkNumber, ..., file=chunk) plus the Drive-specific extras (parent_dir, driveType, ...)
5. classify each chunk response: - 200 / 201 → chunk accepted, advance - permanent codes → fail fast (see permanentStatuses) - everything else → retry up to opts.MaxRetries with backoff

Empty files are routed through CreateEmptyFile (the web app does the same — Resumable.js can't represent a 0-byte chunk).

walker.go: turn a local-path + remote-path pair into a flat list of per-file upload tasks (UploadOpts) plus the empty-directory mkdirs the chunk-only protocol can't express on its own.

Path semantics (deliberately rsync-LIKE-but-not-rsync):

• <local> is a regular file:
• <remote> ends with '/' → upload to <remote>/<basename(local)>
• else → upload to <remote> (treat as full target path, i.e. user is renaming on the way in)
• <local> is a directory:
• <remote> MUST end with '/' (the destination is a directory).
• The walker recursively emits every regular file under <local>; each file's RelativePath includes <basename(local)> as the top-level component so the source folder's name appears under <remote> on the server (i.e. `upload mydir drive/Home/X/` → drive/Home/X/mydir/...; same for sync/<repo_id>/X/). This matches the LarePass folder-upload UI, which always preserves the picked folder's name.
• Empty subdirectories are recorded as EmptyDirs so the cobra command can pre-mkdir them before the chunk uploads start — Resumable.js's chunk pipeline can't represent a 0-byte directory entry on its own.

All wire-level paths use POSIX-style '/' separators regardless of the host OS, because the server expects forward-slash paths.

Index

• Constants
• type Client
  • func (c *Client) CreateEmptyFile(ctx context.Context, fullPath string) error
  • func (c *Client) FetchNodes(ctx context.Context) ([]Node, error)
  • func (c *Client) GetUploadLink(ctx context.Context, node, parentDir string) (string, error)
  • func (c *Client) GetUploadedBytes(ctx context.Context, node, parentDir, filename string) (int64, error)
  • func (c *Client) Mkdir(ctx context.Context, fullPath string) error
  • func (c *Client) UploadFile(ctx context.Context, opts UploadOpts, progress ProgressFunc) (UploadResult, error)
  • func (c *Client) WaitCloudTask(ctx context.Context, node, taskID string, interval time.Duration, ...) error
• type CloudTaskUpdate
• type CloudTaskUpdateFunc
• type FileTask
• type HTTPError
  • func (e *HTTPError) Error() string
• type Node
• type Plan
  • func BuildPlan(localPath, remoteSubPath, apiRootPrefix, chunkRootPrefix string) (*Plan, error)
  • func (p *Plan) ToUploadOpts(t FileTask, node, driveType string, chunkSize int64, maxRetries int) UploadOpts
• type ProgressFunc
• type UploadOpts
• type UploadResult

Constants

const DefaultChunkSize = 8 * 1024 * 1024

DefaultChunkSize is 8 MiB — the same value the web app uses (apps/packages/app/src/api/files/v2/drive/data.ts L55: SIZE = 8MB). Stick to it unless you have a very good reason; the server's already-uploaded-bytes accounting is keyed on the chunk size the previous run used, so changing this mid-stream means the resume boundary computation diverges (the floor() trick still gets you to a safe re-upload offset, but you waste bandwidth re-sending bytes the server already had).

const DefaultCloudTaskPollInterval = 2 * time.Second

DefaultCloudTaskPollInterval is how often WaitCloudTask polls the task-status endpoint when the caller doesn't override it. 2s is a conservative compromise between responsiveness (cloud uploads of small files can finish in <5s end-to-end) and not flooding the server with status checks for big uploads.

const DefaultMaxRetries = 3

DefaultMaxRetries: per-chunk retry budget. Matches the web app's `maxChunkRetries: 3` (resumejs.ts init() L166).

const DefaultRetryBackoff = 5 * time.Second

DefaultRetryBackoff: between failed-chunk retries. Matches the web app's chunkRetryInterval default of 5s.

Types

type Client

type Client struct {
	HTTPClient *http.Client
	BaseURL    string // FilesURL, e.g. https://files.alice.olares.com
}

Client is the per-FilesURL handle used by uploader.go and the cobra command. It is cheap to construct; reuse one per `files upload` invocation.

HTTPClient is expected to be a factory-provided client whose refreshingTransport injects `X-Authorization` (not `Authorization: Bearer`, see pkg/cmdutil/factory.go for why) and transparently refreshes the token on 401/403 — except for chunk-streaming requests whose body is a *os.File (req.GetBody == nil), where retry is impossible and the 401 falls through to the caller.

func (*Client) CreateEmptyFile

func (c *Client) CreateEmptyFile(ctx context.Context, fullPath string) error

CreateEmptyFile POSTs an empty body to /api/resources/<encoded fullPath> (no trailing slash) to materialize a zero-length file. The web app routes empty files through uploadEmptyFile() instead of the chunk pipeline (resumable.js cannot represent a 0-byte chunk), and we mirror that here.

Unlike Mkdir, a 409 here is reported back to the caller — we don't silently overwrite or pretend success when the user explicitly asked to upload a file and a name collision happened.

func (*Client) FetchNodes

func (c *Client) FetchNodes(ctx context.Context) ([]Node, error)

FetchNodes calls GET {filesURL}/api/nodes/ and returns the configured Drive nodes. The CLI uses nodes[0].Name (or a user-supplied --node override) as the per-request `{node}` path segment for the upload endpoints — same convention as the web app's getUploadNode().

Errors:

• any non-2xx status surfaces as fmt.Errorf with status + body
• empty `data.nodes` is reported with a clear message; the upload flow can't proceed without a node identifier.

func (*Client) GetUploadLink

func (c *Client) GetUploadLink(ctx context.Context, node, parentDir string) (string, error)

GetUploadLink calls GET {filesURL}/upload/upload-link/{node}/?file_path=<enc(parentDir)>&from=web. The server replies with a plaintext path (e.g. `/seafhttp/upload-aj/<repo>/?...`) that the browser then POSTs chunks to. The web app appends `?ret-json=1` to that path so the per-chunk response is JSON instead of a redirect — we do the same to match.

`parentDir` is the parent directory path WITH the `/drive/Home/...` prefix and a TRAILING slash (e.g. `/drive/Home/Documents/`). That's what the web app passes through `files.formatPathtoUrl` → `path.pathname` before plumbing it into this call (see apps/packages/app/src/utils/resumejs.ts L412-L416).

Returned string is a relative path (no scheme/host); the chunk POST uses `c.BaseURL + uploadLink` as the target.

func (*Client) GetUploadedBytes

func (c *Client) GetUploadedBytes(ctx context.Context, node, parentDir, filename string) (int64, error)

GetUploadedBytes asks the server how many bytes of `<parentDir>/<filename>` have already been received. New / never-seen files return 0 (or an error the web app silently swallows — see resumejs.ts: any non-2xx is treated as "start from scratch"). We adopt the same lenient policy so a fresh upload doesn't fail just because the server doesn't know about the file yet.

`parentDir` follows the same convention as GetUploadLink: full `/drive/Home/...` path with a trailing slash. `filename` is the bare basename (no directory components).

func (*Client) Mkdir

func (c *Client) Mkdir(ctx context.Context, fullPath string) error

Mkdir POSTs an empty body to /api/resources/<encoded fullPath>/ to create a directory under the selected namespace root. The trailing slash is what the backend uses to discriminate "create directory" from "create empty file" (postCreateFile in v2/common/utils.ts does the same thing — `isDir ? '/' : ”`).

`fullPath` is the absolute frontend path (e.g. `/drive/Home/Documents` or `/sync/<repo_id>/docs`) without the `/api/resources` prefix.

IMPORTANT: This call is NOT idempotent on the server side. The files-backend auto-renames colliding directories ("Documents" exists → POST creates "Documents (1)") instead of returning 409. We treat the 409 path as "already exists" for completeness but most servers won't take that branch — callers should reserve Mkdir for paths they're confident don't exist yet (e.g. brand-new subdirectories they computed from a local walk). The 409 fast-path stays in case some deployments do return 409 for collisions.

func (*Client) UploadFile

func (c *Client) UploadFile(ctx context.Context, opts UploadOpts, progress ProgressFunc) (UploadResult, error)

UploadFile uploads `opts.LocalPath` to `opts.ParentDir` + `opts.RemoteName`, resuming from whatever the server already has. Empty files are routed to CreateEmptyFile (the chunk pipeline can't express 0-byte chunks).

`progress`, if non-nil, is invoked once after the resume probe (with uploaded=<server bytes>, total=<file size>) and once per accepted chunk thereafter. It is NOT invoked per retry attempt.

Returns an UploadResult on success. The caller MUST inspect `result.CloudTaskID` for cloud-drive uploads (awss3 / google / dropbox) and follow up with Client.WaitCloudTask to drive the second leg of the two-stage transfer; doing so is what the web app resumejs.ts onFileUploadSuccess L591-606 path does, and skipping it leaves the file stuck on the Olares-staging side without ever landing in the user's cloud bucket. For files-backend-managed namespaces (drive/sync/cache/external) CloudTaskID is empty and the caller has nothing to wait on.

func (*Client) WaitCloudTask

func (c *Client) WaitCloudTask(
	ctx context.Context,
	node, taskID string,
	interval time.Duration,
	onUpdate CloudTaskUpdateFunc,
) error

WaitCloudTask polls /api/task/<node>/?task_id=<taskID> at `interval` (or DefaultCloudTaskPollInterval if interval == 0) and returns when the task reaches a terminal status:

• completed → nil
• failed → fmt.Errorf with `failed_reason` when the server provided one, otherwise a generic "task failed" message
• canceled / cancelled → fmt.Errorf("task ... was cancelled")

`onUpdate` is called once per poll whenever the task is NOT yet terminal — pass nil if the caller doesn't want progress updates.

ctx cancellation is honored promptly between polls (and at every HTTP request via Client.do). Transient HTTP errors during polling (the task endpoint flapping, an in-cluster service redeploy) surface immediately as errors — we don't paper over them, because a long-running cloud transfer that can't be queried is indistinguishable from a stuck transfer; the caller should bubble up the failure.

type CloudTaskUpdate

type CloudTaskUpdate struct {
	Status        string  // raw server status: pending / running / paused / ...
	Progress      float64 // 0..100 (server-reported, may stay at 0 for short tasks)
	CurrentPhase  int     // 1..TotalPhase, useful when the server splits the transfer in stages
	TotalPhase    int
	TotalFileSize int64
	FailedReason  string
}

CloudTaskUpdate is the per-poll snapshot WaitCloudTask passes to its onUpdate callback. The cobra layer renders progress lines from these without having to know about the JSON envelope shape.

type CloudTaskUpdateFunc

type CloudTaskUpdateFunc func(CloudTaskUpdate)

CloudTaskUpdateFunc is invoked once per poll while the task is still in flight (pending / running / paused / unknown). It is NOT invoked for the terminal status — that arrives via WaitCloudTask's return.

type FileTask

type FileTask struct {
	// LocalPath is the on-disk path (absolute or working-directory-
	// relative — same form the user passed in).
	LocalPath string
	// RelativePath is the file's path relative to the destination
	// parent_dir, in POSIX form. For a single-file upload this is just
	// the basename / target name; for a folder upload this includes
	// the source-folder prefix (e.g. "mydir/sub/foo.txt").
	RelativePath string
	// RemoteName is the bare basename for the upload (typically the
	// last segment of RelativePath; for the "rename on upload" single-
	// file case it differs from filepath.Base(LocalPath)).
	RemoteName string
	// Size is the file size in bytes at plan time. Useful for sorting
	// (largest-first scheduling) and for the progress display.
	Size int64
}

FileTask is one regular file to upload. The cobra command turns each FileTask into an UploadOpts and pushes it through Client.UploadFile; see Plan.ToUploadOpts for the conversion (kept on Plan so a future `--dry-run` can render tasks without actually uploading).

type HTTPError

type HTTPError struct {
	Status int
	Body   string
	URL    string
	Method string
}

HTTPError carries the status + truncated body of a non-2xx response so callers that care (Mkdir's 409 fast-path, the chunk uploader's permanent vs. retryable classification) can branch on the status code without stringly-typed error parsing.

func (*HTTPError) Error

func (e *HTTPError) Error() string

type Node

type Node struct {
	Name   string `json:"name"`
	Master bool   `json:"master"`
}

Node is the projection of files-backend's `FileNode` that we actually use. The full struct also has a `master` boolean (see apps/packages/app/src/stores/files.ts), which we don't need for the upload flow — we just take the first node's name as the path segment for /upload/upload-link/{node}/ and /upload/file-uploaded-bytes/{node}/.

type Plan

type Plan struct {
	// ParentDir is the constant `/<fileType>/<extend>/<sub>/` "API"
	// parent directory (with trailing '/') that the upload session is
	// anchored to. Same value goes into UploadOpts.ParentDir for every
	// file and is used as the `file_path`/`parent_dir` query for
	// upload-link and file-uploaded-bytes.
	ParentDir string
	// ChunkParentDir is the parent_dir VALUE that goes into each chunk
	// POST's multipart form field (NOT the API queries above). For
	// Drive uploads it equals ParentDir; for Sync uploads it's the
	// path INSIDE the Seafile repo (e.g. `/sub/` instead of
	// `/sync/<repo>/sub/`), because the chunk POST hits
	// `/seafhttp/upload-aj/<token>` and Seafile interprets parent_dir
	// relative to the repo root the token pins.
	ChunkParentDir string
	// RelativeRoot is the path RELATIVE to the selected upload root that
	// ParentDir maps to (no leading or trailing slash, e.g.
	// "Documents/Backups"). The
	// cobra command passes this to Client.Mkdir to ensure the
	// destination dir itself exists before any file upload runs. May
	// be empty when uploading directly to /Home.
	RelativeRoot string
	// EmptyDirs lists the additional sub-directories (POSIX-relative
	// to RelativeRoot, no leading slash, no trailing slash) that need
	// to be pre-created because the source contains them but no files
	// were emitted underneath. Sorted shallow-to-deep so naive
	// sequential mkdir works.
	EmptyDirs []string
	// Files is the flat list of per-file upload tasks. Order is
	// deterministic (sorted by RelativePath) so retries / dry-runs are
	// stable.
	Files []FileTask
}

Plan is the structured result of resolving a (<local>, <remote>) pair against the local filesystem. The cobra command consumes it directly: run EmptyDirs through Client.Mkdir, then schedule Files through an errgroup of Client.UploadFile.

func BuildPlan

func BuildPlan(localPath, remoteSubPath, apiRootPrefix, chunkRootPrefix string) (*Plan, error)

BuildPlan validates inputs against the local filesystem and returns a Plan ready to be executed.

`remoteSubPath` is the path RELATIVE to the selected upload root (drive/Home or sync/<repo_id>) as parsed from a FrontendPath (so "Documents/Backups" or "Documents/Backups/" — with or without leading slash; both are accepted). The trailing slash IS significant: it tells BuildPlan to interpret the remote as a directory rather than a file rename target.

`apiRootPrefix` is the prefix used for API queries (upload-link, file-uploaded-bytes), e.g. `/drive/Home` or `/sync/<repo_id>`. `chunkRootPrefix` is the prefix used for the chunk POST's parent_dir form field; for Drive it equals apiRootPrefix, for Sync it's empty (so the resulting form value is just `/<sub>/` — Seafile expects a path INSIDE the repo since the upload token already pins the repo).

Errors:

• <local> doesn't exist
• <local> is a directory but <remote> doesn't end with '/'

func (*Plan) ToUploadOpts

func (p *Plan) ToUploadOpts(t FileTask, node, driveType string, chunkSize int64, maxRetries int) UploadOpts

ToUploadOpts converts one FileTask into an UploadOpts ready for Client.UploadFile, threading per-call settings (node, chunk size, retries) from the cobra command. Side-effect free.

type ProgressFunc

type ProgressFunc func(uploaded, total int64)

ProgressFunc is the per-chunk callback the cobra command uses to surface a one-line text progress indicator. `uploaded` is the total bytes pushed (cumulative) and `total` is the file size; either may be reported as `(0, 0)` to indicate "an empty file just completed".

type UploadOpts

type UploadOpts struct {
	// LocalPath is the absolute or working-directory-relative path to
	// the file on disk that we're uploading.
	LocalPath string

	// Node is the {node} path segment for /upload/upload-link/<node>/
	// and /upload/file-uploaded-bytes/<node>/. Resolved by the cobra
	// command up-front via Client.FetchNodes.
	Node string
	// DriveType matches the web app query field (`Drive`, `Sync`, ...).
	// For CLI upload we currently emit Drive or Sync.
	DriveType string

	// ParentDir is the destination directory on the server in the
	// "API" form: `/<fileType>/<extend>/<sub>/` with a trailing `/`,
	// e.g. `/drive/Home/Documents/` or `/sync/<repo_id>/Documents/`.
	// This is the value passed as the `file_path` query for
	// upload-link AND the `parent_dir` query for file-uploaded-bytes.
	// The two API queries MUST agree byte-for-byte for resume to find
	// the existing partial upload, which is why we plumb a single
	// value rather than recomputing it at each call site.
	//
	// NOTE: this is NOT the value sent as the `parent_dir` multipart
	// form field on the chunk POST itself — that's `ChunkParentDir`,
	// which differs for Seafile-backed namespaces (Sync).
	ParentDir string

	// ChunkParentDir is the value sent as the `parent_dir` MULTIPART
	// form field on each chunk POST. For Drive uploads it's identical
	// to ParentDir (`/drive/Home/<sub>/`); for Sync uploads it's the
	// path INSIDE the Seafile repo (e.g. `/Documents/` or `/`),
	// because the chunk POST goes to `/seafhttp/upload-aj/<token>`
	// where the token already pins the repo and Seafile interprets
	// `parent_dir` as a path relative to that repo's root.
	//
	// Mirror of `pathname` in the web app's resumejs.ts onChunkingComplete:
	// `formatUploaderPath` strips the `/Seahub/<RepoName>/` prefix for
	// Sync and leaves `/drive/Home/<sub>/` as-is for Drive — that's
	// exactly what we replicate here.
	//
	// When unset, normalize() defaults it to ParentDir so legacy
	// drive-only callers keep working unchanged.
	ChunkParentDir string

	// RemoteName is the bare filename on the server (no directory
	// components). For directory uploads, this is the leaf file name —
	// the directory components live in RelativePath.
	RemoteName string

	// RelativePath is the file's path relative to the upload root, in
	// POSIX form (forward slashes). For a single-file upload this is
	// just RemoteName; for a directory upload it includes the in-tree
	// directory components, e.g. `mydir/photos/IMG_001.jpg`. The web
	// app uses this for resumableRelativePath + the per-chunk
	// `relative_path` form field; the server uses both for sub-directory
	// auto-creation under parent_dir.
	RelativePath string

	// ChunkSize: bytes per chunk. Defaults to DefaultChunkSize when
	// zero.
	ChunkSize int64
	// MaxRetries: retries per chunk on transient failures. Defaults to
	// DefaultMaxRetries when zero. Negative disables retries.
	MaxRetries int
	// RetryBackoff: wait between retries. Defaults to
	// DefaultRetryBackoff when zero.
	RetryBackoff time.Duration
}

UploadOpts is everything UploadFile needs to push one local file into a files-backend namespace (drive/Home or sync/<repo_id>). It's a value type so callers (the cobra command, the directory walker) can build one per file and tweak fields per call without sharing mutable state.

type UploadResult

type UploadResult struct {
	// CloudTaskID is the server-side cloud-transfer task identifier
	// returned by the FINAL chunk's response when the destination is
	// a cloud drive (awss3 / google / dropbox). The Olares files-
	// backend kicks off an internal "Olares-staging → cloud-bucket"
	// transfer task when the chunk pipeline finalizes a cloud-bound
	// file, and it surfaces the task handle on the last chunk's
	// response body so the client can poll completion via
	// `/api/task/<node>/?task_id=<id>` (see Client.WaitCloudTask).
	//
	// Empty for files-backend-managed namespaces (drive/sync/cache/
	// external) where the upload is a single stage and no follow-up
	// task is created.
	//
	// Mirror of resumejs.ts onFileUploadSuccess L591-606: the web app
	// does `JSON.parse(message)` on the final chunk's response body,
	// expects an array, reads `arr[0].taskId`, and registers it with
	// Taskmanager. parseFinalChunkTaskID below replicates that exact
	// shape — anything that doesn't match (Drive's empty body, an
	// empty array, a missing taskId) collapses to "no follow-up
	// task".
	CloudTaskID string
}

UploadResult captures the per-file outcome of UploadFile beyond the usual "did it succeed". Currently the only field is CloudTaskID, which the cobra layer uses to drive the second leg of the two-stage cloud-drive upload (Olares-staging → real cloud bucket); future fields can be added without breaking callers that already use the (UploadResult, error) return shape.