postprocess

Documentation

Index

• Variables
• func Cleanup(dir string) error
• func CleanupArchives(res *ExtractDirResult) error
• func Extract(archivePath string, outputDir string, password string) ([]string, error)
• func IsPasswordProtected(archivePath string) bool
• func Par2Available() bool
• func Par2Repair(par2File string) error
• func Par2Verify(par2File string) error
• type ExtractDirResult
  • func ExtractInDir(dir string, password string) (*ExtractDirResult, error)
  • func ExtractInDirTo(dir, destDir string, password string) (*ExtractDirResult, error)
• type ExtractorType
  • func FindExtractor() (ExtractorType, string)
• type Options
• type Par2RepairableError
  • func (e *Par2RepairableError) Error() string
• type Par2UnrepairableError
  • func (e *Par2UnrepairableError) Error() string
  • func (e *Par2UnrepairableError) Unwrap() error
• type PasswordError
  • func (e *PasswordError) Error() string
• type Result
  • func Process(dir string, downloadedFiles map[string]string, opts Options) (*Result, error)

Variables

var ErrPar2NotInstalled = errors.New("par2 not installed")

ErrPar2NotInstalled is returned by Par2Verify/Par2Repair when parity data is present but the `par2` binary is missing. The caller MUST surface this rather than treat it as "verified OK" — a download that shipped parity but could not be checked is delivered UNVERIFIED, not verified.

var ErrPar2Unrepairable = errors.New("par2: verification failed and repair not possible")

ErrPar2Unrepairable is returned by Par2Verify when parity confirms the data is damaged AND par2 reports repair is not possible — the file is definitively corrupt (distinct from a transient par2 probe error). The pipeline marks the delivery Corrupt so the engine treats it as an integrity failure and re-downloads, rather than shipping a broken file with a soft warning.

Functions

func Cleanup

func Cleanup(dir string) error

Cleanup removes archive and parity files from a directory.

func CleanupArchives

func CleanupArchives(res *ExtractDirResult) error

CleanupArchives removes the volumes of the archive that ExtractInDir actually unpacked — and nothing else.

It deliberately does NOT reuse Cleanup(): that one deletes by extension list (.nfo .txt .jpg .png .sfv .url …), which is safe for usenet, where the directory is a scratch space the downloader created and owns. A torrent's directory is what the SWARM served: the same sweep eats the user's subtitles (.txt), the poster (.jpg), the fanart (.png) and any notes they kept there. Measured on a 7-file release, only the .mkv survived.

Passing an ExtractDirResult whose extraction did not happen is a no-op.

func Extract

func Extract(archivePath string, outputDir string, password string) ([]string, error)

Extract extracts an archive, preferring the in-process (native) extractor and falling back to unrar/7z when one is installed.

Native first, measured rather than assumed. Benchmarked against the shell extractors on real payloads (600 MB RAR set, 246 MB compressed):

RAR  -m0 store, 12×50 MB volumes   go 1.14-1.23s   unrar 0.97-2.16s   (go is STEADIER)
RAR  -m3 compressed, 246 MB        go 0.57s        unrar 0.35s        (1.6x)
7z   store, 600 MB                 go 0.56s        7z    0.55s        (parity, disk-bound)
7z   LZMA2 compressed, 246 MB      go 3.42s        7z    0.75s        (4.6x — the worst case)

Output was byte-identical (sha256) in every case. The store rows are the ones that matter: scene releases ship -m0, so the common path is disk-bound and the native decoder is at worst indistinguishable. Only LZMA2 is materially slower, and 3.4s per 250 MB does not threaten a post-processing pipeline.

The shell path is kept — not deleted — because a native decoder that chokes on an exotic scene RAR would otherwise leave the user with nothing, where today 7z rescues it. Every rescue is logged: if the log fills with them, the preference order was wrong and the evidence will say so.

password is optional — pass "" if not needed. Returns the list of extracted file paths.

func IsPasswordProtected

func IsPasswordProtected(archivePath string) bool

IsPasswordProtected checks if an archive requires a password.

The native check runs first and answers for every supported container without spawning a process. It is also the only check available on a machine with no extractor binary — where this used to return a flat false, letting the pipeline walk into an extraction that could only fail.

func Par2Available

func Par2Available() bool

Par2Available checks if par2cmdline is installed.

func Par2Repair

func Par2Repair(par2File string) error

Par2Repair attempts to repair files using par2 parity data.

func Par2Verify

func Par2Verify(par2File string) error

Par2Verify verifies files using a par2 file. Returns nil on success, ErrPar2NotInstalled when the binary is missing (parity present but unchecked — the caller must surface it, NOT treat it as verified), a *Par2RepairableError when repair is possible, or another error on failure.

Types

type ExtractDirResult

type ExtractDirResult struct {
	// Extracted is true when an archive was found AND unpacked.
	Extracted bool
	// Files lists the extracted files. Empty when Extracted is false.
	Files []string
	// Note is non-empty when an archive was present but could NOT be unpacked
	// for a recoverable reason (no extractor installed, or an extractor that
	// produced nothing). The caller keeps the raw payload and surfaces this,
	// rather than failing the download — the user still gets the .rNN files and
	// can unpack them by hand.
	Note string
	// contains filtered or unexported fields
}

ExtractDirResult reports what ExtractInDir did.

func ExtractInDir

func ExtractInDir(dir string, password string) (*ExtractDirResult, error)

ExtractInDir unpacks a RAR/split archive sitting in dir, when there is one.

It exists because Process() — the usenet post-processing pipeline — cannot be reused for torrents: its signature is usenet-native (a segment→path map from the NNTP downloader, plus lazy par2 fetching), and its par2 steps are meaningless for a torrent, which arrives as a plain directory on disk. ExtractInDir is the archive half of that pipeline, addressed by directory, so both download methods share the extractor without sharing usenet semantics.

Contract: it is a NO-OP when the release ships no archive (the common case — most torrents are already a plain .mkv), so callers may invoke it unconditionally before organizing.

A missing extractor binary is NOT an error: unlike usenet — where the payload is USELESS unextracted, since the archive IS the delivery format — a torrent's raw .rNN files are still what the user was given. Failing the download there would turn "cannot improve this" into "you get nothing". Reported via Note.

func ExtractInDirTo

func ExtractInDirTo(dir, destDir string, password string) (*ExtractDirResult, error)

ExtractInDirTo is ExtractInDir with the output written to destDir instead of beside the archive.

A seeding torrent must keep serving the EXACT bytes it downloaded, and its directory belongs to the swarm — so nothing may be added to it (a stray file makes organize's cleanup pass judge the directory differently) nor removed from it. Extracting to a sibling directory leaves the torrent bit-for-bit intact while still producing a playable file for the library.

destDir == dir reproduces the in-place behaviour, which is what the non-seeding path wants: there the parts are deleted right after, so a sibling would only add a pointless cross-directory move.

type ExtractorType

type ExtractorType string

ExtractorType identifies which extraction tool is available.

const (
	ExtractorNone  ExtractorType = ""
	ExtractorUnrar ExtractorType = "unrar"
	Extractor7z    ExtractorType = "7z"
)

const ExtractorNative ExtractorType = "native"

ExtractorNative is reported when no external binary is installed. It is not a "nothing available" answer: the in-process extractor always works.

func FindExtractor

func FindExtractor() (ExtractorType, string)

FindExtractor reports which EXTERNAL archive extractor is available in PATH.

It no longer decides whether extraction is possible at all — extraction is always possible via the native path — so ExtractorNone now means only "no shell fallback installed". Callers that used to treat ExtractorNone as fatal must not: see ExtractInDirTo and the doctor check.

type Options

type Options struct {
	Password string // password for encrypted archives (empty = none)
	Cleanup  bool   // remove intermediate files after extraction
	// FetchParity downloads the par2 recovery volumes into the task dir when
	// verification detects damage — the index alone carries checksums but no
	// recovery blocks, so "repair is not possible" against index-only parity
	// just means the blocks aren't local yet, NOT that the release is beyond
	// saving. par2 discovers the volumes by name next to the index, so the
	// pipeline only needs the call to succeed. nil = no more parity available;
	// verification runs with what's on disk.
	FetchParity func() (map[string]string, error)
}

Options configures post-processing behavior.

type Par2RepairableError

type Par2RepairableError struct {
	Par2File string
	Damaged  []string
}

Par2RepairableError indicates verification failed but repair is possible.

func (*Par2RepairableError) Error

func (e *Par2RepairableError) Error() string

type Par2UnrepairableError

type Par2UnrepairableError struct {
	Output  string
	Damaged []string
	// Err is the underlying exec failure, when there was one. Kept so the exit
	// code survives into the message the user eventually sees.
	Err error
}

Par2UnrepairableError carries par2's verdict alongside the list of target files it could not reconstruct. The caller uses Damaged to invalidate the resume state of exactly those files, so a retry re-fetches the broken ones and keeps everything par2 confirmed intact.

func (*Par2UnrepairableError) Error

func (e *Par2UnrepairableError) Error() string

func (*Par2UnrepairableError) Unwrap

func (e *Par2UnrepairableError) Unwrap() error

Unwrap keeps errors.Is(err, ErrPar2Unrepairable) working for every existing caller that classifies on the sentinel.

type PasswordError

type PasswordError struct {
	Archive string

	// Uncertain marks a verdict that was INFERRED rather than reported. A
	// header-encrypted 7z and a corrupt one produce the same parse failure, so
	// "needs a password" is the better reading of an ambiguous signal, not a
	// fact the decoder stated.
	//
	// Extract reads this to decide whether the shell fallback is still worth
	// running: a certain password error is deterministic and retrying wastes
	// time, while an uncertain one is exactly what a second extractor can
	// resolve. Without the flag the field would have to be re-derived from the
	// error message, which PasswordError does not carry.
	Uncertain bool
}

PasswordError indicates the archive requires a password.

func (*PasswordError) Error

func (e *PasswordError) Error() string

type Result

type Result struct {
	FinalPath string   // path to the main content file (e.g., the video)
	Files     []string // all final files
	Repaired  bool     // whether par2 repair was needed
	Extracted bool     // whether archive extraction was performed
	// VerifyNote is non-empty when par2 verification was DEGRADED — parity shipped
	// but could not be confirmed (par2 missing, repair failed, verify error). The
	// download is still delivered, but the caller surfaces this so the user knows
	// the file is unverified rather than silently assuming it's good. Empty means
	// either "verified OK" or "no parity shipped" — both are non-degraded.
	VerifyNote string
	// Corrupt is true when par2 DEFINITIVELY confirmed the data is damaged and it
	// could not be repaired (repair failed, or corruption detected with no par2
	// binary to fix it). The engine treats this as an integrity failure and
	// re-downloads — distinct from VerifyNote's softer "unverified but delivered"
	// (e.g. no parity shipped, or a transient probe error).
	Corrupt bool
	// DamagedFiles names the target files par2 reported as damaged or missing.
	// The engine invalidates the resume state of exactly these, so a retry
	// re-fetches the broken files and keeps the ones parity confirmed intact.
	DamagedFiles []string
	// Verified is true ONLY when par2 actually ran and vouched for the payload
	// (clean verify, or a repair that succeeded).
	//
	// It exists because VerifyNote cannot answer the question: an empty note
	// means "verified OK" OR "no parity was available to check", and a caller
	// that deliberately downloaded with holes must distinguish those. Inferring
	// verification from an empty note delivered a zero-filled file as complete
	// whenever the par2 index failed to download.
	Verified bool
}

Result holds the outcome of post-processing.

func Process

func Process(dir string, downloadedFiles map[string]string, opts Options) (*Result, error)

Process runs the full post-processing pipeline on downloaded usenet files. Steps: par2 verify → par2 repair → extract archives → cleanup → find main file.