arctic

package
v0.0.0-...-ad489ce Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 20, 2026 License: Apache-2.0 Imports: 36 Imported by: 0

Documentation

Overview

Package arctic acquires, processes, and publishes the public Reddit archive. It pulls the monthly bulk dumps from the public torrent catalog, decompresses the zstd JSONL, writes Parquet with a stable schema, keeps a local index, and can publish the shards to a Hugging Face dataset repository. It carries no CLI knowledge and depends only on the standard library, golang.org/x, and a set of pure-Go libraries for BitTorrent, zstd, Parquet, and SQLite.

Index

Constants

View Source 
const (
	SubredditTorrentInfoHash = "56aa49f9653ba545f48df2e33679f014d2829c10"
	SubredditTorrentURL      = "https://academictorrents.com/download/56aa49f9653ba545f48df2e33679f014d2829c10.torrent"
	SubredditTorrentSubdir   = "subreddits23"
)

Per-subreddit bundle: the top forty thousand communities repackaged by subreddit over 2005-06 .. 2023-12, files at reddit/subreddits23/{name}_{comments,submissions}.zst.

View Source 
const (
	EnvDataDir        = "ARCTIC_DATA_DIR"
	EnvRawDir         = "ARCTIC_RAW_DIR"
	EnvWorkDir        = "ARCTIC_WORK_DIR"
	EnvRepoRoot       = "ARCTIC_REPO_ROOT"
	EnvMinFreeGB      = "ARCTIC_MIN_FREE_GB"
	EnvChunkLines     = "ARCTIC_CHUNK_LINES"
	EnvCommitEvery    = "ARCTIC_COMMIT_EVERY"
	EnvDownloadFloor  = "ARCTIC_DOWNLOAD_FLOOR_GB"
	EnvMaxDownloads   = "ARCTIC_MAX_DOWNLOADS"
	EnvMaxProcess     = "ARCTIC_MAX_PROCESS"
	EnvMaxConvert     = "ARCTIC_MAX_CONVERT"
	EnvMaxDecodes     = "ARCTIC_MAX_DECODES"
	EnvEngine         = "ARCTIC_ENGINE"
	EnvHFToken        = "HF_TOKEN"
	DefaultHFRepo     = "open-index/arctic"
	DefaultChunkLines = 500000
	// DefaultCommitEveryShards keeps a big month landing every few shards
	// instead of one commit at the end, so progress is visible and resumable.
	DefaultCommitEveryShards = 8
	DefaultMinFreeGB         = 30
	// DefaultDownloadFloorGB leaves room for a few in-flight months at once so
	// the pipeline can process them in parallel without exhausting the disk.
	DefaultDownloadFloorGB = 40
)

Environment variable names. The CLI reads these as flag defaults.

View Source 
const HasDuckDB = false

HasDuckDB reports whether this binary carries the DuckDB engine. The default build does not.

View Source 
const MinEpoch int64 = 1104537600

MinEpoch is the earliest instant any source serves: 2005-01-01. Requests before this are clamped.

Variables

View Source 
var ErrCommitStall = errors.New("arctic: commit stall, no forward progress within max-commit-stall")

ErrCommitStall is returned when the pipeline makes no forward progress within cfg.MaxCommitStall: neither a processed shard nor a Hugging Face commit. The publish leaves the in-flight month uncommitted so a supervisor that restarts the process resumes cleanly, and the CLI maps this to exit code 75 (EX_TEMPFAIL).

Functions

func AppendStats 

func AppendStats(path string, row StatsRow) error

AppendStats merges one row into the ledger at path, replacing any existing row with the same key.

func BuiltinMonthly 

func BuiltinMonthly(ym string) bool

BuiltinMonthly reports whether the compiled-in catalog already carries a month, ignoring the runtime override. `catalog refresh` uses it to flag which fetched months are genuinely new.

func CommittedSet 

func CommittedSet(path string) (map[string]bool, error)

CommittedSet reads the ledger and returns the set of committed keys ("YYYY-MM-type"), used to skip work that already landed on a resumed run. A row counts as committed once it carries a CommittedAt timestamp.

func DefaultDataDir 

func DefaultDataDir() string

DefaultDataDir returns the XDG data directory for arctic, honoring ARCTIC_DATA_DIR when set.

func DownloadFile 

func DownloadFile(ctx context.Context, infoHash, pathInTorrent, destPath string,
	trackers []string, cb DownloadCallback) error

DownloadFile fetches one file out of a torrent to destPath. It adds the magnet with the given trackers, waits for metadata, selects only the wanted file, streams progress, and re-adds the torrent on a stall with a widening timeout. On completion it runs QuickValidateZst and reports corruption versus transient failure so the caller can decide whether to delete or resume.

func DownloadMonth 

func DownloadMonth(ctx context.Context, cfg Config, m Month, t Type, cb DownloadCallback) (string, error)

DownloadMonth resolves a month and type to its torrent and in-torrent path, then fetches the .zst into cfg.ZstPath. It returns the destination path.

func DownloadSubreddit 

func DownloadSubreddit(ctx context.Context, destDir, name string, t Type, cb DownloadCallback) (string, error)

DownloadSubreddit fetches a single subreddit's comments or submissions file from the per-subreddit bundle torrent into destDir.

func FetchMonthlyHashes 

func FetchMonthlyHashes(ctx context.Context, client *http.Client) (map[string]string, error)

FetchMonthlyHashes pulls the latest monthly dump releases and returns a month ("YYYY-MM") to info-hash map for the comment/submission dumps.

func FetchZstSizes 

func FetchZstSizes(ctx context.Context, cfg Config) (map[string]int64, error)

FetchZstSizes queries the catalog torrents for each month's .zst file size, downloading no data, and caches the result to zst_sizes.json under DataDir. The keys are "type/YYYY-MM".

func FilePathInTorrent 

func FilePathInTorrent(m Month, t Type) string

FilePathInTorrent returns the path of a month+type file inside its torrent. The bundle nests by type (reddit/comments/RC_YYYY-MM.zst); a monthly torrent holds the bare file at its root.

func GenerateREADME 

func GenerateREADME(cfg Config, rows []StatsRow) string

GenerateREADME builds the dataset card for the published Hugging Face repo. It renders the YAML front matter the hub indexes on, a per-year breakdown, the full monthly ledger table, usage examples, and the static dataset card. rows is the stats ledger; an empty ledger still produces a valid card so the README exists from the first commit.

func InBundle 

func InBundle(m Month) bool

InBundle reports whether a month lives in the bundle torrent.

func InfoHashFor 

func InfoHashFor(m Month) (string, error)

InfoHashFor resolves the torrent info hash that carries a month. Months in the bundle range return the bundle hash; later months return their monthly hash, or ErrNotPublished when the map has not caught up.

func IsCorruption 

func IsCorruption(err error) bool

IsCorruption reports whether err is or wraps an ErrCorruption.

func IsRateLimit 

func IsRateLimit(err error) bool

IsRateLimit reports whether err is a hub rate-limit response.

func IsTransient 

func IsTransient(err error) bool

IsTransient reports whether err is or wraps an ErrTransient.

func KnownMonthlyHashes 

func KnownMonthlyHashes() []string

KnownMonthlyHashes returns the monthly "YYYY-MM" keys in sorted order, for catalog listings.

func LoadMonthlyCache 

func LoadMonthlyCache(path string) (map[string]string, error)

LoadMonthlyCache reads the refreshed month hashes, returning nil when no cache exists yet.

func LoadProgress 

func LoadProgress(path string) (map[string]ShardProgress, error)

LoadProgress reads the progress ledger, returning an empty map when none exists yet.

func MonthlyCachePath 

func MonthlyCachePath(cfg Config) string

MonthlyCachePath is where refreshed month hashes persist between runs.

func ProgressPath 

func ProgressPath(cfg Config) string

ProgressPath is where in-flight month progress persists between runs.

func Publish 

func Publish(ctx context.Context, cfg Config, opts PublishOptions, cb func(string)) error

Publish runs the resumable download to Parquet to Hugging Face pipeline over a month range. It skips months already in the stats ledger, picks pipeline or sequential mode from the hardware budget, and returns ErrCommitStall when a commit wedges so the caller can exit for a restart.

func QuickValidateZst 

func QuickValidateZst(path string) error

QuickValidateZst is a fast sanity check on a .zst file. It verifies the zstd magic at the head, that the final bytes are not zero-padded (which is what a missing boundary piece leaves behind), and that sampled interior regions are not zero-filled holes from incomplete pieces.

func RegisterMonthlyHashes 

func RegisterMonthlyHashes(m map[string]string)

RegisterMonthlyHashes merges refreshed month to info-hash pairs into the runtime override. Malformed months or hashes are ignored so a bad cache can never poison resolution.

func SaveMonthlyCache 

func SaveMonthlyCache(path string, m map[string]string) error

SaveMonthlyCache writes the month hashes as sorted, indented JSON.

func SaveProgress 

func SaveProgress(path string, m map[string]ShardProgress) error

SaveProgress writes the progress ledger as indented JSON, replacing it atomically so a crash mid-write cannot leave a torn file.

func SetDecodeConcurrency 

func SetDecodeConcurrency(n int)

SetDecodeConcurrency sizes the decoder gate to n concurrent decodes. A value below one is treated as one. Call it before processing begins; it replaces the gate wholesale, so it must not run while a decode holds a token.

func SetVerbose 

func SetVerbose(v bool)

SetVerbose turns the package's internal logging on or off.

func ShardHFPath 

func ShardHFPath(t Type, m Month, n int) string

ShardHFPath returns the in-repo path of a shard for the published layout.

func SubredditFilePathInTorrent 

func SubredditFilePathInTorrent(name string, t Type) string

SubredditFilePathInTorrent returns the path of a subreddit+type file inside the per-subreddit bundle.

func SubredditInTorrent 

func SubredditInTorrent(ctx context.Context, name string) (bool, error)

SubredditInTorrent reports whether name has a file in the per-subreddit bundle, so a caller can pick the torrent path over the API without a failed download. It reads the torrent's file manifest once and caches it.

func ValidateParquet 

func ValidateParquet(path string) error

ValidateParquet checks that a shard is non-empty and carries the PAR1 magic at head and tail, which catches a truncated write before a consumer ever opens the file.

func WriteStats 

func WriteStats(path string, rows []StatsRow) error

WriteStats writes the full ledger, deduplicating by key (last write wins) and sorting by month then type. The write is atomic via a temp file and rename.

Types

type Budget 

type Budget struct {
	// MaxDownloads is how many torrent downloads run concurrently.
	MaxDownloads int
	// MaxProcess is how many months decode concurrently. A process-wide
	// semaphore still holds only one 2 GB decoder at a time, but each slot needs
	// headroom for scanner buffers and the conversion that follows.
	MaxProcess int
	// MaxConvertWorkers is how many Parquet conversions run per decoded month.
	MaxConvertWorkers int
	// MaxDecodes is how many months hold the 2 GB-window zstd decoder at once.
	// One is safe on a small box; a host with spare RAM decodes several giant
	// dumps in parallel, which is the throughput lever for a large backlog.
	MaxDecodes int
	// DuckDBMemoryMB caps the DuckDB engine when the build carries it.
	DuckDBMemoryMB int
	// Sequential is set when the machine is too small for any overlap: one
	// download, one decode, one conversion, fully serial.
	Sequential bool
}

Budget caps how much work runs at once. The 2 GB zstd window for the bulk dumps is the dominant memory cost, so the caps are sized to keep a small machine from thrashing or getting OOM-killed in the middle of a decode.

func ComputeBudget 

func ComputeBudget(hw HardwareProfile) Budget

ComputeBudget derives the caps from a hardware profile. The shape follows the spec: a hard sequential floor on tiny machines, then download, process, and convert caps scaled by RAM and CPU, and DuckDB memory that grows on large hosts.

func (Budget) String 

func (b Budget) String() string

String renders the budget for arctic info.

type Comment 

type Comment struct {
	ID              string    `parquet:"id" json:"id"`
	Author          string    `parquet:"author" json:"author"`
	Subreddit       string    `parquet:"subreddit" json:"subreddit"`
	Body            string    `parquet:"body" json:"body"`
	Score           int64     `parquet:"score" json:"score"`
	CreatedUTC      int64     `parquet:"created_utc" json:"created_utc"`
	CreatedAt       time.Time `parquet:"created_at,timestamp" json:"created_at"`
	BodyLength      int32     `parquet:"body_length" json:"body_length"`
	LinkID          string    `parquet:"link_id" json:"link_id"`
	ParentID        string    `parquet:"parent_id" json:"parent_id"`
	Distinguished   string    `parquet:"distinguished" json:"distinguished"`
	AuthorFlairText string    `parquet:"author_flair_text" json:"author_flair_text"`
}

Comment is one Reddit comment as it lands in a Parquet shard. The columns are the queryable common case: identity, placement in time and community, the text, and the fields people filter on. created_utc is the raw epoch the source stores; created_at is the same instant as a timestamp so a reader can show a human time without converting.

func CommentFromJSON 

func CommentFromJSON(line []byte) (Comment, bool)

CommentFromJSON parses one dump line into a Comment. It returns ok=false when the line is not valid JSON, so the caller can count and skip it rather than abort the file. Missing scalars coerce to their zero value, matching the dumps' habit of omitting fields rather than nulling them.

type Config 

type Config struct {
	// DataDir is the root for per-entity imports and the local index.
	DataDir string
	// RawDir is where downloaded .zst files land.
	RawDir string
	// WorkDir is scratch for conversion.
	WorkDir string
	// RepoRoot stages the Hugging Face dataset repo during a publish.
	RepoRoot string
	// HFRepo is the Hugging Face dataset id to publish to.
	HFRepo string

	// Engine selects the conversion backend.
	Engine Engine
	// ChunkLines is the number of JSONL lines per Parquet shard.
	ChunkLines int
	// CommitEveryShards commits to the hub every N shards within a month so a
	// big month lands incrementally and a restart resumes mid-month. Zero
	// commits the whole month in one go at the end.
	CommitEveryShards int
	// DuckDBMemoryMB caps the DuckDB engine's memory.
	DuckDBMemoryMB int

	// MinFreeGB is the free-disk floor a publish refuses to start below.
	MinFreeGB int
	// DownloadFloorGB holds back starting a new month's download while free
	// disk is below this, so months process in parallel without the pipeline
	// piling up more .zst files than the disk can hold. Zero disables the gate.
	DownloadFloorGB int
	// MaxCommitStall is how long a Hugging Face commit may make no progress
	// before the publish exits with the restart code.
	MaxCommitStall time.Duration

	// MaxDownloads, MaxProcess, MaxConvertWorkers, and MaxDecodes bound
	// concurrency. Zero means "use the computed budget".
	MaxDownloads      int
	MaxProcess        int
	MaxConvertWorkers int
	MaxDecodes        int
}

Config holds the resolved paths and tunables for acquisition, processing, and publishing. The CLI fills it from flags and environment; the library reads it and never touches the environment itself.

func DefaultConfig 

func DefaultConfig() Config

DefaultConfig returns a Config with the standard paths and tunables. The ARCTIC_* environment variables override the defaults; a flag the CLI sets later overrides the environment in turn.

func (Config) EntityDir 

func (c Config) EntityDir(kind, name string) string

EntityDir returns the import directory for a subreddit or user.

func (Config) IndexPath 

func (c Config) IndexPath() string

IndexPath is the local catalog database path.

func (Config) ZstPath 

func (c Config) ZstPath(m Month, t Type) string

ZstPath returns where a month+type dump file lives once downloaded.

type DownloadCallback 

type DownloadCallback func(DownloadProgress)

DownloadCallback receives progress samples. It may be nil.

type DownloadProgress 

type DownloadProgress struct {
	Phase      string // "metadata" | "downloading" | "done"
	BytesDone  int64
	BytesTotal int64
	SpeedBps   float64
	Peers      int
	Elapsed    time.Duration
	Message    string
}

DownloadProgress is one progress sample streamed to a DownloadCallback while a torrent file is fetched.

type Engine 

type Engine string

Engine names a conversion backend. 

const (
	// EngineGo is the pure-Go Parquet writer. It is always available and is the
	// only engine on a CGO_ENABLED=0 build.
	EngineGo Engine = "go"
	// EngineDuckDB is the DuckDB-backed engine. It is present only in a binary
	// built with -tags duckdb.
	EngineDuckDB Engine = "duckdb"
)

func DefaultEngine 

func DefaultEngine() Engine

DefaultEngine is the engine a fresh Config uses. Without the duckdb build tag the pure-Go writer is the only option.

type ErrCorruption 

type ErrCorruption struct{ Msg string }

ErrCorruption marks a downloaded file that failed integrity checks. The caller should delete it and pull again from scratch.

func (*ErrCorruption) Error 

func (e *ErrCorruption) Error() string

type ErrNotPublished 

type ErrNotPublished struct{ Month Month }

ErrNotPublished marks a month after the bundle that has no monthly torrent yet. It is a no-data condition, not a failure.

func (*ErrNotPublished) Error 

func (e *ErrNotPublished) Error() string

type ErrTransient 

type ErrTransient struct{ Msg string }

ErrTransient marks a timeout or network failure. The partial data on disk is fine to resume from.

func (*ErrTransient) Error 

func (e *ErrTransient) Error() string

type HFClient 

type HFClient struct {
	Token string
	Repo  string // "namespace/name"
	HTTP  *http.Client
}

HFClient talks to the Hugging Face dataset API over plain HTTP with a bearer token. It carries no SDK dependency.

func NewHFClient 

func NewHFClient(token, repo string) *HFClient

NewHFClient builds a client for repo authenticated with token.

func (*HFClient) CreateDatasetRepo 

func (c *HFClient) CreateDatasetRepo(ctx context.Context, private bool) error

CreateDatasetRepo creates the dataset repo, treating an "already exists" response as success so the call is idempotent.

func (*HFClient) UploadFiles 

func (c *HFClient) UploadFiles(ctx context.Context, ops []HFOp) error

UploadFiles commits ops to the dataset's main branch in batches of ten, with three retries per batch. Files above lfsThreshold go through LFS; smaller files are inlined in the commit.

type HFOp 

type HFOp struct {
	LocalPath  string // source on disk (ignored when Delete is true)
	PathInRepo string // destination path in the dataset repo
	Delete     bool
}

HFOp is one file operation in a commit: add or update a file, or delete one.

type HardwareProfile 

type HardwareProfile struct {
	Hostname       string  `json:"hostname"`
	OS             string  `json:"os"`
	CPUs           int     `json:"cpus"`
	RAMTotalGB     float64 `json:"ram_total_gb"`
	RAMAvailableGB float64 `json:"ram_available_gb"`
	DiskFreeGB     float64 `json:"disk_free_gb"`
}

HardwareProfile describes the machine the tool runs on. The budget reads it to decide how many downloads, decodes, and conversions can run at once without thrashing memory or disk.

func DetectHardware 

func DetectHardware(workDir string) HardwareProfile

DetectHardware probes the current machine. workDir picks the volume the disk check runs against, since that is where downloads and conversion land.

func (HardwareProfile) String 

func (h HardwareProfile) String() string

String renders a one-line summary for arctic info.

type Index 

type Index struct {
	// contains filtered or unexported fields
}

Index is the local catalog of processed shards. It is a pure-Go SQLite database in WAL mode, one row per shard, so stats and coverage queries never re-read the Parquet.

func OpenIndex 

func OpenIndex(path string) (*Index, error)

OpenIndex opens or creates the index at path and ensures the schema exists.

func (*Index) Close 

func (idx *Index) Close() error

Close releases the database.

func (*Index) RecordShard 

func (idx *Index) RecordShard(t Type, year, month int, entity, path string, records, bytes int64) error

RecordShard upserts one processed shard. entity is empty for monthly dumps and the subreddit or user name for per-entity imports; for monthly dumps pass year and month instead.

func (*Index) Stats 

func (idx *Index) Stats(by string) ([]StatRow, error)

Stats returns rollups grouped by "month", "type", or "subreddit" (which also covers user entities). Rows are sorted by group.

type Month 

type Month struct {
	Year  int
	Month int
}

Month is a year and month in the catalog.

func CatalogEnd 

func CatalogEnd() Month

CatalogEnd is the last month the catalog can resolve: the later of the bundle end and the newest monthly torrent in the map.

func CatalogStart 

func CatalogStart() Month

CatalogStart is the first month in the catalog.

func MonthRange 

func MonthRange(start, end Month) []Month

MonthRange returns every month from start to end inclusive.

func ParseMonth 

func ParseMonth(s string) (Month, error)

ParseMonth parses a "YYYY-MM" string into a Month.

func (Month) After 

func (m Month) After(other Month) bool

After reports whether m comes strictly after other.

func (Month) Before 

func (m Month) Before(other Month) bool

Before reports whether m comes strictly before other.

func (Month) Next 

func (m Month) Next() Month

Next returns the month after m.

func (Month) String 

func (m Month) String() string

String renders a month as YYYY-MM.

type ProcessConfig 

type ProcessConfig struct {
	// StartShard skips emitting shards with an index below it. A resumed month
	// whose earlier shards are already committed fast-forwards past them without
	// rewriting or re-uploading.
	StartShard int
	// OnProgress, when set, reports the running count of decoded lines.
	OnProgress func(done int64)
	// OnShard, when set, is called after each shard at or above StartShard is
	// written and validated. Returning an error aborts the file.
	OnShard func(ShardDone) error
}

ProcessConfig tunes a streaming process run.

type ProcessResult 

type ProcessResult struct {
	Shards       int
	Records      int64
	SkippedLines int64
	Bytes        int64
}

ProcessResult summarizes what one .zst file produced.

func ProcessFile 

func ProcessFile(ctx context.Context, cfg Config, zstPath string, t Type, outDir string,
	cb func(done int64)) (ProcessResult, error)

ProcessFile decodes the .zst at zstPath, parses each line into the type's struct, groups lines into chunks of cfg.ChunkLines, and writes one Parquet shard per chunk. Unparseable lines are counted and skipped, never fatal. The cb callback, when set, reports the running count of decoded lines.

func ProcessFileStream 

func ProcessFileStream(ctx context.Context, cfg Config, zstPath string, t Type, pathFn ShardPathFunc,
	pc ProcessConfig) (ProcessResult, error)

ProcessFileStream is ProcessFileTo with per-shard callbacks and mid-month resume, so a caller can commit shards as they are produced and pick up after an interrupted run. The returned ProcessResult counts only the shards this call produced (those at or above pc.StartShard).

func ProcessFileTo 

func ProcessFileTo(ctx context.Context, cfg Config, zstPath string, t Type, pathFn ShardPathFunc,
	cb func(done int64)) (ProcessResult, error)

ProcessFileTo is like ProcessFile but names shards through pathFn, which the publish path uses to lay shards out at data/{type}/YYYY/MM/NNN.parquet. The directory of each shard path is created as needed.

type PublishOptions 

type PublishOptions struct {
	From     Month
	To       Month
	Types    []Type
	HFCommit bool // when false, process and stage shards but skip the hub commit
	Private  bool // create the dataset repo private if it does not exist
	Keep     bool // keep local .zst and shards after a commit instead of deleting
}

PublishOptions configures a publish run.

type ShardDone 

type ShardDone struct {
	N       int    // absolute shard index within the month
	Path    string // where the shard was written
	Records int64  // rows in this shard
	Bytes   int64  // Parquet size on disk
}

ShardDone reports a shard the processor just finished writing. The publish path uses it to commit shards to the hub as they land instead of waiting for the whole month.

type ShardPathFunc 

type ShardPathFunc func(n int) string

ShardPathFunc names a shard's output path given its sequence number. ProcessFile uses it for the publish layout; when nil, shards are written as NNN.parquet under outDir.

type ShardProgress 

type ShardProgress struct {
	Engine  string `json:"engine"` // shard boundaries differ per engine, so resume only within one
	Shards  int    `json:"shards"` // shards committed so far (also the next shard's index)
	Records int64  `json:"records"`
	Bytes   int64  `json:"bytes"`
}

ShardProgress records how much of a (month, type) has been committed to the hub. A publish updates it after every batch of shards, so a restart resumes mid-month instead of reprocessing from shard zero. It stays local; the hub only ever sees finished shards, the ledger, and the README.

type StatRow 

type StatRow struct {
	Group  string // month "YYYY-MM", type, or subreddit/user name
	Type   string // empty when grouping by type only
	Shards int64
	Count  int64
	Bytes  int64
}

StatRow is one rollup line returned by Stats.

type StatsRow 

type StatsRow struct {
	Year         int
	Month        int
	Type         string
	Shards       int
	Count        int64
	SizeBytes    int64 // total Parquet size across the month's shards
	ZstBytes     int64 // size of the source .zst, 0 when not recorded
	DurDownloadS float64
	DurProcessS  float64
	DurCommitS   float64
	CommittedAt  string // RFC3339; empty means not yet committed
}

StatsRow is one committed (month, type) entry in the publish ledger. The durations record how long each stage took so the README can show throughput.

func ReadStats 

func ReadStats(path string) ([]StatsRow, error)

ReadStats reads the stats.csv ledger. A missing file returns no rows and no error so a fresh run starts clean.

func (StatsRow) Key 

func (r StatsRow) Key() string

Key identifies a row as "YYYY-MM-type", the same key CommittedSet uses for resume.

type Submission 

type Submission struct {
	ID              string    `parquet:"id" json:"id"`
	Author          string    `parquet:"author" json:"author"`
	Subreddit       string    `parquet:"subreddit" json:"subreddit"`
	Title           string    `parquet:"title" json:"title"`
	Selftext        string    `parquet:"selftext" json:"selftext"`
	Score           int64     `parquet:"score" json:"score"`
	CreatedUTC      int64     `parquet:"created_utc" json:"created_utc"`
	CreatedAt       time.Time `parquet:"created_at,timestamp" json:"created_at"`
	TitleLength     int32     `parquet:"title_length" json:"title_length"`
	NumComments     int64     `parquet:"num_comments" json:"num_comments"`
	URL             string    `parquet:"url" json:"url"`
	Over18          bool      `parquet:"over_18" json:"over_18"`
	LinkFlairText   string    `parquet:"link_flair_text" json:"link_flair_text"`
	AuthorFlairText string    `parquet:"author_flair_text" json:"author_flair_text"`
}

Submission is one Reddit submission (a link or self post) as it lands in a Parquet shard.

func SubmissionFromJSON 

func SubmissionFromJSON(line []byte) (Submission, bool)

SubmissionFromJSON parses one dump line into a Submission, with the same skip-on-bad-line contract as CommentFromJSON.

type Type 

type Type string

Type names the two record kinds. It is the value of the --type flag and the directory segment in the published layout. 

const (
	TypeComments    Type = "comments"
	TypeSubmissions Type = "submissions"
)

func (Type) Prefix 

func (t Type) Prefix() string

Prefix returns the dump file prefix for the type: RC for comments, RS for submissions.

func (Type) Valid 

func (t Type) Valid() bool

Valid reports whether t is one of the known types.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.