shift

Documentation

Overview

Package shift is the client for the Arctic Shift backfill API at https://arctic-shift.photon-reddit.com. It fetches a single subreddit's or a single user's full public Reddit history as JSONL over HTTP, paginated by the created_utc timestamp.

The API is the only path for individual users (the bulk torrents are keyed by subreddit, not by author) and the only path for data newer than the last published monthly dump. It serves records back to the 2005-01-01 epoch; this package clamps any earlier request up to that floor.

This package speaks HTTP and writes JSONL. It does no torrent work and no Parquet conversion; the arctic package and the CLI orchestrate those around the JSONL this package produces. The Inspect helper reads back a written entity directory to report counts and the date span without re-parsing into the full schema.

Index

• Constants
• Variables
• type Client
  • func NewClient() *Client
  • func (c *Client) FetchRange(ctx context.Context, kind, name string, t arctic.Type, after, before int64, ...) (int64, error)
  • func (c *Client) FetchSubreddit(ctx context.Context, name string, t arctic.Type, after, before int64, ...) (int64, error)
  • func (c *Client) FetchUser(ctx context.Context, name string, t arctic.Type, after, before int64, ...) (int64, error)
  • func (c *Client) GetMinDate(ctx context.Context, kind, name string) (int64, error)
• type Info
  • func Inspect(dir string) (Info, error)
• type Progress
• type ProgressCallback

Constants

const BaseURL = "https://arctic-shift.photon-reddit.com"

BaseURL is the public Arctic Shift API endpoint.

Variables

var ErrBlocked = errors.New("arctic shift api blocked the request")

ErrBlocked means the API rate-limited or refused the request and the caller should slow down or wait rather than retry hard. The CLI maps it to its "blocked" exit code.

Types

type Client

type Client struct {
	HTTP      *http.Client
	BaseURL   string
	UserAgent string
	// Limiter paces requests so a long backfill stays under the API's tolerance.
	// A nil limiter means no client-side pacing.
	Limiter *rate.Limiter
}

Client is a polite HTTP client for the Arctic Shift API. The zero value is not usable; build one with NewClient.

func NewClient

func NewClient() *Client

NewClient returns a Client with a descriptive User-Agent, a 60-second per-request timeout, and a limiter that allows roughly two requests per second with a small burst. Those defaults keep a multi-month backfill from tripping the server's rate limit while still moving at a useful pace.

func (*Client) FetchRange

func (c *Client) FetchRange(ctx context.Context, kind, name string, t arctic.Type, after, before int64, workers int, out io.Writer, cb ProgressCallback) (int64, error)

FetchRange fetches an entity's records of type t over [after, before) by splitting the span into one-month buckets and pulling them concurrently, then writing the buckets to out in chronological order. kind is "subreddit" or "user". It returns the total records written.

Each bucket lands in its own temp file first so concurrent workers never interleave their lines on out, then the files concatenate in order. The result is the same byte stream a single sequential FetchSubreddit or FetchUser would produce, gathered faster.

func (*Client) FetchSubreddit

func (c *Client) FetchSubreddit(ctx context.Context, name string, t arctic.Type, after, before int64, out io.Writer, cb ProgressCallback) (int64, error)

FetchSubreddit pages the API for one subreddit's records of type t with a created_utc in [after, before), writes one JSON object per line to out, and returns the number of records written. after is clamped up to arctic.MinEpoch; a before of zero means "up to now". name carries no r/ prefix.

func (*Client) FetchUser

func (c *Client) FetchUser(ctx context.Context, name string, t arctic.Type, after, before int64, out io.Writer, cb ProgressCallback) (int64, error)

FetchUser pages the API for one account's records of type t with a created_utc in [after, before), writes one JSON object per line to out, and returns the number of records written. Users have no per-subreddit torrent, so the API is the only path. after is clamped up to arctic.MinEpoch; a before of zero means "up to now". name carries no u/ prefix.

func (*Client) GetMinDate

func (c *Client) GetMinDate(ctx context.Context, kind, name string) (int64, error)

GetMinDate returns the earliest available created_utc epoch for an entity, clamped up to arctic.MinEpoch. kind is "subreddit" or "user"; name carries no r/ or u/ prefix. A response with no data is reported as an error so the caller can treat an unknown entity as no-data.

type Info

type Info struct {
	Comments    int64 `json:"comments"`
	Submissions int64 `json:"submissions"`
	Authors     int64 `json:"authors"`
	Subreddits  int64 `json:"subreddits"`
	FirstUTC    int64 `json:"first_utc"`
	LastUTC     int64 `json:"last_utc"`
	SizeBytes   int64 `json:"size_bytes"`
}

Info summarizes what is imported in an entity directory: the record counts by type, the number of distinct authors and subreddits, the first and last created_utc seen, and the total bytes of the JSONL files. The CLI's "sub info" and "user info" read this.

func Inspect

func Inspect(dir string) (Info, error)

Inspect scans the comments.jsonl and submissions.jsonl in dir and returns an Info. It counts lines, tracks distinct authors and subreddits, and records the min and max created_utc across both files. The "[deleted]" placeholder author is not counted as a distinct author.

Inspect writes an info.json cache in dir keyed on the JSONL file sizes, and reuses it on a later call when those sizes still match.

type Progress

type Progress struct {
	Type         string
	Count        int64
	ThroughEpoch int64
}

Progress reports how far a fetch has advanced. Count is the records written so far; ThroughEpoch is the created_utc of the most recent record seen, so a caller can show "caught up to date X".

type ProgressCallback

type ProgressCallback func(Progress)

ProgressCallback receives a Progress after each page. It may be nil.