parser

README

x-auto

X (Twitter) parser for w_popularity.

Strategy

The earlier Nitter-based revision is gone: every public Nitter mirror was returning 5xx, Anubis challenges, or DNS failures from server environments in 2026. The parser now tries four paths in order:

1. Official X API v2 — GET /2/users/by/username/<handle> with a free-tier bearer token (Config.BearerToken / X_BEARER_TOKEN). Returns public_metrics directly. Free tier: 100 user reads/month/app, plenty for daily snapshots. Also the only path that can enumerate a user's recent tweets with engagement metrics.

2. cdn.syndication.twimg.com/timeline/profile — Twitter's embed-syndication CDN, no auth required. The JSON envelope carries an HTML body containing an inline __INITIAL_STATE__ blob with the user payload. This endpoint has been gated down to empty bodies for most unauthenticated callers since ~2024, but we still try it: when it works it is the cheapest path.

3. api.fxtwitter.com — third-party JSON proxy run by the FixTweet project. Reliable plain-JSON profile reads: followers / following / tweets / likes / verification. Rate-limited but free. This is currently the workhorse for the no-auth case.

4. camoufox — Config.CamoufoxURL is reserved for a future browser- driven fallback. Returns shared.ErrNotImplemented until wired.

When every path fails the parser returns shared.ErrAuth wrapped with the hint "all X scraping paths failed; configure X_BEARER_TOKEN or CamoufoxURL".

What this parser collects

Field	Source
ChannelSnapshot.Followers	API v2 / syndication / fxtwitter
ChannelSnapshot.PostsCount	API v2 / syndication / fxtwitter
ChannelSnapshot.TotalLikes	API v2 / syndication / fxtwitter
ChannelSnapshot.Raw[name/description/created_at/verified/listed_count/following_count]	All paths populate as available; source records which path won.
PostSnapshot.Likes/Views/Comments/Shares	API v2 only. Public paths return no posts.

Usage

import parser "github.com/suenot/x-auto"

p := parser.New(parser.Config{
    BearerToken: os.Getenv("X_BEARER_TOKEN"), // optional
    HTTPTimeout: 15 * time.Second,
})

snap, err := p.FetchChannel(ctx, "elonmusk")
posts, err := p.FetchRecentPosts(ctx, "elonmusk", time.Now().Add(-24*time.Hour))

Without a bearer token, FetchRecentPosts returns an empty slice — none of the unauthenticated paths can reliably enumerate a user's timeline with engagement metrics. Use the bearer-token path or the camoufox fallback for post-level data.

Config

type Config struct {
    BearerToken    string          // X API v2; env: X_BEARER_TOKEN
    HTTPClient     *http.Client
    HTTPTimeout    time.Duration   // default 15s
    UserAgent      string
    CamoufoxURL    string          // reserved for future fallback
    APIBaseURL     string          // override https://api.x.com (tests)
    SyndicationURL string          // override https://cdn.syndication.twimg.com (tests)
    FxTwitterURL   string          // override https://api.fxtwitter.com (tests)

    NitterMirrors  []string        // accepted for backward compat, IGNORED
}

Errors

• shared.ErrNotFound — API v2 errors-array marks the handle as missing, or any path returns HTTP 404.
• shared.ErrRateLimited — any path returns HTTP 429.
• shared.ErrAuth — API v2 returns 401/403, OR every public path failed (with the configure-X_BEARER_TOKEN-or-CamoufoxURL hint).
• shared.ErrTransient — single-path transient errors (5xx, partial responses) that the parser silently retried against the next path.
• shared.ErrNotImplemented — camoufox path not yet wired.

License

MIT

Documentation

Overview

Package parser implements the w_popularity x (Twitter) adapter.

History: an earlier revision scraped public Nitter mirrors. As of 2026 every public Nitter mirror returns 5xx, Anubis challenges, or DNS failures from server environments, so that approach has been abandoned.

Strategy (tried in order):

1. Official X API v2 (preferred). Requires Config.BearerToken (or env X_BEARER_TOKEN). One GET /2/users/by/username/<handle> with public_metrics gives Followers / PostsCount / TotalLikes. Free tier: 100 user reads / month / app — fine for daily snapshots.

2. cdn.syndication.twimg.com/timeline/profile (no auth). Twitter's embed-syndication CDN. When it serves a body, the JSON wraps an HTML page containing an `__INITIAL_STATE__` blob with the user payload. The endpoint has been ratcheted down to empty bodies since ~2024 but we still try it: when it works it is the cheapest unauthenticated path.

3. api.fxtwitter.com (third-party proxy). Reliable JSON-only proxy returning followers / following / tweets / likes / verification. Owned by FixTweet; rate-limited but free.

4. camoufox-driven browser fallback. Not yet wired — `Config.CamoufoxURL` is reserved for that path. When set we return an explicit "not implemented" hint so the caller can decide whether to escalate to a manual / staffed fallback.

Index

• Constants
• type Config
• type XParser
  • func New(cfg Config) *XParser
  • func (p *XParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)
  • func (p *XParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)
  • func (p *XParser) Platform() shared.Platform

Constants

const DefaultUserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) " +
	"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36"

DefaultUserAgent is sent on every unauthenticated request. Some endpoints (notably cdn.syndication.twimg.com) reject bare Go clients with empty bodies, so we mimic a recent desktop browser.

Types

type Config

type Config struct {
	BearerToken    string
	HTTPClient     *http.Client
	HTTPTimeout    time.Duration
	UserAgent      string
	CamoufoxURL    string
	APIBaseURL     string
	SyndicationURL string
	FxTwitterURL   string

	// Deprecated. Accepted but ignored. Will be removed.
	NitterMirrors []string
}

Config controls runtime behaviour.

• BearerToken: X API v2 bearer (free-tier OAuth 2.0 app token). When non-empty the API v2 path is tried first.
• HTTPClient: optional; constructed from HTTPTimeout otherwise.
• HTTPTimeout: per-request budget. Default: 15s.
• UserAgent: overrides DefaultUserAgent.
• CamoufoxURL: reserved for a future browser-driven fallback. Not yet consumed by this implementation.
• APIBaseURL: override https://api.x.com (test hook).
• SyndicationURL: override https://cdn.syndication.twimg.com (test hook).
• FxTwitterURL: override https://api.fxtwitter.com (test hook).
• NitterMirrors: accepted for backward compatibility, IGNORED at runtime. Public Nitter mirrors are no longer reliable; the field is kept so existing configs do not fail to parse.

type XParser

type XParser struct {
	// contains filtered or unexported fields
}

XParser is the X (Twitter) adapter.

func New

func New(cfg Config) *XParser

New constructs a parser. It does not touch the network at construction time.

func (*XParser) FetchChannel

func (p *XParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)

FetchChannel tries each strategy in order until one returns a populated snapshot. See package doc for the priority list.

func (*XParser) FetchRecentPosts

func (p *XParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)

FetchRecentPosts returns recent posts. Only the API v2 path returns engagement metrics; the public unauthenticated paths return URL + ID + PublishedAt and leave Likes/Views/Comments at zero (or just return empty when the path cannot reach the user timeline).

func (*XParser) Platform

func (p *XParser) Platform() shared.Platform

Platform returns shared.PlatformX.