parser

README

youtube-auto

YouTube parser for w_popularity.

Status: functional (channel + recent videos), no auth required.

Strategy

Direct HTML scraping of youtube.com/@<handle>/about and youtube.com/@<handle>/videos with a modern desktop User-Agent and Accept-Language: en. Both pages embed a var ytInitialData = {...} blob that carries everything we need.

We deliberately do not use:

• the official YouTube Data API v3 (would require an API key and burns quota)
• the yt-dlp CLI (broke in late 2024 when YouTube migrated several channel pages to the new pageHeaderRenderer / aboutChannelViewModel shape — yt-dlp's extractor returns null for channel_follower_count, view_count, playlist_count)

The parser is tolerant of both YouTube response shapes:

• new (≥2024): aboutChannelViewModel.subscriberCountText / viewCountText / videoCountText
• legacy: header.c4TabbedHeaderRenderer.subscriberCountText / viewCountText / videosCountText
• ultra-new pageHeader: walked as a last-resort string-leaf scan for patterns like "487M subscribers" / "980 videos"

Usage

import parser "github.com/suenot/youtube-auto"

p := parser.New(parser.Config{})

snap, err  := p.FetchChannel(ctx, "@MrBeast")
posts, err := p.FetchRecentPosts(ctx, "@MrBeast", time.Now().AddDate(0, 0, -7))

Config

field	meaning	default
APIKey	no-op, kept for backwards compatibility — ignored	""
HTTPClient	override; used by tests to inject an httptest.Server	shared
HTTPTimeout	per-request budget when HTTPClient is not provided	30s
UserAgent	override outgoing UA	Chrome 120 desktop

Fields populated

ChannelSnapshot:

• Followers — from subscriberCountText (parses "1.2M", "487M", "1,234,567", etc.)
• PostsCount — from videoCountText / videosCountText
• TotalViews — from viewCountText
• Raw["channel_id"], Raw["title"], Raw["source"]="html"

PostSnapshot (one per video on /videos):

• PostID, URL (https://www.youtube.com/watch?v=<id>), Kind = video
• PublishedAt — derived from relative strings like "3 days ago", "1 month ago", "5 years ago", "3 дня назад". Month/year are approximated (30d / 365d).
• Views — from viewCountText
• Raw["title"], Raw["published_ago"], Raw["source"]="html"
• Likes and Comments are not exposed on the channel listing page, so they remain zero. (Per-video pages have them but each one costs an extra request; not worth it for a batch sweep.)

Error mapping

HTTP	mapped to
404, 410	shared.ErrNotFound
HTTP 200 + alerts[].type="ERROR"	shared.ErrNotFound (soft 404)
429	shared.ErrRateLimited
401, 403	shared.ErrAuth
5xx, network errors, parse failures	shared.ErrTransient
Body without ytInitialData	shared.ErrTransient

Rate-limit notes

Anonymous web scraping has no quota but YouTube will eventually 429 if you hammer the same handle. Cache results and stagger requests.

License

MIT

Documentation

Overview

Package parser implements the w_popularity YouTube adapter.

Strategy: direct HTML scraping of the YouTube channel pages. We fetch `https://www.youtube.com/@<handle>/about` (for the channel snapshot) and `https://www.youtube.com/@<handle>/videos` (for recent uploads) with a modern desktop User-Agent and Accept-Language: en. Each response embeds a `var ytInitialData = {...};` blob which carries everything we need.

We do NOT call any official YouTube Data API and we do NOT shell out to yt-dlp — that approach broke in late 2024 when YouTube migrated several channel pages to a new "pageHeaderRenderer"/"aboutChannelViewModel" shape in which yt-dlp's extractor returns nulls for follower / view / playlist counts.

The parser is tolerant of both the legacy `c4TabbedHeaderRenderer` shape and the newer `pageHeaderRenderer` + `aboutChannelViewModel` shape. It walks all string leaves in ytInitialData looking for patterns such as "29.6M subscribers", "123,020,785,579 views", "980 videos" — so it keeps working as long as YouTube renders those strings somewhere in the JSON.

Index

• type Config
• type YouTubeParser
  • func New(cfg Config) *YouTubeParser
  • func (p *YouTubeParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)
  • func (p *YouTubeParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)
  • func (p *YouTubeParser) Platform() shared.Platform

Types

type Config

type Config struct {
	APIKey      string // no-op, accepted for backwards compatibility
	HTTPClient  *http.Client
	HTTPTimeout time.Duration
	UserAgent   string
}

Config controls runtime behaviour. All fields are optional.

• APIKey is kept for backwards compatibility with callers that wired it up for the old yt-dlp+API path. It is currently a no-op; the parser no longer talks to the YouTube Data API.
• HTTPClient overrides the default client (used by tests to inject an httptest.Server via a URL-rewriting RoundTripper).
• HTTPTimeout is the per-request budget when HTTPClient is not provided (default 30s).
• UserAgent overrides the default desktop UA used in outgoing requests.

type YouTubeParser

type YouTubeParser struct {
	// contains filtered or unexported fields
}

YouTubeParser is the shared.Parser implementation for YouTube.

func New

func New(cfg Config) *YouTubeParser

New returns a parser configured per cfg. It does not touch the network at construction time.

func (*YouTubeParser) FetchChannel

func (p *YouTubeParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)

FetchChannel fetches /<handle>/about and pulls Followers, PostsCount, TotalViews and the canonical channel id out of the embedded ytInitialData.

func (*YouTubeParser) FetchRecentPosts

func (p *YouTubeParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)

FetchRecentPosts fetches /<handle>/videos and pulls the most recent video entries out of the embedded ytInitialData. Posts older than `since` are dropped. Ordering is newest-first as returned by YouTube.

func (*YouTubeParser) Platform

func (p *YouTubeParser) Platform() shared.Platform