parser

README

smartlab-auto

smartlab parser for w_popularity.

Scrapes public Smart-Lab profile pages (https://smart-lab.ru/my/<handle>/) for follower count, post count, and a list of recent posts with their views and comment counters.

Status

Implemented. No credentials required — profile pages are fully public.

Fields extracted

ChannelSnapshot

Field	Source
Followers	Читают counter in .blog_statistic
PostsCount	Записи counter in .blog_statistic
TotalViews	not exposed on profile page — 0
Raw[title]	<h3> display name inside .blog_info

PostSnapshot

Field	Source
PostID	tid="..." attribute on <div class="topic ...">
URL	https://smart-lab.ru/blog/<id>.php
Kind	PostKindPost
PublishedAt	<li class="date">10 января 2024, 18:55</li> (MSK → UTC)
Views	<span class="views-span" title="Просмотрели 3.5К"> (K/M)
Comments	<li class="comments-total"><span class="red">37</span></li>
Raw[title]	text of the topic's <h2 class="title"><a>

Charset

Smart-Lab today serves Content-Type: text/html; charset=utf-8. Historical pages used Windows-1251; the parser sniffs Content-Type and <meta charset> and decodes via golang.org/x/text/encoding/charmap when needed.

Usage

import parser "github.com/suenot/smartlab-auto"

p := parser.New(parser.Config{HTTPTimeout: 15 * time.Second})
snap, err := p.FetchChannel(ctx, "suenot")
posts, err := p.FetchRecentPosts(ctx, "suenot", time.Time{})

Errors

• shared.ErrNotFound — HTTP 404 (handle does not exist).
• shared.ErrRateLimited — HTTP 429.
• shared.ErrTransient — 5xx, network error, body-read failure.

Tests

go test ./... runs unit tests against embedded fixtures from testdata/profile_suenot.html (UTF-8) and testdata/profile_cp1251.html (Windows-1251). No network access required.

License

MIT

Documentation

Overview

Package parser implements the w_popularity smart-lab.ru adapter.

Strategy:

primary:  HTML scrape of https://smart-lab.ru/my/<handle>/
fallback: none — Smart-Lab profile pages are fully public

The profile page exposes follower count ("Читают"), post count ("Записи") and a chronological list of the user's blog posts. Each post block carries id, slug-style URL (/blog/<id>.php), human-readable publish date, view count (e.g. "3.5К") and a comment counter. We extract those fields with a combination of tokenizer + regexp, tolerating layout drift by recording uncertain values in Raw rather than failing the snapshot.

Smart-Lab today serves UTF-8; historic pages used Windows-1251. The HTTP helper sniffs Content-Type and the page's <meta charset> to pick the right decoder via golang.org/x/text/encoding/charmap.

Index

• type Config
• type SmartLabParser
  • func New(cfg Config) *SmartLabParser
  • func (p *SmartLabParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)
  • func (p *SmartLabParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)
  • func (p *SmartLabParser) Platform() shared.Platform

Types

type Config

type Config struct {
	HTTPClient  *http.Client
	HTTPTimeout time.Duration
	UserAgent   string
	BaseURL     string

	Credential  string
	CamoufoxURL string
}

Config controls runtime behaviour.

• HTTPClient: if nil, New() builds one with HTTPTimeout (default 15s).
• HTTPTimeout: only used when HTTPClient is nil.
• UserAgent: defaults to a Chrome-on-macOS string. Smart-Lab serves a stub page to clients that look like default Go.
• BaseURL: defaults to "https://smart-lab.ru". Override for tests.
• Credential / CamoufoxURL: reserved for future use; not consulted today.

type SmartLabParser

type SmartLabParser struct {
	// contains filtered or unexported fields
}

SmartLabParser implements shared.Parser for smart-lab.ru.

func New

func New(cfg Config) *SmartLabParser

New constructs a parser. It performs no I/O.

func (*SmartLabParser) FetchChannel

func (p *SmartLabParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)

FetchChannel returns a ChannelSnapshot for the given handle (no leading /).

func (*SmartLabParser) FetchRecentPosts

func (p *SmartLabParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)

FetchRecentPosts returns the user's posts from the profile page, newest-first. The public page paginates aggressively; we return what's on the first page and let the caller request more via a tighter `since` window if needed.

func (*SmartLabParser) Platform

func (p *SmartLabParser) Platform() shared.Platform