parser

README

linkedin-auto

linkedin parser for w_popularity.

Strategy

LinkedIn aggressively blocks unauthenticated scraping. A logged-out curl https://www.linkedin.com/in/<handle>/ returns HTTP 999 with a tiny HTML body that JS-redirects to /authwall. Even the rare 200 responses carry an empty schema.org Person LD-JSON (no follower count), so the previous "public LD-JSON" approach is effectively dead.

This parser takes two paths, in priority order:

1. Authenticated HTML scrape via li_at session cookie (primary). When Config.LIATCookie is set we attach Cookie: li_at=<token> (and an optional JSESSIONID) to the request. LinkedIn then serves the real profile page HTML. We pull structured data out of it via three tolerant extractors, in order:

   • BPR datalets (<code id="datalet-bpr-guid-…"><!--{…}--></code>) — LinkedIn's own browser-proxy payload format. Most reliable surface today; carries followerCount, connectionCount, headline, locationName, industryName, currentCompany, etc.
   • window.__APOLLO_STATE__ — a JS global emitted by some render paths. Walked the same way.
   • <script type="application/ld+json"> with @type=Person — normally empty on logged-out fetches, but populated when authenticated.

   The extractor deep-walks every decoded JSON tree looking for known keys (followerCount, connectionsCount, headline, location, industryName, …) and stops at the first follower hit. Missing fields downgrade to Followers=0 plus a diagnostic Raw["fetch_note"].

2. camoufox via Playwright (fallback — skeleton). When Config.CamoufoxURL is set and LIATCookie is empty, we'd dial the CDP endpoint and pull the rendered HTML through the same extractor. Currently a stub (fetchViaCamoufox returns "camoufox path not implemented"); the branching is in place for a drop-in replacement.

Getting an li_at cookie

1. Log in to https://www.linkedin.com in a regular browser.
2. Open DevTools → Application → Cookies → https://www.linkedin.com.
3. Copy the value of the li_at cookie (it looks like AQEDA… followed by ~200 characters).
4. Optionally copy JSESSIONID too (LinkedIn wraps it in quotes; the parser will re-add the quotes if you strip them).

Caveat: li_at cookies expire roughly every 365 days, and may be invalidated sooner if LinkedIn detects unusual activity (different IP / UA, multiple concurrent sessions, etc). When the cookie is rejected the parser returns shared.ErrAuth with the message "li_at cookie expired" — that's your cue to refresh it.

For production deployments behind a single source IP, rotate the cookie across a small pool of accounts (and a residential proxy) to keep one expiry from taking down the pipeline.

Configuration

import parser "github.com/suenot/linkedin-auto"

p := parser.New(parser.Config{
    LIATCookie:  os.Getenv("LINKEDIN_LI_AT"),     // primary auth
    JSESSIONID:  os.Getenv("LINKEDIN_JSESSIONID"), // optional
    HTTPTimeout: 15 * time.Second,
    CamoufoxURL: os.Getenv("CAMOUFOX_URL"),       // optional fallback (stub)
})

snap, err := p.FetchChannel(ctx, "suenot")
// snap.Followers, snap.Raw["headline"], snap.Raw["location"], ...

Error mapping

Condition	Error
LIATCookie set, HTTP 999 / 403 / authwall body	ErrAuth: li_at cookie expired
LIATCookie empty, no CamoufoxURL	ErrAuth: set LI_AT cookie or CAMOUFOX_URL
HTTP 404	ErrNotFound
HTTP 429	ErrRateLimited
HTTP 5xx / transport error	ErrTransient
2xx with no extractable structure	snapshot, Followers=0, Raw["fetch_note"] set

FetchRecentPosts is best-effort. Profile pages don't expose a lifetime post list without hitting /detail/recent-activity/shares/ as the logged-in user; we extract whatever Article items happen to appear in LD-JSON and return (nil, nil) for everything else.

License

MIT

Documentation

Overview

Package parser implements the w_popularity LinkedIn adapter.

LinkedIn aggressively blocks unauthenticated scraping. A logged-out HTTP GET of https://www.linkedin.com/in/<handle>/ almost always returns HTTP 999 (LinkedIn's own "no scraping" status) and a body that JS-redirects to /authwall. The schema.org Person LD-JSON block is also empty on the logged-out path: there is no follower count to scrape.

Strategy (in priority order):

1. **Authenticated HTML scrape via `li_at` session cookie.** When Config.LIATCookie is set we attach `Cookie: li_at=...` (and an optional JSESSIONID) to the request. LinkedIn then serves the real profile page HTML, which embeds structured data in three possible shapes:

   a. BPR datalets:

   <code id="datalet-bpr-guid-…" style="display:none"> <!--{"data":{"data": {...}, "included": [{...}]}}--> </code>

   Each datalet body is an HTML comment around a JSON object. For profile pages, the `included` array contains nodes with follower / connection counters and headline/location fields. This is LinkedIn's own browser-proxy payload format and is the most reliable extraction surface today.

   b. window.__APOLLO_STATE__: occasionally a different render path emits an Apollo cache as a JS global. Less common, but worth trying as a secondary.

   c. <script type="application/ld+json"> with @type=Person: when authenticated the LD-JSON `interactionStatistic` array carries a real `followers` counter (zero for logged-out fetches, which is why the previous implementation always returned 0).

   The extractor is intentionally tolerant: it deep-walks every decoded JSON tree looking for known keys (followerCount, connectionsCount, headline, location, industryName, …) and stops at the first hit. We never want a LinkedIn UI rotation to break the whole pipeline; missing fields downgrade to 0 plus a diagnostic marker in Raw["fetch_note"].

2. **camoufox via Playwright (skeleton).** Config.CamoufoxURL is the CDP endpoint of a running camoufox instance (same shape as the facebook/instagram adapters). fetchViaCamoufox is currently a stub — the branching is in place so a real implementation is a drop-in replacement.

When both LIATCookie and CamoufoxURL are empty we return ErrAuth with the hint "set LI_AT cookie or CAMOUFOX_URL". When LIATCookie is set but LinkedIn still answers 999 we treat the cookie as expired and return ErrAuth with "li_at cookie expired".

Index

• type Config
• type LinkedInParser
  • func New(cfg Config) *LinkedInParser
  • func (p *LinkedInParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)
  • func (p *LinkedInParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)
  • func (p *LinkedInParser) Platform() shared.Platform

Types

type Config

type Config struct {
	// LIATCookie is the `li_at` session cookie of a logged-in LinkedIn
	// browser session. This is the primary auth surface. Pulled from
	// the LINKEDIN_LI_AT env var by callers. Empty disables the
	// authenticated path.
	LIATCookie string

	// JSESSIONID is an optional companion cookie. Some LinkedIn server
	// endpoints check it; the profile page generally doesn't, but we
	// forward it when present.
	JSESSIONID string

	// HTTPClient is optional; one with HTTPTimeout is constructed
	// otherwise. The default client does not auto-follow LinkedIn's
	// 301-to-authwall chain — we want to inspect the first response.
	HTTPClient *http.Client

	// HTTPTimeout caps every outbound call. Defaults to 15s.
	HTTPTimeout time.Duration

	// UserAgent overrides the default desktop Chrome UA. LinkedIn is
	// pickier about UAs than most sites; a realistic value is required.
	UserAgent string

	// CamoufoxURL is the CDP endpoint of a camoufox instance. When set,
	// and LIATCookie is not, the parser falls back to it. Empty
	// disables the fallback.
	CamoufoxURL string

	// BaseURL overrides https://www.linkedin.com. Test hook.
	BaseURL string
}

Config controls runtime behaviour.

type LinkedInParser

type LinkedInParser struct {
	// contains filtered or unexported fields
}

LinkedInParser is the public entry point.

func New

func New(cfg Config) *LinkedInParser

New constructs a LinkedIn parser.

func (*LinkedInParser) FetchChannel

func (p *LinkedInParser) FetchChannel(ctx context.Context, handle string) (shared.ChannelSnapshot, error)

FetchChannel returns the latest snapshot for handle. The handle may be a bare vanity name (`suenot`) or a full profile URL.

func (*LinkedInParser) FetchRecentPosts

func (p *LinkedInParser) FetchRecentPosts(ctx context.Context, handle string, since time.Time) ([]shared.PostSnapshot, error)

FetchRecentPosts is best-effort. Profile pages don't expose a post stream without hitting /detail/recent-activity/shares/ as the logged-in user — which costs a second request and a second cookie surface. For now we extract whatever Article items happen to appear in LD-JSON (rare, but free) and return (nil, nil) for everything else.

func (*LinkedInParser) Platform

func (p *LinkedInParser) Platform() shared.Platform

Platform implements shared.Parser.