page

package
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 25, 2026 License: Apache-2.0 Imports: 6 Imported by: 0

Documentation

Overview

Package page turns a github.com HTML document into structured data.

github.com is two applications sharing a domain. The React half ships its route props as JSON inside a script tag, and the Rails half ships schema.org microdata. Both are structured, which means most of what looks like scraping is really JSON decoding with an HTML document as the envelope.

Pulling text out of rendered markup is the third choice here, not the first. Where it is unavoidable, the selector lives in selectors.go with the date it was last checked against a live page, so a break is a one-file diff.

Index

Constants

This section is empty.

Variables

View Source 
var (
	ProfileFullName = Sel{Class: "p-name"}
	ProfileNickname = Sel{Class: "p-nickname"}
	ProfileVCardOrg = Sel{Class: "p-org"}
	// p-label holds the location text inside the homeLocation list item.
	ProfileVCardLabel = Sel{Class: "p-label"}
)

The microformat pair inside vcard-names. p-name is the display name and p-nickname is the login, and they carry the same two facts the microdata carries independently, which is why both are read: a disagreement between them is recorded as a conflict rather than resolved, because it would mean the page changed under us. Verified 2026-07-25 against torvalds and sindresorhus.

View Source 
var (
	ProfileDetail    = Sel{Tag: "li", Class: "vcard-detail"}
	ProfileDetailURL = Sel{Tag: "a", Attr: "rel", AttrContains: "me"}
	ProfileAnyLink   = Sel{Tag: "a", Attr: "href"}
)

ProfileDetail is one row of the vcard details list. Each row carries an itemprop naming what it holds (worksFor, homeLocation, url, social, email), so one selector plus the itemprop covers all five instead of five selectors that each break separately. Verified 2026-07-25 against sindresorhus, which has all of them but email.

View Source 
var (
	ProfileFollowers = Sel{Tag: "a", Attr: "href", AttrSuffix: "?tab=followers"}
	ProfileFollowing = Sel{Tag: "a", Attr: "href", AttrSuffix: "?tab=following"}
	ProfileStars     = Sel{Tag: "a", Attr: "href", AttrSuffix: "?tab=stars"}
	ProfileBoldCount = Sel{Tag: "span", Class: "text-bold"}
)

The tab links carrying the follower and following counts. The count is in a bold span inside the anchor and is a compact string like "313k", which is what ParseCompactCount is for. The href is absolute on a profile and relative on an organization, so these match on a suffix. Verified 2026-07-25 against torvalds and sindresorhus.

View Source 
var (
	ProfileTabItem    = Sel{Attr: "data-tab-item"}
	ProfileTabCounter = Sel{Class: "Counter"}
)

ProfileTabItem is one entry of the profile navigation. Each carries a data-tab-item naming the tab and, when the tab is non-empty, a Counter span with the number in it. That is where the repository, project, package, star, and sponsoring counts come from. Verified 2026-07-25 against sindresorhus.

View Source 
var (
	OrgHead       = Sel{Tag: "header", Class: "orghead"}
	OrgHeading    = Sel{Tag: "h1", Class: "h2"}
	OrgAvatarImg  = Sel{Tag: "img", Attr: "itemprop", AttrValue: "image"}
	OrgWebsite    = Sel{Tag: "a", Attr: "itemprop", AttrValue: "url"}
	OrgFollowers  = Sel{Tag: "a", Attr: "href", AttrContains: "/followers"}
	OrgMemberLink = Sel{Tag: "a", Class: "member-avatar"}
	// The location and email rows have no class or itemprop of their own, so
	// they are found by the icon inside them, the same trick the repository
	// licence link needs.
	OrgLocationRow = Sel{Tag: "li", HasDescendantClass: "octicon-location"}
	OrgEmailRow    = Sel{Tag: "li", HasDescendantClass: "octicon-mail"}
)

The organization page is a different template from a user profile: no vcard, no microformats, one pagehead block instead. These five are the whole of it. Verified 2026-07-25 against golang.

View Source 
var (
	ProfilePinnedList = Sel{Tag: "ol", Class: "js-pinned-items-reorder-list"}
	ProfileOrgAvatar  = Sel{Tag: "a", Attr: "data-hovercard-type", AttrValue: "organization"}
	ProfileUserLink   = Sel{Tag: "a", Attr: "data-hovercard-type", AttrValue: "user"}
	// ProfileReadme is the box a user profile puts its readme in. An
	// organization's readme has no class of its own, so the caller falls back
	// to the markdown article, which is the same on both.
	// Verified 2026-07-25 against sindresorhus and github.
	ProfileReadme    = Sel{Class: "profile-readme"}
	ProfileVCardList = Sel{Class: "vcard-details"}
	ProfileAchieve   = Sel{Class: "js-profile-achievements"}
)

ProfilePinned is the pinned-repository list, and ProfileOrgAvatar is the organization strip. The hovercard type attribute is the reliable hook on both: it is what the front end uses to decide which popover to fetch, so it is load-bearing for GitHub too and does not drift casually. Verified 2026-07-25 against torvalds.

View Source 
var (
	// The sidebar carries data-gid (the base64 node id) and data-url, which is
	// the canonical /{owner}/{repo}/discussions/{n}/sidebar path and therefore
	// the authority on which repository an /orgs/ URL belongs to.
	DiscussionSidebar = Sel{Attr: "id", AttrValue: "partial-discussion-sidebar"}
	DiscussionTitle   = Sel{Class: "js-issue-title"}
	DiscussionNumber  = Sel{Class: "gh-header-number"}
	// Every status pill in the header is a span.State. Which one it is comes
	// from the title attribute: "Status: Closed as resolved", "Answered".
	DiscussionState    = Sel{Tag: "span", Class: "State", Attr: "title"}
	DiscussionCategory = Sel{Tag: "a", Attr: "href", AttrContains: "/discussions/categories/"}
	// Labels carry the name in data-name, which is the raw name before the
	// truncation span gets at it.
	DiscussionLabel       = Sel{Tag: "a", Class: "IssueLabel", Attr: "data-name"}
	DiscussionUpvote      = Sel{Class: "js-upvote-button", Attr: "aria-label"}
	DiscussionComment     = Sel{Class: "js-comment-container", Attr: "data-gid"}
	DiscussionBody        = Sel{Class: "js-comment-body"}
	DiscussionAuthor      = Sel{Tag: "a", Class: "author"}
	DiscussionParticipant = Sel{Tag: "a", Class: "participant-avatar"}
	DiscussionAnswerLink  = Sel{Class: "js-discussions-goto-answer-button"}
)

Discussions are the one thread type with no React payload at all: the page is Rails, and the only structured block on it is a schema.org QAPage. So the fields that block does not carry (category, labels, participants, the node id) come from these, and everything here is checked against the JSON-LD where the two overlap. Verified 2026-07-25 against orgs/community#1 and google/docsy-example#479.

View Source 
var (
	HovercardBody  = Sel{Class: "markdown-body-short"}
	HovercardTitle = Sel{Class: "markdown-title"}
	HovercardState = Sel{Tag: "span", Class: "State", Attr: "title"}
	HovercardLabel = Sel{Class: "IssueLabel", Attr: "data-name"}
	HovercardRef   = Sel{Class: "commit-ref"}
)

The hovercard is a small HTML fragment served to XHR requests. It is the only keyless source for a pull request's body: the conversation route ships merge metadata and nothing else. The body it gives is truncated to about a line and a half, which is why the record calls it a snippet. Verified 2026-07-25 against cli/cli#9000.

View Source 
var (
	RepoListItem  = Sel{Tag: "li", Attr: "itemprop", AttrValue: "owns"}
	RepoNameLink  = Sel{Tag: "a", Attr: "itemprop", AttrValue: "name codeRepository"}
	RepoListDesc  = Sel{Attr: "itemprop", AttrValue: "description"}
	RepoListLang  = Sel{Attr: "itemprop", AttrValue: "programmingLanguage"}
	RepoLangColor = Sel{Class: "repo-language-color"}
	RepoStarsLink = Sel{Tag: "a", Attr: "href", AttrSuffix: "/stargazers"}
	RepoForksLink = Sel{Tag: "a", Attr: "href", AttrSuffix: "/forks"}
	RepoTopicTag  = Sel{Tag: "a", Class: "topic-tag"}
	RepoLabel     = Sel{Class: "Label"}
)

RepoListItem is one row. The anchor inside carries itemprop="name codeRepository", which is the microdata hook and the reason this parser is anchored on meaning rather than on layout. Verified 2026-07-25 against torvalds?tab=repositories, 12 rows.

View Source 
var (
	TrendingRow      = Sel{Tag: "article", Class: "Box-row"}
	TrendingHeading  = Sel{Tag: "h2"}
	TrendingDesc     = Sel{Tag: "p"}
	TrendingPeriod   = Sel{Class: "float-sm-right"}
	TrendingBuiltBy  = Sel{Tag: "img", Class: "avatar-user"}
	TrendingDevRow   = Sel{Class: "Box-row"}
	TrendingDevName  = Sel{Tag: "h1", Class: "h3"}
	TrendingDevRepo  = Sel{Tag: "h1", Class: "h4"}
	TrendingSponsors = Sel{Tag: "a", Attr: "href", AttrPrefix: "/sponsors/"}
)

TrendingRow is one repository card. The page is Rails and is the only source anywhere for the trending list: there is no JSON equivalent, tokened or not. Verified 2026-07-25 against /trending.

View Source 
var (
	TopicHeading   = Sel{Tag: "h1"}
	TopicShortDesc = Sel{Tag: "p", Class: "f4"}
	TopicMarkdown  = Sel{Class: "markdown-body"}
	TopicWikipedia = Sel{Tag: "a", Attr: "href", AttrPrefix: "https://en.wikipedia.org"}
	TopicRepoRow   = Sel{Tag: "article", Class: "border"}
)

The topic page carries metadata the search result does not: the long description, the logo, the creator, the release date, the Wikipedia link, and the aliases. Verified 2026-07-25 against /topics/go.

View Source 
var (
	ReleaseBox       = Sel{Class: "Box"}
	ReleaseMarkdown  = Sel{Class: "markdown-body"}
	ReleaseAssetRow  = Sel{Class: "Box-row"}
	ReleaseTagIcon   = Sel{Class: "octicon-tag"}
	ReleaseLabel     = Sel{Class: "Label"}
	ReleaseAssetLink = Sel{Tag: "a", Attr: "href", AttrPrefix: "/"}
)

The release list page is lazy: a cold fetch of /releases carries no /releases/tag/ links at all, which is why the list comes from releases.atom and only the per-release page is parsed here. Download counts exist nowhere else on a keyless surface, which is what makes the extra request worth it. Verified 2026-07-25 against gohugoio/hugo.

View Source 
var (
	DependencyRow = Sel{Attr: "data-test-selector", AttrValue: "dg-repo-pkg-dependency"}
	// The name is an anchor when GitHub resolved the package to a repository
	// and a plain span when it did not, so the class is the only thing both
	// forms share.
	DependencyName     = Sel{Class: "h4"}
	DependencyLink     = Sel{Tag: "a", Attr: "data-hovercard-type", AttrValue: "dependendency_graph_package"}
	DependencyVersion  = Sel{Tag: "span", Class: "text-mono"}
	DependencyRelation = Sel{Tag: "a", Attr: "data-test-selector", AttrValue: "relationship-label-link"}
	DependencyManifest = Sel{Tag: "a", Attr: "data-test-selector", AttrValue: "dg-repo-pkg-manifest"}

	DependentRow   = Sel{Attr: "data-test-id", AttrValue: "dg-repo-pkg-dependent"}
	DependentRepo  = Sel{Tag: "a", Attr: "data-hovercard-type", AttrValue: "repository"}
	DependentUser  = Sel{Tag: "a", Attr: "data-hovercard-type", AttrValue: "user"}
	DependentStars = Sel{Tag: "span", Class: "text-bold", HasDescendantClass: "octicon-star"}
	DependentForks = Sel{Tag: "span", Class: "text-bold", HasDescendantClass: "octicon-repo-forked"}
	// The dependents pager is a cursor in a button, not a rel="next" anchor,
	// so it needs its own selector and its own token.
	DependentNext = Sel{Tag: "a", Class: "BtnGroup-item", Attr: "href", AttrContains: "dependents_after="}
)

The two dependency pages are the only keyless source for who depends on whom, and they are the most fragile markup this tool reads: the rows are identified by test hooks rather than by classes, and one of those hooks carries GitHub's own typo, "dependendency". It is spelled here the way the page spells it, and a fix on their side will show up as a missing edge rather than a wrong one.

The two pages disagree on everything, including which attribute names a row: dependencies uses data-test-selector and dependents uses data-test-id. Verified 2026-07-25 against gohugoio/hugo.

View Source 
var (
	GistFileBox    = Sel{Class: "file"}
	GistFileHeader = Sel{Class: "file-header", Attr: "data-path"}
	GistBlobWrap   = Sel{Class: "blob-wrapper"}
)

Verified 2026-07-25 against gist.github.com.

View Source 
var (
	BoxRow       = Sel{Class: "Box-row"}
	MarkdownBody = Sel{Class: "markdown-body"}
	Pagination   = Sel{Class: "pagination"}
	NextPage     = Sel{Tag: "a", Attr: "rel", AttrValue: "next"}
)

BoxRow is the generic Rails list row. It appears on the org people page, the tags page, the release assets list, and half a dozen others, which is why it is here once rather than in six decoders. Verified 2026-07-25.

View Source 
var LanguageBarItem = Sel{Tag: "a", Attr: "href", AttrContains: "/search?l="}

LanguageBarItem is one segment of the coloured language bar under the About box. Used only when sections.languages arrives empty, which is the common case on a cold page. Verified 2026-07-25 against gohugoio/hugo.

View Source 
var LanguageBarName = Sel{Tag: "span", Class: "text-bold"}

LanguageBarName is the bold span holding just the language name, so the percentage that follows it in the anchor text can be told apart from it without splitting on whitespace and hoping. Verified 2026-07-25 against gohugoio/hugo.

View Source 
var LicenseLink = Sel{Tag: "a", HasDescendantClass: "octicon-law"}

LicenseLink is the About-sidebar licence link. There is no licence field in any JSON payload on any keyless surface, so this anchor is the only source. It is the most fragile selector in the tool: it is identified by the octicon-law SVG inside it rather than by a class on the anchor, because the anchor's classes change more often than the icon does. Verified 2026-07-25 against gohugoio/hugo.

View Source 
var ProfileAchievement = Sel{Tag: "a", Attr: "href", AttrContains: "achievement="}

ProfileAchievement is one achievement badge. The label lives in the img alt as "Achievement: Pair Extraordinaire", and the slug lives in the href query, so the slug is what gets recorded. Verified 2026-07-25 against torvalds.

View Source 
var ProfileBio = Sel{Class: "user-profile-bio", Attr: "data-bio-text"}

ProfileBio is the bio, which is carried in a data attribute as well as in the element text. The attribute is the one to read: it is the source form, before GitHub's own link and emoji rewriting. Verified 2026-07-25 against torvalds.

View Source 
var ProfileNames = Sel{Class: "vcard-names-container"}

ProfileNames anchors the display-name block when the microdata hooks change. The container carries js-profile-editable-names, which was confirmed present. Verified 2026-07-25 against torvalds.

View Source 
var ProfilePinnedItem = Sel{Tag: "li", Class: "js-pinned-item-list-item"}

ProfilePinnedItem is one pinned repository. The repository link inside it is the plain anchor whose href has two path segments. Verified 2026-07-25 against sindresorhus.

View Source 
var RelTimeEl = Sel{Tag: "relative-time", Attr: "datetime"}

RelTimeEl is the <relative-time> custom element. Its datetime attribute is an ISO timestamp; its text is "3 days ago" and is never parsed.

Functions

func Attr 

func Attr(n *html.Node, name string) string

Attr reads one attribute, empty when it is absent.

func BlockText 

func BlockText(n *html.Node) string

BlockText returns the prose of a subtree with the line structure the markup implies, which is what Text deliberately throws away.

Text collapses a whole subtree onto one line, which is right for a label and wrong for a document: a twenty-kilobyte README as a single line is not a readable rendering of anything. This keeps one line per block element, one blank line between paragraphs, and the interior whitespace of a <pre> exactly as it was, since indentation is the meaning of a code block rather than decoration on it.

func CountIn 

func CountIn(s string) (int, string, bool)

CountIn parses the first token of a rendered label, which is how the profile counters read: "313k followers", "1.2k following".

func Find 

func Find(root *html.Node, s Sel) *html.Node

Find returns the first matching element, or nil.

func FindAll 

func FindAll(root *html.Node, s Sel) []*html.Node

FindAll returns every matching element in document order.

func FragmentText 

func FragmentText(s string) string

FragmentText is BlockText over an HTML fragment that arrived as a string. Several of GitHub's payloads carry rendered markup as a JSON value rather than as part of the document, so there is no node to walk until this parses one.

func HasClass 

func HasClass(n *html.Node, want string) bool

HasClass matches one class in the whitespace-separated list, never a substring. `Box-row` must not match `Box-row-hover`.

func OuterHTML 

func OuterHTML(n *html.Node) string

OuterHTML re-renders a subtree. It is how a README makes it into a record with its markup intact: the alternative is refetching the fragment, and the fragment is already here.

func ParseCompactCount 

func ParseCompactCount(s string) (n int, display string, ok bool)

ParseCompactCount reads a rendered count and returns the integer plus the original string. Both are kept because the grouping separator depends on the locale GitHub infers, so "8,112" and "8.112" are the same number, and throwing away the original would make that ambiguity invisible.

Returns ok=false rather than zero when the string is not a number at all. Absent is not zero: a parser that returns 0 on failure is a bug that ships quietly and is discovered a year later in someone's aggregate.

func RelTime 

func RelTime(n *html.Node) string

RelTime returns the datetime attribute of the first <relative-time> descendant. The element's own text is never read: it is localised and relative, and parsing it would be a whole class of bug for no gain.

func Text 

func Text(n *html.Node) string

Text returns the concatenated, whitespace-collapsed text of a subtree. It is the last-resort extractor and every caller of it is marked as such.

func Walk 

func Walk(n *html.Node, fn func(*html.Node) bool)

Walk visits every node depth-first. Returning false from fn stops the traversal of that subtree and, once found is set, the search as a whole.

Types

type Page 

type Page struct {
	URL       string `json:"url"`
	Canonical string `json:"canonical,omitempty"`
	Title     string `json:"title,omitempty"`
	Plane     Plane  `json:"plane"`

	// Payload is the React route props, the union of every route object the
	// server sent for this page.
	Payload map[string]json.RawMessage `json:"payload,omitempty"`

	// Queries holds the Relay results the server preloaded, keyed by query
	// name. This is a GraphQL response without a GraphQL token, and on issue
	// and pull request pages it is where everything lives.
	Queries map[string]json.RawMessage `json:"queries,omitempty"`

	// StructuredData is payload.structured_data, GitHub's own schema.org view
	// of the page. Its url field is wrong, see Canonical.
	StructuredData json.RawMessage `json:"structured_data,omitempty"`

	// LinkedData is every <script type="application/ld+json"> block, verbatim.
	LinkedData []json.RawMessage `json:"linked_data,omitempty"`

	// Partials are the react-partial blocks worth keeping. Most are chrome
	// (header, footer, command palette) and are dropped by size and key count.
	Partials map[string]json.RawMessage `json:"partials,omitempty"`

	// Meta is og: and twitter: content, keyed by the full property name.
	Meta map[string]string `json:"meta,omitempty"`

	// Microdata is a multimap of itemprop name to the values found, which is
	// how the Rails pages publish their fields.
	Microdata map[string][]string `json:"microdata,omitempty"`

	// Fragments are the deferred loads the page names for itself, from
	// <include-fragment src> and <turbo-frame src>. Discovering them by
	// scanning rather than by hardcoded path means a new one shows up in
	// `github page` before any record models it.
	Fragments []string `json:"fragments,omitempty"`

	Bytes int `json:"bytes"`

	// HTML is the original document, kept for the selector passes. It is not
	// serialised: printing 300 KB of markup inside a JSON record helps nobody.
	HTML []byte `json:"-"`
	// contains filtered or unexported fields
}

Page is the whole HTML plane of one document in one struct. Every page-derived record is built from a Page and never from a raw string, which is what makes `github page` possible: it prints this, and every record is a projection of it.

func Extract 

func Extract(url string, doc []byte) *Page

Extract runs the nine-step read: find the script blocks, classify them, unpack the app payload and its preloaded queries, then collect the linked data, the meta tags, the microdata, the canonical URL, and the deferred fragments.

It never returns an error. A page that carries nothing this understands is a Page with nothing in it, and deciding whether that is a failure belongs to the caller who knows what it was looking for.

func (*Page) Doc 

func (p *Page) Doc() *html.Node

Doc parses the page once and caches the tree. Most reads never call it: the React pages carry their data as JSON and the scanner alone is enough.

func (*Page) FirstQuery 

func (p *Page) FirstQuery() (string, json.RawMessage, bool)

FirstQuery returns the first preloaded result, which on the thread pages is the one that matters. Map order is random, so the names are sorted first: a decoder that works on one run and not the next is worse than no decoder.

func (*Page) MetaContent 

func (p *Page) MetaContent(name string) string

MetaContent returns an og: or twitter: value.

func (*Page) Prop 

func (p *Page) Prop(name string) string

Prop returns the first value of a microdata itemprop.

func (*Page) Props 

func (p *Page) Props(name string) []string

Props returns every value of a microdata itemprop.

func (*Page) Query 

func (p *Page) Query(name string) (json.RawMessage, bool)

Query returns one preloaded Relay result by name, or by suffix when the caller does not want to spell out the whole generated name.

func (*Page) Route 

func (p *Page) Route(name string) (json.RawMessage, bool)

Route returns one React route object from the payload. Route props are a delta against the mounted layout, so a route that was present on one fetch can be absent on the next; every caller checks the second return value.

type Plane 

type Plane string

Plane says which of the two applications rendered a page. It is worth knowing because it decides where the data is: React pages carry an app payload and Rails pages carry microdata. 

const (
	PlaneReact Plane = "react"
	PlaneRails Plane = "rails"
)

type Sel 

type Sel struct {
	Tag          string
	ID           string
	Class        string // one class, matched against the whitespace-separated list
	Attr         string // attribute that must be present
	AttrValue    string // and, if set, must equal this
	AttrPrefix   string
	AttrSuffix   string
	AttrContains string
	// HasDescendantClass requires a descendant element carrying this class.
	// It exists for the licence link, which is identified by the icon inside
	// it because the icon changes less often than the anchor's own classes.
	HasDescendantClass string
}

Sel is one selector. A zero field is "do not care", so a selector that only sets Class matches on class alone.

func (Sel) Match 

func (s Sel) Match(n *html.Node) bool

Match reports whether n satisfies every field the selector set.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.