scrape

README

The scrape package provides an experimental client for accessing additional GitHub data via screen scraping. It is designed to be a client of last resort for data that cannot be retrieved via the REST or GraphQL APIs.

What should be added here

Add only what you need. Whereas the main go-github library attempts to implement the entire GitHub REST API, there is little point in trying to do that here. Certainly, feel free to contribution patches to get data you actually need, but I'd rather not try and provide exhaustive coverage of all GitHub data here.

Add only what can't be accessed elsewhere. If the data can be retrieved through the REST or GraphQL API, use the appropriate libraries for that.

Prefer read-only access. For now, I'm only focusing on reading data. It might be that writing works fine as well, but it is of course much riskier.

How to add methods

See apps.go for examples of methods that access data. Basically, fetch the contents of the page using client.get, and then use goquery to dig into the markup on the page. Prefer selectors that grab semantic ID or class names, as they are more likely to be stable.

Documentation

Overview

Package scrape provides a client for interacting with GitHub using screen scraping.

This is intended to be used as a supplement to the standard go-github library to access data that is not currently exposed by either the official REST or GraphQL APIs.

Because of the nature of screen scraping, this package should be treated as HIGHLY EXPERIMENTAL, and potentially unstable. We make no promises relating to compatibility or stability of the exported API. Even though this package is distributed as part of the go-github library, it is explicitly exempt from any stability promises that my be implied by the library version number.

Index

• type AppManifest
• type Client
  • func NewClient(transport http.RoundTripper) *Client
  • func (c *Client) AppRestrictionsEnabled(org string) (bool, error)
  • func (c *Client) Authenticate(username, password, otpseed string) error
  • func (c *Client) CreateApp(m *AppManifest) (*http.Response, error)
  • func (c *Client) ListOAuthApps(org string) ([]OAuthApp, error)
  • func (c *Client) LoadCookies(v []byte) error
  • func (c *Client) OrgPaymentInformation(org string) (PaymentInformation, error)
  • func (c *Client) SaveCookies() ([]byte, error)
• type OAuthApp
• type OAuthAppReviewState
• type PaymentInformation

Types

type AppManifest

type AppManifest struct {
	// The name of the GitHub App.
	Name *string `json:"name,omitempty"`
	//Required. The homepage of your GitHub App.
	URL *string `json:"url,omitempty"`
	// Required. The configuration of the GitHub App's webhook.
	HookAttributes map[string]string `json:"hook_attributes,omitempty"`
	// The full URL to redirect to after the person installs the GitHub App.
	RedirectURL *string `json:"redirect_url,omitempty"`
	// A description of the GitHub App.
	Description *string `json:"description,omitempty"`
	// Set to true when your GitHub App is available to the public or false when
	// it is only accessible to the owner of the app.
	Public *bool `json:"public,omitempty"`
	// The list of events the GitHub App subscribes to.
	DefaultEvents []string `json:"default_events,omitempty"`
	// The set of permissions needed by the GitHub App.
	DefaultPermissions *github.InstallationPermissions `json:"default_permissions,omitempty"`
}

AppManifest represents a GitHub App manifest, used for preconfiguring GitHub App configuration.

type Client

type Client struct {
	*http.Client
	// contains filtered or unexported fields
}

Client is a GitHub scraping client.

func NewClient

func NewClient(transport http.RoundTripper) *Client

NewClient constructs a new Client. If transport is nil, a default transport is used.

func (*Client) AppRestrictionsEnabled

func (c *Client) AppRestrictionsEnabled(org string) (bool, error)

AppRestrictionsEnabled returns whether the specified organization has restricted third-party application access.

func (*Client) Authenticate

func (c *Client) Authenticate(username, password, otpseed string) error

Authenticate client to GitHub with the provided username, password, and if two-factor auth is enabled for the account, otpseed.

otpseed is the OTP Secret provided from GitHub as part of two-factor application enrollment. When registering the application, click the "enter this text code" link on the QR Code page to see the raw OTP Secret.

func (*Client) CreateApp

func (c *Client) CreateApp(m *AppManifest) (*http.Response, error)

CreateApp creates a new GitHub App with the given manifest configuration.

func (*Client) ListOAuthApps

func (c *Client) ListOAuthApps(org string) ([]OAuthApp, error)

ListOAuthApps lists the reviewed OAuth Applications for the specified organization (whether approved or denied).

func (*Client) LoadCookies

func (c *Client) LoadCookies(v []byte) error

LoadCookies loads the provided cookies for github.com.

func (*Client) OrgPaymentInformation

func (c *Client) OrgPaymentInformation(org string) (PaymentInformation, error)

OrgPaymentInformation returns payment information for the specified org.

func (*Client) SaveCookies

func (c *Client) SaveCookies() ([]byte, error)

SaveCookies returns an encoded form of the github.com cookies set on this client. If Authenticate() has been called, this should include the github.com session cookie. These cookies can be loaded onto a new client by calling LoadCookies.

GitHub session cookies are bearer tokens that are not tied to any particular client, so should be treated with the same sensitivity as the account credentials.

type OAuthApp

type OAuthApp struct {
	ID          int
	Name        string
	Description string
	State       OAuthAppReviewState
	RequestedBy string
}

OAuthApp represents an OAuth application that has been reviewed for access to organization data.

type OAuthAppReviewState

type OAuthAppReviewState int

OAuthAppReviewState indicates the current state of a requested OAuth Application.

const (
	// OAuthAppRequested indicates access has been requested, but not reviewed
	OAuthAppRequested OAuthAppReviewState = iota + 1
	// OAuthAppApproved indicates access has been approved
	OAuthAppApproved
	// OAuthAppDenied indicates access has been denied
	OAuthAppDenied
)

type PaymentInformation

type PaymentInformation struct {
	PaymentMethod    string
	LastPayment      string
	Coupon           string
	ExtraInformation string
}

PaymentInformation for an organization on a paid plan.