Documentation
¶
Overview ¶
Package academic is the provider boundary for scholarly sources (§10.2).
arXiv and PubMed/PMC are the keyless defaults; Unpaywall turns a DOI into a legal open-access copy.
§10.2 calls Unpaywall "the single highest-leverage call in the package", and measurement does not bear that out for how mole uses it. Over a 97-paper sweep of the contradiction corpus, 46 papers carried no DOI at all, and of the 28 unreadable ones that did, Unpaywall placed none — leaving it worth about 5%. The binding constraint on academic coverage is missing identifiers, not unresolved ones. It is kept because that 5% is real and because a DOI is what the other providers hand around; the claim about leverage is not. Semantic Scholar and OpenAlex are deliberately absent — they serve citation-graph queries, which is a different capability from finding papers about a question. So is ScienceDirect/Elsevier, for a harder reason: full text there needs an institutional entitlement the API does not grant.
The API surface is the easy part. §10.3 is the milestone: every provider is rate-limited and identifies itself, because a worker pool without that gets the user banned in week one. Both are enforced here rather than left to each implementation, so a new provider cannot forget.
Index ¶
- Constants
- Variables
- func ArXivHTMLURL(arXivID string) string
- func CheckContact(email string) error
- func LimitFor(k Kind) limiter.Limit
- func LimiterKey(k Kind) string
- func PMCArticleURL(pmcid string) string
- func Register(l *limiter.Limiter)
- type ArXiv
- type Config
- type FullTextFormat
- type Kind
- type Options
- type Paper
- type Provider
- type PubMed
- type Resolver
- type Response
- type Unpaywall
Constants ¶
const DefaultArXivBaseURL = "https://export.arxiv.org/api/query"
DefaultArXivBaseURL is arXiv's Atom query endpoint.
https, not the http the documentation still shows: plain http returned an empty body from here, and a search provider that silently yields nothing is worse than one that fails.
const DefaultMaxResults = 10
DefaultMaxResults bounds a search when the caller does not.
const DefaultPubMedBaseURL = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils"
DefaultPubMedBaseURL is NCBI's E-utilities root.
const DefaultUnpaywallBaseURL = "https://api.unpaywall.org/v2"
DefaultUnpaywallBaseURL is Unpaywall's v2 API.
Variables ¶
var ErrNoContact = errors.New("academic: no contact email configured")
ErrNoContact is returned when a provider is built without a contact address.
A sentinel rather than a string because §10.3 makes this a startup check rather than a README line, and a caller has to be able to tell "you have not configured this yet" from "the provider is down".
var ErrRateLimited = errors.New("academic: rate limited")
ErrRateLimited is a 429 or a provider's documented throttle response.
executor.Classify has an arm for it, so §9.5 treats it as transient — retried with backoff, never aborting the lead — and DeadEndCause labels it rate_limited rather than a generic failure. That arm was missing when this sentinel was introduced, and the doc claimed the classification anyway: the error fell through to Degraded and was never retried.
Functions ¶
func ArXivHTMLURL ¶
ArXivHTMLURL is where a paper's HTML full text would live if it has any.
A candidate, not a fact — see entryToPaper. The caller confirms it with a request; this only spells the URL so two places cannot spell it differently.
func CheckContact ¶
CheckContact reports whether an address is usable for provider identification.
Exported so `mole doctor` reports readiness from the same rule that gates the providers. A second, hand-written check in the CLI would be free to drift, and the whole point of a startup check is that it agrees with the thing it gates.
func LimitFor ¶
LimitFor is §10.3's table, in code.
The numbers are the providers' own published constraints, not guesses, and they are conservative where the provider is vague. Getting these wrong is not a performance question: these are shared public services run by libraries and government agencies, and the failure mode is a block on the user's address.
The keys are prefixed rather than bare, because this limiter is shared with the web fetcher, which keys by hostname — an unprefixed "arxiv" would be a different bucket from the "arxiv.org" the fetcher uses, and a prefixed one cannot silently collide with either.
func LimiterKey ¶
LimiterKey is the bucket a provider's calls are charged to.
func PMCArticleURL ¶
PMCArticleURL is where a PMC article is read.
pmc.ncbi.nlm.nih.gov, not the www.ncbi.nlm.nih.gov/pmc path the older documentation uses: that one 301s here, and following a redirect on every fetch is a request NCBI does not need to serve.
func Register ¶
Register installs every provider's limit on a limiter.
Called once where the limiter is built, so the constraints are in force before any provider exists to violate them. Registering per provider construction would mean a provider built twice resets its own bucket — which is exactly the Crawl-delay bug M5's review found in the fetcher.
Types ¶
type ArXiv ¶
type ArXiv struct {
// contains filtered or unexported fields
}
ArXiv searches arXiv's Atom API. Keyless (§10.2).
func NewArXiv ¶
NewArXiv builds the provider, refusing without a contact address (§10.3).
No generic New yet: a constructor that switches over Kind would need an arm per provider, and the arms for PubMed and Unpaywall would be errors saying "not built". One factory arrives when there is something to dispatch.
func (*ArXiv) Resolve ¶
Resolve looks a paper up by identifier.
arXiv resolves arXiv IDs, not arbitrary DOIs — it has no index of other publishers' identifiers, and pretending otherwise would return "not found" for papers that exist somewhere else. A 10.48550/arXiv.NNNN DOI is accepted because arXiv assigns those itself and the id is recoverable from it; anything else is refused as not this provider's to answer.
func (*ArXiv) Search ¶
Search returns papers matching a query.
Tier 0 of the escalation ladder, and the reason it is worth having: the abstract comes back IN THIS RESPONSE. A web lead pays a search plus a fetch plus extraction to reach less text than this, and the publication date here is exact rather than guessed from a page.
type Config ¶
type Config struct {
Kind Kind
// ContactEmail identifies the caller. NOT optional — see Validate.
ContactEmail string
// BaseURL overrides the provider's endpoint, for tests.
BaseURL string
}
Config is what every provider needs.
func (Config) Validate ¶
Validate refuses a provider that cannot identify itself.
§10.3: "Contact email is a required config value before any academic provider is enabled. Ship it as a startup check, not a README line." Unpaywall requires an email parameter and NCBI requires tool and email; sending requests without them is against both providers' documented terms, and the consequence lands on the user's address rather than on mole.
Enforced at construction rather than at call time so a misconfigured install fails before it has made a single request.
type FullTextFormat ¶
type FullTextFormat string
FullTextFormat reports the best text mole can currently READ for a paper.
Deliberately about mole's capability rather than the paper's availability, because the two are different questions and conflating them is how a PDF library gets built for a corpus that turns out to be mostly closed access.
const ( // FormatHTML is full text mole can read today, through the existing fetcher // and extractor. FormatHTML FullTextFormat = "html" // FormatPDFOnly is open access mole cannot read yet. This is the only // bucket a PDF extractor would serve. FormatPDFOnly FullTextFormat = "pdf_only" // FormatClosed has no open-access copy at all. No amount of parsing helps; // the abstract is the whole of what can be used. FormatClosed FullTextFormat = "closed" )
type Options ¶
type Options struct {
// MaxResults caps papers returned. Zero takes the provider's default.
MaxResults int
}
Options bound one search.
type Paper ¶
type Paper struct {
Title string
Abstract string
// Identifiers. A paper often has several; whichever the provider knows is
// recorded, because Unpaywall keys on DOI and PMC keys on PMCID.
DOI string
ArXivID string
PMID string
PMCID string
// PublishedAt is the paper's date, which academic sources report exactly.
//
// Worth more here than anywhere else in mole: §11.2's staleness detection
// needs publication dates on both sides of a contradiction to tell "these
// disagree" from "this one is older", and web sources supply them rarely
// and unreliably. This is the first place they arrive as fact.
PublishedAt *time.Time
// Where the text can be read, discovered from metadata.
//
// HTMLURL is the escalation target: arXiv's LaTeXML HTML and PMC's XML are
// full text the existing extractor already handles. PDFURL is recorded but
// not yet readable — mole has no PDF extraction, so a paper with only a PDF
// is counted rather than parsed, which is what turns "do we need a PDF
// library?" into a number (§10.4).
HTMLURL string
PDFURL string
LandingURL string
Source Kind
}
Paper is one result, normalized across providers.
The full-text locations are recorded from METADATA, not by fetching. Every provider knows which formats exist for a paper without downloading any of them, and that is what makes the escalation ladder affordable: the decision to read more than the abstract is taken on free information.
func (Paper) FullTextFormat ¶
func (p Paper) FullTextFormat() FullTextFormat
type Provider ¶
type Provider interface {
Resolver
Search(ctx context.Context, query string, opts Options) (*Response, error)
}
Provider is a Resolver that can also search by topic.
type PubMed ¶
type PubMed struct {
// contains filtered or unexported fields
}
PubMed searches PubMed and links to PMC full text. Keyless (§10.2).
Two requests per search, not three. esummary carries clean identifiers and a normalized date, and elink would give the PMC link — but efetch has all of it alongside the abstracts, which the other two do not return at all. Going straight from esearch to efetch is a third fewer requests against a shared public service, which is the kind of arithmetic §10.3 is about.
type Resolver ¶
type Resolver interface {
// Resolve looks a paper up by identifier — a DOI for Unpaywall, an arXiv ID
// for arXiv, either for PubMed. Each answers for the identifiers it indexes
// and refuses the rest, rather than reporting "not found" for a paper that
// exists somewhere it cannot see.
Resolve(ctx context.Context, id string) (*Paper, error)
Kind() Kind
}
Resolver turns an identifier into a paper.
Split out from Provider because Unpaywall is one and is not the other. It has no topical search — it answers "where can I legally read this DOI" and nothing else — and giving it a Search method that always errored would be an interface lying about what its implementations do. §10's sketch shows a single AcademicProvider with both; this is the one place the implementation deliberately differs, because the split is what the real APIs are.
type Unpaywall ¶
type Unpaywall struct {
// contains filtered or unexported fields
}
Unpaywall turns a DOI into a legal open-access copy (§10.2).
A Resolver and not a Provider: it has no topical search, and it returns no abstract. What it knows is where a paper can lawfully be read, which is the one thing neither arXiv nor PubMed can answer for a publisher's article.
Source Files