newspaper

README

newspaper

A Go implementation of Python's newspaper3k: download a web page and extract the article — its title, authors, publish date, full text, lead image, keywords, and summary — while throwing away the navigation, ads, and boilerplate around it.

Both a library and a CLI.

a := newspaper.NewArticle("https://example.com/some-story")
if err := a.Build(); err != nil { // Build = Download + Parse + NLP
    log.Fatal(err)
}
fmt.Println(a.Title)     // "City Council Approves New Transit Plan"
fmt.Println(a.Authors)   // ["Jane Reporter"]
fmt.Println(a.Text)      // the clean article body
fmt.Println(a.Keywords)  // ["transit", "council", "city", ...]
fmt.Println(a.Summary)   // an extractive summary
fmt.Println(a.Entities)  // [{Patricia Gonzalez PERSON} {Riverside Coalition ORG} ...]

What's included

• Article extraction — title, authors, publish date, body text, top image, description, canonical URL, favicon, tags, language.
• Two body extractors — the newspaper3k-faithful stopword-density heuristic and a train-free HMM (Viterbi) that adds a contiguity prior. Swap with cfg.Extractor = newspaper.ExtractorHMM.
• NLP — frequency keywords, extractive summary, and heuristic named-entity recognition (PERSON / ORG / LOCATION).
• Topic modeling — unsupervised LDA (collapsed Gibbs) over a corpus.
• Source crawling — discover a site's sections, RSS/Atom feeds, and article URLs (built on Colly).
• Multi-language — bundled stopword sets drive extraction and NLP per locale.
• Fully swappable — every stage is an interface on Config; override any one (downloader, extractor, keyworder, NER, …). See docs/extending.md.

How it works (the 30-second tour)

A news page is ~95% template and ~5% story. Extraction is the craft of keeping the 5%. The pipeline has three verbs, inherited from newspaper3k:

1. Download — fetch raw HTML over HTTP.
2. Parse — load the HTML into a DOM and run focused extractors: metadata (title/author/date/meta), the main body text (via a text-density heuristic), and media (top image).
3. NLP — derive keywords and an extractive summary from the clean text.

Everything is driven by stopwords (the ultra-common words of a language). Real prose is dense with them; menus and link-lists are not — so counting stopwords per DOM node is a cheap, language-agnostic signal for "this is an article paragraph." The same lists let NLP filter noise out of keywords.

See doc.go for the full annotated mental model.

Install

go install github.com/rangertaha/newspaper/cmd/newspaper@latest

Or build from source:

make build   # → bin/newspaper

CLI

# Extract one article (pretty output)
newspaper article https://example.com/story

# JSON for scripting; choose the HMM body extractor; force a language
newspaper article --json --extractor hmm -l en https://example.com/story

# Discover sections, feeds, and article links on a whole site
newspaper build https://example.com

# Model latent topics across several articles (a corpus method)
newspaper topics https://example.com/a https://example.com/b https://example.com/c -k 5

# List the bundled stopword languages
newspaper languages

Library layout

Path	Responsibility
config.go	Config: timeouts, user-agent, language, limits
network.go	HTTP fetching
article.go	Article: Download / Parse / NLP orchestration
source.go	Source: site crawling & feed discovery
internal/extractors	body (density + HMM), metadata, media extraction
internal/stopwords	embedded per-language stopword sets + tokenizer
internal/nlp	keywords, summary, named-entity recognition
internal/topics	LDA topic modeling (collapsed Gibbs sampling)
internal/urls	URL classification (is this link an article?)

License

MIT — see LICENSE.

Documentation

Overview

Package newspaper is a Go implementation of Python's newspaper3k library: a toolkit for downloading web pages and extracting the *article* from the surrounding clutter (navigation, ads, comments, sidebars, footers).

What problem does this solve?

A news web page is mostly NOT the article. A typical page is 95% template (menus, related-links, newsletter prompts, social buttons, legal boilerplate) and 5% the story you actually want. "Article extraction" is the art of throwing away the 95% and keeping the 5% — plus the structured metadata that describes it (title, author, publish date, lead image, and so on).

newspaper3k popularised a pragmatic pipeline for doing this. This package reproduces that pipeline in idiomatic Go. If you have never seen newspaper3k, here is the whole mental model in four steps:

a := newspaper.NewArticle("https://example.com/story")
a.Download()  // 1. fetch the raw HTML over HTTP
a.Parse()     // 2. turn HTML into structured fields (title, text, author…)
a.NLP()       // 3. (optional) derive keywords + a summary from the text
fmt.Println(a.Text)

The same three verbs — Download, Parse, NLP — are the spine of the original library, and we keep them so prior knowledge transfers directly.

The pipeline, stage by stage

1. DOWNLOAD (network.go): an HTTP GET with a browser-like User-Agent and a timeout. The only output is the raw HTML string. Kept deliberately separate from parsing so you can also feed in HTML you already have (see Article.SetHTML) without touching the network.

2. PARSE (article.go + internal/extractors): the raw HTML is loaded into a DOM (we use goquery, Go's jQuery-like wrapper over golang.org/x/net/html). A series of focused "extractors" then each pull one kind of information out of that DOM:

   - metadata: title, authors, publish date, meta description/keywords, canonical URL, language, favicon, OpenGraph tags. - content: the main article body. This is the hard part and uses a text-density heuristic (internal/extractors/density.go) or an optional train-free HMM (internal/extractors/hmm.go). - media: the lead ("top") image and embedded videos.

3. NLP (internal/nlp): with clean article text in hand, we compute keywords (most significant non-stopword terms) and an extractive summary (the few sentences that best represent the whole). This stage is intentionally optional and dependency-light — it is classic frequency analysis, not a neural model.

Why is everything driven by "stopwords"?

Stopwords are the ultra-common words of a language ("the", "is", "and"). newspaper3k uses them as a cheap, language-agnostic signal in two places:

• Extraction: real prose is dense with stopwords; navigation menus and link lists are not. Counting stopwords per DOM node is therefore a surprisingly good proxy for "this node contains a paragraph of an article."
• NLP: keywords are interesting precisely because they are NOT stopwords, so we filter stopwords out before ranking term frequency.

Because the stopword list is per-language, swapping the language swaps the whole behaviour — which is how one extractor handles dozens of languages. See internal/stopwords.

Sources: crawling a whole site, not just one article

Beyond single articles, newspaper3k can point at a news *site* and discover its sections and article links. That lives in source.go (the Source type), built on category detection and RSS/Atom feed parsing.

Package layout (a map for the reader)

newspaper/                 (this package — the public API)
  doc.go                   — you are here
  config.go                — Config: timeouts, user-agent, language, limits
  network.go               — HTTP fetching
  article.go               — Article: the Download/Parse/NLP orchestration
  source.go                — Source: site-level crawling & feed discovery
internal/extractors/       — pure DOM-in, data-out extraction functions
internal/stopwords/        — embedded per-language stopword sets + tokenizer
internal/nlp/              — keyword & summary algorithms
internal/urls/             — URL classification helpers (is this an article?)
cmd/newspaper/             — the CLI (urfave/cli subcommands)

The internal/ packages are pure and side-effect free where possible: they take a DOM or a string and return data. All the stateful orchestration lives in the root package. That separation is what makes the library easy to test (feed fixtures, no network) and easy to extend (add an extractor without touching the pipeline).

Index

• func Languages() []string
• func Version() string
• type Article
  • func NewArticle(url string) *Article
  • func NewArticleWithConfig(url string, cfg *Config) *Article
  • func (a *Article) Build() error
  • func (a *Article) Config() *Config
  • func (a *Article) Download() error
  • func (a *Article) NLP() error
  • func (a *Article) Parse() error
  • func (a *Article) SetHTML(html string)
• type Config
  • func DefaultConfig() *Config
  • func (c *Config) LanguageOrDefault() string
  • func (c *Config) Topics(docs []string, lang string, k int) []Topic
• type ContentExtractor
• type ContentResult
• type Downloader
• type Entity
• type EntityRecognizer
• type ExtractorKind
• type Keyworder
• type MediaExtractor
• type MediaResult
• type Metadata
• type MetadataExtractor
• type Source
  • func NewSource(url string) *Source
  • func NewSourceWithConfig(url string, cfg *Config) *Source
  • func (s *Source) ArticleList(limit int) []string
  • func (s *Source) Build() error
• type Summarizer
• type Topic
• type TopicModeler
• type URLClassifier

Functions

func Languages

func Languages() []string

Languages returns the ISO-639-1 codes of the stopword sets bundled in the binary. These are the languages extraction and NLP fully support.

func Version

func Version() string

Version returns the library/CLI version, resolved in order of precedence:

1. the value injected at build time with -ldflags (release builds);
2. the module version from the build info, e.g. when installed with `go install github.com/rangertaha/newspaper/cmd/newspaper@v1.2.3`;
3. a "dev" value annotated with the VCS revision when building from source.

Types

type Article

type Article struct {
	// URL is the address requested.
	URL string `json:"url"`

	// Metadata (filled by Parse).
	Title        string    `json:"title"`
	Authors      []string  `json:"authors"`
	PublishDate  time.Time `json:"publish_date"`
	Description  string    `json:"description"`
	MetaKeywords []string  `json:"meta_keywords"`
	Language     string    `json:"language"`
	CanonicalURL string    `json:"canonical_url"`
	Favicon      string    `json:"favicon"`
	SiteName     string    `json:"site_name"`
	Tags         []string  `json:"tags"`

	// Body and media (filled by Parse).
	Text     string   `json:"text"`
	TopImage string   `json:"top_image"`
	Images   []string `json:"images"`
	Movies   []string `json:"movies"`

	// NLP results (filled by NLP).
	Keywords []string `json:"keywords"`
	Summary  string   `json:"summary"`
	Entities []Entity `json:"entities"`

	// HTML is the downloaded markup, kept for re-parsing but excluded from JSON.
	HTML string `json:"-"`
	// contains filtered or unexported fields
}

Article is the central type: it represents one web article and walks it through the pipeline. The three stages mirror newspaper3k and are meant to be called in order:

a := newspaper.NewArticle(url)
a.Download()  // network: URL → raw HTML
a.Parse()     // HTML → Title, Authors, Text, TopImage, …
a.NLP()       // Text → Keywords, Summary, Entities

Build() runs all three. For offline use (tests, cached HTML), skip Download and call SetHTML before Parse.

The struct's exported fields are the result of parsing; they are populated as each stage runs. Fields are JSON-tagged so the whole Article serialises cleanly (the raw HTML and internal DOM are excluded).

func NewArticle

func NewArticle(url string) *Article

NewArticle creates an Article for url using the default configuration.

func NewArticleWithConfig

func NewArticleWithConfig(url string, cfg *Config) *Article

NewArticleWithConfig creates an Article with a custom configuration. A nil cfg uses the defaults.

func (*Article) Build

func (a *Article) Build() error

Build runs the full pipeline: Download, Parse, then NLP.

func (*Article) Config

func (a *Article) Config() *Config

Config returns the article's configuration, allowing callers to inspect or adjust it before running stages.

func (*Article) Download

func (a *Article) Download() error

Download fetches the page via the configured Downloader and stores its HTML.

func (*Article) NLP

func (a *Article) NLP() error

NLP derives keywords, an extractive summary, and named entities from the parsed text. Call Parse first.

func (*Article) Parse

func (a *Article) Parse() error

Parse turns the downloaded HTML into structured fields: metadata, body text, and media. Call Download or SetHTML first.

func (*Article) SetHTML

func (a *Article) SetHTML(html string)

SetHTML supplies HTML directly, bypassing the network. Use it to parse cached or fixture markup. The article's URL (if set) is used to resolve relative links.

type Config

type Config struct {

	// UserAgent is sent on every HTTP request. Many news sites serve different
	// (or no) markup to clients that look like bots, so we default to a
	// realistic desktop browser string.
	UserAgent string

	// RequestTimeout bounds a single HTTP fetch end-to-end.
	RequestTimeout time.Duration

	// MaxBytes caps how much of a response body we read. Article pages are
	// small; this protects against accidentally downloading a huge file.
	MaxBytes int64

	// Headers are extra request headers merged into every fetch.
	Headers map[string]string

	// IgnoreRobotsTxt disables robots.txt compliance during crawling. It
	// defaults to false (we respect robots.txt). Set true only for sites you
	// own or are authorised to crawl.
	IgnoreRobotsTxt bool

	// Language is an ISO-639-1 code ("en", "es", "de", …). It selects the
	// stopword set, which in turn drives BOTH content extraction (text-density
	// scoring) and NLP (keyword filtering). Empty means "auto-detect from the
	// page's lang attribute / meta tags, falling back to English."
	Language string

	// MinWordCount is the minimum number of words a block of text must have to
	// be considered part of the article body. Filters out one-line cruft.
	MinWordCount int

	// MinSentenceLength (in words) below which a "sentence" is treated as a
	// caption/label rather than prose during cleaning.
	MinSentenceLength int

	// MaxKeywords is how many keywords NLP() keeps.
	MaxKeywords int

	// SummarySentences is how many sentences the extractive summary keeps.
	SummarySentences int

	// MaxConcurrency caps parallel downloads when crawling a Source.
	MaxConcurrency int

	// FetchImages toggles whether Parse() bothers locating the top image.
	FetchImages bool

	// Extractor selects which built-in body extractor to use when
	// ContentExtractor is nil. Defaults to ExtractorDensity.
	Extractor ExtractorKind

	Downloader        Downloader        // default: Colly-backed fetcher
	ContentExtractor  ContentExtractor  // default: per Extractor (density/hmm)
	MetadataExtractor MetadataExtractor // default: priority-ladder extractor
	MediaExtractor    MediaExtractor    // default: og:image + largest-in-body
	Keyworder         Keyworder         // default: frequency + title bonus
	Summarizer        Summarizer        // default: Luhn-style sentence scoring
	EntityRecognizer  EntityRecognizer  // default: heuristic proper-noun NER
	TopicModeler      TopicModeler      // default: collapsed-Gibbs LDA
	URLClassifier     URLClassifier     // default: URL-shape heuristic
}

Config holds every tunable knob for the download → parse → NLP pipeline.

In newspaper3k this is the `Configuration` object. We bundle the settings in one struct (instead of scattering function arguments) so that an Article and a Source can share identical behaviour, and so callers can tweak one field without re-specifying the rest. Use DefaultConfig() and then override.

func DefaultConfig

func DefaultConfig() *Config

DefaultConfig returns sensible defaults. Copy and tweak:

cfg := newspaper.DefaultConfig()
cfg.Language = "es"
a := newspaper.NewArticleWithConfig(url, cfg)

func (*Config) LanguageOrDefault

func (c *Config) LanguageOrDefault() string

LanguageOrDefault returns the configured language, or "en" when unset. Extraction always needs *some* stopword set to score against.

func (*Config) Topics

func (c *Config) Topics(docs []string, lang string, k int) []Topic

Topics runs the configured TopicModeler (LDA by default) over a corpus of document texts and returns k topics. Topic modeling is a corpus-level method: pass several articles, not one.

type ContentExtractor

type ContentExtractor interface {
	// Name identifies the strategy (e.g. "density", "hmm").
	Name() string
	// Extract returns the article body for doc.
	Extract(doc *goquery.Document, lang string, cfg *Config) (ContentResult, error)
}

ContentExtractor finds the article body within a parsed document. Two built-ins ship — the newspaper3k-faithful density heuristic and a train-free HMM (see Config.Extractor). Assign Config.ContentExtractor to supply a custom one, e.g. a trained CRF. lang is the resolved ISO-639-1 language code.

type ContentResult

type ContentResult struct {
	Text    string
	TopNode *goquery.Selection
}

ContentResult is the output of a ContentExtractor: the article body text and, when available, the DOM subtree it was taken from (used to locate the top image within the body).

type Downloader

type Downloader interface {
	// Download returns the page HTML and the final URL after redirects.
	Download(rawURL string) (html, finalURL string, err error)
}

Downloader fetches raw HTML for a URL. The default is a Colly-backed client (network.go). Override Config.Downloader to read from a cache, local fixtures, or a headless browser.

type Entity

type Entity struct {
	Text  string
	Type  string // PERSON, ORG, LOCATION, or MISC
	Count int
}

Entity is a named entity with a coarse type and how often it occurred.

type EntityRecognizer

type EntityRecognizer interface {
	Entities(text, lang string) []Entity
}

EntityRecognizer extracts named entities (people, organizations, places) from the text. The default is a heuristic, train-free recognizer.

type ExtractorKind

type ExtractorKind string

ExtractorKind names a built-in content-extraction strategy.

const (
	// ExtractorDensity is the newspaper3k-faithful stopword-density heuristic.
	ExtractorDensity ExtractorKind = "density"
	// ExtractorHMM is the train-free HMM extractor (Viterbi over the block
	// sequence) that adds a contiguity prior the density heuristic lacks.
	ExtractorHMM ExtractorKind = "hmm"
)

type Keyworder

type Keyworder interface {
	Keywords(title, text, lang string, max int) []string
}

Keyworder ranks the most significant (non-stopword) terms in the text.

type MediaExtractor

type MediaExtractor interface {
	Extract(doc *goquery.Document, body *goquery.Selection, pageURL string) MediaResult
}

MediaExtractor finds the lead image and other media. body is the extracted article subtree (may be nil) and lets the extractor prefer in-article images.

type MediaResult

type MediaResult struct {
	TopImage string
	Images   []string
	Movies   []string
}

MediaResult holds discovered media URLs (already resolved to absolute form).

type Metadata

type Metadata struct {
	Title        string
	Authors      []string
	PublishDate  time.Time
	Description  string
	MetaKeywords []string // publisher-declared <meta name="keywords">
	Language     string
	Canonical    string
	Favicon      string
	SiteName     string
	Tags         []string
}

Metadata is the structured, non-body information about an article.

type MetadataExtractor

type MetadataExtractor interface {
	Extract(doc *goquery.Document, pageURL string) Metadata
}

MetadataExtractor recovers structured fields from the document. pageURL is the final (post-redirect) URL, used to resolve relative links to absolute ones.

type Source

type Source struct {
	// URL is the site's base/home URL.
	URL string `json:"url"`

	// Discovered during Build (sorted, de-duplicated).
	Categories []string `json:"categories"`
	Feeds      []string `json:"feeds"`
	Articles   []string `json:"articles"`
	// contains filtered or unexported fields
}

Source represents a news *site* (as opposed to a single Article) and discovers what to read on it: its sections, its RSS/Atom feeds, and candidate article URLs. This mirrors newspaper3k's Source / newspaper.build. See docs/sources.md.

Build crawls one level into the site using Colly (which gives concurrency, same-domain scoping, and optional robots.txt compliance), then enriches the article list from any feeds it finds.

func NewSource

func NewSource(url string) *Source

NewSource creates a Source with the default configuration.

func NewSourceWithConfig

func NewSourceWithConfig(url string, cfg *Config) *Source

NewSourceWithConfig creates a Source with a custom configuration. A nil cfg uses the defaults.

func (*Source) ArticleList

func (s *Source) ArticleList(limit int) []string

ArticleList returns up to limit discovered article URLs (all of them when limit <= 0).

func (*Source) Build

func (s *Source) Build() error

Build discovers categories, feeds, and article URLs for the site. It crawls the homepage and one level into each discovered section, classifying links with the configured URLClassifier, then parses any feeds for more articles.

type Summarizer

type Summarizer interface {
	Summarize(title, text, lang string, sentences int) string
}

Summarizer produces an extractive summary (a selection of the text's own sentences) of at most the requested number of sentences.

type Topic

type Topic struct {
	Words   []string
	Weights []float64
}

Topic is one latent topic: its top terms, highest-weight first, with weights.

type TopicModeler

type TopicModeler interface {
	Topics(docs []string, lang string, k int) []Topic
}

TopicModeler discovers latent topics across a corpus of documents (LDA). It is a corpus-level operation: a single article carries no topic structure, so this runs over a collection (e.g. a Source crawl).

type URLClassifier

type URLClassifier interface {
	IsArticle(rawURL string) bool
}

URLClassifier decides whether a discovered link points at an article (vs. a section, tag, or pagination page). Used while crawling a Source.