scraper

package module
v1.0.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 5, 2026 License: MIT Imports: 16 Imported by: 0

README

nebulo-scraper

Test Run

This is the web scraper our app Nebulo uses to scrape air quality data.

It is written in Go and can be used either as a CLI or as a package from another Go application.

CLI Usage

Populate the required environment variables:

cp .env.example .env

Run all scrapers:

go run ./cmd/nebulo-scraper

Run one scraper:

go run ./cmd/nebulo-scraper -source singapore

The CLI writes JSON files into output/ by default. Use -output <dir> to choose a different destination. Google geocoding results are cached in a temp JSON file by default; use -geocode-cache <path> to choose a different cache file, or -geocode-cache "" for an in-memory cache.

Package Usage

package main

import (
	"context"
	"log"

	scraper "github.com/undertideco/nebulo-scraper"
)

func main() {
	config, err := scraper.LoadConfiguration(context.Background())
	if err != nil {
		log.Fatal(err)
	}

cache := scraper.NewMemoryGeocodeCache()
runner := scraper.New(scraper.Options{
	Configuration: config,
	GeocodeCache:  cache,
})

	cities, err := runner.Scrape(context.Background())
	if err != nil {
		log.Fatal(err)
	}

log.Println(len(cities))
}

Consumers that want Redis, SQLite, or another cache can implement scraper.GeocodeCache and pass it through scraper.Options.GeocodeCache.

Output

Each scraper writes its results to output/<scraper>.json. All results are also combined into output/_all.json.

_all.json is a JSON array of city objects with the following shape:

[
  {
    "name": "string — city or station name",
    "region": "string — country or region identifier",
    "location": {
      "lat": "number — latitude",
      "lng": "number — longitude"
    },
    "data": "number — AQI (Air Quality Index) reading"
  }
]
Development
  1. Clone the repo
  2. Copy .env.example to .env and populate the values
  3. Run go test ./...
Releases

Releases are driven by Cocogitto and the Release GitHub Actions workflow.

For the initial stable release, run the workflow on master with bump set to major; with no previous tag, Cocogitto falls back to 0.0.0 and will create v1.0.0. A seed v0.0.0 tag is not required. For later releases, choose auto, major, minor, or patch.

Each release publishes Linux amd64 and arm64 CLI tarballs plus SHA-256 checksums.

Questions?

Feel free to create an issue.

License

MIT

Documentation

Overview

Package scraper fetches air-quality readings from supported regional data sources and returns them as a normalized list of City records.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func SourceNames

func SourceNames() []string

SourceNames returns the stable source names accepted by Runner.ScrapeSource and Runner.ScrapeSources.

Types

type City

type City = model.City

City is the normalized air-quality record returned by scrapers.

type Configuration

type Configuration struct {
	GoogleGeocodingAPIKey string `env:"GOOGLE_GEOCODING_API_KEY"`
	AirNowAPIKey          string `env:"AIRNOW_KEY"`
	MOENVAPIKey           string `env:"MOENV_API_KEY"`
}

Configuration contains environment-backed settings needed by the default scraper integrations.

func LoadConfiguration

func LoadConfiguration(ctx context.Context) (Configuration, error)

LoadConfiguration loads Configuration from the process environment using github.com/sethvargo/go-envconfig.

type GeocodeCache

type GeocodeCache interface {
	// GetLocation returns a cached address-to-location result.
	GetLocation(ctx context.Context, address string) (Location, bool, error)
	// SetLocation stores an address-to-location result.
	SetLocation(ctx context.Context, address string, location Location) error
	// GetAddress returns a cached coordinate-to-address result.
	GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
	// SetAddress stores a coordinate-to-address result.
	SetAddress(ctx context.Context, lat float64, lng float64, address string) error
}

GeocodeCache stores Google geocoding results.

Consumers can implement this interface with Redis, SQLite, object storage, or any other backing store without adding those dependencies to this module.

type Geocoder

type Geocoder = model.Geocoder

Geocoder resolves addresses to coordinates and coordinates back to names.

type GoogleGeocoder

type GoogleGeocoder struct {
	// contains filtered or unexported fields
}

GoogleGeocoder implements Geocoder using the Google Geocoding API.

func NewGoogleGeocoder

func NewGoogleGeocoder(apiKey string, client *http.Client) *GoogleGeocoder

NewGoogleGeocoder creates a GoogleGeocoder with an in-memory geocode cache.

func NewGoogleGeocoderWithCache

func NewGoogleGeocoderWithCache(apiKey string, client *http.Client, cache GeocodeCache) *GoogleGeocoder

NewGoogleGeocoderWithCache creates a GoogleGeocoder with a caller-supplied cache. A nil cache falls back to an in-memory cache.

func (*GoogleGeocoder) Address

func (g *GoogleGeocoder) Address(ctx context.Context, lat float64, lng float64) (string, error)

Address resolves coordinates to a compact place name.

func (*GoogleGeocoder) LatLng

func (g *GoogleGeocoder) LatLng(ctx context.Context, address string) (Location, error)

LatLng resolves an address to a Location.

type JSONFileGeocodeCache

type JSONFileGeocodeCache struct {
	// contains filtered or unexported fields
}

JSONFileGeocodeCache persists geocoding results to a local JSON file.

func NewJSONFileGeocodeCache

func NewJSONFileGeocodeCache(path string) (*JSONFileGeocodeCache, error)

NewJSONFileGeocodeCache opens or creates a JSON file-backed geocode cache.

func (*JSONFileGeocodeCache) GetAddress

func (c *JSONFileGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)

GetAddress returns a cached coordinate-to-address result.

func (*JSONFileGeocodeCache) GetLocation

func (c *JSONFileGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)

GetLocation returns a cached address-to-location result.

func (*JSONFileGeocodeCache) SetAddress

func (c *JSONFileGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error

SetAddress stores a coordinate-to-address result and persists the file.

func (*JSONFileGeocodeCache) SetLocation

func (c *JSONFileGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error

SetLocation stores an address-to-location result and persists the file.

type Location

type Location = model.Location

Location is a latitude/longitude pair.

type MemoryGeocodeCache

type MemoryGeocodeCache struct {
	// contains filtered or unexported fields
}

MemoryGeocodeCache is a process-local GeocodeCache implementation.

func NewMemoryGeocodeCache

func NewMemoryGeocodeCache() *MemoryGeocodeCache

NewMemoryGeocodeCache returns an empty in-memory geocode cache.

func (*MemoryGeocodeCache) GetAddress

func (c *MemoryGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)

GetAddress returns a cached coordinate-to-address result.

func (*MemoryGeocodeCache) GetLocation

func (c *MemoryGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)

GetLocation returns a cached address-to-location result.

func (*MemoryGeocodeCache) SetAddress

func (c *MemoryGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error

SetAddress stores a coordinate-to-address result.

func (*MemoryGeocodeCache) SetLocation

func (c *MemoryGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error

SetLocation stores an address-to-location result.

type Options

type Options struct {
	Configuration Configuration
	HTTPClient    *http.Client
	Geocoder      Geocoder
	GeocodeCache  GeocodeCache
	Logger        *log.Logger
}

Options configures a Runner.

Configuration provides API keys and other environment-backed settings. HTTPClient, Geocoder, GeocodeCache, and Logger are optional dependency injection points for programmatic consumers and tests.

type Runner

type Runner struct {
	// contains filtered or unexported fields
}

Runner coordinates the registered internal scraper sources.

func New

func New(opts Options) *Runner

New builds a Runner from Options.

If Options.Geocoder is nil and Configuration.GoogleGeocodingAPIKey is set, New creates a GoogleGeocoder. If Options.GeocodeCache is nil, that geocoder uses an in-memory cache.

func (*Runner) Scrape

func (r *Runner) Scrape(ctx context.Context) ([]City, error)

Scrape runs all registered sources and returns their combined city records.

Source-level failures are joined and returned with any successful records.

func (*Runner) ScrapeSource

func (r *Runner) ScrapeSource(ctx context.Context, name string) ([]City, error)

ScrapeSource runs one named source.

func (*Runner) ScrapeSources

func (r *Runner) ScrapeSources(ctx context.Context, names []string) []SourceResult

ScrapeSources runs selected sources in registry order.

Passing nil or an empty slice runs every source. Unknown names are returned as SourceResult errors so callers can handle partial success consistently.

type SourceResult

type SourceResult struct {
	Name   string
	Cities []City
	Err    error
}

SourceResult contains the outcome of scraping one source.

Directories

Path Synopsis
cmd
nebulo-scraper command
internal

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL