Documentation
¶
Overview ¶
Package scraper fetches air-quality readings from supported regional data sources and returns them as a normalized list of City records.
Index ¶
- func SourceNames() []string
- type City
- type Configuration
- type GeocodeCache
- type Geocoder
- type GoogleGeocoder
- type JSONFileGeocodeCache
- func (c *JSONFileGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
- func (c *JSONFileGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)
- func (c *JSONFileGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error
- func (c *JSONFileGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error
- type Location
- type MemoryGeocodeCache
- func (c *MemoryGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
- func (c *MemoryGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)
- func (c *MemoryGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error
- func (c *MemoryGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error
- type Options
- type Runner
- type SourceResult
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func SourceNames ¶
func SourceNames() []string
SourceNames returns the stable source names accepted by Runner.ScrapeSource and Runner.ScrapeSources.
Types ¶
type Configuration ¶
type Configuration struct {
GoogleGeocodingAPIKey string `env:"GOOGLE_GEOCODING_API_KEY"`
AirNowAPIKey string `env:"AIRNOW_KEY"`
MOENVAPIKey string `env:"MOENV_API_KEY"`
}
Configuration contains environment-backed settings needed by the default scraper integrations.
func LoadConfiguration ¶
func LoadConfiguration(ctx context.Context) (Configuration, error)
LoadConfiguration loads Configuration from the process environment using github.com/sethvargo/go-envconfig.
type GeocodeCache ¶
type GeocodeCache interface {
// GetLocation returns a cached address-to-location result.
GetLocation(ctx context.Context, address string) (Location, bool, error)
// SetLocation stores an address-to-location result.
SetLocation(ctx context.Context, address string, location Location) error
// GetAddress returns a cached coordinate-to-address result.
GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
// SetAddress stores a coordinate-to-address result.
SetAddress(ctx context.Context, lat float64, lng float64, address string) error
}
GeocodeCache stores Google geocoding results.
Consumers can implement this interface with Redis, SQLite, object storage, or any other backing store without adding those dependencies to this module.
type GoogleGeocoder ¶
type GoogleGeocoder struct {
// contains filtered or unexported fields
}
GoogleGeocoder implements Geocoder using the Google Geocoding API.
func NewGoogleGeocoder ¶
func NewGoogleGeocoder(apiKey string, client *http.Client) *GoogleGeocoder
NewGoogleGeocoder creates a GoogleGeocoder with an in-memory geocode cache.
func NewGoogleGeocoderWithCache ¶
func NewGoogleGeocoderWithCache(apiKey string, client *http.Client, cache GeocodeCache) *GoogleGeocoder
NewGoogleGeocoderWithCache creates a GoogleGeocoder with a caller-supplied cache. A nil cache falls back to an in-memory cache.
type JSONFileGeocodeCache ¶
type JSONFileGeocodeCache struct {
// contains filtered or unexported fields
}
JSONFileGeocodeCache persists geocoding results to a local JSON file.
func NewJSONFileGeocodeCache ¶
func NewJSONFileGeocodeCache(path string) (*JSONFileGeocodeCache, error)
NewJSONFileGeocodeCache opens or creates a JSON file-backed geocode cache.
func (*JSONFileGeocodeCache) GetAddress ¶
func (c *JSONFileGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
GetAddress returns a cached coordinate-to-address result.
func (*JSONFileGeocodeCache) GetLocation ¶
func (c *JSONFileGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)
GetLocation returns a cached address-to-location result.
func (*JSONFileGeocodeCache) SetAddress ¶
func (c *JSONFileGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error
SetAddress stores a coordinate-to-address result and persists the file.
func (*JSONFileGeocodeCache) SetLocation ¶
func (c *JSONFileGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error
SetLocation stores an address-to-location result and persists the file.
type MemoryGeocodeCache ¶
type MemoryGeocodeCache struct {
// contains filtered or unexported fields
}
MemoryGeocodeCache is a process-local GeocodeCache implementation.
func NewMemoryGeocodeCache ¶
func NewMemoryGeocodeCache() *MemoryGeocodeCache
NewMemoryGeocodeCache returns an empty in-memory geocode cache.
func (*MemoryGeocodeCache) GetAddress ¶
func (c *MemoryGeocodeCache) GetAddress(ctx context.Context, lat float64, lng float64) (string, bool, error)
GetAddress returns a cached coordinate-to-address result.
func (*MemoryGeocodeCache) GetLocation ¶
func (c *MemoryGeocodeCache) GetLocation(ctx context.Context, address string) (Location, bool, error)
GetLocation returns a cached address-to-location result.
func (*MemoryGeocodeCache) SetAddress ¶
func (c *MemoryGeocodeCache) SetAddress(ctx context.Context, lat float64, lng float64, address string) error
SetAddress stores a coordinate-to-address result.
func (*MemoryGeocodeCache) SetLocation ¶
func (c *MemoryGeocodeCache) SetLocation(ctx context.Context, address string, location Location) error
SetLocation stores an address-to-location result.
type Options ¶
type Options struct {
Configuration Configuration
HTTPClient *http.Client
Geocoder Geocoder
GeocodeCache GeocodeCache
Logger *log.Logger
}
Options configures a Runner.
Configuration provides API keys and other environment-backed settings. HTTPClient, Geocoder, GeocodeCache, and Logger are optional dependency injection points for programmatic consumers and tests.
type Runner ¶
type Runner struct {
// contains filtered or unexported fields
}
Runner coordinates the registered internal scraper sources.
func New ¶
New builds a Runner from Options.
If Options.Geocoder is nil and Configuration.GoogleGeocodingAPIKey is set, New creates a GoogleGeocoder. If Options.GeocodeCache is nil, that geocoder uses an in-memory cache.
func (*Runner) Scrape ¶
Scrape runs all registered sources and returns their combined city records.
Source-level failures are joined and returned with any successful records.
func (*Runner) ScrapeSource ¶
ScrapeSource runs one named source.
func (*Runner) ScrapeSources ¶
func (r *Runner) ScrapeSources(ctx context.Context, names []string) []SourceResult
ScrapeSources runs selected sources in registry order.
Passing nil or an empty slice runs every source. Unknown names are returned as SourceResult errors so callers can handle partial success consistently.
type SourceResult ¶
SourceResult contains the outcome of scraping one source.