Version: v0.0.0-...-d33463d Latest Latest

This package is not in the latest version of its module.

Go to latest
Published: Jun 12, 2020 License: BSD-3-Clause Imports: 21 Imported by: 0



Package parse of the Dataflow kit is used by parse.d service which parses html content from web pages following the rules described in Payload JSON file.



This section is empty.


This section is empty.


func DecodeParseRequest

func DecodeParseRequest(ctx context.Context, r *http.Request) (interface{}, error)

DecodeParseRequest decodes request sent to Parser if error occures, server returns 400 Bad Request

func EncodeParseResponse

func EncodeParseResponse(_ context.Context, w http.ResponseWriter, response interface{}) error

EncodeParseResponse encodes response returned by Parser

func HealthCheckHandler

func HealthCheckHandler(w http.ResponseWriter, r *http.Request)

HealthCheckHandler is used to check if Parse service is alive

func MakeParseEndpoint

func MakeParseEndpoint(svc Service) endpoint.Endpoint

MakeParseEndpoint creates Parse Endpoint

func NewHttpHandler

func NewHttpHandler(ctx context.Context, endpoint Endpoints) http.Handler

NewHttpHandler mounts all of the service endpoints into an http.Handler.


type Config

type Config struct {
	Host    string
	Version string

Config provides basic configuration

type Endpoints

type Endpoints struct {
	ParseEndpoint endpoint.Endpoint

Endpoints wrapper

func (Endpoints) Parse

func (e Endpoints) Parse(p scrape.Payload) (io.ReadCloser, error)

Parse method is used for sending payload requests to parse service.

type HTMLServer

type HTMLServer struct {
	// contains filtered or unexported fields

HTMLServer represents the web service that serves up HTML

func Start

func Start(cfg Config) *HTMLServer

Start func launches Parsing service

func (*HTMLServer) Stop

func (htmlServer *HTMLServer) Stop() error

Stop turns off the HTML Server

type ParseService

type ParseService struct {

ParseService implements service with empty struct

func (ParseService) Parse

func (ps ParseService) Parse(p scrape.Payload) (io.ReadCloser, error)

Parse service processes fetched page following the rules from Payload.

type Service

type Service interface {
	Parse(scrape.Payload) (io.ReadCloser, error)

Service defines Parse service interface

func NewHTTPClient

func NewHTTPClient(instance string) (Service, error)

NewHTTPClient returns an Parse Service backed by an HTTP server living at the remote instance. We expect instance to come from a service discovery system, so likely of the form "host:port". We bake-in certain middlewares, implementing the client library pattern.

type ServiceMiddleware

type ServiceMiddleware func(Service) Service

ServiceMiddleware defines a middleware for a Parse service

func LoggingMiddleware

func LoggingMiddleware(logger *zap.Logger) ServiceMiddleware

LoggingMiddleware logs Parse Service endpoints

func Metrics

func Metrics(requestCount metrics.Counter,
	requestLatency metrics.Histogram) ServiceMiddleware

metrics function

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL