Documentation

Overview

    Package downloader is the main module of GO_SPIDER for download page.

    Index

    Constants

    This section is empty.

    Variables

    This section is empty.

    Functions

    This section is empty.

    Types

    type Downloader

    type Downloader interface {
    	Download(req *request.Request) *page.Page
    }

      The Downloader interface. You can implement the interface by implement function Download. Function Download need to return Page instance pointer that has request result downloaded from Request.

      type HttpDownloader

      type HttpDownloader struct {
      }

        The HttpDownloader download page by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is saved. The "jsonp" content is modified to json. The "text" content will save body plain text only. The page result is saved in Page.

        func NewHttpDownloader

        func NewHttpDownloader() *HttpDownloader

        func (*HttpDownloader) Download

        func (this *HttpDownloader) Download(req *request.Request) *page.Page