downloader

Documentation

Overview

Package downloader is the main module of GO_SPIDER for download page.

Index

• type Downloader
• type HttpDownloader
  • func NewHttpDownloader() *HttpDownloader
  • func (this *HttpDownloader) Download(req *request.Request) *page.Page
  • func (this *HttpDownloader) DownloadCore(req *request.Request) *page.Page

Types

type Downloader

type Downloader interface {
	Download(req *request.Request) *page.Page
}

The Downloader interface. You can implement the interface by implement function Download. Function Download need to return Page instance pointer that has request result downloaded from Request.

type HttpDownloader

type HttpDownloader struct {
}

The HttpDownloader download page by package net/http. The "html" content is contained in dom parser of package goquery. The "json" content is saved. The "jsonp" content is modified to json. The "text" content will save body plain text only. The page result is saved in Page.

func NewHttpDownloader

func NewHttpDownloader() *HttpDownloader

func (*HttpDownloader) Download

func (this *HttpDownloader) Download(req *request.Request) *page.Page

func (*HttpDownloader) DownloadCore

func (this *HttpDownloader) DownloadCore(req *request.Request) *page.Page