collyredis

README

Colly Redis Store

A redis store for colly use latest go-redis package.

Because the official one seems no longer being maintained. It is hard to make a pull request for it, since the new go-redis client will break the old api.

The logic code of this package is not change, thanks for gocolly/redisstorage.

Install

go get -u github.com/hack-fan/colly-redis

Usage

import (
	"github.com/gocolly/colly"
	"github.com/hack-fan/collyredis"
	"github.com/go-redis/redis/v8"
)

// Make your any kind of go-redis client
rdb := redis.NewClient(&redis.Options{
    Addr:     "localhost:6379",
    Password: "", // no password set
    DB:       0,  // use default DB
})

storage := collyredis.NewStorage(rdb)

c := colly.NewCollector()
err := c.SetStorage(storage)
if err != nil {
    panic(err)
}

Documentation

Index

• type RedisClient
• type Storage
  • func (s *Storage) AddRequest(r []byte) error
  • func (s *Storage) Clear() error
  • func (s *Storage) Cookies(u *url.URL) string
  • func (s *Storage) GetRequest() ([]byte, error)
  • func (s *Storage) Init() error
  • func (s *Storage) IsVisited(requestID uint64) (bool, error)
  • func (s *Storage) QueueSize() (int, error)
  • func (s *Storage) SetCookies(u *url.URL, cookies string)
  • func (s *Storage) Visited(requestID uint64) error

Types

type RedisClient

type RedisClient interface {
	Ping(ctx context.Context) *redis.StatusCmd
	Keys(ctx context.Context, pattern string) *redis.StringSliceCmd
	Get(ctx context.Context, key string) *redis.StringCmd
	Set(ctx context.Context, key string, value interface{}, expiration time.Duration) *redis.StatusCmd
	Del(ctx context.Context, keys ...string) *redis.IntCmd
	Exists(ctx context.Context, keys ...string) *redis.IntCmd
	Expire(ctx context.Context, key string, expiration time.Duration) *redis.BoolCmd
	LPush(ctx context.Context, key string, values ...interface{}) *redis.IntCmd
	RPop(ctx context.Context, key string) *redis.StringCmd
	LLen(ctx context.Context, key string) *redis.IntCmd
}

RedisClient is because go-redis has many kind of clients.

type Storage

type Storage struct {
	// Client any kind of [go-redis](https://github.com/go-redis/redis) client
	Client RedisClient

	// Prefix is an optional string in the keys. It can be used
	// to use one redis database for independent scraping tasks.
	Prefix string

	// Expiration time for Visited keys. After expiration pages
	// are to be visited again.
	Expires time.Duration

	// Context can be used for canceling all redis request, if you supply your own.
	Context context.Context
	// contains filtered or unexported fields
}

Storage implements the redis storage backend for Colly

func (*Storage) AddRequest

func (s *Storage) AddRequest(r []byte) error

AddRequest implements queue.Storage.AddRequest() function

func (*Storage) Clear

func (s *Storage) Clear() error

Clear removes all entries from the storage

func (*Storage) Cookies

func (s *Storage) Cookies(u *url.URL) string

Cookies implements colly/storage.Cookies()

func (*Storage) GetRequest

func (s *Storage) GetRequest() ([]byte, error)

GetRequest implements queue.Storage.GetRequest() function

func (*Storage) Init

func (s *Storage) Init() error

Init initializes the redis storage

func (*Storage) IsVisited

func (s *Storage) IsVisited(requestID uint64) (bool, error)

IsVisited implements colly/storage.IsVisited()

func (*Storage) QueueSize

func (s *Storage) QueueSize() (int, error)

QueueSize implements queue.Storage.QueueSize() function

func (*Storage) SetCookies

func (s *Storage) SetCookies(u *url.URL, cookies string)

SetCookies implements colly/storage..SetCookies()

func (*Storage) Visited

func (s *Storage) Visited(requestID uint64) error

Visited implements colly/storage.Visited()