twitscrape

README

twitscrape

A scraper written in Go that fetches tweets from the public Twitter search. This does not require authentication.

Installation

Obrain the library using:

$ go get -u github.com/kompiuter/twitscrape

About

This library uses the publicly available Twitter Advanced Search to search for tweets, scrapes all tweets from the response and returns a slice of tweets.

Since this is a scraper, no authentication whatsoever is required. You can use it right out of the box!

Usage

Create a Scrape struct and use its Tweets method. You can specify an io.Writer when creating the Scrape struct to enable logging of info messages to that writer. Leaving Info empty means the scraper will not log any info messages.

scr := twitscrape.Scrape{Info: os.Stdout}
start, _ := time.Parse("01/02/2006", "11/10/2009")
until, _ := time.Parse("01/02/2006", "11/11/2009")
// fetch tweets between start and until dates, which contain hashtag #golang
tweets, err := scr.Tweets("#golang", start, until)
if err != nil {
  // Handle err
}
fmt.Print(tweets[0].Permalink) // https://www.twitter.com/repeatedly/status/5603770675

See a complete example which writes tweets to an output file here.

Query Operators

You may use any query operator as a search parameter to refine your results.

For example:

tweets, err := scr.Tweets("#golang from:davecheney", start, until)

Tests

There are tests available that can be run in the root directory using:

go test

They test that correct output is returned when querying old tweets, relying on the premise that Twitter will never ammend or delete them. If they do, ¯\(ツ)/¯

Roadmap

• Return tweets using a chan

Feedback and pull requests are most welcome!

Documentation

Overview

Package twitscrape is a library for scraping tweets from the twitter archive. The archive is publicly available and can be searched through at: https://twitter.com/search-advanced?lang=en

No authentication is required and the package can be run without any prior configurations.

You can start scraping by creating a new instance of the Scrape struct and calling its Tweets method, by providing a search term, a start date and an end date.

scr := twitscrape.Scrape{}
start, _ := time.Parse("01/02/2006", "11/10/2009")
until, _ := time.Parse("01/02/2006", "11/11/2009")
// fetch tweets between start and until dates, which contain hashtag #golang
tweets, err := scr.Tweets("#golang", start, until)

Tweets returns a slice of Tweet, which is a struct with the following fields:

type Tweet struct {
    // Link to tweet in the form https://www.twitter.com/user/status
    Permalink string
    // Screen name (twitter handle) of tweet author
    Name string
    // Timestamp of tweet in UTC
    Timestamp time.Time
    // Contents of tweet
    Contents string
    // Tweet ID
    ID string
}

By default, the Tweets function will not log anything (such as a missing attribute when scraping). To enable logging, pass a io.Writer into the Scrape struct initialization and logging will be written to that writer:

scr := ts.Scrape{Info: os.Stdout}

In order to better refine your search, you may use any Query Operator (as defined by Twitter) in your search term. The query operators can be found here: https://dev.twitter.com/rest/public/search#query-operators

// Tweets by Dave Cheney
tweets, err := scr.Tweets("#golang from:davecheney", startDate, untilDate)

Since a Twitter search is paginated by Twitter (to 20 Tweets), this library abuses the fact more tweets are loaded via AJAX. More information can be found in a great blog post by Tom Dickinson: http://tomkdickinson.co.uk/2015/01/scraping-tweets-directly-from-twitters-search-page-part-1/

MIT License Copyright (c) 2016 Kyriacos Kyriacou Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Index

• type Scrape
  • func (s Scrape) Tweets(search string, start, until time.Time) ([]Tweet, error)
• type Tweet

Types

type Scrape

type Scrape struct {
	Info io.Writer
}

Scrape is responsible for scraping tweets from Twitter. If Info is set to a writer then log messages will be written to that writer, otherwise no log messages will be written

func (Scrape) Tweets

func (s Scrape) Tweets(search string, start, until time.Time) ([]Tweet, error)

Tweets searches the Twitter archive and returns all tweets found between the start and until date. If you have provided a large period of time, this may take some time to complete. Info messages will be written to Scrape.Info

Any query operator may be used in the search string to refine your search, as defined by Twitter: https://dev.twitter.com/rest/public/search#query-operators

type Tweet

type Tweet struct {
	// Link to tweet in the form https://www.twitter.com/user/status
	Permalink string
	// Screen name (twitter handle) of tweet author
	Name string
	// Timestamp of tweet in UTC
	Timestamp time.Time
	// Contents of tweet
	Contents string
	// Tweet ID
	ID string
}

Tweet represents each individual tweet retrieved from the archive