README
¶
About
Creeper is a next-generation crawler which fetches web page by creeper script. As a cross-platform embedded crawler, you can use it for your news app, subscribe program, etc.
Warning: At present this project is still under early stage development, please do not use in the production environment.
Get Started
Installation
$ go get github.com/wspl/creeper
Hello World!
Create hacker_news.crs
page(@page=1) = "https://news.ycombinator.com/news?p={@page}"
news[]: page -> $("tr.athing")
title: $(".title a.storylink").text
site: $(".title span.sitestr").text
link: $(".title a.storylink").href
Then, create main.go
package main
import "github.com/wspl/creeper"
func main() {
c := creeper.Open("./hacker_news.crs")
c.Array("news").Each(func(c *creeper.Creeper) {
println("title: ", c.String("title"))
println("site: ", c.String("site"))
println("link: ", c.String("link"))
println("===")
})
}
Build and run. Console will print something like:
title: Samsung chief Lee arrested as S.Korean corruption probe deepens
site: reuters.com
link: http://www.reuters.com/article/us-southkorea-politics-samsung-group-idUSKBN15V2RD
===
title: ReactOS 0.4.4 Released
site: reactos.org
link: https://reactos.org/project-news/reactos-044-released
===
title: FeFETs: How this new memory stacks up against existing non-volatile memory
site: semiengineering.com
link: http://semiengineering.com/what-are-fefets/
Script Spec
Town
Town is a lambda like expression for saving (in)mutable string. Most of the time, we used it to store url.
page(@page=1, ext) = "https://news.ycombinator.com/news?p={@page}&ext={ext}"
When you need town, use it as if you were calling a function:
news[]: page(ext="Hello World!") -> $("tr.athing")
You might have noticed that the @page parameter is not used. Yeah, it is a special parameter.
Expression in town definition line like name="something", represents parameter name has a default value "something".
Incidentally, @page is a parameter that will automatically increasing when current page has no more content.
Node
Nodes are tree structure that represent the data structure you are going to crawl.
news[]: page -> $("tr.athing")
title: $(".title a.storylink").text
site: $(".title span.sitestr").text
link: $(".title a.storylink").href
Like yaml, nodes distinguishes the hierarchy by indentation.
Node Name
Node has name. title is a field name, represents a general string data. news[] is a array name, represents a parent structure with multiple sub-data.
Page
Page indicates where to fetching the field data. It can be a town expression or field reference.
Field reference is a advanced usage of Node, you can found the details in ./eh.crs.
If a node owned page and fun at the same time, page should on the left of ->, fun should on the right of ->. Which is page -> fun
Fun
Fun represents the data processing process.
There are all supported funs:
| Name | Parameters | Description |
|---|---|---|
| $ | (selector: string) | Relative CSS selector (select from parent node) |
| $root | (selector: string) | Absolute CSS selector (select from body) |
| html | inner HTML | |
| text | inner text | |
| outerHTML | outer HTML | |
| attr | (attr: string) | attribute value |
| style | style attribute value | |
| href | href attribute value | |
| src | src attribute value | |
| class | class attribute value | |
| id | id attribute value | |
| calc | (prec: int) | calculate arithmetic expression |
| match | (regexp: string) | match first sub-string via regular expression |
| expand | (regexp: string, target: string) | expand matched strings to target string |
Author
Plutonist
Documentation
¶
Index ¶
- func MD5(s string) string
- func PowerfulFind(s *goquery.Selection, q string) *goquery.Selection
- type Creeper
- type Formatted
- type Fun
- type MonoStack
- type Node
- func (n *Node) ChildFilter(cb func(*Node) bool) *Node
- func (n *Node) Filter(cb func(*Node) bool) []*Node
- func (n *Node) Next()
- func (n *Node) NextDirectorNode() *Node
- func (n *Node) PrimaryNode() *Node
- func (n *Node) Reset()
- func (n *Node) Search(name string) *Node
- func (n *Node) SearchFlatScope(name string) *Node
- func (n *Node) SearchRef(name string) *Node
- func (n *Node) Value() (string, error)
- type Page
- type Town
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
Types ¶
type Creeper ¶
type Creeper struct {
Nodes []*Node
Towns []*Town
CacheGet func(string) (string, bool)
CacheSet func(string, string)
Node *Node
}
func NewByFormatted ¶
type Formatted ¶
func Formatting ¶
type Fun ¶
type Fun struct {
Raw string
Node *Node
Name string
Params []string
Document *goquery.Document
Selection *goquery.Selection
Result string
TempSkip bool
BundleSize int
PrevFun *Fun
NextFun *Fun
}
func (*Fun) InitSelector ¶
type Node ¶
type Node struct {
Raw string
Creeper *Creeper
Name string
IsArray bool
IsPrimary bool
IndentLen int
Page *Page
Fun *Fun
Index int
Sn map[int]string
PrevNode *Node
NextNode *Node
ParentNode *Node
FirstChildNode *Node
LastChildNode *Node
}
Source Files
Directories