assocentity

README

assocentity

Package assocentity is a social science tool to analyze the relative distance from tokens to entities. The motiviation is to make conclusions based on the distance from interesting tokens to a certain entity and its synonyms.

Features

• Provide your own tokenizer
• Provides a default NLP tokenizer (by Google)
• Define aliases for entities
• Provides a multi-OS, language-agnostic CLI version

Installation

$ go get github.com/ndabAP/assocentity/v13

Prerequisites

If you want to analyze human readable texts you can use the provided Natural Language tokenizer (powered by Google). To do so, sign-up for a Cloud Natural Language API service account key and download the generated JSON file. This equals the credentialsFile at the example below. You should never commit that file.

A possible offline tokenizer would be a white space tokenizer. You also might use a parser depending on your purposes.

Example

We would like to find out which adjectives are how close in average to a certain public person. Let's take George W. Bush and 1,000 NBC news articles as an example. "George Bush" is the entity and synonyms are "George Walker Bush" and "Bush" and so on. The text is each of the 1,000 NBC news articles.

Defining a text source and to set the entity would be first step. Next, we need to instantiate our tokenizer. In this case, we use the provided Google NLP tokenizer. Finally, we can calculate our mean distances. We can use assocentity.Distances, which accepts multiple texts. Notice how we pass tokenize.ADJ to only include adjectives as part of speech. Finally, we can take the mean by passing the result to assocentity.Mean.

// Define texts source and entity
texts := []string{
	"Former Presidents Barack Obama, Bill Clinton and ...", // Truncated
	"At the pentagon on the afternoon of 9/11, ...",
	"Tony Blair moved swiftly to place his relationship with ...",
}
entities := []string{
	"Goerge Walker Bush",
	"Goerge Bush",
	"Bush",
}
source := assocentity.NewSource(entities, texts)

// Instantiate the NLP tokenizer (powered by Google)
nlpTok := nlp.NewNLPTokenizer(credentialsFile, nlp.AutoLang)

// Get the distances to adjectives
ctx := context.TODO()
dists, err := assocentity.Distances(ctx, nlpTok, tokenize.ADJ, source)
if err != nil {
	// Handle error
}
// Get the mean from the distances
mean := assocentity.Mean(dists)

Tokenization

If you provide your own tokenizer you must implement the interface with the method Tokenize and the following signature:

type Tokenizer interface {
	Tokenize(ctx context.Context, text string) ([]Token, error)
}

Token is of type:

type Token struct {
	PoS  PoS    // Part of speech
	Text string // Text
}

// Part of speech
type PoS int

For example, given the text:

text := "Punchinello was burning to get me"

The result from Tokenize would be:

[]Token{
	{
		Text: "Punchinello",
		PoS:  tokenize.NOUN,
	},
	{
		Text: "was",
		PoS:  tokenize.VERB,
	},
	{
		Text: "burning",
		PoS:  tokenize.VERB,
	},
	{
		Text: "to",
		PoS:  tokenize.PRT,
	},
	{
		Text: "get",
		PoS:  tokenize.VERB,
	},
	{
		Text: "me",
		PoS:  tokenize.PRON,
	},
}

CLI

There is also a language-agnostic terminal version available for either Windows, Mac (Darwin) or Linux (only with 64-bit support) if you don't have Go available. The application expects the text from "stdin" and accepts the following flags:

Flag	Description	Type	Default
entities	Define entities to be searched within input, example: -entities="Max Payne,Payne"	string
gog-svc-loc	Google Clouds NLP JSON service account file, example: -gog-svc-loc="/home/max/gog-svc-loc.json"	string
op	Operation to excute: -op="mean"	string	mean
pos	Defines part of speeches to keep, example: -pos=noun,verb,pron	string	any

Example:

echo "Relax, Max. You're a nice guy." | ./bin/assocentity_linux_amd64_v13.0.0-0-g948274a-dirty -gog-svc-loc=/home/max/.config/assocentity/google-service.json -entities="Max Payne,Payne,Max"

The output is written to "stdout" in appropoiate formats.

Projects using assocentity

• entityscrape - Distance between word types (default: adjectives) in news articles and persons

Author

Julian Claus and contributors.

License

MIT

Documentation

Index

• func Distances(ctx context.Context, tokenizer tokenize.Tokenizer, poS tokenize.PoS, ...) (map[tokenize.Token][]float64, error)
• func Mean(dists map[tokenize.Token][]float64) map[tokenize.Token]float64
• func NewSource(entities, texts []string) source
• func Normalize(dists map[tokenize.Token][]float64, norm Normalizer)
• func Threshold(dists map[tokenize.Token][]float64, threshold float64)
• type Normalizer

Functions

func Distances

func Distances(
	ctx context.Context,
	tokenizer tokenize.Tokenizer,
	poS tokenize.PoS,
	source source,
) (map[tokenize.Token][]float64, error)

Distances returns the distances from entities to a list of texts

func Mean

func Mean(dists map[tokenize.Token][]float64) map[tokenize.Token]float64

Mean returns the mean of the provided distances

func NewSource

func NewSource(entities, texts []string) source

NewSource returns a new source consisting of entities and texts

func Normalize

func Normalize(dists map[tokenize.Token][]float64, norm Normalizer)

Normalize normalizes tokens with provided normalizer

func Threshold

func Threshold(dists map[tokenize.Token][]float64, threshold float64)

Threshold excludes results that are below the given threshold. The threshold is described through the amount of distances per token relative to the total amount of tokens

Types

type Normalizer

type Normalizer func(tokenize.Token) tokenize.Token

Normalizer normalizes tokens like lower casing them

var HumandReadableNormalizer Normalizer = func(tok tokenize.Token) tokenize.Token {
	t := tokenize.Token{
		PoS:  tok.PoS,
		Text: strings.ToLower(tok.Text),
	}

	switch tok.Text {
	case "&":
		t.Text = "and"
	}

	return t
}

HumandReadableNormalizer normalizes tokens through lower casing them and replacing them with their synonyms