invertedindex

Documentation

Index

• func Find(index InvertedIndex, searchTerm string)
• func GenerateDocMap(token []string) map[string]bool
• func Preprocessing(wordList []string) []string
• func RemoveDuplicateListings(ids []int) []int
• func RemoveDuplicates(wordList []string) []string
• func Tokenize(Doc string) []string
• type DocMapWithId
• type GlobalDocMapWithId
• type InvertedIndex
  • func CreateInvertedIndex() *InvertedIndex
  • func GenerateInvertedIndex(DocList []string) InvertedIndex
  • func GenerateInvertedIndexWithPreExistingIds(Docs map[int][]string) InvertedIndex
  • func (invertedIndex *InvertedIndex) AddItem(Term string, Document int)
  • func (invertedIndex *InvertedIndex) FindItem(Term string) int
• type InvertedIndexEntry

Functions

func Find

func Find(index InvertedIndex, searchTerm string)

Find for a given inverted index and search term checks if the term exists and then outputs the documents the term is in

func GenerateDocMap

func GenerateDocMap(token []string) map[string]bool

GenerateDocMap creates a hash map of each word in the document

func Preprocessing

func Preprocessing(wordList []string) []string

Preprocessing converts each word to lowercase TODO: Clean up each word for symbols

func RemoveDuplicateListings

func RemoveDuplicateListings(ids []int) []int

func RemoveDuplicates

func RemoveDuplicates(wordList []string) []string

RemoveDuplicates filters out all duplicate words from each document

func Tokenize

func Tokenize(Doc string) []string

Tokenize gets the individual words from each document and generates a wordlist

Types

type DocMapWithId

type DocMapWithId struct {
	Term  string
	DocId int
}

type GlobalDocMapWithId

type GlobalDocMapWithId struct {
	Docs []DocMapWithId
}

type InvertedIndex

type InvertedIndex struct {
	HashMap map[string]*InvertedIndexEntry
	Items   []*InvertedIndexEntry
}

InvertedIndex contains a hash map to easily check if the term is present and an array of InvertedIndexEntry

func CreateInvertedIndex

func CreateInvertedIndex() *InvertedIndex

CreateInvertedIndex initializes an empty Inverted Index

func GenerateInvertedIndex

func GenerateInvertedIndex(DocList []string) InvertedIndex

GenerateInvertedIndex for each document list gets each word as a token, processes it and generates a hash map for each document using them it then generates the inverted index of all words

func GenerateInvertedIndexWithPreExistingIds

func GenerateInvertedIndexWithPreExistingIds(Docs map[int][]string) InvertedIndex

func (*InvertedIndex) AddItem

func (invertedIndex *InvertedIndex) AddItem(Term string, Document int)

AddItem works by first checking if a given term is already present in the inverse index or not by checking the hashmap. If it is present it updates the Items by increasing the frequency and adding the document it is found in. If it is not present it adds it to the hash map and adds it to the items list

func (*InvertedIndex) FindItem

func (invertedIndex *InvertedIndex) FindItem(Term string) int

FindItem returns the position of a given Item in an Inverted Index

type InvertedIndexEntry

type InvertedIndexEntry struct {
	Term            string
	Frequency       int
	DocumentListing []int
}

InvertedIndexEntry contains the term followed by the number of times it has appeared across all documents and an array of documents it is persent in