epgo

README

epgo

epgo is a command line utility utilizing EPrints' REST API to produce alternative feeds and formats. Currently it supports generating a feed of repository items based on publication dates.

Overview

USAGE: epgo [OPTIONS] [EPRINT_URI|JAVASCRIPT_FILES]

epgo wraps the REST API for E-Prints 3.3 or better. It can return a list of uri, a JSON view of the XML presentation as well as generates feeds and web pages.

epgo can be configured with following environment variables

• EPGO_API_URL (required) the URL to your E-Prints installation
• EPGO_DBNAME (required) the BoltDB name for exporting, site building, and content retrieval
• EPGO_SITE_URL (optional) the website URL (might be the same as E-Prints)
• EPGO_HTDOCS (optional) the htdocs root for site building
• EPGO_TEMPLATE_PATH (optional) the template directory to use for site building

If EPRINT_URI is provided then an individual EPrint is return as a JSON structure (e.g. /rest/eprint/34.xml). Otherwise a list of EPrint paths are returned.

Options	Description
-api	display EPrint REST API response
-dbname	BoltDB name
-export int	export N EPrints to local database, if N is negative export all EPrints
-read-api	read the contents from the API without saving in the database
-feed-size int	the number of items included in generated feeds
-published-newest int	list the N newest published records
-published-oldest int	list the N oldest published records
-articles-newest int	list the N newest articles
-articles-oldest int	list the N oldest articles
-p	pretty print JSON output
-h	display help info
-l	show license information
-v	display version info

Documentation

Overview

Package epgo is a collection of structures and functions for working with the E-Prints REST API

@author R. S. Doiel, <rsdoiel@caltech.edu>

Copyright (c) 2016, Caltech All rights not granted herein are expressly reserved by Caltech.

Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.

2. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

3. Neither the name of the copyright holder nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

Index

• Constants
• Variables
• func Slugify(s string) string
• type Document
• type DocumentList
• type EPrintsAPI
  • func New(cfg *cli.Config) (*EPrintsAPI, error)
  • func (api *EPrintsAPI) BuildEPrintMirror() error
  • func (api *EPrintsAPI) BuildPages(feedSize int, title, target string, ...) error
  • func (api *EPrintsAPI) BuildSite(feedSize int, buildEPrintMirror bool) error
  • func (api *EPrintsAPI) ExportEPrints(count int) error
  • func (api *EPrintsAPI) Get(uri string) (*Record, error)
  • func (api *EPrintsAPI) GetAllRecordIDs(direction int) ([]string, error)
  • func (api *EPrintsAPI) GetAllRecords(direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetArticles(start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetEPrint(uri string) (*Record, error)
  • func (api *EPrintsAPI) GetLocalGroupArticles(groupName string, start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetLocalGroupPublications(groupName string, start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetLocalGroups(start, count, direction int) ([]string, error)
  • func (api *EPrintsAPI) GetORCIDArticles(orcid string, start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetORCIDPublications(orcid string, start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) GetORCIDs(start, count, direction int) ([]string, error)
  • func (api *EPrintsAPI) GetPublications(start, count, direction int) ([]*Record, error)
  • func (api *EPrintsAPI) ListEPrintsURI() ([]string, error)
  • func (api *EPrintsAPI) ListURI(start, count int) ([]string, error)
  • func (api *EPrintsAPI) RenderDocuments(docTitle, docDescription, docpath string, records []*Record) error
  • func (api *EPrintsAPI) RenderEPrint(basepath string, record *Record) error
• type File
• type Funder
• type FunderList
  • func (funders FunderList) ToAgencies() []string
  • func (funders FunderList) ToGrantNumbers() []string
• type NumberingSystem
• type Person
• type PersonList
  • func (persons PersonList) ToISNIs() []string
  • func (persons PersonList) ToNames() []string
  • func (persons PersonList) ToORCIDs() []string
• type Record
  • func (record *Record) PubDate() string
  • func (rec *Record) ToBibTeXElement() *bibtex.Element
• type RelatedURL

Constants

const (
	// Version is the revision number for this implementation of epgo
	Version = "v0.0.7"

	// Ascending sorts from lowest (oldest) to highest (newest)
	Ascending = iota
	// Descending sorts from highest (newest) to lowest (oldest)
	Descending = iota

	// EPrintsExportBatchSize sets the summary output frequency when exporting content from E-Prints
	EPrintsExportBatchSize = 1000

	// DefaultFeedSize sets the default size of rss, JSON, HTML include and index lists
	DefaultFeedSize = 25
)

Variables

var (

	// TmplFuncs is the collected functions available in EPGO templates
	TmplFuncs = tmplfn.Join(tmplfn.TimeMap, tmplfn.PageMap)
)

These are our main bucket and index buckets

Functions

func Slugify

func Slugify(s string) string

Turn a string into a URL friendly path part

Types

type Document

type Document struct {
	XMLName   xml.Name `json:"-"`
	ID        string   `xml:"id,attr" json:"id"`
	DocID     int      `xml:"docid" json:"docid"`
	RevNumber int      `xml:"rev_number" json:"rev_number"`
	Files     []*File  `xml:"files>file" json:"files"`
	EPrintID  int      `xml:"eprintid" json:"eprintid"`
	Pos       int      `xml:"pos" json:"pos"`
	Placement int      `xml:"placement" json:"placement"`
	MimeType  string   `xml:"mime_type" json:"mime_type"`
	Format    string   `xml:"format" json:"format"`
	Language  string   `xml:"language" json:"language"`
	Security  string   `xml:"security" json:"security"`
	License   string   `xml:"license" json:"license"`
	Main      string   `xml:"main" json:"main"`
	Content   string   `xml:"content" json:"content"`
}

Document structures in Record

type DocumentList

type DocumentList []*Document

DocumentList is an array of pointers to Document structs

type EPrintsAPI

type EPrintsAPI struct {
	XMLName        xml.Name `json:"-"`
	URL            *url.URL `xml:"epgo>api_url" json:"api_url"`                 // EPGO_API_URL
	DBName         string   `xml:"epgo>dbname" json:"dbname"`                   // EPGO_DBNAME
	BleveName      string   `xml:"epgo>bleve" json:"bleve"`                     // EPGO_BLEVE
	Htdocs         string   `xml:"epgo>htdocs" json:"htdocs"`                   // EPGO_HTDOCS
	TemplatePath   string   `xml:"epgo>template_path" json:"template_path"`     // EPGO_TEMPLATES
	SiteURL        *url.URL `xml:"epgo>site_url" json:"site_url"`               // EPGO_SITE_URL
	RepositoryPath string   `xml:"epgo>repository_path" json:"repository_path"` // EPGO_REPOSITORY_PATH
}

EPrintsAPI holds the basic connectin information to read the REST API for EPrints

func New

func New(cfg *cli.Config) (*EPrintsAPI, error)

New creates a new API instance

func (*EPrintsAPI) BuildEPrintMirror

func (api *EPrintsAPI) BuildEPrintMirror() error

func (*EPrintsAPI) BuildPages

func (api *EPrintsAPI) BuildPages(feedSize int, title, target string, filter func(*EPrintsAPI, int, int, int) ([]*Record, error)) error

BuildPages generates webpages based on the contents of the exported EPrints data. The site builder needs to know the name of the BoltDB, the root directory for the website and directory to find the templates

func (*EPrintsAPI) BuildSite

func (api *EPrintsAPI) BuildSite(feedSize int, buildEPrintMirror bool) error

BuildSite generates a website based on the contents of the exported EPrints data. The site builder needs to know the name of the BoltDB, the root directory for the website and directory to find the templates

func (*EPrintsAPI) ExportEPrints

func (api *EPrintsAPI) ExportEPrints(count int) error

ExportEPrints from highest ID to lowest for cnt. Saves each record in a DB and indexes published ones

func (*EPrintsAPI) Get

func (api *EPrintsAPI) Get(uri string) (*Record, error)

Get retrieves an EPrint record from the database

func (*EPrintsAPI) GetAllRecordIDs

func (api *EPrintsAPI) GetAllRecordIDs(direction int) ([]string, error)

GetAllRecordIDs reads and returns a list of record ids found.

func (*EPrintsAPI) GetAllRecords

func (api *EPrintsAPI) GetAllRecords(direction int) ([]*Record, error)

GetAllRecords reads and returns all records keys returning an array of keys in ascending or decending order

func (*EPrintsAPI) GetArticles

func (api *EPrintsAPI) GetArticles(start, count, direction int) ([]*Record, error)

GetArticles reads the index for published content and returns a populated array of records found in index in decending order

func (*EPrintsAPI) GetEPrint

func (api *EPrintsAPI) GetEPrint(uri string) (*Record, error)

GetEPrint retrieves an EPrint record via REST API and returns a Record structure and error

func (*EPrintsAPI) GetLocalGroupArticles

func (api *EPrintsAPI) GetLocalGroupArticles(groupName string, start, count, direction int) ([]*Record, error)

GetLocalGroupArticles returns a list of EPrint records with groupName

func (*EPrintsAPI) GetLocalGroupPublications

func (api *EPrintsAPI) GetLocalGroupPublications(groupName string, start, count, direction int) ([]*Record, error)

GetLocalGroupPublications returns a list of EPrint records with groupName

func (*EPrintsAPI) GetLocalGroups

func (api *EPrintsAPI) GetLocalGroups(start, count, direction int) ([]string, error)

GetLocalGroups returns a JSON list of unique Group names in index

func (*EPrintsAPI) GetORCIDArticles

func (api *EPrintsAPI) GetORCIDArticles(orcid string, start, count, direction int) ([]*Record, error)

GetORCIDArticles returns a list of EPrint records with a given ORCID

func (*EPrintsAPI) GetORCIDPublications

func (api *EPrintsAPI) GetORCIDPublications(orcid string, start, count, direction int) ([]*Record, error)

GetORCIDPublications returns a list of EPrint records with a given ORCID

func (*EPrintsAPI) GetORCIDs

func (api *EPrintsAPI) GetORCIDs(start, count, direction int) ([]string, error)

GetORCIDs (or ISNI) returns a list unique of ORCID/ISNI IDs in index

func (*EPrintsAPI) GetPublications

func (api *EPrintsAPI) GetPublications(start, count, direction int) ([]*Record, error)

GetPublications reads the index for published content and returns a populated array of records found in index in ascending or decending order

func (*EPrintsAPI) ListEPrintsURI

func (api *EPrintsAPI) ListEPrintsURI() ([]string, error)

ListEPrintsURI returns a list of eprint record ids

func (*EPrintsAPI) ListURI

func (api *EPrintsAPI) ListURI(start, count int) ([]string, error)

ListURI returns a list of eprint record ids from the database

func (*EPrintsAPI) RenderDocuments

func (api *EPrintsAPI) RenderDocuments(docTitle, docDescription, docpath string, records []*Record) error

RenderDocuments writes JSON, HTML, include and rss to the directory indicated by docpath

func (*EPrintsAPI) RenderEPrint

func (api *EPrintsAPI) RenderEPrint(basepath string, record *Record) error

RenderEPrint writes a single EPrint record to disc.

type File

type File struct {
	XMLName   xml.Name `json:"-"`
	ID        string   `xml:"id,attr" json:"id"`
	FileID    int      `xml:"fileid" json:"fileid"`
	DatasetID string   `xml:"datasetid" json:"datasetid"`
	ObjectID  int      `xml:"objectid" json:"objectid"`
	Filename  string   `xml:"filename" json:"filename"`
	MimeType  string   `xml:"mime_type" json:"mime_type"`
	Hash      string   `xml:"hash" json:"hash"`
	HashType  string   `xml:"hash_type" json:"hash_type"`
	FileSize  int      `xml:"filesize" json:"filesize"`
	MTime     string   `xml:"mtime" json:"mtime"`
	URL       string   `xml:"url" json:"url"`
}

File structures in Document

type Funder

type Funder struct {
	XMLName     xml.Name `json:"-"`
	Agency      string   `xml:"agency" json:"agency"`
	GrantNumber string   `xml:"grant_number,omitempty" json:"grant_number"`
}

Funder is a structure describing a funding source for record

type FunderList

type FunderList []*Funder

FunderList is an array of pointers to Funder structs

func (FunderList) ToAgencies

func (funders FunderList) ToAgencies() []string

ToAgencies takes an array of pointers to Funders and returns a list of Agency names

func (FunderList) ToGrantNumbers

func (funders FunderList) ToGrantNumbers() []string

ToGrantNumbers takes an array of pointers to Funders and returns a list of Agency names

type NumberingSystem

type NumberingSystem struct {
	XMLName xml.Name `json:"-"`
	Name    string   `xml:"name" json:"name"`
	ID      string   `xml:"id" json:"id"`
}

NumberingSystem is a structure describing other numbering systems for record

type Person

type Person struct {
	XMLName xml.Name `json:"-"`
	Given   string   `xml:"name>given" json:"given"`
	Family  string   `xml:"name>family" json:"family"`
	ID      string   `xml:"id,omitempty" json:"id"`
	ORCID   string   `xml:"orcid,omitempty" json:"orcid"`
	ISNI    string   `xml:"isni,omitempty" json:"isni"`
}

Person returns the contents of eprint>creators>item>name as a struct

type PersonList

type PersonList []*Person

PersonList is an array of pointers to Person structs

func (PersonList) ToISNIs

func (persons PersonList) ToISNIs() []string

ToISNIs takes an array of pointers to Person and returns a list of ISNI ids

func (PersonList) ToNames

func (persons PersonList) ToNames() []string

ToNames takes an array of pointers to Person and returns a list of names (family, given)

func (PersonList) ToORCIDs

func (persons PersonList) ToORCIDs() []string

ToORCIDs takes an an array of pointers to Person and returns a list of ORCID ids

type Record

type Record struct {
	XMLName              xml.Name           `json:"-"`
	Title                string             `xml:"eprint>title" json:"title"`
	URI                  string             `json:"uri"`
	Abstract             string             `xml:"eprint>abstract" json:"abstract"`
	Documents            DocumentList       `xml:"eprint>documents>document" json:"documents"`
	Note                 string             `xml:"eprint>note" json:"note"`
	ID                   int                `xml:"eprint>eprintid" json:"id"`
	RevNumber            int                `xml:"eprint>rev_number" json:"rev_number"`
	UserID               int                `xml:"eprint>userid" json:"userid"`
	Dir                  string             `xml:"eprint>dir" json:"eprint_dir"`
	Datestamp            string             `xml:"eprint>datestamp" json:"datestamp"`
	LastModified         string             `xml:"eprint>lastmod" json:"lastmod"`
	StatusChange         string             `xml:"eprint>status_changed" json:"status_changed"`
	Type                 string             `xml:"eprint>type" json:"type"`
	MetadataVisibility   string             `xml:"eprint>metadata_visibility" json:"metadata_visibility"`
	Creators             PersonList         `xml:"eprint>creators>item" json:"creators"`
	IsPublished          string             `xml:"eprint>ispublished" json:"ispublished"`
	Subjects             []string           `xml:"eprint>subjects>item" json:"subjects"`
	FullTextStatus       string             `xml:"eprint>full_text_status" json:"full_text_status"`
	Keywords             string             `xml:"eprint>keywords" json:"keywords"`
	Date                 string             `xml:"eprint>date" json:"date"`
	DateType             string             `xml:"eprint>date_type" json:"date_type"`
	Publication          string             `xml:"eprint>publication" json:"publication"`
	Volume               string             `xml:"eprint>volume" json:"volume"`
	Number               string             `xml:"eprint>number" json:"number"`
	PageRange            string             `xml:"eprint>pagerange" json:"pagerange"`
	IDNumber             string             `xml:"eprint>id_number" json:"id_number"`
	Referred             bool               `xml:"eprint>refereed" json:"refereed"`
	ISSN                 string             `xml:"eprint>issn" json:"issn"`
	OfficialURL          string             `xml:"eprint>official_url" json:"official_url"`
	RelatedURL           []*RelatedURL      `xml:"eprint>related_url>item" json:"related_url"`
	ReferenceText        []string           `xml:"eprint>referencetext>item" json:"referencetext"`
	Rights               string             `xml:"eprint>rights" json:"rights"`
	OfficialCitation     string             `xml:"eprint>official_cit" json:"official_citation"`
	OtherNumberingSystem []*NumberingSystem `xml:"eprint>other_numbering_system>item,omitempty" json:"other_numbering_system"`
	Funders              FunderList         `xml:"eprint>funders>item" json:"funders"`
	Collection           string             `xml:"eprint>collection" json:"collection"`
	Reviewer             string             `xml:"eprint>reviewer" json:"reviewer"`
	LocalGroup           []string           `xml:"eprint>local_group>item" json:"local_group"`
}

Record returns a structure that can be converted to JSON easily

func (*Record) PubDate

func (record *Record) PubDate() string

func (*Record) ToBibTeXElement

func (rec *Record) ToBibTeXElement() *bibtex.Element

ToBibTeXElement takes an epgo.Record and turns it into a bibtex.Element record

type RelatedURL

type RelatedURL struct {
	XMLName     xml.Name `json:"-"`
	URL         string   `xml:"url" json:"url"`
	Type        string   `xml:"type" json:"type"`
	Description string   `xml:"description" json:"description"`
}

RelatedURL is a structure containing information about a relationship