span

README

Span

The span tools convert to and from an intermediate schema and support license tagging and quality assurance.

The intermediate schema is a normalization vehicle, spec: https://github.com/ubleipzig/intermediateschema

---

Install with

$ go get github.com/miku/span/cmd/...

or via deb or rpm packages.

Formats

• CrossRef API, works and members
• JATS Journal Archiving and Interchange Tag Set, with various flavours for JSTOR and others
• DOAJ exports
• FINC Intermediate Format
• Various FINC SOLR Schema
• GENIOS Profile XML
• Elsevier Transport
• Thieme TM Style
• Formeta
• IEEE IDAMS Exchange V2.0.0

Also:

• KBART

Addings data sources

The following kinds of data shapes are supported at the moment:

• A stream of XML, containing zero, one or more records, identified by an XML tag. Moderately fast.
• Newline delimited JSON, containing zero, one or more records, one record per line. Fast.
• Single records of arbitrary shape. Slow.

Use span, if metafacture or jq or a Python snippet are not sufficient.

Steps:

• Add a new subpackage for your format, e.g. dummy.
• Add a struct representing the original record (XML, JSON, bytes).
• Implement the conversion functions required, e.g. ToIntermediateSchema
• Add an entry into the format map for span-import
• Decide, which kind of source this is (XML stream, newline delimited JSON, single records, something else)
• Recompile and ship.

Ideas for span 0.2.0

TODO:

• Do not require recompilation for mapping updates (allow various sources)
• Decouple format from source. Things like SourceID and MegaCollection are per source, not format.
• Allow loadable assets from ~/.config/span/maps, some specified location or a single JSON file.

More taggable formats:

Let formats implement a single function interface, that takes a filter value. How to make this filter generic? Or use Tag as an adapeter?

// Tag alters the document by applying a filter to it.
func (doc *Document) Tag(f Filter) error {
    // Adapt.
    record = FilterRecord{
	ISSN: doc.ISSN,
	Title: doc.CombinedTitle(),
	// ...
    }
    // Apply.
    return f.Apply(record)
}

DONE:

• Reuse more generic code, e.g. parallel
• Make conversions a simpler with xmlstream, zek

Licence

• GPLv3
• This project uses the Compact Language Detector 2 - CLD2, Apache License Version 2.0

Documentation

Overview

Package span implements common functions.

Copyright 2015 by Leipzig University Library, http://ub.uni-leipzig.de
                  The Finc Authors, http://finc.info
                  Martin Czygan, <martin.czygan@uni-leipzig.de>

This file is part of some open source application.

Some open source application is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.

Some open source application is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with Foobar. If not, see <http://www.gnu.org/licenses/>.

@license GPL-3.0+ <http://spdx.org/licenses/GPL-3.0+>

Index

• Constants
• Variables
• func DetectLang3(text string) (string, error)
• func LanguageIdentifier(s string) string
• func LoadSet(r io.Reader, m map[string]struct{}) error
• func ReadLines(filename string) (lines []string, err error)
• func UnescapeTrim(s string) string
• func UnfreezeFilterConfig(frozenfile string) (string, string, error)
• type ArrayFlags
  • func (f *ArrayFlags) Set(value string) error
  • func (f *ArrayFlags) String() string
• type FileReader
  • func (r *FileReader) Close() (err error)
  • func (r *FileReader) Read(p []byte) (n int, err error)
• type LinkReader
  • func (r *LinkReader) Read(p []byte) (int, error)
• type SavedLink
  • func (s *SavedLink) Remove()
  • func (s *SavedLink) Save() (filename string, err error)
• type SavedReaders
  • func (r *SavedReaders) Remove()
  • func (r *SavedReaders) Save() (filename string, err error)
• type Skip
  • func (s Skip) Error() string
• type SkipReader
  • func NewSkipReader(r *bufio.Reader) *SkipReader
  • func (r SkipReader) ReadString(delim byte) (s string, err error)
• type WriteCounter
  • func (w *WriteCounter) Count() uint64
  • func (w *WriteCounter) Write(p []byte) (int, error)
• type ZipContentReader
  • func (r *ZipContentReader) Read(p []byte) (int, error)
• type ZipOrPlainLinkReader
  • func (r *ZipOrPlainLinkReader) Read(p []byte) (int, error)

Constants

const (
	// AppVersion of span package. Commandline tools will show this on -v.
	AppVersion = "0.1.226"
	// KeyLengthLimit was a limit imposed by the memcached protocol, which
	// was used for blob storage until Q1 2017. We switched the key-value
	// store, so this limit is somewhat obsolete.
	KeyLengthLimit = 250
)

Variables

var ISO639BibliographicToThree = map[string]string{

	"alb": "sqi",
	"arm": "hye",
	"baq": "eus",
	"bur": "mya",
	"chi": "zho",
	"cze": "ces",
	"dut": "nld",
	"fre": "fra",
	"geo": "kat",
	"ger": "deu",
	"gre": "ell",
	"ice": "isl",
	"mac": "mkd",
	"mao": "mri",
	"may": "msa",
	"per": "fas",
	"rum": "ron",
	"slo": "slk",
	"tib": "bod",
	"wel": "cym",
}

ISO639BibliographicToThree maps 639-2 identifier of the bibliographic applications to three-letter 639-3 identifier.

var ISO639NameToThree = map[string]string{}/* 7849 elements not displayed */

ISO639NameToThree maps a language name to three letter identifier.

var ISO639NameToThreeLower = map[string]string{}/* 7850 elements not displayed */

var ISO639OneToThree = map[string]string{}/* 184 elements not displayed */

ISO639OneToThree maps 639-1 identifier (two letters) (if there is one) to a three-letter 639-3 identifier.

var ISSNPattern = regexp.MustCompile(`[0-9]{4,4}-[0-9]{3,3}[0-9X]`)

ISSNPattern is a regular expression matching standard ISSN.

Functions

func DetectLang3

func DetectLang3(text string) (string, error)

DetectLang3 returns the best guess 3-letter language code for a given text.

func LanguageIdentifier

func LanguageIdentifier(s string) string

LanguageIdentifier returns the three letter identifier from any string. All data from http://www-01.sil.org/iso639-3/codes.asp.

func LoadSet

func LoadSet(r io.Reader, m map[string]struct{}) error

LoadSet reads the content of from a reader and creates a set from each line.

func ReadLines

func ReadLines(filename string) (lines []string, err error)

ReadLines returns a list of trimmed lines in a file. Empty lines are skipped.

func UnescapeTrim

func UnescapeTrim(s string) string

UnescapeTrim unescapes HTML character references and trims the space of a given string.

func UnfreezeFilterConfig

func UnfreezeFilterConfig(frozenfile string) (string, string, error)

UnfreezeFilterConfig takes the name of a zipfile (from span-freeze) and returns of the path the thawed filterconfig (along with the temporary directory and error). All URLs in the filterconfig have then been replaced by absolute path on the file system. Assumes default values for mapping.json and content file (blob). Cleanup of temporary directory is responsibility of caller.

Types

type ArrayFlags

type ArrayFlags []string

ArrayFlags allows to store lists of flag values.

func (*ArrayFlags) Set

func (f *ArrayFlags) Set(value string) error

Set appends a value.

func (*ArrayFlags) String

func (f *ArrayFlags) String() string

String representation.

type FileReader

type FileReader struct {
	Filename string
	// contains filtered or unexported fields
}

FileReader creates a ReadCloser from a filename. If postpones error handling up until the first read. TODO(miku): Throw this out.

func (*FileReader) Close

func (r *FileReader) Close() (err error)

Close closes the file.

func (*FileReader) Read

func (r *FileReader) Read(p []byte) (n int, err error)

Read reads from the file.

type LinkReader

type LinkReader struct {
	Link string
	// contains filtered or unexported fields
}

LinkReader implements io.Reader for a URL.

func (*LinkReader) Read

func (r *LinkReader) Read(p []byte) (int, error)

type SavedLink

type SavedLink struct {
	Link string
	// contains filtered or unexported fields
}

SavedLink saves the content of a URL to a file.

func (*SavedLink) Remove

func (s *SavedLink) Remove()

Remove remove any left over temporary file.

func (*SavedLink) Save

func (s *SavedLink) Save() (filename string, err error)

Save link to a temporary file, return the filename.

type SavedReaders

type SavedReaders struct {
	Readers []io.Reader
	// contains filtered or unexported fields
}

SavedReaders takes a list of readers and persists their content in a temporary file.

func (*SavedReaders) Remove

func (r *SavedReaders) Remove()

Remove remove any left over temporary file.

func (*SavedReaders) Save

func (r *SavedReaders) Save() (filename string, err error)

Save saves all readers to a temporary file and returns the filename.

type Skip

type Skip struct {
	Reason string
}

Skip marks records to skip.

func (Skip) Error

func (s Skip) Error() string

Error returns the reason for skipping.

type SkipReader

type SkipReader struct {
	CommentPrefixes []string
	// contains filtered or unexported fields
}

SkipReader skips empty lines and lines with comments.

func NewSkipReader

func NewSkipReader(r *bufio.Reader) *SkipReader

NewSkipReader creates a new SkipReader.

func (SkipReader) ReadString

func (r SkipReader) ReadString(delim byte) (s string, err error)

ReadString will return only non-empty lines and lines not starting with a comment prefix.

type WriteCounter

type WriteCounter struct {
	// contains filtered or unexported fields
}

WriteCounter counts the number of bytes written through it.

func (*WriteCounter) Count

func (w *WriteCounter) Count() uint64

Count returns the number of bytes written.

func (*WriteCounter) Write

func (w *WriteCounter) Write(p []byte) (int, error)

Write increments the total byte count.

type ZipContentReader

type ZipContentReader struct {
	Filename string
	// contains filtered or unexported fields
}

ZipContentReader returns the concatenated content of all files in a zip archive given by its filename. All content is temporarily stored in memory, so this type should only be used with smaller archives.

func (*ZipContentReader) Read

func (r *ZipContentReader) Read(p []byte) (int, error)

Read returns the content of all archive members.

type ZipOrPlainLinkReader

type ZipOrPlainLinkReader struct {
	Link string
	// contains filtered or unexported fields
}

ZipOrPlainLinkReader is a reader that transparently handles zipped and uncompressed content, given a URL as string.

func (*ZipOrPlainLinkReader) Read

func (r *ZipOrPlainLinkReader) Read(p []byte) (int, error)

Read implements the reader interface.