gnmatcher

command module
v1.1.9 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 3, 2023 License: MIT Imports: 2 Imported by: 2

README

gnmatcher

API GoDoc

import "github.com/gnames/gnmatcher"

The GNmatcher is a utility for checking scientific name-strings against canonical names derived from a variety of biodiversity datasets. The matching can be exact, partial, or fuzzy. GNmatcher does not tell which datasets provided found canonical name. For this purpose use GNames project.

Introduction

The GNmatcher takes an array of strings and returns back zero or more canonical names for each string. If it is not important which biodiversity repositories provided matched canonical names, the project can be used as a stand-alone RESTful service. If such information is important, the project is used as a component of a scientific names verification service (GNames API).

The project aims to do canonical names matching as fast and accurate as possible. Quite often, humans or character-recognition software (OCR) introduce misspellings in the strings. For this reason, GNmatcher uses fuzzy-matching algorithms when no exact match exists. Also, for cases where full string does not have a match, gnmatcher tries to match it against parts of the string. For example, if a string did not get a match on a subspecies level, the algorithm will try to match it on species and genus levels.

Reconciliation is the normalization of lexical variants of the same name. Resolution is a determination of how a nomenclaturally registered name can be interpreted from the point of taxonomy. For example, a name can be an accepted name for species, a synonym, or a discarded one.

The gnmatcher app functions as an HTTP service. A client access it wia HTTP protocol. The API's methods and structures are described in by the RESTful API documentation.

Input and Output

A user calls HTTP resource /match sending an array of strings to the service and gets back matched canonical names, the match type, as well as other metadata described as a Match object in the RESTful API documentation.

The optimal size of the input is 5-10 thousand strings per array. Note that 10,000 is the maximal size, and larger arrays will be truncated.

Performance

For performance measurement we took 100,000 strings where only 30% of them were 'real' names. On a modern CPU with 12 hyper threads and GNM_JOBS_NUM environment variable set to 8, the service was able to process about 8,000 strings per second. For 'clean' data where most of the names are "real", you should see an even higher performance.

Prerequisites

  • You will need PostgreSQL with a restored dump of gnames database.
  • For PostgreSQL collation to work correctly set LC_COLLATE=C in /etc/default/locale
  • Docker service

Usage

Usage with docker
  • Install docker gnmatcher image: docker pull gnames/gnmatcher.

  • Copy .env.example file on user's disk and change values of environment variables accordingly.

  • Start the service:

    docker run -p 8080:8080 -d --env-file your-env-file \
    gnames/gnmatcher -- rest -p 8080`
    

    This command will set the service on port 8080 and will make it available through port 8080 on a local machine.

Usage from command line
  • Download the [latest verion] of gnmatcher binary, untar and put somewhere in PATH.

  • Run gnmatcher -V to generate configuration at ~/.config/gnmatcher.yaml

  • Edit ~/.config/gnmatcher.yaml accordingly.

  • Run gnmatcher rest -p 1234

The service will run on the given port (the default port is 8080).

Usage as a library
package main

import (
  "fmt"
  gnmatcher "github.com/gnames/gnmatcher/pkg"
  "github.com/gnames/gnmatcher/pkg/config"
  "github.com/gnames/gnmatcher/pkg/io/bloom"
  "github.com/gnames/gnmatcher/pkg/io/trie"
	"github.com/gnames/gnmatcher/pkg/io/virusio"
)

func Example() {
	// Note that it takes several minutes to initialize lookup data structures.
	// Requirement for initialization: Postgresql database with loaded
	// http://opendata.globalnames.org/dumps/gnames-latest.sql.gz
	//
	// If data are imported already, it still takes several seconds to
	// load lookup data into memory.
	cfg := config.New()
	em := bloom.New(cfg)
	fm := trie.New(cfg)
	vm := virusio.New(cfg)
	gnm := gnmatcher.New(em, fm, vm, cfg)
	res := gnm.MatchNames([]string{"Pomatomus saltator", "Pardosa moesta"})
	for _, match := range res.Matches {
		fmt.Println(match.Name)
		fmt.Println(match.MatchType)
		for _, item := range match.MatchItems {
			fmt.Println(item.MatchStr)
			fmt.Println(item.EditDistance)
		}
	}
}

Configuration

You can use either cofiguration file, or environment variables. Configuration file is usually located at $HOME/.config/gnmatcher.yaml. When gnmatcher runs first time, it will create the configuration file. Logs provide the location of the configuration file with every run of gnmatcher. The meanings of configuration options are documented in the config file.

To make it easier to run gnmatcher in a container, or a Kubernetes pod, there are also environment variables that override configuration file values.

Env. Var. Configuration
GNM_CACHE_DIR CacheDir
GNM_JOBS_NUM JobsNum
GNM_MAX_EDIT_DIST MaxEditDist
GNM_PG_HOST PgHost
GNM_PG_PORT PgPort
GNM_PG_USER PgUser
GNM_PG_PASS PgPass
GNM_PG_DB PgDB
GNM_NSQD_TCP_ADDRESS NsqdTCPAddress
GNM_NSQD_CONTAINS_FILTER NsqdContainsFilter
GNM_NSQD_REGEX_FILTER NsqdRegexFilter
GNM_WITH_WEB_LOGS WithWebLogs

Client

A user can find an example of a client for the service in this test file.

The API is formally described in the RESTful API documentation

Development

There is a docker-compose file that sets up HTTP service to run tests. To run it to the following:

  1. Copy .env.example file to the .env file in the project's root directory, change the settings accordingly.

  2. Build the gnmatcher binary and docker image using make dc command.

  3. Run docker-compose command docker compose up

  4. Run tests via go test ./... -v

Documentation

Overview

package main provides an CLI interface to http service to run gnmatcher functionality.

Copyright © 2020 Dmitry Mozzherin <dmozzherin@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Directories

Path Synopsis
internal
ent/exact
package exact contains interface for exact-matching strings to known scientific names.
package exact contains interface for exact-matching strings to known scientific names.
ent/fuzzy
package fuzzy contains interfaces and code to facilitate fuzzy-matching of name-strings to scientific names collected in gnames database.
package fuzzy contains interfaces and code to facilitate fuzzy-matching of name-strings to scientific names collected in gnames database.
ent/matcher
package matcher is the central processing unit for matching name-strings to known scientific names.
package matcher is the central processing unit for matching name-strings to known scientific names.
ent/virus
package virus contains an interface for matching strings to names of viruses, plasmids, prions and other non-cellular entities.
package virus contains an interface for matching strings to names of viruses, plasmids, prions and other non-cellular entities.
pkg
package gnmatcher provides the main use-case of the project, which is matching of possible name-strings to scientific names registered in a variety of biodiversity databases.
package gnmatcher provides the main use-case of the project, which is matching of possible name-strings to scientific names registered in a variety of biodiversity databases.
config
package config contains information needed to run gnmatcher project.
package config contains information needed to run gnmatcher project.
io/bloom
package bloom creates and serves bloom filters for stemmed canonical names, and names of viruses.
package bloom creates and serves bloom filters for stemmed canonical names, and names of viruses.
io/dbase
package dbase provides convenience methods for accessing PostgreSQL database.
package dbase provides convenience methods for accessing PostgreSQL database.
io/rest
package rest provides http REST interface to gnmatcher functionality.
package rest provides http REST interface to gnmatcher functionality.
io/trie
package trie implements FuzzyMatcher interface that is responsible for fuzzy-matching strings to canonical forms of scientific names.
package trie implements FuzzyMatcher interface that is responsible for fuzzy-matching strings to canonical forms of scientific names.
The purpose of this script is to find out how fast algorithms can go through a list of 100_000 names.
The purpose of this script is to find out how fast algorithms can go through a list of 100_000 names.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL