newt

package module
v0.0.0-...-97a1f3c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 6, 2023 License: BSD-3-Clause Imports: 15 Imported by: 0

README

Newt, a new take on the web stack

Newt is two things. Newt is a "new take" on building web applications. Newt is an experimental microservice working with other "off the shelf" microservices.

Newt integrates Postgres, PostgREST and Pandoc by function as a data router and a light weight static file server.

Newt routes a request to a JSON data API (e.g. PostgREST, Solr, Elasticsearch) and then optionally send that result through Pandoc for further processing. Newt runs as a localhost service. Inm production you'd use Newt behind a traditional web server like Apache 2 or NginX.

Motivation

My belief is that many web services used by Archives, Libraries and Museums can benefit from a simplified and consistent back-end. If the back-end is "easy" then the limited developer resources can be focused on the front-end. An improved front-end offers opportunities to provide a more humane user experience for staff and patrons.

Newt's approach

From front-end to back-end

  • A front end web server (e.g. Apache 2, NginX) can provide access control where appropriate (e.g. Single Sign on with OAuth2 or Shibboleth)
  • Newt provides static file services but more importantly serves as a data router. I can map a request to a JSON data API, take those results then send them through Pandoc for transformation.
  • Postgres+PostgREST is an example of a JSON data API
  • Solr, Elasticsearch or Opensearch can also function as a JSON data API
  • Pandoc server provides a templating engine to transform data sources

All these can be treated as "off the shelf". Aside from configuration they can run as traditional services on most POSIX systems. Your application is implemented using SQL. It is enhanced by using Pandoc templates used to turn JSON into HTML (or other desired formats).

Exploring Newt's friends

In the repository three demos are provided to show how Pandoc, Postgres+PostgRESt and Newt can work together. The demos are generated using the a simple Bash scripts (see below). The bash scripts will generate all the file needed to run the demos. A read me for each demo describes how to run it.

Birds 1 Demo : Shows a simple use of Pandoc to render a static bird sightings website

Birds 2 Demo : Shows a dynamic bird sightings website using Postgres+PostgREST, but requires JavaScript running in your web browser

Birds 3 Demo : Shows a dynamic bird sightings website using Newt with Postgres+PostgREST and Pandoc

The goal of the three demos is to show an evolution towards simplicity.

Birds 1 Demo, Pandoc only

This is a simple static website. It introduces Pandoc and which we will leverage again in bird 3 demo. It also is a good tool to rendering static content with. Static websites are generally simple to generate and maintain.

  • README.md, demo read me
  • birds.csv, our list of bird sightings
  • build.sh, a shell script that uses Pandoc to render site (3 lines)
  • htdocs, the website document directory
  • page.tmpl, a Pandoc page template using by build.sh (13 lines)

Pros Cons

Pandoc is widely known Updates require changing birds.csv Pandoc is easily scripted and rebuilding the site.

Birds 2 Demo, Postgres + PostgREST

Postgres and PostgREST can be combined to provide a dynamic data source for our birds 2 demo website. In birds 2 demo PostgREST runs on localhost and we use another web server to presents the static files and to proxy to PostgREST. The enhancements over birds 1 demo is our list of birds is held in a Postgres database which is available to our web browser thanks to proxying to PostgREST. The web browser uses JavaScript to call back to the JSON data API and using the results to render content in the web page.

The back-end is written using SQL. This includes setting up access by PostgREST. The font-end JavaScript is required in the browser to assemble pages results from our JSON data API is complex. The complexity is shifted out of the back-end which is using an off the shelf micro service to the front-end.

  • README.md, demo read me
  • birds.csv, our sightings data, loaded into SQL database
  • htdocs, the website document directory
  • htdocs/sightings.js, JavaScript support is required in the web browser to render the page and handle updates (63 lines)
  • postgrest.conf, configuration for PostgREST (3 lines)
  • setup.sql, SQL setting up data models and JSON data API (50 lines)

Pros Cons

Simple back-end, just SQL code JavaScript in the browser is complex You need a static file server You need to know some SQL

Birds 3 demo, assembling responses with Newt

Managing page assembly browser side is a chore. We can skip the complex JavaScript browser side if we let Newt do the routing of requests to PostgREST and then sending those results through Pandoc like we did in our static site demo. With just SQL and Pandoc templates we can build a web application.

Newt needs to know how to map the front-end requests to PostgREST so to do that it reads a CSV file holding data routing instructions. E.g. turn a request URL into a PostgREST URL and run the results through Pandoc running as a microservice.

  • README.md, demo read me
  • birds-routes.csv, CSV holding data routing instructions (3 lines)
  • birds.csv, CSV holding data to be loaded into our SQL database
  • birds.yaml, configuration for Newt
  • htdocs, holds our static content
  • page.tmpl, Pandoc template for listing birds (13 lines)
  • postgrest.conf, configuration for PostgREST
  • setup.sql, SQL setting of data models and JSON data API (50 lines)

This is very similar to both demo 1 and 2. Missing is build.sh from demo 1. We don't need it since we're running Pandoc as a microservice. There is an added configuration file for Newt. The Pandoc template performs a similar duty as the one used in birds 1 demo. Notice there is no sightings.js in our htdocs directory. From the web browsers point of view there is no need to run JavaScript to submit a standard web form to add a bird sighting.

NOTE: Newt provides our static file service so when developing we can skip Apache 2 or NginX.

Pros Cons

Simple back-end, just SQL code Like demo 2 you need to know some SQL No JavaScript required browser side Newt provides the static web server

Conclusion

In the birds 3 demo I've delegated tasks to a series of flexible microservices.

Postgres : provides data storage and defines how our JSON data API works

PostgREST : Turns a Postgres database into a our JSON data API service

Pandoc : Run in server mode is a powerful template engine, it can convert our JSON data into a web page

Newt : Is a data router and static file server. It translates the web form submission into JSON before sending requests to PostgREST. Newt takes the results and sends that through Pandoc. Newt can also service static files. It could be used to talk to a JSON oriented full text search engine like Solr, Elasticsearch or Opensearch.

Front-end web server : A front-end web server can provide access control and proxy to any of our microservices, leverage virtual hosting, etc.

The "coding" of the back-end is reduced to SQL and Pandoc templates. You are free to make the front-end as simple or as complex as you like. The microservices and front-end web server effectively snap together like LEGO bricks.

Documentation

Overview

*

  • newt.go an implementation of the Newt URL router. *
  • @author R. S. Doiel

Index

Constants

View Source 
const (
	StartVar = "${"
	EndVar   = "}"
)
View Source 
const (
	// Version number of release
	Version = "0.0.1"

	// ReleaseDate, the date version.go was generated
	ReleaseDate = "2023-06-05"

	// ReleaseHash, the Git hash when version.go was generated
	ReleaseHash = "c5f3051"

	LicenseText = `` /* 1430-byte string literal not displayed */

)

Variables

View Source 
var (
	// RdslTypes is a map to the types defined in route_dsl_types.go
	RouteTypes = map[string]EvalType{
		"String":   new(RdslString).EvalType,
		"Year":     new(RdslYear).EvalType,
		"Month":    new(RdslMonth).EvalType,
		"Day":      new(RdslDay).EvalType,
		"Basename": new(RdslBasename).EvalType,
		"Extname":  new(RdslExtname).EvalType,
		"Isbn10":   new(RdslIsbn10).EvalType,
		"Isbn13":   new(RdslIsbn13).EvalType,
		"Isbn":     new(RdslIsbn).EvalType,
		"Issn":     new(RdslIssn).EvalType,
		"DOI":      new(RdslDOI).EvalType,
		"Isni":     new(RdslIsni).EvalType,
		"ORCID":    new(RdslORCID).EvalType,
	}
)

Functions

func FmtHelp 

func FmtHelp(src string, appName string, version string, releaseDate string, releaseHash string) string

FmtHelp lets you process a text block with simple curly brace markup.

func JSONSrcToFrontMatter 

func JSONSrcToFrontMatter(src []byte) (string, error)

toFrontMatter converts JSON source to front matter for a Pandoc Markdown document.

func Run 

func Run(in io.Reader, out io.Writer, eout io.Writer, args []string, dryRun bool) int

Run is a runner for Newt URL router and static file server

Types

type Config 

type Config struct {
	// Port is the name of the localhost port Newt will listen on.
	Port string `json:"newt_port,omitempty" yaml:"newt_port,omitempty"`
	// Routes is the CSV filename that data API routes are read from
	Routes string `json:"newt_routes,omitempty" yaml:"newt_routes,omitempty"`
	// Env is a list of environment variables that can be passed
	// through to the RouteDSL when rendering JSON data API calls or
	// calls to Pandoc server.
	Env []string `json:"newt_env,omitempty" yaml:"newt_env,omitempty"`
	// Htdocs holds any static files you want to make available through
	// Newt router.
	Htdocs string `json:"newt_htdocs,omitempty" yaml:"newt_htdocs,omitempty"`
}

func LoadConfig 

func LoadConfig(configFName string) (*Config, error)

LoadConfig loads a configuration return a Config object and error value.

type EvalType 

type EvalType func(string, string) (string, bool)

type RdslBasename 

type RdslBasename struct {
}

func (RdslBasename) EvalType 

func (basename RdslBasename) EvalType(expr string, val string) (string, bool)

type RdslDOI 

type RdslDOI struct {
}

func (RdslDOI) EvalType 

func (doi RdslDOI) EvalType(expr string, val string) (string, bool)

type RdslDay 

type RdslDay struct {
}

func (RdslDay) EvalType 

func (day RdslDay) EvalType(expr string, val string) (string, bool)

type RdslExtname 

type RdslExtname struct {
}

func (RdslExtname) EvalType 

func (extname RdslExtname) EvalType(expr string, val string) (string, bool)

type RdslIsbn 

type RdslIsbn struct {
}

func (RdslIsbn) EvalType 

func (isbn RdslIsbn) EvalType(extr string, val string) (string, bool)

type RdslIsbn10 

type RdslIsbn10 struct {
}

func (RdslIsbn10) EvalType 

func (isbn10 RdslIsbn10) EvalType(expr string, val string) (string, bool)

type RdslIsbn13 

type RdslIsbn13 struct {
}

func (RdslIsbn13) EvalType 

func (isbn13 RdslIsbn13) EvalType(extr string, val string) (string, bool)

type RdslIsni 

type RdslIsni struct {
}

func (RdslIsni) EvalType 

func (isni RdslIsni) EvalType(expr string, val string) (string, bool)

type RdslIssn 

type RdslIssn struct {
}

func (RdslIssn) EvalType 

func (issn RdslIssn) EvalType(expr string, val string) (string, bool)

type RdslMonth 

type RdslMonth struct {
}

func (RdslMonth) EvalType 

func (month RdslMonth) EvalType(expr string, val string) (string, bool)

type RdslORCID 

type RdslORCID struct {
}

func (RdslORCID) EvalType 

func (orcid RdslORCID) EvalType(expr string, val string) (string, bool)

type RdslString 

type RdslString struct {
}

Route DSL types

func (RdslString) EvalType 

func (str RdslString) EvalType(expr string, val string) (string, bool)

type RdslYear 

type RdslYear struct {
}

func (RdslYear) EvalType 

func (year RdslYear) EvalType(expr string, val string) (string, bool)

type Route 

type Route struct {
	// ReqPath, a path described by RouteDSL from a browser or front end web server
	ReqPath *RouteDSL
	// ReqMethod, the request HTTP method (e.g. GET, POST, PUT, DELETE, PATCH, HEAD)
	ReqMethod string
	// ApiURL is the URL used to contact the data source (e.g. PostgREST, Solr, Elasticsearch)
	ApiURL string
	// ApiMethod indicates the HTTP method to be used to contact the data source api
	ApiMethod string
	// ApiContentType is the content type to be used to contact the data source api
	ApiContentType string
	// PandocTemplate holds the source to a Pandoc template
	PandocTemplate string
	// ResHeaders holds any additional response headers to send back to the
	// browser or front end web server
	ResHeaders map[string]string
}

Route describes an individual route, maps to the columns of a route CSV file.

func (*Route) HasReqMethod 

func (route *Route) HasReqMethod(method string) bool

func (*Route) String 

func (route *Route) String() string

type RouteDSL 

type RouteDSL struct {
	Src  string   `json:"src"`
	Dirs []string `json:"dirs,omitempty"`
	Base string   `json:"base,omitempty"`
	Ext  string   `json:"ext,omitempty"`
	// VarToType maps the variable name to a var defn
	VarToType map[string]string `json:"var_to_types,omitempty"`
	// Types maps type implementation description
	Types map[string]string `json:"-"`
	// Type name to function to Eval function (validates a variable's
	// value and extracts a value)
	TypeFn map[string]EvalType `json:"-"`
}

RouteDSL holds the attributes need to decode a RouteDSL expression, match and decode against path values.

func NewRouteDSL 

func NewRouteDSL(src string) (*RouteDSL, error)

NewRouteDSL takes a RouteDSL expression and returns a RouteDSLExpresion structure and error value.

func (*RouteDSL) Eval 

func (rdsl *RouteDSL) Eval(pathValue string) (map[string]string, bool)

Eval takes a path value and compares it with a Path expression. It returns a status boolean, map of variable names to values.

func (*RouteDSL) RegisterType 

func (rdsl *RouteDSL) RegisterType(tName string, t RouteDSLType) error

RegisterType maps a type name to a a RouteDSLType interface. RouteDSLType interface must defined EvalType.

func (*RouteDSL) Resolve 

func (rdsl *RouteDSL) Resolve(m map[string]string, src string) string

Resolve takes a map of varnames and values and replaces any occurrences found in src string resulting to a new string..

func (*RouteDSL) String 

func (rdsl *RouteDSL) String() string

type RouteDSLType 

type RouteDSLType interface {
	// EvalType takes an variable type expression like
	EvalType(string, string) (string, bool)
}

RouteDSLType

type Router 

type Router struct {
	Env    map[string]string
	Routes []*Route
}

func (*Router) Getenv 

func (router *Router) Getenv(key string) string

Getenv retrieves an environment value using a name

func (*Router) Newt 

func (router *Router) Newt(next http.Handler) http.Handler

Newt implements the router as an http middleware. This allows our router to be used more generally and gives us options like having the Newt application be both a data router and static file server.

func (*Router) OverlayEnv 

func (router *Router) OverlayEnv(m map[string]string) map[string]string

OverlayEnv merges the env with m (a map[string]string). For keys in common the value in env replaces that in m.

func (*Router) ReadCSV 

func (router *Router) ReadCSV(fName string) error

ReadCSV filename

func (*Router) RequestDataAPI 

func (router *Router) RequestDataAPI(rNo int, apiURL string, body []byte) ([]byte, string, int)

RequestDataAPI

func (*Router) RequestPandoc 

func (router *Router) RequestPandoc(rNo int, src []byte) ([]byte, string, int)

RequestPandoc

func (*Router) ResolveApiURL 

func (router *Router) ResolveApiURL(no int, m map[string]string) (string, bool)

ResolveApiURL takes an in bound Request URL, matches it to a route and returns a resolved JSON data API URL based the related api_url. It included an error if something went wrong.

func (*Router) ResolveRoute 

func (router *Router) ResolveRoute(u string, method string) (int, map[string]string, bool)

func (*Router) Setenv 

func (router *Router) Setenv(key string, val string)

Setenv adds an explicit environment name and value to Router.Env

func (*Router) WriteResponse 

func (router *Router) WriteResponse(w http.ResponseWriter, rNo int, src []byte)

WriteResponse

Source Files

View all Source files

Directories

Path Synopsis
cmd
newt

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.