kubeless

README

dashboard function

A Golang-based Kubeless function to deploy in a Kubernetes cluster.

Dependencies

• Kubeless v1.0.7
• Minio
• Redis

Pre-deploy setup

Upload file to object storage (Minio)

Either use the Minio web interface or use the Minio command line tool

Add host

$_> mc config add locals3 $(minikube service --url minio --namespace dictybase) ACCESS_KEY SECRET_KEY

Create bucket

$_> mc mb locals3/dashboard

Upload file to any folder inside that bucket

$_> mc copy canonical.gff3 locals3/dashboard/genomes/44689/

The above bucket and folder path are for example only, any name could be used instead.

Create metadata json file

This file specifies the organism information and file location in the object storage. So for D.discoideum, the metadata file should have following information.

{
  "taxon_id": "44689",
  "scientific_name": "Dictyostelium discoideum",
  "common_name": "Slime mold",
  "rank": "Species",
  "bucket": "dashboard",
  "file": "genomes/44689/canonical_core.gff3"
}

Out of all fields, taxon_id,bucket and file are necessary. The taxonomic information is available here.

Deploy function

$_> zip dashfn.zip *.go go.mod
$_> kubeless function deploy \
dashfn --runtime go1.13 --from-file dashfn.zip --handler dashboard.Handler
--dependencies go.mod --namespace dictybase
-e MINIO_ACCESS_KEY=xxxxxxxx -e MINIO_SECRET_KEY=xxxxxxxx

• check the status of function

  $_> kubeless function ls -n dictybase

Add Ingress

Create a YAML file like this GKE Ingress example.

$_> kubectl apply -f gke-ingress.yaml -n dictybase

Endpoints

They will be available through the mapped host, for example through betafunc.dictybase.org assuming the above Ingress.

POST /dashboard/genomes - Generating information for various biological feature types (chromosome,gene etc..). It will use the metadata.json file to download the gff3 file from object storage and persist the information in Redis cache. An example HTTP request to this endpoint will look like this.

$_> curl -k -d @metadata.json https://betafunction.dictybase.local/dashboard/genomes

GET /dashboard/genomes/{taxon_id}/{type} - Information about reference feature such as chromosome.

For reference features such as chromosome, supercontig the JSON format will be the following

$_> curl -k https://betafunction.dictybase.local/dashboard/genomes/44689/chromosomes

{
  "data": [
    {
      "type": "chromosomes",
      "id": "....",
      "attributes": {
        "name": "...",
        "start": "...",
        "end": "...",
        "length": "...."
      }
    }
  ]
}

$_> curl -k https://betafunc.dictybase.org/dashboard/genomes/44689/genes
The format for the other features (gene, mRNA, pseudogene, etc)

{
  "data": [
    {
      "type": "genes",
      "id": "....",
      "attributes": {
        "seq_id": "...",
        "block_id": "...",
        "start": "...",
        "end": "...",
        "strand": "....",
        "source": "...."
      }
    }
  ]
}

GET /dashboard/genomes/{taxon_id}/pseudogenes - Information about pseudogenes.

{
  "data": [
    {
      "type": "pseudogenes",
      "id": "....",
      "attributes": {
        "seq_id": "...",
        "block_id": "...",
        "start": "...",
        "end": "...",
        "strand": "....",
        "source": "...."
      }
    }
  ]
}

The taxon ID for D.discoideum is 44689.

Documentation

Index

• Constants
• func GFF3GenericConsumer(st Storage, key, t string, in <-chan string) (<-chan error, error)
• func GFF3LineProducer(r io.Reader) (<-chan string, <-chan error, error)
• func GFF3RegionConsumer(st Storage, key, t string, in <-chan string) (<-chan error, error)
• func GFF3Splitter(in <-chan string, len int) ([]chan string, <-chan error, error)
• func Handler(event functions.Event, ctx functions.Context) (string, error)
• func JSONAPIError(err error) (string, int, error)
• func MergeErrors(cs ...<-chan error) <-chan error
• func WaitForPipeline(errs ...<-chan error) error
• type GFF3Consumer
• type MetaData
• type RedisStorage
  • func (r *RedisStorage) ClearAll(prefix string) error
  • func (r *RedisStorage) Close() error
  • func (r *RedisStorage) Delete(key string, fields ...string) error
  • func (r *RedisStorage) Get(key, field string) (string, error)
  • func (r *RedisStorage) IsExist(key, field string) bool
  • func (r *RedisStorage) Set(key, field, val string) error
• type Storage
  • func NewRedisStorage(master, slave string) Storage

Constants

const (
	KEY_PREFIX = "dashboard"
)

Functions

func GFF3GenericConsumer

func GFF3GenericConsumer(st Storage, key, t string, in <-chan string) (<-chan error, error)

func GFF3LineProducer

func GFF3LineProducer(r io.Reader) (<-chan string, <-chan error, error)

func GFF3RegionConsumer

func GFF3RegionConsumer(st Storage, key, t string, in <-chan string) (<-chan error, error)

func GFF3Splitter

func GFF3Splitter(in <-chan string, len int) ([]chan string, <-chan error, error)

func Handler

func Handler(event functions.Event, ctx functions.Context) (string, error)

func JSONAPIError

func JSONAPIError(err error) (string, int, error)

JSONAPIError generate JSONAPI formatted http error from an error object

func MergeErrors

func MergeErrors(cs ...<-chan error) <-chan error

MergeErrors merges multiple channels of errors. Based on https://blog.golang.org/pipelines.

func WaitForPipeline

func WaitForPipeline(errs ...<-chan error) error

WaitForPipeline waits for results from all error channels. It returns early on the first error.

Types

type GFF3Consumer

type GFF3Consumer func(Storage, string, string, <-chan string) (<-chan error, error)

type MetaData

type MetaData struct {
	TaxonId    string `json:"taxon_id"`
	SciName    string `json:"scientific_name,omitempty"`
	CommonName string `json:"common_name,omitempty"`
	Rank       string `json:"rank,omitempty"`
	Bucket     string `json:"bucket"`
	File       string `json:"file"`
}

type RedisStorage

type RedisStorage struct {
	// contains filtered or unexported fields
}

func (*RedisStorage) ClearAll

func (r *RedisStorage) ClearAll(prefix string) error

func (*RedisStorage) Close

func (r *RedisStorage) Close() error

func (*RedisStorage) Delete

func (r *RedisStorage) Delete(key string, fields ...string) error

func (*RedisStorage) Get

func (r *RedisStorage) Get(key, field string) (string, error)

func (*RedisStorage) IsExist

func (r *RedisStorage) IsExist(key, field string) bool

func (*RedisStorage) Set

func (r *RedisStorage) Set(key, field, val string) error

type Storage

type Storage interface {
	Get(string, string) (string, error)
	Set(string, string, string) error
	Delete(string, ...string) error
	IsExist(string, string) bool
	Close() error
}

func NewRedisStorage

func NewRedisStorage(master, slave string) Storage