seed-silo

README

= SILO API

image:https://badges.gitter.im/ngageoint/seed.svg[link="https://gitter.im/ngageoint/seed?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"]
image:https://travis-ci.org/ngageoint/seed-silo.svg?branch=master[link="https://travis-ci.org/ngageoint/seed-silo"]

SILO (Seed Images Location Operation) is a REST API provided by the Seed team for discovering Seed images.  The API
allows users to scan one or more repositories for seed images and then search the resulting images and their manifests
by keywords.  Frameworks such as Scale can use these endpoints to assist users in finding algorithm images and creating
jobs out of them.

== Install

To build/install the SILO webapp on linux:

----
Install go, docker and httpd packages
yum install go
yum install docker
yum install httpd
go get github.com/ngageoint/seed-silo
~/go/src/github.com/ngageoint/seed-silo/install-silo.sh
----

Or even better, run SILO from a https://hub.docker.com/r/geoint/seed-silo[seed-silo container]:

----
docker run -p 9000:9000 docker.io/geoint/seed-silo:1.2.1
----

SILO runs on port 9000 so that port needs to be opened to the docker host.  It could be changed to port 80 if desired
with the flag '-p 80:9000'.  The command above is all that is needed, but if customization is desired, the following
environment variables can be passed in:


.Environment Variables
[cols="1,4"]
|===
|Name |Description

|SILO_ADMIN
|Specifies the username of the default admin. If not specified, this defaults to admin.

|SILO_ADMIN_PASSWORD
|Specifies the password of the default admin. If not specified, the default can be found in the codebase.

|DATABASE_URL
|Specifies the connection parameters for an existing database to use. This database should have a silo schema.
 If such a schema does not exist, it will be created. The environment variable follows the syntax outlined at
 https://github.com/kennethreitz/dj-database-url#url-schema[dj-database-url project]. In short, it follows the form:

 postgis://user:password@host[:port]/name


|SILO_LITE_PATH
|Specifies the path to use for a SQLite database if a postgres url is not specified via DATABASE_URL. If
 neither it nor DATABASE_URL is set the path /usr/silo/seed-silo.db is used.  A database will be created inside the container
 if needed or a database can be mounted into the container at the path for persistence between runs.
|===

== Usage

=== Registry

Registries can be added, deleted and scanned. A registry consists of a name, url, organization (optional), username (optional),
and password (optional).

==== Get Registry

Retrieves a registry

[cols="h,5a"]
|===
| URL
| /registries/{id}

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: {"ID":1,"Name":"dockerhub","Url":"https://hub.docker.com","Org":"geointseed","Username":"","Password":""}

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No registry found with that ID" }

|Sample Call
| curl -X "GET" http://localhost:9000/registries/1
|===

==== Add Registry

Adds a registry to the list of registries to be scanned.  An error will be returned and the registry won't be added if
the daemon is unable to connect to the registry.

[cols="h,5a"]
|===
| URL
| /registries/add

| Method
| POST

| URL Params
| None

| Data Params
| {"name":"localhost", "url":"https://localhost:5000", "org":"", "username":"testuser", "password": "testpassword"}

| Success Response
|       Code: 201 +
       Content: {}

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Unable to connect to registry" } +
        Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" }
        Code: 422 Unprocessable Entity +
        Content: { error : "Error unmarshalling json. " }

|Sample Call
| curl -H "Authorization: Token <token>" -H "Content-Type: application/json" -d '{"name":"localhost", "url":"https://localhost:5000", "org":"", "username":"testuser", "password": "testpassword"}' http://localhost:9000/registries/add
|===

==== Delete Registry

Removes a registry from the list of registries along with all images associated with that registry.

[cols="h,5a"]
|===
| URL
| /registries/delete/{id}

| Method
| DELETE

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: { }

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" }

|Sample Call
| curl -H "Authorization: Token <token>" -X "DELETE" http://localhost:9000/registries/delete/1
|===

==== Scan Registries

Removes all existing image entries, scans all registries for seed images and adds them to the database.
Requires admin authorization token

[cols="h,5a"]
|===
| URL
| /registries/scan

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 202 +
        Content: { } +
        Code: 202 +
        Content: {"message":"Scanning Registries"}

|Error Response
|       Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" }

|Sample Call
| curl -H "Authorization: Token <token>" "https://localhost:9000/registries/scan"
|===

==== Scan Registry

Removes all existing image entries, scans all registries for seed images and adds them to the database.
Requires admin authorization token

[cols="h,5a"]
|===
| URL
| /registries/{id}/scan

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 202 +
        Content: { } +
        Code: 202 +
        Content: {"message":"Scanning Registries"}

|Error Response
|       Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" }

|Sample Call
| curl -H "Authorization: Token <token>" "https://localhost:9000/registries/1/scan"
|===

==== List Registries

Retrieves all of the registries that have been successfully added

[cols="h,5a"]
|===
| URL
| /registries

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
                   { +
                     "ID": 1, +
                     "Name": "localhost", +
                     "Url": "https://localhost:5000", +
                     "Org": "", +
                     "Username": "", +
                     "Password": "" +
                   } +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/registries"
|===

=== Image

Images are added/removed by scanning registries. An image consists of a name, registry, organization (optional), and the
Seed manifest.

==== List Images

Retrieves all of the Seed images that have been scanned from registries

[cols="h,5a"]
|===
| URL
| /images

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "RegistryId": 1, +
    "Name": "my-job-0.1.0-seed:0.1.0", +
    "Registry": "docker.io", +
    "Org": "geointseed", +
    "JobName": "my-job", +
    "Title": "My first job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
    "JobVersion": "0.1.0", +
    "PackageVersion": "0.1.0" +
  }, +
  { +
    "ID": 2, +
    "RegistryId": 1, +
    "Name": "extractor-0.1.0-seed:0.1.0", +
    "Registry": "docker.io", +
    "Org": "geointseed", +
    "JobName": "extractor", +
    "Title": "Extractor", +
    "Maintainer": "John Tobe", +
    "Email": "jtobe@example.com", +
    "MaintOrg": "", +
    "Description": "Read's a zip file and extracts the contents", +
    "JobVersion": "0.1.0", +
    "PackageVersion": "0.1.0" +
  }, +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/images"
|===

==== Search Images

Searches the Seed images that have been scanned from registries and returns images matching the given query.  Images are
returned if the name, organization or manifest strings match the given query.

[cols="h,5a"]
|===
| URL
| /images/search/{query}

| Method
| GET

| URL Params
| query = string

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "RegistryId": 1, +
    "Name": "my-job-0.1.0-seed:0.1.0", +
    "Registry": "docker.io", +
    "Org": "geointseed", +
    "JobName": "my-job", +
    "Title": "My first job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
    "JobVersion": "0.1.0", +
    "PackageVersion": "0.1.0" +
  }, +
  { +
    "ID": 2, +
    "RegistryId": 1, +
    "Name": "extractor-0.1.0-seed:0.1.0", +
    "Registry": "docker.io", +
    "Org": "geointseed", +
    "JobName": "extractor", +
    "Title": "Extractor", +
    "Maintainer": "John Tobe", +
    "Email": "jtobe@example.com", +
    "MaintOrg": "", +
    "Description": "Read's a zip file and extracts the contents", +
    "JobVersion": "0.1.0", +
    "PackageVersion": "0.1.0" +
  }, +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/images/search/test"
|===

==== Get Image

Retrieves an image

[cols="h,5a"]
|===
| URL
| /images/{id}

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: +
                   { +
  "ID": 1, +
  "RegistryId": 1, +
  "JobId": 1, +
  "JobVersionId": 1, +
  "FullName": "my-job-0.1.0-seed:0.1.0", +
  "ShortName": "my-job", +
  "Title": "My first job", +
  "Maintainer": "John Doe", +
  "Email": "jdoe@example.com", +
  "MaintOrg": "E-corp", +
  "JobVersion": "0.1.0", +
  "PackageVersion": "0.1.0", +
  "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
  "Registry": "docker.io", +
  "Org": "geointseed", +
                     "Manifest": "{\"seedVersion\":\"0.1.0\",\"job\":{\"name\":\"my-job\",...}}" +
                      <full seed json> link:seed.manifest.json[sample manifest] +
                   }

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No image found with that ID" }

|Sample Call
| curl -X "GET" http://localhost:9000/images/1
|===

==== Image Manifest

Returns the Seed manifest json for the given image id.

[cols="h,5a"]
|===
| URL
| /images/{id}/manifest

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: link:seed.manifest.json[sample manifest]

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No image found with that ID" }

|Sample Call
| curl "https://localhost:9000/images/1/manifest"
|===

=== Job

Jobs are groups of images with the same job name.  A job has a name, title, maintainer, email, organization, description,
latest job version and latest package version.  It also has a list of images and job versions.

==== List Jobs

Retrieves all of the Jobs that have been scanned from registries

[cols="h,5a"]
|===
| URL
| /jobs

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "Name": "my-job", +
    "LatestJobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Title": "My first job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
    "ImageIDs": [0, 1, 2], +
    "JobVersions": [{JobVersion struct}, {JobVersion struct}...] +
  }, +
  { +
    "ID": 2, +
    "Name": "another-job", +
    "LatestJobVersion": "1.0.0", +
    "LatestPackageVersion": "1.0.0", +
    "Title": "My second job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "blah blah blah", +
    "ImageIDs": [3], +
    "JobVersions": [{JobVersion struct}] +
  }, +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/jobs"
|===

==== Search Jobs

Searches the Seed images that have been scanned from registries and returns jobs for the images matching the given query.  Images are
returned if the name, organization or seed manifest match the given query.  Images/job versions that are irrelevant to the query
are omitted from the ImageIDs and JobVersions structures.

[cols="h,5a"]
|===
| URL
| /jobs/search/{query}

| Method
| GET

| URL Params
| query = string

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "Name": "my-job", +
    "LatestJobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Title": "My first job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
    "ImageIDs": [0, 2], +
    "JobVersions": [{JobVersion struct}, {JobVersion struct}...] +
  }, +
  { +
    "ID": 2, +
    "Name": "another-job", +
    "LatestJobVersion": "1.0.0", +
    "LatestPackageVersion": "1.0.0", +
    "Title": "My second job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "blah blah blah", +
    "ImageIDs": [3], +
    "JobVersions": [{JobVersion struct}] +
  }, +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/jobs/search/test"
|===

==== Get Job

Retrieves a job

[cols="h,5a"]
|===
| URL
| /jobs/{id}

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: +
{ +
    "ID": 1, +
    "Name": "my-job", +
    "LatestJobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Title": "My first job", +
    "Maintainer": "John Doe", +
    "Email": "jdoe@example.com", +
    "MaintOrg": "E-corp", +
    "Description": "Reads an HDF5 file and outputs two TIFF images, a CSV and manifest containing cell_count", +
    "ImageIDs": [0, 2], +
    "JobVersions": [{JobVersion struct}, {JobVersion struct}...] +
  }

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No job found with that ID" }

|Sample Call
| curl -X "GET" http://localhost:9000/jobs/1
|===

=== Job Version

Job Versions are groups of images with the same job name and the same job version.  A job version has a job name, job id,
job version, latest package version and a list of images.

==== List Job Versions

Retrieves all of the Job Versions that have been scanned from registries

[cols="h,5a"]
|===
| URL
| /job-versions

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "JobName": "my-job", +
    "JobId": "1", +
    "JobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Images": [{Image struct}, {Image struct}...] +
  }, +
  { +
    "ID": 2, +
    "JobName": "my-job", +
    "JobId": "1", +
    "JobVersion": "0.2.0", +
    "LatestPackageVersion": "0.2.0", +
    "Images": [{Image struct}] +
  }, +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/job-versions"
|===

==== Get Job Versions

Returns the job versions for a specific job

[cols="h,5a"]
|===
| URL
| /jobs/{id}/job-versions

| Method
| GET

| URL Params
| id = int

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
{ +
    "ID": 1, +
    "JobName": "my-job", +
    "JobId": "1", +
    "JobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Images": [{Image struct}, {Image struct}...] +
  }, +
  { +
    "ID": 2, +
    "JobName": "my-job", +
    "JobId": "1", +
    "JobVersion": "0.2.0", +
    "LatestPackageVersion": "0.2.0", +
    "Images": [{Image struct}] +
  }, +
                 ]

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No job found with that ID" }

|Sample Call
| curl "https://localhost:9000/jobs/1/job-versions"
|===

==== Get Job Version

Retrieves a job version

[cols="h,5a"]
|===
| URL
| /job-versions/{id}

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: +
{ +
    "ID": 1, +
    "JobName": "my-job", +
    "JobId": "1", +
    "JobVersion": "0.1.0", +
    "LatestPackageVersion": "0.1.0", +
    "Images": [{Image struct}, {Image struct}...] +
  }

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No job version found with that ID" }

|Sample Call
| curl -X "GET" http://localhost:9000/jobs/1
|===

=== User

Users can be added, deleted, listed and used to login. A user consists of a username, password, and a role.

==== Get User

Retrieves a user

[cols="h,5a"]
|===
| URL
| /users/{id}

| Method
| GET

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: {"ID":1,"username":"admin","role":"admin"}

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 404 File not found +
        Content: { error : "No user found with that ID" }

|Sample Call
| curl -X "GET" http://localhost:9000/user/1
|===

==== Add User

Adds a user to the system.  Requires a valid token from an admin user.

[cols="h,5a"]
|===
| URL
| /users/add

| Method
| POST

| URL Params
| None

| Data Params
| {"username":"admin", "password": "hunter17", "role": "admin"}

| Success Response
|      Code: 201 +
       Content: {"username":"admin", "password": "hunter17", "role": "admin"}

|Error Response
|       Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" } +
        Code: 422 Unprocessable Entity +
        Content: { error : "Error unmarshalling json. " }

|Sample Call
|curl -H "Content-Type: application/json" -d '{"username":"admin", "password": "hunter17", "role": "admin"}' -H "Authorization: Token <token>" http://localhost:9000/users/add
|===

==== Delete User

Removes a user from the system.  Requires a valid token from an admin user.

[cols="h,5a"]
|===
| URL
| /users/delete/{id}

| Method
| DELETE

| URL Params
| id = integer

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: { }

|Error Response
|       Code: 400 Bad Request +
        Content: { error : "Invalid ID" } +
        Code: 401 Unauthorized +
        Content: { error : "Invalid authorization token" } or { error : "Missing authorization token" } +
        Code: 403 Forbidden +
        Content: { error : "User does not have permission to perform this action" }

|Sample Call
| curl -X "DELETE" -H "Authorization: Token <token>" http://localhost:9000/users/delete/1
|===

==== List Users

Retrieves all of the users in the system

[cols="h,5a"]
|===
| URL
| /users

| Method
| GET

| URL Params
| None

| Data Params
| None

| Success Response
|       Code: 200 +
        Content: [ +
                   { +
                     "ID": 1, +
                     "username": "admin", +
                     "role": "admin" +
                   }, +
                   { +
                     "ID": 2, +
                     "username": "user", +
                     "role": "user" +
                   } +
                 ]

|Error Response
|       None

|Sample Call
| curl "https://localhost:9000/users"
|===

==== Login

Authenticates a user and returns a token to be used in subsequent api calls

[cols="h,5a"]
|===
| URL
| /login

| Method
| GET

| URL Params
| None

| Data Params
| {"username":"admin", "password": "password"}

| Success Response
|       Code: 200 +
        Content: {"token":"<token>"}

|Error Response
|       Code: 401 Unauthorized +
        Content: { error : "Invalid login" } +
        Code: 422 Unprocessable Entity +
        Content: { error : "Error unmarshalling json. " }

|Sample Call
| curl -H "Content-Type: application/json" -d '{"username":"admin", "password": "password"}' "http://localhost:9000/login"
|===