secops

README

SecOps Application

The Security Operations (SecOps) server will be the centeral part of maintaining a Vulnerability DataBase (VDB). The SecOps server will provide REST interface to query and update VDB. The VDB will model potential vulnerability detectors like image and real time scan, endpoint or network detection. The VDB can be used also to manage vulnerability protection using elements like WAF and alerts from them. The VDB data model capture security events and vulnerabilities so that we can detect, respond, contain and recover from incidents.

The VDB knowledge base for vulnerabilities could be from many sources, we will start by using National Vulnerability Database (NVD) which is central to image scans. The knowledge base area is complex solution or detection leading to additional databases as you see here the NVD CVE-2016-3382 was replicated by Snort as they develop rules to detect it using Sid-1-40404.

Getting Started

These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. See deployment for notes on how to deploy the project on a live system.

Design

I started using the atlas CLI compared the application to what you have for the contacts-app and decided it was better to create a template using Atlas CLI and remove the proto and stubs and the zserver implementation and replace them with the contacts-app pieces. This gave me the best starting point to start a new application.

Prerequisites

Install go dep

go get -u github.com/golang/dep/cmd/dep

• helm
• kubectl
• minikube
• postgres (pg_ctl, createdb, dropdb)
• go-migrate (migrate)

Database

Design

The process of designing the data model for the application could be iterative and maybe visual tools. I created a sample ./model directory here with a make target for generating ERD:

make erd
Create file://doc/db/out.html and file://doc/db/out.pdf

Which creates targets e.g. secops ERD that you can view.

We can now start to design the proto definiton of our data model using the above model defintion for the messsage and service definitions. Refer to the proto buff defintion for database defintion which is layered on top of Gorm.

We would build out the data model incrementaly starting with the top node in our case the Region. Then add the necessary migration for it, see below for more detail on database migration below.

Database Migration

For migrating the database schema, golang-migrate framework is used.

Steps for local development

1. The migration files must be placed ./db/migrations/ directory.
2. Make sure you have migrate CLI 3a. Make sure you have Postgres Running

/usr/local/Cellar/postgresql/11.5_1/bin/pg_ctl -D /usr/local/var/postgres -l logfile start

3b. Run migration

dropdb secops
createdb secops
make migrate-up

Development

Local development setup

Table creation should be done manually by running the migrations scripts or following the steps defined in database migration section. Scripts can be found at ./db/migrations/

Create vendor directory with required golang packages

make vendor

dropdb secops
createdb secops
make migrate-up

make protobuf
go run ./cmd/server/*.go

Here is the sample of all the APIs ....

curl http://localhost:8080/v1/version
export JWT="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBY2NvdW50SUQiOjF9.GsXyFDDARjXe1t9DPo2LIBKHEal3O7t3vLI3edA7dGU"
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/cloud_providers -d '{"name": "aws-dev", "description": "AWS account for dev, qa and preprod", "provider":1, "account":"43509870"}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/regions -d '{"name": "us-east-1", "description": "us-east-1 for dev, qa and preprod", "cloud_provider_id": "secops-app/cloud_providers/1"}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/vpcs -d '{"name": "dev vpcs", "description": "dev vpc environment", "vpc_id": "vpc-1a2b3c4d", "cidr_block": "172.31.32.0/20", "region_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/amis -d '{"name": "secop ami", "description": "ami to run secop server", "ami_id": "ami-x0123456", "region_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/hosts -d '{"name": "app instance", "description": "application server", "ip": "172.31.32.19", "dns": "ec2-instance-connect.amazonaws.com", "instance_id": "i-90589002959303", "ami_id":1, "vpc_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/registries -d '{"name": "dockerhub", "description": "Public DockerHub Registry"}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/applications -d '{"name": "cmdb server", "description": "CMDB server application", "repo":"https://github.com/seizadi/cmdb"}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/containers -d '{"name": "cmdb service", "description": "CMDB microservice", "image_repo": "soheileizadi/cmdb-server", "registry_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/container_tags -d '{"image_repo": "soheileizadi/cmdb-server", "image_tag": "latest", "digest": "sha256:e80faab6b05b025e2798a776df408b61f1c813f7fef5ca0bf487e3962888e417", "container_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/containers -d '{"name": "Postrgres", "description": "Postgres Database", "image_repo": "postgres", "registry_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/container_tags -d '{"image_repo": "postgres", "image_tag": "latest", "digest": "sha256:f156a3169c9117f395ece786248c95391f6ba9cca655e695aa42505369adb8da", "container_id":2}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/application_versions -d '{"version": "v1.1", "container_tags_list":["secops-app/container_tags/1", "secops-app/container_tags/2"], "application_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/application_versions -d '{"version": "v1.2", "container_tags_list":["secops-app/container_tags/1", "secops-app/container_tags/2"], "application_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/application_versions -d '{"version": "v1.3", "container_tags_list":["secops-app/container_tags/1", "secops-app/container_tags/2"], "application_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/events -d '{"name": "cmdb-scan", "description": "Scan of CMDB Server", "event_time": "2019-11-14T22:25:52+00:00", "event_type": "clair", "event_job":{}, "container_tag_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/knowledge_bases -d '{"name": "CVE-2019-14697", "link": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14697", "severity": "Unknown", "name_space": "alpine:v3.8", "pkg": "musl", "fixed_version": "1.1.19-r11"}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/vulnerabilities -d '{"name": "soheileizadi/cmdb-server/latest/CVE-2019-14697", "severity": "Unknown", "name_space": "alpine:v3.8", "pkg": "musl", "installed_version": "1.1.19-r10", "state": [{"time":"2019-11-14T22:25:52+00:00", "status": "Detect"}], "ticket_link": "", "knowledge_base_id":1, "event_id":1}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/events -d '{"name": "cmdb-scan", "description": "Scan of CMDB Server", "event_time": "2019-11-15T21:27:42+00:00", "event_type": "clair", "event_job":{}, "container_tag_id":1}'
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/vulnerabilities -d '{"name": "soheileizadi/cmdb-server/latest/CVE-2019-14697", "severity": "Unknown", "name_space": "alpine:v3.8", "pkg": "musl", "installed_version": "1.1.19-r10", "state": [{"time":"2019-11-14T22:25:52+00:00", "status": "Detect"}], "ticket_link": "", "knowledge_base_id":1, "event_id":2}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/events -d '{"name": "cmdb-scan", "description": "Scan of CMDB Server", "event_time": "2019-11-16T20:35:55+00:00", "event_type": "clair", "event_job":{}, "container_tag_id":1}'
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/vulnerabilities -d '{"name": "soheileizadi/cmdb-server/latest/CVE-2019-14697", "severity": "Unknown", "name_space": "alpine:v3.8", "pkg": "musl", "installed_version": "1.1.19-r10", "state": [{"time":"2019-11-14T22:25:52+00:00", "status": "Detect"}], "ticket_link": "", "knowledge_base_id":1, "event_id":3}' | jq
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/events -d '{"name": "cmdb-scan", "description": "Scan of CMDB Server", "event_time": "2019-11-17T21:15:32+00:00", "event_type": "clair", "event_job":{}, "container_tag_id":1}'
curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/vulnerabilities -d '{"name": "soheileizadi/cmdb-server/latest/CVE-2019-14697", "severity": "Unknown", "name_space": "alpine:v3.8", "pkg": "musl", "installed_version": "1.1.19-r10", "state": [{"time":"2019-11-14T22:25:52+00:00", "status": "Detect"}], "ticket_link": "", "knowledge_base_id":1, "event_id":4}' | jq

curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/applications | jq

Build docker images

make

If this process finished with errors it's likely that docker doesn't allow to mount host directory in its container. Therefore you are proposed to run su -c "setenforce 0" command to fix this issue.

Local Kubernetes setup

Deployment

To deploy secops on minikube use

cd repo/secops
helm dependency update .
helm install -f minikube.yaml .

If needed you can validate that the migration is applied:

kubectl get pods
NAME                                       READY     STATUS    RESTARTS   AGE
altered-swan-secops-7c48b7b76-jcfc2          1/1       Running   0          55s
altered-swan-postgresql-77f87976ff-s2fwk   1/1       Running   0          55s

kubectl exec -it altered-swan-postgresql-77f87976ff-s2fwk /bin/sh
# psql secops
psql (9.6.2)
Type "help" for help.

secops=# \dt
               List of relations
 Schema |       Name        | Type  |  Owner   
--------+-------------------+-------+----------
 public | applications      | table | postgres
...
(12 rows)

Usage

Try it out by executing following curl commands:

export JWT="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBY2NvdW50SUQiOjF9.GsXyFDDARjXe1t9DPo2LIBKHEal3O7t3vLI3edA7dGU"
curl http://minikube/secops/v1/version
curl -H "Authorization: Bearer $JWT" http://minikube/secops/v1/regions

For the example that used the localhost for local development the minikube usage is similar:

curl -H "Authorization: Bearer $JWT" http://minikube/secops/v1/cloud_providers -d '{"name": "aws-dev", "description": "AWS account for dev, qa and preprod", "provider":1, "account":"43509870"}'
curl -H "Authorization: Bearer $JWT" http://minikube/secops/v1/regions -d '{"name": "us-east-1", "description": "us-east-1 for dev, qa and preprod", "cloud_provider_id": "secops-app/cloud_providers/1"}'

API documentation

API documentation in k8s deployment could be found on following link, note that no credentials needed to access it:

open http://minikube/secops/apidoc

The Swagger JSON file here:

curl http://minikube/secops/swagger

Testing

The test environment will require a source of vulnerability detection, we will use Clair Server to scan docker images. We will use a (helm chart)[https://hub.helm.sh/charts/banzaicloud-stable/clair]to stand up the server.

helm repo add banzaicloud-stable https://kubernetes-charts.banzaicloud.com
helm install banzaicloud-stable/clair --version 0.1.2

This helm chart did not work on minikube see my debug log below, I got it working forking the chart, so from here I assume you either use my fork or that the patch I submit will be used to update it.

To make sure that the installation worked you can check:

$ helm list
NAME                  	REVISION	UPDATED                 	STATUS  	CHART      	APP VERSION	NAMESPACE
invincible-grasshopper	1       	Mon Nov 11 22:38:18 2019	DEPLOYED	clair-0.1.2	v2.0.9     	default  

$ kubectl get pods
NAME                                           READY   STATUS    RESTARTS   AGE
invincible-grasshopper-clair-64cb64c8c-qfhfq   1/1     Running   2          111s
invincible-grasshopper-postgresql-0            1/1     Running   0          111s

$ kubectl logs invincible-grasshopper-clair-64cb64c8c-qfhfq
{"Event":"running database migrations","Level":"info","Location":"pgsql.go:216","Time":"2019-11-12 06:38:41.247774"}
{"Event":"database migration ran successfully","Level":"info","Location":"pgsql.go:223","Time":"2019-11-12 06:38:41.345852"}
{"Event":"sender configured","Level":"info","Location":"notifier.go:66","Time":"2019-11-12 06:38:41.346291","sender name":"webhook"}
{"Event":"notifier service started","Level":"info","Location":"notifier.go:82","Time":"2019-11-12 06:38:41.346445","lock identifier":"cd9c3eff-8fd6-4e10-9b4c-4c9548578164"}
{"Event":"starting health API","Level":"info","Location":"api.go:85","Time":"2019-11-12 06:38:41.346326","port":6061}
{"Event":"starting main API","Level":"info","Location":"api.go:52","Time":"2019-11-12 06:38:41.347372","port":6060}
{"Event":"updater service started","Level":"info","Location":"updater.go:83","Time":"2019-11-12 06:38:41.347820","lock identifier":"81d5e865-6f69-4ab0-b666-bf27af5c3029"}
{"Event":"updating vulnerabilities","Level":"info","Location":"updater.go:192","Time":"2019-11-12 06:38:41.355885"}
{"Event":"fetching vulnerability updates","Level":"info","Location":"updater.go:239","Time":"2019-11-12 06:38:41.356183"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"amzn.go:84","Time":"2019-11-12 06:38:41.356322","package":"Amazon Linux 2018.03"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"oracle.go:119","Time":"2019-11-12 06:38:41.356447","package":"Oracle Linux"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"rhel.go:92","Time":"2019-11-12 06:38:41.356983","package":"RHEL"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"amzn.go:84","Time":"2019-11-12 06:38:41.357645","package":"Amazon Linux 2"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"ubuntu.go:85","Time":"2019-11-12 06:38:41.357660","package":"Ubuntu"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"alpine.go:52","Time":"2019-11-12 06:38:41.358615","package":"Alpine"}
{"Event":"Start fetching vulnerabilities","Level":"info","Location":"debian.go:63","Time":"2019-11-12 06:38:41.359239","package":"Debian"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2019-11-12 06:38:44.948141","updater name":"alpine"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2019-11-12 06:38:47.040369","updater name":"amzn2"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2019-11-12 06:38:51.149659","updater name":"amzn1"}
{"Event":"finished fetching","Level":"info","Location":"updater.go:253","Time":"2019-11-12 06:39:01.846013","updater name":"debian"}

$ kubectl get services
NAME                                         TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                         AGE
invincible-grasshopper-clair                 LoadBalancer   10.101.234.143   <pending>     6060:31437/TCP,6061:31076/TCP   13h
invincible-grasshopper-postgresql            ClusterIP      10.100.69.67     <none>        5432/TCP                        13h
invincible-grasshopper-postgresql-headless   ClusterIP      None             <none>        5432/TCP                        13h
kubernetes                                   ClusterIP      10.96.0.1        <none>        443/TCP                         19h

$ minikube service invincible-grasshopper-clair
|-----------|------------------------------|-----------------------|--------------------------------|
| NAMESPACE |             NAME             |      TARGET PORT      |              URL               |
|-----------|------------------------------|-----------------------|--------------------------------|
| default   | invincible-grasshopper-clair | clair-api             | http://192.168.64.3:31437      |
|           |                              | clair-health          | http://192.168.64.3:31076      |
|-----------|------------------------------|-----------------------|--------------------------------|

To talk to Clair Server:

$ curl http://192.168.64.3:31437/v1/namespaces | jq
{
  "Namespaces": [
    {
      "Name": "alpine:v3.10",
      "VersionFormat": "dpkg"
    },
    {
      "Name": "alpine:v3.3",
      "VersionFormat": "dpkg"
    },
    {
      "Name": "alpine:v3.8",
      "VersionFormat": "dpkg"
    },
....
    {
      "Name": "ubuntu:18.10",
      "VersionFormat": "dpkg"
    },
    {
      "Name": "ubuntu:19.04",
      "VersionFormat": "dpkg"
    }
  ]
}

To scan an image you can find a Clair client, there are two popular ones, clair-scanner and klar.

The Clair clients send Clair server each layer of a Docker image to be scanned using the scanning APIs. After all layers are scanned, the clients request scan results from the Clair server. You look at the returned vulnerabilities. There are features to provide white list to avoid vulnerabilities that are approved.

The main difference between clair-scanner and klar, is how they interact with the docker registry. The klar client does not download container images, but it does pass docker auth token to Clair for retrieving layers, so the layers are kept in the registry. The clair-scanner downloads images and starts up an http server serving the layers locally. If you need to scan images before pushing them into a registry the clair-scanner will be your best choice.

The klar client is simpler and active in terms of github repo comits. It is used in our testing below.

$ go get github.com/optiopay/klar
$ CLAIR_ADDR=192.168.64.3:31437 klar alpine
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
Analysing 1 layers
Got results from Clair API v1
Found 0 vulnerabilities

Here is one that returns many (689) vulnerabilities showing the importance of thresholds and whitelist in doing scanning. Supported severity levels by Clair server are: Unknown, Negligible, Low, Medium, High, Critical, Defcon1.

$ CLAIR_ADDR=192.168.64.3:31437 klar node:10
clair timeout 1m0s
docker timeout: 1m0s
no whitelist file
Analysing 9 layers
Got results from Clair API v1
Found 689 vulnerabilities
Unknown: 196
Negligible: 329
Low: 162
Medium: 2

CVE-2018-5710: [Unknown] 
Found in: krb5 [1.15-1+deb9u1]
Fixed By: 
An issue was discovered in MIT Kerberos 5 (aka krb5) through 1.16. The pre-defined function "strlen" is getting a "NULL" string as a parameter value in plugins/kdb/ldap/libkdb_ldap/ldap_principal2.c in the Key Distribution Center (KDC), which allows remote authenticated users to cause a denial of service (NULL pointer dereference) via a modified kadmin client.
https://security-tracker.debian.org/tracker/CVE-2018-5710
-----------------------------------------
CVE-2018-5729: [Unknown] 
Found in: krb5 [1.15-1+deb9u1]
Fixed By: 
MIT krb5 1.6 or later allows an authenticated kadmin with permission to add principals to an LDAP Kerberos database to cause a denial of service (NULL pointer dereference) or bypass a DN container check by supplying tagged data that is internal to the database module.
https://security-tracker.debian.org/tracker/CVE-2018-5729
-----------------------------------------
.....
-----------------------------------------
CVE-2017-7475: [Low] 
Found in: cairo [1.14.8-1]
Fixed By: 
Cairo version 1.15.4 is vulnerable to a NULL pointer dereference related to the FT_Load_Glyph and FT_Render_Glyph resulting in an application crash.
https://security-tracker.debian.org/tracker/CVE-2017-7475
-----------------------------------------
CVE-2009-3546: [Medium] 
Found in: libwmf [0.2.8.4-10.6]
Fixed By: 
The _gdGetColors function in gd_gd.c in PHP 5.2.11 and 5.3.x before 5.3.1, and the GD Graphics Library 2.x, does not properly verify a certain colorsTotal structure member, which might allow remote attackers to conduct buffer overflow or buffer over-read attacks via a crafted GD file, a different vulnerability than CVE-2009-3293. NOTE: some of these details are obtained from third party information.
https://security-tracker.debian.org/tracker/CVE-2009-3546
-----------------------------------------
CVE-2007-3996: [Medium] 
Found in: libwmf [0.2.8.4-10.6]
Fixed By: 
Multiple integer overflows in libgd in PHP before 5.2.4 allow remote attackers to cause a denial of service (application crash) and possibly execute arbitrary code via a large (1) srcW or (2) srcH value to the (a) gdImageCopyResized function, or a large (3) sy (height) or (4) sx (width) value to the (b) gdImageCreate or the (c) gdImageCreateTrueColor function.
https://security-tracker.debian.org/tracker/CVE-2007-3996
-----------------------------------------

$ CLAIR_ADDR=192.168.64.3:31437 FORMAT_OUTPUT=json klar soheileizadi/cmdb-server | jq
{
  "LayerCount": 6,
  "Vulnerabilities": {
    "Unknown": [
      {
        "Name": "CVE-2019-14697",
        "NamespaceName": "alpine:v3.8",
        "Link": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14697",
        "Severity": "Unknown",
        "FixedBy": "1.1.19-r11",
        "FeatureName": "musl",
        "FeatureVersion": "1.1.19-r10"
      }
    ]
  }
}

Versioning

We use SemVer for versioning. For the versions available, see the tags on this repository.

Debugging

Clair Helm Probelms

Tried to run Clair helm and had issues...

$ helm install banzaicloud-stable/clair --version 0.1.2
Error: validation failed: unable to recognize "": no matches for kind "Deployment" in version "extensions/v1beta1"

It looks like this problem with deprecated kubernetes APIs. To fix forked the chart and fixed the problem and tested it locally:

$ pwd
/Users/seizadi/projects/banzai-charts/clair
$ helm dependency update .
$ helm install . 

Protobuf problem

The toolkit used to create protobuf is containerized to help eliminate developers from having to download all the tools on their local machine. The problem is sometimes when you go to compile the image on your machine from the stubs you will run into issues.

So here is some tips on how to debug it. I started with looking at the source on container:

seizadi$ docker run -it --entrypoint /bin/sh infoblox/atlas-gentool:latest

Join Table Problem

Trying to figure out how to get join tables and associations to work with Gorm.

Setup join table:

CREATE TABLE application_versions_container_tags
(
    application_version_id int REFERENCES application_versions(id) ON DELETE CASCADE,
    container_tag_id int REFERENCES container_tags(id) ON DELETE CASCADE,
    primary key (application_version_id, container_tag_id)
);

You can use this approach:

In proto...

message ApplicationVersion {
  option (gorm.opts) = {
      ormable: true,
      multi_account: true
  };

  // The ApplicationVersion identifier.
  atlas.rpc.Identifier id = 1 [(gorm.field).tag = {type: "serial"  primary_key: true}];

  // The version of the Application
  string version = 2;

  repeated ContainerTag container_tags = 3 [(gorm.field).many_to_many = {jointable: "application_versions_container_tags"}];
}

message ContainerTag {
  option (gorm.opts) = {
      ormable: true,
      multi_account: true
  };

  // The ContainerTag identifier.
  atlas.rpc.Identifier id = 1 [(gorm.field).tag = {type: "serial"  primary_key: true}];

....
  // ApplicationVersions using this Image
  repeated ApplicationVersion application_versions = 5 [(gorm.field).many_to_many = {jointable: "application_versions_container_tags"}];

}

Here is the API call to create an association:

$ curl -H "Authorization: Bearer $JWT" -X PATCH http://localhost:8080/v1/application_versions/1 -d '{"version": "v1.2", "container_tags":[{"image_tag": "latest", "digest": "sha256:e80faab6b05b025e2798a776df408b61f1c813f7fef5ca0bf487e3962888e417", "container_id":1}]}'

Here are the API calls to view them:

$ curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/application_versions/1
{"result":{"id":"secops-app/application_versions/1","version":"v1.2","container_tags":[{"id":"secops-app/container_tags/2","image_tag":"latest","digest":"sha256:e80faab6b05b025e2798a776df408b61f1c813f7fef5ca0bf487e3962888e417","container_id":"secops-app/containers/1"}]}}

$ curl -H "Authorization: Bearer $JWT" http://localhost:8080/v1/container_tags
{"results":[{"id":"secops-app/container_tags/1","image_tag":"latest","digest":"sha256:e80faab6b05b025e2798a776df408b61f1c813f7fef5ca0bf487e3962888e417","container_id":"secops-app/containers/1"},{"id":"secops-app/container_tags/2","image_tag":"latest","digest":"sha256:e80faab6b05b025e2798a776df408b61f1c813f7fef5ca0bf487e3962888e417","container_id":"secops-app/containers/1","application_versions":[{"id":"secops-app/application_versions/1","version":"v1.2"}]}]}

The problem with this approach is that GORM is mananging the lifecycle of resources and adding/removing them from backend. If you want to reuse resources, you want to use references for association and manage the life cycle of resources through API. In order to perform this we need to connect to the Atlas GORM lifecycle hooks and provide code to support the above behavior.

Now the protobuf definition from above change to support this pattern:

message ApplicationVersion {
  option (gorm.opts) = {
      ormable: true,
      multi_account: true
      include: [{
          type: "[]*ContainerTagORM",
          name: "ContainerTagsList",
          tag: {
            many_to_many: "application_versions_container_tags",
            jointable_foreignkey: "application_version_id",
            association_jointable_foreignkey: "container_tag_id",
            association_autoupdate: false,
            association_autocreate: false
          }
      }]
  };

  // The ApplicationVersion identifier.
  atlas.rpc.Identifier id = 1 [(gorm.field).tag = {type: "serial"  primary_key: true}];

  // The version of the Application
  string version = 2;

  // ContainerTags associated with the ApplicationVersion
  repeated atlas.rpc.Identifier container_tags_list = 3 [(gorm.field).drop = true];

  // Images Linked to this instance of ApplicationVersion
  repeated ContainerTag container_tags = 4 [(gorm.field).many_to_many = {jointable: "application_versions_container_tags"}];
}


message ContainerTag {
  option (gorm.opts) = {
      ormable: true,
      multi_account: true
  };

  // The ContainerTag identifier.
  atlas.rpc.Identifier id = 1 [(gorm.field).tag = {type: "serial"  primary_key: true}];

....
  // ApplicationVersions using this Image
  repeated ApplicationVersion application_versions = 5 [(gorm.field).many_to_many = {jointable: "application_versions_container_tags"}];

}

Note that the changes are only to the ApplicationVersion resource, we could allow resource references from ContainerTag, but we just drive it from ApplicationVersion for now since the current use case the life cycle of these resources is driven by Application discovery.

The gorm.opts changes above:

      include: [{
          type: "[]*ContainerTagORM",
          name: "ContainerTagsList",
          tag: {
            many_to_many: "application_versions_container_tags",
            jointable_foreignkey: "application_version_id",
            association_jointable_foreignkey: "container_tag_id",
            association_autoupdate: false,
            association_autocreate: false
          }
      }]

Note this compile with a WARNING that can be ignored!

$ make protobuf
2019/12/12 22:20:29 WARNING: included field "ContainerTagsList" of type "[]*ContainerTagORM" is not a recognized special type, and no package specified. This type is assumed to be in the same package as the generated code

We add additional detail to the code generated the protobuf GORM plugin in secops.pb.gorm.go:

type ApplicationVersionORM struct {
	AccountID         string
	ApplicationId     *int64
	ContainerTags     []*ContainerTagORM `gorm:"foreignkey:Id;association_foreignkey:Id;many2many:application_versions_container_tags;jointable_foreignkey:application_version_id;association_jointable_foreignkey:container_tag_id"`
	ContainerTagsList []*ContainerTagORM `gorm:"many2many:application_versions_container_tags;jointable_foreignkey:application_version_id;association_jointable_foreignkey:container_tag_id;association_autoupdate:false;association_autocreate:false"`
	Id                int64              `gorm:"type:serial;primary_key"`
	Version           string
}

The definition for ContainerTagsList is driven from the gorm.opts:

	ContainerTagsList []*ContainerTagORM `gorm:"many2many:application_versions_container_tags;jointable_foreignkey:application_version_id;association_jointable_foreignkey:container_tag_id;association_autoupdate:false;association_autocreate:false"`

The 'gorm:' tags inform GORM how process the references, some of the parameters are related to how the join table schema which you see above, the other two parameters shown below, are critical to inform GORM that we will manage the resource association lifecycle and it should not automatically update/create resources.

    association_autoupdate: false,
    association_autocreate: false

The toolkit GORM generates lifecycle hooks for processing requests in secops.pb.gorm.go, we generate code for taking over the association lifecycle from GORM, this code is shown below:

// AfterToORM will marshall ContainerTag from request to ORM object
func (m *ApplicationVersion) AfterToORM(ctx context.Context, a *ApplicationVersionORM) error {
	for _, g := range m.ContainerTagsList {
		containerTagId, err := resource.DecodeInt64(&ContainerTag{}, g)
		if err != nil {
			return err
		}
		a.ContainerTagsList = append(a.ContainerTagsList, &ContainerTagORM{Id: containerTagId})
	}
	
	return nil
}

// AfterToPB copies the ContainerTag resource information from ORM object to PB object
func (m *ApplicationVersionORM) AfterToPB(ctx context.Context, a *ApplicationVersion) error {
	
	for _, ct := range m.ContainerTagsList {
		containerTagId, err := resource.Encode(&ContainerTag{}, ct.Id)
		if err != nil {
			return err
		}
		a.ContainerTagsList = append(a.ContainerTagsList, containerTagId)
	}
	return nil
}

func (m *ApplicationVersionORM) BeforeStrictUpdateSave(ctx context.Context, db *gorm.DB) (*gorm.DB, error) {
	if err := db.Model(m).Association("ContainerTags").Replace(m.ContainerTagsList).Error; err != nil {
		return nil, err
	}
	return db, nil
}

We will now show how the above code is called through the lifecycle hooks in toolkit GORM code. Lets take the case of create request, here is the high level flow and where the callouts. We have added // **** to draw attention to places where the callouts happen:

// DefaultCreateApplicationVersion executes a basic gorm create call
func DefaultCreateApplicationVersion(ctx context.Context, in *ApplicationVersion, db *gorm1.DB) (*ApplicationVersion, error) {
	if in == nil {
		return nil, errors1.NilArgumentError
	}
// **** Callout 1
	ormObj, err := in.ToORM(ctx)
	if err != nil {
		return nil, err
	}
	if hook, ok := interface{}(&ormObj).(ApplicationVersionORMWithBeforeCreate_); ok {
		if db, err = hook.BeforeCreate_(ctx, db); err != nil {
			return nil, err
		}
	}
	if err = db.Create(&ormObj).Error; err != nil {
		return nil, err
	}
	if hook, ok := interface{}(&ormObj).(ApplicationVersionORMWithAfterCreate_); ok {
		if err = hook.AfterCreate_(ctx, db); err != nil {
			return nil, err
		}
	}
// **** Callout 2
	pbResponse, err := ormObj.ToPB(ctx)
	return &pbResponse, err
}

Callout 1

The first callout gets called from posthook.AfterToORM(ctx, &to) shown below in code copied from secops.pb.gorm.go, will redirect call to func (m *ApplicationVersion) AfterToORM() and populates the ContainerTagsList from the client request.

// ToORM runs the BeforeToORM hook if present, converts the fields of this
// object to ORM format, runs the AfterToORM hook, then returns the ORM object
func (m *ApplicationVersion) ToORM(ctx context.Context) (ApplicationVersionORM, error) {
	to := ApplicationVersionORM{}
	var err error
	if prehook, ok := interface{}(m).(ApplicationVersionWithBeforeToORM); ok {
		if err = prehook.BeforeToORM(ctx, &to); err != nil {
			return to, err
		}
	}
....
	if posthook, ok := interface{}(m).(ApplicationVersionWithAfterToORM); ok {
// **** Callout 1
		err = posthook.AfterToORM(ctx, &to)
	}
	return to, err
}

Callout 2

The second call out from func (m *ApplicationVersionORM) ToPB(), shown below in code copied from secops.pb.gorm.go, will redirect call to func (m *ApplicationVersionORM) AfterToPB() and populates the ContainerTagsList going back to the client in the response.

// ToPB runs the BeforeToPB hook if present, converts the fields of this
// object to PB format, runs the AfterToPB hook, then returns the PB object
func (m *ApplicationVersionORM) ToPB(ctx context.Context) (ApplicationVersion, error) {
	to := ApplicationVersion{}
	var err error
....
	if posthook, ok := interface{}(m).(ApplicationVersionWithAfterToPB); ok {
// **** Callout 2
		err = posthook.AfterToPB(ctx, &to)
	}
	return to, err
}

Callout 3

The third call out is needed for an Update (e.g. PATCH or PUT) requests from clients. The high level code for Update is shown below, note that the previous callouts used for Create, Callout 1 & 2 still happen for Update and we have an additional Callout 3 that are shown here:

// DefaultStrictUpdateApplicationVersion clears first level 1:many children and then executes a gorm update call
func DefaultStrictUpdateApplicationVersion(ctx context.Context, in *ApplicationVersion, db *gorm1.DB) (*ApplicationVersion, error) {
	if in == nil {
		return nil, fmt.Errorf("Nil argument to DefaultStrictUpdateApplicationVersion")
	}
// **** Callout 1
	ormObj, err := in.ToORM(ctx)
	if err != nil {
		return nil, err
	}
	accountID, err := auth1.GetAccountID(ctx, nil)
	if err != nil {
		return nil, err
	}
	db = db.Where(map[string]interface{}{"account_id": accountID})
	lockedRow := &ApplicationVersionORM{}
	db.Model(&ormObj).Set("gorm:query_option", "FOR UPDATE").Where("id=?", ormObj.Id).First(lockedRow)
	if hook, ok := interface{}(&ormObj).(ApplicationVersionORMWithBeforeStrictUpdateCleanup); ok {
		if db, err = hook.BeforeStrictUpdateCleanup(ctx, db); err != nil {
			return nil, err
		}
	}
	if hook, ok := interface{}(&ormObj).(ApplicationVersionORMWithBeforeStrictUpdateSave); ok {
// **** Callout 3
		if db, err = hook.BeforeStrictUpdateSave(ctx, db); err != nil {
			return nil, err
		}
	}
	if err = db.Save(&ormObj).Error; err != nil {
		return nil, err
	}
	if hook, ok := interface{}(&ormObj).(ApplicationVersionORMWithAfterStrictUpdateSave); ok {
		if err = hook.AfterStrictUpdateSave(ctx, db); err != nil {
			return nil, err
		}
	}
// **** Callout 2
	pbResponse, err := ormObj.ToPB(ctx)
	if err != nil {
		return nil, err
	}
	return &pbResponse, err
}

The code for callout 3 shown here as hook.BeforeStrictUpdateSave()

		if db, err = hook.BeforeStrictUpdateSave(ctx, db); err != nil {

will redirect to func (m *ApplicationVersionORM) BeforeStrictUpdateSave() and will update the associations based on the updated references from client request.