graphik

README

Graphik

https://graphikdb.github.io/graphik/

git clone git@github.com:graphikDB/graphik.git

docker pull graphikdb/graphik:v0.4.0

Graphik is an identity-aware, permissioned, persistant document/graph database & pubsub server written in Go

• Helpful Links
• Features
• Key Dependencies
• Flags
• gRPC Client SDKs
• Implemenation Details
  • Primitives
  • Login/Authorization/Authorizers
    • Authorizers Examples
  • Secondary Indexes
    • Secondary Index Examples
  • Type Validators
    • Type Validator Examples
  • Identity Graph
  • GraphQL vs gRPC API
  • Additional Details
• Sample GraphQL Queries
  • Node Traversal
• Deployment
  • Docker-Compose
  • Kubernetes
  • Linux
  • Mac/Darwin
  • Windows

Helpful Links

• GraphQL Documentation Site
• Protobuf/gRPC API Spec
• Graphql API Spec
• Common Expression Language Code Lab
• CEL Standard Functions/Definitions
• OpenID Connect
• Directed Graph Wiki

Features

• 100% Go
• Native gRPC Support
• GraphQL Support
• Native Document & Graph Database
• Index-free Adjacency
• Native OAuth/OIDC Support & Single Sign On
• Embedded SSO protected GraphQl Playground
• Persistant(bbolt LMDB)
• Identity-Aware PubSub with Channels & Message Filtering(gRPC & graphQL)
• Change Streams
• Common Expression Language Query Filtering
• Common Expression Language Request Authorization
• Common Expression Language Type Validators
• Loosely-Typed(mongo-esque)
• Prometheus Metrics
• Pprof Metrics
• Safe to Deploy Publicly(with authorizers/tls/validators/cors)
• Read-Optimized
• Full Text Search(CEL)
• Regular Expressions(CEL)
• Client to Server streaming(gRPC only)

Key Dependencies

• google.golang.org/grpc
• github.com/autom8ter/machine
• github.com/google/cel-go/cel
• go.etcd.io/bbolt
• go.uber.org/zap
• golang.org/x/oauth2
• github.com/99designs/gqlgen

Flags

      --allow-headers strings             cors allow headers (env: GRAPHIK_ALLOW_HEADERS) (default [*])
      --allow-methods strings             cors allow methods (env: GRAPHIK_ALLOW_METHODS) (default [HEAD,GET,POST,PUT,PATCH,DELETE])
      --allow-origins strings             cors allow origins (env: GRAPHIK_ALLOW_ORIGINS) (default [*])
      --metrics                           enable prometheus & pprof metrics (emv: GRAPHIK_METRICS = true) (default true)
      --open-id string                    open id connect discovery uri ex: https://accounts.google.com/.well-known/openid-configuration (env: GRAPHIK_OPEN_ID)
      --playground-client-id string       playground oauth client id (env: GRAPHIK_PLAYGROUND_CLIENT_ID)
      --playground-client-secret string   playground oauth client secret (env: GRAPHIK_PLAYGROUND_CLIENT_SECRET
      --playground-redirect string        playground oauth redirect (env: GRAPHIK_PLAYGROUND_REDIRECT) (default "http://localhost:7820/playground/callback")
      --root-users strings                a list of email addresses that bypass registered authorizers(env: GRAPHIK_ROOT_USERS)
      --storage string                    persistant storage path (env: GRAPHIK_STORAGE_PATH) (default "/tmp/graphik")
      --tls-cert string                   path to tls certificate (env: GRAPHIK_TLS_CERT)
      --tls-key string                    path to tls key (env: GRAPHIK_TLS_KEY)

gRPC Client SDKs

• Go
• Python
• PHP
• Javascript
• Java
• C#
• Ruby

Implemenation Details

Please see GraphQL Documentation Site for additional details

Primitives

• Ref == direct pointer to an doc or connection.

message Ref {
  // gtype is the type of the doc/connection ex: pet
  string gtype =1 [(validator.field) = {regex : "^.{1,225}$"}];
  // gid is the unique id of the doc/connection within the context of it's type
  string gid =2 [(validator.field) = {regex : "^.{1,225}$"}];
}

• Doc == JSON document in document storage terms AND vertex/node in graph theory

message Doc {
    // ref is the ref to the doc
    Ref ref =1 [(validator.field) = {msg_exists : true}];
    // k/v pairs
    google.protobuf.Struct attributes =2;
}

• Connection == graph edge/relationship in graph theory. Connections relate Docs to one another.

message Connection {
  // ref is the ref to the connection
  Ref ref =1 [(validator.field) = {msg_exists : true}];
  // attributes are k/v pairs
  google.protobuf.Struct attributes =2;
  // directed is false if the connection is bi-directional
  bool directed =3;
  // from is the doc ref that is the source of the connection
  Ref from =4 [(validator.field) = {msg_exists : true}];
  // to is the doc ref that is the destination of the connection
  Ref to =5 [(validator.field) = {msg_exists : true}];
}

Login/Authorization/Authorizers

• an access token Authorization: Bearer ${token} from the configured open-id connect identity provider is required for all database functionality
• the access token is used to fetch the users info from the oidc userinfo endpoint fetched from the oidc metadata url
• if a user is not present in the database, one will be automatically created under the gtype: user with their email address as their gid
• once the user is fetched, it is evaluated(along with the request & request method) against any registered authorizers(CEL expression) in the database.
  • if an authorizer evaluates false, the request will be denied
  • authorizers may be used to restrict access to functionality by domain, role, email, etc
  • registered root users(see flags) bypass these authorizers
• authorizers are completely optional but highly recommended

Authorizers Examples

Coming Soon

Secondary Indexes

• secondary indexes are CEL expressions evaluated against a particular type of Doc or Connection
• indexes may be used to speed up queries that iterate over a large number of elements
• secondary indexes are completely optional but recommended

Secondary Index Examples

Coming Soon

Type Validators

• type validators are CEL expressions evaluated against a particular type of Doc or Connection to enforce custom constraints
• type validators are completely optional

Type Validator Examples

Coming Soon

Identity Graph

• any time a document is created, a connection of type created from the origin user to the new document is also created
• any time a document is created, a connection of type created_by from the new document to the origin user is also created
• any time a document is edited, a connection of type edited from the origin user to the new document is also created(if none exists)
• any time a document is edited, a connection of type edited_by from the new document to the origin user is also created(if none exists)
• every document a user has ever interacted with may be queried via the Traverse method with the user as the root document of the traversal

GraphQL vs gRPC API

In my opinion, gRPC is king for svc-svc communication & graphQL is king for developing user interfaces & exploring data.

In graphik the graphQL & gRPC are nearly identical, but every request flows through the gRPC server natively - the graphQL api is technically a wrapper that may be used for developing user interfaces & querying the database from the graphQL playground.

The gRPC server is more performant so it is advised that you import one of the gRPC client libraries as opposed to utilizing the graphQL endpoint when developing backend APIs.

The graphQL endpoint is particularly useful for developing public user interfaces against since it can be locked down to nearly any extent via authorizers, cors, validators, & tls.

Streaming/PubSub

Graphik supports channel based pubsub as well as change-based streaming.

All server -> client stream/subscriptions are started via the Stream() endpoint in gRPC or graphQL. All messages received on this channel include the user that triggered/sent the message. Messages on channels may be filtered via CEL expressions so that only messages are pushed to clients that they want to receive. Messages may be sent directly to channels via the Broadcast() method in gRPC & graphQL. All state changes in the graph are sent by graphik to the state channel which may be subscribed to just like any other channel.

Graphik Playground

If the following environmental variables/flags are set, an SSO protected graphQL playground will be served on /playground

GRAPHIK_PLAYGROUND_CLIENT_ID=${client_id} # the oauth2 application/client id
GRAPHIK_PLAYGROUND_CLIENT_SECRET=${client_secret} # the oauth2 application/client secret
GRAPHIK_PLAYGROUND_REDIRECT=${playground_redirect} # the oauth2 authorization code redirect: the playground exposes an endpoint to handle this redirect /playground/callback

Additional Details

• any time a Doc is deleted, so are all of its connections

Sample GraphQL Queries

Get Currently Logged In User(me)

query {
  me(where: {}) {
    ref {
      gid
      gtype
    }
		attributes
  }
}

{
  "data": {
    "me": {
      "ref": {
        "gid": "coleman.word@graphikdb.io",
        "gtype": "user"
      },
      "attributes": {
        "email": "coleman.word@graphikdb.io",
        "email_verified": true,
        "family_name": "Word",
        "given_name": "Coleman",
        "hd": "graphikdb.io",
        "locale": "en",
        "name": "Coleman Word",
        "picture": "https://lh3.googleusercontent.com/--LNU8XICB1A/AAAAAAAAAAI/AAAAAAAAAAA/AMZuuckp6gwH9JVkhlRkk-PTZdyDFctArg/s96-c/photo.jpg",
        "sub": "105138978122958973720"
      }
    }
  },
  "extensions": {}
}

Get the Graph Schema

query {
  getSchema(where: {}) {
    doc_types
		connection_types
    authorizers {
      authorizers {
        name
        expression
      }
    }
		validators {
			validators {
				name
				expression
			}
		}
		indexes {
			indexes {
				name
				expression
			}
		}
  }
}

{
  "data": {
    "getSchema": {
      "doc_types": [
        "dog",
        "human",
        "note",
        "user"
      ],
      "connection_types": [
        "created",
        "created_by",
        "edited",
        "edited_by",
        "owner"
      ],
      "authorizers": {
        "authorizers": [
          {
            "name": "testing",
            "expression": "this.user.attributes.email.contains(\"coleman\")"
          }
        ]
      },
      "validators": {
        "validators": [
          {
            "name": "testing",
            "expression": "this.user.attributes.email.contains(\"coleman\")"
          }
        ]
      },
      "indexes": {
        "indexes": [
          {
            "name": "testing",
            "expression": "this.attributes.primary_owner"
          }
        ]
      }
    }
  },
  "extensions": {}
}

Create a Document

mutation {
  createDoc(input: {
    ref: {
  		gtype: "note"
    }
    attributes: {
      title: "do the dishes"
    }
  }){
    ref {
      gid
      gtype
    }
    attributes
  }
}

{
  "data": {
    "createDoc": {
      "ref": {
        "gid": "1lU0w0QjiI0jnNL8XMzWJHqQmTd",
        "gtype": "note"
      },
      "attributes": {
        "title": "do the dishes"
      }
    }
  },
  "extensions": {}
}

Traverse Documents

# Write your query or mutation here
query {
  traverse(input: {
    root: {
      gid: "coleman.word@graphikdb.io"
      gtype: "user"
    }
    algorithm: BFS
    limit: 6
		max_depth: 1
		max_hops: 10
  }){
    traversals {
      doc {
        ref {
          gid
          gtype
        }
      }
      traversal_path {
        gid
        gtype
      }
			depth
			hops
    }
  }
}

Traverse Documents Related to Logged In User

query {
	traverseMe(where: {
		max_hops: 100
		max_depth:1
		limit: 5
	}){
		traversals {
			traversal_path {
				gtype
				gid
			}
			depth
			hops
			doc {
				ref {
					gid
					gtype
				}
			}
		}
	}
}

{
  "data": {
    "traverseMe": {
      "traversals": [
        {
          "traversal_path": null,
          "depth": 0,
          "hops": 0,
          "doc": {
            "ref": {
              "gid": "coleman.word@graphikdb.io",
              "gtype": "user"
            }
          }
        },
        {
          "traversal_path": [
            {
              "gtype": "user",
              "gid": "coleman.word@graphikdb.io"
            }
          ],
          "depth": 1,
          "hops": 1,
          "doc": {
            "ref": {
              "gid": "1lU0w0QjiI0jnNL8XMzWJHqQmTd",
              "gtype": "note"
            }
          }
        }
      ]
    }
  },
  "extensions": {}
}

Change Streaming

subscription {
	stream(where: {
		channel: "state"
	}){
		data
		user {
			gid
			gtype
		}
	}
}

{
  "data": {
    "stream": {
      "data": {
        "attributes": {
          "title": "do the dishes"
        },
        "ref": {
          "gid": "1lUAK3uwwmhQ503ByzC9nCvdH6W",
          "gtype": "note"
        }
      },
      "user": {
        "gid": "coleman.word@graphikdb.io",
        "gtype": "user"
      }
    }
  },
  "extensions": {}
}

Deployment

Regardless of deployment methodology, please set the following environmental variables or include them in a ${pwd}/.env file

GRAPHIK_PLAYGROUND_CLIENT_ID=${client_id}
GRAPHIK_PLAYGROUND_CLIENT_SECRET=${client_secret}
GRAPHIK_PLAYGROUND_REDIRECT=http://localhost:7820/playground/callback
GRAPHIK_OPEN_ID=${open_id_connect_metadata_url}
#GRAPHIK_ALLOW_HEADERS=${cors_headers}
#GRAPHIK_ALLOW_METHOD=${cors_methos}
#GRAPHIK_ALLOW_ORIGINS=${cors_origins}
#GRAPHIK_ROOT_USERS=${root_users}
#GRAPHIK_TLS_CERT=${tls_cert_path}
#GRAPHIK_TLS_KEY=${tls_key_path}

Docker-Compose

add this docker-compose.yml to ${pwd}:

version: '3.7'
services:
  graphik:
    image: graphikdb/graphik:v0.4.0
    env_file:
      - .env
    ports:
      - "7820:7820"
      - "7821:7821"
    volumes:
      - default:/tmp/graphik
    networks:
      default:
        aliases:
          - graphikdb
networks:
  default:

volumes:
  default:

then run:

docker-compose -f docker-compose.yml pull
docker-compose -f docker-compose.yml up -d

to shutdown:

docker-compose -f docker-compose.yml down --remove-orphans

Kubernetes

Coming Soon

Linux

curl -L https://github.com/graphikDB/graphik/releases/download/v0.4.0/graphik_linux_amd64 \
--output /usr/local/bin/graphik && \
chmod +x /usr/local/bin/graphik 

Mac/Darwin

curl -L https://github.com/graphikDB/graphik/releases/download/v0.4.0/graphik_darwin_amd64 \
--output /usr/local/bin/graphik && \
chmod +x /usr/local/bin/graphik

Windows

Nope - use docker