README
¶
Datum Core
This repository is experimental meaning that it's based on untested ideas or techniques and not yet established or finalized or involves a radically new and innovative style! This means that support is best effort (at best!) and we strongly encourage you to NOT use this in production - reach out to @matoszz with any questions
This repo will hold the core server / handler for Datum services - for complete detailed references please check out the go-template used to generate this repository; it can be found here
Development
Datum's core server operates with the following utilities:
- ent - insane entity mapping tool, definitely not an ORM but kind of an ORM
- atlas - Schema generation and migration
- gqlgen - Code generation from schema definitions
- openfga - Authorization
Dependencies
Setup Taskfile by following the instructions and using one of the various convenient package managers or installation scripts. Two of the more common installation methods are below for your convenience:
brew install go-task
sh -c "$(curl --location https://taskfile.dev/install.sh)" -- -d
(by default, this installs on the ./bin directory relative to the working directory)
After installation, you can then simply run task install to load the associated dependencies. Nearly everything in this repository assumes you already have a local golang environment setup so this is not included. Please see the associated documentation.
To include Taskfile's created in other directories / to call the respective tasks, you would add an includes per the Taskfile documentation and then reference it by name, e.g. task cli:createorg
Updating Environment Variables
Within the config directory in the root of this repository there are several .example files prefixed with .env-dev or similar; these hold examples of environment configurations which you should review and potentially override depending on your needs. Anything which is launched out of the Taskfile will source it's environment from these files and their configurations. Different tasks can be made to source from different files as can be seen by several of the tasks within the Taskfile.
You will need to perform a 1-time action of either removing the .example suffix from these files or creating your own files which match the naming convensions called for {{.ENV}} within the Taskfile. These files are intentionally added to the .gitignore within this repository to prevent you from accidentally committing secrets or other sensitive information which may live inside the server's environment variables.
Pre-requisites to a PR
This repository contains a number of code generating functions / utilities which take schema modifications and scaffold out resolvers, graphql API schemas, openAPI specifications, among other things. To ensure you've generated all the necessary dependencies run task pr; this will run the entirety of the commands required to safely generate a PR. If for some reason one of the commands fails / encounters an error, you will need to debug the individual steps. It should be decently easy to follow the Taskfile in the root of this repository.
Querying
The best method of forming / testing queries against the server is to run task rover which will launch an interactive query UI.
OpenFGA Playground
You can load up a local openFGA environment with the compose setup in this repository; task fga:up - this will launch an interactive playground where you can model permissions model(s) or changes to the models
Migrations
task atlas or task atlas:create will generate the necessary migrations
Creating a new Schema
To ease the effort required to add additional schemas into the system a template + task function has been created. This isn't doing anything terribly complex, but it's attempting to ensure you have the minimum set of required things needed to create a schema - most notably: you need to ensure the IDMixin is present (otherwise you will get ID type conflicts) and a standard set of schema annotations.
**NOTE: you still have to make intelligent decisions around things like the presence / integration of hooks, interceptors, policies, etc. This is saving you about 10 seconds of copy-paste, so don't over estimate the automation, here.
To generate a new schema, you can run task newschema -- [yourschemaname] where you replace the name within []. Please be sure to note that this isn't a command line flag so there's a space between -- and the name.
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd is our cobra/viper cli implementation
|
Package cmd is our cobra/viper cli implementation |
|
cli/cmd
Package datum is our cobra/viper cli implementation
|
Package datum is our cobra/viper cli implementation |
|
cli/cmd/group
Package datumgroup is our cobra/viper cli for group endpoints
|
Package datumgroup is our cobra/viper cli for group endpoints |
|
cli/cmd/login
Package datumlogin is our cobra/viper cli for authentication endpoints
|
Package datumlogin is our cobra/viper cli for authentication endpoints |
|
cli/cmd/org
Package datumorg is our cobra/viper cli for organization endpoints
|
Package datumorg is our cobra/viper cli for organization endpoints |
|
cli/cmd/register
Package register allows user registration
|
Package register allows user registration |
|
cli/cmd/reset
Package reset allows user password reset
|
Package reset allows user password reset |
|
cli/cmd/tokens
Package datumtokens is our cobra/viper cli for token endpoints
|
Package datumtokens is our cobra/viper cli for token endpoints |
|
cli/cmd/user
Package datumuser is our cobra/viper cli for user endpoints
|
Package datumuser is our cobra/viper cli for user endpoints |
|
internal
|
|
|
constants
Package constants contains constants used throughout the application
|
Package constants contains constants used throughout the application |
|
datumclient
Package datumclient contains the client to interact with the datum server
|
Package datumclient contains the client to interact with the datum server |
|
ent/enums
Package enums has enums
|
Package enums has enums |
|
ent/generated
Package generated is the ent generated package
|
Package generated is the ent generated package |
|
ent/hooks
Package hooks is middleware to alter the graphql mutation
|
Package hooks is middleware to alter the graphql mutation |
|
ent/interceptors
Package interceptors is middleware to alter the graphql query
|
Package interceptors is middleware to alter the graphql query |
|
ent/mixin
Package mixin contains the mixin package
|
Package mixin contains the mixin package |
|
ent/privacy/rule
Package rule contains policy rules
|
Package rule contains policy rules |
|
ent/privacy/token
Package token provides policy wrappers / getters for queries and mutations during password reset, signup
|
Package token provides policy wrappers / getters for queries and mutations during password reset, signup |
|
ent/privacy/utils
Package utils is a helper for working with ent policies
|
Package utils is a helper for working with ent policies |
|
ent/privacy/viewer
Package viewer contains role definitions for policies
|
Package viewer contains role definitions for policies |
|
ent/schema
Package schema contains the ent schema
|
Package schema contains the ent schema |
|
entclean
nolint
|
nolint |
|
entdb
Package entdb extends the ent db library and satisfies matt's needs for consistency
|
Package entdb extends the ent db library and satisfies matt's needs for consistency |
|
entx
Package entx is a package of tools for interacting with ent
|
Package entx is a package of tools for interacting with ent |
|
fga
Package fga is a wrapper around openfga client credit to https://github.com/canonical/ofga/blob/main/tuples.go TODO: can we contribute this back once we have this in a working place
|
Package fga is a wrapper around openfga client credit to https://github.com/canonical/ofga/blob/main/tuples.go TODO: can we contribute this back once we have this in a working place |
|
fga/mocks
Package mock_client is a generated GoMock package.
|
Package mock_client is a generated GoMock package. |
|
graphapi
Package graphapi is the graph api package
|
Package graphapi is the graph api package |
|
httpserve/config
Package config holds the echo server configuration utilities
|
Package config holds the echo server configuration utilities |
|
httpserve/handlers
Package handlers contains custom handler functions
|
Package handlers contains custom handler functions |
|
httpserve/middleware
Package middleware provides middleware for http Handlers.
|
Package middleware provides middleware for http Handlers. |
|
httpserve/middleware/auth
Package auth handles auth
|
Package auth handles auth |
|
httpserve/middleware/authtest
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication
|
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication |
|
httpserve/middleware/cachecontrol
Package cachecontrol creates a cache control echo middleware
|
Package cachecontrol creates a cache control echo middleware |
|
httpserve/middleware/cors
Package cors does cors stuff
|
Package cors does cors stuff |
|
httpserve/middleware/debug
Package debug contains utility helper functions for debugging when working with echo server
|
Package debug contains utility helper functions for debugging when working with echo server |
|
httpserve/middleware/echocontext
Package echocontext adds the echo context to the parent context
|
Package echocontext adds the echo context to the parent context |
|
httpserve/middleware/mime
Package mime does mime stuff
|
Package mime does mime stuff |
|
httpserve/middleware/ratelimit
Package ratelimit implements a rate limiting middleware
|
Package ratelimit implements a rate limiting middleware |
|
httpserve/middleware/redirect
Package redirect does redirect stuff
|
Package redirect does redirect stuff |
|
httpserve/middleware/secure
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks
|
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks |
|
httpserve/middleware/session
Package session provides session middleware to the echo server
|
Package session provides session middleware to the echo server |
|
httpserve/middleware/transaction
Package transaction implements a transaction middleware for REST endpoints using the ent db client
|
Package transaction implements a transaction middleware for REST endpoints using the ent db client |
|
httpserve/route
Package route will hold the routes and route groups
|
Package route will hold the routes and route groups |
|
httpserve/server
Package server contains the server functions
|
Package server contains the server functions |
|
httpserve/serveropts
Package serveropts contains an echo server options wrapper
|
Package serveropts contains an echo server options wrapper |
|
keygen
Package keygen provides functionality for generating API client IDs and secrets
|
Package keygen provides functionality for generating API client IDs and secrets |
|
otelx
Package otelx provides configuration options for various OpenTelemetry integrations.
|
Package otelx provides configuration options for various OpenTelemetry integrations. |
|
passwd
Package passwd provides fancy crypto shit for passwords
|
Package passwd provides fancy crypto shit for passwords |
|
store
Package store is for storing cookies, sessions, and maybe other fun stuff
|
Package store is for storing cookies, sessions, and maybe other fun stuff |
|
testutils
Package testutils provides utilities for use in tests
|
Package testutils provides utilities for use in tests |
|
tokens
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens
|
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens |
|
utils/emails
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice.
|
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice. |
|
utils/emails/mock
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here
|
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here |
|
utils/gravatar
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars
|
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars |
|
utils/marionette
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in.
|
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in. |
|
utils/responses
Package responses holds human readable string responses that can be surfaced up to an end user
|
Package responses holds human readable string responses that can be surfaced up to an end user |
|
utils/rows
Package rows makes CLI output pretty
|
Package rows makes CLI output pretty |
|
utils/sendgrid
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider
|
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider |
|
utils/sqlite
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups.
|
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups. |
|
utils/ulids
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages
|
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages |
|
utils/viperconfig
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag()
|
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag() |