README
¶
Datum Core
This repository is experimental meaning that it's based on untested ideas or techniques and not yet established or finalized or involves a radically new and innovative style! This means that support is best effort (at best!) and we strongly encourage you to NOT use this in production - reach out to @matoszz with any questions
Development
Datum's core server operates with the following utilities:
- ent - insane entity mapping tool, definitely not an ORM but kind of an ORM
- atlas - Schema generation and migration
- gqlgen - Code generation from schema definitions
- gqlgenc - client building utilities with GraphQL
- openfga - Authorization
- echo - High performance, extensible, minimalist Go web framework
- koanf - configuration management
- viper - command line flags / management
We also leverage many secondary technologies in use, including (but not limited to!):
- redis - in-memory datastore used for sessions, caching
- sqlite - currently planned database system but also offer additional support for PostgreSQL
- golangci-lint - an annoyingly opinionated linter
- buildkite - our CI system of choice (with github actions providing some intermediary support)
- sonar - used for code scanning, vulnerability scanning
- posthog - product analytics
- sentry - error montioring, tracing
- sendgrid - transactional email send provider
All of these components are bundled into our respective Docker images; for additional information / instructions, see the contributing guide in this repository.
Dependencies
Setup Taskfile by following the instructions and using one of the various convenient package managers or installation scripts. After installation, you can then simply run task install to load the associated dependencies. Nearly everything in this repository assumes you already have a local golang environment setup so this is not included. Please see the associated documentation.
Updating Configuration Settings
Within the config directory in the root of this repository there are several example.yaml files prefixed with config or similar; these hold examples of environment configurations which you should review and potentially override depending on your needs. Anything which is launched out of the Taskfile will source it's configuration from these files.
You will need to perform a 1-time action of creating a .config.yaml file based on the .example files.
The Taskfiles will also source a .dotenv files which match the naming conventions called for {{.ENV}} to ease the overriding of environment variables. These files are intentionally added to the .gitignore within this repository to prevent you from accidentally committing secrets or other sensitive information which may live inside the server's environment variables.
All settings in the yaml configuration can also be overwritten with environment variables prefixed with DATUM_. For example, to override the Google client_secret set in the yaml configuration with an environment variable you can use:
export DATUM_AUTH_PROVIDERS_GOOGLE_CLIENTSECRET
Configuration precedence is as follows, the latter overriding the former:
defaultvalues set in the config struct within the code.config.yamlvalues- Environment variables
Documentation
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
Package cmd is our cobra/viper cli implementation
|
Package cmd is our cobra/viper cli implementation |
|
cli/cmd
Package datum is our cobra/viper cli implementation
|
Package datum is our cobra/viper cli implementation |
|
cli/cmd/group
Package datumgroup is our cobra/viper cli for group endpoints
|
Package datumgroup is our cobra/viper cli for group endpoints |
|
cli/cmd/groupmembers
Package datumgroupmembers is our cobra/viper cli for group member endpoints
|
Package datumgroupmembers is our cobra/viper cli for group member endpoints |
|
cli/cmd/groupsetting
Package datumgroupsetting provides commands for managing group settings
|
Package datumgroupsetting provides commands for managing group settings |
|
cli/cmd/invite
Package datuminvite creates invitation emails + tokens for external users to join a Datum organization
|
Package datuminvite creates invitation emails + tokens for external users to join a Datum organization |
|
cli/cmd/login
Package datumlogin is our cobra/viper cli for authentication endpoints
|
Package datumlogin is our cobra/viper cli for authentication endpoints |
|
cli/cmd/org
Package datumorg is our cobra/viper cli for organization endpoints
|
Package datumorg is our cobra/viper cli for organization endpoints |
|
cli/cmd/orgmembers
Package datumorgmembers is our cobra/viper cli for org member endpoints
|
Package datumorgmembers is our cobra/viper cli for org member endpoints |
|
cli/cmd/orgsetting
Package datumorgsetting provides commands for managing organization settings
|
Package datumorgsetting provides commands for managing organization settings |
|
cli/cmd/register
Package register allows user registration
|
Package register allows user registration |
|
cli/cmd/reset
Package reset allows user password reset
|
Package reset allows user password reset |
|
cli/cmd/subscribers
Package datumsubscribers is our cobra/viper cli for subscriber endpoints
|
Package datumsubscribers is our cobra/viper cli for subscriber endpoints |
|
cli/cmd/tokens
Package datumtokens is our cobra/viper cli for token endpoints
|
Package datumtokens is our cobra/viper cli for token endpoints |
|
cli/cmd/user
Package datumuser is our cobra/viper cli for user endpoints
|
Package datumuser is our cobra/viper cli for user endpoints |
|
cli/cmd/usersetting
Package datumusersetting is our cobra/viper cli for user setting endpoints
|
Package datumusersetting is our cobra/viper cli for user setting endpoints |
|
cli/cmd/version
Package version contains the version information for the CLI
|
Package version contains the version information for the CLI |
|
Package config holds configuration stuff to configure the things
|
Package config holds configuration stuff to configure the things |
|
Package db provides an embedded filesystem containing all the database migrations
|
Package db provides an embedded filesystem containing all the database migrations |
|
internal
|
|
|
constants
Package constants contains constants used throughout the application
|
Package constants contains constants used throughout the application |
|
datumclient
Package datumclient contains the client to interact with the datum server
|
Package datumclient contains the client to interact with the datum server |
|
ent/enums
Package enums has enums
|
Package enums has enums |
|
ent/generated
Package generated is the ent generated package
|
Package generated is the ent generated package |
|
ent/hooks
Package hooks is middleware to alter the graphql mutation
|
Package hooks is middleware to alter the graphql mutation |
|
ent/interceptors
Package interceptors is middleware to alter the graphql query
|
Package interceptors is middleware to alter the graphql query |
|
ent/mixin
Package mixin contains the mixin package
|
Package mixin contains the mixin package |
|
ent/privacy/rule
Package rule contains policy rules
|
Package rule contains policy rules |
|
ent/privacy/token
Package token provides policy wrappers / getters for queries and mutations during password reset, signup
|
Package token provides policy wrappers / getters for queries and mutations during password reset, signup |
|
ent/privacy/utils
Package utils is a helper for working with ent policies
|
Package utils is a helper for working with ent policies |
|
ent/privacy/viewer
Package viewer contains role definitions for policies
|
Package viewer contains role definitions for policies |
|
ent/schema
Package schema contains the ent schema
|
Package schema contains the ent schema |
|
entdb
Package entdb extends the ent db library and satisfies matt's needs for consistency
|
Package entdb extends the ent db library and satisfies matt's needs for consistency |
|
graphapi
Package graphapi is the graph api package
|
Package graphapi is the graph api package |
|
httpserve/config
Package config holds the echo server configuration utilities
|
Package config holds the echo server configuration utilities |
|
httpserve/handlers
Package handlers contains custom handler functions
|
Package handlers contains custom handler functions |
|
httpserve/route
Package route will hold the routes and route groups
|
Package route will hold the routes and route groups |
|
httpserve/server
Package server contains the server functions
|
Package server contains the server functions |
|
httpserve/serveropts
Package serveropts contains an echo server options wrapper
|
Package serveropts contains an echo server options wrapper |
|
envparse
Package envparse provides a way to parse environment variables from a struct
|
Package envparse provides a way to parse environment variables from a struct |
|
pkg
|
|
|
analytics
Package analytics offers datum system analytics
|
Package analytics offers datum system analytics |
|
analytics/machine
Package machine provides a basic wrapper for getting machine ID
|
Package machine provides a basic wrapper for getting machine ID |
|
analytics/posthog
Package posthog contains posthog helper functions and utilities
|
Package posthog contains posthog helper functions and utilities |
|
auth
Package auth handles auth
|
Package auth handles auth |
|
cache
Package cache holds the library for interacting with redis
|
Package cache holds the library for interacting with redis |
|
keygen
Package keygen provides functionality for generating API client IDs and secrets
|
Package keygen provides functionality for generating API client IDs and secrets |
|
middleware
Package middleware provides middleware for http Handlers.
|
Package middleware provides middleware for http Handlers. |
|
middleware/auth
Package auth handles auth
|
Package auth handles auth |
|
middleware/authtest
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication
|
Package authtest is a mini server and token generator with a mock test to confirm keygen works correctly; not intended to test the whole api but rather just the token issuance and authentication |
|
middleware/cachecontrol
Package cachecontrol creates a cache control echo middleware
|
Package cachecontrol creates a cache control echo middleware |
|
middleware/cors
Package cors does cors stuff
|
Package cors does cors stuff |
|
middleware/debug
Package debug contains utility helper functions for debugging when working with echo server
|
Package debug contains utility helper functions for debugging when working with echo server |
|
middleware/echocontext
Package echocontext adds the echo context to the parent context
|
Package echocontext adds the echo context to the parent context |
|
middleware/mime
Package mime does mime stuff
|
Package mime does mime stuff |
|
middleware/ratelimit
Package ratelimit implements a rate limiting middleware
|
Package ratelimit implements a rate limiting middleware |
|
middleware/redirect
Package redirect does redirect stuff
|
Package redirect does redirect stuff |
|
middleware/secure
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks
|
Package secure is middleware that provides protection against cross-site scripting (XSS) attack, content type sniffing, clickjacking, insecure connection and other code injection attacks |
|
middleware/sentry
Package sentry is a middleware package for sentry integration
|
Package sentry is a middleware package for sentry integration |
|
middleware/transaction
Package transaction implements a transaction middleware for REST endpoints using the ent db client
|
Package transaction implements a transaction middleware for REST endpoints using the ent db client |
|
otelx
Package otelx provides configuration options for various OpenTelemetry integrations.
|
Package otelx provides configuration options for various OpenTelemetry integrations. |
|
passwd
Package passwd provides fancy crypto shit for passwords
|
Package passwd provides fancy crypto shit for passwords |
|
providers/github
Package github provides GitHub OAuth2 login and callback handlers.
|
Package github provides GitHub OAuth2 login and callback handlers. |
|
providers/google
Package google provides Google OAuth2 login and callback handlers.
|
Package google provides Google OAuth2 login and callback handlers. |
|
providers/oauth2
Package oauth2 provides handlers for OAuth2 login and callback requests.
|
Package oauth2 provides handlers for OAuth2 login and callback requests. |
|
providers/oidc
Package oidc provides oidc authentication helpers and flow
|
Package oidc provides oidc authentication helpers and flow |
|
providers/webauthn
Package webauthn is a provider package offering Passkey login functionality
|
Package webauthn is a provider package offering Passkey login functionality |
|
rout
Package rout is a semi-centralized method of handling and surfacing user facing errors
|
Package rout is a semi-centralized method of handling and surfacing user facing errors |
|
sessions
Package sessions manages users sessions
|
Package sessions manages users sessions |
|
testutils
Package testutils provides utilities for use in tests
|
Package testutils provides utilities for use in tests |
|
tokens
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens
|
Package tokens creates tokenmanager, responsible for signing, issuing, and validating tokens |
|
utils/cli/rows
Package rows makes CLI output pretty
|
Package rows makes CLI output pretty |
|
utils/cli/useragent
Package useragent is a utility package for generating the user agent string for the CLI
|
Package useragent is a utility package for generating the user agent string for the CLI |
|
utils/dumper
Package dumper is a utility package for dumping http request contents
|
Package dumper is a utility package for dumping http request contents |
|
utils/emails
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice.
|
Package emails contains all of the logic and templating for sending emails within the datum services - it is built to allow for other email providers other than sendgrid if required but is mostly specific to sendgrid as that's our email provider of choice. |
|
utils/emails/mock
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here
|
Package mock provides intermediary data structures to assist in testing and debugging sending an email without actually sending the email - testing the creation and sending of email typically requires some kind of tracking of what was sent or not and metadata about the message which is what is setup here |
|
utils/gravatar
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars
|
Package gravatar generates random avatar images for users or organizations on sign up which they can override with their own avatars |
|
utils/marionette
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in.
|
Package marionette is a golang based task manager with scheduling, backoff, future scheduling built in. |
|
utils/sendgrid
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider
|
Package sendgrid is a utiliuty package for interfacing with Sendgrid, a third party transactional email provider |
|
utils/sentry
Package sentry is a helper package for sentry integration
|
Package sentry is a helper package for sentry integration |
|
utils/sqlite
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups.
|
Package sqlite implements a connect hook around the sqlite3 driver so that the underlying connection can be fetched from the driver for more advanced operations such as backups. |
|
utils/totp
Package totp provides code generation for TOTP (RFC 6238) and HOTP (RFC 4226)
|
Package totp provides code generation for TOTP (RFC 6238) and HOTP (RFC 4226) |
|
utils/ulids
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages
|
Package ulids is a lightweight wrapper around the github.com/oklog/ulid package that provides some common functionality (like checking if a ULID is null or is zero) as well as a process-global, cryptographically random, monotonic, and thread-safe ulid generation mechanism that can be used from external packages |
|
utils/viperconfig
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag()
|
Package viperconfig is a fancy interface allowing you to bind viper commands in your individual packages without having to make some insane monolithic file Credit goes to the stacklok folks: https://github.com/stacklok/minder/blob/main/internal/util/configutils.go TO DO: convert allthethings over to using utils.BindConfigFlag() |
Click to show internal directories.
Click to hide internal directories.