Flowify is an Equinor developed workflow manager based on the Argo workflows project.
The aim of the project is to provide a simple and non-technical user interface to help users build and execute data- or compute-intensive
workflows on a Kubernetes platform. This repository contains the server part of the Flowify project, the client is hosted elsewhere.
Installation and deployment
To deploy on a Kubernetes cluster, see https://equinor.github.io/flowify-documentation/run_k8s/
Development
Tested requirements (other versions may work)
- Docker 20.10.8
- Docker compose 2.10.2
To start-up development environment
git clone git@github.com:equinor/flowify-workflows-server.git
cd dev
docker compose up -d
Note: The kind cluster will take some time to spin up.
To rebuild the server after code changes
docker compose up -d --build server
The following services will be available at:
Service |
Port |
Flowify server |
8842 |
MongoDB |
27017 |
To Get cluster container IP Address:
docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' cluster-control-plane
Load kubeconfig from host machine
docker exec cluster-control-plane cat /etc/kubernetes/admin.conf | sed "s/cluster-control-plane/$KUBERNETES_SERVICE_HOST/g" > ~/.kube/config
Load example database to mongoDB
docker cp ./dev/example_data/dump mongo_server:/
docker exec mongo_server mongorestore dump
Optional: To start an developer instance of the Frontend:
Set the following dummy JWT token as environmental variable on your local host
export FLOWIFY_AUTH_TOKEN="Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJzYW5kYm94IiwiYXVkIjoiZmxvd2lmeSIsImlhdCI6MTY2MzY3NDU0NywibmJmIjoxNjYzNjc0NTQ3LCJleHAiOjI2MTA0NDU3NDcsIm9pZCI6IjgwNDgiLCJuYW1lIjoiRi4gTG93ZSIsImVtYWlsIjoiZmxvd0BzYW5kLmJveCIsInJvbGVzIjpbInNhbmRib3gtZGV2ZWxvcGVyIl19.Hc4gXrL6hsE91S6qlJpFfsONq7L-jTN9WsHxtC1fhGk"
Alternatively encode the following claims to a JWT token and set as value of environmental variable FLOWIFY_AUTH_TOKEN
"name": "F. Lowe",
"email": "flow@flowify.io",
"roles": [""], <= needs to match the dev-environment
"iat": time.Now().Unix(),
"nbf": time.Now().Unix(),
"exp": time.Now().Add(time.Minute * 5).Unix(), // at least enough into the future
"aud": "flowify", <= needs to match the dev-environment
"iss": "sandbox", <= needs to match the dev-environment
Run the frontend and attach to the docker network
git clone git@github.com:equinor/flowify-workflows-UI.git
docker build . -t dev_frontend
docker run -d --rm --name flowify_ui -p 8080:8080 --network kind -e FLOWIFY_AUTH_TOKEN dev_frontend
Frontend will be available at http://localhost:8080
Deployment architecture of development environment
C4Container
Deployment_Node(c1, "Docker network: kind") {
Deployment_Node(d_com, "docker-compose"){
Container(kind, "kind Cluster running Argo Workflows")
Container(mongo, "MongoDB")
Container(server, "Flowify server")
}
Container(ctrl, "cluster-control-plane")
Container(ui, "Flowify UI (Optional)")
}
System_Ext(host, "Local host")
Rel(kind, ctrl, "SIGTERM")
BiRel(host, mongo, "Port 27017")
BiRel(host, server, "Port 8842")
BiRel(host, ui, "Port 8080")
UpdateLayoutConfig($c4ShapeInRow="2", $c4BoundaryInRow="2")
Summary of Makefile commands
Command |
Description |
make init |
pre-commit hooks for gofmt applied before patches |
make clean |
remove protoc and swagger files, and clear Go cache |
make codegen |
This will install all the relevant code generator tools, and create the Go interfaces and REST gateway from the gRPC specification |
make server |
Build the flowify server binary |
make all |
alias for the previous make codegen server |
make tests |
run the tests and create coverage files |
Local sandbox deployment
The project contains a sandbox setup with scripts and tokens that allow the server to run and be tested. Either by manually curling the endpoints, or together with a locally deployed frontend. In the latter case the appropriate Authorization
header is required, either from a proxy, or directly injected in the browser.
The application does not verify the signature on the authorization token, but
expects it to be there and have the roles
and email
fields set.
Add workspace access
A Flowify workspace is a compartmentalization within the Flowify application. Every
object and every running of a workflow needs to be associated with a workspace.
In order to define Flowify workspaces, we need to have several components in
place. At the time of writing, this is still a manual process.
First, we need to create the actual workspaces in the Flowify system. To implement
them, currently we use the Kubernetes namespace functionality. We also need to
create a Kubernetes ConfigMap that sets to rules on how to access this workspace.
Second, using the Azure Identity management system, we create access attributes
(implemented using Azure App roles) and connect Azure users or groups to these
App roles. If a user has all App roles assigned (either directly or via his
group membership) that are in the ConfigMap rules, access is granted.
Azure Active Directory
Every Equinor user has a Azure Active Directory account. The Flowify project
leverage this account to use the Microsoft Single-sign-on functionality to
authenticate a user that logs on to Flowify. Via the Azure Enterprise apps these users, or a group that they are a member of, can get a role assigned.
These roles are managed and created via the Azure App registrations. Note that
the Enterprise Application and the App Role are two different views on the
same Flowify "entity" with the Azure ecosystem.
App roles
The app roles are a set of roles a user acquires when getting the access token
for the application. In the Azure App registration there is
a list of roles
than can be assigned to a user. This list only shows the available roles, it does
not tie them to any user or group.
The Value field of the App role shows what string will be injected in the token
when a user gets assigned this role; they are required to define the access
permissions for the Flowify workspaces.
If a user has all the roles assigned to him that are required to access a
workspace (see next paragraph), then the Flowify server will give a user
permission to work in that workspace.
Flowify kubernetes configuration
To define a workspace for the Flowify application, we need two Kubernetes components:
-
A core.v1.Namespace
object. The name of the namespace defines the name of
the workspace.
-
A core.v1.Configmap
object. This object declares the workspace. To be
recognized as such, it is required to set the label
app.kubernetes.io/component: workspace-config
Other required fields are the roles
field. This is required to be a JSON
formatted array of strings containing the role attributes require to access
the workspace.
An example Configmap could look like
apiVersion: v1
kind: ConfigMap
metadata:
labels:
app.kubernetes.io/component: workspace-config
app.kubernetes.io/part-of: flowify
name: workspace
namespace: flowify
data:
roles: "[\"role-x\", \"role-y\", \"role-z\"]"
projectName: example-workspace
description: >
A short description of the workspace (optional).
The strings in the roles
field need to match the value
of the App role
that is required to access the workspace, as defined in the Azure App
registration's _App roles. It is also possible to specify multiple role
lists by providing an array of token arrays.
All core.v1.Configmap
objects that hold the configuration for a workspace need
to reside in the same namespace. The name of this namespace is set by the
namespace
command-line flag. The Flowify server application
needs to have permissions to read ConfigMaps from this namespace. The current
available workspaces in the deployed application can be found here.