fynca

README

Fynca Render System

Fynca is a lightweight distributed render system for Blender.

                 ┌─────────────────┐
                 │  Fynca CLI / UI │
                 └─────────────────┘
                          │
                          │
                          ▼
           ┌────────────────────────────┐
           │                            │
           │        Fynca Server        │─ ─ ─ ┐
           │                            │      ▼
           └────────────────────────────┘  ┌───────┐
                          │                │       │
                          │                │ Minio │◀ ┐
                          │                │       │
                          ▼                └───────┘  │
┌──────────────────────────────────────────────────┐
│                                                  │  │
│                 Fynca Workers                    │
│                                                  │  │
│                                                  │
│ ┌────────┐  ┌────────┐   ┌────────┐  ┌────────┐  │  │
│ │j1      │  │        │   │        │  │ j3     │  │
│ │        │  │        │   │        │  │        │  │  │
│ │        │  │j2s0    │   │j2s1    │  │        │  │
│ └────────┘  └────────┘   └────────┘  └────────┘  │─ ┘
│                                                  │
│ ┌────────┐  ┌────────┐   ┌────────┐  ┌────────┐  │
│ │        │  │        │   │        │  │        │  │
│ │        │  │        │   │        │  │        │  │
│ │ j2s2   │  │ j2s3   │   │ j4     │  │ j5     │  │
│ └────────┘  └────────┘   └────────┘  └────────┘  │
│                                                  │
└──────────────────────────────────────────────────┘

Fynca Server

The Fynca server manages job submissions. It provides a GRPC API that is used by clients to submit and manage render jobs.

Fynca CLI

The Fynca CLI manages render jobs from the command line. It is intended as an administrative interface to the Fynca server as it provides all configuration options.

NAME:
   fctl queue - queue render job

USAGE:
   fctl queue [command options] [arguments...]

OPTIONS:
   --name value                          job name
   --project-file value, -f value        path to project file
   --resolution-x value, -x value        render resolution (x) (default: 1920)
   --resolution-y value, -y value        render resolution (y) (default: 1080)
   --resolution-scale value              render resolution scale (default: 100)
   --render-start-frame value, -s value  render start frame (default: 1)
   --render-end-frame value, -e value    render end frame (default: 0)
   --render-samples value                render samples (default: 100)
   --render-use-gpu, -g                  use gpu for rendering (default: false)
   --render-priority value, -p value     set render priority (0-100) (default: 50)
   --cpu value                           specify cpu (in Mhz) for each task in the render job (default: 0)
   --memory value                        specify memory (in MB) for each task in the render job (default: 0)
   --render-slices value                 number of slices to subdivide a frame for rendering (default: 0)
   --help, -h                            show help (default: false)

NATS

Fynca uses the NATS queue for processing render jobs.

Redis

Redis is used for database storage for job metadata and user accounts.

Minio

Minio is recommended for both project storage and render task results. Minio is lightweight and works very well with Fynca but any compatible S3 system should work.

Fynca Worker

The Fynca worker is a small application that manages Blender to process jobs. It accesses the NATS queue for jobs and stores the results in MinIO.

Documentation

Overview

Copyright 2022 Evan Hazlett

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Copyright 2022 Evan Hazlett

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Index

• Constants
• type AuthenticatorConfig
• type Config
  • func DefaultConfig() *Config
  • func LoadConfig(configPath string) (*Config, error)
  • func (c *Config) GetJobTimeout() time.Duration
  • func (c *Config) GetWorkerConfig(name string) *Worker
• type RenderEngine
• type Worker

Constants

const (
	// ServerQueueGroupName is the name for the server queue group
	ServerQueueGroupName = "fynca-servers"
	// WorkerQueueGroupName is the name for the worker queue group
	WorkerQueueGroupName = "fynca-workers"

	QueueSubjectJobPriorityNormal    = "normal"
	QueueSubjectJobPriorityUrgent    = "urgent"
	QueueSubjectJobPriorityAnimation = "animation"
	QueueSubjectJobPriorityLow       = "low"

	// S3ProjectPath is the project for project files
	S3ProjectPath = "projects"
	// S3RenderPath is the s3 bucket for final renders
	S3RenderPath = "render"
	// S3JobPath is the path to the render job config
	S3JobPath = "job.json"
	// S3JobLogPath is the path to the job log
	S3JobLogPath = "job.log"
	// S3JobStatusContentType is the content type for job status objects
	S3JobContentType = "application/json"

	// WorkerTTL is the TTL for the worker heartbeat
	WorkerTTL = time.Second * 10

	// CtxTokenKey is the key stored in the context for the token
	CtxTokenKey = "token"
	// CtxTokenKey is the key stored in the context for the username
	CtxUsernameKey = "username"
	// CtxTokenKey is the key stored in the context if the user is an admin
	CtxAdminKey = "isAdmin"
	// CtxNamespaceKey is the key stored in the context for the namespace
	CtxNamespaceKey = "namespace"

	// CtxDefaultNamespace is the default key used when unauthenticated and no auth
	CtxDefaultNamespace = "default"
)

Types

type AuthenticatorConfig

type AuthenticatorConfig struct {
	// Name is the name of the authenticator
	Name string
	// Options are passed to the authenticator
	Options map[string]string
}

type Config

type Config struct {
	// GRPCAddress is the address for the grpc server
	GRPCAddress string
	// TLSCertificate is the certificate used for grpc communication
	TLSServerCertificate string
	// TLSKey is the key used for grpc communication
	TLSServerKey string
	// TLSClientCertificate is the client certificate used for communication
	TLSClientCertificate string
	// TLSClientKey is the client key used for communication
	TLSClientKey string
	// TLSInsecureSkipVerify disables certificate verification
	TLSInsecureSkipVerify bool
	// S3Endpoint is the endpoint for the S3 compatible service
	S3Endpoint string
	// S3AccessID is the S3 access id
	S3AccessID string
	// S3AccessKey is the S3 key
	S3AccessKey string
	// S3Bucket is the S3 bucket
	S3Bucket string
	// S3UseSSL enables SSL for the S3 service
	S3UseSSL bool
	// NATSUrl is the URL for the NATS server
	NATSURL string
	// NATSJobStreamName is the queue subject for the workers
	NATSJobStreamName string
	// NATSJobStatusStreamName is the queue subject for the servers
	NATSJobStatusStreamName string
	// NATSKVBucketWorkerControl is the name of the kv store in the for worker control
	NATSKVBucketWorkerControl string
	// DatabaseAddress is the address of the database
	DatabaseAddress string
	// JobTimeout is the maximum amount of time a job can take
	JobTimeout duration
	// Workers are worker specific configuration
	Workers []*Worker
	// ProfilerAddress enables the performance profiler on the specified address
	ProfilerAddress string
	// TraceEndpoint is the endpoint of the telemetry tracer
	TraceEndpoint string
	// Environment is the environment the app is running in
	Environment string
	// InitialAdminPassword is the password used when creating the initial admin account. If empty, a random one is generated.
	InitialAdminPassword string
	// Authenticator is the auth configuration
	Authenticator *AuthenticatorConfig

	// TODO: cleanup
	// JobPrefix is the prefix for all queued jobs
	JobPrefix string
	// JobPriority is the priority for queued jobs
	JobPriority int
	// JobCPU is the amount of CPU (in Mhz) that will be configured for each job
	JobCPU int
	// JobMemory is the amount of memory (in MB) that will be configured for each job
	JobMemory int
}

Config is the configuration used for the server

func DefaultConfig

func DefaultConfig() *Config

func LoadConfig

func LoadConfig(configPath string) (*Config, error)

LoadConfig returns a Fynca config from the specified file path

func (*Config) GetJobTimeout

func (c *Config) GetJobTimeout() time.Duration

func (*Config) GetWorkerConfig

func (c *Config) GetWorkerConfig(name string) *Worker

GetWorkerConfig returns the node specific configuration or default if not found

type RenderEngine

type RenderEngine struct {
	// Name is the name of the render engine
	Name string
	// Path is the path to the render engine executable
	Path string
}

type Worker

type Worker struct {
	// Name is the name of the worker to apply the configuration
	// leave blank for all
	Name string
	// MaxJobs is the maximum number of jobs this worker will perform and then exit
	MaxJobs int
	// RenderEngines is a list of enabled render engines for the worker
	RenderEngines []*RenderEngine
}