ogx-k8s-operator

README

ogx-k8s-operator

This repo hosts a Kubernetes operator that creates and manages OGX (Open GenAI Stack) servers.

Features

• Automated deployment of OGX servers
• Support for multiple distributions (includes Ollama, vLLM, and others)
• Declarative runtime config generation from OGXServer CR fields
• Customizable server configurations
• Volume management for model storage
• Kubernetes-native resource management

Table of Contents

• Quick Start
  • Installation
  • Deploying the OGX Server
  • Runtime Config via CR
• Network Policies (internal-only / Praxis-fronted)
• Monitoring
• Developer Guide
  • Prerequisites
  • Building the Operator
  • Deployment
• Running E2E Tests
• API Overview

Quick Start

Installation

You can install the operator directly from a released version or the latest main branch using kubectl apply -f.

To install the latest version from the main branch:

kubectl apply -f https://raw.githubusercontent.com/ogx-ai/ogx-k8s-operator/main/release/operator.yaml

To install a specific released version (e.g., v1.0.0), replace main with the desired tag:

kubectl apply -f https://raw.githubusercontent.com/ogx-ai/ogx-k8s-operator/v1.0.0/release/operator.yaml

Deploying the OGX Server

1. Deploy the inference provider server (ollama, vllm)

Ollama Examples:

Deploy Ollama with default model llama3.2:1b

./hack/deploy-quickstart.sh

Deploy Ollama with other model:

./hack/deploy-quickstart.sh --provider ollama --model llama3.2:7b

vLLM Examples:

This would require a secret "hf-token-secret" in namespace "vllm-dist" for HuggingFace token (required for downloading models) to be created in advance.

Deploy vLLM with default model (meta-llama/Llama-3.2-1B):

./hack/deploy-quickstart.sh --provider vllm

Deploy vLLM with GPU support:

./hack/deploy-quickstart.sh --provider vllm --runtime-env "VLLM_TARGET_DEVICE=gpu,CUDA_VISIBLE_DEVICES=0"

1. Create an OGXServer CR to get the server running. Example:

apiVersion: ogx.io/v1beta1
kind: OGXServer
metadata:
  name: ogxserver-sample
spec:
  distribution:
    name: starter
  workload:
    replicas: 1
    storage:
      size: "20Gi"
      mountPath: "/.ogx"
    overrides:
      env:
      - name: OLLAMA_INFERENCE_MODEL
        value: "llama3.2:1b"
      - name: OLLAMA_URL
        value: "http://ollama-server-service.ollama-dist.svc.cluster.local:11434"

1. Verify the server pod is running in the user defined namespace.

Local Vector Storage (inline::milvus)

To enable the inline::milvus local vector storage provider, set ENABLE_INLINE_MILVUS in spec.workload.overrides.env. This is only supported in single-worker, single-replica deployments. Milvus-Lite uses SQLite internally and does not support concurrent access from multiple processes.

Runtime Config via CR

The operator supports two ways to provide OGX config.yaml:

1. Declarative generation from CR fields (recommended) via:
   • spec.baseConfig (optional base config input)
   • spec.providers
   • spec.resources
   • spec.storage
   • spec.disabledAPIs
2. Direct override via spec.overrideConfig pointing to a user-managed ConfigMap.

When declarative fields are present and spec.overrideConfig is not set, the operator:

• Resolves base config from spec.baseConfig when set, otherwise from OCI labels com.ogx.distribution.default-config + com.ogx.config.<filename>
• Generates a final config.yaml
• Creates immutable ConfigMap ${name}-config-${hash}
• Mounts that config to /etc/ogx/config.yaml
• Injects required secret-based env vars from provider/storage secret refs
• Rolls the Deployment when referenced config/secret inputs change

The mounted runtime config always comes from either spec.overrideConfig or the generated ConfigMap. spec.baseConfig is only used as an input to generation and is never mounted into the pod directly.

Example declarative OGXServer:

apiVersion: ogx.io/v1beta1
kind: OGXServer
metadata:
  name: runtime-config-sample
spec:
  distribution:
    name: starter
  providers:
    inference:
      remote:
        openai:
          - id: openai-primary
            apiKey:
              name: openai-creds
              key: api-key
  resources:
    models:
      - name: gpt-4o-mini
        provider: openai-primary
  storage:
    sql:
      type: postgres
      connectionString:
        name: db-credentials
        key: connection-string

Ready-to-apply sample:

kubectl apply -f config/samples/example-with-generated-config.yaml

Required labels for referenced resources (same namespace as OGXServer):

metadata:
  labels:
    ogx.io/watch: "true"

See Runtime Config Generation Guide for detailed flow, examples, and troubleshooting.

Using a ConfigMap for config.yaml override

A ConfigMap can be used to store config.yaml configuration for each OGXServer. Updates to the ConfigMap will restart the Pod to load the new data.

Example to create a config.yaml ConfigMap, and an OGXServer that references it:

kubectl apply -f config/samples/example-with-configmap.yaml

spec.overrideConfig always takes precedence over declarative generation fields.

Praxis upgrade migration (3.5 → 3.6)

For operator-managed Responses/Conversations migration and the PraxisCutoverReady gate, see docs/upgrade-praxis-migration.md.

Network Policies (internal-only / Praxis-fronted)

OGX is an internal-only backend: in the target topology it is fronted by Praxis, which is the sole public entrypoint. The operator can enforce this at the network layer via Praxis-fronted mode.

Praxis-fronted mode (spec.praxisMode)

spec.praxisMode.enabled selects the per-CR operating mode:

• true — Praxis-fronted (internal-only): the locked-down NetworkPolicy below, and no external exposure. (default when spec.praxisMode is provided)
• false — legacy behavior: the pre-Praxis NetworkPolicy peers (all pods in the same namespace + the OpenShift router) and network.externalAccess.enabled is honored (an Ingress is created when enabled).

The rest of this section describes Praxis mode. In Praxis mode, for every OGXServer the operator creates a NetworkPolicy whose ingress on the service port (8321) admits traffic only from:

1. Praxis pods — identified per-CR by spec.praxisMode.praxisSelector (see below), defaulting to pods labeled app: payload-processing (the MaaS Gateway contract). This is the only application traffic OGX accepts.
2. The operator namespace — so the operator can poll OGX status (/v1/providers, /v1/version). This is control-plane traffic, not application traffic.

The broad "all pods in the same namespace" rule and the OpenShift router rule are not included, and the operator does not create any external exposure (Ingress) for OGX. Setting spec.network.externalAccess.enabled: true is not honored in Praxis mode — it is treated as false and surfaced as an admission warning (rather than a hard rejection, so existing CRs and GitOps applies keep working). status.serviceURL is populated with the internal cluster DNS endpoint; status.externalURL is empty. (In legacy mode, external access is honored and status.externalURL reflects the created Ingress.)

In Praxis mode the operator also disables the Responses API and Conversations API in OGX's generated config (they are served by Praxis instead). This is applied internally during config generation and does not mutate your CR's spec.disabledAPIs.

Every generated Praxis config replaces server.auth with upstream-header authentication using x-user-id and x-tenant-id, applies ownership-based access rules, and sets server.tenancy.mode: multi. When no declarative config fields are set, the operator preserves the distribution's default config and changes only auth, tenancy, and an explicit spec.network.port. spec.overrideConfig remains unchanged and takes precedence.

Praxis-mode readiness conditions

Because OGX and Praxis are deployed independently, the operator surfaces two Praxis-mode preconditions as status conditions rather than admission rejections (the operator does not manage Praxis, and a Praxis instance or TLS Secret may legitimately appear after OGX):

• PraxisReachable — True when the effective Praxis selector (spec.praxisMode.praxisSelector, or the fail-safe default) resolves to at least one Ready Praxis pod. When False, internal-only OGX has no valid path until Praxis is available.
• TLSConfigured — True when spec.network.tls.secretName is set and the referenced Secret exists (it must carry the ogx.io/watch: "true" label to be detected). When False, OGX is not serving its internal endpoint over the expected mTLS.

These conditions are only evaluated in Praxis mode.

Adding your own ingress rules (additive)

spec.network.policy.ingress rules are appended on top of the mandatory Praxis + operator rules — they cannot remove the lock-down:

apiVersion: ogx.io/v1beta1
kind: OGXServer
metadata:
  name: my-ogxserver
spec:
  distribution:
    name: starter
  network:
    policy:
      enabled: true
      ingress:
        - from:
            - namespaceSelector:
                matchLabels:
                  kubernetes.io/metadata.name: my-app-namespace
          ports:
            - protocol: TCP
              port: 8321

To fully disable the NetworkPolicy (emergency escape hatch), set spec.network.policy.enabled: false. In Praxis mode this removes the mandatory Praxis + operator lock-down entirely (OGX may then be reachable directly by co-located workloads, bypassing Praxis), so it is surfaced as an admission warning. Prefer additive spec.network.policy.ingress rules instead.

Configuring the Praxis peer (per-CR)

Which pods count as "Praxis" is configured per-CR via spec.praxisMode.praxisSelector — a namespace plus a Pod label selector. It becomes the NetworkPolicy ingress peer, so different OGXServers can point at different Praxis instances independently. If omitted, the operator fails safe to pods labeled app: payload-processing in the openshift-ingress namespace (never an allow-all peer):

apiVersion: ogx.io/v1beta1
kind: OGXServer
metadata:
  name: my-ogxserver
spec:
  distribution:
    name: starter
  praxisMode:
    enabled: true
    praxisSelector:
      namespace: praxis
      podSelector:
        matchLabels:
          app: payload-processing

Note (Praxis namespace): the fail-safe default peer pins the openshift-ingress namespace, matching where the PoC operator places the Praxis/extproc servers. This is provisional and pending confirmation with the MaaS team — if Praxis runs elsewhere, set praxisSelector with the correct namespace.

Field	Description
praxisMode.enabled	Praxis-fronted (internal-only) mode: true (default when praxisMode is provided) or false (force legacy). Omit praxisMode for legacy behavior.
praxisMode.praxisSelector	Per-CR NetworkPolicyPeer (namespace + pod selector) identifying the Praxis instance. Fails safe to app: payload-processing in the openshift-ingress namespace when omitted.
network.policy.enabled	When true (default), the operator creates a NetworkPolicy for the OGXServer workload. Set to false to disable it entirely.
network.policy.ingress	In Praxis mode, additional ingress rules appended to the mandatory Praxis + operator rules. In legacy mode, rules that replace the defaults.
network.externalAccess.enabled	Honored only in legacy mode. In Praxis mode it is not honored (admission warning; no external exposure created).

Monitoring

The operator provides built-in Prometheus monitoring for OGXServer instances. Monitoring is enabled by default and requires no configuration when the prometheus-operator CRDs are installed on the cluster.

When enabled, the operator creates:

• A ServiceMonitor with label monitoring.opendatahub.io/scrape: "true" for ODH/RHOAI Prometheus scraping
• A PrometheusRule with telemetry recording rules for Red Hat Insights

Configure monitoring via spec.monitoring:

apiVersion: ogx.io/v1beta1
kind: OGXServer
metadata:
  name: my-ogxserver
spec:
  distribution:
    name: starter
  monitoring:
    enabled: true        # default: true
    metricsPort: 9090    # default: 9464

Field	Description
monitoring.enabled	When true (default), the operator creates a ServiceMonitor and PrometheusRule
monitoring.metricsPort	Port for the /metrics endpoint (default: 9464)

If the prometheus-operator CRDs are not installed on the cluster, monitoring resources are silently skipped.

Ready-to-apply sample:

kubectl apply -f config/samples/example-with-monitoring.yaml

See Monitoring Integration Guide for detailed architecture, pipelines, and troubleshooting.

Image Mapping Overrides

The operator supports ConfigMap-driven image updates for OGX distribution images. This allows independent patching for security fixes or bug fixes without requiring a new operator version.

Configuration

Create or update the operator ConfigMap with an image-overrides key:

image-overrides: |
  starter-gpu: quay.io/custom/ogx:starter-gpu
  starter: quay.io/custom/ogx:starter

Configuration Format

Use the distribution name directly as the key (e.g., starter-gpu, starter). The operator will apply these overrides automatically

Example Usage

To update the OGX distribution image for all starter distributions:

kubectl patch configmap ogx-operator-config -n ogx-k8s-operator-system --type merge -p '{"data":{"image-overrides":"starter: quay.io/ogx-ai/ogx-server:latest"}}'

This will cause all OGXServer resources using the starter distribution to restart with the new image.

Developer Guide

Prerequisites

• Kubernetes cluster (v1.20 or later)
• Go version go1.24
• operator-sdk v1.39.2 (v4 layout) or newer
• kubectl configured to access your cluster
• A running inference server:
  • For local development, you can use the provided script: /hack/deploy-quickstart.sh

Building the Operator

• Prepare release files with specific versions

  make release VERSION=0.2.1 LLAMASTACK_VERSION=0.2.12
  

  This command updates distribution configurations and generates release manifests with the specified versions.

• Custom operator image can be built using your local repository

  make image IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>
  

  The default image used is quay.io/ogx-ai/ogx-k8s-operator:latest when not supply argument for make image To create a local file local.mk with env variables can overwrite the default values set in the Makefile.

• Building multi-architecture images (ARM64, AMD64, etc.)

  The operator supports building for multiple architectures including ARM64. To build and push multi-arch images:

  make image-buildx IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>
  

  By default, this builds for linux/amd64,linux/arm64. You can customize the platforms by setting the PLATFORMS variable:

  # Build for specific platforms
  make image-buildx IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag> PLATFORMS=linux/amd64,linux/arm64
  
  # Add more architectures (e.g., for future support)
  make image-buildx IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag> PLATFORMS=linux/amd64,linux/arm64,linux/s390x,linux/ppc64le
  

  Note:

  • The image-buildx target works with both Docker and Podman. It will automatically detect which tool is being used.

  • Native builds in CI: CI workflows use a matrix strategy with native runners for each architecture (AMD64 and ARM64). Each architecture is built on its own runner, avoiding QEMU emulation entirely. Per-architecture images are pushed separately, then combined into a single multi-arch manifest list. This ensures CGO_ENABLED=1 with full OpenSSL FIPS support for all architectures.

  • Local cross-compilation: For local development, the Dockerfile uses --platform=$BUILDPLATFORM to run Go compilation natively on the build host. When cross-compiling (e.g., building ARM64 on an AMD64 host), CGO_ENABLED=0 is used with pure Go FIPS (via GOEXPERIMENT=strictfipsruntime). Native local builds use CGO_ENABLED=1 with full OpenSSL FIPS support.

  • FIPS adherence: All CI-produced images use CGO_ENABLED=1 with full OpenSSL FIPS support via native builds on architecture-matched runners.

  • For Docker: Multi-arch builds require Docker Buildx. Ensure Docker Buildx is set up:

    docker buildx create --name x-builder --use
    

  • For Podman: Podman 4.0+ supports podman buildx (experimental). If buildx is unavailable, the Makefile will automatically fall back to using podman's native manifest-based multi-arch build approach.

  • The resulting images are multi-arch manifest lists, which means Kubernetes will automatically select the correct architecture when pulling the image.

  CI Build Targets:

  The CI workflows use the following Makefile targets for the matrix-based build strategy:

  # Build and push a single-arch image (used by each matrix job on its native runner)
  make image-build-push-single PLATFORM=linux/amd64 IMG=quay.io/<username>/ogx-k8s-operator:<tag>-amd64
  
  # Create a multi-arch manifest from per-arch images (used by the final manifest job)
  make image-create-manifest IMG=quay.io/<username>/ogx-k8s-operator:<tag> \
    ARCH_IMGS="quay.io/<username>/ogx-k8s-operator:<tag>-amd64 quay.io/<username>/ogx-k8s-operator:<tag>-arm64"
  

• Building ARM64-only images

  To build a single ARM64 image (useful for testing or ARM-native systems):

  make image-build-arm IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>
  make image-push IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>
  

  This works with both Docker and Podman.

• Once the image is created, the operator can be deployed directly. For each deployment method a kubeconfig should be exported

  export KUBECONFIG=<path to kubeconfig>
  

Deployment

Deploying on vanilla Kubernetes (cert-manager)

• Deploy the created image in your cluster using following command:

  make deploy IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>
  

• To remove resources created during installation use:

  make undeploy
  

Deploying on OpenShift

OpenShift clusters use the built-in service-serving-cert-signer for webhook TLS (no cert-manager required):

make deploy-openshift IMG=quay.io/<username>/ogx-k8s-operator:<custom-tag>

• To remove resources:

  make undeploy-openshift
  

Running E2E Tests

The operator includes end-to-end (E2E) tests to verify the complete functionality of the operator. To run the E2E tests:

1. Ensure you have a running Kubernetes cluster
2. Run the E2E tests using one of the following commands:
   • If you want to deploy the operator and run tests:

     make deploy test-e2e
     

   • If the operator is already deployed:

     make test-e2e
     

The make target will handle prerequisites including deploying ollama server.

API Overview

Please refer to api documentation