m3tal-core

README

M3TAL System Documentation

Overview

This document provides technical details and operational procedures for the M3TAL system.

Prerequisites

Docker Engine and Docker Compose V2 are strictly REQUIRED and must be installed prior to M3TAL installation and operation. M3TAL internally orchestrates Docker containers using Docker Engine and Docker Compose V2.

Installation

To install the M3TAL CLI and API daemon on a Debian-based system, follow these steps:

# 1. Add the GPG signing key
curl -fsSL https://jakej985-rgb.github.io/m3tal-core/KEY.gpg | sudo gpg --dearmor -o /usr/share/keyrings/m3tal-archive-keyring.gpg

# 2. Add the APT repository
echo "deb [signed-by=/usr/share/keyrings/m3tal-archive-keyring.gpg] https://jakej985-rgb.github.io/m3tal-core stable main" | sudo tee /etc/apt/sources.list.d/m3tal.list

# 3. Install
sudo apt update && sudo apt install -y m3tal

Firewall Considerations

If you are using Traefik for public-facing access (i.e., DASHBOARD_EXPOSE_MODE=traefik or exposing other services via Traefik), ensure that host port 80 (and 443 if HTTPS is configured) is open in your firewall (e.g., ufw allow 80/tcp).

Components

The M3TAL system comprises the following core components:

• CLI binary (/usr/bin/m3tal): A unified Go binary serving as the single entrypoint for all M3TAL operations.
• API daemon (m3tal-api.service): A Go binary running as a systemd service, listening on host-local port 8080. It manages Docker interactions, the internal state database, and API routes.
• Dashboard container (m3tal-dashboard): A Python/Flask application running within a Docker container, listening internally on port 8082. It communicates with the M3TAL API daemon via http://host.docker.internal:8080.
• Traefik gateway (routing-compose.yml): A Docker container acting as a reverse proxy, exposing services by domain name on host port 80. It uses a file provider for dynamic routing configuration.
• Cloudflared (routing-compose.yml): An optional Cloudflare tunnel Docker container, facilitating zero-configuration internet access for services.

Filesystem Contract

The following table details the critical filesystem paths and their purposes within the M3TAL system:

Path	Purpose
/etc/m3tal/.env	The primary configuration file, managed by the m3tal config wizard command.
/var/lib/m3tal/state.db	The SQLite state database, automatically created and managed by the API daemon.
/opt/m3tal/stack/	The canonical directory containing all Docker Compose files and Traefik configuration.
/docker	A symlink to /opt/m3tal/stack/. This is the user-facing path for all stack operations.
/docker/users.json	The credential store for the M3TAL Dashboard, managed by the m3tal dashpass command.

Deployment Lifecycle

M3TAL orchestrates Docker containers using Docker Compose V2. The m3tal up command is a wrapper around docker compose that operates on all *-compose.yml files located within the /docker/ directory, effectively deploying each as an independent stack.

The /opt/m3tal/stack/ directory serves as the canonical source of truth where all Docker Compose stack files and associated configurations reside. For user convenience, a symlink alias, /docker, points to /opt/m3tal/stack/. All user-facing stack operations, such as adding new services, should interact with the /docker/ directory.

Adding a New Stack

To deploy a new Docker Compose stack:

1. Place your Docker Compose file (e.g., my-stack-compose.yml) directly into the /docker/ directory. This file will be automatically included by m3tal up.
2. Ensure any required environment variables are configured in /etc/m3tal/.env using m3tal config wizard or m3tal config set KEY value.
3. Run m3tal up to deploy your new stack along with all other managed services.

Dashboard Access

The M3TAL Dashboard supports two primary access modes, determined by the DASHBOARD_EXPOSE_MODE variable in /etc/m3tal/.env.

Local Mode (Default)

• Configuration: DASHBOARD_EXPOSE_MODE=local (This is the default setting upon initial installation).
• Mechanism: M3TAL uses the m3tal-compose.local.yml override file, which adds a direct port binding to the dashboard container (${DASHBOARD_PORT:-8082}:8082). Traefik is not involved in this access method.
• Access: The dashboard is directly accessible via http://HOST_IP:8082 or http://localhost:8082.
• Behavior for new users: A new user performing a default installation will access the dashboard directly via port 8082, as this aligns with the DASHBOARD_EXPOSE_MODE=local setting.

Traefik Mode

• Configuration: DASHBOARD_EXPOSE_MODE=traefik
• Mechanism: M3TAL uses the m3tal-compose.traefik.yml override file, which applies Traefik labels to the dashboard container. Traefik (if running via m3tal up) then routes traffic for dash.DOMAIN to the dashboard's internal port 8082.
• Access: The dashboard is accessible via http://dash.DOMAIN (e.g., http://dash.example.com). This mode requires Traefik to be running.

Traefik Gateway

Traefik automatically discovers and routes traffic to Docker services by interpreting Traefik labels defined within their Docker Compose service definitions. Crucially, services are not exposed by Traefik by default and require traefik.enable=true along with other relevant labels to be discoverable and routable.

Traefik also utilizes dynamic configuration files, such as dynamic/api.yml, to route requests to services that are not directly Docker containers but listen on host-local ports. For example, api.DOMAIN is routed to the M3TAL Go API daemon, which listens on the host-local port 8080, via http://host.docker.internal:8080. Similarly, dash.DOMAIN routes to the dashboard container (when DASHBOARD_EXPOSE_MODE=traefik) by targeting its internal port 8082 based on its Traefik labels.

Exposing a Custom User Service via Traefik

To expose a custom Docker Compose service (e.g., defined in my-app-compose.yml) via Traefik, add the following labels to your service definition:

# /docker/my-app-compose.yml
services:
  my-app:
    image: nginx:alpine
    container_name: my-custom-nginx
    labels:
      - "traefik.enable=true" # Enable Traefik for this service
      - "traefik.http.routers.myapp.rule=Host(`app.DOMAIN`)" # Route based on hostname
      - "traefik.http.services.myapp.loadbalancer.server.port=80" # Target internal container port
      - "traefik.http.routers.myapp.entrypoints=web" # Use the 'web' entrypoint (HTTP port 80)
    networks:
      - proxy # Connect to the Traefik proxy network

networks:
  proxy:
    external: true # Ensure the 'proxy' network is recognized as external

Service Management

The M3TAL API daemon is managed as a systemd service. Use the following systemctl commands for interaction:

• Check status: systemctl status m3tal-api.service
• Restart service: systemctl restart m3tal-api.service
• View logs: journalctl -u m3tal-api.service -f

Quick Demo

• Start the M3TAL Dashboard: To specifically start or update only the dashboard container, use:

  m3tal dash up
  

  This command will download the latest dashboard compose files, read your DASHBOARD_EXPOSE_MODE setting, and start the dashboard with the appropriate configuration.
• Deploy all M3TAL stacks: To orchestrate and deploy all other stacks in the /docker/ directory, including the routing stack (Traefik) and any user-defined compose files you've placed there, use:

  m3tal up
  

Port Map

The following table lists the primary network ports utilized by the M3TAL system:

Port	Service	Access	Description
80	Traefik HTTP entry point	Public	The public-facing HTTP port for services exposed via Traefik.
8080	M3TAL API daemon (Go)	Host-local	The internal port the M3TAL API daemon listens on.
8081	Traefik dashboard	Host-local only	The internal Traefik dashboard port, accessible only from the host machine.
8082	M3TAL Dashboard	Direct port (local mode) or via Traefik (traefik mode)	The port the M3TAL Dashboard container listens on internally. Access method depends on DASHBOARD_EXPOSE_MODE.

Note: These are the primary M3TAL-managed ports. User-added Docker Compose stacks may expose additional ports on their own, depending on their configurations.

Configuration Variables

M3TAL configuration is managed via environment variables stored in /etc/m3tal/.env. These variables can be set and modified using the m3tal config wizard or m3tal config set KEY value commands.