README
¶
OffGrid LLM
Run AI models locally. No cloud. No API keys. Complete privacy.
Install
curl -fsSL https://raw.githubusercontent.com/takuphilchan/offgrid-llm/main/install.sh | bash
Run
offgrid run llama3
That's it. The model downloads automatically and you start chatting.
Why OffGrid?
| Feature | OffGrid | Ollama | LM Studio |
|---|---|---|---|
| Air-gapped USB deployment | Yes | - | - |
| P2P model sharing | Yes | - | - |
| Built-in RAG | Yes | - | - |
| Multi-user + audit logs | Yes | - | - |
| AI Agents with MCP | Yes | - | Partial |
| Voice assistant | Yes | - | - |
| OpenAI-compatible API | Yes | Yes | Yes |
Built for:
- Healthcare (HIPAA) - patient data never leaves
- Government - air-gapped deployment
- Enterprise - audit logging and multi-user
- Remote sites - ships, rigs, expeditions
- Research - unlimited use, zero API costs
Quick Start
Web UI
offgrid serve
Chat with AI models locally - full privacy, no cloud
Download, manage, and switch between models with size info
CLI Chat
offgrid run llama3 # Chat with Llama 3
offgrid run mistral # Chat with Mistral
offgrid run codellama # Chat with Code Llama
Python
pip install offgrid
import offgrid
client = offgrid.Client()
response = client.chat("Hello!")
print(response)
Models
Built-in shortcuts:
| Alias | Model | RAM |
|---|---|---|
tiny |
TinyLlama 1.1B | 2 GB |
phi |
Phi 3 Mini | 4 GB |
llama3 |
Llama 3.2 3B | 4 GB |
qwen |
Qwen 2.5 3B | 4 GB |
mistral |
Mistral 7B | 8 GB |
codellama |
Code Llama 7B | 8 GB |
offgrid alias list # See all shortcuts
offgrid search llama # Search HuggingFace
offgrid list # Show installed
Features
AI Agents
Autonomous task execution with tool use:
offgrid agent chat --template coder
offgrid agent run "Analyze this data and create a chart"
Templates: researcher, coder, analyst, writer, sysadmin, planner
Knowledge Base (RAG)
Chat with your documents:
offgrid kb add ./documents/
offgrid kb search "project requirements"
Voice
Speech-to-text and text-to-speech:
offgrid audio transcribe recording.wav
offgrid audio speak "Hello world" --output hello.wav
Offline Transfer
Export models to USB for air-gapped systems:
offgrid export llama3 /media/usb
offgrid import /media/usb
Audit Logs
Tamper-evident security logging:
offgrid audit show
offgrid audit export-csv report.csv
offgrid audit verify
Documentation
| Guide | Description |
|---|---|
| Quick Start | Get running in 3 minutes |
| CLI Reference | All commands |
| API Reference | REST endpoints |
| Python SDK | Python library |
| Agents | AI agent system |
| RAG | Knowledge base |
| Docker | Container deployment |
System Requirements
| RAM | Models |
|---|---|
| 4 GB | TinyLlama, SmolLM, Phi 3 Mini |
| 8 GB | Llama 3.2 3B, Qwen 2.5 3B, Gemma 2B |
| 16 GB | Mistral 7B, Llama 3.1 8B, Code Llama |
| 32 GB+ | Llama 3 70B, Mixtral, DeepSeek |
GPU optional. Supports NVIDIA (CUDA), AMD (ROCm), Apple Silicon (Metal).
Contributing
git clone https://github.com/takuphilchan/offgrid-llm.git
cd offgrid-llm
go build -o bin/offgrid ./cmd/offgrid
./bin/offgrid serve
See Contributing Guide.
License
MIT - See LICENSE
Built with llama.cpp
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
offgrid
command
|
|
|
dev
|
|
|
examples
command
|
|
|
internal
|
|
|
audio
Package audio provides speech-to-text (ASR) and text-to-speech (TTS) capabilities using whisper.cpp for ASR and Piper for TTS, both running fully offline.
|
Package audio provides speech-to-text (ASR) and text-to-speech (TTS) capabilities using whisper.cpp for ASR and Piper for TTS, both running fully offline. |
|
audit
Package audit provides security audit logging for air-gapped and compliance environments.
|
Package audit provides security audit logging for air-gapped and compliance environments. |
|
config
Package config provides configuration export/import for fleet deployment.
|
Package config provides configuration export/import for fleet deployment. |
|
degradation
Package degradation provides graceful degradation under resource pressure.
|
Package degradation provides graceful degradation under resource pressure. |
|
docs
Package docs provides offline documentation and help system.
|
Package docs provides offline documentation and help system. |
|
inference
Package inference provides inference-related functionality including load balancing
|
Package inference provides inference-related functionality including load balancing |
|
integrity
Package integrity provides offline model verification using bundled SHA256 hashes.
|
Package integrity provides offline model verification using bundled SHA256 hashes. |
|
logging
Package logging provides structured logging for OffGrid LLM.
|
Package logging provides structured logging for OffGrid LLM. |
|
maintenance
Package maintenance provides disk space management and cleanup for long-running edge deployments with limited storage.
|
Package maintenance provides disk space management and cleanup for long-running edge deployments with limited storage. |
|
mcp
Package mcp provides Model Context Protocol server and client functionality
|
Package mcp provides Model Context Protocol server and client functionality |
|
p2p
Package p2p provides secure P2P model transfer with encryption and resume support.
|
Package p2p provides secure P2P model transfer with encryption and resume support. |
|
power
Package power provides power and battery awareness for edge devices.
|
Package power provides power and battery awareness for edge devices. |
|
prewarm
Package prewarm provides model pre-warming on boot for faster first-request response.
|
Package prewarm provides model pre-warming on boot for faster first-request response. |
|
rag
Package rag provides distributed RAG index functionality
|
Package rag provides distributed RAG index functionality |
|
tools
Package tools provides a plugin system for custom tools
|
Package tools provides a plugin system for custom tools |
|
users
Package users provides user management and authentication
|
Package users provides user management and authentication |
|
validation
Package validation provides input validation utilities for API requests.
|
Package validation provides input validation utilities for API requests. |
|
watchdog
Package watchdog provides process monitoring and automatic recovery for llama-server and other critical services in offline/edge deployments.
|
Package watchdog provides process monitoring and automatic recovery for llama-server and other critical services in offline/edge deployments. |
|
pkg
|
|
Click to show internal directories.
Click to hide internal directories.