web-search-mcp

README

web-search-mcp 🕸️

MCP server for intelligent web search — no API keys required, with local AI-powered semantic analysis running on your machine.

---

💡 Why This Exists & The Magic

When an AI assistant needs to find information on the web, the typical approach is wasteful:

❌ BAD APPROACH (without web-search-mcp):
   AI: "find me Go trends 2025"
   → AI invokes a search engine
   → AI reads ALL pages in full
   → AI loads MEGABYTES of text into context
   → Wastes tons of tokens analyzing everything
   → Expensive, slow, inefficient

With web-search-mcp the analysis happens locally:

✅ GOOD APPROACH (with web-search-mcp):
   AI: "find me Go trends 2025"
   → web_search_analyze:
      ├─ DuckDuckGo → 8 URLs (0 tokens)
      ├─ chromedp + readability → clean text (0 tokens)
      └─ Ollama embedding → semantic ranking (0 tokens!)
   → AI gets: [{url, title, relevance: 66%, snippet}, ...]
   → AI PICKS only 1-2 best pages
   → AI reads ONLY their full text (few tokens)
   → Fast, cheap, intelligent

The killer feature: semantic analysis (comparing query meaning with page meaning) is done locally via Ollama — it consumes zero AI API tokens! The AI receives a pre-ranked list and can decide which page to read without burning through kilobytes of garbage.

---

🚀 Quick Start

1. Install dependencies

# Ollama + embedding model (required for semantic analysis)
ollama pull embeddinggemma:latest
ollama serve

# Chromium (for JavaScript-rendered pages, optional)
# Arch:  sudo pacman -S chromium
# Ubuntu: sudo apt install chromium-browser
# macOS:  brew install chromium

2. Build and run

git clone https://github.com/kirill-scherba/web-search-mcp
cd web-search-mcp
go build -o web-search-mcp .

# Start — server listens on stdin/stdout for MCP
./web-search-mcp

3. Connect to an AI agent

Any MCP-capable agent can use this server: it speaks the standard stdio transport, so the only thing a client needs is a command to start and the name it should call it by. The settings file differs per client, the entry does not.

The entry itself (same everywhere, only the path changes):

{
  "mcpServers": {
    "web-search-mcp": {
      "command": "/path/to/web-search-mcp",
      "args": [],
      "env": {},
      "disabled": false
    }
  }
}

Cline

Paste the entry into the Cline MCP settings file, then press "Restart MCP Servers" in the Cline panel (or reload the page). Optionally add the tool names to autoApprove so searches run without a confirmation prompt:

"autoApprove": ["web_search", "web_search_analyze", "web_fetch", "web_semantic_search"]

Codex

Codex reads MCP servers from its own configuration and starts them the same way:

# ~/.codex/config.toml
[mcp_servers.web-search-mcp]
command = "/path/to/web-search-mcp"
args = []

Restart the session; the tools appear as web_search, web_search_analyze, web_fetch and web_semantic_search.

Claude

Claude Desktop takes the same JSON entry in claude_desktop_config.json (Settings → Developer → Edit Config), after which the application is restarted:

{
  "mcpServers": {
    "web-search-mcp": {
      "command": "/path/to/web-search-mcp",
      "args": []
    }
  }
}

On Linux and macOS the file lives at ~/Library/Application Support/Claude/ (macOS) or ~/.config/Claude/ (Linux), on Windows under %APPDATA%\Claude\.

Windows users: the command is the path to web-search-mcp.exe, and forward slashes or doubled backslashes both work in JSON.

---

🎮 How to Use — Examples

Just tell the AI what you need. The tools are invoked automatically:

🔍 Simple URL Search

User: "find information about Go in 2025"
AI calls: web_search("Go 2025 trends", limit=8)
→ Gets: URL list + snippets
→ Shows: top 3 links with descriptions

🏆 Smart Search with Semantic Ranking (saves tokens!)

User: "analyze what people are writing about Go trends"
AI calls: web_search_analyze("Go programming trends 2025", limit=8)
→ 1. DuckDuckGo → fetches 8 URLs
→ 2. Downloads each page (in parallel!)
→ 3. Ollama embeds each page's text
→ 4. Ollama embeds the query
→ 5. Cosine similarity → ranks by meaning
→ 6. Saves everything to DB for future searches
→ Returns: [{url, title, relevance: 66%, snippet}, ...]

AI: "Most relevant results:
     1. Go Developer Survey 2025 — 66% match
     2. JetBrains Go Ecosystem — 57%
     3. GeeksForGeeks Future of Go — 51%
     Want me to read the first article in full?"

The magic: AI doesn't waste tokens analyzing all pages — that work is done by local Ollama.

📄 Read Full Page Content

User: "open the developer survey article"
AI calls: web_fetch("https://go.dev/blog/survey2025")
→ Checks cache (if already fetched — returns from DB)
→ If not: chromedp + readability → clean text
→ Returns: full article text (18K chars)
→ AI reads it and answers your question

🔎 Semantic Search Over Previously Indexed Content

User: "what was there about Go popularity?"
AI calls: web_semantic_search("Go popularity among developers")
→ Ollama embeds the query
→ Searches all stored chunks in DB
→ Returns: relevant text fragments with match %

AI: "From the Go Developer Survey 2025:
     Go ranks in the top 5 languages by developer
     satisfaction (relevance: 48%)"

---

🛠️ Tool Reference

Tool	What it does	When to use
web_search	Searches DuckDuckGo, returns URL + snippet	Need a quick list of links
web_search_analyze 🏆	Search + fetch + semantic analysis	Primary tool! Saves tokens
web_fetch	Fetches page, renders JS, extracts text	Need full content of a specific page
web_semantic_search	Semantic search over indexed content	"What was there about X?" — no need to re-google

---

💰 Why This Saves Tokens

Without web-search-mcp:

1. AI → search → gets HTML result list
2. AI → reads first result (full HTML)
3. AI → if wrong, reads second... and so on
4. Each page = thousands of tokens
5. Total: 10-50K tokens per search

With web-search-mcp:

1. AI → web_search_analyze → gets ranked list
2. AI picks 1-2 best pages based on relevance
3. AI → web_fetch → reads only the best one
4. Total: 3-5K tokens per search (10x savings!)

---

⚙️ Configuration

./web-search-mcp \
  --db ~/.config/web-search-mcp/web_search.db \
  --ollama-url http://localhost:11434 \
  --embedding-model embeddinggemma:latest \
  --chromium-path /usr/bin/chromium

# Or via environment variables
export OLLAMA_BASE_URL=http://localhost:11434
export CHROME_PATH=/usr/bin/chromium
./web-search-mcp

CLI flags

Flag	Default	Description
--db	~/.config/web-search-mcp/web_search.db	Database path (cache)
--ollama-url	http://localhost:11434	Ollama endpoint
--embedding-model	embeddinggemma:latest	Embedding model
--chromium-path	auto-detect	Chromium binary path
-h	—	Help

---

🏗️ Architecture

                    ┌─────────────────────┐
                    │    MCP Client (AI)   │
                    └──────────┬──────────┘
                               │ JSON-RPC / stdin-stdout
                    ┌──────────▼──────────┐
                    │   web-search-mcp    │
                    │  ┌────────────────┐ │
                    │  │  web_search    │─┼──→ DuckDuckGo
                    │  ├────────────────┤ │
                    │  │  web_fetch     │─┼──→ chromedp + readability
                    │  ├────────────────┤ │
                    │  │  search_analyze│─┼──→ search + fetch + embed
                    │  ├────────────────┤ │
                    │  │ semantic_search│─┼──→ vector search (libSQL)
                    │  └────────────────┘ │
                    └──────┬────────┬─────┘
                           │        │
                    ┌──────▼┐  ┌────▼──────┐
                    │Ollama │  │  libSQL   │
                    │embed  │  │  (cache)  │
                    └───────┘  └───────────┘

Stack: Go + chromedp + go-readability + Ollama + go-libsql + mcp-go Transport: stdin/stdout (JSON-RPC 2.0) License: MIT

---

🎨 Logo Prompt

Send this prompt to Gemini or any image generation AI to create a logo for the project:

Minimalist logo for "web-search-mcp" — an MCP server for intelligent web search.

A stylized spider web (representing "web" and "MCP") with a magnifying glass in the center. The web is drawn with thin, clean cyan/blue lines on a dark background. In the center, the magnifying glass overlaps the web intersection — the glass lens contains a subtle glow effect symbolizing "intelligence" and "AI." The style is flat vector, modern, cyberpunk-lite. No text needed. Simple icon, suitable for a 128x128 app icon.

Style: flat vector, dark theme, cyan/blue accent palette (#00ADD8 for Go, #3B82F6). Minimal lines, tech aesthetics.

---

🧪 Quick Test

# Direct test (without MCP)
printf '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}\n{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}\n' | timeout 3 ./web-search-mcp 2>/dev/null

# If you see 4 tools — everything works!

---

License

MIT © Kirill Scherba

---

Built with Go + chromedp + go-readability + Ollama + go-libsql + mcp-go.