techdetect

README

TechDetect

A enhanced web technology detection engine inspired by ProjectDiscovery's wappalyzergo, featuring improved fingerprint organization, MongoDB-style query evaluation, and browser-based detection capabilities.

Key Features

• 🎯 Dual-Stage Detection: HTTP-based (fast) + Browser-based (accurate) detection
• 🖼️ Favicon Hash Detection: Shodan mmh3 and OWASP/WhatWeb MD5 hashes of /favicon.ico and HTML icon links
• 📁 Organized Fingerprints: Technologies categorized into separate files for easier management
• 📦 Embedded Fingerprints: All fingerprints bundled into binary - no external dependencies needed
• 🔍 MongoDB-Style Queries: Advanced query evaluation with 9 operators ($or, $and, $not, $nor, $regex, $eq, $ne, $exists, $in, $nin) for precise detection and reduced false positives
• 🌐 Browser Detection: Chromedp integration for JavaScript execution and accurate version extraction (e.g., Next.js, React)
• 🔄 Smart Redirect Handling: Follows same-domain redirects with data accumulation
• 🚀 Pipeline-Friendly: Clean JSON/JSONL output for seamless integration with other tools
• 🌐 Proxy Support: HTTP and SOCKS5 proxies with authentication
• 🔒 SSL Options: Skip verification for self-signed certificates

Installation

Option 1: Download Binary (Recommended)

Download the pre-built binary from releases. All fingerprints are embedded - no additional files needed!

# Just run it
./techdetect https://example.com

Option 2: Build from Source

git clone <repository-url>
cd wappalyzergo_extra
go mod tidy
go build -o techdetect ./cmd/techdetect

Quick Start

# Basic HTTP detection
./techdetect https://example.com

# With browser detection
./techdetect -browser https://example.com

# JSON output
./techdetect -format json https://example.com

# JSONL output (streaming)
./techdetect -format jsonl https://example.com

# Skip SSL verification
./techdetect -insecure true https://self-signed.example.com

# Using HTTP proxy
./techdetect -proxy http://proxy.example.com:8080 https://example.com

# Using HTTP proxy with authentication
./techdetect -proxy http://user:pass@proxy.example.com:8080 https://example.com

# Using SOCKS5 proxy
./techdetect -proxy socks5://proxy.example.com:1080 https://example.com

# Using SOCKS5 proxy with authentication
./techdetect -proxy socks5://user:pass@proxy.example.com:1080 https://example.com

# Pipe input from other tools
cat urls.txt | ./techdetect -format jsonl
echo https://example.com | ./techdetect -format json

Output Formats

Text (Human-Readable)

🔍 https://nextjs.org - Detected 11 technologies:

  ✓ Turbopack
  ✓ Next.js (v16.2.0-canary.19)
  ✓ Framer Motion
  ✓ Webpack
  ✓ HSTS
  ✓ Next.js App Router
  ✓ Vercel Speed Insights
  ✓ Node.js
  ✓ Vercel
  ✓ React
  ✓ Vercel Analytics

JSON (Batch)

{
  "results": [
    {
      "url": "https://example.com",
      "technologies": {
        "React": "18.2.0",
        "Next.js": "13.4.0"
      },
      "mode": "http"
    }
  ]
}

JSONL (Streaming)

{"url":"https://example.com","technologies":{"React":"18.2.0"},"mode":"http"}
{"url":"https://another.com","technologies":{"Vue.js":"3.0"},"mode":"http"}

Command-Line Options

Flag	Description	Default
-url	Target URL to analyze	-
-format	Output format: text, json, or jsonl	text
-browser	Enable browser detection (slower but more accurate)	false
-insecure	Skip SSL certificate verification	false
-fingerprints	Path to fingerprints directory	./data/fingerprints
-proxy	Proxy URL (http://[user:pass@]host:port or socks5://[user:pass@]host:port)	-

Integration with ProjectDiscovery Tools

# Subdomain discovery → Tech detection
subfinder -d target.com | httpx -silent | ./techdetect -format jsonl

# Port scanning → Tech detection
echo target.com | naabu -silent | ./techdetect -format jsonl

# URL probing → Tech detection
cat urls.txt | httpx -silent | ./techdetect -format jsonl -browser

Architecture

┌─────────────────────────────────────────┐
│       Technology Detection Engine        │
└─────────────────────────────────────────┘
                    │
        ┌───────────┴───────────┐
        │                       │
┌───────▼────────┐    ┌────────▼────────┐
│  HTTP Detector  │    │ Browser Detector│
│                 │    │                 │
│ • Path classify │    │ • Chromedp      │
│ • Favicon hash  │    │ • JS execution  │
│ • Retry logic   │    │ • Version       │
│ • Query eval    │    │   extraction    │
│ • Redirects     │    │                 │
└────────┬────────┘    └────────┬────────┘
         │                      │
         └──────────┬───────────┘
                    │
         ┌──────────▼──────────┐
         │   Query Evaluator    │
         │                      │
         │ MongoDB-Style:       │
         │ $or, $and, $not,     │
         │ $nor, $regex, $eq,   │
         │ $ne, $exists, $in,   │
         │ $nin                 │
         └─────────────────────┘

Fingerprint Structure

Fingerprints are organized by technology category in data/fingerprints/:

data/
├── categories.json
└── fingerprints/
    ├── 001-cms.json
    ├── 012-javascript-frameworks.json
    ├── 035-maps.json
    └── ...

Each fingerprint supports MongoDB-style queries for precise detection:

{
  "apps": {
    "WordPress": {
      "cats": [1, 11],
      "implies": ["MySQL", "PHP"],
      "paths": [
        {
          "path": "/",
          "detect": {
            "$or": [
              {"body": {"$regex": "/wp-content/"}},
              {"headers.x-pingback": {"$regex": "/xmlrpc\\.php$"}}
            ]
          },
          "extract_version": [
            {"body": "<meta[^>]+WordPress\\s+([\\d.]+)"}
          ]
        }
      ],
      "browser": [
        {
          "path": "/",
          "detection": "return typeof wp !== 'undefined';",
          "version": "try { return String(wp.version || ''); } catch(e){ return ''; }"
        }
      ]
    }
  }
}

Detection Operators

Logical Operators

• $or - Match ANY condition
• $and - Match ALL conditions
• $not - Negate condition
• $nor - Match NONE

Comparison Operators

• $regex - Regular expression match (supports version extraction)
• $eq - Exact equality
• $ne - Not equal
• $exists - Field existence check
• $in - Value in array
• $nin - Value NOT in array

See SCHEMA_GUIDE.md for detailed documentation.

Advanced Features

Smart Redirect Detection

• Follows same-domain redirects (max 3)
• Accumulates bodies and headers from all redirect steps
• Performs technology detection at each redirect

Fatal Error Detection

• Stops immediately on fatal network errors (no such host, network unreachable)
• Avoids wasting time on unreachable domains

Browser Detection

• Executes JavaScript to detect client-side technologies
• Extracts accurate version information
• Conditional execution based on HTTP detection results

Performance

• HTTP Detection: < 2s for 10 paths
• Browser Detection: < 5s per page
• Memory Usage: < 100MB for 100 technologies

Project Structure

.
├── cmd/
│   └── techdetect/
│       └── main.go              # CLI application
├── data/
│   ├── categories.json          # Technology categories
│   └── fingerprints/
│       └── *.json               # Organized fingerprint files
├── browser_detector.go          # Browser-based detection (Chromedp)
├── detector.go                  # Main detector orchestration
├── favicon.go                   # Favicon mmh3 and MD5 hash detection
├── http_detector.go             # HTTP-based detection
├── loader.go                    # Fingerprint loader
├── query.go                     # MongoDB-style query evaluator
├── types.go                     # Core data structures
├── go.mod
└── README.md

Credits

This project is inspired by and uses fingerprint data converted from ProjectDiscovery's wappalyzergo.

Key Enhancements:

• Reorganized fingerprints into categorized files for better maintainability
• Implemented MongoDB-style query evaluation for more accurate detection
• Added browser detection capabilities with JavaScript execution
• Enhanced redirect handling and error detection

License

MIT

Documentation

Index

• Constants
• func ComputeFaviconHash(data []byte) int32
• func ComputeFaviconMD5(data []byte) string
• func ShouldRunBrowserDetection(techName string, results map[string]*Technology, probe BrowserProbe) bool
• type BatchResults
• type BrowserDetector
  • func NewBrowserDetector() *BrowserDetector
  • func NewBrowserDetectorWithOptions(proxyURL string) *BrowserDetector
  • func (bd *BrowserDetector) DetectBrowser(baseURL string, fingerprints map[string]Fingerprint, ...) (map[string]*Technology, error)
• type BrowserPathClassification
  • func ClassifyBrowserByPath(fingerprints map[string]Fingerprint) []BrowserPathClassification
• type BrowserProbe
  • func (bp *BrowserProbe) HasDetectionCapability() bool
  • func (bp *BrowserProbe) HasVersionCapability() bool
• type DetectResult
• type DetectionContext
• type Detector
  • func NewDetector(fingerprintsDir string) (*Detector, error)
  • func NewDetectorWithOptions(fingerprintsDir string, insecureSkipVerify bool, proxyURL string) (*Detector, error)
  • func (d *Detector) Detect(url string, useBrowser bool) (*DetectResult, error)
  • func (d *Detector) DetectFull(url string) (*DetectResult, error)
  • func (d *Detector) DetectHTTPOnly(url string) (*DetectResult, error)
• type Fingerprint
• type FingerprintDB
• type HTTPDetector
  • func NewHTTPDetector() *HTTPDetector
  • func NewHTTPDetectorWithOptions(insecureSkipVerify bool, proxyURL string) *HTTPDetector
  • func (hd *HTTPDetector) DetectHTTP(baseURL string, fingerprints map[string]Fingerprint) (map[string]*Technology, []string)
• type Loader
  • func NewLoader(fingerprintsDir string) *Loader
  • func (l *Loader) LoadAll() (map[string]Fingerprint, error)
• type PathClassification
  • func ClassifyByPath(fingerprints map[string]Fingerprint) []PathClassification
• type PathProbe
• type QueryEvaluator
  • func NewQueryEvaluator() *QueryEvaluator
  • func (qe *QueryEvaluator) Evaluate(query map[string]interface{}, ctx *DetectionContext) (bool, string)
  • func (qe *QueryEvaluator) ExtractVersion(rules []map[string]string, ctx *DetectionContext) string
• type RequestConfig
• type ScanResult
• type Technology

Constants

const (
	MaxRetries     = 1
	RequestTimeout = 10 * time.Second
	MaxRedirects   = 3
	InitialBackoff = 1 * time.Second
)

Functions

func ComputeFaviconHash

func ComputeFaviconHash(data []byte) int32

ComputeFaviconHash returns the Shodan-compatible MurmurHash3 x86/32 of a favicon: standard base64 with a newline every 76 characters and a trailing newline, then signed mmh3 (seed 0).

func ComputeFaviconMD5

func ComputeFaviconMD5(data []byte) string

ComputeFaviconMD5 returns the hex MD5 of raw favicon bytes (OWASP/WhatWeb style).

func ShouldRunBrowserDetection

func ShouldRunBrowserDetection(techName string, results map[string]*Technology, probe BrowserProbe) bool

ShouldRunBrowserDetection determines if browser detection should run for a technology

Types

type BatchResults

type BatchResults struct {
	Results []ScanResult `json:"results"`
}

BatchResults wraps multiple scan results for JSON array output

type BrowserDetector

type BrowserDetector struct {
	// contains filtered or unexported fields
}

BrowserDetector performs browser-based detection

func NewBrowserDetector

func NewBrowserDetector() *BrowserDetector

NewBrowserDetector creates a new browser detector

func NewBrowserDetectorWithOptions

func NewBrowserDetectorWithOptions(proxyURL string) *BrowserDetector

NewBrowserDetectorWithOptions creates a new browser detector with proxy support

func (*BrowserDetector) DetectBrowser

func (bd *BrowserDetector) DetectBrowser(baseURL string, fingerprints map[string]Fingerprint, httpResults map[string]*Technology) (map[string]*Technology, error)

DetectBrowser performs browser-based detection

type BrowserPathClassification

type BrowserPathClassification struct {
	Path         string
	Technologies map[string][]BrowserProbe // tech name -> probes
}

BrowserPathClassification groups browser probes by path

func ClassifyBrowserByPath

func ClassifyBrowserByPath(fingerprints map[string]Fingerprint) []BrowserPathClassification

ClassifyBrowserByPath groups browser fingerprints by path

type BrowserProbe

type BrowserProbe struct {
	Path      string `json:"path"`
	Detection string `json:"detection,omitempty"`
	Version   string `json:"version,omitempty"`
}

BrowserProbe represents a browser-based detection probe

func (*BrowserProbe) HasDetectionCapability

func (bp *BrowserProbe) HasDetectionCapability() bool

HasDetectionCapability checks if browser probe can detect technology

func (*BrowserProbe) HasVersionCapability

func (bp *BrowserProbe) HasVersionCapability() bool

HasVersionCapability checks if browser probe can extract version

type DetectResult

type DetectResult struct {
	Technologies []Technology `json:"technologies"`
	FailedPaths  []string     `json:"failed_paths,omitempty"`
}

DetectResult contains detection results

type DetectionContext

type DetectionContext struct {
	Body       string
	Headers    map[string]string
	StatusCode int
}

DetectionContext holds data available for detection

type Detector

type Detector struct {
	// contains filtered or unexported fields
}

Detector is the main detection engine

func NewDetector

func NewDetector(fingerprintsDir string) (*Detector, error)

NewDetector creates a new detection engine

func NewDetectorWithOptions

func NewDetectorWithOptions(fingerprintsDir string, insecureSkipVerify bool, proxyURL string) (*Detector, error)

NewDetectorWithOptions creates a new detection engine with custom options

func (*Detector) Detect

func (d *Detector) Detect(url string, useBrowser bool) (*DetectResult, error)

Detect performs full detection (HTTP + Browser) on a target URL

func (*Detector) DetectFull

func (d *Detector) DetectFull(url string) (*DetectResult, error)

DetectFull performs full detection including browser stage

func (*Detector) DetectHTTPOnly

func (d *Detector) DetectHTTPOnly(url string) (*DetectResult, error)

DetectHTTPOnly performs HTTP-only detection (fast, no browser)

type Fingerprint

type Fingerprint struct {
	Cats          []int          `json:"cats"`
	Implies       []string       `json:"implies,omitempty"`
	FaviconHashes []int32        `json:"favicon_hashes,omitempty"`
	FaviconMD5    []string       `json:"favicon_md5,omitempty"`
	Paths         []PathProbe    `json:"paths,omitempty"`
	Browser       []BrowserProbe `json:"browser,omitempty"`
	Description   string         `json:"description,omitempty"`
	Website       string         `json:"website,omitempty"`
	Icon          string         `json:"icon,omitempty"`
	CPE           string         `json:"cpe,omitempty"`
}

Fingerprint represents the detection rules for a technology

type FingerprintDB

type FingerprintDB struct {
	Apps map[string]Fingerprint `json:"apps"`
}

FingerprintDB represents the entire fingerprint database

type HTTPDetector

type HTTPDetector struct {
	// contains filtered or unexported fields
}

HTTPDetector performs HTTP-based detection

func NewHTTPDetector

func NewHTTPDetector() *HTTPDetector

NewHTTPDetector creates a new HTTP detector

func NewHTTPDetectorWithOptions

func NewHTTPDetectorWithOptions(insecureSkipVerify bool, proxyURL string) *HTTPDetector

NewHTTPDetectorWithOptions creates a new HTTP detector with custom options

func (*HTTPDetector) DetectHTTP

func (hd *HTTPDetector) DetectHTTP(baseURL string, fingerprints map[string]Fingerprint) (map[string]*Technology, []string)

DetectHTTP performs HTTP-based detection on a target URL

type Loader

type Loader struct {
	// contains filtered or unexported fields
}

Loader handles loading fingerprints from disk or embedded FS

func NewLoader

func NewLoader(fingerprintsDir string) *Loader

NewLoader creates a new fingerprint loader that uses embedded fingerprints

func (*Loader) LoadAll

func (l *Loader) LoadAll() (map[string]Fingerprint, error)

LoadAll loads all fingerprints from either embedded FS or external directory

type PathClassification

type PathClassification struct {
	Path         string
	RequestConf  *RequestConfig
	Technologies map[string][]PathProbe // tech name -> probes
}

PathClassification groups fingerprints by path

func ClassifyByPath

func ClassifyByPath(fingerprints map[string]Fingerprint) []PathClassification

ClassifyByPath groups all fingerprints by their request paths

type PathProbe

type PathProbe struct {
	Path           string                 `json:"path"`
	Request        *RequestConfig         `json:"request,omitempty"`
	Detect         map[string]interface{} `json:"detect"`
	ExtractVersion []map[string]string    `json:"extract_version,omitempty"`
}

PathProbe represents an HTTP-based detection probe

type QueryEvaluator

type QueryEvaluator struct{}

QueryEvaluator evaluates MongoDB-style queries against a context

func NewQueryEvaluator

func NewQueryEvaluator() *QueryEvaluator

NewQueryEvaluator creates a new query evaluator

func (*QueryEvaluator) Evaluate

func (qe *QueryEvaluator) Evaluate(query map[string]interface{}, ctx *DetectionContext) (bool, string)

Evaluate evaluates a query against the detection context

func (*QueryEvaluator) ExtractVersion

func (qe *QueryEvaluator) ExtractVersion(rules []map[string]string, ctx *DetectionContext) string

ExtractVersion attempts to extract version from context using extraction rules

type RequestConfig

type RequestConfig struct {
	Method  string            `json:"method,omitempty"`
	Headers map[string]string `json:"headers,omitempty"`
	Body    interface{}       `json:"body,omitempty"`
}

RequestConfig represents optional HTTP request configuration

type ScanResult

type ScanResult struct {
	URL          string            `json:"url"`
	Technologies map[string]string `json:"technologies"`    // tech name -> version
	Mode         string            `json:"mode"`            // "http", "browser", or "hybrid"
	Error        string            `json:"error,omitempty"` // error message if scan failed
}

ScanResult represents the result for a single URL in JSON/JSONL format

type Technology

type Technology struct {
	Name    string `json:"name"`
	Version string `json:"version"`
}

Technology represents a detected technology