GSEER

README

GSEER

Internet Asset Intelligence & Search Engine. Discover. Identify. Search.

GSEER is an open-source, self-hosted internet-asset intelligence and search engine. You point it at identifiers you care about, its passive collectors gather public facts about them (DNS records, ASN ownership), and you search everything it has indexed with a simple query language — through a CLI, a REST API, and a built-in dashboard.

Everything runs on your machine: your database, your data, your rules.

---

Features

• Search (GQL) — one query language, three surfaces (CLI, REST API, dashboard), all backed by a single SearchService. Search is always read-only: a query can never trigger collection or mutate data.
• Faceted search — port / protocol / technology / service / country / source / freshness aggregates over the same filter (gseer search --facets).
• Export — streamed JSON / CSV / NDJSON with cursor-stable pagination and RFC 4180 escaping (gseer export).
• Passive collection — explicit, operator-driven collectors with a terms-review gate, checkpoints, dead-letter bookkeeping, and re-drive. Nothing ever runs uninvoked.
• Suppression policy — block IPs/CIDRs/domains from ever entering your index, retroactively if needed (gseer suppress).
• Fingerprint rules — a lint-gated, versioned rule collection that classifies technology evidence an observation already carries into products with confidence (gseer rules).
• Signed updates — gseer update verifies an Ed25519-signed manifest against a key embedded in the binary before anything is downloaded or replaced.
• Full Docker stack — one command writes a db + server compose file with healthchecks, a persistent volume, and a generated API token.

Requirements

Component	Requirement
Go	1.26+ (only for go install / building from source — the binary itself has zero runtime dependencies)
Docker + Compose	for the bundled PostgreSQL 17 (recommended path for the server profile)
PostgreSQL	16+ if you run your own (OD-7 floor enforced by gseer setup and gseer doctor; verified end-to-end on 16.15 and 17.2 — docs/DEPLOYMENT.md)
OS	Linux / macOS / Windows

v0.2.0 is a source-only self-hosted release. The artifact is the source archive GSEER-v0.2.0-source.zip together with its SHA-256 checksum sidecar GSEER-v0.2.0-source.zip.sha256. This source archive is the only downloadable release artifact for v0.2.0. There are no prebuilt platform binaries, no binary packages, and no published Docker image for this release. Build from source (below), or build the Docker image locally via the Dockerfile. Docker is the easiest way to satisfy the PostgreSQL dependency.

v0.2.1 (compatible increment) adds the GQL compatibility layer (*=, free text, field aliases), personal mode on SQLite — a single-file database with no PostgreSQL dependency — and release engineering (validated release config, go install support, draft-gated releases). See docs/releases/RELEASE_NOTES_v0.2.1.md. PostgreSQL remains the primary engine for the self-hosted server profile.

Quick Start

Install the binary

Copy-paste; needs Go 1.26+ (the built binary has zero runtime dependencies):

# Easiest — no clone, no build script:
go install github.com/Lutfifakee-Project/GSEER/cmd/gseer@v0.2.1
gseer version
#   → gseer v0.2.1 (version/commit/built_at come from the toolchain's
#     own build info; `gseer doctor` prints the binary's SHA-256)

# Or build from source:
git clone https://github.com/Lutfifakee-Project/GSEER.git
cd GSEER
make build VERSION=v0.2.1
./bin/gseer version
# (optional: sudo cp bin/gseer /usr/local/bin/ so `gseer` is on PATH)

Platform archives land on the Releases page after a maintainer publishes the draft; verify any download against checksums.txt (SHA-256) and the sha256= line of gseer doctor.

Path A — Personal mode (one SQLite file, no PostgreSQL, no Docker)

For a single operator on a laptop or VPS. CLI surfaces only — search, export, suppression, backup (no API server; that is Path B).

# 1. Initialize the single-file database (~/.gseer/gseer.db; override
#    the location with --db-path)
gseer setup --personal

# 2. Verify — doctor runs real personal checks now
#    (database file, migration state, index honesty)
gseer doctor

# 3. The index starts EMPTY (honest — GSEER is the engine, not the
#    data).  Load the labeled SYNTHETIC evaluation corpus to try every
#    surface without collecting anything:
gseer setup --load-fixtures          # asks for a y/yes confirmation
gseer search 'port==443 && title CONTAINS "login"'
gseer export 'port==443' --format csv --out results.csv

# 4. Keep classes of records out of your index
gseer suppress add cidr 10.0.0.0/8 --reason "RFC1918"
gseer suppress list

# 5. Consistent online backup / restore (VACUUM INTO)
gseer db backup --out backup.db
gseer db restore backup.db

# 6. Real collection needs the self-hosted profile — the collection
#    registry (sources, jobs, checkpoints, dead letters) is
#    PostgreSQL-only.  `gseer source`/`collect` on a personal install
#    refuse with E_PROFILE_UNSUPPORTED and point here:
gseer setup --self-hosted --with-compose   # → Path B

Path B — Self-hosted server (PostgreSQL + dashboard + API)

# 1. Write the compose file + a .env with a random database password
gseer setup --self-hosted --with-compose

# 2. Start PostgreSQL (healthcheck-gated)
docker compose up -d db
docker compose ps db          # wait until it shows "healthy"

# 3. Hardened setup — bootstrap → migrate → grants → audit, one command
gseer setup --self-hosted --db-url "$(grep '^GSEER_DSN=' .env | cut -d= -f2-)"
#   → [ ok ] migrations: 8 applied, schema at head v17
#   → [ ok ] privilege audit: PASSED (fail-closed gate, OD-5)
#   → prints GSEER_DB_RUNTIME_DSN (the gseer_app credential) EXACTLY ONCE

# 4. Make this shell self-hosted (setup does not rewrite your shell)
export GSEER_PROFILE=self-hosted
export GSEER_DB_DSN="$(grep '^GSEER_DSN=' .env | cut -d= -f2-)"
export GSEER_DB_RUNTIME_DSN='postgres://gseer_app:<from step 3>@127.0.0.1:5432/gseer?sslmode=disable'
#   Without this, the CLI defaults to the client profile and doctor
#   fails with E_CONFIG_REQUIRED (client.endpoint missing).  The server
#   additionally REQUIRES the runtime credential — it never serves
#   traffic under the operator credential (docs/DEPLOYMENT.md).

# 5. Verify the installation
gseer doctor
#   → 6 PASS, 0 FAIL.  (The index is reported as EMPTY — that is honest.)

# 6. Your first search (a fresh install has an empty index)
gseer search 'port==443'
#   → The index is empty — no assets have been ingested yet.

# 7. (Optional) load the labeled SYNTHETIC evaluation corpus so you can
#    try search/export/facets without collecting anything
gseer setup --load-fixtures          # asks for a y/yes confirmation
gseer search 'port==443 && title CONTAINS "login"'
#   Every surface labels this data: SYNTHETIC EVALUATION DATA.

# 8. Register a source and run it — explicit, one command each
#    (docs/SOURCES.md holds the terms verdicts)
gseer source add my-dns --type dns --seed-mode config \
    --seeds example.net,example.org --enable --yes
gseer collect run --source my-dns
gseer search 'domain IN_DOMAIN "example.net"'

# 9. Start the dashboard + API (runs the privilege audit first, in the foreground)
gseer server start
#   Dashboard:  http://127.0.0.1:8443/
#   API health: http://127.0.0.1:8443/api/v1/health

Prefer everything in Docker? gseer setup --with-compose --full writes a docker-compose.full.yml (db + server, generated API token, database not exposed to the host) — then docker compose -f docker-compose.full.yml up -d --build. See docs/DOCKER.md.

Stop / restart the service:

# server start runs in the foreground:
#   stop with Ctrl-C in its terminal, or from another terminal:
gseer server stop
gseer server status
gseer server start          # start it again
docker compose stop db      # stop PostgreSQL;  docker compose start db

First searches (GQL)

GQL is the GSEER Query Language. = is sugar for ==; strings are double-quoted; numbers are bare integers.

gseer search 'port=443'                            # `=` is sugar for `==`
gseer search 'title="phpMyAdmin"'                  # strings are double-quoted
gseer search 'server="nginx" && port=443'          # boolean combine
gseer search 'ip IN_CIDR "203.0.113.0/24"'         # CIDR containment
gseer search 'domain IN_DOMAIN "example.net"'      # apex + subdomains
gseer search 'ANY(host){ http.status == 200 }'     # host has ANY service with 200
gseer search 'port==443' --facets                  # facet aggregates over the same rows
gseer search 'stale == "false"' --facets --facet port,protocol,country --max-values 5
gseer search 'port==443' --mode asset              # one row per asset
gseer search 'port==443' --explain                 # print the compiled query plan
gseer search 'port==443' --validate                # validate only, no DB contact
gseer search '"wordpress"'                         # bare-keyword free text (stored text surfaces)
gseer search 'ip*="203.0.113.*"'                   # wildcard fuzzy match (* = any run, ? = one)
gseer search 'after="2026-01-01" && status_code=200'  # compat aliases: time filter + status

The compatibility layer adds C-style operators (&&, ||, !), wildcard fuzzy matching (*=), common field aliases (host, org, base_protocol, status_code, cert.subject.cn, cert.issuer.cn), directional time filters (after/before), derived booleans (is_domain, is_ipv6, cert.is_expired), and empty-value semantics (field="" = missing-or-empty). The full mapping — including what is deliberately NOT supported and why (Search ≠ Scan) — is in docs/SEARCH-SYNTAX.md.

The full field registry (every field and its operators) is in gseer search --help. Export the results:

gseer export 'stale == "false"' --format csv --out results.csv
gseer export 'port == 443' --format ndjson --out -   # stdout; stats go to stderr

CLI Command Reference

Run gseer --help for the live list; gseer <command> --help documents every flag. Profile support per command — P = personal mode (SQLite), S = self-hosted (PostgreSQL), C = client (remote server). Verified against v0.2.1:

Command	Purpose
gseer search <query> (P S C)	Search the index with GQL (--facets, --mode, --limit, --sort, --explain, --validate, -o json)
gseer host <ip|hostname|domain> (S)	Read-only host inspection: asset + services + latest HTTP/TLS/fingerprints + provenance (-o json)
gseer export <query> (P S)	Stream results as JSON / CSV / NDJSON (--format, --out, --max)
gseer setup (P S)	--personal initializes the single-file personal mode; --self-hosted runs the hardened deployment order (--with-compose [--full], --load-fixtures, --purge-fixtures, --reset-runtime-password)
gseer doctor (P S C)	Diagnose config / DB / schema / index / privilege-audit health (read-only; personal mode checks the SQLite file, migration state, and index)
gseer server start|status|stop (S)	Manage the local API + dashboard (start runs the fail-closed privilege audit before binding)
gseer source add|list|enable|disable (S)	Register and manage passive data sources
gseer collect run|status|redrive (S)	Run collectors; inspect jobs; re-drive dead letters
gseer suppress add|list|remove (P S)	Manage the suppression policy
gseer rules list|validate (S)	Fingerprint rules (list; lint + false-positive tests)
gseer update --check|--apply	Verify / apply signed updates
gseer db migrate status|diagnose|force (S)	Inspect and recover schema migrations
gseer db backup|restore (P)	Consistent personal-mode snapshot (VACUUM INTO) and atomic restore; self-hosted backups use pg_dump (docs/DEPLOYMENT.md)
gseer version	Version, commit, build info
gseer completion bash|zsh|fish|powershell	Shell completion

Planned but not in this release: gseer scan, gseer asset. Invoking them prints a pointer and exits with code 64 — no scanning exists in GSEER.

Dashboard

gseer server start serves a server-rendered dashboard at http://127.0.0.1:8443/ (loopback by default). It is read-only (GET-only, no cookies, no login) and runs the same SearchService as the CLI: the search box accepts GQL, results show the same rows, and the facet panel mirrors --facets. The dashboard never triggers collection.

REST API

The API lives under /api/v1 on the same port. On the default loopback bind no token is required; a non-loopback bind requires one (GSEER_TOKEN, enforced by the config layer).

Endpoint	Method	Purpose
/api/v1/health	GET	Liveness ({"status":"ok"})
/api/v1/ready	GET	Readiness
/api/v1/version	GET	Version info
/api/v1/search	GET / POST	Search (same GQL, same semantics as the CLI)
/api/v1/facets	GET	Facet aggregates for a GQL filter
/api/v1/export	GET	Streamed JSON / CSV / NDJSON (Content-Disposition attachment)
/api/v1/validate	POST	Validate a GQL query
/api/v1/explain	POST	Compiled query plan
/api/v1/stats	GET	Index statistics

curl 'http://127.0.0.1:8443/api/v1/search?q=port%3D%3D443'
curl -H "Authorization: Bearer $GSEER_TOKEN" 'http://127.0.0.1:8443/api/v1/search?q=port%3D%3D443'

Data Collection

Nothing collects uninvoked. You register a source, enable it, and run it — one explicit command each. Targets are identifiers you named or assets already in your index; there are no third-party target lists and no enumeration. Collection requires the self-hosted profile (the registry that tracks sources, jobs, checkpoints, and dead letters is PostgreSQL-only; on a personal install the commands refuse with E_PROFILE_UNSUPPORTED).

# Register a passive DNS source (targets = domains you choose)
gseer source add my-dns --type dns --seed-mode config \
    --seeds example.net,example.org --enable --yes

# Run it once (explicit; no daemon, no scheduler)
gseer collect run --source my-dns

# What landed (passive facts are asset-level: a DNS answer proves the
# name was published — it never invents ports, banners or HTTP titles)
gseer search 'domain IN_DOMAIN "example.net"'

# Bookkeeping and recovery
gseer collect status                       # jobs, checkpoints, dead letters
gseer collect redrive --all                # re-ingest dead-lettered records
gseer collect redrive --source my-dns --discard

# Keep classes of records out of your index entirely
gseer suppress add cidr 10.0.0.0/8 --reason "RFC1918"
gseer suppress add domain_suffix internal.example --reason "our own space"
gseer suppress list

Connector status in v0.1.0:

Connector	Status
dns	Operational — verified end-to-end against operator-chosen names
ip2asn	Reviewed and enableable (Team Cymru free community service)
rdap	Disabled — RIR terms do not grant indexing/redistribution yet
ct (crt.sh)	Disabled — no published terms covering automated collection

Disabled connectors are enforced in code: enabling or running them refuses with E_SOURCE_TERMS. Verdicts and the review register live in docs/SOURCES.md.

Configuration

Precedence: CLI flags > environment variables > YAML config file > defaults. The config file lives at ~/.gseer/config.yaml (override with --config or GSEER_CONFIG); see .env.example for the full annotated set.

Variable	Meaning
GSEER_PROFILE	client (default) or self-hosted
GSEER_DB_DSN	Full PostgreSQL DSN for the operator credential (overrides the components below)
GSEER_DB_RUNTIME_DSN	Full PostgreSQL DSN for the runtime credential gseer_app (required by server start; printed once by setup)
GSEER_DB_HOST / GSEER_DB_PORT / GSEER_DB_USER / GSEER_DB_PASSWORD / GSEER_DB_NAME / GSEER_DB_SSLMODE	DSN components (operator credential)
GSEER_LISTEN	Server bind address (default 127.0.0.1:8443; non-loopback requires GSEER_TOKEN)
GSEER_TOKEN	API token for the REST API (Authorization: Bearer …)
GSEER_LOG_LEVEL / GSEER_LOG_FORMAT	debug|info|warn|error / text|json
GSEER_POLICY_EVALUATION_FIXTURES	Standing consent for the fixture load/purge gate
GSEER_FINGERPRINTS_RULES_DIR / GSEER_FINGERPRINTS_DISABLED	Operator fingerprint rules / kill switch
GSEER_UPDATE_MANIFEST_URL	Signed-update channel
GSEER_SERVER_TRUSTED_PROXIES	Reverse-proxy CIDRs allowed to set X-Forwarded-For

gseer setup --with-compose generates a .env (mode 0600) with a random database password — never commit a real .env.

Troubleshooting

Symptom	What it means / what to do
E_CONFIG_REQUIRED: client.endpoint … right after setup --self-hosted	Setup provisioned the database; your shell is still in the client profile. Export the handoff: export GSEER_PROFILE=self-hosted, `export GSEER_DB_DSN="$(grep '^GSEER_DSN=' .env
E_PROFILE_UNSUPPORTED on source/collect/host (personal install)	Those commands need the self-hosted profile: the collection registry and the host report are PostgreSQL-only. For evaluation data run gseer setup --load-fixtures (personal); for real collection deploy Path B (gseer setup --self-hosted --with-compose).
API_E_UNAUTHORIZED (client search, exit 5)	The server requires a Bearer token and the CLI client cannot present one — query with curl -H "Authorization: Bearer $GSEER_TOKEN" …, or from inside the compose network.
E_RUNTIME_CREDENTIAL_REQUIRED on server start (exit 3)	No runtime credential configured. gseer setup --self-hosted provisions gseer_app and prints GSEER_DB_RUNTIME_DSN once — export it.
E_RUNTIME_CREDENTIAL_MISMATCH (exit 5)	database.runtime_dsn connects as something other than gseer_app (e.g. the operator credential). Point it at the gseer_app credential — the server never serves traffic under another identity.
E_PRIVILEGE_AUDIT_FAILED (exit 5)	The fail-closed audit found a real deployment problem (missing grants, unsafe roles, escalation path). Fix the deployment per docs/DEPLOYMENT.md §3.4 — never grant gseer_app more privileges to "make it pass".
E_BOOTSTRAP during setup	The operator credential could not create/validate the hardened roles. Run setup with the database-owner credential, or remediate the named role state.
E_DB_UNREACHABLE: cannot reach PostgreSQL	The database is not running. docker compose up -d db, then docker compose ps db until healthy.
E_CONFIG_INSECURE: non-loopback bind requires a token	Set GSEER_TOKEN (or server.token) before binding a non-loopback address.
E_SOURCE_TERMS when enabling/running a source	That connector's provider terms are not cleared for automated collection (docs/SOURCES.md). This is a policy gate, not a bug.
E_QUERY_* on search/facets	The GQL did not validate. The error names the position and a Try: fix; gseer search '<q>' --validate checks without touching the DB.
E_UPDATE_UNSIGNED	The update manifest has no valid signature against the embedded key. It is refused — see docs/UPDATE.md.
Doctor reports HINT index: EMPTY	Normal on a fresh install. Load fixtures for evaluation, or register a source and run gseer collect run.
Migration failed mid-setup	gseer db migrate status, then gseer db migrate diagnose; force exists for recovery and asks for confirmation.
server start says the port is taken	Another instance is running (gseer server status) or something else uses 8443 — change GSEER_LISTEN.

Security, Limitations, and License

Security posture

• Search ≠ Scan — no code path in this repository port-scans, banner-grabs, or enumerates the internet. A query can only read. Planned active observation (gseer scan) does not exist in this release; invoking it exits 64.
• Nothing runs uninvoked — no daemon, no scheduler. Every collection run is an explicit operator command, gated by the terms register.
• Least-privilege runtime, fail-closed — the server serves traffic ONLY as the least-privilege gseer_app credential, and the fail-closed effective-privilege audit refuses startup (exit 5) unless the deployment's roles/grants/ownership are exactly the approved matrix. There is no bypass, flag, or override (docs/DEPLOYMENT.md).
• Suppression is enforced twice — in every ingestion path AND by a database trigger that refuses suppressed writes if any path forgets; after migration 0016 all suppression writes go through the gseer_policy SECURITY DEFINER functions, never direct DML.
• Loopback by default — the DB binds 127.0.0.1 only (compose full stack: no host port at all); the server binds loopback and a non-loopback bind requires a token; security headers on every response; API tokens are hashed at rest.
• Updates are verified before anything is downloaded — Ed25519 signature against the key embedded in the binary, then version gate, platform match, SHA-256, atomic replace with rollback.

Limitations (honest)

• Self-hosted index only. A fresh install has an empty database; you search exactly what your sources collected. There is no global, hosted-engine-scale index — GSEER is the engine, not a hosted service.
• Passive facts are asset-level. A DNS answer or a certificate-log entry proves an identifier was published; it never invents a port, a banner, or an HTTP title. Service-level rows appear only from evidence that actually carries them; DNS-discovered IPs do not mirror the resolving name into the asset's hostname/domain columns (the name is on the observation's evidence — docs/DEPLOYMENT.md §7).
• RDAP and crt.sh/CT are disabled until their providers' terms are reviewed and approved. The enforcement is in code, not just docs.
• Fingerprint coverage is deliberately conservative — the seed set classifies only well-documented header/banner evidence; CMS/framework identification extends via contributions (docs/RULES.md).
• No exploitation signatures. The rule linter rejects anything shaped like an exploit or credential payload.

License

GNU Affero General Public License v3.0 — see LICENSE. Contributions are welcome: fingerprint rules (docs/RULES.md) are the easiest starting point; every Go file carries the AGPL header and CI enforces it.

Documentation

• docs/QUICKSTART.md — the canonical fresh-install workflow
• docs/INSTALL.md — install methods, checksums, build from source
• docs/DEPLOYMENT.md — credential model, hardened startup, the fail-closed privilege audit
• docs/DOCKER.md — full stack: startup, upgrade, backup/restore, recovery
• docs/UPDATE.md — the signed-update chain, the v0.1.0 → v0.2.0 upgrade path, key ceremony, recovery
• docs/SOURCES.md — passive data-source review register (terms verdicts)
• docs/RULES.md — fingerprint rule format, confidence tiers, contributions
• docs/ROADMAP.md — release register (what is live per release)
• CHANGELOG.md — per-release changes
• docs/releases/ — per-release release notes (v0.1.0, v0.2.0, v0.2.1)

Documentation

Overview

Package main exists for exactly ONE reason (REL-01): the Go module root is the package path resolved by the short-form install command

go install github.com/Lutfifakee-Project/GSEER@latest

and `go install` requires a main package at the module root. Without this file that command fails with "does not contain package" and the easy-install path documented in docs/INSTALL.md is broken.

This file is a deliberate mirror of cmd/gseer/main.go — the canonical release entry point that GoReleaser builds (builds.main in .goreleaser.yaml). It contains NO command logic of its own: all dispatch lives in internal/cli, so there is exactly one command tree and both entry points cannot diverge. TestMain-equivalence is enforced by main_test.go, which builds BOTH entry points and asserts identical stdout and exit codes for representative invocations.