virgo

README

virgo

Turn web pages into Markdown using the Chrome DevTools Protocol

Install

You can install the primary CLI utility using go install:

go install github.com/mjc-gh/virgo/cmd/virgo@latest

Usage

virgo
NAME:
   virgo - A tool for converting webpages to Markdown and plaintext.

USAGE:
   virgo [global options] [command [command options]]

VERSION:
   0.0.0

COMMANDS:
    screenshot  Screenshot one or more URLs
    markdown    Get the markdown content of a URL
    plaintext   Get the plaintext content of a URL
    help, h     Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help
   --version, -v  print the version

Markdown subcommand

virgo markdown --help
NAME:
   virgo markdown - Get the markdown content of a URL

USAGE:
   virgo markdown [options] url

OPTIONS:
   --include-images, -i       include images in markdown output (default: false)
   --debug, -d                enable debug logging (default: false)
   --headfull, -H             run browser in headfull mode (default: false)
   --concurrency int, -c int  number of concurrent workers (default: 0)
   --remote-port int          remote DevTools port (default: 0)
   --remote-host string       remote DevTools host
   --device-type string       device type (desktop/mobile/tablet) (default: "desktop")
   --device-size string       device size preset (default: "large")
   --user-agent string        browser user-agent preset (default: "chrome")
   --help, -h                 show help

Headless Shell

You can use the chromedp headless shell container if you do not have Chrome installed locally. This works well on Linux servers:

docker pull chromedp/headless-shell:latest
docker run -d -p 9222:9222 --rm --name headless-shell chromedp/headless-shell
virgo markdown --remote-port 9222 --remote-host 127.0.0.1 cnn.com

Docker

Using the virgo-web REST API Container

The virgo-web service is available as a Docker container with Chrome pre-installed. This is useful for running the REST API in isolated environments like Kubernetes, Docker Compose, or any container orchestration platform.

Quick Start

# Build the container locally
make docker.build

# Run the container
docker run -d -p 8888:8888 virgo-web:latest

# Test the API
curl http://localhost:8888/markdown?url=https://example.com

Using Pre-built Images

Pre-built container images are available from GitHub Container Registry:

# Pull the latest image
docker pull ghcr.io/mjc-gh/virgo:latest

# Run the container
docker run -d -p 8888:8888 ghcr.io/mjc-gh/virgo:latest

Images are tagged with semantic versions (e.g., v1.2.3) and latest for the most recent release.

Environment Variables

Configure the virgo-web service using environment variables:

Variable	Default	Description
VIRGO_HOST	0.0.0.0	Web server bind host
VIRGO_PORT	8888	Web server bind port
VIRGO_CONCURRENCY	1	Number of concurrent workers
VIRGO_DEBUG	false	Enable debug logging
VIRGO_JSON_LOGS	true	Output logs as JSON

Example: Running with Custom Configuration

docker run -d \
  -p 8888:8888 \
  -e VIRGO_CONCURRENCY=4 \
  -e VIRGO_DEBUG=true \
  -e VIRGO_JSON_LOGS=false \
  ghcr.io/mjc-gh/virgo:latest

Docker Compose Example

version: '3.8'

services:
  virgo-web:
    image: ghcr.io/mjc-gh/virgo:latest
    ports:
      - "8888:8888"
    environment:
      VIRGO_CONCURRENCY: 4
      VIRGO_DEBUG: false
      VIRGO_JSON_LOGS: true
    restart: unless-stopped

REST API Endpoints

The virgo-web service provides the following endpoints:

• POST /markdown - Convert URL to Markdown

  curl -X POST http://localhost:8888/markdown \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'
  

• POST /plaintext - Convert URL to plaintext

  curl -X POST http://localhost:8888/plaintext \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}'
  

• POST /screenshot - Get screenshot of URL

  curl -X POST http://localhost:8888/screenshot \
    -H "Content-Type: application/json" \
    -d '{"url": "https://example.com"}' \
    --output screenshot.png
  

• GET /health_check - Service health status

  curl http://localhost:8888/health_check
  

Development

You need Go and golangci-lint

mise use -g go@1.26.3
mise use -g golangci-lint@v2.12.2

You can build and test the CLI tool with the following:

make build.cli
make test

Documentation

Index

• func Logger() *zerolog.Logger
• func SetupLogger(debug bool, jsonLogs bool) *zerolog.Logger
• type SpecificLevelWriter
  • func (w SpecificLevelWriter) Write(p []byte) (int, error)
  • func (w SpecificLevelWriter) WriteLevel(level zerolog.Level, p []byte) (int, error)

Functions

func Logger

func Logger() *zerolog.Logger

func SetupLogger

func SetupLogger(debug bool, jsonLogs bool) *zerolog.Logger

Types

type SpecificLevelWriter

type SpecificLevelWriter struct {
	Writer io.Writer
	Levels []zerolog.Level
}

SpecificLevelWriter is a LevelWriter that only writes logs for specific levels.

func (SpecificLevelWriter) Write

func (w SpecificLevelWriter) Write(p []byte) (int, error)

Write implements io.Writer and writes using the configured writer.

func (SpecificLevelWriter) WriteLevel

func (w SpecificLevelWriter) WriteLevel(level zerolog.Level, p []byte) (int, error)

WriteLevel writes the log message if the level matches one of the configured levels.