llmsink

package module
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: MIT Imports: 11 Imported by: 0

README

llms-ink-go

Go Reference

Go client for llms.ink: turn any public link — a site, docs page, GitHub repo, PDF or video — into llms.txt or llms-full.txt, and check a site's own llms.txt. Standard library only, Go 1.21+.

go get github.com/lukax/llms-ink-go
import llmsink "github.com/lukax/llms-ink-go"

c := llmsink.New() // reads LLMS_INK_KEY if set; anonymous otherwise
r, err := c.Llms(ctx, "https://llmstxt.org", llmsink.LlmsOptions{Mode: llmsink.ModeFull})
fmt.Println(r.Title, r.Tokens, r.Text)

chk, err := c.Check(ctx, "llmstxt.org")
fmt.Println(chk.Found, chk.Summary.Errors, chk.Summary.Warnings)

LlmsOptions{Mode, Fresh}: ModeLLMS (default, the index) or ModeFull (the content); Fresh: true skips the cache.

Options: WithKey(key), WithBaseURL(url), WithHTTPClient(*http.Client).

API key (optional)

Without a key, requests are anonymous and share per-IP limits. A key (Authorization: Bearer lk_…) gets its own daily quota. Set LLMS_INK_KEY or pass WithKey. An invalid or revoked key fails with 401 invalid_key; it never falls back to anonymous. Request a key.

Errors

API errors are *llmsink.Error (Status, Code, Title, Detail, RetryAfter), from the API's problem details:

var e *llmsink.Error
switch {
case llmsink.IsInvalidKey(err):
case llmsink.IsRateLimited(err), llmsink.IsQuotaExceeded(err):
	errors.As(err, &e)
	time.Sleep(e.RetryAfter)
case errors.As(err, &e):
	log.Print(e.Code, e.Detail)
}

langchaingo

A langchaingo document loader lives in a separate module so the root stays dependency-free:

go get github.com/lukax/llms-ink-go/langchaingo
l := langchaingo.NewLoader(nil, llmsink.LlmsOptions{Mode: llmsink.ModeFull}, "https://llmstxt.org")
docs, err := l.Load(ctx)

Development

go vet ./... && go test ./...
go test -tags live -run Live -v .   # hits https://llms.ink
(cd langchaingo && go test ./...)

Release: tag vX.Y.Z for the root and langchaingo/vX.Y.Z for the loader; then drop the replace in langchaingo/go.mod.

MIT License.

Documentation

Overview

Package llmsink is a client for the llms.ink API (https://llms.ink/docs/): turn any public link — a site, docs page, GitHub repo, PDF or video — into llms.txt or llms-full.txt, and check a site's own llms.txt.

An API key is optional. Without one, requests are anonymous and share the per-IP limits; set LLMS_INK_KEY or use WithKey for your own daily quota.

Index

Examples

Constants

View Source
const DefaultBaseURL = "https://llms.ink"

DefaultBaseURL is the production API origin.

View Source
const Version = "0.1.0"

Version is this module's version, sent in the User-Agent.

Variables

This section is empty.

Functions

func IsInvalidKey

func IsInvalidKey(err error) bool

IsInvalidKey reports whether err is a 401 invalid_key (unknown or revoked key).

func IsQuotaExceeded

func IsQuotaExceeded(err error) bool

IsQuotaExceeded reports whether err is a 429 quota_exceeded (daily quota); see Error.RetryAfter.

func IsRateLimited

func IsRateLimited(err error) bool

IsRateLimited reports whether err is a 429 rate_limited (burst limit); see Error.RetryAfter.

Types

type BrokenLink struct {
	URL    string `json:"url"`
	Status int    `json:"status"`
}

BrokenLink is a link in llms.txt that failed; Status is 0 on timeout.

type CheckIssue

type CheckIssue struct {
	Level   string `json:"level"` // error, warning, info
	Code    string `json:"code"`  // e.g. missing_title, broken_link
	Line    *int   `json:"line"`
	Message string `json:"message"`
}

CheckIssue is one finding of Client.Check.

type CheckResult

type CheckResult struct {
	Site     string  `json:"site"`
	URL      string  `json:"url"`
	Found    bool    `json:"found"`
	Status   int     `json:"status"`
	Title    *string `json:"title"`
	Tokens   int     `json:"tokens"`
	Sections int     `json:"sections"`
	Links    struct {
		Total   int          `json:"total"`
		Checked int          `json:"checked"`
		Broken  []BrokenLink `json:"broken"`
	} `json:"links"`
	LlmsFull bool         `json:"llmsFull"`
	Issues   []CheckIssue `json:"issues"`
	Summary  struct {
		Errors   int `json:"errors"`
		Warnings int `json:"warnings"`
	} `json:"summary"`
	CheckedAt string `json:"checkedAt"`
}

CheckResult is the response of GET /api/v0/check.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client calls the llms.ink API. The zero value is not usable; call New.

func New

func New(opts ...Option) *Client

New returns a Client. The key defaults to the LLMS_INK_KEY environment variable.

func (*Client) Check

func (c *Client) Check(ctx context.Context, link string) (*CheckResult, error)

Check validates a site's own llms.txt: format, broken links, llms-full.txt.

Example
package main

import (
	"context"
	"fmt"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New()
	r, err := c.Check(context.Background(), "llmstxt.org")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r.Found, r.Summary.Errors, r.Summary.Warnings)
}

func (*Client) Llms

func (c *Client) Llms(ctx context.Context, link string, opts LlmsOptions) (*Result, error)

Llms returns llms.txt (or llms-full.txt with ModeFull) for any public link.

Example
package main

import (
	"context"
	"fmt"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New() // uses LLMS_INK_KEY if set
	r, err := c.Llms(context.Background(), "https://llmstxt.org", llmsink.LlmsOptions{Mode: llmsink.ModeFull})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r.Title, r.Tokens)
}

type Error

type Error struct {
	Status     int           // HTTP status
	Code       string        // e.g. invalid_key, rate_limited, quota_exceeded, invalid_url
	Title      string        // HTTP status title
	Detail     string        // for people
	RetryAfter time.Duration // from Retry-After on 429; 0 if absent
}

Error is an API error (RFC 9457 problem details; codes at https://llms.ink/docs/#errors). Use errors.As, or IsInvalidKey, IsRateLimited and IsQuotaExceeded.

Example
package main

import (
	"context"
	"errors"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New(llmsink.WithKey("lk_..."))
	_, err := c.Llms(context.Background(), "example.com", llmsink.LlmsOptions{})
	var e *llmsink.Error
	switch {
	case llmsink.IsInvalidKey(err):
		log.Fatal("unknown or revoked API key")
	case llmsink.IsRateLimited(err), llmsink.IsQuotaExceeded(err):
		errors.As(err, &e)
		log.Printf("slow down; retry in %s", e.RetryAfter)
	case err != nil:
		log.Fatal(err)
	}
}

func (*Error) Error

func (e *Error) Error() string

type LlmsOptions

type LlmsOptions struct {
	// Mode defaults to ModeLLMS.
	Mode Mode
	// Fresh skips the cache and rebuilds (counts against the daily quota).
	Fresh bool
}

LlmsOptions are optional parameters for Client.Llms.

type Media

type Media struct {
	Duration   *float64    `json:"duration"` // seconds; nil if unknown
	HasVideo   bool        `json:"hasVideo"`
	HasAudio   bool        `json:"hasAudio"`
	Thumbnail  *string     `json:"thumbnail"`
	Downloads  []string    `json:"downloads"` // "mp4", "m4a"
	IsLive     bool        `json:"isLive,omitempty"`
	Transcript *Transcript `json:"transcript,omitempty"`
}

Media describes a video result.

type Mode

type Mode string

Mode selects the output: ModeLLMS (the llms.txt index) or ModeFull (llms-full.txt).

const (
	ModeLLMS Mode = "llms"
	ModeFull Mode = "full"
)

type Option

type Option func(*Client)

Option configures a Client.

func WithBaseURL

func WithBaseURL(u string) Option

WithBaseURL sets the API origin, e.g. for tests. Defaults to DefaultBaseURL.

func WithHTTPClient

func WithHTTPClient(h *http.Client) Option

WithHTTPClient sets the HTTP client. Defaults to one with a 120s timeout.

func WithKey

func WithKey(key string) Option

WithKey sets the API key (sent as "Authorization: Bearer <key>"). An empty key means anonymous. Overrides LLMS_INK_KEY.

type Result

type Result struct {
	URL         string  `json:"url"`         // the link after normalization
	Kind        string  `json:"kind"`        // site, section, page, repo, document, video
	Source      string  `json:"source"`      // native or generated
	Mode        Mode    `json:"mode"`        // llms or full
	Title       string  `json:"title"`       //
	Text        string  `json:"text"`        // the llms.txt / llms-full.txt Markdown
	Tokens      int     `json:"tokens"`      // approximate (characters ÷ 4)
	Pages       int     `json:"pages"`       // links (llms) or pages inlined (full)
	Model       *string `json:"model"`       // nil for native files
	Cached      bool    `json:"cached"`      //
	GeneratedAt string  `json:"generatedAt"` // RFC 3339
	Media       *Media  `json:"media,omitempty"`
}

Result is the response of GET /api/v0/llms.

type Segment

type Segment struct {
	Start float64 `json:"start"`
	End   float64 `json:"end"`
	Text  string  `json:"text"`
}

Segment is a transcript segment, in source-video seconds.

type Transcript

type Transcript struct {
	Source   string    `json:"source"`
	Segments []Segment `json:"segments"`
}

Transcript is a video transcript with timestamps.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL