llmsink

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 14, 2026 License: MIT Imports: 11 Imported by: 0

README

llms-ink-go

Go Reference

Go client for llms.ink: turn any public link — a site, docs page, GitHub repo, PDF or video — into llms.txt or llms-full.txt, and check a site's own llms.txt. Standard library only, Go 1.21+.

go get github.com/lukax/llms-ink-go
import llmsink "github.com/lukax/llms-ink-go"

c := llmsink.New() // reads LLMS_INK_KEY if set; anonymous otherwise
r, err := c.Llms(ctx, "https://llmstxt.org", llmsink.LlmsOptions{Mode: llmsink.ModeFull})
fmt.Println(r.Title, r.Tokens, r.Text)

chk, err := c.Check(ctx, "llmstxt.org")
fmt.Println(chk.Found, chk.Summary.Errors, chk.Summary.Warnings)

// lint llms.txt you haven't published yet (POST /api/v0/check)
res, err := c.CheckText(ctx, text, llmsink.CheckTextOptions{Base: "https://example.com", Links: true})
for _, s := range res.Parsed.Sections {
	fmt.Println(s.Name, len(s.Links))
}

LlmsOptions{Mode, Fresh}: ModeLLMS (default, the index) or ModeFull (the content); Fresh: true skips the cache.

CheckResult.Parsed (nil when nothing was found) holds the parsed llms.txt: Title, Summary, Details, and Sections with their Links (Title, URL, Notes, Line). CheckTextOptions{Base, Links}: Base resolves relative links and is reported as Site/URL (nil without it); Links also checks linked URLs and, with Base, llms-full.txt (LlmsFull is nil otherwise).

v0.2.0 is breaking: CheckResult.Site and URL are now *string and LlmsFull is *bool. See CHANGELOG.md.

Options: WithKey(key), WithBaseURL(url), WithHTTPClient(*http.Client).

API key (optional)

Without a key, requests are anonymous and share per-IP limits. A key (Authorization: Bearer lk_…) gets its own daily quota. Set LLMS_INK_KEY or pass WithKey. An invalid or revoked key fails with 401 invalid_key; it never falls back to anonymous. Request a key.

Errors

API errors are *llmsink.Error (Status, Code, Title, Detail, RetryAfter), from the API's problem details:

var e *llmsink.Error
switch {
case llmsink.IsInvalidKey(err):
case llmsink.IsRateLimited(err), llmsink.IsQuotaExceeded(err):
	errors.As(err, &e)
	time.Sleep(e.RetryAfter)
case errors.As(err, &e):
	log.Print(e.Code, e.Detail)
}

langchaingo

A langchaingo document loader lives in a separate module so the root stays dependency-free:

go get github.com/lukax/llms-ink-go/langchaingo
l := langchaingo.NewLoader(nil, llmsink.LlmsOptions{Mode: llmsink.ModeFull}, "https://llmstxt.org")
docs, err := l.Load(ctx)

Development

go vet ./... && go test ./...
go test -tags live -run Live -v .   # hits https://llms.ink
(cd langchaingo && go test ./...)

Release: tag vX.Y.Z for the root and langchaingo/vX.Y.Z for the loader; then drop the replace in langchaingo/go.mod.

MIT License.

Documentation

Overview

Package llmsink is a client for the llms.ink API (https://llms.ink/docs/): turn any public link — a site, docs page, GitHub repo, PDF or video — into llms.txt or llms-full.txt, and check a site's own llms.txt.

An API key is optional. Without one, requests are anonymous and share the per-IP limits; set LLMS_INK_KEY or use WithKey for your own daily quota.

Index

Examples

Constants

View Source
const DefaultBaseURL = "https://llms.ink"

DefaultBaseURL is the production API origin.

View Source
const Version = "0.2.0"

Version is this module's version, sent in the User-Agent.

Variables

This section is empty.

Functions

func IsInvalidKey

func IsInvalidKey(err error) bool

IsInvalidKey reports whether err is a 401 invalid_key (unknown or revoked key).

func IsQuotaExceeded

func IsQuotaExceeded(err error) bool

IsQuotaExceeded reports whether err is a 429 quota_exceeded (daily quota); see Error.RetryAfter.

func IsRateLimited

func IsRateLimited(err error) bool

IsRateLimited reports whether err is a 429 rate_limited (burst limit); see Error.RetryAfter.

Types

type BrokenLink struct {
	URL    string `json:"url"`
	Status int    `json:"status"`
}

BrokenLink is a link in llms.txt that failed; Status is 0 on timeout.

type CheckIssue

type CheckIssue struct {
	Level   string `json:"level"` // error, warning, info
	Code    string `json:"code"`  // e.g. missing_title, broken_link
	Line    *int   `json:"line"`
	Message string `json:"message"`
}

CheckIssue is one finding of Client.Check.

type CheckResult

type CheckResult struct {
	Site     *string `json:"site"`
	URL      *string `json:"url"`
	Found    bool    `json:"found"`
	Status   int     `json:"status"`
	Title    *string `json:"title"`
	Tokens   int     `json:"tokens"`
	Sections int     `json:"sections"`
	Links    struct {
		Total   int          `json:"total"`
		Checked int          `json:"checked"`
		Broken  []BrokenLink `json:"broken"`
	} `json:"links"`
	LlmsFull *bool        `json:"llmsFull"`
	Issues   []CheckIssue `json:"issues"`
	Summary  struct {
		Errors   int `json:"errors"`
		Warnings int `json:"warnings"`
	} `json:"summary"`
	Parsed    *Parsed `json:"parsed"`
	CheckedAt string  `json:"checkedAt"`
}

CheckResult is the response of Client.Check and Client.CheckText. Site and URL are nil only for CheckText without a base; LlmsFull is nil when llms-full.txt wasn't checked; Parsed is nil when nothing was found.

type CheckTextOptions added in v0.2.0

type CheckTextOptions struct {
	// Base is a public URL that resolves relative links and is reported as Site/URL.
	Base string
	// Links also checks up to 25 linked URLs, plus llms-full.txt next to Base when set.
	Links bool
}

CheckTextOptions are optional parameters for Client.CheckText.

type Client

type Client struct {
	// contains filtered or unexported fields
}

Client calls the llms.ink API. The zero value is not usable; call New.

func New

func New(opts ...Option) *Client

New returns a Client. The key defaults to the LLMS_INK_KEY environment variable.

func (*Client) Check

func (c *Client) Check(ctx context.Context, link string) (*CheckResult, error)

Check validates a site's own llms.txt: format, broken links, llms-full.txt.

Example
package main

import (
	"context"
	"fmt"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New()
	r, err := c.Check(context.Background(), "llmstxt.org")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r.Found, r.Summary.Errors, r.Summary.Warnings)
}

func (*Client) CheckText added in v0.2.0

func (c *Client) CheckText(ctx context.Context, text string, opts CheckTextOptions) (*CheckResult, error)

CheckText validates llms.txt content you already have (up to 2,000,000 bytes) without hosting it: format, structure and, with Links, broken links.

Example
package main

import (
	"context"
	"fmt"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New()
	text := "# Example\n\n> One line.\n\n## Docs\n\n- [Guide](/guide.md): start here\n"
	r, err := c.CheckText(context.Background(), text, llmsink.CheckTextOptions{Base: "https://example.com"})
	if err != nil {
		log.Fatal(err)
	}
	if r.Parsed != nil {
		for _, s := range r.Parsed.Sections {
			for _, l := range s.Links {
				fmt.Println(s.Name, l.Title, l.URL)
			}
		}
	}
	fmt.Println(r.Summary.Errors, r.Summary.Warnings)
}

func (*Client) Llms

func (c *Client) Llms(ctx context.Context, link string, opts LlmsOptions) (*Result, error)

Llms returns llms.txt (or llms-full.txt with ModeFull) for any public link.

Example
package main

import (
	"context"
	"fmt"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New() // uses LLMS_INK_KEY if set
	r, err := c.Llms(context.Background(), "https://llmstxt.org", llmsink.LlmsOptions{Mode: llmsink.ModeFull})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(r.Title, r.Tokens)
}

type Error

type Error struct {
	Status     int           // HTTP status
	Code       string        // e.g. invalid_key, rate_limited, quota_exceeded, invalid_url
	Title      string        // HTTP status title
	Detail     string        // for people
	RetryAfter time.Duration // from Retry-After on 429; 0 if absent
}

Error is an API error (RFC 9457 problem details; codes at https://llms.ink/docs/#errors). Use errors.As, or IsInvalidKey, IsRateLimited and IsQuotaExceeded.

Example
package main

import (
	"context"
	"errors"
	"log"

	llmsink "github.com/lukax/llms-ink-go"
)

func main() {
	c := llmsink.New(llmsink.WithKey("lk_..."))
	_, err := c.Llms(context.Background(), "example.com", llmsink.LlmsOptions{})
	var e *llmsink.Error
	switch {
	case llmsink.IsInvalidKey(err):
		log.Fatal("unknown or revoked API key")
	case llmsink.IsRateLimited(err), llmsink.IsQuotaExceeded(err):
		errors.As(err, &e)
		log.Printf("slow down; retry in %s", e.RetryAfter)
	case err != nil:
		log.Fatal(err)
	}
}

func (*Error) Error

func (e *Error) Error() string

type LlmsOptions

type LlmsOptions struct {
	// Mode defaults to ModeLLMS.
	Mode Mode
	// Fresh skips the cache and rebuilds (counts against the daily quota).
	Fresh bool
}

LlmsOptions are optional parameters for Client.Llms.

type Media

type Media struct {
	Duration   *float64    `json:"duration"` // seconds; nil if unknown
	HasVideo   bool        `json:"hasVideo"`
	HasAudio   bool        `json:"hasAudio"`
	Thumbnail  *string     `json:"thumbnail"`
	Downloads  []string    `json:"downloads"` // "mp4", "m4a"
	IsLive     bool        `json:"isLive,omitempty"`
	Transcript *Transcript `json:"transcript,omitempty"`
}

Media describes a video result.

type Mode

type Mode string

Mode selects the output: ModeLLMS (the llms.txt index) or ModeFull (llms-full.txt).

const (
	ModeLLMS Mode = "llms"
	ModeFull Mode = "full"
)

type Option

type Option func(*Client)

Option configures a Client.

func WithBaseURL

func WithBaseURL(u string) Option

WithBaseURL sets the API origin, e.g. for tests. Defaults to DefaultBaseURL.

func WithHTTPClient

func WithHTTPClient(h *http.Client) Option

WithHTTPClient sets the HTTP client. Defaults to one with a 120s timeout.

func WithKey

func WithKey(key string) Option

WithKey sets the API key (sent as "Authorization: Bearer <key>"). An empty key means anonymous. Overrides LLMS_INK_KEY.

type Parsed added in v0.2.0

type Parsed struct {
	Title    *string         `json:"title"`
	Summary  *string         `json:"summary"`
	Details  *string         `json:"details"`
	Sections []ParsedSection `json:"sections"`
}

Parsed is the structure of an llms.txt file: the H1 title, the blockquote summary, details (markdown before the first H2) and sections.

type ParsedLink struct {
	Title string  `json:"title"`
	URL   string  `json:"url"`
	Notes *string `json:"notes"`
	Line  int     `json:"line"`
}

ParsedLink is a link in an llms.txt section. URL is resolved against the base or fetched URL when possible, else as written; Notes is the text after ":" (nil if none); Line is the 1-based source line.

type ParsedSection added in v0.2.0

type ParsedSection struct {
	Name     string       `json:"name"`
	Optional bool         `json:"optional"`
	Text     *string      `json:"text"`
	Links    []ParsedLink `json:"links"`
}

ParsedSection is an H2 section of llms.txt. Optional is true for a section named "Optional" (case-insensitive); Text is non-link content (nil if none).

type Result

type Result struct {
	URL         string  `json:"url"`         // the link after normalization
	Kind        string  `json:"kind"`        // site, section, page, repo, document, video
	Source      string  `json:"source"`      // native or generated
	Mode        Mode    `json:"mode"`        // llms or full
	Title       string  `json:"title"`       //
	Text        string  `json:"text"`        // the llms.txt / llms-full.txt Markdown
	Tokens      int     `json:"tokens"`      // approximate (characters ÷ 4)
	Pages       int     `json:"pages"`       // links (llms) or pages inlined (full)
	Model       *string `json:"model"`       // nil for native files
	Cached      bool    `json:"cached"`      //
	GeneratedAt string  `json:"generatedAt"` // RFC 3339
	Media       *Media  `json:"media,omitempty"`
}

Result is the response of GET /api/v0/llms.

type Segment

type Segment struct {
	Start float64 `json:"start"`
	End   float64 `json:"end"`
	Text  string  `json:"text"`
}

Segment is a transcript segment, in source-video seconds.

type Transcript

type Transcript struct {
	Source   string    `json:"source"`
	Segments []Segment `json:"segments"`
}

Transcript is a video transcript with timestamps.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL