computeruse

README

Expelimental Computer Use

• computer use by RobotGO
• browser use by Rod

Usage

export ANTHROPIC_API_KEY="sk-ant-..."
cd examples/browsercmd
go run .

• Accessibility permission is required on macOS [https://support.apple.com/guide/mac-help/mh43185/mac]

WARNING

Computer use is a beta feature. Please be aware that computer use poses unique risks that are distinct from standard API features or chat interfaces. These risks are heightened when using computer use to interact with the internet. To minimize risks, consider taking precautions such as:

• Use a dedicated virtual machine or container with minimal privileges to prevent direct system attacks or accidents.
• Avoid giving the model access to sensitive data, such as account login information, to prevent information theft.
• Limit internet access to an allowlist of domains to reduce exposure to malicious content.
• Ask a human to confirm decisions that may result in meaningful real-world consequences as well as any tasks requiring affirmative consent, such as accepting cookies, executing financial transactions, or agreeing to terms of service. In some circumstances, Claude will follow commands found in content even if it conflicts with the user’s instructions. For example, Claude instructions on webpages or contained in images may override instructions or cause Claude to make mistakes. We suggest taking precautions to isolate Claude from sensitive data and actions to avoid risks related to prompt injection.

Finally, please inform end users of relevant risks and obtain their consent prior to enabling computer use in your own products. https://docs.anthropic.com/en/docs/build-with-claude/computer-use

LICENSE

• JSON License

The Software shall be used for Good, not Evil.

Documentation

Index

• type Browser
  • func NewBrowser() *Browser
  • func (b *Browser) CursorPosition() (x, y int)
  • func (b *Browser) Key(key string)
  • func (b *Browser) LeftClick()
  • func (b *Browser) MouseMove(x, y int)
  • func (b *Browser) Open(url string)
  • func (b *Browser) RightClick()
  • func (b *Browser) Screenshot() []byte
  • func (b *Browser) Type(text string)
• type Computer
  • func NewRobot(bounds Rect) Computer
• type ContentContent
• type ContentSource
• type Message
  • func NewToolResponseMessage(toolresults []ToollCallResult) Message
  • func RunMessages(msgs []Message, width, height int) (Message, error)
• type MessageContent
• type MessageInput
• type Rect
• type Robot
  • func (r *Robot) CursorPosition() (x int, y int)
  • func (r *Robot) Key(key string)
  • func (r *Robot) LeftClick()
  • func (r *Robot) MouseMove(x, y int)
  • func (r *Robot) RightClick()
  • func (r *Robot) Screenshot() []byte
  • func (r *Robot) Type(text string)
• type ToollCallResult
  • func ToolCall(com Computer, msg Message) ([]ToollCallResult, error)

Types

type Browser

type Browser struct {
	// contains filtered or unexported fields
}

func NewBrowser

func NewBrowser() *Browser

func (*Browser) CursorPosition

func (b *Browser) CursorPosition() (x, y int)

func (*Browser) Key

func (b *Browser) Key(key string)

func (*Browser) LeftClick

func (b *Browser) LeftClick()

func (*Browser) MouseMove

func (b *Browser) MouseMove(x, y int)

func (*Browser) Open

func (b *Browser) Open(url string)

func (*Browser) RightClick

func (b *Browser) RightClick()

func (*Browser) Screenshot

func (b *Browser) Screenshot() []byte

func (*Browser) Type

func (b *Browser) Type(text string)

type Computer

type Computer interface {
	MouseMove(x, y int)
	LeftClick()
	RightClick()
	Type(text string)
	Key(key string)
	Screenshot() []byte
	CursorPosition() (x int, y int)
}

func NewRobot

func NewRobot(bounds Rect) Computer

type ContentContent

type ContentContent struct {
	Type   string        `json:"type"`
	Source ContentSource `json:"source,omitempty"`
}

type ContentSource

type ContentSource struct {
	Type      string `json:"type"`
	MediaType string `json:"media_type"`
	Data      string `json:"data"`
}

type Message

type Message struct {
	Role    string           `json:"role"`
	Content []MessageContent `json:"content"`
}

func NewToolResponseMessage

func NewToolResponseMessage(toolresults []ToollCallResult) Message

func RunMessages

func RunMessages(msgs []Message, width, height int) (Message, error)

type MessageContent

type MessageContent struct {
	Type string `json:"type"` // text, image, tool_use, tool_result

	// for text
	Text string `json:"text,omitempty"`

	// for tool_use
	ID    string        `json:"id,omitempty"`
	Name  string        `json:"name,omitempty"`
	Input *MessageInput `json:"input,omitempty"`

	// for tool_result
	Content   []ContentContent `json:"content,omitempty"`
	ToolUseID string           `json:"tool_use_id,omitempty"`
	IsError   bool             `json:"is_error,omitempty"`
}

type MessageInput

type MessageInput struct {
	Action     string `json:"action,omitempty"`
	Text       string `json:"text,omitempty"`
	Coordinate []int  `json:"coordinate,omitempty"`
}

type Rect

type Rect struct {
	Top    int
	Left   int
	Width  int
	Height int
}

type Robot

type Robot struct {
	Bounds Rect
}

func (*Robot) CursorPosition

func (r *Robot) CursorPosition() (x int, y int)

func (*Robot) Key

func (r *Robot) Key(key string)

func (*Robot) LeftClick

func (r *Robot) LeftClick()

func (*Robot) MouseMove

func (r *Robot) MouseMove(x, y int)

func (*Robot) RightClick

func (r *Robot) RightClick()

func (*Robot) Screenshot

func (r *Robot) Screenshot() []byte

func (*Robot) Type

func (r *Robot) Type(text string)

type ToollCallResult

type ToollCallResult struct {
	Action     string
	Text       string
	ToolUseID  string
	IsError    bool
	Screenshot []byte // PNG bytes
}

func ToolCall

func ToolCall(com Computer, msg Message) ([]ToollCallResult, error)