browser

README

Gost-DOM - A headless browser for Go

The Go-to headless browser for TDD workflows.

Gost-DOM is a headless browser written in Go intended to write tests of web application in Go that relies on JavaScript. Properties of Gost-DOM-based tests:

• Tests run in parallel due to complete complete isolation[^1]
• No erratic behaviour due to 100% predictable UI reactions.
• "Blazingly fast". No out-of-process calls, not even thread boundaries for web API calls. Web application code runs in the test thread, so a panic in your code keeps a full stack trace to the test case. [^2]
• Dependencies can be replaced while testing.

Yet Gost-DOM still uses HTTP request and responses for verification, testing the entire stack, as well as middlewares.

of usage.

[!NOTE]

This is 0.x version still, and breaking API changes do occur, but will be announced before release in the Gost-DOM discussions (do say Hi! 👋)

Looking for sponsors

This tool has reached a level where it can be used to test some web applications with JavaScript, e.g., simple HTMX applications. But there is still a lot to build to support just the most relevant Web APIs.

I've made good progress because of too much spare time; but that will not last. If I could find enough sponsors, it could mean the difference between continued development, or death 😢

For companies wanting to sponsor, I can send formal invoices too. More information on the project's Sponsor page

Getting started

• Read Getting Started
• Familiarize yourself with the Feature list to know what is implemented.
• Join my discord server to chat with me, and stay up-to-date on progress.
• say hi! on the github discussions page.
• Read the contribution guide to see how you can help.

Project status

This is still in an early phase, but it's approaching a design that seems promising for the purpose.

At the moment there's an emphasis on high-risk features that can expose poor design choices, but the "primary API" has been reasonably stable for a good amount of time.

Upcoming work

The near-future work is prioritised around

• Provide distinct error messages when using unsupported JS functions
• Improving fetch implementation
• Fix incorrectly implemented features
• Simulate user interaction, such as typing

Notify when using unsupported JS functions

Most of the JS mapping layer is auto-generated, including operations and attribute getter/setters that are not yet supported. This generates explicit error messages in the callback functions, making it clear when using, that the failed test case is caused by lack of support in Gost-DOM - not a bug in your code.

This should be improved further - including when calling functions accepting an options object, and clien code pass a valid, but unsupported option.

Implement more of fetch

The current fetch implementation is extremely basic, having only support for simple GET requests.

This will be improved in the future for more advanced scenarios, including correct headers support, cookies, body - including streaming response bodies, etc.

Fix incorrectly implemented features

Many of the already implemented features or APIs are not completely implemented, a few examples.

• Assigning to the history doesn't navigate
• Live collections are not live.
• Submit buttons cannot override form method and action.

To give users a better chance of predicting what works, and what doesn't, it is an aim to make sure that existing features work as they would in a real browser.

Future goals

There is much to do, which includes (but this is not a full list):

• Support web-sockets and server events.
• Implement all standard JavaScript classes that a browser should support; but not part of the ECMAScript standard itself.
  • JavaScript polyfills would be a good starting point; which is how xpath is implemented at the moment.
    • Conversion to native go implementations would be prioritized on usage, e.g. fetch would be high in the list of priorities.
• Implement default browser behaviour for user interaction, e.g. pressing enter when an input field has focus should submit the form.

Pure Go script engine

V8 is a "feature complete" JavaScript environment, so V8 support will never go away. But it has some overhead, and it depends on Cgo. Also, the current V8 layer leaks memory in the scope of a browsing context.

Goja is a pure Go JavaScript engine, and is alsmost fully supported as an alternative. JavaScript bindings in code target a layer of abstraction, allowing the script engine to be replaced.

CSS Parsing

Parsing CSS woule be nice, allowing test code to verify the resulting styles of an element; but having a working DOM with a JavaScript engine is higher priority.

Mock external sites

The system may depend on external sites in the browser, most notably identity providers (IDP), where your app redirects to the IDP, which redirects on successful login; but could be other services such as map providers, etc.

For testing purposes, replacing this with a dummy replacement would have some benefits:

• The verification of your system doesn't depend on the availability of an external service; when working offline
• Avoid tests breaking due to a new UI in your external dependency.
• For an identity provider
  • Avoid pollution of dummy accounts to run your test suite.
  • Avoid locking out test accounts due to "suspiscious activity".
  • The IDP may use a Captcha or 2FA that can be impossible; or difficult to control from tests, and would cause a significant slowdown to the test suite.
• For applications like map providers
  • Avoid being billed for API use during testing.

Out of scope.

Full Spec Compliance

A goal is not always meant to be reached, it often serves simply as something to aim at.

• Bruce Lee

While it is a goal to reach whatwg spec compliance, the primary goal is to have a useful tool for testing modern web applications.

Some specs don't really have any usage in modern web applications. For example, you generally wouldn't write an application that depends on quirks mode.

Another example is document.write. I've yet to work on any application that depends on this. However, implementing support for this feature require a complete rewrite of the HTML parser. You would need a really good case (or sponsorship level) to have that prioritised.

Accessibility tree

It is not currently planned that this library should maintain the accessibility tree; nor provide higher level testing capabilities like what Testing Library provides for JavaScript.

These problems should eventually be solved, but could easily be implemented in a different library with dependency to the DOM alone.

Visual Rendering

It is not a goal to be able to provide a visual rendering of the DOM.

But just like the accessibility tree, this could be implemented in a new library depending only on the interface from here.

Terminology

Some words inherntly have multiple meanings.

• Interface. The IDL Specification defines interfaces; which are exposed in certain scopes, implemented by "classes" in JavaScript.
  • The interfaces can be composed of partial or mixin interfaces.
  • IDL Interfaces and mixin interfaces are represented in Go, and typically exposed as Go interface types.

Attribution / 3rd party included code.

Some web APIs are implemented by embedding polyfills from other open-source JavaScript libraries.

• XPath evaluator uses code from the jsdom project distributed under the MIT license.
• FastestSmallestTextEncoderDecoder distributed under the Creative Commons Zero v1.0 Universal license.

---

[^1]: Complete isolation depends on your code, e.g., if you don't replace database dependencies, they are not isolated. [^2]: This depends on how you configure Gost-DOM.

Documentation

Overview

Package browser is the main entry point for Gost, helping create a window initialized with a script enging, connected to a server.

Index

• type Browser
  • func New(options ...BrowserOption) *Browser
  • func NewBrowser() *Browser
  • func NewBrowserFromHandler(handler http.Handler) *Browser
  • func NewFromHandler(handler http.Handler) *Browser
  • func (b *Browser) Close()
  • func (b *Browser) Closed() bool
  • func (b *Browser) NewWindow() Window
  • func (b *Browser) Open(location string) (window Window, err error)
• type BrowserOption
  • func WithContext(ctx context.Context) BrowserOption
  • func WithHandler(h http.Handler) BrowserOption
  • func WithLogger(l *slog.Logger) BrowserOption
  • func WithScriptHost(host html.ScriptHost) BrowserOption

Types

type Browser

type Browser struct {
	Client     http.Client
	ScriptHost ScriptHost
	Logger     log.Logger
	// contains filtered or unexported fields
}

Pretty stupid right now, but should _probably_ allow handling multiple windows/tabs. This used to be the case for _some_ identity providers, but I'm not sure if that even work anymore because of browser security.

func New

func New(options ...BrowserOption) *Browser

New initialises a new Browser with the default script engine.

func NewBrowser

func NewBrowser() *Browser

Deprecated: NewBrowser should not be called. Call New instead.

This method will selfdestruct in 10 commits

func NewBrowserFromHandler

func NewBrowserFromHandler(handler http.Handler) *Browser

Deprecated: NewBrowserFromHandler should not be called, call, NewFromHandler instead.

This method will selfdestruct in 10 commits

func NewFromHandler

func NewFromHandler(handler http.Handler) *Browser

NewFromHandler initialises a new Browser with with an http.Handler

Deprecated: Prefer browser.New(browser.WithHandler(...)) instead.

func (*Browser) Close

func (b *Browser) Close()

func (*Browser) Closed

func (b *Browser) Closed() bool

func (*Browser) NewWindow

func (b *Browser) NewWindow() Window

NewWindow creates a new window. Panics if the browser has been closed

func (*Browser) Open

func (b *Browser) Open(location string) (window Window, err error)

Open will open a new html.Window, loading the specified location. If the server does not respons with a 200 status code, an error is returned.

See html.NewWindowReader about the return value, and when the window returns.

type BrowserOption

type BrowserOption func(*browserConfig)

func WithContext

func WithContext(ctx context.Context) BrowserOption

func WithHandler

func WithHandler(h http.Handler) BrowserOption

WithHandler configures the browser's http.Client to use an http.Roundtripper that bypasses the TCP stack and calls directly into the specified handler as a normal function call.

Note: There is a current limitation that NO requests from the browser will be sent when using this. So sites will not work if they

• Depend on content from CDN
• Depend on an external service, e.g., an identity provider.

That is a limitation that was the result of prioritising more important, and higher risk features.

func WithLogger

func WithLogger(l *slog.Logger) BrowserOption

func WithScriptHost

func WithScriptHost(host html.ScriptHost) BrowserOption