http

README

HTTP

http defines an HTTP client implementation. It is a thin wrapper around the Go standard package net/http but in Python requests style.

Functions

get(url,params={},headers={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP GET request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

put(url,params={},headers={},body="",form_body={},form_encoding="",json_body={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP PUT request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
body	string	optional. raw string body to provide to the request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data.
json_body	any	optional. JSON data to supply as a request. handy for working with JSON-API's.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

post(url,params={},headers={},body="",form_body={},form_encoding="",json_body={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP POST request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
body	string	optional. raw string body to provide to the request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data.
json_body	any	optional. JSON data to supply as a request. handy for working with JSON-API's.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

postForm(url,params={},headers={},form_body={},form_encoding="",auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP POST request with form data, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

delete(url,params={},headers={},body="",form_body={},form_encoding="",json_body={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP DELETE request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
body	string	optional. raw string body to provide to the request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data
json_body	any	optional. JSON data to supply as a request. handy for working with JSON-API's.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

patch(url,params={},headers={},body="",form_body={},form_encoding="",json_body={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP PATCH request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
body	string	optional. raw string body to provide to the request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data.
json_body	any	optional. JSON data to supply as a request. handy for working with JSON-API's.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

options(url,params={},headers={},body="",form_body={},form_encoding="",json_body={},auth=(),timeout=30,allow_redirects=True,verify=True) response

Perform an HTTP OPTIONS request, returning a response.

Parameters

name	type	description
url	string	URL to request.
headers	dict	optional. dictionary of headers to add to request.
body	string	optional. raw string body to provide to the request.
form_body	dict	optional. dict of values that will be encoded as form data.
form_encoding	string	optional. application/x-www-form-url-encoded (default) or multipart/form-data.
json_body	any	optional. JSON data to supply as a request. handy for working with JSON-API's.
auth	tuple	optional. (username,password) tuple for HTTP Basic authorization.
timeout	float	optional. how many seconds to wait for the server to send all the data before giving up. 0 means no timeout.
allow_redirects	bool	optional. whether to follow redirects.
verify	bool	optional. whether to verify the server's SSL certificate.

set_timeout(timeout)

Set the global timeout for all HTTP requests.

Parameters

name	type	description
timeout	float	The timeout in seconds. Must be non-negative. This timeout will be used for all subsequent HTTP requests made by the module.

get_timeout() float

Get the current global timeout setting for HTTP requests. returns: The current timeout in seconds used for HTTP requests.

Types

response

The result of performing a HTTP request.

Fields

name	type	description
url	string	the URL that was ultimately requested (may change after redirects).
status_code	int	response status code (for example: 200 == OK).
headers	dict	dictionary of response headers.
encoding	string	transfer encoding. example: "octet-stream" or "application/json".

Methods

body() string

output response body as a string

json() object

attempt to parse response body as json, returning a JSON-decoded result.

Documentation

Overview

Package http provides tools for integrating HTTP request handling within Go-based web servers and clients with Starlark scripting capabilities. This enables dynamic inspection and modification of HTTP requests and responses through scripts, enhancing flexibility and control over processing.

Migrated from: https://github.com/qri-io/starlib/tree/master/http with modifications.

Index

• Constants
• Variables
• func AsString(x starlark.Value) (string, error)
• func ConvertServerRequest(r *http.Request) *starlarkstruct.Struct
• func LoadModule() (starlark.StringDict, error)
• type ExportedServerRequest
  • func NewExportedServerRequest(r *http.Request) (*ExportedServerRequest, error)
  • func (r *ExportedServerRequest) Struct() *starlarkstruct.Struct
  • func (r *ExportedServerRequest) Write(req *http.Request) (err error)
• type ExportedServerResponse
  • func (d *ExportedServerResponse) Write(w http.ResponseWriter) (err error)
• type Module
  • func (m *Module) StringDict() starlark.StringDict
  • func (m *Module) Struct() *starlarkstruct.Struct
• type RequestGuard
• type Response
  • func (r *Response) HeadersDict() *starlark.Dict
  • func (r *Response) JSON(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, ...) (starlark.Value, error)
  • func (r *Response) Struct() *starlarkstruct.Struct
  • func (r *Response) Text(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, ...) (starlark.Value, error)
• type ServerResponse
  • func NewServerResponse() *ServerResponse
  • func (r *ServerResponse) Export() *ExportedServerResponse
  • func (r *ServerResponse) Struct() *starlarkstruct.Struct
  • func (r *ServerResponse) Write(w http.ResponseWriter) (err error)

Constants

const ModuleName = "http"

ModuleName defines the expected name for this Module when used in starlark's load() function, eg: load('http', 'get')

Variables

var (
	// UserAgent is the default user agent for http requests, override with a custom value before calling LoadModule.
	UserAgent = "Starlet-http-client/" + itn.StarletVersion
	// TimeoutSecond is the default timeout in seconds for http requests, override with a custom value before calling LoadModule.
	TimeoutSecond = 30.0
	// SkipInsecureVerify controls whether to skip TLS verification, override with a custom value before calling LoadModule.
	SkipInsecureVerify = false
	// DisableRedirect controls whether to follow redirects, override with a custom value before calling LoadModule.
	DisableRedirect = false
	// Client is the http client used to create the http module, override with a custom client before calling LoadModule.
	Client *http.Client
	// Guard is a global RequestGuard used in LoadModule, override with a custom implementation before calling LoadModule.
	Guard RequestGuard
	// ConfigLock is a global lock for settings, use it to ensure thread safety when setting.
	ConfigLock sync.RWMutex
)

Functions

func AsString

func AsString(x starlark.Value) (string, error)

AsString unquotes a starlark string value

func ConvertServerRequest

func ConvertServerRequest(r *http.Request) *starlarkstruct.Struct

ConvertServerRequest converts a http.Request to a Starlark struct for use in Starlark scripts on the server side.

func LoadModule

func LoadModule() (starlark.StringDict, error)

LoadModule creates an http Module

Types

type ExportedServerRequest

type ExportedServerRequest struct {
	Method   string         // The HTTP method (e.g., GET, POST, PUT, DELETE)
	URL      *url.URL       // The request URL
	Proto    string         // The protocol used for the request (e.g., HTTP/1.1)
	Host     string         // The host specified in the request
	Remote   string         // The remote address of the client
	Header   http.Header    // The HTTP headers included in the request
	Encoding []string       // The transfer encodings specified in the request
	Body     []byte         // The request body data
	JSONData starlark.Value // The request body data as Starlark value
}

ExportedServerRequest encapsulates HTTP request data in a format accessible to both Go code and Starlark scripts. This struct bridges Go's HTTP handling features with Starlark's dynamic scripting capabilities, enabling seamless interaction and manipulation of request properties in Go, while providing a structured, read-only view of the request data to Starlark scripts.

Key Features:

• Full access to HTTP request properties (method, URL, headers, body) for reading and modification in Go.
• Structured, read-only representation of request data for Starlark scripts, enhancing scripting flexibility.
• JSON body support simplifies working with JSON payloads directly in scripts.

Usage Pattern:

1. Convert an incoming http.Request to ExportedServerRequest with NewExportedServerRequest for access in Go.
2. Modify the ExportedServerRequest properties as needed in Go before handing off to Starlark.
3. Use the Struct method to convert the ExportedServerRequest to a Starlark struct, passing it to Starlark scripts for read-only access. This step allows scripts to inspect the request's properties.
4. Since the Starlark struct is read-only, modifications to the request must be performed in Go, either before or after script execution.

This design prioritizes ease of use, security, and performance, facilitating dynamic and complex request processing logic through Go and Starlark. It ensures the integrity of the HTTP request handling by preventing unauthorized modifications and protecting against potential security threats. Developers are encouraged to validate all modifications and interactions with the request data to maintain the server's security posture.

func NewExportedServerRequest

func NewExportedServerRequest(r *http.Request) (*ExportedServerRequest, error)

NewExportedServerRequest creates a new ExportedServerRequest from an http.Request.

func (*ExportedServerRequest) Struct

func (r *ExportedServerRequest) Struct() *starlarkstruct.Struct

Struct returns a Starlark struct representation of the ExportedServerRequest, which exposes the following fields to Starlark scripts:

• method: The HTTP method (e.g., GET, POST, PUT, DELETE)
• url: The request URL
• proto: The protocol used for the request (e.g., HTTP/1.1)
• host: The host specified in the request
• remote: The remote address of the client
• header: The HTTP headers included in the request
• query: The query parameters included in the request URL
• encoding: The transfer encodings specified in the request
• body: The request body data
• json: The request body data as Starlark value, if it is valid JSON or None otherwise

func (*ExportedServerRequest) Write

func (r *ExportedServerRequest) Write(req *http.Request) (err error)

Write writes the request data back to a provided http.Request instance.

type ExportedServerResponse

type ExportedServerResponse struct {
	StatusCode int         // StatusCode is the status code of the response.
	Header     http.Header // Header is the header of the response, a map of string to list of strings. Content-Type is set automatically.
	Data       []byte      // Data is the data of the response, usually the body content.
}

ExportedServerResponse is a struct to export the response data to Go.

func (*ExportedServerResponse) Write

func (d *ExportedServerResponse) Write(w http.ResponseWriter) (err error)

type Module

type Module struct {
	// contains filtered or unexported fields
}

Module joins http tools to a dataset, allowing dataset to follow along with http requests

func (*Module) StringDict

func (m *Module) StringDict() starlark.StringDict

StringDict returns all module methods in a starlark.StringDict

func (*Module) Struct

func (m *Module) Struct() *starlarkstruct.Struct

Struct returns this module's methods as a starlark Struct

type RequestGuard

type RequestGuard interface {
	Allowed(thread *starlark.Thread, req *http.Request) (*http.Request, error)
}

RequestGuard controls access to http by checking before making requests if Allowed returns an error the request will be denied

type Response

type Response struct {
	http.Response
}

Response represents an HTTP response, wrapping a go http.Response with starlark methods

func (*Response) HeadersDict

func (r *Response) HeadersDict() *starlark.Dict

HeadersDict flops

func (*Response) JSON

func (r *Response) JSON(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error)

JSON attempts to parse the response body as JSON

func (*Response) Struct

func (r *Response) Struct() *starlarkstruct.Struct

Struct turns a response into a *starlark.Struct

func (*Response) Text

func (r *Response) Text(thread *starlark.Thread, _ *starlark.Builtin, args starlark.Tuple, kwargs []starlark.Tuple) (starlark.Value, error)

Text returns the raw data as a string

type ServerResponse

type ServerResponse struct {
	// contains filtered or unexported fields
}

ServerResponse is a struct that enables HTTP response manipulation within Starlark scripts, facilitating dynamic preparation of HTTP responses in Go-based web servers executing such scripts.

Key Features:

• Setting HTTP status codes.
• Adding and managing HTTP headers.
• Specifying the content type of the response.
• Setting the response body with support for various data types (e.g., binary, text, HTML, JSON).

Usage Pattern:

1. Create a ServerResponse instance using NewServerResponse().
2. Utilize the Struct() method to obtain a Starlark struct that exposes ServerResponse functionalities to Starlark scripts.
3. In the Starlark script, utilize provided methods (e.g., set_status, add_header, set_content_type) to prepare the response.
4. Back in Go, the ServerResponse instance can directly write its content to an http.ResponseWriter using its Write() method. Alternatively, you can call the Export() method to convert the ServerResponse into an ExportedServerResponse for modification, which is then capable of being written to an http.ResponseWriter using its Write() method.

Internally, ServerResponse uses a private contentDataType enum to manage the intended type of the response data, allowing for automatic adjustment of the Content-Type header based on the set data type by the Starlark script.

The ExportedServerResponse struct simplifies ServerResponse for interoperability with Go's standard http package, comprising an HTTP status code, headers, and data for the HTTP response. Its Write() method allows for the prepared response to be efficiently written to an http.ResponseWriter, ensuring correct header setting and response body data writing.

Note: Direct manipulation of ServerResponse and its methods by Starlark scripts necessitates validation of script inputs to mitigate potential security issues like header injection attacks. This design allows scripts to dynamically prepare HTTP responses while maintaining a secure and controlled server environment.

func NewServerResponse

func NewServerResponse() *ServerResponse

NewServerResponse creates a new ServerResponse.

func (*ServerResponse) Export

func (r *ServerResponse) Export() *ExportedServerResponse

Export dumps the response data to a struct for later use in Go.

func (*ServerResponse) Struct

func (r *ServerResponse) Struct() *starlarkstruct.Struct

Struct returns a Starlark struct representation of the ServerResponse, which exposes the following methods to Starlark scripts:

• set_status(code): Sets the HTTP status code for the response.
• set_code(code): An alias for set_status.
• add_header(key, value): Adds a header with the given key and value to the response.
• set_content_type(contentType): Sets the Content-Type header for the response.
• set_data(data): Sets the response data as binary data.
• set_json(data): Sets the response data as JSON, marshaling the given Starlark value to JSON.
• set_text(data): Sets the response data as plain text.
• set_html(data): Sets the response data as HTML.

func (*ServerResponse) Write

func (r *ServerResponse) Write(w http.ResponseWriter) (err error)

Write writes the response to http.ResponseWriter.