README

forest-go

builds.sr.ht status GoDoc

This repo contains:

  • A golang library for working with nodes in the Arbor Forest. This repo is based on the work-in-progress specification available here.
  • A CLI for creating, manipulating, and viewing nodes in the Arbor Forest in cmd/forest

For information about each component of this repo, see later in this file.

NOTE: this package requires using a fork of golang.org/x/crypto, and you must therefore include the following in your go.mod:

    replace golang.org/x/crypto => github.com/ProtonMail/crypto <version-from-forest-go's-go.mod>

About Arbor

arbor logo

Arbor is a chat system that makes communication clearer. It explicitly captures context that other platforms ignore, allowing you to understand the relationship between each message and every other message. It also respects its users and focuses on group collaboration.

You can get information about the Arbor project here.

For news about the project, join our mailing list!

Using the CLI

Right now, the CLI works with files in its current working directory, though this will change in the future. For the meantime, create a directory to play around in:

mkdir arbor-forest
cd arbor-forest

Important: About OpenPGP Keys

Arbor Forest nodes are signed by OpenPGP private keys. This gives Arbor strong guarantees about the authenticity of messages. The below procedures assume that you have gpg2 installed and have already generated a private key. Wherever you see --gpguser <email> below, substitute the email address associated with your GPG private key for <email>.

If you do not have gpg2 or a key and you do not want to install them, you can omit the --gpguser <email> flag in the commands below. If you do this, the CLI will create a new one for you and store it in ./arbor.privkey. This private key is not encrypted (has no passphrase), and should not be used for anything of importance.

Identities

Since all nodes must be signed by an Identity node, you must create one of those before you can create any others.

forest create identity --name <your-name> --gpguser <email>

This will print the base64url-encoded ID of your identity node, which will be stored in a file by that name in your current working directory.

To view your identity in a human-readable format, try the following (install jq if you don't have it, it's really handy):

forest show <id> | jq .

Substitute the base64url-encoded ID of your identity node for <id>. jq will pretty-print the JSON to make it easier to read.

Communities

To create a community, use:

forest create community --as <id> --name <community-name> --gpguser <email>

Substitute the base64url-encoded ID of your identity node for <id> and provide appropriate values for name and metadata.

To view your community in a human-readable format, try the following:

forest show <id> | jq .

Substitute the base64url-encoded ID of your community node for <id>. jq will pretty-print the JSON to make it easier to read.

Replies

To create a reply, use:

forest create reply --as <id> --to <parent-id> --content <your message> --gpguser <email>

Substitute the base64url-encoded ID of your identity node for <id> and the base64url-encoded ID of another reply or conversation node for <parent-id>. Substitute <your message> for the content of your reply. Usually this will be a response to the content of the node referenced by <parent-id>.

To view your reply in a human-readable format, try the following:

forest show <id> | jq .

Substitute the base64url-encoded ID of your reply node for <id>. jq will pretty-print the JSON to make it easier to read.

Build

Must use Go 1.11+

go build

Test

go test -v -cover

Documentation

Overview

Package forest is a library for creating nodes in the Arbor Forest data structure.

The specification for the Arbor Forest can be found here: https://github.com/arborchat/protocol/blob/forest/spec/Forest.md

NOTE: this package requires using a fork of golang.org/x/crypto, and you must therefore include the following in your go.mod:

replace golang.org/x/crypto => github.com/ProtonMail/crypto <version-from-forest-go's-go.mod>

All nodes in the Arbor Forest are cryptographically signed by an Identity node. Identity nodes sign themselves. To create a new identity, first create or load an OpenPGP private key using golang.org/x/crypto/openpgp. Then you can use that key and name to create an identity.

privkey := getPrivateKey() // do this however
name, err := fields.NewQualifiedContent(fields.ContentTypeUTF8, "example")
// handle error
metadata, err := fields.NewQualifiedContent(fields.ContentTypeJSON, "{}")
// handle error
identity, err := forest.NewIdentity(privkey, name, metadata)
// handle error

Identities (and their private keys) can be used to create other nodes with the Builder type. You can create community nodes using a builder like so:

builder := forest.As(identity, privkey)
communityName, err := fields.NewQualifiedContent(fields.ContentTypeUTF8, "example")
// handle error
communityMetadata, err := fields.NewQualifiedContent(fields.ContentTypeJSON, "{}")
// handle error
community, err := builder.NewCommunity(communityName, communityMetadata)
// handle error

Builders can also create reply nodes:

message, err := fields.NewQualifiedContent(fields.ContentTypeUTF8, "example")
// handle error
replyMetadata, err := fields.NewQualifiedContent(fields.ContentTypeJSON, "{}")
// handle error
reply, err := builder.NewReply(community, message, replyMetadata)
// handle error
message2, err := fields.NewQualifiedContent(fields.ContentTypeUTF8, "reply to reply")
// handle error
reply2, err := builder.NewReply(reply, message2, replyMetadata)
// handle error

The Builder type can also be used fluently like so:

// omitting creating the qualified content and error handling
community, err := forest.As(identity, privkey).NewCommunity(communityName, communityMetadata)
reply, err := forest.As(identity, privkey).NewReply(community, message, replyMetadata)
reply2, err := forest.As(identity, privkey).NewReply(reply, message2, replyMetadata)

Index

Constants

View Source
const MaxNameLength = 256

Variables

This section is empty.

Functions

func FindGPG

func FindGPG() (path string, err error)

FindGPG returns the path to the local gpg executable if one can be found. Otherwise it returns an error.

func NodeTypeOf

func NodeTypeOf(b []byte) (fields.NodeType, error)

NodeTypeOf returns the NodeType of the provided binary-marshaled node. If the provided bytes are not a forest node or the type cannot be determined, an error will be returned and the first return value must be ignored.

func ValidateID

func ValidateID(h Hashable, expected fields.QualifiedHash) (bool, error)

ValidateID returns whether the ID of this commonNode matches the data. The first return value indicates the result of the comparison. If there is an error, the first return value will always be false and the second will indicate what went wrong when computing the hash.

func ValidateNode

func ValidateNode(n Node, store Store) error

ValidateNode calls ValidateInternal and ValidateReferences on a node, returning the first error encountered (or nil if there were no errors).

func ValidateSignature

func ValidateSignature(v SignatureValidator, identity *Identity) (bool, error)

ValidateSignature returns whether the signature contained in this SignatureValidator is a valid signature for the given Identity. When validating an Identity node, you should pass the same Identity as the second parameter.

func VersionAndNodeTypeOf

func VersionAndNodeTypeOf(b []byte) (fields.Version, fields.NodeType, error)

Types

type Builder

type Builder struct {
	User *Identity
	Signer
}

Builder creates nodes in the forest on behalf of the given user.

func As

func As(user *Identity, signer Signer) *Builder

As creates a Builder that can write new nodes on behalf of the provided user. It is intended to be able to be used fluently, like:

community, err := forest.As(user, privkey).NewCommunity(name, metatdata)

func (*Builder) NewCommunity

func (n *Builder) NewCommunity(name string, metadata []byte) (*Community, error)

NewCommunity creates a community node (signed by the given identity with the given privkey).

func (*Builder) NewCommunityQualified

func (n *Builder) NewCommunityQualified(name *fields.QualifiedContent, metadata *fields.QualifiedContent) (*Community, error)

func (*Builder) NewReply

func (n *Builder) NewReply(parent interface{}, content string, metadata []byte) (*Reply, error)

NewReply creates a reply node as a child of the given community or reply

func (*Builder) NewReplyQualified

func (n *Builder) NewReplyQualified(parent interface{}, content, metadata *fields.QualifiedContent) (*Reply, error)

type CommonNode

type CommonNode struct {
	SchemaInfo `arbor:"order=0,recurse=always"`
	Parent     fields.QualifiedHash    `arbor:"order=1,recurse=serialize"`
	IDDesc     fields.HashDescriptor   `arbor:"order=2,recurse=always"`
	Depth      fields.TreeDepth        `arbor:"order=3"`
	Created    fields.Timestamp        `arbor:"order=4"`
	Metadata   fields.QualifiedContent `arbor:"order=5,recurse=serialize"`
	Author     fields.QualifiedHash    `arbor:"order=6,recurse=serialize"`
	// contains filtered or unexported fields
}

generic node

func (*CommonNode) AuthorID

func (n *CommonNode) AuthorID() *fields.QualifiedHash

AuthorID returns the Author of a common node. It wraps SignatureIdentityHash to satisfy the Node interface.

func (CommonNode) CreatedAt

func (n CommonNode) CreatedAt() time.Time

func (*CommonNode) Equals

func (n *CommonNode) Equals(n2 *CommonNode) bool

func (CommonNode) HashDescriptor

func (n CommonNode) HashDescriptor() *fields.HashDescriptor

func (CommonNode) ID

Compute and return the CommonNode's ID as a fields.Qualified Hash

func (CommonNode) IsIdentity

func (n CommonNode) IsIdentity() bool

func (CommonNode) ParentID

func (n CommonNode) ParentID() *fields.QualifiedHash

func (*CommonNode) SignatureIdentityHash

func (n *CommonNode) SignatureIdentityHash() *fields.QualifiedHash

SignatureIdentityHash returns the node identitifer for the Identity that signed this node.

func (CommonNode) TreeDepth

func (n CommonNode) TreeDepth() fields.TreeDepth

func (*CommonNode) TwigMetadata

func (n *CommonNode) TwigMetadata() (*twig.Data, error)

TwigMetadata returns the metadata of this node parsed into a *twig.Data

func (*CommonNode) ValidateInternal

func (n *CommonNode) ValidateInternal() error

ValidateInternal checks all fields for internal validity. It does not check the existence or validity of nodes referenced from this node. If the node validates, ValidateInternal returns `nil`.

func (*CommonNode) ValidateReferences

func (n *CommonNode) ValidateReferences(store Store) error

ValidateReferences checks for the existence of all referenced nodes within the provided store.

type Community

type Community struct {
	CommonNode `arbor:"order=0,recurse=always"`
	Name       fields.QualifiedContent `arbor:"order=1,recurse=serialize"`
	Trailer    `arbor:"order=2,recurse=always"`
}

func UnmarshalCommunity

func UnmarshalCommunity(b []byte) (*Community, error)

func (*Community) Equals

func (c *Community) Equals(other interface{}) bool

func (*Community) MarshalBinary

func (c *Community) MarshalBinary() ([]byte, error)

func (*Community) MarshalSignedData

func (c *Community) MarshalSignedData() ([]byte, error)

func (*Community) UnmarshalBinary

func (c *Community) UnmarshalBinary(b []byte) error

func (*Community) ValidateInternal

func (c *Community) ValidateInternal() error

ValidateInternal checks all fields for internal validity. It does not check the existence or validity of nodes referenced from this node.

func (*Community) ValidateReferences

func (c *Community) ValidateReferences(store Store) error

ValidateReferences checks all referenced nodes for existence within the store.

type GPGSigner

type GPGSigner struct {
	GPGUserName string
	// Rewriter is invoked on each invocation of exec.Command that spawns GPG. You can use it to modify
	// flags or any other property of the subcommand (environment variables). This is especially useful
	// to control how GPG prompts for key passphrases.
	Rewriter func(*exec.Cmd) error
	// contains filtered or unexported fields
}

GPGSigner uses a local gpg2 installation for key management. It will invoke gpg2 as a subprocess to sign data and to acquire the public key for its signing key. The public fields can be used to modify its behavior in order to change how it prompts for passphrases and other details.

func NewGPGSigner

func NewGPGSigner(gpgUserName string) (*GPGSigner, error)

NewGPGSigner wraps the private key so that it can sign using the local system's implementation of GPG.

func (GPGSigner) PublicKey

func (s GPGSigner) PublicKey() ([]byte, error)

PublicKey returns the bytes of the OpenPGP public key used by this signer.

func (*GPGSigner) Sign

func (s *GPGSigner) Sign(data []byte) ([]byte, error)

Sign invokes gpg2 to sign the data as this Signer's configured PGP user. It returns the signature or an error (if any).

type Hashable

type Hashable interface {
	HashDescriptor() *fields.HashDescriptor
	encoding.BinaryMarshaler
}

type Identity

type Identity struct {
	CommonNode `arbor:"order=0,recurse=always"`
	Name       fields.QualifiedContent `arbor:"order=1,recurse=serialize"`
	PublicKey  fields.QualifiedKey     `arbor:"order=2,recurse=serialize"`
	Trailer    `arbor:"order=3,recurse=always"`
}

Identity nodes represent a user. They associate a username with a public key that the user will sign messages with.

func NewIdentity

func NewIdentity(signer Signer, name string, metadata []byte) (*Identity, error)

NewIdentity builds an Identity node for the user with the given name and metadata, using the OpenPGP Entity privkey to define the Identity. That Entity must contain a private key with no passphrase.

func NewIdentityQualified

func NewIdentityQualified(signer Signer, name *fields.QualifiedContent, metadata *fields.QualifiedContent) (*Identity, error)

func UnmarshalIdentity

func UnmarshalIdentity(b []byte) (*Identity, error)

func (*Identity) Equals

func (i *Identity) Equals(other interface{}) bool

func (*Identity) MarshalBinary

func (i *Identity) MarshalBinary() ([]byte, error)

func (*Identity) MarshalSignedData

func (i *Identity) MarshalSignedData() ([]byte, error)

MarshalSignedData writes all data that should be signed in the correct order for signing. This can be used both to generate and validate message signatures.

func (*Identity) UnmarshalBinary

func (i *Identity) UnmarshalBinary(b []byte) error

func (*Identity) ValidateInternal

func (i *Identity) ValidateInternal() error

ValidateInternal checks all fields for internal validity. It does not check the existence or validity of nodes referenced from this node.

func (*Identity) ValidateReferences

func (i *Identity) ValidateReferences(store Store) error

ValidateReferences checks all referenced nodes for existence within the store.

type NativeSigner

type NativeSigner openpgp.Entity

NativeSigner uses golang's native openpgp operation for signing data. It only supports private keys without a passphrase.

func (NativeSigner) PublicKey

func (s NativeSigner) PublicKey() ([]byte, error)

PublicKey returns the raw bytes of the binary openpgp public key used by this signer.

func (NativeSigner) Sign

func (s NativeSigner) Sign(data []byte) ([]byte, error)

Sign signs the input data with the contained private key and returns the resulting signature.

type Node

type Node interface {
	AuthorID() *fields.QualifiedHash
	CreatedAt() time.Time
	Equals(interface{}) bool
	ID() *fields.QualifiedHash
	ParentID() *fields.QualifiedHash
	TreeDepth() fields.TreeDepth
	TwigMetadata() (*twig.Data, error)
	ValidateReferences(Store) error
	ValidateInternal() error
	encoding.BinaryMarshaler
	encoding.BinaryUnmarshaler
}

func UnmarshalBinaryNode

func UnmarshalBinaryNode(b []byte) (Node, error)

UnmarshalBinaryNode unmarshals a node of any type. If it does not return an error, the concrete type of the first return parameter will be one of the node structs declared in this package (e.g. Identity, Community, etc...)

type Reply

type Reply struct {
	CommonNode     `arbor:"order=0,recurse=always"`
	CommunityID    fields.QualifiedHash    `arbor:"order=1,recurse=serialize"`
	ConversationID fields.QualifiedHash    `arbor:"order=2,recurse=serialize"`
	Content        fields.QualifiedContent `arbor:"order=3,recurse=serialize"`
	Trailer        `arbor:"order=4,recurse=always"`
}

func UnmarshalReply

func UnmarshalReply(b []byte) (*Reply, error)

func (*Reply) Equals

func (r *Reply) Equals(other interface{}) bool

func (*Reply) MarshalBinary

func (r *Reply) MarshalBinary() ([]byte, error)

func (*Reply) MarshalSignedData

func (r *Reply) MarshalSignedData() ([]byte, error)

func (*Reply) UnmarshalBinary

func (r *Reply) UnmarshalBinary(b []byte) error

func (*Reply) ValidateInternal

func (r *Reply) ValidateInternal() error

ValidateInternal checks all fields for internal validity. It does not check the existence or validity of nodes referenced from this node.

func (*Reply) ValidateReferences

func (r *Reply) ValidateReferences(store Store) error

ValidateReferences checks all referenced nodes for existence within the store.

type SchemaInfo

type SchemaInfo struct {
	Version fields.Version  `arbor:"order=0"`
	Type    fields.NodeType `arbor:"order=1"`
}

type SignatureValidator

type SignatureValidator interface {
	MarshalSignedData() ([]byte, error)
	GetSignature() *fields.QualifiedSignature
	SignatureIdentityHash() *fields.QualifiedHash
	IsIdentity() bool
}

SignatureValidator is a type that has a signature and can supply the ID of the node that signed it.

type Signer

type Signer interface {
	Sign(data []byte) (signature []byte, err error)
	PublicKey() (key []byte, err error)
}

Signer can sign any binary data

func NewNativeSigner

func NewNativeSigner(privatekey *openpgp.Entity) (Signer, error)

NewNativeSigner creates a native Golang PGP signer. This will fail if the provided key is encrypted. GPGSigner should be used for all encrypted keys.

type Store

type Store interface {
	CopyInto(Store) error
	Get(*fields.QualifiedHash) (Node, bool, error)
	GetIdentity(*fields.QualifiedHash) (Node, bool, error)
	GetCommunity(*fields.QualifiedHash) (Node, bool, error)
	GetConversation(communityID, conversationID *fields.QualifiedHash) (Node, bool, error)
	GetReply(communityID, conversationID, replyID *fields.QualifiedHash) (Node, bool, error)
	Children(*fields.QualifiedHash) ([]*fields.QualifiedHash, error)
	// Recent returns recently-created (as per the timestamp in the node) nodes.
	// It may return both a slice of nodes and an error if some nodes in the
	// store were unreadable.
	Recent(nodeType fields.NodeType, quantity int) ([]Node, error)
	// Add inserts a node into the store. It is *not* an error to insert a node which is already
	// stored. Implementations must not return an error in this case.
	Add(Node) error

	RemoveSubtree(*fields.QualifiedHash) error
}

type Trailer

type Trailer struct {
	Signature fields.QualifiedSignature `arbor:"order=0,recurse=serialize,signature"`
}

Trailer is the final set of fields in every arbor node

func (*Trailer) Equals

func (t *Trailer) Equals(t2 *Trailer) bool

func (*Trailer) GetSignature

func (t *Trailer) GetSignature() *fields.QualifiedSignature

GetSignature returns the signature for the node, which must correspond to the Signature Authority for the node in order to be valid.

type Validator

type Validator interface {
	Validate() error
}

Directories

Path Synopsis
cmd
Package grove implements an on-disk storage format for arbor forest nodes.
Package grove implements an on-disk storage format for arbor forest nodes.
Package testkeys provides PGP private keys SUITABLE ONLY FOR WRITING TEST CASES.
Package testkeys provides PGP private keys SUITABLE ONLY FOR WRITING TEST CASES.
Package testutil provides utilities for easily making test arbor nodes and content.
Package testutil provides utilities for easily making test arbor nodes and content.
Package twig implements the twig key-value data format.
Package twig implements the twig key-value data format.