tfgateway

package module
v0.3.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 19, 2021 License: Apache-2.0 Imports: 28 Imported by: 0

README

TFGateway Tests and Build

Supported primitives

Deployment

architecture_overview

The TFGateway works be reading the reservation detail from the TFExplorer. It then convert these reservation into configuration readable by the TCP Router server or CoreDNS and write them into a redis server.

Both CoreDNS and the TCP router are watching redis and reloads there internal configuration evrytime there is a change in redis.

Binaries needed
Delegation of domain

If you want people to be able to delegate domain to the TFGateway. User needs to create a NS record pointing to the a domain of the TFGateway. Which means you need to have an A record pointing to the IP of the TFGateway and use the --nameservers flag when starting the TFGateway.

TCP Router server example configuration
[server]
addr = "0.0.0.0"
port = 443
httpport = 80
clientsport = 18000
[server.dbbackend]
type 	 = "redis"
addr     = "127.0.0.1"
port     = 6379
refresh  = 10
CoreDNS example configuration
. {
    redis  {
        address 127.0.0.1:6379
    }
}
Gateway 4 to 6 prerequisite
  • The host needs to masquerade the ipv6 traffic going out ip6tables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

NODES and their configuration

There are 7 nodes that have the tfgateway installed, 6 of them are DO nodes, one is a separate machine in the freefarm env.

https://cloud.digitalocean.com/projects/92a99fbe-5fa1-48f0-b088-1d93a56ac817/resources?i=68c689

the nodes are all configured the same way, where systemd handles the daemons necessary to run the nodes.

Installation

First, install a systemd driven system (DO droplets default to ubuntu, good enough).

And upgrade it / install some stuff (archlinux users, you know what to do ;-) )

#!/usr/bin/env bash

sed -ie "s/^#Port.*/Port 22\nPort 34022\n/" /etc/ssh/sshd_config
systemctl restart sshd

apt -y update
export DEBIAN_FRONTEND=noninteractive

# To make for a happy wireguard (better check if this kernel is still available)
apt-get -o Dpkg::Options::='--force-confold' --force-yes -fuy dist-upgrade
apt -o Dpkg::Options::='--force-confold' --force-yes -fuy install linux-image-5.3.0-46-generic linux-headers-5.3.0-46-generic

apt -y autoremove

# That is, if ufw isn't installed by default
apt -o Dpkg::Options::='--force-confold' --force-yes -fuy install ufw
ufw allow 34022/tcp

ech | ufw enable

systemctl enable ufw --now

# you might want to install docker for some ungoldly reason too
# curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
# add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
#
# apt update
# apt-get install docker-ce docker-ce-cli containerd.io -y
#
# systemctl enable docker --now

# wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -
# echo "deb [ arch=amd64,arm64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.2.list
# apt-get update
# apt-get install -y mongodb-org

# but you need wireguard
add-apt-repository ppa:wireguard/wireguard
apt-get update
apt-get -y install wireguard

# you might want to reboot before going further... with the kernel upgrade and all, ya know...

For networking, I assume you know your way ;-)

Anyway, so the gateway does several things when you want to expose a service. As your container with a service is not directly available for Internet access, you'll want to expose it in some way.

First, you want your service to be reachable by a DNS name (fqdn) in a domain (that can be readily available or that you want to handle). Secondly, the service you want to expose needs to be reachable.

TFGateway manages as well a dns server (coredns) as a tcp proxy that accepts connections from clients an forwards it to your service. Configuration of both services happen through a redis backend that is used as a queue, so that coredns and the tcp proxy can add/remove entries in their live configuration..

So all 4 need to be running on a node, with some ip/tcp ports properly opened up to handle traffic.

notes after installations

your system might be running resolved and using port 53, needed to be freed for coredns, also the 6379 port of redis will be used as soon as the default redis unit starts

  • stop and disable redis unit systemctl stop redis
  • stop and disable systemd-resolved unit systemctl stop systemd-resolved

Hence:

Some systemd units

/etc/systemd/system/tfredis.service

[Unit]
Description=The Redis server for TFGateway
After=network.target

[Service]
Type=simple
Environment=statedir=/run/redis
PIDFile=/run/redis/redis.pid
ExecStartPre=/bin/touch /var/log/redis.log
ExecStartPre=/bin/mkdir -p ${statedir}
ExecStart=/usr/local/bin/redis-server /etc/tfredis.conf
ExecReload=/bin/kill -USR2 $MAINPID
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G
LimitNOFILE=10050

[Install]
WantedBy=multi-user.target

/etc/systemd/system/tcprouter.service

[Unit]
Description=TCP router server
After=network.target

[Service]
ExecStart=/usr/local/bin/trs --config /etc/tcprouter/router.toml
Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target

/etc/systemd/system/coredns.service

[Unit]
Description=CoreDNS
After=network.target
Requires=tfredis.service

[Service]
ExecStart=/usr/local/bin/coredns -conf /etc/coredns/Corefile
Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target

/etc/systemd/system/tfgateway.service

[Unit]
Description=TCP router server
After=network.target

[Service]
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment ip=$(/sbin/ip r get 1.1.1.1 | awk '{print $7}')"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment hostname=$(/bin/hostname)"
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment subdom=tfgw$${hostname/tf-gateway}"
ExecStart=/usr/local/bin/tfgateway --seed /etc/identity.seed --nameservers ${hostname}.gateway.tf --endpoint ${ip}:3443 --domains ${subdom}.gateway.tf --domains ${subdom}.3x0.me --domains ${subdom}.ava.tf --domains ${subdom}.base.tf --farm 1

Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target

These services need some config, for our usecases it's all static.

/etc/tfredis.conf

bind 127.0.0.1

/etc/coredns/Corefile

. {
    redis  {
        address 127.0.0.1:6379
    }
}

/etc/tcprouter/router.toml

[server]
addr = "0.0.0.0"
port = 443
httpport = 80
clientsport = 18000
[server.dbbackend]
type 	 = "redis"
addr     = "127.0.0.1"
port     = 6379
refresh  = 10

Functionality

So how does all that fit together?

There are 2 parts:

  • resolve the dns name to an ip address, so that the client request arrives at the gatway that has been configured to accept requests for the specified fqdn (e.g. www.myspecialname.gateway.tf)
  • forwarding traffic from a client to the service that you want to expose.
DNS

Let's start by playing a dns server for your client, that wants to reach www.myspecialname.gateway.tf.

Your client has a dns server (Recursive DNS server,Rdns) configured in it's IP configuration. A Recursive DNS server handles the queries for your client, ultimately returning an IP address for and FQDN. That RDSN server ahs a list of all Top-Level-Domain nameservers it can send queries to.

Client -> Rdns: hey, Rdns, I want the IP address of www.myspecialname.gateway.tf

Rdns -> TLD server : Hey, tld server, what nameserver can I reach for a .tf domain

TLD Server -> Rdns : here is an ip address where you can ask

Rdns -> .tf nameserver: Hey, .tf nameserver, I'm looking for a nameserver that can answer questions about gateway.tf

.tf nameserver -> Rdns : you can go ask the server with that ip.

Rdns -> gateway.tf nameserver: hey gateway.tf server, I want the IP address of www.myspecialname.gateway.tf.

gateway.tf nameserver -> Rdns: Uh-oh, I'm not authoritative for that fqdn, actually myspecialname is a subdomain with it's own nameserver, go ask him, here is the ip address

Rdns -> myspecialname.gateway.tf nameserver : Hey myspecialname.gateway.tf, can you give me the IP address of www in your domain ?

myspecialname.gateway.tf nameserver -> Rdns: well of course, Rdns, here is the IP address.

Now the DNS server for myspecialname.gateway.tf is that coredns server on the tfgatways. Entries are added by a reservation on the explorer, which the gets picked up by the gateway and set up, zo that above process for resolving the fqdn (www.myspecialname.gateway.tf) to an ip address for the client.

TCP Proxy

The TCP Proxy is special in the sense that it contains 2 parts:

  • a server to accept connections from external clients (like your web browser)
  • and a client-server part that effectively forwards the above connection stream towards the exposed service

The client-server part seems a bit convoluted, but please bear with me:

The moment you want to expose a service, the grid adds a container in the same user network of the running service container, and starts a proxy, that is also a client towards the tfgateway server. The tfgateway's tcp proxy connects the outside listener with the the proxy client in that addon container, and as such the the proxy in the addon container can forward the queries towards the service.

example deployment script

apt update -y
apt install redis-server redis-tools -y
mkdir -p /etc/coredns
mkdir -p /etc/tcprouter



cat << EOF > /etc/identity.seed
"1.1.0"{"mnemonic":"$$3BOT_WORDS","threebotid":$$3BOT_ID}
EOF

cat << EOF > /etc/tfredis.conf
bind 127.0.0.1
EOF


cat << EOF > /etc/coredns/Corefile
. {
    log
    errors
    redis  {
        address 127.0.0.1:6379
    }
    forward . 174.138.6.79 8.8.8.8 1.1.1.1
}

EOF



cat << EOF > /etc/tcprouter/router.toml
[server]
addr = "0.0.0.0"
port = 443
httpport = 80
clientsport = 18000
[server.dbbackend]
type     = "redis"
addr     = "127.0.0.1"
port     = 6379
refresh  = 10
EOF




cat << EOF > /etc/systemd/system/tfredis.service
[Unit]
Description=The Redis server for TFGateway
After=network.target

[Service]
Type=simple
Environment=statedir=/run/redis
PIDFile=/run/redis/redis.pid
ExecStartPre=/bin/touch /var/log/redis.log
ExecStartPre=/bin/mkdir -p /run/redis
ExecStart=redis-server /etc/tfredis.conf
ExecReload=/bin/kill -USR2 $MAINPID
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G
LimitNOFILE=10050

[Install]
WantedBy=multi-user.target
EOF


cat << EOF > /etc/systemd/system/coredns.service
[Unit]
Description=CoreDNS
After=network.target
After=tfredis.target

[Service]
ExecStart=/usr/local/bin/coredns -conf /etc/coredns/Corefile
Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target
EOF



cat << EOF > /etc/systemd/system/tcprouter.service
[Unit]
Description=TCP router server
After=network.target
After=coredns.target

[Service]
ExecStart=/usr/local/bin/trs --config /etc/tcprouter/router.toml
Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target
EOF




cat << EOF > /etc/systemd/system/tfgateway.service
[Unit]
Description=tfgateway server
After=network.target
After=tcprouter.target

[Service]
ExecStartPre=/bin/bash -c "/bin/systemctl set-environment ip=$(/sbin/ip r get 1.1.1.1 | awk '{print $7}')"
ExecStart=/usr/local/bin/tfgateway --seed /etc/identity.seed --explorer $$EXPLORER_URL/api/v1 --nameservers $$NAMESERVER --endpoint ${ip}:3443 --domains $$DOMAIN --farm $$FARM_ID

Type=simple
Restart=on-failure
MemoryAccounting=true
MemoryHigh=800M
MemoryMax=1G

[Install]
WantedBy=multi-user.target
EOF

systemctl stop redis
systemctl stop systemd-resolved
systemctl start tfredis && systemctl start coredns && systemctl start tcprouter && systemctl start tfgateway

Make sure to replace variables starting with $$ in the above with the right values

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var (
	ProxyReservation         = provision.ReservationType(workloads.WorkloadTypeProxy.String())
	ReverseProxyReservation  = provision.ReservationType(workloads.WorkloadTypeReverseProxy.String())
	SubDomainReservation     = provision.ReservationType(workloads.WorkloadTypeSubDomain.String())
	DomainDeleateReservation = provision.ReservationType(workloads.WorkloadTypeDomainDelegate.String())
	Gateway4To6Reservation   = provision.ReservationType(workloads.WorkloadTypeGateway4To6.String())
)

ReservationType enum list all the supported primitives but the tfgateway

View Source 
var ProvisionOrder = map[provision.ReservationType]int{
	DomainDeleateReservation: 0,
	SubDomainReservation:     1,
	ProxyReservation:         2,
	ReverseProxyReservation:  3,
	Gateway4To6Reservation:   4,
}

ProvisionOrder is used to sort the workload type in the right order for provision engine

Functions

func ResultToSchemaType 

func ResultToSchemaType(r provision.Result) (*workloads.Result, error)

ResultToSchemaType converts result to schema type

func WorkloadToProvisionType 

func WorkloadToProvisionType(w workloads.Workloader) (*provision.Reservation, error)

WorkloadToProvisionType TfgridReservationWorkload1 to provision.Reservation

Types

type Counters 

type Counters struct {
	NRU primitives.CounterUint64 // network units
	// contains filtered or unexported fields
}

Counters tracks the amount of primitives workload deployed and the amount of resource unit used

func (*Counters) CheckMemoryRequirements 

func (c *Counters) CheckMemoryRequirements(r *provision.Reservation, totalMemAvailable uint64) error

CheckMemoryRequirements implements the provision.Statser interface on the tfgateway Counters.

func (*Counters) CurrentUnits 

func (c *Counters) CurrentUnits() directory.ResourceAmount

CurrentUnits return the number of each resource units reserved on the system

func (*Counters) CurrentWorkloads 

func (c *Counters) CurrentWorkloads() directory.WorkloadAmount

CurrentWorkloads return the number of each workloads provisioned on the system

func (*Counters) Decrement 

func (c *Counters) Decrement(r *provision.Reservation) error

Decrement is called by the provision.Engine when a reservation has been decommissioned

func (*Counters) Increment 

func (c *Counters) Increment(r *provision.Reservation) error

Increment is called by the provision.Engine when a reservation has been provisionned

type Delegate 

type Delegate struct {
	Domain string `json:"domain"`
}

Delegate is the primitives that allow a user to delegate a or part of a domain to us

type Feedback 

type Feedback struct {
	// contains filtered or unexported fields
}

Feedback is an implementation of the provision.Feedbacker that u sends results to the TFExplorer: https://github.com/threefoldtech/tfexplorer

func NewFeedback 

func NewFeedback(client *client.Client, converter provision.ResultConverterFunc) *Feedback

NewFeedback creates an ExplorerFeedback

func (*Feedback) Deleted 

func (e *Feedback) Deleted(nodeID, id string) error

Deleted implements provision.Feedbacker

func (*Feedback) Feedback 

func (e *Feedback) Feedback(nodeID string, r *provision.Result) error

Feedback implements provision.Feedbacker

func (*Feedback) UpdateStats 

func (e *Feedback) UpdateStats(nodeID string, w directory.WorkloadAmount, u directory.ResourceAmount) error

UpdateStats implements provision.Feedbacker

type Fs 

type Fs struct {
	sync.RWMutex
	// contains filtered or unexported fields
}

Fs is a in reservation cache using the filesystem as backend

func NewFSCache 

func NewFSCache(root string) (*Fs, error)

NewFSCache creates a in memory reservation store

func (*Fs) Add 

func (s *Fs) Add(r *provision.Reservation) error

Add a reservation to the store

func (*Fs) Close 

func (s *Fs) Close() error

Close makes sure the backend of the store is closed properly

func (*Fs) Exists 

func (s *Fs) Exists(id string) (bool, error)

Exists checks if the reservation ID is in the store

func (*Fs) Get 

func (s *Fs) Get(id string) (*provision.Reservation, error)

Get retrieves a specific reservation using its ID if returns a non nil error if the reservation is not present in the store

func (*Fs) GetExpired 

func (s *Fs) GetExpired() ([]*provision.Reservation, error)

GetExpired returns all id the the reservations that are expired at the time of the function call

func (*Fs) Remove 

func (s *Fs) Remove(id string) error

Remove a reservation from the store

func (*Fs) Sync 

func (s *Fs) Sync(statser provision.Statser) error

Sync update the statser with all the reservation present in the cache

type Gateway4to6 

type Gateway4to6 struct {
	PublicKey string `json:"public_key"`
}

Gateway4to6 is a primitive that allow client to have a ipv6 gateway

type Gateway4to6Result 

type Gateway4to6Result struct {
	IPs   []string  `json:"ips"`
	Peers []wg.Peer `json:"peers"`
}

Gateway4to6Result contains the configuration required by the user to start its 4to6 gateway tunnel

type Provisioner 

type Provisioner struct {
	Provisioners    map[provision.ReservationType]provision.ProvisionerFunc
	Decommissioners map[provision.ReservationType]provision.DecomissionerFunc
	// contains filtered or unexported fields
}

Provisioner hold all the logic responsible to provision and decomission the different primitives workloads defined by this package

func NewProvisioner 

func NewProvisioner(proxy *proxy.Mgr, dns *dns.Mgr, wg *wg.Mgr, kp identity.KeyPair, explorer *client.Client) *Provisioner

NewProvisioner creates a new 0-OS provisioner

type Proxy 

type Proxy struct {
	Domain  string `json:"domain"`
	Addr    string `json:"addr"`
	Port    uint32 `json:"port"`
	PortTLS uint32 `json:"port_tls"`
}

Proxy defines the configuration for a TCP proxy

type ReverseProxy 

type ReverseProxy struct {
	Domain string `json:"domain"`
	Secret string `json:"secret"`
}

ReverseProxy define a reverse tunnel TCP proxy

type Subdomain 

type Subdomain struct {
	Domain string   `json:"domain"`
	IPs    []net.IP `json:"destination"`
}

Subdomain defines a sub-domain from a mangaged or delagated domain

Source Files

View all Source files

Directories

Path Synopsis
cmd
tfgateway

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.