gateboard

README

gateboard

gateboard resolves AWS Private API Gateway ID.

Services

Service	Description
gateboard	Holds database of key value mappings: gateway_name => gateway_id. You can populate the database as you wish.
gateboard-discovery	Can be used to scan AWS API Gateway APIs and to save the name x ID mappings into gateboard.
gateboard-cache	Can be used as local fast cache to save resources on a centralized main gateboard service.

TODO

• SQS listener
• Client with async update
• Create mongodb index on startup
• Define TTL on server record (60s), restrict acceptable TTL range on client (60s..600s)
• Repository tests
• HTTP server tests
• SQS tests
• Docker image
• Client tests
• Refactor config
• Repository DynamoDB
• Optional authentication
• AWS Secrets Manager
• Generate token for optional authentication
• Gateway load sharing
• Repository redis
• Cache service
• Discovery service
• Metrics
• Repository S3
• Tracing
• Benchmark
• User guide
• Zap logging

Build

git clone https://github.com/udhos/gateboard
cd gateboard
CGO_ENABLED=0 go install ./...

Supported Repositories (Persistent Storage)

export REPO=mem      ;# testing-only pseudo-storage
export REPO=mongo    ;# MongoDB
export REPO=redis    ;# redis
export REPO=dynamodb ;# DynamoDB
export REPO=s3       ;# S3

Testing repository mongo

Start mongodb:

docker run --rm --name mongo-main -p 27017:27017 -d mongo

Run repository tests:

export TEST_REPO_MONGO=true ;# enable mongodb tests
go test -count=1 -run TestRepository ./cmd/gateboard

Testing repository dynamodb

Create a dynamodb table named gateboard_test with partition key gateway_name.

Make sure the table is empty before running the tests.

Run repository tests:

export TEST_REPO_DYNAMO=true ;# enable dynamodb tests
go test -count=1 -run TestRepository ./cmd/gateboard

Testing repository redis

Start redis:

docker run --rm --name redis-main -p 6379:6379 -d redis

Run repository tests:

export TEST_REPO_REDIS=true ;# enable redis tests
go test -count=1 -run TestRepository ./cmd/gateboard

Testing repository S3

Create a bucket.

Make sure the bucket is empty before running the tests.

Run repository tests:

export TEST_REPO_S3=put_bucket_name_here ;# enable S3 tests
go test -count=1 -run TestRepository ./cmd/gateboard

Optional Authentication

Enable WRITE_TOKEN=true in order to require token authentication for write requests.

export WRITE_TOKEN=true

Make sure the repository has the token token1 assigned to gateway gw2.

Example for mongodb:

db.gateboard.insertOne({"gateway_name":"gw2","token":"token1"})

Now requests to update gateway gw2 must include the token token1.

curl -X PUT -s -d '{"gateway_id":"id1","token":"token1"}' localhost:8080/gateway/gw2

{"gateway_name":"gw2","gateway_id":"id1"}

Otherwise the request will be denied.

curl -X PUT -v -d '{"gateway_id":"id2"}' localhost:8080/gateway/gw2
*   Trying ::1:8080...
* TCP_NODELAY set
* Connected to localhost (::1) port 8080 (#0)
> PUT /gateway/gw2 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.68.0
> Accept: */*
> Content-Length: 20
> Content-Type: application/x-www-form-urlencoded
> 
* upload completely sent off: 20 out of 20 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
< Content-Type: application/json; charset=utf-8
< Date: Sat, 17 Dec 2022 00:59:22 GMT
< Content-Length: 65
< 
* Connection #0 to host localhost left intact
{"gateway_name":"gw2","gateway_id":"id2","error":"invalid token"}

Example

curl localhost:8080/gateway/gate1
{"gateway_name":"gate1","gateway_id":"","error":"gatewayGet: not found: repository gateway not found error"}

curl -X PUT -d '{"gateway_id":"id1"}' localhost:8080/gateway/gate1
{"gateway_name":"gate1","gateway_id":"id1"}

curl localhost:8080/gateway/gate1
{"gateway_name":"gate1","gateway_id":"id1"}

curl -X PUT -d '{"gateway_id":"id2"}' localhost:8080/gateway/gate1
{"gateway_name":"gate1","gateway_id":"id2"}

curl localhost:8080/gateway/gate1
{"gateway_name":"gate1","gateway_id":"id2"}

gateway-discovery

Save to server

Discovery writes directly to server.

Start server.

export REPO=mem
gateboard

Run discovery.

export SAVE=server
export DRY_RUN=false
gateboard-discovery

Dump database.

curl localhost:8080/dump | jq

Save to webhook

Discovery writes to webhook that forwards to SQS queue.

Start server.

export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/gateboard
export REPO=mem
gateboard

Run discovery.

export SAVE=webhook
# use lambda function url as webhook
export WEBHOOK_URL=https://xxxxxxxxxxxxx.lambda-url.us-east-1.on.aws
export DRY_RUN=false
gateboard-discovery

Dump database.

curl localhost:8080/dump | jq

Save to SQS

Discovery writes to SQS queue.

Start server.

export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/gateboard
export REPO=mem
gateboard

Run discovery.

export SAVE=sqs
export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/gateboard
export DRY_RUN=false
gateboard-discovery

Dump database.

curl localhost:8080/dump | jq

Save to SNS

Discovery writes to SNS topic that forwards to SQS queue.

Start server.

export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/gateboard
export REPO=mem
gateboard

Run discovery.

export SAVE=sns
export TOPIC_ARN=arn:aws:sns:us-east-1:123456789012:gateboard
export DRY_RUN=false
gateboard-discovery

Dump database.

curl localhost:8080/dump | jq

Save to lambda

Discovery writes to lambda function that forwards to SQS queue.

Start server.

export QUEUE_URL=https://sqs.us-east-1.amazonaws.com/123456789012/gateboard
export REPO=mem
gateboard

Run discovery.

export SAVE=lambda
export LAMBDA_ARN=arn:aws:lambda:us-east-1:123456789012:function:forward_to_sqs
export DRY_RUN=false
gateboard-discovery

Dump database.

curl localhost:8080/dump | jq

AWS Secrets Manager

Retrieve config vars from AWS Secrets Manager.

export CONFIG_VAR=aws-secretsmanager:region:name:json_field

See detailed documentation at https://github.com/udhos/boilerplate.

Example:

export MONGO_URL=aws-secretsmanager::mongo_uri

# The secret `mongo_uri` must store a scalar value like: `mongodb://127.0.0.1:27017`

Example with JSON field uri:

export MONGO_URL=aws-secretsmanager::mongo:uri

# The secret `mongo` must store a JSON value like: `{"uri":"mongodb://127.0.0.2:27017"}`

Metrics

# HELP http_server_requests_seconds Spring-like server request duration in seconds.
# TYPE http_server_requests_seconds histogram

Example: http_server_requests_seconds_bucket{method="GET",status="200",uri="/gateway/*gateway_name",le="0.001"} 4

# HELP repository_requests_seconds Repository request duration in seconds.
# TYPE repository_requests_seconds histogram

Example: repository_requests_seconds_bucket{method="get",status="success",le="0.00025"} 4

Docker

Docker hub:

https://hub.docker.com/r/udhos/gateboard

Pull from docker hub:

docker pull udhos/gateboard:0.0.0

Build recipe:

./docker/build.sh

Multiarch build recipe:

./docker/build-multiarch.sh

gateboard-discovery docker image

Docker hub:

https://hub.docker.com/r/udhos/gateboard-discovery

Pull from docker hub:

docker pull udhos/gateboard-discovery:0.0.0

Build recipe:

./docker/build-discovery.sh

Push:

docker push -a udhos/gateboard-discovery