ExperienceOne Golang APIKit
Overview
The APIKit enables the rapid development of Web APIs with Golang by providing the functionality to (re-)generate the communication layer of the API based on an OpenAPIv2 (Swagger) definition, both for the client and server part. It also helps with one-time generation of stubs for the server-side endpoint handlers.
The generated API code does fully handle (and thus hide) the HTTP layer of the application, so the developer can focus on business logic implementation. Integration between API core and HTTP layer is handled via generated structs and interfaces.
The API HTTP layer can be regenerated from the OpenAPIv2 definition without breaking the integration with the API core code. This enables a "definition first" approach that ensures a 100% match of OpenAPI / Swagger definition and the implemented API.
Specifically the APIKit contains a CLI to generate and update the following API related items:
- standard project directory structures and common files
- validation of OpenAPIv2 (Swagger) definition
- HTTP API client based on an OpenAPIv2 (Swagger) definition
- HTTP API server based on an OpenAPIv2 (Swagger) definition
- stubs for server API endpoint handlers
See the compliance list for a detailed list of supported OpenAPIv2 features.
Requirements
- Golang 1.11 or higher
- make (if you want to use the Makefile)
Installation
git clone git@gitlab.nolteundlauth.de:golang/apikit.git
cd apikit
make install
Usage
Generate standart project structure
The command apikit project <dest.dir> <path/of/package>
generates a standard project directory.
cd go/src
$GOPATH/bin/apikit project myproject myproject
The project
command takes the name of the directory to create and the path of the main package as arguments It creates a CLI stub in the cmd
subdirectory and a .gitignore
file.
Note that it is not necessary to use the standard project structure with the APIKit.
Define the API with OpenAPIv2
The next step is to define your API. It can be useful to use a Swagger editor for this task. By convention the Swagger definition is stored in the doc
subdirectory.
For showcasing the following example definition will be used:
---
swagger: "2.0"
info:
title: "vis-admin"
version: "1.0.0"
consumes:
- "application/json"
produces:
- "application/json"
paths:
/api/client/{clientId}:
put:
summary: "Create or update client"
operationId: "CreateOrUpdateClient"
consumes: []
parameters:
- name: "X-Auth"
in: "header"
required: true
type: "string"
description: "Authentication token"
- name: "clientId"
in: "path"
required: true
type: "string"
description: "Client ID"
- name: "body"
in: "body"
required: true
schema:
$ref: "#/definitions/Client"
responses:
200:
description: "Updated"
schema:
$ref: "#/definitions/Client"
201:
description: "Created"
schema:
$ref: "#/definitions/Client"
400:
description: "Malformed request body"
403:
description: "Not authenticated"
405:
description: "Not allowed"
definitions:
Client:
type: "object"
required:
- "id"
- "name"
properties:
id:
type: "string"
name:
type: "string"
activePresets:
type: "string"
Validate the OpenAPIv2 definition
The command apikit validate <api.yaml>
validates against the given OpenAPIv2 / Swagger definition.
cd myproject
$GOPATH/bin/apikit validate doc/myproject.yaml
Generate API server and client
The apikit generate <api.yaml> <dest.dir> <package>
command generates the client and server API based on an OpenAPIv2 definition. Use --only-client
if the server component and --only-server
if the client component is not needed.
$GOPATH/bin/apikit generate doc/myproject.yaml api api
Note that all endpoints need to have an operation ID to generate successfully. The generated code should not be edited manually. Instead, the OpenAPI definition should be updated and the source regenerated with APIKit generate
command.
Using the API client
The client.go
file contains an interface
defining the programming interface of the API.
type VisAdminClient interface {
CreateOrUpdateClient(request *CreateOrUpdateClientRequest) (CreateOrUpdateClientResponse, error)
}
The request
parameter contains all data that is sent to the server with the API call. Every API call returns a response interface that can be casted to the actual, HTTP return code specific struct, and an error
code if the communication with the server went wrong. The content of the response is only valid if the error
return value is nil
.
The client.go
file also contains an implementation of the programming interface using HTTP calls to the actual API. There is a constructor for the client implementation which takes an http.Client
and the API base URL as parameters. Via the http.Client
network handling and additional http.RoundTripper
can be configured and integrated with the API client.
The steps for using the client to make an API call are:
- create the client implementation via constructor
- fill request struct with appropriate data
- call the API via the function for the endpoint
- check error return value
- check HTTP status code of the response
- cast response to correct struct
This is illustrated in the following example:
package myproject_test
import (
"myproject/api"
"net/http"
"testing"
)
const (
baseUrl = "http://api.myproject.com/"
testClientId = "clientId"
testApiKey = "apiKey"
testName = "name"
)
func TestClient(t *testing.T) {
// create instance of API client
client := api.NewVisAdminClient(&http.Client{}, baseUrl, api.Opts{})
// create request struct
request := api.CreateOrUpdateClientRequest{
ClientId: testClientId,
XAuth: testApiKey,
Body: api.Client{
Id: testClientId,
Name: testName,
},
}
// call API with request
response, err := client.CreateOrUpdateClient(&request)
if err != nil {
t.Fatal(err)
}
// check HTTP return code
if response.StatusCode() != http.StatusOK {
t.Fatalf("API returned error code %d", response.StatusCode())
}
// cast response appropriatly and use returned information
t.Log(response.(*api.CreateOrUpdateClient200Response).Body.Name)
}
Note that for testing the API can be mocked by a local implementation of the API client programming interface.
Using the API server
The file server.go
contains an a full HTTP server that serves the specified API. Additional to the defined endpoints it contains the /spec
endpoint that delivers the OpenAPIv2 / Swagger definition the server was generated with. The /spec
endpoint can be used to visualize the API via UIs like Swagger UI.
The implementation of the API endpoints has to be done in handler functions that take the request data as parameter and return an implementation of the endpoint-specific response interface. The handler functions need to be registered before starting the server. If there is no handler registered for an endpoint, the server will respond with status code 404 as default.
The steps for using the API server are:
- create the server via constructor
- register endpoint handlers
- start listening on network socket
The following example shows how the server is initialized and than called by the generated client (see above).
package myproject_test
import (
"context"
"myproject/api"
"net/http"
"testing"
"time"
)
const (
baseUrl = "http://localhost:8080"
testClientId = "clientId"
testApiKey = "apiKey"
testName = "name"
)
func CreateOrUpdateClient(ctx context.Context, request *api.CreateOrUpdateClientRequest) api.CreateOrUpdateClientResponse {
return &api.CreateOrUpdateClient200Response{
Body: request.Body,
}
}
func TestServer(t *testing.T) {
// create server instance
server := api.NewVisAdminServer(nil)
// register endpoint handler
server.SetCreateOrUpdateClientHandler(CreateOrUpdateClient)
// start server to listen for requests
go func() {
t.Log(server.Start(8080))
}()
time.Sleep(1 * time.Second)
// create instance of API client
client := api.NewVisAdminClient(&http.Client{}, baseUrl, api.Opts{})
// create request struct
request := api.CreateOrUpdateClientRequest{
ClientId: testClientId,
XAuth: testApiKey,
Body: api.Client{
Id: testClientId,
Name: testName,
},
}
// call API with request
response, err := client.CreateOrUpdateClient(&request)
if err != nil {
t.Fatal(err)
}
// check HTTP return code
if response.StatusCode() != http.StatusOK {
t.Fatalf("API returned error code %d", response.StatusCode())
}
// cast response appropriate and use returned information
t.Log(response.(*api.CreateOrUpdateClient200Response).Body.Name)
// stop server
server.Stop()
}
Generate handler stubs
For one-time generation of stubs for the endpoint handlers the command apikit handlers <api.yaml> <dest.go> <package> <api/package/path>
can be used. Especially for large APIs it saves some typing work by writing out the boilerplate code for the handlers.
$GOPATH/bin/apikit handlers doc/myproject.yaml handlers.go myproject myproject/api
Generate service stubs
For larger projects grouping the endpoint handlers in service classes is convenient. A service stub can be generated with apikit service <api.yaml> <dest.go> <package> <tag> <api/package/path>
. The command will generate the endpoints that are tagged with tag
only.
$GOPATH/bin/apikit service doc/myproject.yaml client_service.go myproject client myproject/api
Validation of request data
The generated server does validate the request against the constraints defined in the OpenAPIv2 specification. If the validation fails, the server will respond with a 400 Bad Request
status code to the client.
To pass more information about the invalid data to the client, the 400 Bad Request
response with the body type ValidationErrors
has to be added to the endpoint. The generated server will automatically use it to send details about malformed data to the client.
# ... endpoint definition ...
responses:
400:
description: "Bad request"
schema:
$ref: "#/definitions/ValidationErrors"
definitions:
ValidationErrors:
type: "object"
properties:
Message:
type: "string"
Errors:
type: "array"
items:
$ref: "#/definitions/ValidationError"
ValidationError:
type: "object"
properties:
Message:
type: "string"
Field:
type: "string"
Code:
type: "string"
Required and non-required fields
The generated types do reflect if a field is required or not by making not required fields pointers. If a non-required field is not present in the request data or a field is present, but has the JSON value null
, the pointer will be set to nil
. Otherwise the pointer will point to the actual data. Required fields are not allowed to be not present or to be null
. If they are, a 400 Bad Request
response is returned to the client.
The proper handling of required and non-required fields allows to correctly distinguish between zero values ("", 0, false) and non-present values. Note that the default status for fields in OpenAPIv2 is non-required. To avoid nil pointer checks mark as many fields required
as appropriate.
String validation
The string validation supports the minLength
, maxLength
and the pattern
attributes. pattern
allows to specify a regular expression that the string has to match. If the string has the format uuid
, url
or email
, it is automatically matched against the corresponding regular expression.
Integer validation
The integer validation supports the minimum
, maximum
, exclusiveMinimum
and exclusiveMaximum
attributes.
Content types
The client needs to send the correct content type header matching the consumes attribute in the OpenAPIv2 definition. If not, the server responds with a 415 Unsupported media type
error.
Enum support
The APIKit generator has limited enum support. It is required that every enum definition occurs only once in the whole OpenAPIv2 specification. The best way to achieve this is to reference enum types where they are used and have a global definition of the enum type in the definitions
section of the specification.
Car:
type: object
required:
- driveConcept
properties:
driveConcept:
$ref: '#/definitions/DriveConcept'
DriveConcept:
type: string
enum:
- COMBUSTOR
- HYBRID
- ELECTRIC
- FUELCELL
- UNDEFINED
This produces the following code in types.go
:
type DriveConcept string
const (
DriveConceptCOMBUSTOR DriveConcept = "COMBUSTOR"
DriveConceptHYBRID DriveConcept = "HYBRID"
DriveConceptELECTRIC DriveConcept = "ELECTRIC"
DriveConceptFUELCELL DriveConcept = "FUELCELL"
DriveConceptUNDEFINED DriveConcept = "UNDEFINED"
)
type (
Car struct {
DriveConcept DriveConcept `json:"driveConcept" bson:"driveConcept"`
}
)
Advanced features
Error logging
An error logger can be passed to the server via the ErrorHandler
attribute of the ServerOpts
struct. The logger will be called when something unexpected goes wrong inside the HTTP layer.
Additional routes for the server
The ServerOpts
struct can be used to pass a function to the generated server that is executed during server startup and has access to the internal router object. It is defined as func(router *routing.Router)
.
With this function routes that are not defined in the OpenAPIv2 specification or are implemented by packages that come with an own HTTP handler can be added to the server.
import(
...
"github.com/go-ozzo/ozzo-routing"
)
func NewServer() {
onStart := func(router *routing.Router) {
router.Any("/metrics", func(ctx *routing.Context) error {
metrics.ServeHTTP(ctx.Response, ctx.Request)
return nil
})
}
server := api.NewVisAdminServer(&api.ServerOpts{OnStart: onStart})
Middleware components
Functions that shall be executed every time an endpoint is called can be added to the server and to individual handlers via middleware components. A middleware is defined by a struct that holds the handler function (defined in ozzo-routing
) and a flag, if the middleware shall be executed before or after the endpoint handler.
Middleware struct {
Handler routing.Handler
After bool
}
Server-wide middleware components can be passed via the ServerOpts
struct.
middleware := []api.Middleware{{Handler: middleware.RequestID().Handler}}
server := api.NewVisAdminServer(&api.ServerOpts{Middleware: middleware})
Handler-specific middleware can be added via a var-arg list at handler registration.
sessionMiddleware := xcustomer.Middleware{Handler: middleware.SessionHandler(s.sessions)}
server.SetCreateOrUpdateClientHandler(CreateOrUpdateClient, sessionMiddleware)
Middleware can use c.Request.Context()
object to pass data to the handler function. For example the session handler can check for a valid session token and load session data into the context
object of the handler if successful.
func SessionHandler(service session.Service) routing.Handler {
return func(c *routing.Context) error {
authToken := c.Request.Header.Get(XAuthTokenHeader)
if len(authToken) == 0 {
return xcustomer.NewHTTPStatusCodeError(http.StatusBadRequest)
}
sess, err := service.GetSession(authToken)
if err != nil {
if err.StatusCode == http.StatusInternalServerError {
return xcustomer.NewHTTPStatusCodeError(http.StatusInternalServerError)
}
return xcustomer.NewHTTPStatusCodeError(http.StatusUnauthorized)
}
if sess == nil {
return xcustomer.NewHTTPStatusCodeError(http.StatusUnauthorized)
}
// add session data to the context of the endpoint handler
ctx := context.WithValue(c.Request.Context(), xcontext.SessionKey, sess)
c.Request = c.Request.WithContext(ctx)
return c.Next()
}
}
Server-side request and response logging
The APIKit middleware
package provides a component for server-side request and response logging. The middleware records request and response and forwards it to a user-defined log function before sending the data to the client. LogResponseWriter
is part of the APIKit middleware
package.
rrLogger := func(r *http.Request, w *LogResponseWriter, elapsed time.Duration) {
// ... print request and response infos as suitable
}
middleware := []api.Middleware{{Handler: middleware.Log(rrLogger)}}
There is also a convenience wrapper for the log function.
timeFormat := time.RFC3339
pathsToIgnore := []string{...} // endpoints not to log
logger := logFunc func(entity middleware.LogEntry, values ...interface{}) {
// use entity object to print log
}
rrLogger := middleware.NewLogger(timeFormat, pathsToIgnore, logger)
GDPR compliant request and response logging
In order to comply with the GDPR no data that could identify a person must be stored in log files. Therefore, a middleware logger is provided that can mask specified fields in JSON request and response bodies.
fieldsToMask := []string{...} // list of JSON paths
typesToMask := []reflect.Type{...} // list of types
rrLogger := middleware.NewMaskingLogger(timeFormat, fieldsToMask, typesToMask, pathsToIgnore, logger)
Client-side request and response logging
The APIKit roundtripper
package contains components for client-side request and response logging. The simplest roundtripper forwards the HTTP request and response to a log function. The roundtripper.Use()
function wraps the transport of an HTTP client with the given RoundTripper
.
logFunc := func(req *http.Request, resp *http.Response) {
// ... log request and response content ...
}
client := roundtripper.Use(&http.Client{}, roundtripper.Logger(logFunc))
Additionally there is a convenience roundtripper that does an httputil.Dump..()
on request and response and forwards the result a generic logging function.
logger := func(a ...interface{}) {
// log a[0]
}
client := roundtripper.Use(&http.Client{}, roundtripper.Dump(logger))
The logging function is defined as an interface var-arg list so that it's possible to use standard functions of logging frameworks like logrus
. The roundtripper does always call it with one string parameter.
Up- and downloading files as streams
Up- and downloading binary data in JSON format requires to put whole files in memory while marshalling / unmarshalling the JSON. This can quickly overwhelm the server. A better approach is to handle files as streams. The APIKit supports this via the type: file
attribute.
When uploading a file the content type multipart/form-data
(consumes
attribute) is required.
/upload:
post:
summary: Upload a file with others data
consumes:
- multipart/form-data
operationId: PostUpload
parameters:
- in: formData
name: upfile
type: file
description: the file to upload
- in: formData
name: note
type: string
maxLength: 4000
pattern: "^[0-9a-zA-Z ]*$"
description: Description of file
responses:
'200':
description: Status 200
'500':
description: Status 500
The request struct that is passed to the endpoint handler now contains a FormData object that includes an io.ReadCloser
to access the file content. Note that the io.ReadCloser
needs to be closed by the handler to prevent memory leaks.
func PostUpload(ctx context.Context, request *PostUploadRequest) PostUploadResponse {
defer request.FormData.Upfile.Content.Close()
data, err := ioutil.ReadAll(request.FormData.Upfile.Content)
if err != nil {
log.Println(fmt.Sprintf("error reading uploaded file (%v)", err))
return new(PostUpload500Response)
}
// .. do something with data
return new(PostUpload200Response)
}
To deliver a file as stream to the client, the response body must have the type: file
. In the example below the Content-Type
header specified explicitly to overwrite the Content-Type
generated by the produces
attribute.
'/download/{image}':
get:
produces:
- image/png
summary: Retrieve a image
description: Retrieve a image
operationId: DownloadImage
parameters:
- in: path
name: image
description: The image name of the image
type: string
required: true
responses:
'200':
description: image to download
schema:
type: file
headers:
Content-Type:
type: "string"
'500':
description: Internal server error
The handle than has to pass an io.ReadCloser
with the response. The open stream is closed in the HTTP layer of the generated server.
func DownloadImage(ctx context.Context, request *DownloadImageRequest) DownloadImageResponse {
file, err := os.Open(request.Name)
if err != nil {
return &DownloadImage500Response{}
}
return &DownloadImage200Response{Body: file, ContentType: "image/png"}
}
APIKit development notes
If you have found a bug or would like contributed to the project please check out our Contribution Guidelines.
After making changes to the APIKit source do run the testsuite via the Makefile
.
make test
When there were changes to the internal/framework
package, update the framework
files before committing and building the executable.
make framework
When debugging code generation consider enabling the debug mode of the generate
command. When activated the generator will save the generated code regardless of errors into a temporary file and it print its location to the terminal.