bricksllm

README

BricksLLM: AI Gateway For Putting LLM In Production

[!TIP] A managed version of BricksLLM is also available! It is production ready, and comes with a dashboard to make interacting with BricksLLM easier. Try us out for free today!

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM:

• Set LLM usage limits for users on different pricing tiers
• Track LLM usage on a per user and per organization basis
• Block or redact requests containing PIIs
• Improve LLM reliability with failovers, retries and caching
• Distribute API keys with rate limits and cost limits for internal development/production use cases
• Distribute API keys with rate limits and cost limits for students

Features

• PII detection and masking
• Rate limit
• Cost control
• Cost analytics
• Request analytics
• Caching
• Request Retries
• Failover
• Model access control
• Endpoint access control
• Native support for all OpenAI endpoints
• Native support for Anthropic
• Native support for Azure OpenAI
• Native support for vLLM
• Native support for Deepinfra
• Support for custom deployments
• Integration with custom models
• Datadog integration
• Logging with privacy control

Getting Started

The easiest way to get started with BricksLLM is through BricksLLM-Docker.

Step 1 - Clone BricksLLM-Docker repository

git clone https://github.com/bricks-cloud/BricksLLM-Docker

Step 2 - Change to BricksLLM-Docker directory

cd BricksLLM-Docker

Step 3 - Deploy BricksLLM locally with Postgresql and Redis

docker compose up

You can run this in detach mode use the -d flag: docker compose up -d

Step 4 - Create a provider setting

curl -X PUT http://localhost:8001/api/provider-settings \
   -H "Content-Type: application/json" \
   -d '{
          "provider":"openai",
          "setting": {
             "apikey": "YOUR_OPENAI_KEY"
          }
      }'   

Copy the id from the response.

Step 5 - Create a Bricks API key

Use id from the previous step as settingId to create a key with a rate limit of 2 req/min and a spend limit of 25 cents.

curl -X PUT http://localhost:8001/api/key-management/keys \
   -H "Content-Type: application/json" \
   -d '{
	      "name": "My Secret Key",
	      "key": "my-secret-key",
	      "tags": ["mykey"],
        "settingIds": ["ID_FROM_STEP_FOUR"],
        "rateLimitOverTime": 2,
        "rateLimitUnit": "m",
        "costLimitInUsd": 0.25
      }'   

Congratulations you are done!!!

Then, just redirect your requests to us and use OpenAI as you would normally. For example:

curl -X POST http://localhost:8002/api/providers/openai/v1/chat/completions \
   -H "Authorization: Bearer my-secret-key" \
   -H "Content-Type: application/json" \
   -d '{
          "model": "gpt-3.5-turbo",
          "messages": [
              {
                  "role": "system",
                  "content": "hi"
              }
          ]
      }'

Or if you're using an SDK, you could change its baseURL to point to us. For example:

// OpenAI Node SDK v4
import OpenAI from 'openai';

const openai = new OpenAI({
  apiKey: "some-secret-key", // key created earlier
  baseURL: "http://localhost:8002/api/providers/openai/v1", // redirect to us
});

How to Update?

For updating to the latest version

docker pull luyuanxin1995/bricksllm:latest

For updating to a particular version

docker pull luyuanxin1995/bricksllm:1.4.0

Documentation

Environment variables

Name	type	description	default
POSTGRESQL_HOSTS	required	Hosts for Postgresql DB. Separated by ,	localhost
POSTGRESQL_DB_NAME	optional	Name for Postgresql DB.
POSTGRESQL_USERNAME	required	Postgresql DB username
POSTGRESQL_PASSWORD	required	Postgresql DB password
POSTGRESQL_SSL_MODE	optional	Postgresql SSL mode	disable
POSTGRESQL_PORT	optional	The port that Postgresql DB runs on	5432
POSTGRESQL_READ_TIME_OUT	optional	Timeout for Postgresql read operations	2m
POSTGRESQL_WRITE_TIME_OUT	optional	Timeout for Postgresql write operations	5s
REDIS_HOSTS	required	Host for Redis. Separated by ,	localhost
REDIS_PASSWORD	optional	Redis Password
REDIS_PORT	optional	The port that Redis DB runs on	6379
REDIS_READ_TIME_OUT	optional	Timeout for Redis read operations	1s
REDIS_WRITE_TIME_OUT	optional	Timeout for Redis write operations	500ms
IN_MEMORY_DB_UPDATE_INTERVAL	optional	The interval BricksLLM API gateway polls Postgresql DB for latest key configurations	1s
STATS_PROVIDER	optional	This value can only be datadog. Required for integration with Datadog.
PROXY_TIMEOUT	optional	Timeout for proxy HTTP requests.	600s
NUMBER_OF_EVENT_MESSAGE_CONSUMERS	optional	Number of event message consumers that help handle counting tokens and inserting event into db.	3
AWS_SECRET_ACCESS_KEY	optional	It is for PII detection feature.	5s
AWS_ACCESS_KEY_ID	optional	It is for using PII detection feature.	5s
AMAZON_REGION	optional	Region for AWS.	us-west-2
AMAZON_REQUEST_TIMEOUT	optional	Timeout for amazon requests.	5s
AMAZON_CONNECTION_TIMEOUT	optional	Timeout for amazon connection.	10s
AWS_SECRET_ACCESS_KEY	optional	Required for PII detection features.
AWS_ACCESS_KEY_ID	optional	Required for PII detection features.

Configuration Endpoints

The configuration server runs on Port 8001.

Headers

name	type	data type	description
X-API-KEY	optional	string	Key authentication header.

Get keys: GET /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Query Parameters

name	type	data type	description
tag	optional	string	Identifier attached to a key configuration
tags	optional	[]string	Identifiers attached to a key configuration
provider	optional	string	Provider attached to a key provider configuration. Its value can only be openai.
keyIds	optional	[]string	Unique identifiers for keys.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Response

Response Body
[]KeyConfiguration

KeyConfiguration

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	bool	false	Flag controls whether or not the key should be hashed.

Get keys V2: POST /api/v2/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request

Field	required	type	example	description
keyIds	optional	[]string	["98daa3ae-961d-4253-bf6a-322a32fdca3d"]	Array of key IDs.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with keys.
name	optional	string	key	Filter keys by the substring match on the name field.
revoked	optional	bool	false	Filters keys by revoked status.
limit	optional	int	5	Pagination limit.
offset	optional	int	5	Pagination offset.
order	optional	string	asc	asc and desc are the two available values.
returnCount	optional	bool	true	Flag that controls whether the response should contain total key counts.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/v2/key-management/keys

Response

Field	type	example	description
keys	[]key	Array of keys.
count	int	123	Total number of keys.

key

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	bool	false	Flag controls whether or not the key should be hashed.

Create key: PUT /api/key-management/keys

Description

This endpoint is set up for retrieving key configurations using a query param called tag.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
name	required	string	spike's developer key	Name of the API key.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with the key.
key	required	string	abcdef12345	API key.
settingId	depercated	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	required	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Setting ids associated with the key.
costLimitInUsd	optional	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	enum	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	optional	string	24h	time to live. Available units are [s, m, h].
allowedPaths	optional	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Paths allowed for access.
shouldLogRequest	optional	bool	false	Should request be stored.
shouldLogResponse	optional	bool	true	Should response be stored.
rotationEnabled	optional	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	optional	bool	false	Flag controls whether or not the key should be hashed.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Responses

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d].
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	bool	false	Flag controls whether or not the key should be hashed.

Update key: PATCH /api/key-management/keys/{keyId}

Description

This endpoint is set up for updating key configurations using key id.

Parameters

name	type	data type	description
keyId	required	string	Unique key configuration identifier.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
settingId	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Setting ids associated with the key.
name	optional	string	spike's developer key	Name of the API key.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with the key.
revoked	optional	boolean	true	Indicator for whether the key is revoked.
revokedReason	optional	string	The key has expired	Reason for why the key is revoked.
costLimitInUsd	optional	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
allowedPaths	optional	[{ "path": "/api/providers/openai/v1/chat/completions", "method": "POST"}]	``	Paths allowed for access.
shouldLogRequest	optional	bool	false	Should request be stored.
shouldLogResponse	optional	bool	true	Should response be stored.
rotationEnabled	optional	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	optional	bool	false	Flag controls whether or not the key should be hashed.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/key-management/keys

Response

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	bool	false	Flag controls whether or not the key should be hashed.

Get top spending key Ids: POST /api/reporting/top-keys

Description

This endpoint is set up for getting a list of key ids ordered by spend.

Request

Field	required	type	example	description
start	required	int64	1257894000	Start unix timestamp.
end	required	int64	1257894000	End unix timestamp
keyIds	optional	[]string	["98daa3ae-961d-4253-bf6a-322a32fdca3d"]	Array of key IDs.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with keys.
name	optional	string	key	Filter keys by the substring match on the name field.
revoked	optional	bool	false	Filters keys by revoked status.
limit	optional	int	5	Pagination limit.
offset	optional	int	5	Pagination offset.
order	optional	string	asc	asc and desc are the two available values.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/reporting/top-keys

Response

Field	type	example	description
name	string	spike's developer key	Name of the API key.
createdAt	int64	1257894000	Key configuration creation time in unix.
updatedAt	int64	1257894000	Key configuration update time in unix.
revoked	boolean	true	Indicator for whether the key is revoked.
revokedReason	string	The key has expired	Reason for why the key is revoked.
tags	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyId	string	550e8400-e29b-41d4-a716-446655440000	Unique identifier for the key.
costLimitInUsd	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	2d	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", method: "POST"}]	Allowed paths that can be accessed using the key.
settingId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This field is DEPERCATED. Use settingIds field instead.
settingIds	string	[98daa3ae-961d-4253-bf6a-322a32fdca3d]	Setting ids associated with the key.
shouldLogRequest	bool	false	Should request be stored.
shouldLogResponse	bool	true	Should response be stored.
rotationEnabled	bool	false	Should key rotate setting used to access third party endpoints in order to circumvent rate limits.
policyId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Policy id associated with the key.
isKeyNotHashed	bool	false	Flag controls whether or not the key should be hashed.

Create a provider setting: POST /api/provider-settings

Description

This endpoint is creating a provider setting.

Request

Field	required	type	example	description
provider	required	enum	openai	This value can only be openai, anthropic and azure as for now.
setting	required	Setting	{ "apikey": "YOUR_OPENAI_KEY" }	A map of values used for authenticating with the selected provider.
name	optional	string	YOUR_PROVIDER_SETTING_NAME	This field is used for giving a name to provider setting
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Setting

Field	required	type	example	description
apiKey	required	string	xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	This value is required.
url	required	string	https://your.deployment.url	This value is required when the provider is vllm
resourceName	required	string	YOUR_AZURE_RESOURCE_NAME	This value is required when the provider is azure.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier.
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Get provider settings: GET /api/provider-settings

Description

This endpoint is getting provider settings.

Query Parameters

name	type	data type	description
ids	optional	[]string	Provider setting ids

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

[]ProviderSetting

ProviderSetting

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier.
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Update a provider setting: PATCH /api/provider-settings/:id

Description

This endpoint is updating a provider setting .

Parameters

name	type	data type	description
id	required	string	Unique identifier for the provider setting that you want to update.

Request

Field	required	type	example	description
setting	required	Setting	{ "apikey": "YOUR_OPENAI_KEY" }	A map of values used for authenticating with the selected provider.
name	optional	string	YOUR_PROVIDER_SETTING_NAME	This field is used for giving a name to provider setting
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Setting

Field	required	type	example	description
apiKey	required	string	xx-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx	This value is required.
url	required	string	https://your.deployment.url	This value is required when the provider is vllm
resourceName	required	string	YOUR_AZURE_RESOURCE_NAME	This value is required when the provider is azure.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
createdAt	int64	1699933571	Unix timestamp for creation time.
updatedAt	int64	1699933571	Unix timestamp for update time.
provider	enum	openai	This value can only be openai as for now.
id	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	This value is a unique identifier
name	string	YOUR_PROVIDER_SETTING_NAME	Provider setting name.
allowedModels	[]string	["text-embedding-ada-002"]	Allowed models for this provider setting.

Retrieve Metrics: POST /api/reporting/events

Description

This endpoint is retrieving aggregated metrics given an array of key ids and tags.

Request

Field	required	type	example	description
keyIds	required	[]string	["key-1", "key-2", "key-3" ]	Array of ids that specicify the keys that you want to aggregate stats from.
tags	required	[]string	["tag-1", "tag-2"]	Array of tags that specicify the key tags that you want to aggregate stats from.
customIds	required	[]string	["customId-1", "customId-2"]	A list of custom IDs that you want to aggregate stats from.
filters	required	[]string	["model", "keyId"]	Group by data points through different filters(model,keyId, userId or customId).
start	required	int64	1699933571	Start timestamp for the requested timeseries data.
end	required	int64	1699933571	End timestamp for the requested timeseries data.
increment	required	int	60	This field is the increment in seconds for the requested timeseries data.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

Field	type	example	description
dataPoints	[]dataPoint	[{ "timeStamp": 1699933571, "numberOfRequests": 1, "costInUsd": 0.8, "latencyInMs": 600, "promptTokenCount": 0, "completionTokenCount": 0, "successCount": 1 }]	Unix timestamp for creation time.
latencyInMsMedian	float64	656.7	Median latency for the given time period.
latencyInMs99th	float64	555.7	99th percentile latency for the given time period.

Datapoint

Field	type	example	description
timeStamp	int64	1702504746	Unix timestamp for the data point
numberOfRequests	int64	100	Aggregated number of http requests over the given time increment.
costInUsd	float64	1.7	Aggregated cost of proxied requests in USD over the given time increment.
latencyInMs	int	555	Aggregated latency in milliseconds of http requests over the given time increment.
promptTokenCount	int	25	Aggregated prompt token counts over the given time increment.
completionTokenCount	int	4000	Aggregated completion token counts over the given time increment.
successCount	int	555	Aggregated number of successful http requests over the given time increment.
keyId	int	555.7	key Id associated with the event.
model	string	gpt-3.5-turbo	model associated with the event.
customId	string	customId	customId associated with the event.

Get events: GET /api/events

Description

This endpoint is for getting events.

Query Parameters

name	type	data type	description
customId	optional	string	Custom identifier attached to an event.
keyIds	optional	[]string	A list of key IDs.
start	required if keyIds is specified	int64	Start timestamp.
end	required if keyIds is specified	int64	End timestamp.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/provider-settings

Response

[]Event

Event

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
tags	int64	["YOUR_TAG"]	Tags of the key.
key_id	string	YOUR_KEY_ID	Key Id associated with the proxy request.
cost_in_usd	float64	0.0004	Cost incured by the proxy request.
model	string	gpt-4-1105-preview	Model used in the proxy request.
provider	string	openai	Provider for the proxy request.
status	int	200	Http status.
prompt_token_count	int	8	Prompt token count of the proxy request.
completion_token_count	int	16	Completion token counts of the proxy request.
latency_in_ms	int	160	Provider setting name.
path	string	/api/v1/chat/completion	Provider setting name.
method	string	POST	Http method for the assoicated proxu request.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
request	[]byte	{}	Request in bytes to providers.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
response	[]byte	{}	Response in bytes from providers.

Get events V2: GET /api/v2/events

Description

This endpoint retrieves a list of events based on provided filters.

Request

name	type	data type	example	description
userIds	optional	[]string	["user1", "user2"]	List of user IDs to filter events by specific users.
customIds	optional	[]string	["id1", "id2"]	List of custom identifiers for filtering events.
keyIds	optional	[]string	["key1", "key2"]	List of key IDs used to filter events.
tags	optional	[]string	["tag1", "tag2"]	List of tags associated with events for filtering.
start	required	int64	1609459200	Start timestamp for filtering events.
end	required	int64	1612137600	End timestamp for filtering events.
limit	optional	int	100	Maximum number of events to return.
offset	optional	int	0	Offset for pagination.
policyIds	optional	[]string	["policy1", "policy2"]	List of policy IDs to filter events related to specific policies.
actions	optional	[]string	["allowed"]	Values can include allowed, redacted, and blocked.
costOrder	optional	string	asc	Order by cost, specify asc or desc.
dateOrder	optional	string	desc	Order by date, specify 'asc' or 'desc'.
returnCount	optional	bool	true	Whether to return the count of events instead of event details.

Response

Field	type	example	description
events	[]Event	[]	List of events.
count	int	20	Total number of events returned.

Event

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
tags	int64	["YOUR_TAG"]	Tags of the key.
key_id	string	YOUR_KEY_ID	Key Id associated with the proxy request.
cost_in_usd	float64	0.0004	Cost incured by the proxy request.
model	string	gpt-4-1105-preview	Model used in the proxy request.
provider	string	openai	Provider for the proxy request.
status	int	200	Http status.
prompt_token_count	int	8	Prompt token count of the proxy request.
completion_token_count	int	16	Completion token counts of the proxy request.
latency_in_ms	int	160	Provider setting name.
path	string	/api/v1/chat/completion	Provider setting name.
method	string	POST	Http method for the assoicated proxu request.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
request	[]byte	{}	Request in bytes to providers.
custom_id	string	YOUR_CUSTOM_ID	Custom Id passed by the user in the headers of proxy requests.
response	[]byte	{}	Response in bytes from providers.

Create custom provider: POST /api/custom/providers

Description

This endpoint is creating custom providers.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
provider	required	string	bricks	Unique identifier associated with the route config.
route_configs	required	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Update custom provider: PATCH /api/custom/providers/:id

Description

This endpoint is updating a custom provider.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
route_configs	optional	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500, 404, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

Field	type	example	description
id	int64	1699933571	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Create a policy: POST /api/policies

Description

This endpoint is for creating a policy.

Request

Field	required	type	example	description
name	optional	string	Pivacy Policy #1	Name for the policy.
tags	required	[]string	["orgId-1"]	Associated tags.
config	optional	config	{"rules": { "address": "block" }}	PII detection rules.
regexConfig	optional	regexConfig	{"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]}	Regular expression rules.

Config

Field	required	type	example	description
rules	required	map[Rule]Action	{ "address": "block" }	PII entities mapped to their associated actions.

RegexConfig

Field	required	type	example	description
definition	required	string	[2-9]|[12]\d|3[0-6]	Regex definition.
action	required	Action	block	Action associated with the regex definition.

Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400, 500
title	string	request body reader error
type	string	/errors/policies
detail	string	something is wrong
instance	string	/api/policies

Response

Field	type	example	description
id	int64	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
tags	[]string	["org-111"]	Tags attached to policies.
config	config	{"rules": { "address": "block" }}	PII detection rules.
regexConfig	regexConfig	{"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]}	Regular expression rules.

Update a policy: PATCH /api/policies/:id

Description

This endpoint is for updating a policy.

Request

Field	required	type	example	description
name	optional	string	Pivacy Policy #1	Name for the policy.
tags	required	[]string	["orgId-1"]	Associated tags.
config	optional	config	{"rules": { "address": "block" }}	PII detection rules.
regexConfig	optional	regexConfig	{"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]}	Regular expression rules.

Config

Field	required	type	example	description
rules	required	map[Rule]Action	{ "address": "block" }	PII entities mapped to their associated actions.

RegexConfig

Field	required	type	example	description
definition	required	string	[2-9]|[12]\d|3[0-6]	Regex definition.
action	required	Action	block	Action associated with the regex definition.

Rule

| type | example | description | | enum| address | Possible values are address,age,all,aws_access_key,aws_secret_key,bank_account_number,bank_routing,ca_health_number,ca_social_insurance_number,credit_debit_cvv,credit_debit_expiry,credit_debit_number,date_time,driver_id,email,in_aadhaar,in_nrega,in_permanent_account_number,in_voter_number,international_bank_account_number,ip_address,license_plate,mac_address,name,passport_number,password,phone,pin,ssn,swift_code,uk_national_health_service_number,uk_national_insurance_number,uk_unique_taxpayer_reference_number,url,us_individual_tax_identification_number,username, and vehicle_identification_number. |

Action

| type | example | description | | enum| block | Possible values are block,allow_but_redact, and allow. |

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400, 500
title	string	request body reader error
type	string	/errors/policies
detail	string	something is wrong
instance	string	/api/policies

Response

Field	type	example	description
id	int64	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
tags	[]string	["org-111"]	Tags attached to policies.
config	config	{"rules": { "address": "block" }}	PII detection rules.
regexConfig	regexConfig	{"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]}	Regular expression rules.

Get policies by tags: GET /api/policies

Description

This endpoint is for retrieving policies by tags.

Query Parameters

| name | type | data type | description | |--------|------------|----------------|------------------------------------------------------|s | tags | required | []string | Tags attached to the policies. |

Response

[]Policy

Policy

Field	type	example	description
id	int64	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
tags	[]string	["org-111"]	Tags attached to policies.
config	config	{"rules": { "address": "block" }}	PII detection rules.
regexConfig	regexConfig	{"rules": [{"definition": "[2-9]|[12]\d|3[0-6]", "action": "block"}]}	Regular expression rules.

Get custom providers: GET /api/custom/providers

Description

This endpoint is for getting custom providers.

RouteConfig

Field	required	type	example	description
path	required	string	/chat/completion	Path associated with the custom provider route. It must be unique within the custom provider.
target_url	required	string	https://api.openai.com/v1/chat/completions	Proxy destination URL for the custom provider route.
model_location	required	string	model	JSON field for the model in the HTTP request.
request_prompt_location	required	string	messages.#.content	JSON field for the prompt request in the HTTP request.
response_completion_location	required	string	choices.#.message.content	JSON field for the completion content in the HTTP response.
stream_location	required	string	stream	JSON field for the stream boolean in the HTTP request.
stream_end_word	required	string	[DONE]	End word for the stream.
stream_response_completion_location	required	string	choices.#.delta.content	JSON field for the completion content in the streaming response.
stream_max_empty_messages	required	int	10	Number of max empty messages in stream.

Request

Field	required	type	example	description
route_configs	optional	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Route configurations for the custom provider.
authentication_param	optional	string	apikey	The authentication parameter required for.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/custom/providers

Response

[]Provider

Provider

Field	type	example	description
id	int64	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb	Unique identifier associated with the event.
created_at	int64	1699933571	Unix timestamp for creation time.
updated_at	int64	1699933571	Unix timestamp for update time.
provider	string	bricks	Unique identifier associated with the route config.
route_configs	[]RouteConfig	{{ "path": "/chat/completions", "target_url": "https://api.openai.com/v1/chat/completions" }}	Start timestamp for the requested timeseries data.
authentication_param	string	apikey	The authentication parameter required for.

Create routes: POST /api/routes

Description

This endpoint is for creating routes.

StepConfig

Field	required	type	example	description
provider	required	enum	azure	Provider for the step. Can only be either azure or openai.
model	required	string	gpt-3.5-turbo	Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries	optional	int	2	Number of retries.
params	optional	object	{ deploymentId: "ada-test",apiVersion: "2022-12-01" }	Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout	optional	string	5s	Timeout desired for each request. Default value is 5m.

CacheConfig

Field	required	type	example	description
enabled	required	bool	false	Boolean flag indicating whether caching is enabled.
ttl	optional	string	5s	TTL for the cache. Default value is 168h.

Request

Field	required	type	example	description
name	required	string	staging-openai-azure-completion-route	Name for the route.
path	required	string	/	Unique identifier for.
steps	required	[]StepConfig	apikey	The authentication parameter required for.
keyIds	required	[]string	[]	The authentication parameter required for.
cacheConfig	required	CacheConfig	[]	The authentication parameter required for.

Error Response

http code	content-type
500, 400	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes

Response

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Retrieve a route: GET /api/routes/:id

Description

This endpoint is for retrieving a route.

Parameters

name	type	data type	description
id	required	string	Unique identifier for the route.

Error Response

http code	content-type
500, 404	application/json

Field	type	example
status	int	404
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes/:id

Response

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Retrieve routes: GET /api/routes

Description

This endpoint is for retrieving routes.

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	404
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/routes/:id

CacheConfig

Field	required	type	example	description
enabled	required	bool	false	Boolean flag indicating whether caching is enabled.
ttl	optional	string	5s	TTL for the cache. Default value is 168h.

StepConfig

Field	required	type	example	description
provider	required	enum	azure	Provider for the step. Can only be either azure or openai.
model	required	string	gpt-3.5-turbo	Model that the step should call. Can only be chat completion or embedding models from OpenAI or Azure OpenAI.
retries	optional	int	2	Number of retries.
params	optional	object	{ deploymentId: "ada-test",apiVersion: "2022-12-01" }	Params required for maing API requests to desired modela and provider combo. Required if the provider is azure
timeout	optional	string	5s	Timeout desired for each request. Default value is 5m.

RouteConfig

Field	type	example	description
id	required	string	9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb
createdAt	required	string	1699933571
updatedAt	required	string	1699933571
name	required	string	staging-openai-azure-completion-route
path	required	string	/production/chat/completion
steps	required	[]StepConfig	[{"retries": 2, "provider": "openai", "params": {}, "model": "gpt-3.5-turbo", "timeout": "1s"}]
keyIds	required	[]string	["9e6e8b27-2ce0-4ef0-bdd7-1ed3916592eb"]
cacheConfig	required	CacheConfig	{ "enabled": false, "ttl": "5s" }

Response

[]RouteConfig

Get user ids: GET /api/reporting/users-ids

Description

This endpoint is for getting user ids given a key id.

Query Parameters

name	type	data type	description
keyId	required	string	Key Id

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	500
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/reporting/user-ids

Response

[]string

Get custom ids: GET /api/reporting/custom-ids

Description

This endpoint is for getting custom ids given a key id.

Query Parameters

name	type	data type	description
keyId	required	string	Key Id

Error Response

http code	content-type
500	application/json

Field	type	example
status	int	500
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/reporting/custom-ids

Response

[]string

Create a user: POST /api/users

Description

This endpoint is creating a user.

Query Parameters

name	type	data type	description
tag	optional	string	Identifier attached to a key configuration
tags	optional	[]string	Identifiers attached to a key configuration
provider	optional	string	Provider attached to a key provider configuration. Its value can only be openai.
keyIds	optional	[]string	Unique identifiers for keys.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
name	required	string	spike's developer key	Name of the API key.
tags	optional	[]string	["org-tag-12345"]	Identifiers associated with the key.
keyIds	optional	[]string	["98daa3ae-961d-4253-bf6a-322a32fdca3d"]	Setting ids associated with the key.
costLimitInUsd	optional	float64	5.5	Total spend limit of the API key.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	enum	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	optional	string	2d	time to live. Available units are [s, m, h].
allowedPaths	optional	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	optional	[]string	["gpt-4"]	List of models that can be accessed by the user.
userId	optional	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Client defined user ID.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/users

Response

Field	type	example	description
id	string	550e8400-e29b-41d4-a716-446655440000	Id of the user object.
name	string	spike's developer key	Name of the user.
createdAt	int64	1257894000	UserKey configuration creation time in unix.
updatedAt	int64	1257894000	User configuration update time in unix.
tags	[]string	["org-tag-12345"]	Identifiers associated with the user.
keyIds	[]string	["550e8400-e29b-41d4-a716-446655440000"]	Array of key IDs.
revoked	boolean	true	Indicator for whether the user is revoked.
revokedReason	string	The key has expired	Reason for why the user is revoked.
costLimitInUsd	float64	5.5	Total spend limit of the user.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	[]string	["gpt-4"]	List of models that can be accessed by the user.
userId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Client defined user id.

Update a user via id: PATCH /api/users/:id

Description

This endpoint is updating a user via id.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
name	required	string	spike	Name of the user.
keyIds	optional	[]string	["98daa3ae-961d-4253-bf6a-322a32fdca3d"]	Keys ids associated with the user.
costLimitInUsd	optional	float64	5.5	Total spend limit of the user.
revoked	optional	boolean	true	Indicator for whether the user is revoked.
revokedReason	optional	string	expired	Reason for why the user is revoked.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	Rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	enum	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	optional	string	24h	time to live. Available units are [s, m, h].
allowedPaths	optional	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	optional	[]string	["gpt-4"]	List of models that can be accessed by the user.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/users

Response

Field	type	example	description
id	string	550e8400-e29b-41d4-a716-446655440000	Id of the user object.
name	string	spike	Name of the user.
createdAt	int64	1257894000	User configuration creation time in unix.
updatedAt	int64	1257894000	User configuration update time in unix.
tags	[]string	["org-tag-12345"]	Identifiers associated with the user.
keyIds	[]string	["550e8400-e29b-41d4-a716-446655440000"]	Array of key IDs.
revoked	boolean	true	Indicator for whether the user is revoked.
revokedReason	string	expired	Reason for why the user is revoked.
costLimitInUsd	float64	5.5	Total spend limit of the user.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	[]string	["gpt-4"]	List of models that can be accessed by the user.
userId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Client defined user id.

Update a user via tags and user id: PATCH /api/users

Description

This endpoint is updating a user via tags and id.

Query Parameters

name	type	data type	description
userId	optional	string	User identifier.
tags	optional	[]string	Array of identifiers associated with a user.

Request

PathConfig

Field	required	type	example	description
path	required	string	/api/providers/openai/v1/chat/completion	Allowed path
method	required	string	POST	HTTP Method

Field	required	type	example	description
name	required	string	spike	Name of the user.
keyIds	optional	[]string	["98daa3ae-961d-4253-bf6a-322a32fdca3d"]	Keys ids associated with the user.
costLimitInUsd	optional	float64	5.5	Total spend limit of the user.
revoked	optional	boolean	true	Indicator for whether the user is revoked.
revokedReason	optional	string	expired	Reason for why the user is revoked.
costLimitInUsdOverTime	optional	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	optional	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	optional	int	2	Rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	optional	enum	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	optional	string	24h	time to live. Available units are [s, m, h].
allowedPaths	optional	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	optional	[]string	["gpt-4"]	List of models that can be accessed by the user.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/users

Response

Field	type	example	description
id	string	550e8400-e29b-41d4-a716-446655440000	Id of the user object.
name	string	spike	Name of the user.
createdAt	int64	1257894000	User configuration creation time in unix.
updatedAt	int64	1257894000	User configuration update time in unix.
tags	[]string	["org-tag-12345"]	Identifiers associated with the user.
keyIds	[]string	["550e8400-e29b-41d4-a716-446655440000"]	Array of key IDs.
revoked	boolean	true	Indicator for whether the user is revoked.
revokedReason	string	expired	Reason for why the user is revoked.
costLimitInUsd	float64	5.5	Total spend limit of the user.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	[]string	["gpt-4"]	List of models that can be accessed by the user.
userId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Client defined user id.

Get users: GET /api/users

Description

This endpoint is retrieving users.

Query Parameters

name	type	data type	description
userIds	optional	[]string	Array of user identifiers.
tags	optional	[]string	Array of identifiers associated with a user.
offset	optional	int	Pagination offset.
limit	optional	int	Pagination limit.
keyIds	optional	[]string	Array of key IDs associated with a user.

Error Response

http code	content-type
400, 500	application/json

Field	type	example
status	int	400
title	string	request body reader error
type	string	/errors/request-body-read
detail	string	something is wrong
instance	string	/api/users

Response

Response Body
[]User

User

Field	type	example	description
id	string	550e8400-e29b-41d4-a716-446655440000	Id of the user object.
name	string	spike	Name of the user.
createdAt	int64	1257894000	User configuration creation time in unix.
updatedAt	int64	1257894000	User configuration update time in unix.
tags	[]string	["org-tag-12345"]	Identifiers associated with the user.
keyIds	[]string	["550e8400-e29b-41d4-a716-446655440000"]	Array of key IDs.
revoked	boolean	true	Indicator for whether the user is revoked.
revokedReason	string	expired	Reason for why the user is revoked.
costLimitInUsd	float64	5.5	Total spend limit of the user.
costLimitInUsdOverTime	float64	2	Total spend within period of time. This field is required if costLimitInUsdUnit is specified.
costLimitInUsdUnit	enum	d	Time unit for costLimitInUsdOverTime. Possible values are [m, h, d, mo].
rateLimitOverTime	int	2	rate limit over period of time. This field is required if rateLimitUnit is specified.
rateLimitUnit	string	m	Time unit for rateLimitOverTime. Possible values are [h, m, s, d]
ttl	string	24h	time to live. Available units are [s, m, h]
allowedPaths	[]PathConfig	[{ "path": "/api/providers/openai/v1/chat/completion", "method": "POST"}]	List of paths that can be accessed by the user.
allowedModels	[]string	["gpt-4"]	List of models that can be accessed by the user.
userId	string	98daa3ae-961d-4253-bf6a-322a32fdca3d	Client defined user id.

OpenAI Proxy

The OpenAI proxy runs on Port 8002.

Headers

name	type	data type	description
x-custom-event-id	optional	string	Custom Id that can be used to retrieve an event associated with each proxy request.

Chat Completion

Call OpenAI chat completions: POST /api/providers/openai/v1/chat/completions

Description

This endpoint is set up for proxying OpenAI chat completion requests. Documentation for this endpoint can be found here.

Embeddings

Call OpenAI embeddings: POST /api/providers/openai/v1/embeddings

Description

This endpoint is set up for proxying OpenAI embedding requests. Documentation for this endpoint can be found here.

Moderations

Call OpenAI moderations: POST /api/providers/openai/v1/moderations

Description

This endpoint is set up for proxying OpenAI moderation requests. Documentation for this endpoint can be found here.

Models

Get OpenAI models: GET /api/providers/openai/v1/models

Description

This endpoint is set up for retrieving OpenAI models. Documentation for this endpoint can be found here.

Retrieve an OpenAI model: GET /api/providers/openai/v1/models/:model

Description

This endpoint is set up for retrieving an OpenAI model. Documentation for this endpoint can be found here.

Files

List files: GET /api/providers/openai/v1/files

Description

This endpoint is set up for list OpenAI files. Documentation for this endpoint can be found here.

Upload a file: POST /api/providers/openai/v1/files

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Delete a file: POST /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for creating an OpenAI file. Documentation for this endpoint can be found here.

Retrieve a file: GET /api/providers/openai/v1/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI file. Documentation for this endpoint can be found here.

Retrieve file content: GET /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for retrieving an OpenAI file content. Documentation for this endpoint can be found here.

Images

Generate images: POST /api/providers/openai/v1/images/generations

Description

This endpoint is set up for generating OpenAI images. Documentation for this endpoint can be found here.

Edit images: POST /api/providers/openai/v1/images/edits

Description

This endpoint is set up for editting OpenAI generated images. Documentation for this endpoint can be found here.

Create image variations: POST /api/providers/openai/v1/images/variations

Description

This endpoint is set up for creating OpenAI image variations. Documentation for this endpoint can be found here.

Voices

Create speech: POST /api/providers/openai/v1/audio/speech

Description

This endpoint is set up for creating speeches. Documentation for this endpoint can be found here.

Create transcriptions: POST /api/providers/openai/v1/audio/transcriptions

Description

This endpoint is set up for editting generated images. Documentation for this endpoint can be found here.

Create translations: POST /api/providers/openai/v1/audios/translations

Description

This endpoint is set up for creating translations. Documentation for this endpoint can be found here.

Assistants

Create assistant: POST /api/providers/openai/v1/assistants

Description

This endpoint is set up for creating an OpenAI assistant. Documentation for this endpoint can be found here.

Retrieve assistant: GET /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for retrieving an OpenAI assistant. Documentation for this endpoint can be found here.

Modify assistant: POST /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for modifying an OpenAI assistant. Documentation for this endpoint can be found here.

Delete assistant: DELETE /api/providers/openai/v1/assistants/:assistant_id

Description

This endpoint is set up for deleting an OpenAI assistant. Documentation for this endpoint can be found here.

List assistants: GET /api/providers/openai/v1/assistants

Description

This endpoint is set up for listing OpenAI assistants. Documentation for this endpoint can be found here.

Create assistant file: POST /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for creating an OpenAI assistant file. Documentation for this endpoint can be found here.

Retrieve assistant file: GET /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI assistant file. Documentation for this endpoint can be found here.

Delete assistant file: DELETE /api/providers/openai/v1/assistants/:assistant_id/files/:file_id

Description

This endpoint is set up for deleting an OpenAI assistant file. Documentation for this endpoint can be found here.

List assistant files: GET /api/providers/openai/v1/assistants/:assistant_id/files

Description

This endpoint is set up for retrieving OpenAI assistant files. Documentation for this endpoint can be found here.

Create thread: POST /api/providers/openai/v1/threads

Description

This endpoint is set up for creating an OpenAI thread. Documentation for this endpoint can be found here.

Retrieve thread: GET /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for retrieving an OpenAI thread. Documentation for this endpoint can be found here.

Modify thread: POST /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for modifying an OpenAI thread. Documentation for this endpoint can be found here.

Delete thread: DELETE /api/providers/openai/v1/threads/:thread_id

Description

This endpoint is set up for deleting an OpenAI thread. Documentation for this endpoint can be found here.

Create message: POST /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for creating an OpenAI message. Documentation for this endpoint can be found here.

Retrieve message: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id

Description

This endpoint is set up for retrieving an OpenAI message. Documentation for this endpoint can be found here.

Modify message: POST /api/providers/openai/v1/files/:file_id/content

Description

This endpoint is set up for modifying an OpenAI message. Documentation for this endpoint can be found here.

List messages: GET /api/providers/openai/v1/threads/:thread_id/messages

Description

This endpoint is set up for listing OpenAI messages. Documentation for this endpoint can be found here.

Retrieve message file: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files/:file_id

Description

This endpoint is set up for retrieving an OpenAI message file. Documentation for this endpoint can be found here.

List message files: GET /api/providers/openai/v1/threads/:thread_id/messages/:message_id/files

Description

This endpoint is set up for retrieving OpenAI message files. Documentation for this endpoint can be found here.

Create run: POST /api/providers/openai/v1/threads/:thread_id/runs

Description

This endpoint is set up for creating an OpenAI run. Documentation for this endpoint can be found here.

Retrieve run: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for retrieving an OpenAI run. Documentation for this endpoint can be found here.

Modify run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id

Description

This endpoint is set up for modifying an OpenAI run. Documentation for this endpoint can be found here.

List runs: GET /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for retrieving OpenAI runs. Documentation for this endpoint can be found here.

Submit tool outputs to run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for submitting tool outputs to an OpenAI run. Documentation for this endpoint can be found here.

Cancel a run: POST /api/providers/openai/v1/threads/:thread_id/runs/:run_id/cancel

Description

This endpoint is set up for cancellling an OpenAI run. Documentation for this endpoint can be found here.

Create thread and run: POST /api/providers/openai/v1/threads/runs

Description

This endpoint is set up for creating an OpenAI thread and run. Documentation for this endpoint can be found here.

Retrieve run step: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps/:step_id

Description

This endpoint is set up for retrieving an OpenAI run step. Documentation for this endpoint can be found here.

List run steps: GET /api/providers/openai/v1/threads/:thread_id/runs/:run_id/steps

Description

This endpoint is set up for listing OpenAI run steps. Documentation for this endpoint can be found here.

Azure OpenAI Proxy

The custom provider proxy runs on Port 8002.

Create Azure OpenAI chat completion: POST /api/providers/azure/openai/deployments/:deployment_id/chat/completions?api-version={API_VERSION}

Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Create Azure OpenAI embeddings: POST /api/providers/azure/openai/deployments/:deployment_id/embeddings?api-version={API_VERSION}

Description

This endpoint is set up for proxying Azure OpenAI completion requests. Documentation for this endpoint can be found here.

Anthropic Proxy

The custom provider proxy runs on Port 8002.

Create Anthropic completion: POST /api/providers/anthropic/v1/complete

Description

This endpoint is set up for proxying Anthropic completion requests. Documentation for this endpoint can be found here.

Create Anthropic messages: POST /api/providers/anthropic/v1/messages

Description

This endpoint is set up for proxying Anthropic messages requests. Documentation for this endpoint can be found here.

vllm Provider Proxy

The vllm provider proxy runs on Port 8002.

Create chat completions: POST /api/providers/vllm/v1/chat/completions

Description

This endpoint is set up for proxying vllm chat completions requests. Documentation for this endpoint can be found here.

Create completions: POST /api/providers/vllm/v1/completions

Description

This endpoint is set up for proxying vllm completions requests. Documentation for this endpoint can be found here.

Deepinfra Provider Proxy

The deepinfra provider proxy runs on Port 8002.

Create chat completions: POST /api/providers/deepinfra/v1/chat/completions

Description

This endpoint is set up for proxying deepinfra chat completions requests. Documentation for this endpoint can be found here.

Create completions: POST /api/providers/deepinfra/v1/completions

Description

This endpoint is set up for proxying deepinfra completions requests. Documentation for this endpoint can be found here.

Create embeddings: POST /api/providers/deepinfra/v1/embeddings

Description

This endpoint is set up for proxying deepinfra embeddings requests. Documentation for this endpoint can be found here.

Custom Provider Proxy

The custom provider proxy runs on Port 8002.

Call custom providers: POST /api/custom/providers/:provider/*

Description

First you need to use create custom providers endpoint to create custom providers. Then create corresponding provider setting for the newly created custom provider. Afterward, you can start creating keys associated with the custom provider, and use the keys to access this endpoint by placing the created key in Authorization: Bearer YOUR_BRICKSLLM_KEY as part of your HTTP request headers.

Route Proxy

The custom provider proxy runs on Port 8002.

Call a route: POST /api/route/*

Description

Route helps you interpolate different models (embeddings or chat completion models) and providers (OpenAI or Azure OpenAI) to gurantee API responses.

First you need to use create route endpoint to create routes. If the route uses both Azure and OpenAI, you need to create API keys with corresponding provider settings as well. If the route is for chat completion, just call the route using the OpenAI chat completion format. On the other hand, if the route is for embeddings, just call the route using the embeddings format.