README
¶
eve-marketwatch
Self contained microservice to collect market and contract data and stream all additions, changes, and deletions over a websocket. On connect a dump of the current state will be sent (it is quite big) followed by a stream of all changes.
This can be used to keep a database synchronized with the current market state, try to estimate completed orders for history, track players persistently making frequent changes (cough bots cough), or to find high value items sold to try to gank the player later. The possibilities are endless!
The microservice will spawn one goroutine (think lightweight thread) per market and collect the next available set of data when the cache time expires. Pages are also concurrently pulled with a concurrency limit of 100 requests in flight to keep everything to one https connection.
dockerized
The docker containers are from scratch and do not have ca-certs available, provide your systems ca-certs or an alternative location.
docker run antihax/eve-marketwatch -p 3005:3005 -v /etc/ssl/certs:/etc/ssl/certs
compilation
- Build the cmd directory.
environment
You can optionally pass an SSO configuration and a refresh_token from CCP to also gather market information from public structures. This requires the esi-markets.structure_markets.v1 scope. You can register an application to receive the clientID and secret at CCP's Third Party Applications site.
You will need to use another tool to obtain a refresh_token using the clientID and secret to provide this tool.
| Variable | Description |
|---|---|
| ESI_CLIENTID_TOKENSTORE | SSO ClientID |
| ESI_SECRET_TOKENSTORE | SSO Secret |
| ESI_REFRESHKEY | a refresh_token from the ClientID and Secret above |
Note: turning on structures will cause an initial performance hit as the service discovers which structures actually have a market. The consumer will spew errors and hit the error limit, but after an hour, this should settle and then operate smoothly.
operation
Subscription parameters can be sent in the websocket URL to determine which channel to subscribe to. The following will subscribe to both market and contract streams.
Connect to the websocket on port 3005 ws://address:3005/?market=1&contract=1 and receive a stream of JSON data of market and contract changes. On initial connect, you will receive a dump of the current market state.
Recommendation is to read messages asap and put them into queues so as not to hit timeout states on the websocket.
The :3000 port has prometheus stats and golang pprof information. This port should not be exposed, please protect it.
data received
Data will be encapsulated in a json frame.
{"action": "actionstring", "payload": { json payload }}
Payloads are as follows
addition
ESI formatted market orders
change and deletion
Changes and deletions will be an array of the following types. Deletions will also have a volume_remain of 0.
int64 `json:"order_id"`
int64 `json:"location_id"`
int32 `json:"type_id"`
int32 `json:"volume_change,omitempty"`
int32 `json:"volume_remain,omitempty"`
float64 `json:"price"`
int32 `json:"duration,omitempty"`
bool `json:"is_buy_order,omitempty"`
time.Time `json:"issued,omitempty"`
time.Time `json:"time_changed"`
contractAddition
Wrapped ESI formatted
type FullContract struct {
Contract esi.GetContractsPublicRegionId200Ok `json:"contract"`
Items []esi.GetContractsPublicItemsContractId200Ok `json:"items,omitempty"`
Bids []esi.GetContractsPublicBidsContractId200Ok `json:"bids,omitempty"`
}
contractChange and contractDeletion
type ContractChange struct {
ContractId int32 `json:"contract_id"`
LocationId int64 `json:"location_id"`
Expired bool `json:"expired,omitempty"`
DateExpired time.Time `json:"date_expired,omitempty"`
Bids []esi.GetContractsPublicBidsContractId200Ok `json:"bids,omitempty"`
Price float64 `json:"price,omitempty"`
Type_ string `json:"type,omitempty"`
TimeChanged time.Time `json:"time_changed,omitempty"`
}
Directories