rtb

package module
Version: v0.0.0-...-d28c1d8 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 29, 2016 License: MIT Imports: 3 Imported by: 1

README

rtb Real Time Bidding Interface

Proof of concept real time bidding library. This library can be tested using the p.o.c server implementation at https://github.com/evandigby/rtbhost.

About This Project -- The "Why?"

This project was completed over a couple days (~10 hours of work) as part of an interview process. This was my first crack at RTB logic, as well as my first attempt at programming in Go.

Having successfully gotten the job and after working with a production RTB in Go for a year, I would do most of it entirely differenty if I started from scratch. There are also a few places I would tweak to make it look more like proper Go.

If anyone takes notice, I would be happy to elaborate on things I would do differently, within the context of this project.

Implementation Notes

Target Matching
  • The redis server is used to match campaigns to targets by storing sets that point in "both directions": Sets of all targets a campaign requires, and sets of all campaigns for a specific target.
  • When a request comes in, it compiles a list of targets for that request, and then does a union on the sets associated with each target. This produces a list of campaign Ids that contain ANY of the targets (an OR relationship).
  • There is still a to-do item to implement the AND relationship for targeting, allowing campaigns to require all of their targets, or combinations of their targets, not just any of the targets.
Bid Decisions
  • The target matching sets in redis are stored as sorted sets whose score is their bid CPM.
  • When the set union is returned from target matching, it's already sorted from highest bid CPM to lowest.
  • In most cases, the bidder will pick the first one. If the first one does not have available budget, the bidder will move down the list.
  • Pacing can be implemented using the "BidPacer" interface. Every time a bidder matches a campaign, it will first ask that campaign if it "can bid" through the pacer.
  • There is a sample "time segmented" pacer that will divide the remaining daily budget over the remaining time in the day, and break that into chunks of a specified time segment length. It will not allow any campaign to bid that has exceeded its number of bids for that time segment. This essentially granulates remaining daily budget into smaller chunks.
Bid Responses
  • The system will respond with a 200 HTTP Status Code for a bid, and a 204 HTTP Status Code for a no-bid.
  • Right now bid responses are largely stubbed out. These of course need to be completed for any production implementation.
  • It's also worth implementing no-bid reasons on 204.
Remaining Daily Spending Budget
  • The redis server is used as a quick way to cache remaining daily budgets to allow multiple instances of a host using this library to coordinate over the network.
Transaction logging
  • The redis server is NOT designed to act as a reliable transaction log.
  • Any production implementation of this real time bidder should implement a bomb proof transaction log to maintain accurate accounting records.
  • It would also be prudent for another system to consume the transaction log, either directly from the bidder, or through the transaction logger, and periodically audit and update the values stored in the redis instance
Error handling
  • The current implementation should be stable within the context of what is implemented, but will panic at any errors that arise from unforseen circumstances. This should be fleshed out to provide more detailed error logging and reporting for better debugging in a production environment.
Testing
  • You will find tests in the root, inmemory, and redis folders.
  • The root tests are simple unit tests for core functions, such as money conversions.
  • inmemory tests are pure unit tests of the bidder using mocks (see rtb/mocks) to stub out any data access
  • redis tests are integration tests of the redis components.

System Requirements

The rtbhost requires the following:

  • Go
  • Access to a redis instance.
  • Access to an amqp server (if you want to utilize logging).

Usage

Regarding usage:

  • Build rtbhost in the developer home directory:

    go build github.com/evandigby/rtbhost

  • Reset the data store in redis by using the shell script, found in the developer home directory:

    ./clear

  • Run the host on the default port by executing

    ./rtbhost

  • See the many usage options by executing

    ./rtbhost --help

  • Launch two processes running at ports 8000 and 8001 using the shell script “launch”:

    ./launch

  • Any parameters passed into launch will be passed onto both rtbhost instances.

  • By default, the rtbhost will output a status update on both campaigns every 5 seconds. The shell script will wait 2 seconds before starting the second process to ensure the logging doesn’t overlap.

  • You can use the “logverbose” option to force rtbhost to log every request (to stdout by default):

    ./launch --logverbose

  • There are many other command line options. Please feel free to explore them!

  • You can run tests test using:

    go test github.com/evandigby/rtb

    go test github.com/evandigby/rtb/inmemory

    go test github.com/evandigby/rtb/redis

Documentation

Overview

Package rtb defines the types for implementing a real time bidding application.

Index

Constants

View Source
const (
	// Placement defines a target based on the app or site name
	Placement TargetType = 1
	// CreativeSize defines a target based on the size of the creative the impression is looking for
	CreativeSize = 2
	// Country defines a target based on the country of the user the request came form
	Country = 3
	// OS defines the OS of the device the request came form
	OS = 4
)
View Source
const CPMConversionFactor = int64(1000)
View Source
const MicroCentsConversionFactorFloat = float64(MicroCentsConversionFactorInt64)
View Source
const MicroCentsConversionFactorInt64 = int64(1000000)

Are micro cents small enough to remove rounding error?

Variables

This section is empty.

Functions

func CpmToMicroCents

func CpmToMicroCents(cpm float64) int64

CpmToMicroCents converts from CPM (measured in dollars) to micro cents

func DollarsToMicroCents

func DollarsToMicroCents(dollars float64) int64

DollarsToMicroCents converts from dollars to micro cents

func MicroCentsPerImpression

func MicroCentsPerImpression(cpmInMicroCents int64) int64

MicroCentsPerImpression calculates the number of micro cents per impression based on a CPM value

func MicroCentsToCpm

func MicroCentsToCpm(microCents int64) float64

MicroCentsToCpm converts from micro cents to CPM (measured in dollars)

func MicroCentsToDollars

func MicroCentsToDollars(microCents int64) float64

MicroCentsToDollars converts from micro cents to dollars

func MicroCentsToDollarsRounded

func MicroCentsToDollarsRounded(microCents int64, decimals int) float64

func NewTransactionError

func NewTransactionError(msg string, valueValid bool) error

Types

type App

type App struct {
	Bundle    string     `json:"bundle,omitempty"`
	Cat       []string   `json:"cat,omitempty"`
	ID        string     `json:"id,omitempty"`
	Name      string     `json:"name,omitempty"`
	Publisher *Publisher `json:"publisher,omitempty"`
	Storeurl  string     `json:"storeurl,omitempty"`
	Ver       string     `json:"ver,omitempty"`
}

type Banker

type Banker interface {
	// DebitAccount subtracts an amount from an account
	// Returns the remaining remainingDailyBudgetInMicroCents after the transaction, and an error if the transaction was unsuccessful
	DebitAccount(account int64, amount int64, dailyBudget int64, dailyBudgetExpiration time.Time) (remainingDailyBudgetInMicroCentsInMicroCents int64, err error)
	// Returns the remainingDailyBudgetInMicroCents for the account, or zero for a non-existant account
	RemainingDailyBudgetInMicroCents(account int64) int64
	// Deletes an account
	DeleteAccount(account int64)
	// Sets the account's remainingDailyBudgetInMicroCents to a specific amount, expiring at a certain time
	SetRemainingDailyBudgetInMicroCents(account int64, amount int64, dailyBudgetExpiration time.Time)
}

Banker defines an interface for a campaign provider to track daily campaign budgets

Implementations of this interface should be designed with speed in mind.

Although this should be close to 100% accurate, users of this interface should not depended on implementations for a true transaction log and accounting purposes.

type Banner struct {
	Api   []float64   `json:"api,omitempty"`
	Battr []float64   `json:"battr,omitempty"`
	Btype []float64   `json:"btype,omitempty"`
	Ext   interface{} `json:"ext,omitempty"`
	H     int32       `json:"h,omitempty"`
	Pos   int32       `json:"pos,omitempty"`
	W     int32       `json:"w,omitempty"`
}

type Bid

type Bid struct {
	Adid    string                 `json:"adid,omitempty"`
	Adm     string                 `json:"adm,omitempty"`
	Adomain []string               `json:"adomain,omitempty"`
	Attr    []float64              `json:"attr,omitempty"`
	Cid     string                 `json:"cid,omitempty"`
	Crid    string                 `json:"crid,omitempty"`
	Crtype  string                 `json:"crtype,omitempty"`
	Ext     map[string]interface{} `json:"ext,omitempty"`
	ID      string                 `json:"id,omitempty"`
	Impid   string                 `json:"impid,omitempty"`
	Iurl    string                 `json:"iurl,omitempty"`
	Nurl    string                 `json:"nur,omitempty"`
	Price   float64                `json:"price,omitempty"`
}

type BidLogConsumer

type BidLogConsumer interface {
	LogChannel() chan *BidLogItem
}

BidLogConsumer defines a type that can consume bid log requests

type BidLogItem

type BidLogItem struct {
	Domain                            string           `json:"d,omitempty"`
	BidRequest                        *BidRequest      `json:"rq,omitempty"`
	BidResponse                       *BidResponse     `json:"rp,omitempty"`
	RemainingDailyBudgetsInMicroCents map[string]int64 `json:"b,omitempty"`
	StartTimestampInNanoseconds       int64            `json:"sts,omitempty"`
	EndTimestampInNanoseconds         int64            `json:"ets,omitempty"`
}

BidLogItem defines the structure of a bidder log

type BidLogProducer

type BidLogProducer interface {
	// Safe inside goroutine
	LogItem(logItem *BidLogItem)
}

BidLogProducer defines a type that can log bid requests

type BidLogger

type BidLogger interface {
	BidLogProducer
	BidLogConsumer
}

BidLogger defines a type that can both produce and consume bid log requests

type BidRequest

type BidRequest struct {
	App    *App        `json:"app,omitempty"`
	At     float64     `json:"at,omitempty"`
	Badv   []string    `json:"badv,omitempty"`
	Bcat   []string    `json:"bcat,omitempty"`
	Device *Device     `json:"device,omitempty"`
	Ext    interface{} `json:"ext,omitempty"`
	ID     string      `json:"id,omitempty"`
	Imp    []Imp       `json:"imp,omitempty"`
	Site   *Site       `json:"site,omitempty"`
	User   *User       `json:"user,omitempty"`
}

BidRequest defines a bid request object and how to parse it from JSON

func (*BidRequest) Targeting

func (r *BidRequest) Targeting() []Target

Targeting creates a list of targets from a bid request

type BidResponse

type BidResponse struct {
	Bidid   string    `json:"bidid,omitempty"`
	Cur     string    `json:"cur,omitempty"`
	ID      string    `json:"id,omitempty"`
	Seatbid []Seatbid `json:"seatbid,omitempty"`
}

BidRequest defines a bid response object and how to parse it from JSON

type Bidder

type Bidder interface {
	Bid() (response *BidResponse, campaignRemainingDailyBudget map[string]int64, err error)
}

Bidder defines a type which can bid on bid requests

type Campaign

type Campaign interface {
	// Id defines the unique identifier of this campaign
	Id() int64
	// BidCpmInMicroCents defines the CPM this campaign is willing to bid for a matching impression
	BidCpmInMicroCents() int64
	// DailyBudgetInMicroCents defines the daily budget for this campaign
	DailyBudgetInMicroCents() int64
	// Targets define the type of request this campaign is targeting
	Targets() *map[TargetType]string
}

Campaign defines the structure of a campaign and what requests it is targeting

type CampaignProvider

type CampaignProvider interface {
	// ReadByTargeting returns any campaigns that have available funds and meet the target criteria passed in.
	// Campaigns are returned in order from highest cpm to lowest
	// Available funds are is measured at the time of the query, and may be spent by the time DebitCampaign is called.
	ReadByTargeting(bidFloorInMicroCents int64, targets []Target) []Campaign

	// DebitCampaign subtracts an amount from the daily budget of the campaign
	// Returns the remaining daily budget after the transaction, and an error if the transaction was unsuccessful
	DebitCampaign(campaignId int64, amountInMicroCents int64, dailyBudgetExpiration time.Time) (remainingDailyBudgetInMicroCents int64, err error)

	// Creates a new persisted campaign
	CreateCampaign(campaignId int64, bidCpmInMicroCents int64, dailyBudgetInMicroCents int64, targets []Target) Campaign

	// Reads a persisted campaign
	ReadCampaign(campaignId int64) Campaign

	// Lists campaigns
	ListCampaigns() []int64
}

CampaignProvider defines a way to access a campaign data store

type Deal

type Deal struct {
	At       float64 `json:"at,omitempty"`
	Bidfloor float64 `json:"bidfloor,omitempty"`
	ID       string  `json:"id,omitempty"`
}

type Device

type Device struct {
	Carrier        string      `json:"carrier,omitempty"`
	Connectiontype float64     `json:"connectiontype,omitempty"`
	Devicetype     float64     `json:"devicetype,omitempty"`
	Dnt            float64     `json:"dnt,omitempty"`
	Dpidmd5        string      `json:"dpidmd5,omitempty"`
	Dpidsha1       string      `json:"dpidsha1,omitempty"`
	Ext            interface{} `json:"ext,omitempty"`
	Geo            *Geo        `json:"geo,omitempty"`
	Ip             string      `json:"ip,omitempty"`
	Js             float64     `json:"js,omitempty"`
	Language       string      `json:"language,omitempty"`
	Make           string      `json:"make,omitempty"`
	Model          string      `json:"model,omitempty"`
	Os             string      `json:"os,omitempty"`
	Osv            string      `json:"osv,omitempty"`
	Ua             string      `json:"ua,omitempty"`
}

type Geo

type Geo struct {
	City    string  `json:"city,omitempty"`
	Country string  `json:"country,omitempty"`
	Lat     float64 `json:"lat,omitempty"`
	Lon     float64 `json:"lon,omitempty"`
	Metro   string  `json:"metro,omitempty"`
	Region  string  `json:"region,omitempty"`
	Zip     string  `json:"zip,omitempty"`
}

type Imp

type Imp struct {
	Banner            *Banner `json:"banner,omitempty"`
	Bidfloor          float64 `json:"bidfloor,omitempty"`
	Displaymanager    string  `json:"displaymanager,omitempty"`
	Displaymanagerver string  `json:"displaymanagerver,omitempty"`
	ID                string  `json:"id,omitempty"`
	Instl             float64 `json:"instl,omitempty"`
	Tagid             string  `json:"tagid,omitempty"`
}

func (*Imp) Targeting

func (i *Imp) Targeting() []Target

Targeting creates a list of targets from a specific impression

type Mraid

type Mraid struct {
	Functions []string `json:"functions,omitempty"`
	Version   string   `json:"version,omitempty"`
}

type Pacer

type Pacer interface {
	CanBid(campaign Campaign) bool
}

Defines an object that can set the pace of campaign spending

type Pmp

type Pmp struct {
	Deals   []Deal  `json:"deals,omitempty"`
	Private float64 `json:"private,omitempty"`
}

type Publisher

type Publisher struct {
	ID   string `json:"id,omitempty"`
	Name string `json:"name,omitempty"`
}

type Seatbid

type Seatbid struct {
	Bid  []Bid  `json:"bid,omitempty"`
	Seat string `json:"seat,omitempty"`
}

type Segment

type Segment struct {
	ID    string `json:"id,omitempty"`
	Value string `json:"value,omitempty"`
}

type Site

type Site struct {
	Cat       []string   `json:"cat,omitempty"`
	Domain    string     `json:"domain,omitempty"`
	ID        string     `json:"id,omitempty"`
	Name      string     `json:"name,omitempty"`
	Publisher *Publisher `json:"publisher,omitempty"`
}

type Target

type Target struct {
	Type  TargetType
	Value string
}

Target defines the type and value of a specific targeting

type TargetType

type TargetType int

TargetType defines the type of targeting

type TimeSegmentedPacer

type TimeSegmentedPacer interface {
	Pacer
	Segment() time.Duration
}

Defines a specific type of pacer that will pace bids of a time period

type Transaction

type Transaction struct {
	CampaignId             int64
	BidResponseId          string
	AmountInMicroCents     int64
	TimestampInNanoSeconds int64
	Ext                    interface{} // Extended data
}

A transaction is kept light intentionally, as we don't want to "bog down" any accounting system

type TransactionError

type TransactionError struct {
	// contains filtered or unexported fields
}

Defines a transaction error used by the banker

func (*TransactionError) Error

func (e *TransactionError) Error() string

type TransactionLogger

type TransactionLogger interface {
	// A way to determine if someone is listening to this log. This is important, as we should not start bidding if nobody is going to consume the transaction
	ConsumerListening() (bool, error)
	// A return of nil (no error) means that a transaction was logged and acknoleged
	// Any error returned should be treated as though the transaction was not logged.
	LogTransaction(transaction *Transaction) error
}

The transaction logger interface is designed to assist with implementing a bomb proof transaction logger

type User

type User struct {
	Data     []UserData `json:"data,omitempty"`
	Gender   string     `json:"gender,omitempty"`
	Keywords string     `json:"keywords,omitempty"`
	Yob      string     `json:"yob,omitempty"`
}

type UserData

type UserData struct {
	ID      string    `json:"id,omitempty"`
	Name    string    `json:"name,omitempty"`
	Segment []Segment `json:"segment,omitempty"`
}

type Video

type Video struct {
	Linearity   float64  `json:"linearity,omitempty"`
	Maxduration float64  `json:"maxduration,omitempty"`
	Minduration float64  `json:"minduration,omitempty"`
	Type        []string `json:"type,omitempty"`
}

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL