rlearning

README

Go-ML rlearning

Package rlearning is a ml reinforcement learning package for Go.

Example Usage

Below contains example usage of the rlearning package on the frozen lake environment. This example can be found in ../tests/frozenlake_test.go.

Q-Learning on Frozen Lake

    // Initialize frozen lake env with 4x4 map
	env := frozenlake.InitFrozenLake(4, 4, 1.25, false)
	_, _, _, err := env.Step([]float64{})
	if err == nil {
		t.Fatalf(`Failed to get error from step with no action\n`)
	}

	// Initialize parameters
	max_episodes := 1000
	max_actions := 99
	learning_rate := 0.83
	discount := 0.95
	exploration_rate := 1.0
	exploration_decay := 1.0 / float64(max_episodes)

	// Initialize agent and set policy
	agent := rlearning.NewQAgent(&env, max_episodes, max_actions, learning_rate, discount)
	agent.SetPolicy("DecayExploration", []float64{exploration_rate, exploration_decay})

	// Train the agent
	agent.Train(false)

	// Test the agent
	agent.Test(true)

Result

    ===MAP LAYOUT===
    S F F F 
    H F H F 
    F F F F 
    H F F G 
    ===END MAP LAYOUT===
    ===QAGENT TEST===
            Action     1: 2
            Action     2: 1
            Action     3: 1
            Action     4: 2
            Action     5: 2
            Action     6: 1
            Test Total Reward: 1.000000
    ===END QAGENT TEST===

Documentation

Overview

The rlearning package contains code for reinforcement learning algorithms and other related functions needed to run them.

As of now, the list of functionality supported is Q-Learning and the DecayExploration policy.

Index

• type Agent
• type DecayExplorationPolicy
  • func (policy *DecayExplorationPolicy) SelectAction(mode string, values *mat.Dense, args []float64) float64
  • func (policy *DecayExplorationPolicy) Update()
• type Policy
  • func NewPolicy(policyType string, args []float64) *Policy
• type QAgent
  • func NewQAgent(env *env.Environment, max_eps int, max_acts int, learning_rate float64, ...) QAgent
  • func (agt *QAgent) SetPolicy(policyType string, args []float64)
  • func (agt *QAgent) Test(info bool) (bool, error)
  • func (agt *QAgent) Train(info bool) (bool, error)

Types

type Agent

type Agent interface {
	SetPolicy(policyType string, args []float64)
	Train(info bool) (bool, error)
	Test(info bool)
}

Agent is a representation for reinforcement learning agents with setting a learning policy, training function, and testing function.

type DecayExplorationPolicy

type DecayExplorationPolicy struct {
	EXPLORATION_RATE  float64
	EXPLORATION_DECAY float64
}

DecayExplorationPolicy implements the Policy interface to represent a decayed exploration policy for reinforcement learning.

func (*DecayExplorationPolicy) SelectAction

func (policy *DecayExplorationPolicy) SelectAction(
	mode string,
	values *mat.Dense,
	args []float64,
) float64

SelectAction returns an action given a state from `args`.

func (*DecayExplorationPolicy) Update

func (policy *DecayExplorationPolicy) Update()

Update updates the `EXPLORATION_RATE` using the `EXPLORATION_DECAY` value.

type Policy

type Policy interface {
	SelectAction(mode string, values *mat.Dense, args []float64) float64
	Update()
}

Policy is a representation for reinforcement learning policies with a function to select an action given values and a mode type. An update function is needed to update any policy parameters.

func NewPolicy

func NewPolicy(policyType string, args []float64) *Policy

NewPolicy returns a policy defined by `policyType` acting as a wrapper initialization function.

type QAgent

type QAgent struct {
	ENV           *env.Environment
	MAX_EPISODES  int
	MAX_ACTIONS   int
	LEARNING_RATE float64
	DISCOUNT      float64
	POLICY        *Policy
	// contains filtered or unexported fields
}

QAgent implements the Agent interface for a Q-Learning agent. It contains the max iterations for training, max actions in each iteration, learning rate, discount rate, the learning policy, and table of Q values.

func NewQAgent

func NewQAgent(
	env *env.Environment,
	max_eps int,
	max_acts int,
	learning_rate float64,
	discount float64,
) QAgent

NewQAgent returns a new QAgent instance given the environment and other parameters.

func (*QAgent) SetPolicy

func (agt *QAgent) SetPolicy(policyType string, args []float64)

SetPolicy sets the learning policy for the QAgent.

func (*QAgent) Test

func (agt *QAgent) Test(info bool) (bool, error)

Test performs one episode on the environment. It returns `true` if it successfully performs the episode. If `info` is `true`, then print statements are given for the episode.

func (*QAgent) Train

func (agt *QAgent) Train(info bool) (bool, error)

Train performs Q-learning on its environment. It returns `true` if it successfully trains. It returns `false` if an error is thrown. If `info` is `true`, then print statements are given for the reward at each episode.